{"nbformat":4,"nbformat_minor":0,"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.7.8"},"colab":{"name":"CAMeL_Tools_Guided_Tour.ipynb","provenance":[],"collapsed_sections":[],"toc_visible":true}},"cells":[{"cell_type":"markdown","metadata":{"id":"JW2oOVmkJgg-"},"source":["# Introduction\n","\n","This notebook provides a quick overview of the functionalities provided by [CAMeL Tools](https://github.com/CAMeL-Lab/camel_tools). While it doesn't cover every component in detail, it does provide a starting point for learning CAMeL Tools and navigating its APIs. For more detailed information please refer to the [CAMeL Tools documentation](https://camel-tools.readthedocs.io/en/latest/index.html)."]},{"cell_type":"markdown","metadata":{"id":"QOwO9Z7VdWf-"},"source":["# Arabic NLP Refresher\n","\n","Before diving into CAMeL Tools, we recommend a quick refresher on Arabic NLP and the challenges it presents. [This webinar](https://wti.kaust.edu.sa/upcoming-events/Machine-Learning-Arabic-NLP-Webinar/ml-webinar-2020-keynote-1) presented by Dr. Nizar Habash provides a short introcuction on Arabic NLP as well as short examples of how CAMeL Tools can be used to solve various problems. A similar presentation of this webinar in Arabic is also available [here](https://www.youtube.com/watch?v=JQrmRvtx_VY).\n","\n","*Note: The examples in this talk may not be compatible with current or newer versions of CAMeL Tools.*\n"]},{"cell_type":"markdown","metadata":{"id":"flp2VnX-14Kz"},"source":["# Installation and Setup\n","\n","The following steps are needed if you want to run the examples in this notebook on Google Colaboratory. If you want to run this notebook on your own machine, please follow the [installation instructions](https://camel-tools.readthedocs.io/en/latest/getting_started.html#installation) instead."]},{"cell_type":"markdown","metadata":{"id":"47wgvS4-6UOD"},"source":["First, we install the CAMeL Tools Python package."]},{"cell_type":"code","metadata":{"id":"WLCWm2nj2LX6"},"source":["%pip install camel-tools"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"7c8-zs9m45B0"},"source":["In order to use all the components provided in CAMeL Tools, we need to install all the datasets required by these components.\n","To do this in Colab, we need to first mount a Google Drive and create a directory where the data will be installed.\n","\n","Run the code below and follow the instructions in the output."]},{"cell_type":"code","metadata":{"id":"rLDEdYgz1OZN"},"source":["from google.colab import drive\n","import os\n","\n","drive.mount('/gdrive')\n","\n","%mkdir /gdrive/MyDrive/camel_tools"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"Fcc0ZaAK5f5F"},"source":["Next, we need to tell CAMeL Tools to install the data in the newly created directory. This will take a couple of minutes to complete.\n","\n","**NOTE:** You will need at least 2.3GB of available space on your Google Drive to install all the CAMeL Tools data."]},{"cell_type":"code","metadata":{"id":"aYsFJl0A5mua"},"source":["os.environ['CAMELTOOLS_DATA'] = '/gdrive/MyDrive/camel_tools'\n","\n","!export | camel_data full"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"0zZvcWRx6tQJ"},"source":["We also provide a lightweight dataset for the Morphology and Disambiguation components **only** that can be installed by calling `camel_data light` instead of `camel_data full`.\n"]},{"cell_type":"markdown","metadata":{"id":"Z-O0aoMczyUj"},"source":["**Once the data has been installed on your Google Drive, you only need to run the following the next time you want to run this notebook.**"]},{"cell_type":"code","metadata":{"id":"9knGLLGg7cnm"},"source":["%pip install camel-tools\n","\n","from google.colab import drive\n","import os\n","\n","drive.mount('/gdrive')\n","os.environ['CAMELTOOLS_DATA'] = '/gdrive/MyDrive/camel_tools'"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"Q2mkVx2f088f"},"source":["# Preprocessing"]},{"cell_type":"markdown","metadata":{"id":"gBNnYjuE9T_6"},"source":["CAMeL Tools provides a suite of preprocessing utilities for cleaning and preparing Arabic text. Some of these may need to be used before passing text to other CAMeL Tools components."]},{"cell_type":"markdown","metadata":{"id":"r-V9yJNU088w"},"source":["## Simple Transliteration\n","\n","Simple (or one-to-one) transliteration is the process of translating each Arabic character to/from a single non-Arabic character mapping (or encoding). The most common of these encoding schemes is the Buckwalter scheme and its variants that map Arabic characters to ASCII counterparts. You can see the various encoding schemes supported in CAMeL Tools and how they map to Arabic script [here](https://camel-tools.readthedocs.io/en/latest/reference/encoding_schemes.html).\n","\n","This has many uses including:\n","\n","- Smaller size of Arabic text on disk\n","- Allow non-Arabic speakers and people not familiar with Arabic script to read Arabic text\n","- Easier debugging by eliminating right-to-left display issues\n","\n","Additionally, some Arabic NLP systems and resources use these transliteration schemes internally.\n","\n","The first utility we provide to convert to/from different encodings is the [`CharMapper`](https://camel-tools.readthedocs.io/en/latest/api/utils/charmap.html). In the example below, we use a `CharMapper` to convert text in Arabic script to a Buckwalter encoding."]},{"cell_type":"code","metadata":{"id":"2jzBCQ8A088w","colab":{"base_uri":"https://localhost:8080/"},"outputId":"19a8acca-4463-4956-cc6f-9970444cb1da"},"source":["from camel_tools.utils.charmap import CharMapper\n","\n","sentence = \"ذهبت إلى المكتبة.\"\n","print(sentence)\n","\n","ar2bw = CharMapper.builtin_mapper('ar2bw')\n","\n","sent_bw = ar2bw(sentence)\n","print(sent_bw)"],"execution_count":null,"outputs":[{"output_type":"stream","text":["ذهبت إلى المكتبة.\n","*hbt <lY Almktbp.\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"89ZesZuWKzIR"},"source":["`CharMapper` is a very flexible component that can be used for many other types of text transformations and can be initialized with any character-to-string mapping. "]},{"cell_type":"markdown","metadata":{"id":"8zHN9cNP088m"},"source":["### Unicode Normalization\n","\n","Text in Python 3 is represented as a series of [Unicode](https://unicode.org/standard/standard.html) characters. Some characters have different variants, for example the character 'ع' can also be represented as 'ﻊ', 'ﻋ', 'ﻌ' depending on where it appears in a word. For all these forms, we would call 'ع' their canonical form. Furthermore, some Unicode characters are a composition of multiple characters and in some cases can comprise entire phrases. For example, the character 'ﰷ' is a single Unicode character that represents the composition of 'ك' and 'ا'. Unicode decomposition is the process of splitting these composed characters into their individual constituents.\n","\n","These variants and composed forms are generally used for display purposes and are problematic in text processing tasks. To convert Unicode text into something more suitable for text processing, we provide the [`normalize_unicode`](https://camel-tools.readthedocs.io/en/latest/api/utils/normalize.html#camel_tools.utils.normalize.normalize_unicode) function.\n","\n","In the example below, we demonstrate a more extreme example of how `normalize_unicode` converts a composed character into its decomposed form.\n"]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"xsR9wuFT088o","outputId":"ddac73bb-c208-4cc9-d8db-f000b3ff4f8d"},"source":["from camel_tools.utils.normalize import normalize_unicode\n","\n","sentence = 'ﷺ'\n","print(sentence)\n","\n","sent_norm = normalize_unicode(sentence)\n","print(sent_norm)"],"execution_count":null,"outputs":[{"output_type":"stream","text":["ﷺ\n","صلى الله عليه وسلم\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"Tbdr5KrDlQIY"},"source":["**NOTE:** It is advised to run this function on all text prior to any further preprocessing or use."]},{"cell_type":"markdown","metadata":{"id":"fUJ8IaJo088q"},"source":["### Orthographic Normalization\n","\n","It is common for Arabic speakers to use shortcuts when typing Arabic text.\n","For example, the different variants of the letter alef ('ا', 'آ', 'أ', 'إ') may be typed as just 'ا'. Some of these substitutions can just be the result of typos. The presence of these variations can cause data sparsity and are usually normalized to a single form. Orthographic normalization is the process of converting letter variants or visually similar letters into a single form.\n","\n","We provide a collection of orthographic normalization functions in CAMeL Tools. The example below demonstrates a few of them."]},{"cell_type":"code","metadata":{"id":"hVN6ZsVV088r","colab":{"base_uri":"https://localhost:8080/"},"outputId":"d5b2950b-00e7-40f3-ba36-80db006027c0"},"source":["from camel_tools.utils.normalize import normalize_alef_maksura_ar\n","from camel_tools.utils.normalize import normalize_alef_ar\n","from camel_tools.utils.normalize import normalize_teh_marbuta_ar\n","\n","sentence = \"هل ذهبت إلى المكتبة؟\"\n","print(sentence)\n","\n","# Normalize alef variants to 'ا'\n","sent_norm = normalize_alef_ar(sentence)\n","print(sent_norm)\n","\n","# Normalize alef maksura 'ى' to yeh 'ي'\n","sent_norm = normalize_alef_maksura_ar(sent_norm)\n","print(sent_norm)\n","\n","# Normalize teh marbuta 'ة' to heh 'ه'\n","sent_norm = normalize_teh_marbuta_ar(sent_norm)\n","print(sent_norm)"],"execution_count":null,"outputs":[{"output_type":"stream","text":["هل ذهبت إلى المكتبة؟\n","هل ذهبت الى المكتبة؟\n","هل ذهبت الي المكتبة؟\n","هل ذهبت الي المكتبه؟\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"rawo1cAJPCOC"},"source":["The example above performs orthographic normalization on Arabic script. We provide variants of these functions for other encoding schemes as well (e.g. Buckwalter, Safe Buckwalter, etc). See [here](https://camel-tools.readthedocs.io/en/latest/api/utils/normalize.html) for more information.\n"]},{"cell_type":"markdown","metadata":{"id":"vhD5hMgA088t"},"source":["## Dediacritization\n","\n","Dediacritization is the process of removing Arabic diacritical marks. Diacritics increase data sparsity and so most Arabic NLP techniques ignore them. The example below shows how diacritics can be removed from Arabic text using the [`dediac_ar`](https://camel-tools.readthedocs.io/en/latest/api/utils/dediac.html#camel_tools.utils.dediac.dediac_ar) function:"]},{"cell_type":"code","metadata":{"id":"Np_JWmFq088u","colab":{"base_uri":"https://localhost:8080/"},"outputId":"bfd3de1a-00d5-473d-8967-0d99293fae1d"},"source":["from camel_tools.utils.dediac import dediac_ar\n","\n","sentence = \"هَلْ ذَهَبْتَ إِلَى المَكْتَبَةِ؟\"\n","print(sentence)\n","\n","sent_dediac = dediac_ar(sentence)\n","print(sent_dediac)"],"execution_count":null,"outputs":[{"output_type":"stream","text":["هَلْ ذَهَبْتَ إِلَى المَكْتَبَةِ؟\n","هل ذهبت إلى المكتبة؟\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"i2I9gP03CHrR"},"source":["We provide dediacritization functions for other encoding schemes (eg. Buckwalter, Safe Buckwalter, etc) which you can learn more about [here](https://camel-tools.readthedocs.io/en/latest/api/utils/dediac.html).\n"]},{"cell_type":"markdown","metadata":{"id":"wQw096Uw088u"},"source":["## Word Tokenization"]},{"cell_type":"markdown","metadata":{"id":"7-Dr0J4W_-Bp"},"source":["Some CAMeL Tools components expect text to be pretokenized by whitespace and punctuation. This means that an input sentence should be an array of words and punctuation instead of a single string. This is to preserve the alignment of inputs and outputs.\n","\n","Python does provide the `split()` method to tokenize words by whitespace but it doesn't seperate punctuation from words. For example:"]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"zqLIIGGim5nm","outputId":"bca8345e-38d1-4de2-dbcf-77988d1ddc4d"},"source":["sentence = \"هَلْ ذَهَبْتَ إِلَى المَكْتَبَةِ؟\"\n","print(sentence)\n","\n","sent_split = sentence.split()\n","print(sent_split)"],"execution_count":null,"outputs":[{"output_type":"stream","text":["هَلْ ذَهَبْتَ إِلَى المَكْتَبَةِ؟\n","['هَلْ', 'ذَهَبْتَ', 'إِلَى', 'المَكْتَبَةِ؟']\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"6Sju7VQ6m3h5"},"source":["To split by whitespace and seperate punctuation, we currently provide the utility function [`simple_word_tokenize`](https://camel-tools.readthedocs.io/en/latest/api/tokenizers/word.html#camel_tools.tokenizers.word.simple_word_tokenize). The example below is similar to the one above, but this time we use `simple_word_tokenize()` instead of `split()`."]},{"cell_type":"code","metadata":{"id":"WxbPNRCU088v","colab":{"base_uri":"https://localhost:8080/"},"outputId":"9d7c4f48-75fd-4378-effb-e333e2d58738"},"source":["from camel_tools.tokenizers.word import simple_word_tokenize\n","\n","sentence = \"هَلْ ذَهَبْتَ إِلَى المَكْتَبَةِ؟\"\n","print(sentence)\n","\n","sent_split = simple_word_tokenize(sentence)\n","print(sent_split)"],"execution_count":null,"outputs":[{"output_type":"stream","text":["هَلْ ذَهَبْتَ إِلَى المَكْتَبَةِ؟\n","['هَلْ', 'ذَهَبْتَ', 'إِلَى', 'المَكْتَبَةِ', '؟']\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"7AMRqZE9A8tj"},"source":["**NOTE:** `simple_word_tokenize`, as the name suggests is very simple and does not do anything fancy. It will seperate all punctuation regardless of context (ie. hashtags, emails, abbreviations, etc.)"]},{"cell_type":"markdown","metadata":{"id":"Nv4xaPCX088y"},"source":["# Morphology"]},{"cell_type":"markdown","metadata":{"id":"-7zdJFwRDUQ1"},"source":["CAMeL Tools provides a powerful morphological analysis, generation and reinflection system. This system is powered by [morphological databases](https://camel-tools.readthedocs.io/en/latest/api/morphology/database.html#databases) that are comprised of lexicons and compatibility tables. See [here](https://www.aclweb.org/anthology/W18-5816.pdf) for more details."]},{"cell_type":"markdown","metadata":{"id":"si9YcVbY088z"},"source":["## Analysis\n"]},{"cell_type":"markdown","metadata":{"id":"U7JXVSeA8-rr"},"source":["Morphological analysis is the process of generating all possible readings (analyses) of a given word out of context. All analyses are generated from the undiacritized form of the input word. Each of these analyses is defined by a set lexical and morphological features. "]},{"cell_type":"code","metadata":{"id":"MtzSgEBl088z","colab":{"base_uri":"https://localhost:8080/"},"outputId":"b11678d4-569c-40ee-dea7-9e6873a8427d"},"source":["from camel_tools.morphology.database import MorphologyDB\n","from camel_tools.morphology.analyzer import Analyzer\n","\n","# First, we need to load a morphological database.\n","# Here, we load the default database which is used for analyzing\n","# Modern Standard Arabic. \n","db = MorphologyDB.builtin_db()\n","\n","analyzer = Analyzer(db)\n","\n","analyses = analyzer.analyze('موظف')\n","\n","for analysis in analyses:\n","    print(analysis, '\\n')"],"execution_count":null,"outputs":[{"output_type":"stream","text":["{'diac': 'مُوَظَّف', 'lex': 'مُوَظَّف_2', 'bw': 'مُوَظَّف/ADJ', 'gloss': 'employed;hired', 'pos': 'adj', 'prc3': '0', 'prc2': '0', 'prc1': '0', 'prc0': '0', 'per': 'na', 'asp': 'na', 'vox': 'na', 'mod': 'na', 'stt': 'i', 'cas': 'u', 'enc0': '0', 'rat': 'n', 'source': 'lex', 'form_gen': 'm', 'form_num': 's', 'pattern': 'مُوَ2َّ3', 'root': '#.ظ.ف', 'catib6': 'NOM', 'ud': 'ADJ', 'd1seg': 'مُوَظَّف', 'd1tok': 'مُوَظَّف', 'atbseg': 'مُوَظَّف', 'd3seg': 'مُوَظَّف', 'd2seg': 'مُوَظَّف', 'd2tok': 'مُوَظَّف', 'atbtok': 'مُوَظَّف', 'd3tok': 'مُوَظَّف', 'bwtok': 'مُوَظَّف', 'pos_lex_logprob': -5.400551, 'caphi': 'm_u_w_a_dh._dh._a_f', 'pos_logprob': -0.9868824, 'gen': 'm', 'lex_logprob': -5.400551, 'num': 's', 'stem': 'مُوَظَّف', 'stemgloss': 'employed;hired', 'stemcat': 'N-ap'} \n","\n","{'diac': 'مُوَظَّفِ', 'lex': 'مُوَظَّف_2', 'bw': 'مُوَظَّف/ADJ+ِ/CASE_DEF_GEN', 'gloss': 'employed;hired+[def.gen.]', 'pos': 'adj', 'prc3': '0', 'prc2': '0', 'prc1': '0', 'prc0': '0', 'per': 'na', 'asp': 'na', 'vox': 'na', 'mod': 'na', 'stt': 'c', 'cas': 'g', 'enc0': '0', 'rat': 'n', 'source': 'lex', 'form_gen': 'm', 'form_num': 's', 'pattern': 'مُوَ2َّ3ِ', 'root': '#.ظ.ف', 'catib6': 'NOM', 'ud': 'ADJ', 'd1seg': 'مُوَظَّفِ', 'd1tok': 'مُوَظَّفِ', 'atbseg': 'مُوَظَّفِ', 'd3seg': 'مُوَظَّفِ', 'd2seg': 'مُوَظَّفِ', 'd2tok': 'مُوَظَّفِ', 'atbtok': 'مُوَظَّفِ', 'd3tok': 'مُوَظَّفِ', 'bwtok': 'مُوَظَّف_+ِ', 'pos_lex_logprob': -5.400551, 'caphi': 'm_u_w_a_dh._dh._a_f_i', 'pos_logprob': -0.9868824, 'gen': 'm', 'lex_logprob': -5.400551, 'num': 's', 'stem': 'مُوَظَّف', 'stemgloss': 'employed;hired', 'stemcat': 'N-ap'} \n","\n","{'diac': 'مُوَظَّفَ', 'lex': 'مُوَظَّف_2', 'bw': 'مُوَظَّف/ADJ+َ/CASE_DEF_ACC', 'gloss': 'employed;hired+[def.acc.]', 'pos': 'adj', 'prc3': '0', 'prc2': '0', 'prc1': '0', 'prc0': '0', 'per': 'na', 'asp': 'na', 'vox': 'na', 'mod': 'na', 'stt': 'c', 'cas': 'a', 'enc0': '0', 'rat': 'n', 'source': 'lex', 'form_gen': 'm', 'form_num': 's', 'pattern': 'مُوَ2َّ3َ', 'root': '#.ظ.ف', 'catib6': 'NOM', 'ud': 'ADJ', 'd1seg': 'مُوَظَّفَ', 'd1tok': 'مُوَظَّفَ', 'atbseg': 'مُوَظَّفَ', 'd3seg': 'مُوَظَّفَ', 'd2seg': 'مُوَظَّفَ', 'd2tok': 'مُوَظَّفَ', 'atbtok': 'مُوَظَّفَ', 'd3tok': 'مُوَظَّفَ', 'bwtok': 'مُوَظَّف_+َ', 'pos_lex_logprob': -5.400551, 'caphi': 'm_u_w_a_dh._dh._a_f_a', 'pos_logprob': -0.9868824, 'gen': 'm', 'lex_logprob': -5.400551, 'num': 's', 'stem': 'مُوَظَّف', 'stemgloss': 'employed;hired', 'stemcat': 'N-ap'} \n","\n","{'diac': 'مُوَظَّفُ', 'lex': 'مُوَظَّف_2', 'bw': 'مُوَظَّف/ADJ+ُ/CASE_DEF_NOM', 'gloss': 'employed;hired+[def.nom.]', 'pos': 'adj', 'prc3': '0', 'prc2': '0', 'prc1': '0', 'prc0': '0', 'per': 'na', 'asp': 'na', 'vox': 'na', 'mod': 'na', 'stt': 'c', 'cas': 'n', 'enc0': '0', 'rat': 'n', 'source': 'lex', 'form_gen': 'm', 'form_num': 's', 'pattern': 'مُوَ2َّ3ُ', 'root': '#.ظ.ف', 'catib6': 'NOM', 'ud': 'ADJ', 'd1seg': 'مُوَظَّفُ', 'd1tok': 'مُوَظَّفُ', 'atbseg': 'مُوَظَّفُ', 'd3seg': 'مُوَظَّفُ', 'd2seg': 'مُوَظَّفُ', 'd2tok': 'مُوَظَّفُ', 'atbtok': 'مُوَظَّفُ', 'd3tok': 'مُوَظَّفُ', 'bwtok': 'مُوَظَّف_+ُ', 'pos_lex_logprob': -5.400551, 'caphi': 'm_u_w_a_dh._dh._a_f_u', 'pos_logprob': -0.9868824, 'gen': 'm', 'lex_logprob': -5.400551, 'num': 's', 'stem': 'مُوَظَّف', 'stemgloss': 'employed;hired', 'stemcat': 'N-ap'} \n","\n","{'diac': 'مُوَظَّفٌ', 'lex': 'مُوَظَّف_2', 'bw': 'مُوَظَّف/ADJ+ٌ/CASE_INDEF_NOM', 'gloss': 'employed;hired+[indef.nom.]', 'pos': 'adj', 'prc3': '0', 'prc2': '0', 'prc1': '0', 'prc0': '0', 'per': 'na', 'asp': 'na', 'vox': 'na', 'mod': 'na', 'stt': 'i', 'cas': 'n', 'enc0': '0', 'rat': 'n', 'source': 'lex', 'form_gen': 'm', 'form_num': 's', 'pattern': 'مُوَ2َّ3ٌ', 'root': '#.ظ.ف', 'catib6': 'NOM', 'ud': 'ADJ', 'd1seg': 'مُوَظَّفٌ', 'd1tok': 'مُوَظَّفٌ', 'atbseg': 'مُوَظَّفٌ', 'd3seg': 'مُوَظَّفٌ', 'd2seg': 'مُوَظَّفٌ', 'd2tok': 'مُوَظَّفٌ', 'atbtok': 'مُوَظَّفٌ', 'd3tok': 'مُوَظَّفٌ', 'bwtok': 'مُوَظَّف_+ٌ', 'pos_lex_logprob': -5.400551, 'caphi': 'm_u_w_a_dh._dh._a_f_u_n', 'pos_logprob': -0.9868824, 'gen': 'm', 'lex_logprob': -5.400551, 'num': 's', 'stem': 'مُوَظَّف', 'stemgloss': 'employed;hired', 'stemcat': 'N-ap'} \n","\n","{'diac': 'مُوَظَّفٍ', 'lex': 'مُوَظَّف_2', 'bw': 'مُوَظَّف/ADJ+ٍ/CASE_INDEF_GEN', 'gloss': 'employed;hired+[indef.gen.]', 'pos': 'adj', 'prc3': '0', 'prc2': '0', 'prc1': '0', 'prc0': '0', 'per': 'na', 'asp': 'na', 'vox': 'na', 'mod': 'na', 'stt': 'i', 'cas': 'g', 'enc0': '0', 'rat': 'n', 'source': 'lex', 'form_gen': 'm', 'form_num': 's', 'pattern': 'مُوَ2َّ3ٍ', 'root': '#.ظ.ف', 'catib6': 'NOM', 'ud': 'ADJ', 'd1seg': 'مُوَظَّفٍ', 'd1tok': 'مُوَظَّفٍ', 'atbseg': 'مُوَظَّفٍ', 'd3seg': 'مُوَظَّفٍ', 'd2seg': 'مُوَظَّفٍ', 'd2tok': 'مُوَظَّفٍ', 'atbtok': 'مُوَظَّفٍ', 'd3tok': 'مُوَظَّفٍ', 'bwtok': 'مُوَظَّف_+ٍ', 'pos_lex_logprob': -5.400551, 'caphi': 'm_u_w_a_dh._dh._a_f_i_n', 'pos_logprob': -0.9868824, 'gen': 'm', 'lex_logprob': -5.400551, 'num': 's', 'stem': 'مُوَظَّف', 'stemgloss': 'employed;hired', 'stemcat': 'N-ap'} \n","\n","{'diac': 'مُوَظَّف', 'lex': 'مُوَظَّف_1', 'bw': 'مُوَظَّف/NOUN', 'gloss': 'employee', 'pos': 'noun', 'prc3': '0', 'prc2': '0', 'prc1': '0', 'prc0': '0', 'per': 'na', 'asp': 'na', 'vox': 'na', 'mod': 'na', 'stt': 'i', 'cas': 'u', 'enc0': '0', 'rat': 'r', 'source': 'lex', 'form_gen': 'm', 'form_num': 's', 'pattern': 'مُوَ2َّ3', 'root': '#.ظ.ف', 'catib6': 'NOM', 'ud': 'NOUN', 'd1seg': 'مُوَظَّف', 'd1tok': 'مُوَظَّف', 'atbseg': 'مُوَظَّف', 'd3seg': 'مُوَظَّف', 'd2seg': 'مُوَظَّف', 'd2tok': 'مُوَظَّف', 'atbtok': 'مُوَظَّف', 'd3tok': 'مُوَظَّف', 'bwtok': 'مُوَظَّف', 'pos_lex_logprob': -3.747339, 'caphi': 'm_u_w_a_dh._dh._a_f', 'pos_logprob': -0.4344233, 'gen': 'm', 'lex_logprob': -3.747339, 'num': 's', 'stem': 'مُوَظَّف', 'stemgloss': 'employee', 'stemcat': 'Nall'} \n","\n","{'diac': 'مُوَظَّفِ', 'lex': 'مُوَظَّف_1', 'bw': 'مُوَظَّف/NOUN+ِ/CASE_DEF_GEN', 'gloss': 'employee+[def.gen.]', 'pos': 'noun', 'prc3': '0', 'prc2': '0', 'prc1': '0', 'prc0': '0', 'per': 'na', 'asp': 'na', 'vox': 'na', 'mod': 'na', 'stt': 'c', 'cas': 'g', 'enc0': '0', 'rat': 'r', 'source': 'lex', 'form_gen': 'm', 'form_num': 's', 'pattern': 'مُوَ2َّ3ِ', 'root': '#.ظ.ف', 'catib6': 'NOM', 'ud': 'NOUN', 'd1seg': 'مُوَظَّفِ', 'd1tok': 'مُوَظَّفِ', 'atbseg': 'مُوَظَّفِ', 'd3seg': 'مُوَظَّفِ', 'd2seg': 'مُوَظَّفِ', 'd2tok': 'مُوَظَّفِ', 'atbtok': 'مُوَظَّفِ', 'd3tok': 'مُوَظَّفِ', 'bwtok': 'مُوَظَّف_+ِ', 'pos_lex_logprob': -3.747339, 'caphi': 'm_u_w_a_dh._dh._a_f_i', 'pos_logprob': -0.4344233, 'gen': 'm', 'lex_logprob': -3.747339, 'num': 's', 'stem': 'مُوَظَّف', 'stemgloss': 'employee', 'stemcat': 'Nall'} \n","\n","{'diac': 'مُوَظَّفَ', 'lex': 'مُوَظَّف_1', 'bw': 'مُوَظَّف/NOUN+َ/CASE_DEF_ACC', 'gloss': 'employee+[def.acc.]', 'pos': 'noun', 'prc3': '0', 'prc2': '0', 'prc1': '0', 'prc0': '0', 'per': 'na', 'asp': 'na', 'vox': 'na', 'mod': 'na', 'stt': 'c', 'cas': 'a', 'enc0': '0', 'rat': 'r', 'source': 'lex', 'form_gen': 'm', 'form_num': 's', 'pattern': 'مُوَ2َّ3َ', 'root': '#.ظ.ف', 'catib6': 'NOM', 'ud': 'NOUN', 'd1seg': 'مُوَظَّفَ', 'd1tok': 'مُوَظَّفَ', 'atbseg': 'مُوَظَّفَ', 'd3seg': 'مُوَظَّفَ', 'd2seg': 'مُوَظَّفَ', 'd2tok': 'مُوَظَّفَ', 'atbtok': 'مُوَظَّفَ', 'd3tok': 'مُوَظَّفَ', 'bwtok': 'مُوَظَّف_+َ', 'pos_lex_logprob': -3.747339, 'caphi': 'm_u_w_a_dh._dh._a_f_a', 'pos_logprob': -0.4344233, 'gen': 'm', 'lex_logprob': -3.747339, 'num': 's', 'stem': 'مُوَظَّف', 'stemgloss': 'employee', 'stemcat': 'Nall'} \n","\n","{'diac': 'مُوَظَّفُ', 'lex': 'مُوَظَّف_1', 'bw': 'مُوَظَّف/NOUN+ُ/CASE_DEF_NOM', 'gloss': 'employee+[def.nom.]', 'pos': 'noun', 'prc3': '0', 'prc2': '0', 'prc1': '0', 'prc0': '0', 'per': 'na', 'asp': 'na', 'vox': 'na', 'mod': 'na', 'stt': 'c', 'cas': 'n', 'enc0': '0', 'rat': 'r', 'source': 'lex', 'form_gen': 'm', 'form_num': 's', 'pattern': 'مُوَ2َّ3ُ', 'root': '#.ظ.ف', 'catib6': 'NOM', 'ud': 'NOUN', 'd1seg': 'مُوَظَّفُ', 'd1tok': 'مُوَظَّفُ', 'atbseg': 'مُوَظَّفُ', 'd3seg': 'مُوَظَّفُ', 'd2seg': 'مُوَظَّفُ', 'd2tok': 'مُوَظَّفُ', 'atbtok': 'مُوَظَّفُ', 'd3tok': 'مُوَظَّفُ', 'bwtok': 'مُوَظَّف_+ُ', 'pos_lex_logprob': -3.747339, 'caphi': 'm_u_w_a_dh._dh._a_f_u', 'pos_logprob': -0.4344233, 'gen': 'm', 'lex_logprob': -3.747339, 'num': 's', 'stem': 'مُوَظَّف', 'stemgloss': 'employee', 'stemcat': 'Nall'} \n","\n","{'diac': 'مُوَظَّفٌ', 'lex': 'مُوَظَّف_1', 'bw': 'مُوَظَّف/NOUN+ٌ/CASE_INDEF_NOM', 'gloss': 'employee+[indef.nom.]', 'pos': 'noun', 'prc3': '0', 'prc2': '0', 'prc1': '0', 'prc0': '0', 'per': 'na', 'asp': 'na', 'vox': 'na', 'mod': 'na', 'stt': 'i', 'cas': 'n', 'enc0': '0', 'rat': 'r', 'source': 'lex', 'form_gen': 'm', 'form_num': 's', 'pattern': 'مُوَ2َّ3ٌ', 'root': '#.ظ.ف', 'catib6': 'NOM', 'ud': 'NOUN', 'd1seg': 'مُوَظَّفٌ', 'd1tok': 'مُوَظَّفٌ', 'atbseg': 'مُوَظَّفٌ', 'd3seg': 'مُوَظَّفٌ', 'd2seg': 'مُوَظَّفٌ', 'd2tok': 'مُوَظَّفٌ', 'atbtok': 'مُوَظَّفٌ', 'd3tok': 'مُوَظَّفٌ', 'bwtok': 'مُوَظَّف_+ٌ', 'pos_lex_logprob': -3.747339, 'caphi': 'm_u_w_a_dh._dh._a_f_u_n', 'pos_logprob': -0.4344233, 'gen': 'm', 'lex_logprob': -3.747339, 'num': 's', 'stem': 'مُوَظَّف', 'stemgloss': 'employee', 'stemcat': 'Nall'} \n","\n","{'diac': 'مُوَظَّفٍ', 'lex': 'مُوَظَّف_1', 'bw': 'مُوَظَّف/NOUN+ٍ/CASE_INDEF_GEN', 'gloss': 'employee+[indef.gen.]', 'pos': 'noun', 'prc3': '0', 'prc2': '0', 'prc1': '0', 'prc0': '0', 'per': 'na', 'asp': 'na', 'vox': 'na', 'mod': 'na', 'stt': 'i', 'cas': 'g', 'enc0': '0', 'rat': 'r', 'source': 'lex', 'form_gen': 'm', 'form_num': 's', 'pattern': 'مُوَ2َّ3ٍ', 'root': '#.ظ.ف', 'catib6': 'NOM', 'ud': 'NOUN', 'd1seg': 'مُوَظَّفٍ', 'd1tok': 'مُوَظَّفٍ', 'atbseg': 'مُوَظَّفٍ', 'd3seg': 'مُوَظَّفٍ', 'd2seg': 'مُوَظَّفٍ', 'd2tok': 'مُوَظَّفٍ', 'atbtok': 'مُوَظَّفٍ', 'd3tok': 'مُوَظَّفٍ', 'bwtok': 'مُوَظَّف_+ٍ', 'pos_lex_logprob': -3.747339, 'caphi': 'm_u_w_a_dh._dh._a_f_i_n', 'pos_logprob': -0.4344233, 'gen': 'm', 'lex_logprob': -3.747339, 'num': 's', 'stem': 'مُوَظَّف', 'stemgloss': 'employee', 'stemcat': 'Nall'} \n","\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"X3KV4vNDDtG8"},"source":["Let's examine just one of the output analyses generated from another word ('وسيكتبونها') to get a better idea of the verious morphological and lexical features that we provide. We provide a [handy reference page](https://camel-tools.readthedocs.io/en/latest/reference/camel_morphology_features.html) that explains what these features are and what their values mean."]},{"cell_type":"markdown","metadata":{"id":"EcfmX4eZ088z"},"source":["```\n","diac: وَسَيَكْتُبُونَها\n","lex: كَتَب-u_1\n","bw: وَ/CONJ+سَ/FUT_PART+يَ/IV3MP+كْتُب/IV+ُونَ/IVSUFF_SUBJ:MP_MOOD:I+ها/IVSUFF_DO:3FS\n","gloss: and_+_will_+_they_(people)+write+it;them;her\n","pos: verb\n","prc3: 0\n","prc2: wa_conj\n","prc1: sa_fut\n","prc0: 0\n","per: 3\n","asp: i\n","vox: a\n","mod: i\n","stt: na\n","cas: na\n","enc0: 3fs_dobj\n","rat: n\n","source: lex\n","form_gen: m\n","form_num: p\n","pattern: وَسَيَ1ْ2ُ3ُونَها\n","root: ك.ت.ب\n","catib6: PRT+PRT+VRB+NOM\n","ud: CONJ+AUX+VERB+PRON\n","d1seg: وَ+_سَيَكْتُبُونَها\n","d1tok: وَ+_سَيَكْتُبُونَها\n","atbseg: وَ+_سَ+_يَكْتُبُونَ_+ها\n","d3seg: وَ+_سَ+_يَكْتُبُونَ_+ها\n","d2seg: وَ+_سَ+_يَكْتُبُونَها\n","d2tok: وَ+_سَ+_يَكْتُبُونَها\n","atbtok: وَ+_سَ+_يَكْتُبُونَ_+ها\n","d3tok: وَ+_سَ+_يَكْتُبُونَ_+ها\n","bwtok: وَ+_سَ+_يَ+_كْتُب_+ُونَ_+ها\n","pos_lex_logprob: -3.648503\n","caphi: w_a_s_a_y_a_k_t_u_b_uu_n_a_h_aa\n","pos_logprob: -1.023208\n","gen: m\n","lex_logprob: -3.648503\n","num: p\n","stem: كْتُب\n","stemgloss: write\n","stemcat: IV\n","```"]},{"cell_type":"markdown","metadata":{"id":"UfO9gxQy0880"},"source":["## Generation\n","\n","Generation is the process of inflecting a lemma\n","for a set of morphological features. The example below generates all the possible inflections for the lemma 'مُوَظَّف' (employee) that are masculine plural nouns."]},{"cell_type":"code","metadata":{"id":"ESM-zdQ50881","colab":{"base_uri":"https://localhost:8080/"},"outputId":"207b8b89-f539-4faf-f034-eabe665bf4c8"},"source":["from camel_tools.morphology.database import MorphologyDB\n","from camel_tools.morphology.generator import Generator\n","\n","# We need to indicate that the database we are loading will be\n","# used for generation.\n","db = MorphologyDB.builtin_db(flags='g')\n","\n","generator = Generator(db)\n","\n","lemma = 'مُوَظَّف'\n","features = {\n","    'pos': 'noun',\n","    'gen': 'm',\n","    'num': 'p'\n","}\n","\n","analyses = generator.generate(lemma, features)\n","\n","# Extract and print unique diacritizations from generated analyses\n","for diac in set([a['diac'] for a in analyses]):\n","    print(diac)"],"execution_count":null,"outputs":[{"output_type":"stream","text":["مُوَظَّفِينَ\n","مُوَظَّفُو\n","مُوَظَّفُونَ\n","مُوَظَّفِي\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"kHa3zXmnFl15"},"source":["**NOTE:** `'pos'` is the only *required* feature that needs to be specified."]},{"cell_type":"markdown","metadata":{"id":"0Z45XwXQ0881"},"source":["## Reinflection\n","\n","Reinflection is the process of converting a given word in any form to a different form (i.e. tense, gender, etc). The CAMeL Tools reinflector works similar to the generator except that the word doesn't have to be a lemma and it is not have to be restricted to a specific `'pos'`. The example below reinflects the word 'شوارع' (streets) into its dual form with a proclitic 'ب'."]},{"cell_type":"code","metadata":{"id":"gAffYSc60881","colab":{"base_uri":"https://localhost:8080/"},"outputId":"6d0f7d3d-0c63-4648-a21b-23ebec08cb6d"},"source":["from camel_tools.morphology.database import MorphologyDB\n","from camel_tools.morphology.reinflector import Reinflector\n","\n","# We need to indicate that the database we are loading will be\n","# used for reinflection.\n","db = MorphologyDB.builtin_db(flags='r')\n","\n","reinflector = Reinflector(db)\n","\n","word = 'شوارع'\n","features = {\n","    'num': 'd',\n","    'prc1': 'bi_prep'\n","}\n","\n","analyses = reinflector.reinflect(word, features)\n","\n","# Extract and print unique diacritizations from reinflected analyses\n","for diac in set([a['diac'] for a in analyses]):\n","    print(diac)"],"execution_count":null,"outputs":[{"output_type":"stream","text":["بِشارِعَيْنِ\n","بِشارِعَيْ\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"XYTXsWdA0882"},"source":["# Disambiguation"]},{"cell_type":"markdown","metadata":{"id":"AQgWp4XiGIUT"},"source":["Disambiguation is the process of determining what is the most likely analysis of a word in a given context. In CAMeL Tools, disambiguation is the backbone for many Arabic NLP tasks such as diacritization, POS tagging and morphological tokenization. At the moment, we provide a disambiguator that uses a Maximum Likelihood Estimation model, [`MLEDisambiguator`](https://camel-tools.readthedocs.io/en/latest/api/disambig/mle.html#camel_tools.disambig.mle.MLEDisambiguator). More advanced disambiguation techniques will be added to CAMeL Tools in the near future.\n","\n","In the example below, we use a disambiguator to compute the most likely diacritization, POS tag, and lemma for all words in a given sentence."]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"XRic4ub10888","executionInfo":{"status":"ok","timestamp":1612959390260,"user_tz":-240,"elapsed":5274,"user":{"displayName":"Ossama Obeid","photoUrl":"","userId":"06767072059928592512"}},"outputId":"d8cb1d83-ee37-4564-8443-639f48186f03"},"source":["from camel_tools.tokenizers.word import simple_word_tokenize\n","from camel_tools.disambig.mle import MLEDisambiguator\n","\n","mle = MLEDisambiguator.pretrained()\n","\n","# The disambiguator expects pre-tokenized text\n","sentence = simple_word_tokenize('نجح بايدن في الانتخابات')\n","\n","disambig = mle.disambiguate(sentence)\n","\n","# For each disambiguated word d in disambig, d.analyses is a list of analyses\n","# sorted from most likely to least likely. Therefore, d.analyses[0] would\n","# be the most likely analysis for a given word. Below we extract different\n","# features from the top analysis of each disambiguated word into seperate lists.\n","diacritized = [d.analyses[0].analysis['diac'] for d in disambig]\n","pos_tags = [d.analyses[0].analysis['pos'] for d in disambig]\n","lemmas = [d.analyses[0].analysis['lex'] for d in disambig]\n","\n","# Print the combined feature values extracted above\n","for triplet in zip(diacritized, pos_tags, lemmas):\n","    print(triplet)\n"],"execution_count":7,"outputs":[{"output_type":"stream","text":["('نَجَحَ', 'verb', 'نَجَح-a_1')\n","('بايدن', 'noun_prop', 'بايدن_0')\n","('فِي', 'prep', 'فِي_1')\n","('الاِنْتِخاباتِ', 'noun', 'ٱِنْتِخاب_1')\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"kMTRdw4g0888"},"source":["# Tagging"]},{"cell_type":"markdown","metadata":{"id":"YoYy16RApKQX"},"source":["The disambiguation example above is not very readable due to the amount of list and dictionary dereferencing just to extract a feature.\n","Additionally, for general tagging applications, extracting a feature from the top analysis is not sufficient.\n","There are some edge cases that need to be handled. For example, a word could return no analyses at all if the word isn't in the morphological database. We would then need to check the result of each disambiguated word to see if an analysis exists and then output some default value if not. Furthermore, the default value would differ depending on what feature we are tagging.\n","\n","To handle these edge cases and to simplify the usage of a disambiguator, we provide the [`DefaultTagger`](https://camel-tools.readthedocs.io/en/latest/api/tagger/default.html#camel_tools.tagger.default.DefaultTagger) class which wraps a disambiguator and handles edge cases by providing sensible default values. The example below shows "]},{"cell_type":"code","metadata":{"id":"6ToGeNYB0889","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1612959410631,"user_tz":-240,"elapsed":6168,"user":{"displayName":"Ossama Obeid","photoUrl":"","userId":"06767072059928592512"}},"outputId":"359ec837-6742-464e-e13d-cf67e9a18694"},"source":["from camel_tools.tokenizers.word import simple_word_tokenize\n","from camel_tools.disambig.mle import MLEDisambiguator\n","from camel_tools.tagger.default import DefaultTagger\n","\n","mle = MLEDisambiguator.pretrained()\n","tagger = DefaultTagger(mle, 'pos')\n","\n","# The tagger expects pre-tokenized text\n","sentence = simple_word_tokenize('نجح بايدن في الانتخابات')\n","\n","pos_tags = tagger.tag(sentence)\n","\n","print(pos_tags)"],"execution_count":8,"outputs":[{"output_type":"stream","text":["['verb', 'noun_prop', 'prep', 'noun']\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"7uuXeQjd0889"},"source":["# Morphological Tokenization"]},{"cell_type":"markdown","metadata":{"id":"KmeKhpPVsI69"},"source":["Previously we discussed word tokenization as the process of splitting sentences by whitspace and seperating punctuation. Another type of tokenization is morphological tokenization whereby Arabic words are split into component prefixes, stems, and suffixes.\n","\n","There are also different tokenization schemes for Arabic each having different use cases.\n","\n","We provide the [`MorphologicalTokenizer`](https://camel-tools.readthedocs.io/en/latest/api/tokenizers/morphological.html#camel_tools.tokenizers.morphological.MorphologicalTokenizer) class to tokenize words in different schemes. It behaves very much like the `DefaultTagger` in that it uses a disambiguator to first disambiguate words and then extracts a particular tokenization feature, but it has the following differences:\n","\n","- While the `DefaultTagger` produces exactly one output for each input word, the `MorphologicalTokenizer` might produce multiple output tokens.\n","-  The `MorphologicalTokenizer` can be configured to produce diacritized and undiacritized output."]},{"cell_type":"code","metadata":{"id":"Upf_eeVE0889","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1612959448103,"user_tz":-240,"elapsed":5991,"user":{"displayName":"Ossama Obeid","photoUrl":"","userId":"06767072059928592512"}},"outputId":"b676e705-4d91-4797-b47d-c21012be279a"},"source":["from camel_tools.tokenizers.word import simple_word_tokenize\n","from camel_tools.disambig.mle import MLEDisambiguator\n","from camel_tools.tokenizers.morphological import MorphologicalTokenizer\n","\n","# The tokenizer expects pre-tokenized text\n","sentence = simple_word_tokenize('فتنفست الصعداء')\n","print(sentence)\n","\n","# Load a pretrained disambiguator to use with a tokenizer\n","mle = MLEDisambiguator.pretrained('calima-msa-r13')\n","\n","# Without providing additional arguments, the tokenizer will output undiacritized\n","# morphological tokens for each input word delimited by an underscore.\n","tokenizer = MorphologicalTokenizer(mle, scheme='d3tok')\n","tokens = tokenizer.tokenize(sentence)\n","print(tokens)\n","\n","# By specifying `split=True`, the morphological tokens are output as seperate\n","# strings.\n","tokenizer = MorphologicalTokenizer(mle, scheme='d3tok', split=True)\n","tokens = tokenizer.tokenize(sentence)\n","print(tokens)\n","\n","# We can output diacritized tokens by setting `diac=True`\n","tokenizer = MorphologicalTokenizer(mle, scheme='d3tok', split=True, diac=True)\n","tokens = tokenizer.tokenize(sentence)\n","print(tokens)"],"execution_count":9,"outputs":[{"output_type":"stream","text":["['فتنفست', 'الصعداء']\n","['ف+_تنفست', 'ال+_صعداء']\n","['ف+', 'تنفست', 'ال+', 'صعداء']\n","['فَ+', 'تَنَفَّسْتُ', 'ال+', 'صُعَداءَ']\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"MaxVSNFH5QHi"},"source":["We can see a list of supported tokenization schemes by calling the `MorphologicalTokenizer.scheme_set()`.\n","\n","**NOTE:** Different morphological databases will have differing support for each of these tokenization schemes."]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"Pb9dvHok43XH","outputId":"97005bee-cd92-4bec-a025-9ec9ad9748c1"},"source":["print(list(MorphologicalTokenizer.scheme_set()))"],"execution_count":null,"outputs":[{"output_type":"stream","text":["['d2seg', 'd1seg', 'atbtok', 'd1tok', 'atbseg', 'd3seg', 'd2tok', 'd3tok', 'bwtok']\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"e7hm9JjJ40u8"},"source":["Try running the previous example using a different scheme than 'd3tok' to see how the output tokenization might vary."]},{"cell_type":"markdown","metadata":{"id":"-GMeBGri088-"},"source":["# Dialect Identification\n","\n","We provide a pretrained dialect identification system that can distinguish between 25 city dialects as well as Modern Standard Arabic. The model can be accessed using the [`DialectIdentifier`](https://camel-tools.readthedocs.io/en/latest/api/dialectid.html#camel_tools.dialectid.DialectIdentifier) class. In addition to city dialects, we can provide results aggregated by region and by country. While these agregated results are less fine-grained, they tend to be more accurate.\n","\n","The example below illustrates how `DialectIdentifier` can be used to predict the dialects of given sentences by city, country and region. \n","\n","**NOTE:** You may get some warnings when running the example below in Colab. These can be safely ignored."]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"9YpWxBAm088-","outputId":"f2dd0967-c3d5-4b96-93d2-0ce7f346923f"},"source":["from camel_tools.dialectid import DialectIdentifier\n","\n","did = DialectIdentifier.pretrained()\n","\n","sentences = [\n","    'مال الهوى و مالي شكون اللي جابني ليك  ما كنت انايا ف حالي بلاو قلبي يانا بيك',\n","    'بدي دوب قلي قلي بجنون بحبك انا مجنون ما بنسى حبك يوم'\n","]\n","\n","predictions = did.predict(sentences, 'city')\n","print([p.top for p in predictions])\n","\n","predictions = did.predict(sentences, 'country')\n","print([p.top for p in predictions])\n","\n","predictions = did.predict(sentences, 'region')\n","print([p.top for p in predictions])"],"execution_count":null,"outputs":[{"output_type":"stream","text":["/usr/local/lib/python3.6/dist-packages/sklearn/base.py:318: UserWarning: Trying to unpickle estimator LabelEncoder from version 0.23.2 when using version 0.22.2.post1. This might lead to breaking code or invalid results. Use at your own risk.\n","  UserWarning)\n","/usr/local/lib/python3.6/dist-packages/sklearn/base.py:318: UserWarning: Trying to unpickle estimator TfidfTransformer from version 0.23.2 when using version 0.22.2.post1. This might lead to breaking code or invalid results. Use at your own risk.\n","  UserWarning)\n","/usr/local/lib/python3.6/dist-packages/sklearn/base.py:318: UserWarning: Trying to unpickle estimator TfidfVectorizer from version 0.23.2 when using version 0.22.2.post1. This might lead to breaking code or invalid results. Use at your own risk.\n","  UserWarning)\n","/usr/local/lib/python3.6/dist-packages/sklearn/base.py:318: UserWarning: Trying to unpickle estimator FeatureUnion from version 0.23.2 when using version 0.22.2.post1. This might lead to breaking code or invalid results. Use at your own risk.\n","  UserWarning)\n","/usr/local/lib/python3.6/dist-packages/sklearn/base.py:318: UserWarning: Trying to unpickle estimator MultinomialNB from version 0.23.2 when using version 0.22.2.post1. This might lead to breaking code or invalid results. Use at your own risk.\n","  UserWarning)\n","/usr/local/lib/python3.6/dist-packages/sklearn/base.py:318: UserWarning: Trying to unpickle estimator LabelBinarizer from version 0.23.2 when using version 0.22.2.post1. This might lead to breaking code or invalid results. Use at your own risk.\n","  UserWarning)\n","/usr/local/lib/python3.6/dist-packages/sklearn/base.py:318: UserWarning: Trying to unpickle estimator OneVsRestClassifier from version 0.23.2 when using version 0.22.2.post1. This might lead to breaking code or invalid results. Use at your own risk.\n","  UserWarning)\n"],"name":"stderr"},{"output_type":"stream","text":["['Rabat', 'Beirut']\n","['Morocco', 'Lebanon']\n","['Maghreb', 'Levant']\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"IiIPcZcx088_"},"source":["# Sentiment Analysis\n","\n","We provide a pretrained [sentiment analysis](https://en.wikipedia.org/wiki/Sentiment_analysis) system that has been trained using a combination of multiple data sets. The model can be accessed using the [`SentimentAnalyzer`](https://camel-tools.readthedocs.io/en/latest/api/sentiment.html#camel_tools.sentiment.SentimentAnalyzer) class and it outputs one of three sentiment tags for each sentence: `'positive'`, `'negative'` and `'neutral'`. The example below demonstrates how it can be used."]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"nWSDfRYd088_","executionInfo":{"status":"ok","timestamp":1612959008426,"user_tz":-240,"elapsed":18039,"user":{"displayName":"Ossama Obeid","photoUrl":"","userId":"06767072059928592512"}},"outputId":"6a758f91-8edd-4d96-8a1d-e883d88d9f75"},"source":["from camel_tools.sentiment import SentimentAnalyzer\n","\n","sa = SentimentAnalyzer.pretrained()\n","\n","sentences = [\n","    'أنا بخير',\n","    'أنا لست بخير'\n","]\n","\n","sentiments = sa.predict(sentences)\n","\n","print(sentiments)"],"execution_count":4,"outputs":[{"output_type":"stream","text":["['positive', 'negative']\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"1jo6RMNz088_"},"source":["# Named Entitiy Recognition\n","\n","CAMeL Tools comes with an easy-to-use, pretrained [named-entitity recognition](https://en.wikipedia.org/wiki/Named-entity_recognition)  (NER) system that can be accessed using the [`NERecognizer`](https://camel-tools.readthedocs.io/en/latest/api/ner.html#camel_tools.ner.NERecognizer) class. \n","\n","For each token in an input sentence, `NERecognizer` outputs a label that indicates the type of named-entity.The system outputs one of the following labels for each token: `'B-LOC'`, `'B-ORG'`, `'B-PERS'`, `'B-MISC'`, `'I-LOC'`, `'I-ORG'`, `'I-PERS'`, `'I-MISC'`, `'O'`.\n","Named-entites can either be a `LOC` (location), `ORG` (organization), `PERS` (person), or `MISC` (miscallaneous).\n","\n","Labels beginning with `B` indicate that their corresponding tokens are the begininging of a multi-word named-entity or is a single-token named-entity'. Those begining with `I` indicate that their corresponding tokens are continuations of a multi-word named-entity. Words that aren't named-entities are given the `'O'` label.\n","\n","The example below illustrates how `NERecognizer` can be used to label named-entities in a given sentence.\n"]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"gwBGDtiY088_","outputId":"a12b11ae-e87f-438e-95cc-712d045ddcb8"},"source":["from camel_tools.tokenizers.word import simple_word_tokenize\n","from camel_tools.ner import NERecognizer\n","\n","ner = NERecognizer.pretrained()\n","\n","# NERecognizer expects pre-tokenized text\n","sentence = simple_word_tokenize('إمارة أبوظبي هي إحدى إمارات دولة الإمارات العربية المتحدة السبع.')\n","\n","labels = ner.predict_sentence(sentence)\n","\n","# Print each token paired with it's NER label\n","print(list(zip(sentence, labels)))"],"execution_count":null,"outputs":[{"output_type":"stream","text":["[('إمارة', 'O'), ('أبوظبي', 'B-LOC'), ('هي', 'O'), ('إحدى', 'O'), ('إمارات', 'O'), ('دولة', 'O'), ('الإمارات', 'B-LOC'), ('العربية', 'I-LOC'), ('المتحدة', 'I-LOC'), ('السبع', 'O'), ('.', 'O')]\n"],"name":"stdout"}]}]}