From 8296947c896ab2aeb246363834662be049a21c7d Mon Sep 17 00:00:00 2001 From: Dmitri Soshnikov Date: Thu, 13 Jan 2022 13:26:22 +0300 Subject: [PATCH] Add embeddings --- .gitignore | 1 + 5-NLP/13-TextRep/README.md | 4 +- .../TextRepresentationPyTorch.ipynb | 19 +- 5-NLP/13-TextRep/TextRepresentationTF.ipynb | 164 +-- 5-NLP/14-Embeddings/EmbeddingsPyTorch.ipynb | 133 ++- 5-NLP/14-Embeddings/EmbeddingsTF.ipynb | 1030 +++++------------ 5-NLP/14-Embeddings/README.md | 0 .../images/embedding-classifier-example.png | Bin 0 -> 39645 bytes ...rithms-for-converting-words-to-vectors.png | Bin 0 -> 15187 bytes .../images/offset-sequence-representation.png | Bin 0 -> 38594 bytes 5-NLP/15-LanguageModeling/README.md | 20 + 5-NLP/README.md | 16 + README.md | 4 +- 13 files changed, 493 insertions(+), 898 deletions(-) create mode 100644 5-NLP/14-Embeddings/README.md create mode 100644 5-NLP/14-Embeddings/images/embedding-classifier-example.png create mode 100644 5-NLP/14-Embeddings/images/example-algorithms-for-converting-words-to-vectors.png create mode 100644 5-NLP/14-Embeddings/images/offset-sequence-representation.png create mode 100644 5-NLP/15-LanguageModeling/README.md diff --git a/.gitignore b/.gitignore index 55d493e7..23696872 100644 --- a/.gitignore +++ b/.gitignore @@ -15,6 +15,7 @@ .ipynb_checkpoints/ data/ +.vector_cache/ # Mono auto generated files mono_crash.* diff --git a/5-NLP/13-TextRep/README.md b/5-NLP/13-TextRep/README.md index 13ab39bc..3853e33a 100644 --- a/5-NLP/13-TextRep/README.md +++ b/5-NLP/13-TextRep/README.md @@ -14,7 +14,7 @@ Our goal would be to classify the news item into one of the categories based on If we want to solve Natural Language Processing (NLP) tasks with neural networks, we need some way to represent text as tensors. Computers already represent textual characters as numbers that map to fonts on your screen using encodings such as ASCII or UTF-8. -![Image showing diagram mapping a character to an ASCII and binary representation](images/ascii-character-map.png) +Image showing diagram mapping a character to an ASCII and binary representation We understand what each letter **represents**, and how all characters come together to form the words of a sentence. However, computers by themselves do not have such an understanding, and neural network has to learn the meaning during training. @@ -34,7 +34,7 @@ In some cases, we may consider using tri-grams -- combinations of three words -- When solving tasks like text classification, we need to be able to represent text by one fixed-size vector, which we will use as an input to final dense classifier. One of the simplest ways to do that is to combine all individual word representations, eg. by adding them. If we add one-hot encodings of each word, we will end up with a vector of frequencies, showing how many times each word appears inside the text. Such representation of text is called **bag of words** (BOW). - + BOW essentially represents which words appear in text and in which quantities, which can indeed be a good indication of what the text is about. For example, news article on politics is likely to contains words such as *president* and *country*, while scientific publication would have something like *collider*, *discovered*, etc. Thus, word frequencies can in many cases be a good indicator of text content. diff --git a/5-NLP/13-TextRep/TextRepresentationPyTorch.ipynb b/5-NLP/13-TextRep/TextRepresentationPyTorch.ipynb index 463039a8..4cb34c33 100644 --- a/5-NLP/13-TextRep/TextRepresentationPyTorch.ipynb +++ b/5-NLP/13-TextRep/TextRepresentationPyTorch.ipynb @@ -8,7 +8,7 @@ "\n", "As we have mentioned, we will focus on simple text classification task based on **AG_NEWS** dataset, which is to classify news headlines into one of 4 categories: World, Sports, Business and Sci/Tech.\n", "\n", - "### The Dataset\n", + "## The Dataset\n", "\n", "This dataset is built into [`torchtext`](https://github.com/pytorch/text) module, so we can easily access it." ] @@ -116,7 +116,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "### Tokenization\n", + "## Tokenization\n", "\n", "Now we need to convert text into **numbers** that can be represented as tensors. If we want word-level representation, we need to do two things:\n", "* use **tokenizer** to split text into **tokens**\n", @@ -200,7 +200,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "### Bag of Words text representation\n", + "## Bag of Words text representation\n", "\n", "Because words represent meaning, sometimes we can figure out the meaning of a text by just looking at the individual words, regardless of their order in the sentence. For example, when classifying news, words like *weather*, *snow* are likely to indicate *weather forecast*, while words like *stocks*, *dollar* would count towards *financial news*.\n", "\n", @@ -285,7 +285,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "### Training BoW classifier\n", + "## Training BoW classifier\n", "\n", "Now that we have learned how to build Bag-of-Words representation of our text, let's train a classifier on top of it. First, we need to convert our dataset for training in such a way, that all positional vector representations are converted to bag-of-words representation. This can be achieved by passing `bowify` function as `collate_fn` parameter to standard torch `DataLoader`:" ] @@ -396,7 +396,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "### BiGrams, TriGrams and N-Grams\n", + "## BiGrams, TriGrams and N-Grams\n", "\n", "One limitation of a bag of words approach is that some words are part of multi word expressions, for example, the word 'hot dog' has a completely different meaning than the words 'hot' and 'dog' in other contexts. If we represent words 'hot` and 'dog' always by the same vectors, it can confuse our model.\n", "\n", @@ -488,7 +488,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "### Term Frequency Inverse Document Frequency TF-IDF\n", + "## Term Frequency Inverse Document Frequency TF-IDF\n", "\n", "In BoW representation, word occurrences are evenly weighted, regardless of the word itself. However, it is clear that frequent words, such as *a*, *in*, etc. are much less important for the classification, than specialized terms. In fact, in most NLP tasks some words are more relevant than others.\n", "\n", @@ -538,10 +538,15 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "### Conclusion \n", + "## Conclusion \n", "\n", "However even though TF-IDF representations provide frequency weight to different words they are unable to represent meaning or order. As the famous linguist J. R. Firth said in 1935, “The complete meaning of a word is always contextual, and no study of meaning apart from context can be taken seriously.”. We will learn later in the course how to capture contextual information from text using language modeling.\n" ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [] } ], "metadata": { diff --git a/5-NLP/13-TextRep/TextRepresentationTF.ipynb b/5-NLP/13-TextRep/TextRepresentationTF.ipynb index 2675380e..ebb5799c 100644 --- a/5-NLP/13-TextRep/TextRepresentationTF.ipynb +++ b/5-NLP/13-TextRep/TextRepresentationTF.ipynb @@ -8,59 +8,16 @@ "\n", "In this module, we will start with a simple text classification task based on the **[AG_NEWS](http://www.di.unipi.it/~gulli/AG_corpus_of_news_articles.html)** dataset: we'll classify news headlines into one of 4 categories: World, Sports, Business and Sci/Tech. \n", "\n", - "### The Dataset\n", + "## The Dataset\n", "\n", "To load the dataset, we will use the **[TensorFlow Datasets](https://www.tensorflow.org/datasets)** API." ] }, { "cell_type": "code", - "execution_count": 2, + "execution_count": 1, "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "\u001b[1mDownloading and preparing dataset Unknown size (download: Unknown size, generated: Unknown size, total: Unknown size) to C:\\Users\\dmitryso\\tensorflow_datasets\\ag_news_subset\\1.0.0...\u001b[0m\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - "Dl Completed...: 0 url [00:00, ? url/s]\n", - "Dl Completed...: 0%| | 0/1 [00:00" + "" ] }, - "execution_count": 18, + "execution_count": 7, "metadata": {}, "output_type": "execute_result" } @@ -241,16 +198,16 @@ }, { "cell_type": "code", - "execution_count": 19, + "execution_count": 8, "metadata": {}, "outputs": [ { "data": { "text/plain": [ - "array([[1, 1, 0, 2, 0, 0, 0, 0, 0]])" + "array([[1, 1, 0, 2, 0, 0, 0, 0, 0]], dtype=int64)" ] }, - "execution_count": 19, + "execution_count": 8, "metadata": {}, "output_type": "execute_result" } @@ -264,8 +221,7 @@ " 'Its hot outside.',\n", " ]\n", "sc_vectorizer.fit_transform(corpus)\n", - "sc_vectorizer.transform(['My dog likes hot dogs on a hot day.']).toarray()\n", - "\n" + "sc_vectorizer.transform(['My dog likes hot dogs on a hot day.']).toarray()" ] }, { @@ -277,16 +233,16 @@ }, { "cell_type": "code", - "execution_count": 20, + "execution_count": 9, "metadata": {}, "outputs": [ { "data": { "text/plain": [ - "array([0., 0., 0., ..., 0., 0., 0.], dtype=float32)" + "array([0., 5., 0., ..., 0., 0., 0.], dtype=float32)" ] }, - "execution_count": 20, + "execution_count": 9, "metadata": {}, "output_type": "execute_result" } @@ -316,7 +272,7 @@ }, { "cell_type": "code", - "execution_count": 21, + "execution_count": 11, "metadata": {}, "outputs": [], "source": [ @@ -335,23 +291,23 @@ }, { "cell_type": "code", - "execution_count": 22, + "execution_count": 12, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ - "938/938 [==============================] - 88s 94ms/step - loss: 0.5466 - acc: 0.8759 - val_loss: 0.3682 - val_acc: 0.8950\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n" + "938/938 [==============================] - 66s 70ms/step - loss: 0.6144 - acc: 0.8427 - val_loss: 0.4416 - val_acc: 0.8697\n" ] }, { "data": { "text/plain": [ - "" + "" ] }, - "execution_count": 22, + "execution_count": 12, "metadata": {}, "output_type": "execute_result" } @@ -379,41 +335,44 @@ }, { "cell_type": "code", - "execution_count": 23, + "execution_count": 13, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ - "Model: \"functional_1\"\n", + "Model: \"model\"\n", "_________________________________________________________________\n", - "Layer (type) Output Shape Param # \n", + " Layer (type) Output Shape Param # \n", "=================================================================\n", - "input_1 (InputLayer) [(None, 1)] 0 \n", - "_________________________________________________________________\n", - "text_vectorization_6 (TextVe (None, None) 0 \n", - "_________________________________________________________________\n", - "tf_op_layer_OneHot (TensorFl [(None, None, 50000)] 0 \n", - "_________________________________________________________________\n", - "tf_op_layer_Sum (TensorFlowO [(None, 50000)] 0 \n", - "_________________________________________________________________\n", - "dense_1 (Dense) (None, 4) 200004 \n", + " input_1 (InputLayer) [(None, 1)] 0 \n", + " \n", + " text_vectorization (TextVec (None, None) 0 \n", + " torization) \n", + " \n", + " tf.one_hot (TFOpLambda) (None, None, 5335) 0 \n", + " \n", + " tf.math.reduce_sum (TFOpLam (None, 5335) 0 \n", + " bda) \n", + " \n", + " dense_2 (Dense) (None, 4) 21344 \n", + " \n", "=================================================================\n", - "Total params: 200,004\n", - "Trainable params: 200,004\n", + "Total params: 21,344\n", + "Trainable params: 21,344\n", "Non-trainable params: 0\n", "_________________________________________________________________\n", - "938/938 [==============================] - 79s 84ms/step - loss: 0.5221 - acc: 0.8804 - val_loss: 0.3447 - val_acc: 0.9024\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n" + "938/938 [==============================] - 73s 77ms/step - loss: 0.6057 - acc: 0.8414 - val_loss: 0.4202 - val_acc: 0.8736\n" ] }, { "data": { "text/plain": [ - "" + "" ] }, - "execution_count": 23, + "execution_count": 13, "metadata": {}, "output_type": "execute_result" } @@ -451,7 +410,7 @@ }, { "cell_type": "code", - "execution_count": 24, + "execution_count": 14, "metadata": {}, "outputs": [ { @@ -465,10 +424,11 @@ { "data": { "text/plain": [ - "array([[1, 0, 1, 0, 2, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]])" + "array([[1, 0, 1, 0, 2, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]],\n", + " dtype=int64)" ] }, - "execution_count": 24, + "execution_count": 14, "metadata": {}, "output_type": "execute_result" } @@ -507,7 +467,7 @@ }, { "cell_type": "code", - "execution_count": 25, + "execution_count": 15, "metadata": {}, "outputs": [ { @@ -515,16 +475,16 @@ "output_type": "stream", "text": [ "Training vectorizer\n", - "938/938 [==============================] - 10s 11ms/step - loss: 0.5207 - acc: 0.8826 - val_loss: 0.3430 - val_acc: 0.9051\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n" + "938/938 [==============================] - 7s 7ms/step - loss: 0.5929 - acc: 0.8486 - val_loss: 0.4168 - val_acc: 0.8772\n" ] }, { "data": { "text/plain": [ - "" + "" ] }, - "execution_count": 25, + "execution_count": 15, "metadata": {}, "output_type": "execute_result" } @@ -566,7 +526,7 @@ }, { "cell_type": "code", - "execution_count": 20, + "execution_count": 16, "metadata": {}, "outputs": [ { @@ -578,7 +538,7 @@ " 0. ]])" ] }, - "execution_count": 20, + "execution_count": 16, "metadata": {}, "output_type": "execute_result" } @@ -599,7 +559,7 @@ }, { "cell_type": "code", - "execution_count": 21, + "execution_count": 17, "metadata": {}, "outputs": [ { @@ -607,16 +567,16 @@ "output_type": "stream", "text": [ "Training vectorizer\n", - "938/938 [==============================] - 94s 101ms/step - loss: 0.3203 - acc: 0.9039 - val_loss: 0.2542 - val_acc: 0.9186\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n" + "938/938 [==============================] - 12s 12ms/step - loss: 0.4197 - acc: 0.8662 - val_loss: 0.3432 - val_acc: 0.8849\n" ] }, { "data": { "text/plain": [ - "" + "" ] }, - "execution_count": 21, + "execution_count": 17, "metadata": {}, "output_type": "execute_result" } @@ -638,7 +598,7 @@ "source": [ "## Conclusion \n", "\n", - "Even though TF-IDF representations provide frequency weights to different words, they are unable to represent meaning or order. As the famous linguist J. R. Firth said in 1935, \"The complete meaning of a word is always contextual, and no study of meaning apart from context can be taken seriously.\" We will learn how to capture contextual information from text using language modeling in a later unit." + "Even though TF-IDF representations provide frequency weights to different words, they are unable to represent meaning or order. As the famous linguist J. R. Firth said in 1935, \"The complete meaning of a word is always contextual, and no study of meaning apart from context can be taken seriously.\" We will learn how to capture contextual information from text using language modeling later in the course." ] } ], diff --git a/5-NLP/14-Embeddings/EmbeddingsPyTorch.ipynb b/5-NLP/14-Embeddings/EmbeddingsPyTorch.ipynb index beb6320e..c77795cf 100644 --- a/5-NLP/14-Embeddings/EmbeddingsPyTorch.ipynb +++ b/5-NLP/14-Embeddings/EmbeddingsPyTorch.ipynb @@ -8,22 +8,9 @@ "\n", "In our previous example, we operated on high-dimensional bag-of-words vectors with length `vocab_size`, and we were explicitly converting from low-dimensional positional representation vectors into sparse one-hot representation. This one-hot representation is not memory-efficient, in addition, each word is treated independently from each other, i.e. one-hot encoded vectors do not express any semantic similarity between words.\n", "\n", - "In this unit, we will continue exploring **News AG** dataset. To begin, let's load the data and get some definitions from the previous unit.\n" + "In this unit, we will continue exploring **News AG** dataset. To begin, let's load the data and get some definitions from the previous notebook.\n" ] }, - { - "cell_type": "code", - "execution_count": 1, - "metadata": { - "jupyter": { - "outputs_hidden": true - } - }, - "outputs": [], - "source": [ - "!wget -q https://raw.githubusercontent.com/MicrosoftDocs/pytorchfundamentals/main/nlp-pytorch/torchnlp.py" - ] - }, { "cell_type": "code", "execution_count": 1, @@ -33,7 +20,21 @@ "name": "stdout", "output_type": "stream", "text": [ - "Loading dataset...\n", + "Loading dataset...\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "d:\\WORK\\ai-for-beginners\\5-NLP\\14-Embeddings\\data\\train.csv: 29.5MB [00:01, 18.8MB/s] \n", + "d:\\WORK\\ai-for-beginners\\5-NLP\\14-Embeddings\\data\\test.csv: 1.86MB [00:00, 11.2MB/s] \n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ "Building vocab...\n", "Vocab size = 95812\n" ] @@ -54,7 +55,7 @@ "metadata": {}, "source": [ "\n", - "### What is embedding?\n", + "## What is embedding?\n", "\n", "The idea of **embedding** is to represent words by lower-dimensional dense vectors, which somehow reflect semantic meaning of a word. We will later discuss how to build meaningful word embeddings, but for now let's just think of embeddings as a way to lower dimensionality of a word vector. \n", "\n", @@ -62,7 +63,7 @@ "\n", "By using embedding layer as a first layer in our network, we can switch from bag-or-words to **embedding bag** model, where we first convert each word in our text into corresponding embedding, and then compute some aggregate function over all those embeddings, such as `sum`, `average` or `max`. \n", "\n", - "![Image showing an embedding classifier for five sequence words.](./images/embedding-classifier-example.png)\n", + "![Image showing an embedding classifier for five sequence words.](images/embedding-classifier-example.png)\n", "\n", "Our classifier neural network will start with embedding layer, then aggregation layer, and linear classifier on top of it:" ] @@ -136,19 +137,19 @@ "name": "stdout", "output_type": "stream", "text": [ - "3200: acc=0.6428125\n", - "6400: acc=0.68453125\n", - "9600: acc=0.7123958333333333\n", - "12800: acc=0.725703125\n", - "16000: acc=0.7365625\n", - "19200: acc=0.7464583333333333\n", - "22400: acc=0.7548214285714285\n" + "3200: acc=0.6415625\n", + "6400: acc=0.6865625\n", + "9600: acc=0.7103125\n", + "12800: acc=0.726953125\n", + "16000: acc=0.739375\n", + "19200: acc=0.75046875\n", + "22400: acc=0.7572321428571429\n" ] }, { "data": { "text/plain": [ - "(0.9526769402541186, 0.7595969289827256)" + "(0.889799795315499, 0.7623160588611644)" ] }, "execution_count": 4, @@ -176,7 +177,7 @@ "\n", "In the previous architecture, we needed to pad all sequences to the same length in order to fit them into a minibatch. This is not the most efficient way to represent variable length sequences - another apporach would be to use **offset** vector, which would hold offsets of all sequences stored in one large vector.\n", "\n", - "![Image showing an offset sequence representation](./images/offset-sequence-representation.png)\n", + "![Image showing an offset sequence representation](images/offset-sequence-representation.png)\n", "\n", "> **Note**: On the picture above, we show a sequence of characters, but in our example we are working with sequences of words. However, the general principle of representing sequences with offset vector remains the same.\n", "\n", @@ -246,19 +247,19 @@ "name": "stdout", "output_type": "stream", "text": [ - "3200: acc=0.6334375\n", - "6400: acc=0.68234375\n", - "9600: acc=0.7072916666666667\n", - "12800: acc=0.72375\n", - "16000: acc=0.73575\n", - "19200: acc=0.743125\n", - "22400: acc=0.7497767857142857\n" + "3200: acc=0.6153125\n", + "6400: acc=0.6615625\n", + "9600: acc=0.6932291666666667\n", + "12800: acc=0.715078125\n", + "16000: acc=0.7270625\n", + "19200: acc=0.7382291666666667\n", + "22400: acc=0.7486160714285715\n" ] }, { "data": { "text/plain": [ - "(23.37446267194498, 0.754118682021753)" + "(22.771553103007037, 0.7551983365323096)" ] }, "execution_count": 7, @@ -338,13 +339,13 @@ "name": "stdout", "output_type": "stream", "text": [ - "neuronal -> 0.780479907989502\n", + "neuronal -> 0.7804799675941467\n", "neurons -> 0.7326500415802002\n", - "neural_circuits -> 0.7252851128578186\n", + "neural_circuits -> 0.7252851724624634\n", "neuron -> 0.7174385190010071\n", - "cortical -> 0.6941086053848267\n", - "brain_circuitry -> 0.6923245787620544\n", - "synaptic -> 0.6699119210243225\n", + "cortical -> 0.6941086649894714\n", + "brain_circuitry -> 0.6923246383666992\n", + "synaptic -> 0.6699118614196777\n", "neural_circuitry -> 0.6638563275337219\n", "neurochemical -> 0.6555314064025879\n", "neuronal_activity -> 0.6531826257705688\n" @@ -360,7 +361,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "We can also extract vector embeddings from the word, to be used in training classification model (we only show first 20 components of the vector for clarity):" + "We can also compute vector embeddings from the word, to be used in training classification model (we only show first 20 components of the vector for clarity):" ] }, { @@ -396,7 +397,7 @@ }, { "cell_type": "code", - "execution_count": 11, + "execution_count": 10, "metadata": {}, "outputs": [ { @@ -405,7 +406,7 @@ "('queen', 0.7118192911148071)" ] }, - "execution_count": 11, + "execution_count": 10, "metadata": {}, "output_type": "execute_result" } @@ -424,7 +425,7 @@ "\n", "Another method, **GloVe**, leverages the idea of co-occurence matrix, uses neural methods to decompose co-occurrence matrix into more expressive and non linear word vectors.\n", "\n", - "You can play with the example by changing embeddings to FastText and GloVe, since gensim supports " + "You can play with the example by changing embeddings to FastText and GloVe, since gensim supports several different word embedding models." ] }, { @@ -438,7 +439,7 @@ }, { "cell_type": "code", - "execution_count": 12, + "execution_count": 11, "metadata": { "tags": [] }, @@ -481,29 +482,29 @@ }, { "cell_type": "code", - "execution_count": 13, + "execution_count": 12, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ - "3200: acc=0.63875\n", - "6400: acc=0.693125\n", - "9600: acc=0.7176041666666667\n", - "12800: acc=0.7321875\n", - "16000: acc=0.7454375\n", - "19200: acc=0.7559375\n", - "22400: acc=0.7631696428571428\n" + "3200: acc=0.6359375\n", + "6400: acc=0.68109375\n", + "9600: acc=0.7067708333333333\n", + "12800: acc=0.723671875\n", + "16000: acc=0.73625\n", + "19200: acc=0.7463541666666667\n", + "22400: acc=0.7560714285714286\n" ] }, { "data": { "text/plain": [ - "(218.64081493921944, 0.7667146513115803)" + "(214.1013875559821, 0.7626759436980166)" ] }, - "execution_count": 13, + "execution_count": 12, "metadata": {}, "output_type": "execute_result" } @@ -528,7 +529,15 @@ "cell_type": "code", "execution_count": 14, "metadata": {}, - "outputs": [], + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "100%|█████████▉| 399999/400000 [00:15<00:00, 25411.14it/s]\n" + ] + } + ], "source": [ "vocab = torchtext.vocab.GloVe(name='6B', dim=50)" ] @@ -660,12 +669,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "One of the reasons we are not seeing significant increase in accuracy is due to the fact that some words from our dataset are missing in the pre-trained GloVe vocabulary, and thus they are essentially ignored. To overcome this fact, we can train our own embeddings on our dataset. \n", - "\n", - "\n", - "## Training your own embeddings\n", - "\n", - "In our examples, we have been using pre-trained semantic embeddings, but it is interesting to see how those embeddings can be trained using either CBoW, or Skip-gram architectures. This exercise goes beyond this module, but those interested might want to check out this [official PyTorch tutorial on Language Modeling](https://pytorch.org/tutorials/beginner/nlp/word_embeddings_tutorial.html). Also, **gensim** framework can be used to train most commonly used embeddings in a few lines of code, as described [in this documentation](https://pytorch.org/tutorials/beginner/nlp/word_embeddings_tutorial.html)." + "One of the reasons we are not seeing significant increase in accuracy is due to the fact that some words from our dataset are missing in the pre-trained GloVe vocabulary, and thus they are essentially ignored. To overcome this fact, we can train our own embeddings on our dataset. " ] }, { @@ -685,10 +689,13 @@ } ], "metadata": { + "interpreter": { + "hash": "0cb620c6d4b9f7a635928804c26cf22403d89d98d79684e4529119355ee6d5a5" + }, "kernelspec": { "display_name": "py37_pytorch", "language": "python", - "name": "conda-env-py37_pytorch-py" + "name": "python3" }, "language_info": { "codemirror_mode": { @@ -700,7 +707,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.7.7" + "version": "3.8.12" } }, "nbformat": 4, diff --git a/5-NLP/14-Embeddings/EmbeddingsTF.ipynb b/5-NLP/14-Embeddings/EmbeddingsTF.ipynb index ec520127..5d47a26f 100644 --- a/5-NLP/14-Embeddings/EmbeddingsTF.ipynb +++ b/5-NLP/14-Embeddings/EmbeddingsTF.ipynb @@ -2,48 +2,32 @@ "cells": [ { "cell_type": "markdown", + "metadata": {}, "source": [ "## Embeddings\n", "\n", "In our previous example, we operated on high-dimensional bag-of-words vectors with length `vocab_size`, and we explicitly converted low-dimensional positional representation vectors into sparse one-hot representation. This one-hot representation is not memory-efficient. In addition, each word is treated independently from each other, so one-hot encoded vectors don't express semantic similarities between words.\n", "\n", "In this unit, we will continue exploring the **News AG** dataset. To begin, let's load the data and get some definitions from the previous unit." - ], - "metadata": {} + ] }, { "cell_type": "code", - "source": [ - "import sys\n", - "!{sys.executable} -m pip install --quiet tensorflow_datasets==4.4.0\n", - "!cd ~ && wget -q -O - https://mslearntensorflowlp.blob.core.windows.net/data/tfds-ag-news.tgz | tar xz" - ], - "outputs": [], "execution_count": 2, - "metadata": {} - }, - { - "cell_type": "code", + "metadata": {}, + "outputs": [], "source": [ "import tensorflow as tf\n", "from tensorflow import keras\n", "import tensorflow_datasets as tfds\n", "import numpy as np\n", "\n", - "# In this tutorial, we will be training a lot of models. In order to use GPU memory cautiously,\n", - "# we will set tensorflow option to grow GPU memory allocation when required.\n", - "physical_devices = tf.config.list_physical_devices('GPU') \n", - "if len(physical_devices)>0:\n", - " tf.config.experimental.set_memory_growth(physical_devices[0], True)\n", - "\n", "ds_train, ds_test = tfds.load('ag_news_subset').values()" - ], - "outputs": [], - "execution_count": 3, - "metadata": {} + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "\n", "### What's an embedding?\n", @@ -62,11 +46,38 @@ "* `Embedding` layer, which takes $n$ numbers, and reduces each number to a dense vector of a given length (100 in our example). Thus, the input tensor of shape $n$ will be transformed into an $n\\times 100$ tensor. \n", "* Aggregation layer, which takes the average of this tensor along the first axis, i.e. it will compute the average of all $n$ input tensors corresponding to different words. To implement this layer, we will use a `Lambda` layer, and pass into it the function to compute the average. The output will have shape of 100, and it will be the numeric representation of the whole input sequence.\n", "* Final `Dense` linear classifier." - ], - "metadata": {} + ] }, { "cell_type": "code", + "execution_count": 3, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Model: \"sequential\"\n", + "_________________________________________________________________\n", + " Layer (type) Output Shape Param # \n", + "=================================================================\n", + " text_vectorization (TextVec (None, None) 0 \n", + " torization) \n", + " \n", + " embedding (Embedding) (None, None, 100) 3000000 \n", + " \n", + " lambda (Lambda) (None, 100) 0 \n", + " \n", + " dense (Dense) (None, 4) 404 \n", + " \n", + "=================================================================\n", + "Total params: 3,000,404\n", + "Trainable params: 3,000,404\n", + "Non-trainable params: 0\n", + "_________________________________________________________________\n" + ] + } + ], "source": [ "vocab_size = 30000\n", "batch_size = 128\n", @@ -80,45 +91,41 @@ " keras.layers.Dense(4, activation='softmax')\n", "])\n", "model.summary()" - ], - "outputs": [ - { - "output_type": "stream", - "name": "stdout", - "text": [ - "Model: \"sequential_1\"\n", - "_________________________________________________________________\n", - "Layer (type) Output Shape Param # \n", - "=================================================================\n", - "text_vectorization_1 (TextVe (None, None) 0 \n", - "_________________________________________________________________\n", - "embedding_1 (Embedding) (None, None, 100) 3000000 \n", - "_________________________________________________________________\n", - "lambda_1 (Lambda) (None, 100) 0 \n", - "_________________________________________________________________\n", - "dense_1 (Dense) (None, 4) 404 \n", - "=================================================================\n", - "Total params: 3,000,404\n", - "Trainable params: 3,000,404\n", - "Non-trainable params: 0\n", - "_________________________________________________________________\n" - ] - } - ], - "execution_count": 6, - "metadata": {} + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "In the `summary` printout, in the **output shape** column, the first tensor dimension `None` corresponds to the minibatch size, and the second corresponds to the length of the token sequence. All token sequences in the minibatch have different lengths. We'll discuss how to deal with it in the next section.\n", "\n", "Now let's train the network:" - ], - "metadata": {} + ] }, { "cell_type": "code", + "execution_count": 4, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Training vectorizer\n", + "938/938 [==============================] - 20s 20ms/step - loss: 0.7891 - acc: 0.8155 - val_loss: 0.4470 - val_acc: 0.8642\n" + ] + }, + { + "data": { + "text/plain": [ + "" + ] + }, + "execution_count": 4, + "metadata": {}, + "output_type": "execute_result" + } + ], "source": [ "def extract_text(x):\n", " return x['title']+' '+x['description']\n", @@ -131,128 +138,140 @@ "\n", "model.compile(loss='sparse_categorical_crossentropy',metrics=['acc'])\n", "model.fit(ds_train.map(tupelize).batch(batch_size),validation_data=ds_test.map(tupelize).batch(batch_size))" - ], - "outputs": [ - { - "output_type": "stream", - "name": "stdout", - "text": [ - "Training vectorizer\n", - "938/938 [==============================] - 12s 13ms/step - loss: 0.7953 - acc: 0.8113 - val_loss: 0.4496 - val_acc: 0.8657\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n" - ] - }, - { - "output_type": "execute_result", - "execution_count": 7, - "data": { - "text/plain": "" - }, - "metadata": {} - } - ], - "execution_count": 7, - "metadata": {} + ] }, { "cell_type": "markdown", - "source": [ - "> **Note** that we are building vectorizer based on a subset of the data. This is done in order to speed up the process, and it might result in a situation when not all tokens from our text is present in the vocabulary. In this case, those tokens would be ignored, which may result in slightly lower accuracy. However, in real life a subset of text often gives a good vocabulary estimation." - ], "metadata": { "nteract": { "transient": { "deleting": false } } - } + }, + "source": [ + "> **Note** that we are building vectorizer based on a subset of the data. This is done in order to speed up the process, and it might result in a situation when not all tokens from our text is present in the vocabulary. In this case, those tokens would be ignored, which may result in slightly lower accuracy. However, in real life a subset of text often gives a good vocabulary estimation." + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "### Dealing with variable sequence sizes\n", "\n", "Let's understand how training happens in minibatches. In the example above, the input tensor has dimension 1, and we use 128-long minibatches, so that actual size of the tensor is $128 \\times 1$. However, the number of tokens in each sentence is different. If we apply the `TextVectorization` layer to a single input, the number of tokens returned is different, depending on how the text is tokenized:" - ], - "metadata": {} + ] }, { "cell_type": "code", - "source": [ - "print(vectorizer('Hello, world!'))\n", - "print(vectorizer('I am glad to meet you!'))" - ], + "execution_count": 5, + "metadata": {}, "outputs": [ { - "output_type": "stream", "name": "stdout", + "output_type": "stream", "text": [ "tf.Tensor([ 1 45], shape=(2,), dtype=int64)\n", "tf.Tensor([ 112 1271 1 3 1747 158], shape=(6,), dtype=int64)\n" ] } ], - "execution_count": 8, - "metadata": {} + "source": [ + "print(vectorizer('Hello, world!'))\n", + "print(vectorizer('I am glad to meet you!'))" + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "However, when we apply the vectorizer to several sequences, it has to produce a tensor of rectangular shape, so it fills unused elements with the PAD token (which in our case is zero):" - ], - "metadata": {} + ] }, { "cell_type": "code", + "execution_count": 6, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "" + ] + }, + "execution_count": 6, + "metadata": {}, + "output_type": "execute_result" + } + ], "source": [ "vectorizer(['Hello, world!','I am glad to meet you!'])" - ], - "outputs": [ - { - "output_type": "execute_result", - "execution_count": 9, - "data": { - "text/plain": "" - }, - "metadata": {} - } - ], - "execution_count": 9, - "metadata": {} + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "Here we can see the embeddings:" - ], - "metadata": {} + ] }, { "cell_type": "code", - "source": [ - "model.layers[1](vectorizer(['Hello, world!','I am glad to meet you!'])).numpy()" - ], + "execution_count": 7, + "metadata": {}, "outputs": [ { - "output_type": "execute_result", - "execution_count": 10, "data": { - "text/plain": "array([[[-0.02485236, -0.00416857, -0.06599288, ..., -0.02404598,\n 0.03529833, -0.02100844],\n [ 0.22493948, 0.01383338, 0.12420551, ..., 0.19531338,\n 0.13524376, 0.04216914],\n [ 0.04510409, 0.00708018, -0.0310419 , ..., -0.0188726 ,\n -0.0179676 , -0.04813331],\n [ 0.04510409, 0.00708018, -0.0310419 , ..., -0.0188726 ,\n -0.0179676 , -0.04813331],\n [ 0.04510409, 0.00708018, -0.0310419 , ..., -0.0188726 ,\n -0.0179676 , -0.04813331],\n [ 0.04510409, 0.00708018, -0.0310419 , ..., -0.0188726 ,\n -0.0179676 , -0.04813331]],\n\n [[-0.00226152, -0.0972852 , -0.00063103, ..., 0.00504377,\n 0.22460397, 0.1497297 ],\n [-0.15621698, -0.13758421, -0.02889572, ..., -0.02577994,\n 0.03472563, 0.08767739],\n [-0.02485236, -0.00416857, -0.06599288, ..., -0.02404598,\n 0.03529833, -0.02100844],\n [-0.06490357, -0.08200071, -0.06175491, ..., -0.02477042,\n -0.06802022, -0.01040947],\n [ 0.03279151, 0.12563369, 0.06062867, ..., -0.04349922,\n -0.12154414, -0.12533969],\n [-0.14435016, -0.304014 , -0.00378676, ..., 0.05609043,\n 0.20370889, 0.28518862]]], dtype=float32)" + "text/plain": [ + "array([[[ 1.53059261e-02, 6.80514947e-02, 3.14026810e-02, ...,\n", + " -8.92002955e-02, 1.52911525e-04, -5.65562584e-02],\n", + " [ 2.57456154e-01, 2.79364467e-01, -2.03605562e-01, ...,\n", + " -2.07474351e-01, 8.31158683e-02, -2.03911960e-01],\n", + " [ 3.98201384e-02, -8.03454965e-03, 2.39790026e-02, ...,\n", + " -7.18549127e-04, 2.66963355e-02, -4.30646613e-02],\n", + " [ 3.98201384e-02, -8.03454965e-03, 2.39790026e-02, ...,\n", + " -7.18549127e-04, 2.66963355e-02, -4.30646613e-02],\n", + " [ 3.98201384e-02, -8.03454965e-03, 2.39790026e-02, ...,\n", + " -7.18549127e-04, 2.66963355e-02, -4.30646613e-02],\n", + " [ 3.98201384e-02, -8.03454965e-03, 2.39790026e-02, ...,\n", + " -7.18549127e-04, 2.66963355e-02, -4.30646613e-02]],\n", + "\n", + " [[ 1.89674050e-01, 2.61548996e-01, -3.67433839e-02, ...,\n", + " -2.07366899e-01, -1.05442435e-01, -2.36952081e-01],\n", + " [ 6.16133213e-02, 1.80511594e-01, 9.77298319e-02, ...,\n", + " -5.46628237e-02, -1.07340455e-01, -1.06589928e-01],\n", + " [ 1.53059261e-02, 6.80514947e-02, 3.14026810e-02, ...,\n", + " -8.92002955e-02, 1.52911525e-04, -5.65562584e-02],\n", + " [-4.84890305e-02, -8.41715634e-02, 1.51529670e-01, ...,\n", + " 1.28192469e-01, -7.77286515e-02, 1.26041949e-01],\n", + " [-4.17212099e-02, -5.60694858e-02, 4.08860669e-02, ...,\n", + " 8.70475471e-02, 8.92383084e-02, 1.67974353e-01],\n", + " [ 2.85779923e-01, 4.57767487e-01, 4.52292450e-02, ...,\n", + " -1.97419018e-01, -2.04659685e-01, -2.79758364e-01]]],\n", + " dtype=float32)" + ] }, - "metadata": {} + "execution_count": 7, + "metadata": {}, + "output_type": "execute_result" } ], - "execution_count": 10, - "metadata": {} + "source": [ + "model.layers[1](vectorizer(['Hello, world!','I am glad to meet you!'])).numpy()" + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "> **Note**: To minimize the amount of padding, in some cases it makes sense to sort all sequences in the dataset in the order of increasing length (or, more precisely, number of tokens). This will ensure that each minibatch contains sequences of similar length." - ], - "metadata": {} + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "\n", "## Semantic embeddings: Word2Vec\n", @@ -271,473 +290,26 @@ "To experiment with the Word2Vec embedding pretrained on Google News dataset, we can use the **gensim** library. Below we find the words most similar to 'neural'.\n", "\n", "> **Note:** When you first create word vectors, downloading them can take some time!" - ], - "metadata": {} + ] }, { "cell_type": "code", + "execution_count": 8, + "metadata": {}, + "outputs": [], "source": [ "import gensim.downloader as api\n", "w2v = api.load('word2vec-google-news-300')" - ], - "outputs": [ - { - "output_type": "stream", - "name": "stdout", - "text": [ - "[==================================================] 100.0% 1662.8/1662.8MB downloaded\n" - ] - }, - { - "output_type": "stream", - "name": "stderr", - "text": [ - "IOPub message rate exceeded.\n", - "The notebook server will temporarily stop sending output\n", - "to the client in order to avoid crashing it.\n", - "To change this limit, set the config variable\n", - "`--NotebookApp.iopub_msg_rate_limit`.\n", - "\n", - "Current values:\n", - "NotebookApp.iopub_msg_rate_limit=1000.0 (msgs/sec)\n", - "NotebookApp.rate_limit_window=3.0 (secs)\n", - "\n", - "IOPub message rate exceeded.\n", - "The notebook server will temporarily stop sending output\n", - "to the client in order to avoid crashing it.\n", - "To change this limit, set the config variable\n", - "`--NotebookApp.iopub_msg_rate_limit`.\n", - "\n", - "Current values:\n", - "NotebookApp.iopub_msg_rate_limit=1000.0 (msgs/sec)\n", - "NotebookApp.rate_limit_window=3.0 (secs)\n", - "\n", - "IOPub message rate exceeded.\n", - "The notebook server will temporarily stop sending output\n", - "to the client in order to avoid crashing it.\n", - "To change this limit, set the config variable\n", - "`--NotebookApp.iopub_msg_rate_limit`.\n", - "\n", - "Current values:\n", - "NotebookApp.iopub_msg_rate_limit=1000.0 (msgs/sec)\n", - "NotebookApp.rate_limit_window=3.0 (secs)\n", - "\n", - "IOPub message rate exceeded.\n", - "The notebook server will temporarily stop sending output\n", - "to the client in order to avoid crashing it.\n", - "To change this limit, set the config variable\n", - "`--NotebookApp.iopub_msg_rate_limit`.\n", - "\n", - "Current values:\n", - "NotebookApp.iopub_msg_rate_limit=1000.0 (msgs/sec)\n", - "NotebookApp.rate_limit_window=3.0 (secs)\n", - "\n", - "IOPub message rate exceeded.\n", - "The notebook server will temporarily stop sending output\n", - "to the client in order to avoid crashing it.\n", - "To change this limit, set the config variable\n", - "`--NotebookApp.iopub_msg_rate_limit`.\n", - "\n", - "Current values:\n", - "NotebookApp.iopub_msg_rate_limit=1000.0 (msgs/sec)\n", - "NotebookApp.rate_limit_window=3.0 (secs)\n", - "\n", - "IOPub message rate exceeded.\n", - "The notebook server will temporarily stop sending output\n", - "to the client in order to avoid crashing it.\n", - "To change this limit, set the config variable\n", - "`--NotebookApp.iopub_msg_rate_limit`.\n", - "\n", - "Current values:\n", - "NotebookApp.iopub_msg_rate_limit=1000.0 (msgs/sec)\n", - "NotebookApp.rate_limit_window=3.0 (secs)\n", - "\n", - "IOPub message rate exceeded.\n", - "The notebook server will temporarily stop sending output\n", - "to the client in order to avoid crashing it.\n", - "To change this limit, set the config variable\n", - "`--NotebookApp.iopub_msg_rate_limit`.\n", - "\n", - "Current values:\n", - "NotebookApp.iopub_msg_rate_limit=1000.0 (msgs/sec)\n", - "NotebookApp.rate_limit_window=3.0 (secs)\n", - "\n", - "IOPub message rate exceeded.\n", - "The notebook server will temporarily stop sending output\n", - "to the client in order to avoid crashing it.\n", - "To change this limit, set the config variable\n", - "`--NotebookApp.iopub_msg_rate_limit`.\n", - "\n", - "Current values:\n", - "NotebookApp.iopub_msg_rate_limit=1000.0 (msgs/sec)\n", - "NotebookApp.rate_limit_window=3.0 (secs)\n", - "\n", - "IOPub message rate exceeded.\n", - "The notebook server will temporarily stop sending output\n", - "to the client in order to avoid crashing it.\n", - "To change this limit, set the config variable\n", - "`--NotebookApp.iopub_msg_rate_limit`.\n", - "\n", - "Current values:\n", - "NotebookApp.iopub_msg_rate_limit=1000.0 (msgs/sec)\n", - "NotebookApp.rate_limit_window=3.0 (secs)\n", - "\n", - "IOPub message rate exceeded.\n", - "The notebook server will temporarily stop sending output\n", - "to the client in order to avoid crashing it.\n", - "To change this limit, set the config variable\n", - "`--NotebookApp.iopub_msg_rate_limit`.\n", - "\n", - "Current values:\n", - "NotebookApp.iopub_msg_rate_limit=1000.0 (msgs/sec)\n", - "NotebookApp.rate_limit_window=3.0 (secs)\n", - "\n", - "IOPub message rate exceeded.\n", - "The notebook server will temporarily stop sending output\n", - "to the client in order to avoid crashing it.\n", - "To change this limit, set the config variable\n", - "`--NotebookApp.iopub_msg_rate_limit`.\n", - "\n", - "Current values:\n", - "NotebookApp.iopub_msg_rate_limit=1000.0 (msgs/sec)\n", - "NotebookApp.rate_limit_window=3.0 (secs)\n", - "\n", - "IOPub message rate exceeded.\n", - "The notebook server will temporarily stop sending output\n", - "to the client in order to avoid crashing it.\n", - "To change this limit, set the config variable\n", - "`--NotebookApp.iopub_msg_rate_limit`.\n", - "\n", - "Current values:\n", - "NotebookApp.iopub_msg_rate_limit=1000.0 (msgs/sec)\n", - "NotebookApp.rate_limit_window=3.0 (secs)\n", - "\n", - "IOPub message rate exceeded.\n", - "The notebook server will temporarily stop sending output\n", - "to the client in order to avoid crashing it.\n", - "To change this limit, set the config variable\n", - "`--NotebookApp.iopub_msg_rate_limit`.\n", - "\n", - "Current values:\n", - "NotebookApp.iopub_msg_rate_limit=1000.0 (msgs/sec)\n", - "NotebookApp.rate_limit_window=3.0 (secs)\n", - "\n", - "IOPub message rate exceeded.\n", - "The notebook server will temporarily stop sending output\n", - "to the client in order to avoid crashing it.\n", - "To change this limit, set the config variable\n", - "`--NotebookApp.iopub_msg_rate_limit`.\n", - "\n", - "Current values:\n", - "NotebookApp.iopub_msg_rate_limit=1000.0 (msgs/sec)\n", - "NotebookApp.rate_limit_window=3.0 (secs)\n", - "\n", - "IOPub message rate exceeded.\n", - "The notebook server will temporarily stop sending output\n", - "to the client in order to avoid crashing it.\n", - "To change this limit, set the config variable\n", - "`--NotebookApp.iopub_msg_rate_limit`.\n", - "\n", - "Current values:\n", - "NotebookApp.iopub_msg_rate_limit=1000.0 (msgs/sec)\n", - "NotebookApp.rate_limit_window=3.0 (secs)\n", - "\n", - "IOPub message rate exceeded.\n", - "The notebook server will temporarily stop sending output\n", - "to the client in order to avoid crashing it.\n", - "To change this limit, set the config variable\n", - "`--NotebookApp.iopub_msg_rate_limit`.\n", - "\n", - "Current values:\n", - "NotebookApp.iopub_msg_rate_limit=1000.0 (msgs/sec)\n", - "NotebookApp.rate_limit_window=3.0 (secs)\n", - "\n", - "IOPub message rate exceeded.\n", - "The notebook server will temporarily stop sending output\n", - "to the client in order to avoid crashing it.\n", - "To change this limit, set the config variable\n", - "`--NotebookApp.iopub_msg_rate_limit`.\n", - "\n", - "Current values:\n", - "NotebookApp.iopub_msg_rate_limit=1000.0 (msgs/sec)\n", - "NotebookApp.rate_limit_window=3.0 (secs)\n", - "\n", - "IOPub message rate exceeded.\n", - "The notebook server will temporarily stop sending output\n", - "to the client in order to avoid crashing it.\n", - "To change this limit, set the config variable\n", - "`--NotebookApp.iopub_msg_rate_limit`.\n", - "\n", - "Current values:\n", - "NotebookApp.iopub_msg_rate_limit=1000.0 (msgs/sec)\n", - "NotebookApp.rate_limit_window=3.0 (secs)\n", - "\n", - "IOPub message rate exceeded.\n", - "The notebook server will temporarily stop sending output\n", - "to the client in order to avoid crashing it.\n", - "To change this limit, set the config variable\n", - "`--NotebookApp.iopub_msg_rate_limit`.\n", - "\n", - "Current values:\n", - "NotebookApp.iopub_msg_rate_limit=1000.0 (msgs/sec)\n", - "NotebookApp.rate_limit_window=3.0 (secs)\n", - "\n", - "IOPub message rate exceeded.\n", - "The notebook server will temporarily stop sending output\n", - "to the client in order to avoid crashing it.\n", - "To change this limit, set the config variable\n", - "`--NotebookApp.iopub_msg_rate_limit`.\n", - "\n", - "Current values:\n", - "NotebookApp.iopub_msg_rate_limit=1000.0 (msgs/sec)\n", - "NotebookApp.rate_limit_window=3.0 (secs)\n", - "\n", - "IOPub message rate exceeded.\n", - "The notebook server will temporarily stop sending output\n", - "to the client in order to avoid crashing it.\n", - "To change this limit, set the config variable\n", - "`--NotebookApp.iopub_msg_rate_limit`.\n", - "\n", - "Current values:\n", - "NotebookApp.iopub_msg_rate_limit=1000.0 (msgs/sec)\n", - "NotebookApp.rate_limit_window=3.0 (secs)\n", - "\n", - "IOPub message rate exceeded.\n", - "The notebook server will temporarily stop sending output\n", - "to the client in order to avoid crashing it.\n", - "To change this limit, set the config variable\n", - "`--NotebookApp.iopub_msg_rate_limit`.\n", - "\n", - "Current values:\n", - "NotebookApp.iopub_msg_rate_limit=1000.0 (msgs/sec)\n", - "NotebookApp.rate_limit_window=3.0 (secs)\n", - "\n", - "IOPub message rate exceeded.\n", - "The notebook server will temporarily stop sending output\n", - "to the client in order to avoid crashing it.\n", - "To change this limit, set the config variable\n", - "`--NotebookApp.iopub_msg_rate_limit`.\n", - "\n", - "Current values:\n", - "NotebookApp.iopub_msg_rate_limit=1000.0 (msgs/sec)\n", - "NotebookApp.rate_limit_window=3.0 (secs)\n", - "\n", - "IOPub message rate exceeded.\n", - "The notebook server will temporarily stop sending output\n", - "to the client in order to avoid crashing it.\n", - "To change this limit, set the config variable\n", - "`--NotebookApp.iopub_msg_rate_limit`.\n", - "\n", - "Current values:\n", - "NotebookApp.iopub_msg_rate_limit=1000.0 (msgs/sec)\n", - "NotebookApp.rate_limit_window=3.0 (secs)\n", - "\n", - "IOPub message rate exceeded.\n", - "The notebook server will temporarily stop sending output\n", - "to the client in order to avoid crashing it.\n", - "To change this limit, set the config variable\n", - "`--NotebookApp.iopub_msg_rate_limit`.\n", - "\n", - "Current values:\n", - "NotebookApp.iopub_msg_rate_limit=1000.0 (msgs/sec)\n", - "NotebookApp.rate_limit_window=3.0 (secs)\n", - "\n", - "IOPub message rate exceeded.\n", - "The notebook server will temporarily stop sending output\n", - "to the client in order to avoid crashing it.\n", - "To change this limit, set the config variable\n", - "`--NotebookApp.iopub_msg_rate_limit`.\n", - "\n", - "Current values:\n", - "NotebookApp.iopub_msg_rate_limit=1000.0 (msgs/sec)\n", - "NotebookApp.rate_limit_window=3.0 (secs)\n", - "\n", - "IOPub message rate exceeded.\n", - "The notebook server will temporarily stop sending output\n", - "to the client in order to avoid crashing it.\n", - "To change this limit, set the config variable\n", - "`--NotebookApp.iopub_msg_rate_limit`.\n", - "\n", - "Current values:\n", - "NotebookApp.iopub_msg_rate_limit=1000.0 (msgs/sec)\n", - "NotebookApp.rate_limit_window=3.0 (secs)\n", - "\n", - "IOPub message rate exceeded.\n", - "The notebook server will temporarily stop sending output\n", - "to the client in order to avoid crashing it.\n", - "To change this limit, set the config variable\n", - "`--NotebookApp.iopub_msg_rate_limit`.\n", - "\n", - "Current values:\n", - "NotebookApp.iopub_msg_rate_limit=1000.0 (msgs/sec)\n", - "NotebookApp.rate_limit_window=3.0 (secs)\n", - "\n", - "IOPub message rate exceeded.\n", - "The notebook server will temporarily stop sending output\n", - "to the client in order to avoid crashing it.\n", - "To change this limit, set the config variable\n", - "`--NotebookApp.iopub_msg_rate_limit`.\n", - "\n", - "Current values:\n", - "NotebookApp.iopub_msg_rate_limit=1000.0 (msgs/sec)\n", - "NotebookApp.rate_limit_window=3.0 (secs)\n", - "\n", - "IOPub message rate exceeded.\n", - "The notebook server will temporarily stop sending output\n", - "to the client in order to avoid crashing it.\n", - "To change this limit, set the config variable\n", - "`--NotebookApp.iopub_msg_rate_limit`.\n", - "\n", - "Current values:\n", - "NotebookApp.iopub_msg_rate_limit=1000.0 (msgs/sec)\n", - "NotebookApp.rate_limit_window=3.0 (secs)\n", - "\n", - "IOPub message rate exceeded.\n", - "The notebook server will temporarily stop sending output\n", - "to the client in order to avoid crashing it.\n", - "To change this limit, set the config variable\n", - "`--NotebookApp.iopub_msg_rate_limit`.\n", - "\n", - "Current values:\n", - "NotebookApp.iopub_msg_rate_limit=1000.0 (msgs/sec)\n", - "NotebookApp.rate_limit_window=3.0 (secs)\n", - "\n", - "IOPub message rate exceeded.\n", - "The notebook server will temporarily stop sending output\n", - "to the client in order to avoid crashing it.\n", - "To change this limit, set the config variable\n", - "`--NotebookApp.iopub_msg_rate_limit`.\n", - "\n", - "Current values:\n", - "NotebookApp.iopub_msg_rate_limit=1000.0 (msgs/sec)\n", - "NotebookApp.rate_limit_window=3.0 (secs)\n", - "\n", - "IOPub message rate exceeded.\n", - "The notebook server will temporarily stop sending output\n", - "to the client in order to avoid crashing it.\n", - "To change this limit, set the config variable\n", - "`--NotebookApp.iopub_msg_rate_limit`.\n", - "\n", - "Current values:\n", - "NotebookApp.iopub_msg_rate_limit=1000.0 (msgs/sec)\n", - "NotebookApp.rate_limit_window=3.0 (secs)\n", - "\n", - "IOPub message rate exceeded.\n", - "The notebook server will temporarily stop sending output\n", - "to the client in order to avoid crashing it.\n", - "To change this limit, set the config variable\n", - "`--NotebookApp.iopub_msg_rate_limit`.\n", - "\n", - "Current values:\n", - "NotebookApp.iopub_msg_rate_limit=1000.0 (msgs/sec)\n", - "NotebookApp.rate_limit_window=3.0 (secs)\n", - "\n", - "IOPub message rate exceeded.\n", - "The notebook server will temporarily stop sending output\n", - "to the client in order to avoid crashing it.\n", - "To change this limit, set the config variable\n", - "`--NotebookApp.iopub_msg_rate_limit`.\n", - "\n", - "Current values:\n", - "NotebookApp.iopub_msg_rate_limit=1000.0 (msgs/sec)\n", - "NotebookApp.rate_limit_window=3.0 (secs)\n", - "\n", - "IOPub message rate exceeded.\n", - "The notebook server will temporarily stop sending output\n", - "to the client in order to avoid crashing it.\n", - "To change this limit, set the config variable\n", - "`--NotebookApp.iopub_msg_rate_limit`.\n", - "\n", - "Current values:\n", - "NotebookApp.iopub_msg_rate_limit=1000.0 (msgs/sec)\n", - "NotebookApp.rate_limit_window=3.0 (secs)\n", - "\n", - "IOPub message rate exceeded.\n", - "The notebook server will temporarily stop sending output\n", - "to the client in order to avoid crashing it.\n", - "To change this limit, set the config variable\n", - "`--NotebookApp.iopub_msg_rate_limit`.\n", - "\n", - "Current values:\n", - "NotebookApp.iopub_msg_rate_limit=1000.0 (msgs/sec)\n", - "NotebookApp.rate_limit_window=3.0 (secs)\n", - "\n", - "IOPub message rate exceeded.\n", - "The notebook server will temporarily stop sending output\n", - "to the client in order to avoid crashing it.\n", - "To change this limit, set the config variable\n", - "`--NotebookApp.iopub_msg_rate_limit`.\n", - "\n", - "Current values:\n", - "NotebookApp.iopub_msg_rate_limit=1000.0 (msgs/sec)\n", - "NotebookApp.rate_limit_window=3.0 (secs)\n", - "\n", - "IOPub message rate exceeded.\n", - "The notebook server will temporarily stop sending output\n", - "to the client in order to avoid crashing it.\n", - "To change this limit, set the config variable\n", - "`--NotebookApp.iopub_msg_rate_limit`.\n", - "\n", - "Current values:\n", - "NotebookApp.iopub_msg_rate_limit=1000.0 (msgs/sec)\n", - "NotebookApp.rate_limit_window=3.0 (secs)\n", - "\n", - "IOPub message rate exceeded.\n", - "The notebook server will temporarily stop sending output\n", - "to the client in order to avoid crashing it.\n", - "To change this limit, set the config variable\n", - "`--NotebookApp.iopub_msg_rate_limit`.\n", - "\n", - "Current values:\n", - "NotebookApp.iopub_msg_rate_limit=1000.0 (msgs/sec)\n", - "NotebookApp.rate_limit_window=3.0 (secs)\n", - "\n", - "IOPub message rate exceeded.\n", - "The notebook server will temporarily stop sending output\n", - "to the client in order to avoid crashing it.\n", - "To change this limit, set the config variable\n", - "`--NotebookApp.iopub_msg_rate_limit`.\n", - "\n", - "Current values:\n", - "NotebookApp.iopub_msg_rate_limit=1000.0 (msgs/sec)\n", - "NotebookApp.rate_limit_window=3.0 (secs)\n", - "\n", - "IOPub message rate exceeded.\n", - "The notebook server will temporarily stop sending output\n", - "to the client in order to avoid crashing it.\n", - "To change this limit, set the config variable\n", - "`--NotebookApp.iopub_msg_rate_limit`.\n", - "\n", - "Current values:\n", - "NotebookApp.iopub_msg_rate_limit=1000.0 (msgs/sec)\n", - "NotebookApp.rate_limit_window=3.0 (secs)\n", - "\n", - "IOPub message rate exceeded.\n", - "The notebook server will temporarily stop sending output\n", - "to the client in order to avoid crashing it.\n", - "To change this limit, set the config variable\n", - "`--NotebookApp.iopub_msg_rate_limit`.\n", - "\n", - "Current values:\n", - "NotebookApp.iopub_msg_rate_limit=1000.0 (msgs/sec)\n", - "NotebookApp.rate_limit_window=3.0 (secs)\n", - "\n" - ] - } - ], - "execution_count": 11, - "metadata": {} + ] }, { "cell_type": "code", - "source": [ - "for w,p in w2v.most_similar('neural'):\n", - " print(f\"{w} -> {p}\")" - ], + "execution_count": 12, + "metadata": {}, "outputs": [ { - "output_type": "stream", "name": "stdout", + "output_type": "stream", "text": [ "neuronal -> 0.7804799675941467\n", "neurons -> 0.7326500415802002\n", @@ -752,70 +324,94 @@ ] } ], - "execution_count": 12, - "metadata": {} + "source": [ + "for w,p in w2v.most_similar('neural'):\n", + " print(f\"{w} -> {p}\")" + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "We can also extract the vector embedding from the word, to be used in training the classification model. The embedding has 300 components, but here we only show the first 20 components of the vector for clarity:" - ], - "metadata": {} + ] }, { "cell_type": "code", + "execution_count": 13, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "array([ 0.01226807, 0.06225586, 0.10693359, 0.05810547, 0.23828125,\n", + " 0.03686523, 0.05151367, -0.20703125, 0.01989746, 0.10058594,\n", + " -0.03759766, -0.1015625 , -0.15820312, -0.08105469, -0.0390625 ,\n", + " -0.05053711, 0.16015625, 0.2578125 , 0.10058594, -0.25976562],\n", + " dtype=float32)" + ] + }, + "execution_count": 13, + "metadata": {}, + "output_type": "execute_result" + } + ], "source": [ "w2v['play'][:20]" - ], - "outputs": [ - { - "output_type": "execute_result", - "execution_count": 13, - "data": { - "text/plain": "array([ 0.01226807, 0.06225586, 0.10693359, 0.05810547, 0.23828125,\n 0.03686523, 0.05151367, -0.20703125, 0.01989746, 0.10058594,\n -0.03759766, -0.1015625 , -0.15820312, -0.08105469, -0.0390625 ,\n -0.05053711, 0.16015625, 0.2578125 , 0.10058594, -0.25976562],\n dtype=float32)" - }, - "metadata": {} - } - ], - "execution_count": 13, - "metadata": {} + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "The great thing about semantic embeddings is that you can manipulate the vector encoding based on semantics. For example, we can ask to find a word whose vector representation is as close as possible to the words *king* and *woman*, and as far as possible from the word *man*:" - ], - "metadata": {} + ] }, { "cell_type": "code", - "source": [ - "w2v.most_similar(positive=['king','woman'],negative=['man'])[0]" - ], + "execution_count": 14, + "metadata": {}, "outputs": [ { - "output_type": "execute_result", - "execution_count": 14, "data": { - "text/plain": "('queen', 0.7118192911148071)" + "text/plain": [ + "('queen', 0.7118192911148071)" + ] }, - "metadata": {} + "execution_count": 14, + "metadata": {}, + "output_type": "execute_result" } ], - "execution_count": 14, - "metadata": {} + "source": [ + "w2v.most_similar(positive=['king','woman'],negative=['man'])[0]" + ] }, { "cell_type": "markdown", - "source": [ - "An example above uses some internal GenSym magic, but the underlying logic is actually quite simple. An interesting thing about embeddings is that you can perform normal vector operations on embedding vectors, and that would reflect operations on word **meanings**. The example above can be expressed in terms of vector operations: we calculate the vector corresponding to **KING-MAN+WOMAN** (operations `+` and `-` are performed on vector representations of corresponding words), and then find the closest word in the dictionary to that vector:" - ], "metadata": { "tags": [] - } + }, + "source": [ + "An example above uses some internal GenSym magic, but the underlying logic is actually quite simple. An interesting thing about embeddings is that you can perform normal vector operations on embedding vectors, and that would reflect operations on word **meanings**. The example above can be expressed in terms of vector operations: we calculate the vector corresponding to **KING-MAN+WOMAN** (operations `+` and `-` are performed on vector representations of corresponding words), and then find the closest word in the dictionary to that vector:" + ] }, { "cell_type": "code", + "execution_count": 15, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "'queen'" + ] + }, + "execution_count": 15, + "metadata": {}, + "output_type": "execute_result" + } + ], "source": [ "# get the vector corresponding to kind-man+woman\n", "qvec = w2v['king']-1.7*w2v['man']+1.7*w2v['woman']\n", @@ -824,31 +420,20 @@ "min_idx = np.argmin(d)\n", "# find the corresponding word\n", "w2v.index2word[min_idx]" - ], - "outputs": [ - { - "output_type": "execute_result", - "execution_count": 15, - "data": { - "text/plain": "'queen'" - }, - "metadata": {} - } - ], - "execution_count": 15, - "metadata": {} + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "> **NOTE**: We had to add a small coefficients to *man* and *woman* vectors - try removing them to see what happens.\n", "\n", "To find the closest vector, we use TensorFlow machinery to compute a vector of distances between our vector and all vectors in the vocabulary, and then find the index of minimal word using `argmin`." - ], - "metadata": {} + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "While Word2Vec seems like a great way to express word semantics, it has many disadvantages, including the following:\n", "\n", @@ -860,11 +445,11 @@ "Another method, **GloVe**, uses a different approach to word embeddings, based on the factorization of the word-context matrix. First, it builds a large matrix that counts the number of word occurences in different contexts, and then it tries to represent this matrix in lower dimensions in a way that minimizes reconstruction loss.\n", "\n", "The gensim library supports those word embeddings, and you can experiment with them by changing the model loading code above." - ], - "metadata": {} + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "## Using pretrained embeddings in Keras\n", "\n", @@ -873,11 +458,24 @@ "### Using tokenizer vocabulary\n", "\n", "When using the tokenizer vocabulary, some of the words from the vocabulary will have corresponding Word2Vec embeddings, and some will be missing. Given that our vocabulary size is `vocab_size`, and the Word2Vec embedding vector length is `embed_size`, the embedding layer will be repesented by a weight matrix of shape `vocab_size`$\\times$`embed_size`. We will populate this matrix by going through the vocabulary:" - ], - "metadata": {} + ] }, { "cell_type": "code", + "execution_count": 9, + "metadata": { + "tags": [] + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Embedding size: 300\n", + "Populating matrix, this will take some time...Done, found 4551 words, 784 words missing\n" + ] + } + ], "source": [ "embed_size = len(w2v.get_vector('hello'))\n", "print(f'Embedding size: {embed_size}')\n", @@ -895,33 +493,22 @@ " not_found+=1\n", "\n", "print(f\"Done, found {found} words, {not_found} words missing\")" - ], - "outputs": [ - { - "output_type": "stream", - "name": "stdout", - "text": [ - "Embedding size: 300\n", - "Populating matrix, this will take some time...Done, found 4551 words, 784 words missing\n" - ] - } - ], - "execution_count": 16, - "metadata": { - "tags": [] - } + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "For words that are not present in the Word2Vec vocabulary, we can either leave them as zeroes, or generate a random vector.\n", "\n", "Now we can define an embedding layer with pretrained weights:" - ], - "metadata": {} + ] }, { "cell_type": "code", + "execution_count": 10, + "metadata": {}, + "outputs": [], "source": [ "emb = keras.layers.Embedding(vocab_size,embed_size,weights=[W],trainable=False)\n", "model = keras.models.Sequential([\n", @@ -929,47 +516,47 @@ " keras.layers.Lambda(lambda x: tf.reduce_mean(x,axis=1)),\n", " keras.layers.Dense(4, activation='softmax')\n", "])" - ], - "outputs": [], - "execution_count": 17, - "metadata": {} + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "Now let's train our model. " - ], - "metadata": {} + ] }, { "cell_type": "code", + "execution_count": 11, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "938/938 [==============================] - 10s 10ms/step - loss: 1.1075 - acc: 0.7822 - val_loss: 0.9134 - val_acc: 0.8175\n" + ] + }, + { + "data": { + "text/plain": [ + "" + ] + }, + "execution_count": 11, + "metadata": {}, + "output_type": "execute_result" + } + ], "source": [ "model.compile(loss='sparse_categorical_crossentropy',metrics=['acc'])\n", "model.fit(ds_train.map(tupelize).batch(batch_size),\n", " validation_data=ds_test.map(tupelize).batch(batch_size))" - ], - "outputs": [ - { - "output_type": "stream", - "name": "stdout", - "text": [ - "938/938 [==============================] - 6s 7ms/step - loss: 1.1098 - acc: 0.7849 - val_loss: 0.9145 - val_acc: 0.8159\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n" - ] - }, - { - "output_type": "execute_result", - "execution_count": 18, - "data": { - "text/plain": "" - }, - "metadata": {} - } - ], - "execution_count": 18, - "metadata": {} + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "> **Note**: Notice that we set `trainable=False` when creating the `Embedding`, which means that we're not retraining the Embedding layer. This may cause accuracy to be slightly lower, but it speeds up the training.\n", "\n", @@ -980,29 +567,58 @@ "* Load our dataset with the vocabulary from the pretrained Word2Vec model. Vocabularies used to load the dataset can be specified during loading.\n", "\n", "The latter approach seems easier, so let's implement it. First of all, we will create a `TextVectorization` layer with the specified vocabulary, taken from the Word2Vec embeddings:" - ], - "metadata": {} + ] }, { "cell_type": "code", + "execution_count": 12, + "metadata": {}, + "outputs": [], "source": [ "vocab = list(w2v.vocab.keys())\n", "vectorizer = keras.layers.experimental.preprocessing.TextVectorization(input_shape=(1,))\n", "vectorizer.set_vocabulary(vocab)" - ], - "outputs": [], - "execution_count": 19, - "metadata": {} + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "The gensim word embeddings library contains a convenient function, `get_keras_embeddings`, which will automatically create the corresponding Keras embeddings layer for you." - ], - "metadata": {} + ] }, { "cell_type": "code", + "execution_count": 13, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Epoch 1/5\n", + "938/938 [==============================] - 20s 14ms/step - loss: 1.3377 - acc: 0.4978 - val_loss: 1.2995 - val_acc: 0.5647\n", + "Epoch 2/5\n", + "938/938 [==============================] - 10s 10ms/step - loss: 1.2587 - acc: 0.5722 - val_loss: 1.2339 - val_acc: 0.5842\n", + "Epoch 3/5\n", + "938/938 [==============================] - 10s 10ms/step - loss: 1.1980 - acc: 0.5884 - val_loss: 1.1826 - val_acc: 0.5954\n", + "Epoch 4/5\n", + "938/938 [==============================] - 12s 13ms/step - loss: 1.1503 - acc: 0.6002 - val_loss: 1.1417 - val_acc: 0.6018\n", + "Epoch 5/5\n", + "938/938 [==============================] - 11s 12ms/step - loss: 1.1120 - acc: 0.6097 - val_loss: 1.1083 - val_acc: 0.6104\n" + ] + }, + { + "data": { + "text/plain": [ + "" + ] + }, + "execution_count": 13, + "metadata": {}, + "output_type": "execute_result" + } + ], "source": [ "model = keras.models.Sequential([\n", " vectorizer, \n", @@ -1012,50 +628,18 @@ "])\n", "model.compile(loss='sparse_categorical_crossentropy',metrics=['acc'])\n", "model.fit(ds_train.map(tupelize).batch(128),validation_data=ds_test.map(tupelize).batch(128),epochs=5)" - ], - "outputs": [ - { - "output_type": "stream", - "name": "stdout", - "text": [ - "Epoch 1/5\n", - "938/938 [==============================] - 7s 7ms/step - loss: 1.3381 - acc: 0.4961 - val_loss: 1.2996 - val_acc: 0.5682\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n", - "Epoch 2/5\n", - "938/938 [==============================] - 7s 7ms/step - loss: 1.2591 - acc: 0.5714 - val_loss: 1.2340 - val_acc: 0.5839\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n", - "Epoch 3/5\n", - "938/938 [==============================] - 7s 7ms/step - loss: 1.1983 - acc: 0.5883 - val_loss: 1.1827 - val_acc: 0.5951\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n", - "Epoch 4/5\n", - "938/938 [==============================] - 7s 7ms/step - loss: 1.1505 - acc: 0.6001 - val_loss: 1.1417 - val_acc: 0.6021\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n", - "Epoch 5/5\n", - "938/938 [==============================] - 7s 7ms/step - loss: 1.1122 - acc: 0.6093 - val_loss: 1.1084 - val_acc: 0.6103\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n" - ] - }, - { - "output_type": "execute_result", - "execution_count": 20, - "data": { - "text/plain": "" - }, - "metadata": {} - } - ], - "execution_count": 20, - "metadata": {} + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ - "One of the reasons we're not seeing higher accuracy is because some words from our dataset are missing in the pretrained GloVe vocabulary, and thus they are essentially ignored. To overcome this, we can train our own embeddings based on our dataset. \n", - "\n", - "\n", - "## Training your own embeddings\n", - "\n", - "In our examples, we have been using pretrained semantic embeddings, but it is interesting to see how those embeddings can be trained using either CBoW, or skip-gram architectures. This exercise goes beyond this module, but those interested might want to check out this [official TensorFlow tutorial on training Word2Vec model](https://www.tensorflow.org/tutorials/text/word2vec). Also, the **gensim** framework can be used to train the most commonly used embeddings in a few lines of code, as described [in the official documentation](https://radimrehurek.com/gensim/auto_examples/tutorials/run_word2vec.html#training-your-own-model)." - ], - "metadata": {} + "One of the reasons we're not seeing higher accuracy is because some words from our dataset are missing in the pretrained GloVe vocabulary, and thus they are essentially ignored. To overcome this, we can train our own embeddings based on our dataset. " + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "## Contextual embeddings\n", "\n", @@ -1066,30 +650,32 @@ "- John wants to **play** with his friends.\n", "\n", "The pretrained embeddings we talked about represent both meanings of the word 'play' in the same embedding. To overcome this limitation, we need to build embeddings based on the **language model**, which is trained on a large corpus of text, and *knows* how words can be put together in different contexts. Discussing contextual embeddings is out of scope for this tutorial, but we will come back to them when talking about language models in the next unit.\n" - ], - "metadata": {} + ] } ], "metadata": { + "interpreter": { + "hash": "0cb620c6d4b9f7a635928804c26cf22403d89d98d79684e4529119355ee6d5a5" + }, + "kernel_info": { + "name": "conda-env-py37_tensorflow-py" + }, "kernelspec": { - "name": "conda-env-py37_tensorflow-py", + "display_name": "py37_tensorflow", "language": "python", - "display_name": "py37_tensorflow" + "name": "python3" }, "language_info": { - "name": "python", - "version": "3.7.9", - "mimetype": "text/x-python", "codemirror_mode": { "name": "ipython", "version": 3 }, - "pygments_lexer": "ipython3", + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", "nbconvert_exporter": "python", - "file_extension": ".py" - }, - "kernel_info": { - "name": "conda-env-py37_tensorflow-py" + "pygments_lexer": "ipython3", + "version": "3.8.12" }, "nteract": { "version": "nteract-front-end@1.0.0" @@ -1097,4 +683,4 @@ }, "nbformat": 4, "nbformat_minor": 4 -} \ No newline at end of file +} diff --git a/5-NLP/14-Embeddings/README.md b/5-NLP/14-Embeddings/README.md new file mode 100644 index 00000000..e69de29b diff --git a/5-NLP/14-Embeddings/images/embedding-classifier-example.png b/5-NLP/14-Embeddings/images/embedding-classifier-example.png new file mode 100644 index 0000000000000000000000000000000000000000..3958e3fa36d60f75f226da10e97c10bd74f355c5 GIT binary patch literal 39645 zcmcG!2Ut|g(>DmhB`6?>lCzRTK}kc-5+tc4$skCEA;Z9s1VJ)L5{4*Avg8~T$qYG2 z9CFS>8seV8d++;y-*0!HZ}-{ddFVOaU0q#W^{eVSeLDD+iX7o>s@oVC7=#M)GH)<2 z2xc)buotkgfR?9OLxI2#wz;ITBnC!lM~!^JoufeDq35{l)e5HSjn;KTl6jk3=F#NJzxQ#0(4!pipRccXwi9Vrgk9cEgY& z1_lF$f{diP`{dRP+-|C!3TKXCfRKP-hwumWeBmuAojbQc+V^ht6XKL?j81BdrGLr; z9Q5oIfC)SUrNk`8f0hhj44~!TG75x#2FgzA|4#ic#UJW_>HeYqGv>d9 z`6JlB!u%2JA7TCo_Wz$qzAF`xu|rU4))jc=5C-^M!8MjXdo~Yf!IQM~1zoQ!u{HXa zgvR1Tfmrymqd_f^s#YDRE|bDt)Y+Is>|pQtpL<+-cYVkqnY=B=%kVq{VfjWJ0x(eZ zoi7@HnK|w|{}ddH^Q_@W$MTSh0Nm&rdmw9x(W+Y#CtHHqsyoX@Xym?GIn+rF zkWET%@EN)H0PF!`O{zv5B`cygXsVJpG?)LS8uh_@5ksrCiw^tr?l1?pE z)ol2!i@Fhq$OIUfwiKSIRpJMtntf3}JkQ$VTa#p!HHKMOB?)!l4%)AI2wQbhF9Bx8GG) z3h3@+t%3Q7_|Kn)nEM1Ypo%wPSD{>uZ~a}6<;NOG?R`2ALK>^ttd9Jrw&HmlRmVFf z2E84S+4_bFNXMKp4Z=}u#?rQ*)nT{+)HKx{dw5au``G+q-%KZ$Ue(971mQB<(Bxgt zrev@?#-PC%$3m~VdtUzlmR0WKjj}%`0=V1F#u~3fS;5^~O@`CcV~hm-yH&Y)U!8Qd z7{l79iu@;>X+VS;UM_|~EKp&#S1*o-F?wE7a_jD>dE^+;{O2t|)uIx##fGT0b`LvS}@>C+efmWzRusT9x%) zWqq-v7xMJX=$>x3dY?NpL_=goi0^Lh_yNwT285?1<^pvDPq2KDUq_d1xpC!yJZyYh zqJo%Ghq+ZW#fP>FH40pKl1dM^MO<^Tq%HIcd8|+3Bm8ROhG4}I*}HJ*=u@+DD8P&UbFXrnzjRf)I>vTAxn za&E>`)6?XiliEx&OP0}P^6c^F_HA-pNw42no~)*)yi`@|UolQsXptaPh|Vx5vUHMD z;`&)JseCy2&=G6(i5F7%m|UpU((0Nw?)y?`e;7e$o?&m!*Kymq00Pf)Olw?>J$U^A z?7|$f_~ZoMTeHnVJ~=yj%2IqP`!qyI74hkZ&o`droMTlUFnDA0L(szMw)8gZlQ(Zd zS+hFr?vW!f=3tixuxe;|2qEIW;^>mRCJ3e7~i`tE}+#=e33)5{t}@Vu2+&wYA`cm)1n=NFy$dg86S5OyKtCB zg%2Lr2UYQn$zg6J$Qo)(n|Gi}Fq z(45cFf`iNEpX+e;7hi>UgPjA1M=ruY`9+)DUiI|gmJx2rx?V=DEyC#X zhm&{H#H#nC#`C)O;*_DrhO49+vK>WdEY!p)ErP-bo5R3frdE9Fq~_@7Y|V2e%O5OI z$+56gWC{~#Bf@MbP0Xz@B})5mp-{@69paL0pbThD&Ntji9w_P4@qCPke1-3euuVgH z@R8Rl+xqe-&Bcq73=}K`d~msuT-*EnbC-uDS+k{JB$meQcl%}fo=5hqrl~WXZ{%oC zl_(hJN02bbE7{pbl7#NRmj5a$@3Sr(Ic?wdt{JtvmS?9DRhltd0XNy9+AAXS<>ovT zeCj0(YADxq1lQ|K76Tgk%c-?7iNs zU)6vk(we4vuPhruUW13hdW!{uPOl?X(M zMM-haOlnsI?JMzHKxqO#nO>xx%RUol6|Ww=M`|#qNHRCq5+G`VPL#K=EFFbmAn5RI z(4zNe8*=S9$=|(a8?Lz^QG3B#`@5c}EACL9HjsT%Q(j{8>yzP&5sBt@$i4OO>x(gu zwVpjVn_bQoeClLa{c-uUQmp+Iw;Ubo7T$6v!RmmU-qHnJ#4JmkNQD2pe@t!KaR-Xp zg^M+XVr`RME=-^+#o$P9#&E7t16&23q(OQ32~c*xdlAq}5aQE)bH0S(li6!m%@JW<<`JHv9nD3!&h};15Cwz3aj&S2jg? z*HCN6JDYz+TtNEW-A|^izg>kl)V}rpA-o)T802yPs=0W_rktunR$&Y)HA*FQ{@JwG z&YT3soH`OVc{z+@2dC@b;Gc3baP3v`b}DQdJfFxyp0C9mt4>YsGpah!fv1xuUHi!Q z3%sX(;wPF55V)PN;%8SLdTsuhE{fL0wy} zum4n|Hf#U@G(pfj5dxzL^~&31?BS`fD=Qx={P~(-LIR?5E9b`<$fW!VzB_4Ub@cI@ zu9oJ9@#Ipp~DjJeb{)fhJBU_2uw>&SrN04jZeh_6Z8bAt{r6-`A(@7cV*lE?U2W zm)w3V2BH-q)-;iO6`th4N&ni=>>5M|(aN0Xbw`Hx!$Cc5UZ&g9V|@s=4y6OIFuSIe zKuJm4xLq;%T2cKnxIKRCjwxjyu&*pow}683e1_{@>i1t~%e~Ew(K`Ozgi^emz+c9S z8a{@Bp=M2%M7{YyiEYstUlyr){XRC3AW z8n1wgG^%*atA{7DP;)Lv)fAmgJ=G-AI!p$8#P@eon)egW+BbsK3ozByd9`gc9fgAu zY(Od8k(4#nq1hfOv;DI4yW!LG7cXdMJ8f|b&v4KtnFgiSj+%jl3%w88@UpMcSutmm znqT|O6D_BvlzmFLeq}0QXGlYHVgly8PCQ>|EC*c;$gU@BPz(C-@JjLbzah&w$tsaD z+R|kW^~}$V8I%}n=vjKCmY(IqN|amn$nWE22IT^JCMraY89V0oZS!Jps-@+jRDY&s z{0%6Q4gBRuL!WQFCl$+~$rQKkuS5H%yZh7RyD5$PK|6W+j@~g8j6G_JwHI|Rcpfm=`gfZFLQ zXEbCoLy zRZa0$6?qIDJXB+%g)Zvi+MW4HDZe|PO^bd+i#$v}@^#zF7Zta!65a9*>X0~S!(|>Z za(l+kl*sCJR^&PY&XrkEP_hsA$q?gziuxTXMr>pF#*{IlYWRx;ZXu~`?jTy9RsK{( zEW8y|2#kToeZnFfD3&u=N7-VJunDscRJ0;JWCg{NIB5q30FWJMwB*ak7(K=Dl?=bD zP=JZRn4v}xQ?-C3PI5+}@;c#QjJ4U*zyXY#v46PEz%-GJmV9&6>7I+!Qdp-rYqJ$e z0|Wn)D;h!Q9)RsW;eWCX2-F1fM;tz=^v!%QP167S`I<7&k`SV~{SQgd>&w9Wdwmvs z|E>=Q7u^Pxw!i?^gXaIwIsi4Jb-==VigVM0UPlw=|FVvj^!A{vQD1QSj~y>Wt$+vX z!p%DSHexpn8FmzIr0_!uWpjQW-n$XtG*B{pV2&D!wU?%nH)uX9glh1a1d}E`Y%~AV z3bN0EE{yCwzWjxV*jgdpbzj;}m1^($K7xikzCIamHeG8_&@;Bj&vh{z4 z{QIPQ@NXNU_x9hj7rs{dM~*lDJZ-&J`Mb^LY=!=K$DB3VCK5}4@#xkAwR0i|X}!eT z(?yIn9@>n(ccR+Qv5j`H!hY31Y|Mc8IyjFPkQwfzs=e|VNz4+i-Qc}MnQ~{oC8JwC z5k@5EKU|W;gYE5RG+;uoO5RUz9-U<*jn6lmG!sRE(vu{EK3L&6vwbrg$VgAZqz(Dc z%I$ngbdN;PSD@$L3P68T`9}1D4@>C2I?vjxS8(6|UioX30Ns-={Tk@`w<5{skFKn} zLihYvrQ8q;%>v!@{|UeUWd1K(^tAt0{!e9o!B=pj1j(R{{~_T2Yli=^oPU+@zoC(T zFfSrHh_&3Q}qZK}@DA1qs3nZLs2_TjDyY)}~Pibb5i39O(cy9H7-Fd_B zl%%37zC?h=v*J~h8L>woDn4ptq3swGR8 zs*`dj;BDnG;^KO==Io(t zAP8&mbpcaNot`TV(eWA|;k5XmH1s(YjmE4qIm%glU6mF2*#0y4g%E1G&=<$@*!%(K z<))-&Y>;6wEav`mFr84Q2Bhk7;aT~Q3buZ+6unBWfti({juXR%!c;e5`q)TSg&>^`1{ml#E)Bp#yr?$2U=^Ip9~e&N{n6K zZ)H_n?BZ%3>&YNi+}}GSev108BNFabzOla?xg8x3VgR28ttvaxvYbGO$C+C#K<@+se zDbTgIQ#wvgi&cnRe=PyPvt;V4=^%K`vRa`OeX4a$;-~W{Rvq z#Pgq#F|CMe%oe^ojYZU@WXE{0O6du|*V^RHHG|IsM~j3VQJ@r4y{GziFy&kO@yd;o z+a{mmbv_kN%qwtS4zBMX8z2t5%sBE#PPU6k^?pI?R93TT=@?N~@q>IT^zy`2XPtEg z2U13HVjmJqEgkZliWLUCMsxK$jA{|#;CTc4;=6_kUM?K{#`?aU-I81lF4=*}&{s`n zQ!?l|za8XM*BiuwWn=g^VP=bO&$~6>hF93Ze-*mBmo-iJ#6Kce@H)8CN8t$Uzt7yf zyj9dV##o-uQM?Dc5_r7f7ed?UqT1HXATeXI|5uya-CHegsWCcT`$yoq!& z{N*;@b(a;q9VBMCTQ9bxJRRYMC{X1DOU&BqK}Zp0EIkH5<5Nuoy3ncN{Dd8XWh#Vt z`e**y*DLUzgx_0StzE`4dE74jaOCW!)oN zZ_HWO2SSNl3Cjlv%vOP~zbW-oW4dj&D z^&Aw0&8MI$>`eYj>zjAVQ$;vv%mI^1&coME4j1MeB$8I=Jv5dPRJnBLZKC&X4Abx_ z4hEuyYZG>N^if*mMIWe%Gs>!ce0W%0>~ObA)~MkgLf*pkY28fXj@L)eN>>&icZ8I( znegSxahoh5kaQPsQ4dN?sn2zSTy!l&MU_!G0`xA)UF5Ii9FweuybuI!GLI1qeeq|z zKdU$2k^!$FdQ%|Sw*;>ol&nFc54x{!f+)5oPNDBmxl?kU)!8`pOpjO^8*Njxo^tvHv-F50|j+guid{5E&q_AvEHp4YY z{HwIEB@}p@m=i^rSM%)E-oq zlAUm!<#2`0UWL2Ps;NvWsboHaoJRF&&Kuf|0IxW68%-pHyn?r6Nsj!<^GwUBPV&Wi z=t9T&cb$Kxf>jPOR$mVr>-5E01>0{r72Q#Pfd!Z5xRJXG>a>Pyv}!hdkQ3^*(~YZG zawc-Y!@a0qMEUTowQXj9%1I%VbUYz|KFeU$xK!y@;)L_a zr?PR$U5{d)jSb@tR~%AtmtB9BYYbZQ=qdPX`TkC?4__0+<5|)5H=zN~JgFHRRmr2T zcKYRJrRAOnirf!uDL1D@n^gnQn!otm6#Y3V?WM*m9zG7%rA%+==cE2#6lE(9xNJO4 zr%PcQBN*>;~fo}%=eKxUFt2MLw ztux1F@|y$;t#0alY6~f{qpxQ7UnqQ>DnMz^$XL1ND7YLwyzwUCrit-OF!y*x1KdB4 zcBgP^oMwBATII2*?Z#x^N^biL*=Pkw($cqJUu^luQ40C=U1lnzWfZDuF~vobS(_6% z8?yBq_@h&l`ZhO)oNS`RUI>s0G55$8<@02{ptUS<<=qal9%ZU4-fplquoTd6iwg2o z9-^ro`%JGKUxsw*l*Q|r-VRu-%(Ivm6BYT994q#>XwUf_l%3h*dnO15} z&LCELRx80L!R6AX@02o~l+(i*=s^j6vaU%L_*j8zro~bQB0EGgh2^E6m5;)pkRPwH z=BRzG&&`Opn8)l{?@ZDE{+P=u(_;x6wmsj#zxUA{Ra~?!lAN>(Rl_&up)F99Ynoru zKf-uuqca_=;hw!*NqKDcI9ipT<7mk01}r7Jupckqh1;-{hLpk&HJ)UCV~QHCSkic2 zM3;77pXrDF4&9yymt8DUol_qb=NYo~$n<%#@^Yr?!%O^osHM>5k8(G$WE=LXjmqyf zt5iL?MyKje@>HXsNmT=zZpwV_feb^$_F-GN5&gP`4UX}v@$h5dETzeh8VMhLP9S}r z@R)xNarF%oeIjVRX;wL8ov8pzp|>l)asD|NxtCh}8Oz`^9`qq=Mlo<;?v(7=aF&^6(sz_ zkQLj~d2N@yQjipUM0XQP5qhvag#Z(6mo#dQYy5fOkg5gb3s!lSY&jz(mlad<@sjgz zI`J0#V@QtBOV5}#WTy{(CU89B3ofRgFANolvlEP&=<^2s+eUNdiIZ{d{b%GeD?sv= z-X64`o*}u>wbQ`YSb*qe251W3#00d=82_ziwDKN}}pEZrUszGUh4bgO5$eK zunol zY0V#z8-ChRR>DG|@3&))R==sJ_8hqg;5%w+M3GWK8-T^9vp9DvGlvQY%J3@0zn}c7 zbK#Puv^XPt_bj--4omWF&SQTgb#`mv{%P+fo!TYjU%(0A&=;PH6h~n&$hPp2%CwEn zd5-sh=P@llbfLpI-#oJ`L$xb7o`!D|CQgC9smz?573#?1$i_PN3Vo>OTs}#rrgeW1 zo7sSWigqi;RE;F4l7bR;#x<`-J68i|iQrHMt4BQ4HV=ShDNF#PQwdH3_vxx2|i z*dg?|_vYS~Kb}*7{lak56txr=dpvLR2|n$FvJ#^g2Y`6;Q-K5%1lv|GRrVBB)v%9Q z%GS>_K4mU5%am;NdPptMge~9}sUdG7tlS3s4evHeIX91WjV)R6KpiLEC}K9q`l@4g zVo4yx#<)WL9TwepKVs5ARYsmF;ut!ePm>Za_cVACkv;cv zEOJWobW?H)eRh{fZ(nV7J>T8b#XIqq-F{mCh~w@TEiB*;qBq9X>jXR*gU?#;xQ2z6 z_C0!39lL+dM;p~%cO{35G#TRxG=CV@oECm)$LTGS6+BFD=Mp2`U6;rm@A#|)2RMfk z;ew0kOhn`DK`JhaEb792lToC&RV2!Y_t1-@H`Snn(?I&Uw{k}{%A08^{JCRyVOl&1 zjttT0+mNYbj3l3g?)f&C^@I zxlRize}C7t)R8^?6|VgLP%c{T%MQyMQI&vf8sK8(oMlZEo((D^+&jhG$@2L*1P0Hx zC;1bf(oXKu|B1#^8%TpEjUd2E%uw$YfbdjB?sF=@G69zjz z%@J&=l_z-CpA)7Ow`<;xU$dxwDC! zf#%`NOqohWptzr}hbW|#V9@aaE9WO<+xUx9dDr^b@|R`z?pdP6bXb(7o9tZq6>|^s zvQ9;X>B&Na{TbM%l$ol!c;)Mr518${B^F<9ysg;v&nJx0J5QDa?*AG2To)$9K!7Zd z#ryKwG1kd)9mdm?RHWCNSG2WMF35MZ8Ja5iSNC$gR&_$ zH*IDF#CPVzUxo@WXU$^aJkla}Bo?oM{L=iwuV7{9;9ZiHX14`Qb7`{I=-zrl6YH|k zjT|-oV}aY1&UbB`;9Xyg~tmLywW6}3^rzu}D zNr*?|QIJF41lRJOq4XwOtg5=-_T{yW_vR@;bLyTy8hYD}zSFx?SRQ@PPvfvurlqDK z4Yp$Tz&vR4&HSWQlM3a``0`$MaR@#rRj01TGt{!Fqn-i|*;%SkxbY`BIhDK$%|aYIq?56lqL zlWQxhk<7QuoSatN8mg3VL{<3;0G4T@lmb&WrS<+5U;|tKT|utR-#s>flkvO)4lzo$uwX$31uUBR!7a~KP$`Y z<+f5_ws~#!fR)bm)5BYv?n94;UVo0m?Jd9rFdxb-&y>FCW;)RN#>j#XSO5K?uOJ_4 zN;_-g-ntn(F478ZV{UB|dF-k*MeMMg%sz}#=a>dTCoQLfFgl^!9D^TG^5xmdAgO^R z^d0Ft75t6&Fj;G49oFezI7ydx1;9*u(Chg@4I);4${@-31_xhC%IDn%aTKr*d-!3B<v z!l-C15>A|bI01))<;7` zNi@C%U*>igq@wz@Fb}3+b8$y!x_O>iL0DUIlGkE#Qpk6c@>r++ESfuU*udp?C>`-j zrtd(IJSPd_|Eemq?e4^*hX%j3lT~ZnU>dLBNR&J|8#7>-x#7^&k21cZ(nS>H5B#{^ z4;UY;Ov|Ai&lc&{lVj=d<>TR@)ypgnz>dH*dHVL&pnHA}a&uR&A883l;fU(;RXbUk z#0`2$!V?_pkXC>ZfmNP;k!3RC2f_7{;44G?ZtC!6H|NF&-4@&v^J2Q)3YO=1Lc%KF zW3d2PzbG-03~)6o;mkCQuu&c+k1EMC^S>P@?aixHCYp*6ZEWAJ(J7DhZ$a~Qe|D&V z-7tR-bMZwSK~l~Od+?}DE#Ppe8^m%A_O#PC{`?ZStG(B{o`5%f_REyr-)?27u)e^o zATuo0R7F{Fcc&rgEgBzat@cq#GZ}FNw}h7zm<|{V{m`3yGCKV@bNgCla^b=%-m&hU z__IvFi^6SJTr6~e?$S#XD?YA)^w^5Aywwj&2en7bCgE`iUKeSY&NTd zy_s=2GILeZ)DETZxh<}YDxb}gpXkA@9+j?5sj(R)#*7X=MtH#zrZ2HC zrnmwzVTw}+tkI)zW&82b>EeY~NlM&hMQ(x-?9s)rWWb>TqlvRvv&7@A(QjaB^<_gK z4%efFv4d4w(UcK~%E1-e+2Jx&%0v>gzN?Y2VCGvwoXG5bZ%ZJaW2u+FkuCX&ujO#) z%-Z3~Zf<=inD-Vgpb1hZMv~tzgIi?Cn|;;aafYi1Ch%MHsg{K8;PaKnngAgWt*$j2 zv3NWsB<|^IPOWy^gmDDNiO8r(Eq5L3@UJ(0#Sf!%=x&@snJ7&RoZ!e*8|yD@jm~`7?Upjali^1>QFv=`xMX+wa0oUm zD1W7xTc~*Dj1DijqBa&Qm>0b&OIth#;UIoBAnK3-gw;wom+qszv4ij_q`9hDL(gdW z-;}I=d(zaCv+c(B{NBB*4RPr-xuWT3x<3lrE3n z>0D>N)&3$jd6+NFEpxpd`DN5bqkdv_L_Nmj$?|u!GxM~`2mxmMF5;*A+)&wbmnCG_ zu^Yl0+;2Ki6)P!j#ixtx2jClV%0SE>&1z(k2U?tnRTyd#`)GAp$Zv2o@c?-D#QJ$$;6*VBgtknveGa&EiN!T=~vjLLaV< zG+6EIMx(N5gB4BpXbV2mbHk}0>-}C0bMLx59tz!n7Po0$w3ejm#?ZDA2ka=sq zW5>Pm47U$;N{tmT{&HZ$<0#urb)S|oF5^lqNs8n4cv1f0$+&T6cJen{$Co$ysJZh& zx>?3(Hh6L_3Itb<9f2$F1)u|fG*!Ro+6=^GA{X~fQ_|nsJDo)*RIJP-zx!B82E;GU zi<1<4Ld}a-5RNLgFF4B9M2qNxd2bv53br2RAhG|D*cdrBe;>K(F1z1k{~@=?Y++^L zxy`(klhrmZen%`i3k3Y?iY)37=q~Or3W3{@@Z2 zeBT}-xtc?(Vr3*c#CK0)>@*#`-=0!6xbxxmuxFkpC7pXyZg(AAfu$nUHtA?)a!}6`GZ)2s4+$ z?w!xk^xD}g8-VwYikyxZ_=csfxseiE(lN$jWgsN_N9}Gi5sQ*D8QlfBp}g4zWU3w6 zM=zR@zF}tNHb|z}m~bh}&e zaK5`w(TyoXCDrWb?Ol*Sj42j~r#G8MMDpxGq|ty4T|>9csEDj=^K>r1l3*!q_r!sk zLu(XTY;pcc8=FrplG7AgTq(~#_tRDb!F1SK8pN9qk&wmp>E~Zi3UCZCR1R=Vh3$Q0 zIE1}mD^3#OVZ8N%gYi}g&GQ8vLeRyK-DhS^r)fEdAwj5BYm|vpmfBy@#dBKz1K@P= zsN>9(r{a_61z^=j1RfmHpowjlt(WgcMWkdA-P+Y|gwfu3%NR_AfHa7yWzSD7vwNA1 z(>NSnmght``YiQrnn>a{;A6#V{!11qL0@_4HPFC`+x1*bL$GMe^LwL&)fP~F7n)njfy>O+|q5A}9?XTb) z{k*3moY=C-{kXST8HNamE%7dK&OF=_##Z;M_B}rN3Qnt!n^56VIL~t5_9TzCqJvuD zrdDDvIHlC`M8rv zc<9T6U#Wb2g(j@&TCx7ra-sXpSF!}ZB|YVvS-Z}IzL_`sc}G5-_@63HRp^2B0CRhU z*HtiLk$mT8|I2?icQ9M&75w^)p6vK_a-k`;)hK|$iM{S)Byon6pnEw^2I5Kj%$g!K z7E=pFG_5Z?#i7p6bggeIK45_-4aZ`YJrO#!E*Z%S&;Ub1sb=lp7Oigu>MAH1ZmH8n zjZ)z#cPGf!Cuz04mUB?af?8c-jVT*@vE!n_(GG4qn8Bf+we7u2@|}Kf8a2f9u8DVH zREdC6)MX%e`&D0j8b);B0Jvd91u--90q+a!v#(^X{#xJ{u5c*d$!nV6Fcy^0332xz z@%jwWCaF~OzP!Tkq@`8f9%?4J7MWqARb`%kceSSoD@FhOAxz?I*&>Chd7esyraL2{>x_%XfK*07fgR~Ij9GzXCxa@I8xn2m+qI3F-d-lNno!{DlL)_On&qljI&Z{ z@3iO^H5Vhn3T%1xntsvGfU>A zf7T@JSV0ohiu zaXPpsj4SeCWOE8$q_%R_AAkDTe_0@x(a>cfw-|FU#uCsjwrT)|pr zbIIa*8U3Hb@RrqYVy06S^&O_Ddx3aw&w~%Lv$bE_>2Ws-X~GMlRl~jSn?va!0rE(^ z3>v&kv*N+_N8%+%+2L43Ljj{IE$Lq0R-#p(I{nHWqkp+H-aq^UD^vQ9dp2BrY~@8I zUVSG3JM*#;PSp?iEO`ft;GH1%zBJGhL$vplZ+7VmwV{ z$qQ=bWJ^%lrotU`O?J1Dih_b@?7@4EH|0LHET{0ww9we)Z@zw~NF)fMsNfnEVaU8I z!a;j3o}oFu2eJR z1Fy^+)ER*GHr1B*j1ac}abA*^at_qW&svQCOH;x>cL6G!%pZR`lxDk$Eh}h*S00A7vR^Lx)MQ2eaKO%=5Hkm3$;BpoGu`hBZ2@p=(q^vxPk zO;SO?R(Hi%Rg? zOiwbuG^s#mNqq(j-=ur7Wpn}@*Sw91f8Nf9c*g)+uZ|<2x|F;J1a-*lqE~RF-S9)x z3Yb`>ap!YsL2FucRt0R+yFKy}*dcUG#Lx(SN6K13DuQQg(mTam@LVgpsKFlyyGQ|_ zn>_F$dZErA@6f$sO}%2LIK4*9Ej-3hCq6yNk_OZHKC8=kuGw9|>BE9byybL+M6<@K zoHfoTiB&nL17NGgxrOsoywJ7NjyXTE_Y)68Cb*9=Gna_Zm5o)74HppOa%EZ)d+52$ z+zORU%k)?Pmo1lg+>zk)qi<$H!p`(Smb6qRsjn)?9UJp7*5BLT8%b|?M??afX=49pJ%!sXvq8W|ky(Ydm_8R2jf2?J^u!N^lLNxXZUO)Q$@T@ScV50~s)C zX}*}5luA|jd(*>TQEH()dP2$bDhW9s3<)pICbne_Cn<Aj8D@KY zVdaYUfjPGeqpF1YnJUx~mu|Mx1QL(MrGl0Q^t{l@IMjYq8C;~UD1UdFCz9J&ueej*b z_@l?S62@C}+uaH@v&*}80FVS(x!U|16yx&?JKn?PMrA}Y#+HleQ%Pi%N_i#QGzH?h zR~X|{I1l_t?=b~v?S5I_1o5C|hirzS%vXJWSWMv$Hnd9yi7|JmWm#BH}C=})X13SN@j}wWw+HWrl(SZ zO{cB{;CQyNNn%W{&VJI6-POKDT79%<>5B=3eH92VnF4?18DIY-GIzx)k)C}Fiz6h# zy8yDH0uT|xv|izEu5PfEYlAZ%w+Eby8F|{m#W2*7$F;GWmH<4at8>+_37tejY zQ7Qi`wcXJ0s(lQ=oLQ>keuHGy7&F#1(#cdClJKlEOmp>!J@8Hb6369UCukY#@ z$h2%bkoa}mRB}z#e3h6xJ_kN%{I*GMosAFEJUn)fkE!xt$9<9YT)MunQJVylXAKNc z+x>b^dp#6RxkFS1@Mn)+19w@BGRc{SjBjMq+>dT2*nVb=`${H#$g+RpeS%#d|L^JY ztY!o=aAnfveuCoy(yE+<%FH4+w)%|>!g+J1z~rm95--*ShQ7zd%A6&(y~weAX^C zWrW_^_TeULwGwO-NOpDiH#YH;k^8HhdCuIm;3nFv$hJ)zde^P$C7v5^9?9hOggjb> zb9t@-kVcDat_QQRLLv9P4uJ<<&EPC9%oC=mS6(0>Swr8OWBEp=4*EwGx?_4TNc<9- z$Vk{gC9eo3>C`VHGiD4pcaVlAtL?v0nJa!JL&Z~3nGMfUmoZ8uBM~5A zoH+T2!@N2)J<7m2Lm)7FUr|=~nzHYn9H(P({Q`uKgM%sj1(DgB*6K-+Q}%(Ari>9r zZIhc4IcR4^X#dB`Y-IIz;V^=Y;ukOxQ%AL}_6|$uxBw>SvQeP7ZBGbZVS+z9)2q~v zWfX^M&~+B*jsu_EFPK!m4+rIC1O3>#(f`5OTZcu}c5UMzNC-nq4WZHo4MTT`G)R|- zl*CX{(%pi90z;>iG)j(0Nq55x(nt>7{5Ib4+|Tzt&wISz@8IB%z4vvkb*{C}wbr%w z%v{@p<|%nxZ4vOf>t<4x$^9=V_LRpDIO(4e`&SsGF}hz`ec z<2}TS5Pll_$_KY3*Yl!9GaFv0c(Omial7Fop@MUooj&(8?GLTg;Rlu)bqQ6}jJQ}3 z=`C`D!Z3*oo98Iw?T^(lU*}U&72{DWyJG{}SRk5~6>l3==*VrA1d2hNnSF^!Ue*q< z^K>?zn#pRx*Eb00x) z^Ou)!Op6t7{Y(hXZ_jJITYWHJH24+6mNrXraDfpM^biokkElMWJgQ7Ee`(+$Z$y$p zZsdS75UNdQ=p@-7W`+7B#bPzTIXiCrArMQ8K9i3Is=r_37{$LeTTP`ClrLKEF{1-# zBGC$3F_BJvnwIJBNMIn0k57$XGtn9eVaO?k@y~@rzTf7-=__CC`|*V(%>lqSzUt27 zp0(b~7)UsO>jVvi!ji(sLI+7;w1L~eIG`XbUNjI9^ycp#FA?=18m~m7WTXF}gp~xw zAQ}qmAH#Y7ZTvq*|CjOqHvGSeM`6X|OJPv^VTD1$AVkzSAS*QLdx(4g@^7GfpgTsE z|DYdMCi((s0YLnJ5C3O8{(ba+9{;}+|KITNaEr{;-gV6|Jz92~6f;d9%P2hK1xK)F z-5+Eb&}#9@DZA?IRg!;CH^Ri*_LO)H6h4>kHQ2jJw$2qnn_~4%Zf9dH;XRfuD`-SQ zu^=fXH_Iyq%ZfG}(m(zNaya10GLhypxS7T<@h0b^LNL{Y)p~vL>Y10&XQN|ym2lP? z3H7ZeKM%e7k%^(sLHgRPjgW@c&R!AuVY(4U-cdkdTABpUoS`K>H;=HaXVi;OcIxsW zn-|!U80-o8)|q*lBJ%Im#1t{A?c2cDlqoAkM|#)gX^)zvtd9tVQM48(Ub{VlLu~j; zHuWB*%p8bzzS7kxZ#5TOkiQRa%2WZ%h-Ycb^XxAiHWR!yBjKF;GzxE(;s@0jWHXOr zhK6}5idVZfF)SC<#@NeZRZQ#Y~RG7BiMN|&u_L&ZA z-ve=CxsmDsrwIhmX9;&0< zS2WzzsITa4E=8Q_ShJMf(G{_YX>tI4W9^nYo;)5KD#V4^#JjOO{_S$tV2c5JD%Qcqwg_Ly? zR*F!&6C>{OKCi@vRHNfK_r|AXya#SvcOR7Tf+^x6bWK^GT3j~7=6jKbhY5n1VC3zk zCX1AoLVSlyl0!_qWM^@XuB&%s)pL%=7pm-8_v`PaG+9~*y+23-XeSpT0o}~Kk01O1 zbEK;Qu>a{h;>|%h40cb@W~SA*pkZFnflI{d(>vX?fPRTa1&z3ApHIHJusd-&uxMY* zcyizq%Wva$hc+~~x5!P&>in(m>dYQ7t(e()^ao_CT8*yBKB$SNI2a3Vrf#ZAAR(4@ zdij}oC)p#0Q8ZuC>Qmf> zU>u$*&4f-ISF4*akh>D`&I?sqQ(LW*jl_{9yano_XCsO>&QRCiV2&^mTTaQE-8XpJ z){AuHfltMHSer#0blH=ub%`FnvLc;6+}5dBBAeK(bOS@m*D-$e1IsOcpT@*}Gjb?6 zEXek#7U04{GH~@b6O#)BR~_|!*s}4IX}0mO-Ee^r0HMY^FKL~MB~SZFbK?^~*)xXmfuDS-AJ=+< zvekdbgDHttLDTU8(VBcruZ)@bu8b7_9Nf%8NaIq;PD(n8EZFDqe2Gunh{>y3!sNF% z?)YV?u{kJ@LFcKwsW#$=o#%Ly6%%o1-vN)O%lc@P0jcl0SbImFywDi2I>TjDsU zt|={^a-AKy&!EMEYa@eX9vDA;w*G1^&4@O>} z#Ws#djWi>nu;iHM;+i>wL=1Uy^sDJz=RXBV2i~iG@7*9<=He6P9rg<~CKSe@-p%Ay z#dwnTQs}*(69s5+ItH=`Z?l3PAqrDDC=2A{G#gc765Vt7-=&I}aH;I6b-g?2X-(CB z;N7*x^ZvAZ!DFGcXPtyFLM}@^=N(c<#e{kB*@A2)r1muZ;8@(ALz|6fAg+z)5EsOy zL3Y+-R-41_HdU1d`VN$2lu`ignbkMxvT)iq-bKMm$3G~7nQ#tJB-nj1o#j1x`=wf( zk#1xk;*$#I_s67ZjJW&0T_cC*wgwUo45pK+ramXa8SyO#LSUU?COWccDd|iKUXv1I zOCVpfDIQ($;yN!q2^6=4CMU-*rWXR?W01}IXux#-oDj;QdBc$ehOx~-DyZ&SO?hB8 zI^nZL-oX(dS=i(dvKZC1i+yVX6op~E0uMQqKRj5a9FsA&WAA;pgW@Z5!H{5w#fx{M z%qx^XpxuTAa%g_3^xVtjz0b>Zuvqix5k%{Ts2B2L;;(!@D$9KDbzxlQ`ppl21W~-$ zspfVYjOtbQ?~wfIiC|hXkh>anfq%LRibg@A?j4p@DI9)UJG&Ww*WQ}62T!(NeHGvI z0|%dPZg{^4(%}QWq9CJ2&Np73Gh;&!Q+LA|f)$A&R^HbK3})bNc*j_@IiKea8}i8s zABDFv3%%)z;Cs(}{;VvZZgIqKzY3GtW_#CkmEL4(;To}*lyVag*|I=dOisxj`?M;5H?Cf^C zzs;*;hoB;JP03bkO+IJoSm1R;whVo%qS%UvXjrq@g-V*l5mXD@V;Hk$0VW(RD$`Lv{CGA7bd8?q+11BG z9#j2&oLBOv_8g}gary0?C|@#oUbAppVD=Nlry&ZgT&F<8U}HL@(tgFr;;-0C`Wpgl zaVh^U5LX5W(YUvCANF~vSb7^(hUge3fo^Ska9kCb-Ljg=D~BPPwjlUE#EB3Dd=s^1 zlRxH>dg~w|j{k8;T=Y=!xAKXApUNKsex9P6rA`sP0AHfhGF0Jt#`j|%*i00+T{=r^ zWT!t?4}Cz}sf>QCqOsmq5FxKs5aD^UxV53oF@h((EOGn{x@{R~$n0WVRjeIQdXb}) zFS;}n>=9VnolZ%n7)W;eY&bkK^?Ra9v*A0gL;epb37mocqcXh#jmqX`#sW&Lm^MqJTa zFv0VwMfP&eT651PLAYn2#JVvQ=b$=3D{J*rJZV_rmz^nzpnvEJzUona2uVJ=5I}Z= zWpH>_5}03ZpMiM}(ttI_H~a`5>Pg2tc)0`-J5YqLFKO}-L}|t_Ji7I2!rd*ci_Qnv z+e!I~D?Pf1u*7N;sxMs;YlU-=5~XqX?eRJ;QB*MG`o(SjrC zc!e^!JZlN;FFzkam4@6*vB!`}l8$m1k5v5qM#-+HYH5&Deteu2ZNS@3G+Xy zS*}&07T5rr08M9L%>$!jE#_y6h@ zNNogFY4DZKoKRWoX}#^yBa@Nl)#^o^$sY)OV;4I5NLDa$x%H#_QF7UY?NgX!feW#7 zt*=>FGoEnuAy&Y@Vs#8ne^jsfsy88?_a$kjgbD6ICH`|JjChVPs%aMb4wR`C_KlDvyLOyIULWiR8P5{k z8#O3|t0%Weq!+;pKa=?zTS(-K`ab-nJQgLG7n!ti3I6ScGceI#*@YE>gxCGJ%D=n< z`8t?9aM|lT$$PBuYFcjx`Dw6~)RTQjZ28a!CoKQfs?{%+B4#kw?0w-~~Au+|ntSQRMKq;&?X z2iP0FVqF7ddI6O~>R*z9>^r*Pl?GrB%_1|^Qis09XzJeMQ@hw7xtjOTf2f^aAhi}o z@6#{n+(6E)>@~6H!T%FH>#HeU*2fwVs=(Ftg+Bk81N}`YVM!?%(~gkHcqMUkE%nyo z9WL`9rFaYKa3fzA`r?Lb?QEQ})IjRh`CWBXBN$sJ+dt~!q|GbVA)*Spa^9`%otE33 z!AQr!cbZ(KOzH#@2&*&?$9w4Q3V~RPsQYRfuyq$Q($kS_bzK?IRE+_w1Nhuzj3$o^ zrtLe#3It?%ZyfDodt(tKfwHRbmZks)XSgZ+G1Wy0r4E(E3 z1umS!j*&>2{QYVJcJbUnwo#aG@{*pH@~e~uTyM+0EATB*{wxZXZ*PhacMA0Y@$1cS zb)aF9Q4~yQyv16#qfGcsN6T|a2#}G1uW$Ee3gr0TE%+w!I`iZb7P$%aJ65=km0}SH zGQAR8pA$&5USNE^?{XQxgCtt3z!jChoM6?0w!2Y11eyjP@kY8;SYq~|xN~93Yt;`_ z3oZ_-NnAHo_Q*%|CeHutQGeC*GKeqF13JyD_@<+>3Am;2fkJn+SUW-t>zxBn3Cxrn@Y{m+J*4i`S@ zWK693qRD1^Q*-}^e%lnTbPo1eov>%nniMLAY`#-DX}H#q<+0+U6*}A>VLg~^3mTBX z3UxnXO*EOlCbf7RyS-}N2Gn$qrvxL*Zfv&ZV$XrD1Z3Rd1+V5!%GKPGQ=1+77V&pp zEP153-!`3QJMKuPBmJ1jnf&3TCdMXwMI=2#E_1>nf&1)~7oR2Y-QGp4XB4N|O#zb1 z1{@|az>%C1$eYDg?XUwULYP|jX4!xhkx^i@BvNYEWV+h+ zFqMhtn((ya{yDed6H@ZNnWDpZ6Ap>2Q04=}is0hn=S|Ci2B5LdA=MtqX0CRb6K%hj z*?*5~&28Yz^(ra%y!z}ovD)QM*b_`5vS{r5qZQG)2m)Hl^+S@Qe|H6 zs5@Fh)E+ZZ-}0^sD9t7mYtOnEn}Ju6#seB`V}PWPE&+JkD zOaGo`)$|Rcd~bx0#^uEvgoK^)Da0qP4zAgKR5h#3pf39o5!kK*awVzp3!-5ONo?KLQzPNUqblQ$LR zOX&q89|Z$Tg!OuO)(wM>ed0e>iIm}$iDm(fJet^97uq}MW(KTlNLdN2f3Itz)x4TK zz8qSJEntt(xiMTVKj7mC(lgW&ABJ^aRJ$*T>xJIVYNZl7@9I^Z;WV_^Vi+HOA&l=fXt^itXb8J7v9pHxEPi=POTZrb1d$_D=R`c&5Wsu8rT5$_o<#q~q zPQtT=2GEG!LS(31)eB>hsjb@huP#}O6t4A3+pY~=_mntBDcVTk4X}`(a$YD(Dv1hR z=;&G?61SI=1}AARLg|*MaN6Wlr_zR25<+J$N=FbU;d z0r&Rg)@WH@T7g?B-Fs;xznk12n7))jW5ugr{zA9%%Rw0DgVmBmuC9S1X=%_q&9b|m`=+R7kd&_&KvM)4m^ zY^EKz6np?EKoc1*oR2LT4!YEkwA8fdWjnMa*(_JUz-hV*JP(w}6e#e2QV=Vl5w$6U z;oIolluXdpqRkhzm}oe4<$AYDI%T;Pj>}_)fb{1jIWJmfwHtZ;JPtznin;$7L+yjE z87`nf7dgv09bByBq|wx(@n((@Jmj3A(Q9ubQ6FF@*AbVYKz6-9Q+oSdyGbKK^JSob zzu=>nS-r8EGHs)q<FI#kiZ zgA2!$Tx0c{FtShD!{$Dv_u|}|$n z-WT^&)u0KFWQE3H;yXzfh6vH9X`uU!5<8nfi4I&RB}dS3bDgOto$lZ=qPI`L+>D{) z@e5q>3Rlgq(l^0<0UqM6y=uyeUc#~7QtQ>Y>5no_U*BqfovoflXxyFgh@MU^b5-24 z9;w6(lQwsgHSf4JwVOOn68$=lI`vmHfdSU5>*;QQa#6t33vFw^%rD<#ugPzD$ou>>EGko1nfKuV-r=go z+S|SQA=&4}7H@dSe}oKFf3SM~Xl&{EovJh{mm=9Va&{}UOn}rWSZHLcyyYuoL{B5) zjDSP~NKHX08Sm*_cYl{__vSye3?epkBC}{cZ#};e{eK)ZBY-QU(FOhm{$-H=esEpo zCN3PFXF%B@{0fOy$O*}1&X-T5r6Fz24GB`)C)ISvUbxHK#l2dOU~RejDvU zw*9jbSKoR)xaGJn8$ANrL5_P<-k@Is7(>e~OkvQu1O& zQ-=BfM>zil!~ZYbRM=z=nOgsQQ~!kRjq$fn@`t_mEqQ1{;O z=o%M+UCemfCGgwklQ9yeAUi)$^JQzQOgHlVo7plp^L@u+E|oOCq2H!igOTBa4>@uO zsqYA6`eAr$ikS@j8bTK2+$bM};0oA}y=~E{C3BEK$3A0jxqX&z0Btj#`q$hiB;;?+ z6>rdGLP>THON<*F{`i4qJvYTef+>TLcHn0eAjo9r<9%Q#GmwZ8BCn|~?EeL89N$dv1ENazN5GoKzA z{Nc8EztUj4e@S)lN9a#t{VfS_vFJ11wtp$j3AXF~FGmjkKGo%*y1$G8EcJX}?*1=5 zz@8&v|K(`@*5xZxu)|;00Hz*1HXYLbD|qMwO24JKWzPU&$QxSgKy)_ zSp7?DK(Jl)-|!v$*>xwe4F2*OVC!Ax$HRX~V$5_f1>5`uiA3#dd34QRw&Ai52II}B z{Vn%_X$~!JWYyo7Co9UMtN+Up`=Bs+t^eNu6~60V3iA)?U%%n#|1)z2A%O3i4Y@Y$ z5!8mHmV-e$fAAd>7dEG)V{1PgUEBcYx+FFq1hVcX5inhZ3KWG*;3cF->tzFvgwj}5 z>HlQOIbR8>E7^&4)aO5#O(OS4IxfaK0(mhA`Qtbf${6)K5zjjAF9PKI>sSK$l=63O zA4CVK|JDV@{heQa&5l2Avaa#GYQO1F87c7UrUG;`1N$=C;X41mlrRU0?~?iR%CiE^ z%Xb?eIDd2QPnmzJjQtNMC6E%|S#Pr7p6T!W_4`Bn|8x@VPhkHx^8YybXLUu#{&$^! zD*2yI{=4QsNB;ZC{{==s5ztqF>hAwPq5Q-0e}|6tXWahH$p7PHGndsrefA$g{>PL5 z3*`TfW`3T;^#jt;Nv0(Zu!geD)sMT2`M8?RDiW|x(iWRFaYgixGJdZfBjMRO!JTQK zi@yAZ{{Qe2u6esPmk;o)CTYxFgVj-Mv~DwQx~0?810GTwRet)kYl{w-GjktxnH zh)M^kJpNB>gkkls!~J6ByArq@Yxa9S-ja(;s=+^Uwlx_wx;98}IMr6f{CaDbupYSG=k8L+Fi%xD9Z`V|D>ylVNkc zf|1kB+a^TQJp~QD*lP(F*S_fpZ;o@zcn4;c)K4SnmA&+(tm(e zT?n$sb6m(1vXoq{f11xA3$DoIB+Qehdnw?8y-?+Y?yG6Y= z&-b^ByUv>sr#^(5U@@AI4iAHSeoaM-mb8@(P%(-|GTPa)YVbA$n?4I=#X0#1a{U=P z?)IzYh3*R9FLx0`U%vC^Qmh;&|wE4%!0Sz(Afk%fu5uYP{td{-n zTnKG)2!6Q!@+$I8yb9;m#f=AHIbv9`;$)i*xCEBj6t6()qsdb#?)ANPriUwUJ{ zJ6rLKVBRQcZZWr)*PUwC^R_zuwLg@?frK7;CE2bW6MJ1yga;OAp1m?o{Neyi!-GSg z2>z zVOTzBjUvpa@;Fh7m#|!N%7B+bh>&yYz)mH8*GZhHyypd}R6&E?bgH&+TeOj@_n}r| z@x!mgBs+8{SnpXxgwJsLHp`Q(pWey*HK}kdSwj;#XtUrdt8jIdd*@CVkj8&hhPb}H#wbh?w7!Xf!2V6m9PC&eNe=khrU+SU)*~9 zRTc(oKgxd&=4?9w<%F%AVLl3nOn|b0shc|M7N#@mMaRLpp&#X;mEP4_3;{g%8qsxh zSc(k@4)UHP-s`fAy~uy}VA!333Ss!>;OHl0%DTL=^+la0&yQ1)Cnt9d;bwgFE6mj2*yz`$z7O@yg{=*I`x2O`dn8+S)o=PJD*x3YC8miKtsvx!#KV{)x>@F z?+?hBFmHLdGQ<^WFE%HvKp-?5S(|#aa4Jf8T?nKy`EI}1xT^|0(IEVJO<7yofawCL z9tyWWp@xdxhCYu*CWXqrctQIO5T)OZC0}UG zjGgj%rC5TBcuSrLEvfJd_@oym*$H)tkLU1*kkly~S;CVa+Whc2S{%eG@o9wlI%~Yp zyVf>UAg-&r<@w}^+npnW!1!SlEGAWZA1sT?Z|z7>?^UU1!#A*7J0KAPA?HgI%QA)U zpZiUy$gb4UgyaOfT;%h>hunG4d~z_O`!@7oU8Z@Uv@6A@tq`B}Sj@}k zA=XZmSsI5rjg1;nqY=nVM#0WUB=dCFqcdBsS@FA|D|LkgepYQ81wQz?rJNJpm!cB0 zI&P^^yAY1#sP*m*!zP#gx~sS~jrvci2g~yF6rg*op>~N3diTEc7@P`c+=WZN5;`_G zv0=#07E`&(wWb-`ZFOh)V9KBRS^V4Gvw(5MN9HlmNge4DxP5xkE5(E zA1*VM7~q?5Cv#3eQ`{ZAq6|KjXU{PkK3ZU`D&OL4=4EY2$j4ul34=_5Ps@Lvjgb*p zr!RhbS{lAEYOB|A!C}d<%;1d=w~p{|dMrcrdF1V!EM#E$R=Au@mm_K7=~Kyh(>w1E zG5J{HfBxFDsJ8#O@X?#~V^)_{wLqf2pf5gLeb#x@A33;?KJO?Di;LNn{|uwyH;vn* zVE5qr_r|Gl5_YemSp%^Ro=azIQk0g`c9WXaKf|;yQk0!srq>k+6 zWtT}-E-PQ@YRGY`@-T*+sq4 zKJ6zG?X6)NyI8?Mtx&@yLxbnsEebwcELlnW-mTJ5c{#(8HadxSktY+)R#b+Syn~e_ z+9`4?o{Rh3g&aomY)0g9zNEF_TYuWhG^V1UYg^SEY5q4H(9)0LBwMZ7HA%o%-60=u z2V6vFCgjjMqhsf(TYS->*uC|(?@P~+BNtx;N89OB=y%v_vGOJcq6N2f*X~o$vuurB zfnCeTOaU{=5VFHnPMA++MZx4F`6a4}-Xf}@*9?GmoY_j@dJm9~G`<4heYI<$wjib&OTQA)bJFMF72 zV7Qh!uunZbEwf?qYb3mQ{O8p3wEcio&B#c|LG)%gNxx}2OuoatvqvZLociLs)oLR; z^fMWz$5sC-w00mK#e zLXt0U^eA>0>&-mGu+bWfn@KDr5>Q{F)+)f&O1D=;s%X6(9;0nnUld;3pU+Q)F*D5f zS@!hJn_8#)W;QzdZjI8Y1YLAtm!w-;!mIPqq^il}s8qFC#*~+?-WOq{)oCS3ao;at zC(6b6KfBCNqMgx{ajJG+Md59CSha!;D`d-miBr!M_>k+eUA`KxK%ooZycvPSH9hTr zxcTJeNyGw(&gUYZ0)GEm=!|JsG@J2t)cM8ht}l;`b~}`{X&_;?R75Lf)>W{Esp`E> zcpr{S0e8!x#lnogyP5Lj@Q%-j=~8~*VB6jjf0{j>t8{9U+ahFsiTk1>V4`mSy7dGb zx(b>43^6-W%%0|7O}l;OJ|NyAhyw zIzRz|{nv{ntD7}f;e>~8s*xb*i|0Y%$fQH_m(++HgHvUdYyLxGn=9RIy?d%p!?}ex zFv+Y06G$|UJ<{_*uc#z*tV)KRK#xqmeT^572E!#ezNKN|&DF26>EP-PKT*VPbo?66 zWuakiR8MGVRKr-4%Zi}Ui5=pkt`+)31XVfozw}%QZJm+slL�PjGP}2%#%pjZ+-c z4^fDIDEApfttHy-fK;68`vMHV5{ct`5qm!tsbd$CMfgQ5qn0|Z`^J)^VvxHhAc)VR z-@{UaTgQS0Sw&PA763HY;d<=a?G=t_(24U2%~Ih$<%vI zGKJWc$?YP2_7!Y8!C?>Pl^duOk}n%8xu#Lfc2_~`CrmpzyMUI(vzIpeJSh^NwsQzio z8+|lXK2-{8>Q-aSJrOPoTmsKz1SwR_ve6&vbnCd>>vHP?i=JYuv-5Hhb(@s_T_>s1 z5%`jLz-rA7&(IAaf6#{B9t2Hj2h+lTM$4c=LX5d&NUABU$rW%^-d=<>LGRZJRmfmk zRuc8He!}TbJP{Ih^Mla1zubX*qiN?4Sgelk7GCVz03LTe=AG+>y5n?|3szY7+hfsO}bY)uAuGNwo)xqvc+h(hF{dy!FrcXhUkk6C9 z4)PpHBj4ObZ%4SdD)NVDxbSEDcMkT-lx|n6w!yf{{q0vFhP6K!AK@@AmbumBRyp@% z4}=(vk>+j0^TCazH5>JCtA6sqPsyaOu2A4Lo;1n*CCL&!FE)0?FCEa=uo`o2fdr8-?l|6wMLR~!L?N*E4jBvp>#|2#OX0x}Uu7+Ks zGW)6An4Nkq2D_4C$q(229X`=#%=bmIkQA;!GR%G;#eFbhNJ3W8sl%)6UZ$|A@o5NP z=`TWTU9!{eD3E*%nr%U*D>)NJVx`)*z%+p$W%@A%+*M<7Bg1Tk{743QpSv`8tv4Smt`e)lG1v_L`e z6KRcNnl$iA)<+IvzJ{k6_Iytki65CC-O`j8{{?v>1{J#(%in`{3A0QQSDWsL8t-Gi zrHfl$r>LwqQs6%@Nbb%~)VSBc-t8NpZ~9y{JG66J3JbyVQ@Q{9JUzB;E(g3zE-2XQ z(>b31G=_ju-zFzbxK+~mz@R~_j!2UWyBg=!oZ>3^Med_k^kv)b;AR>$gEx)9;t=sP zQROW3Dr3v-voyWD?0tfz!RNYctdefS-(^_t{6gzWd2B81C)`WoW?ZeB>9;>2EAZC6 zheu(JS#{!<@i)XpQhfSjqL3kT)ko+ z^}lb^aq@)>j9HyhM0q<_Wom@P%IMl^sD#_M2?Jf14sn9Ak2%^!u&^22F1)+WNLYfm zU}^pxQe{>fdv0~c@1uEyx%(OwCr9W{4M*kJ#*}CCl!z;y6T8o$m~k}mTIy{xU!z9n z!$K&K0q&%29_CP91;USVL{+Ai(!P%tAaVc)BpXSL9ML)w6pNlqY}vF!t-$OKFY18| zsln@L!dI#{UmioF8rHyc=&kj!;fxTz6HWcb@0Id<6|jU>T=CYo0lwxRc+X2`^`Acw zs9Xsn!(n^N#d$W$;feQE_1c{#l=w~jkjkFQEbnAGohe;iFz^o4&Ihukm*YX{5JNJ! zi=-53^@krjEXDoqfswry$JRqGUEdqoZn1|kiY>yMouiH-SS_Ca;2Tgcb?uZm?K87J zd}At*W38;7^|JM}iy^eKs2!Tn3ERe|?r)I3!1@q!g%<--68hzuzp9cAvsv>Lwz z%Zs+kQM$!qFKO|0etkq6?0v8@-AsJ6{2VGDzZFe5GU@)Vs<^WdeV`GxFM$|qxE%+j z4;T35DS$7tUOE|}kt9{Mgz8oqyjW>kbxK3-qAU>H!^mXL(4pNQLWE4PtgZXbJNH$h5#M1^91{?75d@E5s z2q)))VPq8Y@}fdy#P~te$uF^y2C5x3kPW3Tp?6xjJPZZ5Am<---ZXlkUCP~so^Zae z2-A+a1KBteqS0s`#<7?b}G%Sx>w`lP0MN7X@ z{KFrxUGUwI*2TV4ba#Dsqb?PzlfE&Og&H-Y94>2P6;dHl@t}k*Ggi*#;VOhjNQ`!E z6zKTCt``EbStFPV-(Vwi(O1V_lPmMe4|`7bAB!};Db$=vMTys3$%{UR&h0ElGc5zZ zvf`o=e5@}9uXui{%>4=@%zKzSiZ69t#a|xt#OmV`q;qR!R4gsThQ(5Vs!>ohuVuJI z3|=ULFjRn>mSiO--_c-1L!%~9kdb`uCQV0e8S4c)*tj%50o6%XRG9zNu*pIhIU^61<|=))CMyl1zN3y{KuL)bhd9$i+H zt#Tmhs~LO{1ONCH4TU*==MwkPYC-k2DgRktQ`xfq83^kNu6vddfxBjeAxLjmg1vke zeHveqGWKZkH39bXYc>-;Pd0>?+Nvv~h=d%zVr!|dhb+w~F8)EomvJaboI(sF;(}R^ z{-LmLvIbLF|Ty>L&v))w>eRA+RVT7UljeBPTo(8i8 zK;D!@e_R07s#bSlCf=t4%jN1Gg}p#k7XSbOY%!~A2x&(5RXRlL-f`IQ%PByzWk9BdeWl+V)* zS-p-5>f+ua9l)YmLwbk4@SwKf6il3ddAX9@%`T_4l?nZMhNhAs+(Z?0#d;SSu@2E$ zhD^=L_zK2;`4&b{>Jo7J*&Dqbl~&ZuEuFtdNe~KAKkf@1T7$U3KRjB8e1DA|HAkt5 zTgfbc>_uHaH4kq5EEmHccIl3EPt3$@>>5?b{D{e(*;%2N|M{ZsRU=y_a`N`Bg@Yg> z_5y2T1lSJX&WcLE$c{n5N!M8MuSYy!ws$`r+)3XczsltxpMOApJvC+2m3Bs`5TUqz z)twu&l1r=}CCz;ho01enq_Tg;yksVXPTuZ)UNRgpL3;o`=D%1G1n*pCJWXr)yb$HQ zg?hP4mMf5ZY;SJzGNg4udDeT8I@(85^Rgr3PM?It{VnnMYh%1l6~0S37uKItZpWMV zycF`uFnq*3*=6As`BaVT!zxa@`&G&PE3e%(pTaAkB}kkzQC|)9UPX(J7I$^q^bEc- z;@Krm8UkqcHaKSp`l|>j5U=FMhiax*d^(ta73%43ShNn|VMEL!(?jc^Ul(jQe5uYr z*C~L<{Wh!nfVysIK^axE39|j8nKZx&a?soA^Wz`$;Rs~v8X(8C3}X~Zd@!`m=9cww>6elpLzMlkZZ2Hf;n zv?XAtpu1T@1Z2pwaIw$q7bJ?KR&-hzn|vRaHw5^(u|mMeoxqCh5KVC(=b#xQd$B(6 zx60i*)d!rwz7RXVAgURyw(m1PoJF&`exTKUmyNUn9>|SA9q!l{ADUG-r(WV`*T9qB z9WmecUfRCQX3(G0*Q#{(Vcr9|1Zt7} zF-;Vw@Vq2RPv!o*HX^Pd=w=u=sib(10_%w4tM}ky(4DoOW@0wtf%1bpRu6A%3hOkx0tiE)f)@LeN&!(WLJPE?-$-P_u*-JfUqh>(#*VG z1HV#Te-*b=0!j8YEoK~j-3|@f3`wr$+F{xQDHRWQl6)P4b`L@g9qgA@-rTtq8u^%h zH8@Zg{xbSBn(yZw`>HY@g)`6&;gh2crR#(BOc{>u&S7Xwl(`&5TA!>S_o~e}p7gES zJXMFZmC`0WHltqwuT>?R(J6~q5xi%hQw8isJ@}miN-vd<2b;Pku)cd^P?>$4D>||} zB2aRAyW5MN(HaM^$EAVC(Xwvm2ed*XDdv$DykF{alEV zxohj!%p?79uR!;zPt1UZXx6CBOpfQUnA#d>{rds_?k5>{#lsojp4epe+DH8$Ig3&(m1*{0BeE#u zzsM&z1I0fi;B3Qwme~{@$CR<7pyLRe|N3c6WWDk0XUs;by=nq~ox9Ly8DH}2e4O32 zD^IHtFB6g9*WdQ9upp(h%o^7pR)GVl_OY|>)h<6zM1CD>em5};m4KP??BOFWa!!AB z9fBp-(9bWgRFuOY@X?cKM>a&Y({8?zNXT3K%Z|uMn}y0M&dumrjXmg={BD$1SJrZX z=*8R2e*2T`+^5a{21n1;8v%}+QihAs>XsBHjzw+)Yb8$h4eUmAxjy7I4cd?x$F)I# z&;Izbb(*0_m(h>VL1>Ax4D|c@+Cj#UOBOHi(9^c~{UQpoZ_Vx6`UOp94>bKo8#fhv zDJKc%>LsqX`}(T7^lc)Jvqn7f?ys^;^QmlKuQG~E6B8;Z`%egM?wi%)J~BL(`NssG ze?VWkI}-xXCq7+^iE3>sug>HR=Vf_Y=4r^Z1J7@qqoo}Of zEQB#-$xwv{-j`T>4Xgn&IGzJjFSJ7qe|+KIwcug>4xOFZ1>b9};JvSN_Sht9*tK-u znhmc00%Caf`dW`Uf}iA@&=U~v z!=(Gk`?=mGaIUOqMzJp@PZ95XT85d-a7LEh8D^lzW6<|n@sW4>ErnTT3n6f-PgSti z8Pl1(J!SoL?L4P7{dHiq>pOeP-E3(Oyq%K_j*?Zvvp@ARDO-+P05RJPpBV8xZ5M8N zXP~Te0W5j~r*9E|z7Oq6V5t3h-B(>Zk=`i|fdp!tMrUp*H)$Xukac= z_jLAyjZZ-85LWtFIe9CE_aS1xD1~h8df6ZE4j!lVRV(j;b8rxiUkbq+7Ij*_*lsh& z<7STyQq3o#9Dl)SpjNnD6uUQTCFWBW?FfkP-n-$&+*Epl(5nUN0}rDNK?6eCRJ=+koJjdrIDgpLR3gvdtBzo{DzJrcy8+5vx75kMZR2y` z`Yw~1ux`_N*jU%2YX^Emq)CMW`$B!oaoiH5jrhG70aUu}{%Y<$UHOlPQ#OwVdz{eX zgBF(^J@AokM-JLeS*o+!Cj`jz%jx0bcANs~$&p0gdZM0VZJ6helVvhG*x^C4gJtkge&J9-LbZ%w{BbO96^gQ_Fdz1qIH8}L7wA1r^eHFP3KdNniCiT zbkXkqpI(~W=U3Ocd&e!32v!wEw0~>pzNK&8{d$gA);TT9_!V@Ol+FGk9*;V_(@;=t zeDWYwSTp?_O*Ya1uB%haK-!4?ut@W`%x#Nwfw(u`KKl;#m?CBj4wo0vl)(#5mzN5) zdfMy@w!U;ZZ5cZe2FyE8!XCN0dcV4U3G$J3W%wc*Z#Pt8*D;s!tG-;zv_ShSISy3d zfPO#Z$J8)u|2Bih^5};{twzFASUKI>a@>BcQS1e9~2r_6cROLwgO~%CK z91jy+Unj5=b@bU)H;2Zq2mgHsSetK>Eh=Up(XEkh3NYrf*DBvCgzOc$Zo_*Z(fA&KI!L48c8aXKFPWsL<1djOoG zrtd+RyFF^xb^A+krz{9c=qS?gszaw2$@>9`gVH@ue5Yxu8Zc-mma%<)2t&L+yG-mm z)bY;SkW!zsX*|snc!HXf+vvWKtHCd;Zu8^xLA5$e+%ZCoGhua4gd5{8FMeTk{(=?q z$^97{EE1xj&qA0neuq^p|9x}^_%O}p(f?Q5l}59jXz{;xp3aP__SIA?s?%C3TB_9w zMVaYCTRR~tBSuN>f)`s9Z54w?H)>6(wPGu=Ol+Y_Y|%C}XvqI18nKp$rI9pwnQ!mw ze0X2)x##}wz2|T~;O?=_aJL&LVtgx?9Rv`x?-(XzPK=-M|B12EY*z>R z*1T|s@3i;JX)I5^26sni%}QD+Gc+1FC4IsITP5dzTfJ@)EVbnmt{>)G=ax$#NB{X~ zr+~6(E_-dr)vNDD>WuY%o9&B&c7gc!cN+MWRe5$#wrj{2D9&MJI*PB*zdDpShOb<& z2|G?(N&b2b{c5?S^>o+y)5g?E33wFKxf_R=uUGsb%VS*Go@jpJ(m@Pv>35{M=qe90 zx;MQl@ZCaYHOK>Z#slpcvGGj&6KXsjHaw&pLEcml()`^L55L;2z)TwvGo9Da2YNf}nHXy5a+_>?n#W}wiTgRiWKzu{sL zEg86Qg&muSG92CX8FHZPgnP;I#{H!`o`!)xgK{G^VKfl#{`Dv8#nKb)A3mmx*WZW7 zj8*I-T$(PBlI$I?1egwPZ@wP<2H{~=R@XT|+@Fl*90?pL6gdVZ^hRaDztAhFQdZ+R%IzSYQMv^OnvS2#UW-Q+f^Xl=Wd?e&O0y>Nb1ouy@D8I#%v28k zGTBs5vfe0weTsicwlw(aY8b6noroQR+ocHm(|R@Hh>kG%kv!@ZZnJf-?d zJ7RUkWXX~b&)rZ&R2zi`IR3-k|JWi!QFnF?(GLOZuo`ZGUd=8aOJj9yDuX-*MWtNN zi*0WqQIRUeprLe1Sja&gMMb1p*=|ugQ+yA3!4U~gHBryTI*-It_}Xg;?w1|XU+1x9 zm>Yt(jmGsS!a6-;^!W1*?!!%Fdb4*>a$pdwv^Ksh{Z!q?!65Kltxh*r`SA# z`-ObDZUi|IbTaD}rQJO8t7d{LOC@1&Br&N7GHG!jSY>cb>4dCAV+twLpu(I_X)sFN zi|xDRI33ZhfOA4^-Q*If<6rEsv!R_s@x>czRz*MbEw$0WYsiMdnvMrGWW?$#z1fnS zevP&q)+U)=_V7(^v&(~btEM>tWOc+E#UJMKwy-27P%;lSFOYX3ss_{BUwi)X4eN?L zcqaUTdMAN3!aW?Uq~WNYY6x?MU|L<3hEMr|u0@WZX_tmlQeM+t0$xUxg= zFxw^lM-(GQpb-tqk2W@6d2%8DBcOIbJsjomgSMC~t-cmm0hXc7r-ek` zr677mQK+#4P)fs!7LIBa&A!Jx{Jf2mBP@ zU(WJr)+(x(a0@R{Pm-zQ@KEsXW80jz^V}t#E$a^l{0= zw-C~VSKqD77UB5Ij>W2R8Xcbvox1IPNRTI4Ib!1XnW6g$;v3qi(7U)UsKdtEx)p11 zyAnb*N_MTeKEoX7DkCXKos_;o%*(w+w*YJOr}2wK6+I-bbHVgNF9XZ=zU?R)^Dttb z1skf>I@LO&0*$~H<4Ia@D9dhb#J~IFemRw1L9Bk^_SlUtSH*0*O6Gt(sW10iwd1_0 zYij$<=!G?>U*j%*xOgxvBkwk{N;n*SMLbWYIbv@n6W0CQ(0G2Qx1s3-DW)6I!E8cP zGskRi=q(9f>^=$G%yKi<>4%yXMUyEpO;SY=eip)yx3!|cbh zVJ_&Tn0)2{_&jqHU0b{ib?uJnLYOj}QrP@bR}a7`STlthvLa$ZRe|-FayA|x*I8|? za)3CR+Iy86amM7~Yf+JXryMX75QqC@TGOp5vP|3y1Z1>a?lM9m{jEF><_IA`58mC| zW?##t7U|CvIpAV0_)t0X`xV7O3;O>jI5vd%9Ij7?0e1(o z%WfC9%8of!1;r=<0IU3tZV#$RWp7jM;Pn3Bj>mg|b$2E1_F2APhuK-@rL*2IQWMl2 z_W~DHFb~jlasPH4h0zlUQ;A+^m7Zb&K$vF8DbEo1n5mG^ydonDzg>CYyWpxP8RLkprU|2dyc?Bwvu*?p*wyXHm;>qB?qW-M$OTwzEla!dK>`oTE^_4c_@eG z$0w}Fko{6g4qcjyxwg!QPyH6`W5O(_Wn+=i@KgS6`BjVON3PSc?NgWar)5}y$+Oxc z^gbLR@wr~BZKTXs!F`=b0?8obd)VJ#X8+$*d!)?dt9$-3|BKYS@Xp_1i~o;L3J!my Vr z`(Q9L-dmsJ_kEA=AMf$Lf4u!Mjd`BuUa#l6&g(p{J4#DKMC|mE*-7RN`4RW%4`zDuLJzS`2IuqJe%AP3 zz0C6PIm-{VWlo=0ClH}&&Q-Z${gT8N>zw1_()n*xRQaP7Z7a1>v@WbqArgy&I%yXq zv@|BK9ZcO{Drw%ncHxNTw2LW-i*u$#b?kwL9QltdOa4a!4dyyCb zm&snlfByU$3=R@BX-TVo`sE>>?DW6c^QWJooTs1k@YB!#E{Jf)=Usq6ESW_5_V@R@ zMBehMG4&P{+|Xu9z`xgKnykM~8C+T#jm&vNCI}vTCnHB+TyDxdY`qUqTR!qY-=QKBZH0SqHNzR zD;1zTBS;SE3wKQ4^J`mwY{hC&rjM?7nQ8U)Tt(#nT-|f3lg`?vip?MwVGqEd#BTro zlf=swKH^k2>avWx5``>=Mq9B~tlA+QYbJkswXFvs9n+oB%MFQWEqaP>ktukgt&wN2 z;GnangBbfHKSbpI7VV9}>doWrm&jGWRv2G6wKNCfv59ecDnhkp@@ns1|M`nb+SQt? zVh%wc$Y(h(gj?ARH}I-CiLuL4AyATh${%Q$l+=dqrGK-lf6-qg?7Vo(tcVseL@TzQ zYJ0DxhF7fJdzQ0_mtY~yc4cWN?A#sFCFvtYZ84gVRnM~VYG;#FTdJSsm6f7>qb~dk zudUJbZj^`9NNA~^z_Rzfc-?Y?rskH4!TxI~ucneq6rYFi``H8Y<(JC~PZ)lpvTWUAPH*Bc>nZffk`;&<*rDoF1;z05YUH*L~A z*ffUHYJNX2H@Nk{rgo2!D)`4lw&voNtJ&PGM#Obz=Pk`b?I)-`9RF@YFKGCKIgR(s z@44S1*tk^Nk)PRp$#ykrCkyrx>=zdj`?%d$GI`Z9&xO8-$&QVnsd<3C(JbH0+byzm z25-x+AxabSjM;`sE$J<*hL@QklTzw}#_ac=#iywSP^vJM+kPyvycLqQBs=x3Ma8tVMF46ziC^617>~j2+=^XfwvCkZ z;x$7x!C-fC2PCK=@5CET9n&XTj3y26=;m{xq1m`tx|?qN7MrgS0nk4e9xlJhb*@af zFZu$#ABgcXOPAfUQ^QaiK6hu}6@>0Mi)E-|4h|O^oavj5%uwqJ__i?>Z38cvjeCOw zU+76zUHo4r|H6gMDsG$FVt>E$-}n2_S?JHjf)AA&=BL1U;gkofr(W$v@!udzJkP*a z35`xK?3LY>rGlY~LhbCS5pyPu!F;AmZ3*&(R}9^cjf0w;6Q6#f??|if(sa5eIVUOz zJu1elj^F&jmMzQsfr8*S#b+sx|IGC@b3cMk3GW+g!PJZ2oih01|30&)pK_)RtIIq? zTv{z|;1x>HT-;tIPn4(|Ge4+j#D#8o>kwbeYTkO;%5vbqny^VOcH887=cXcdS)_cyIYljc##p!@yAbE$AQ7fYj8~_?>-d^yuCvR%^{M;L`fO`**?#c|WdUh-?YZqNfKtio;2A`hI+{Um`bY(+$J-_0UWhjuM zr(Y0vA($(tE&i_E1qd_ZwUM#0)BWQ@v!<0j>G#xbV*!8Il%Ktz3g(hlOyW%^`yt5t z;2vf7k6oD^{75lx^mRWTh$+kC0v&!$@!KzMr99gHYIp(4W2Ud)_q5LVnn5P#jM~_( zjU~UkiM)>1*4<+2>6Nu+*V2d=0>4>;e(&0fkpzW>cjGkXI4(hEhtDSQ?vVnLtL;=! z44f~G449O%7DqbfU09Uf+^O&Az+ew{ZatA&aBx?bh~QNdo92DuE6soB-cvKPUQIfP z*NOF9g)Ly*k$~iMPcU=;5zEQke1gdA1{ug3K zA`y?ycmF#%n_iPef)lS}Q&`iO`HI)YsP^9Z_XLGLVOswfWMfm)@i%3QgMFCWrj--vTC6qQ~IG5^hzX z!@Znj+q6D^|IPrP_#W-@(RyrEO3LL2l#1H5)z$Lk+Gx^)cO%(VRBuR6Ei4ky#CR<3 zS#tg&eH1~Z-_5|lUWo%VJ&`UYRwvTQl7vR`d9K$y(k zr{)i*M0sJy&cIQ6?IO8SY{v|}F;}(VoR~GKc4t=oOW*KUH)7X}`7fm$cm8OWb&k7i zIA>SGX2Fc9AG}e&aKjWOPQzo~(L81oe58QXN!dImlveYf--e@0ei{Z)W-P=BL6*cQ zd;Fhwy>?+$ZYH-=)kZ!H4Bt9#&;In~CQlkIkO+y_=L>e>k~Cu zV-KppLe_67{3sO9!y`juM-GnKVZY+J6<+WEjIw}SWG8OSPCAGMVEVbpXN74p3khuZ zZM@Z2^UQYcwu`m%>znB2t=jJL2Gx6YG$|xlFu)xke2=-;46C8^Y(p3N{UpWYIz;+{ zx&AzV!OX&PS&2=d{EGSF-K-a})BcdcB;d`&UvC~WuKg`^)Tu`p>#|xee$ULCpW}GW z5cDSa%mbcK9pmP|8?@I-cVhO>Lt-zsOIHE6$_P0WDDsKs4B8@xK%f%og#*=39B7H}K_FdprwvoWPCx(a~k~XiJ^fT(% z;x`^rOdJa;+41&v>79RqK9hxXds3a85Qa~{GrzF+N!-@g<3xXrF}u9JB);=@xhFO@ zD9oR9N6ON*^Un{Kv#xt405d(>9ILqQSK!yN*5jCJyZ4Jb4@%Yj=|EXXOO#_MS)4%(2(cCKM!x!M_>YWp6Z z2ja@AbS7W)GeTb=rlXvxg7D2>#w19)im7FL3UN@RQs2_Z+nwSAq2S&WA3seuG=)IB?PuOV-@f*w+R1@d?^eY7NhRkuNUlN5C)A_x-ev8 zD$y|pqN)3j-r*-b^GhF$%O@n-FCY9pY{x7=YldqV^>M=5Y6rC0Ihiie(4IE)M9rO3 zRi-DCUg-|W0`xc;I2Xn5cYRbcobERHJLA~|&81*2hx^3*g4D2U-%wbOiWA?Z_1;BlcUO$xy?A_(yuD(%&A;GfwsK~lSzxWjDC*E z6&*gPzTsJN^3%QU`~#Uiv3P+l%q}&rjIerIZgBWBAznq>Cf)AY-wgC%d`)w+fZFKM zk+kgBJ-^?(tv&C{%F2Sd-W95yl%rO0lGiLOELbRB-%-95Ank=RZ-?^Rm^y8C_@MiS z5C2aWu6k5y9o@Tni~~}sw%ggyFmU|4>Je${gGW!gFt0pNn zB$`)^?SD|t``mH8J&Awj3;}&VP+J+R8Lg2vwalW4aYz5}Z4E71KB>JPMpAE`lIA5= z*wsw#`)CNYteuw;%?E^GgASJqDwca_e{9g-Hngu!+y0w^; zP+pgnX8Xm}Mb&GJ%g@REk%})Z{d9bMcJ-8Rn5Q1z{ZfY5{rjzPg)xcuuwApiis1?~ zm~T**0&Rd9N;C=oWQ-=5D;g<6L!G|5=sUBq@zQTuj-H}@lR7x3)alub0%Io5?bJgD zA-h<4JS_CbW5<}3-tgXxMwXYC2U8)Kl-@Qs*4Hz4oJH5m4(Am{Qp>;)zP^W047K6u zV^^yh8dMOTDIRJ`szm|i>hUj^I5dY$($Oz5wI0Egj| zBNJ_=QId~QC&B!+!do`(1fzo?j%GyF;k0w);h|H1=8urHwfa=8-#gi2$+%n7DCylI??> zElq(lV3{rZ>R>Gb*;uJyu1KWM+c4+D5x9<~P^I2Sk(W!!RNhI30-jK{Bxf`dY9Nq8 z><_5`;cFy)ymLbdA(o*0B@qtIadZ3LC#$3?nmc^zvF2APfx+ zvEkl?0*#Hh^>yc|j)&SzgVi=h-``1m+w)OpdY!`~b4HG(Ka)%$_F?RM$r&H)xS@rT zJvk&uCTCT8`ODFfpR~SAG!nqXsgEw!)_G$Q1R~}n@UHj6l{gcSW=?hf#mkpTGzD%~i=fP~C_5&l&)lY)hcrPkFRrI#eK;IF zzlBYCq?D8|c<=ra8H1x4ijI~RO3NKR_xZD2&162EW1TTUtXpI>`-fBADRt9*0746e zoP$J$Sx4H-9ZMWE+I_AkYOF|;6ZHP&Vm^zr*_r(N&fvKu-jNTMGE_+D*AbWGC(l7n zEGd65n9MPHyIZ_r+=(q*PA7xtB`Z!9rnBVa^EAvHX%xQN3}}m=U)wi}V4v%qjZzO0 zwniI%w${uf{_KFeBOgH@d>Uor?q(=F1alzD3loaYKKfY{eVtIT+P5gSoi<0kTyxRR zC{9^8x}%EYtc}=s0nH3+lnoY8{USGnnlPLr&)v zIuZ&STluDss|D@##B;=H0i7mEdk5uD5q$k(=5=}#6O(01icC!tF_ z3w+n_7Vnb7xnX)~4hV!P4B$T2Thp)FOl2ASL@jQlwK-Kyz6iJS5mYvyj=gLdy_CxMnpvb!i7Bi+~w9uX9uFHbw?A zN#}V{Cz|WEN`9em+|y z?jLZ$Vw#QKd)fO)c1q~d5P59XeeaE}q&5&c6J+;)Jr-{3VQq24o&#Acj!}dI6TW#*?}G)X^L7W zSoVSy7r|FW`>jdC%QyS|ugie_xl`+}4eZLhfz;xS?cqeZ3BP7mEN8&hA2Q=};HoN< zl3q@OEBdnn9<4BvZk@;9j;RqGv^Y6JVW&>oHUsJC0Jl9)KUiH&jf*#uqxuX2&U9a2 z-!Eo6z@YTjon&ALe?5j}5B_GjPz7tfRnH`YSf2Y9s6IAj@Fpr@B`+LRbCaTx4 zR~vR`Kc)j8lTUQG0cLZhwDg?Zj~5bEuR4V3v73?5=#w4c6Zv7?HglB_5M|*xzb!6y z2d#9^ao*Kt3Y_QRY;v6iRNceprx%BaZEbB082FXv2Ugp3Y-&jlcHa+&T0L#VfxUlc zaJ4`8Oq*8VQv%QHJFkFU0Kyt`Falz{h})wAkiEnylXua@Ab2B@#jDc2$*0j@#OTRb zEHW~BH&5JhJ0Thx4Q@n2ML;TI8JtM{%E~Ht@&)qr5`F2Ji+Nye`KT{1)j3~LCyU`! z2G(SG_1q`CiY0EWunlZ$AI*AC2+;=^R>)U5?0ewxwi&~7u~N+ygVnJUQdF*c2P$>O zURTdwz79GV%6TX(2xldNQQZ2!zbd?aR6ryKGA)Jm(w%l#$l33Pgv!4*e@gf_tRt(~ zltEjNjaQv#hQ*$e?{KF&_RC93x}9?oBLJ$5xolV`)J`&xL}(i7>-%=ioLh8syC_s2J@Vl|ixkoO?BR(c~i2(T@X4xEHT5lM`rT$F-?^7&w*;gP2= zUY3*_Qcs}Qp8S7sNH{x0aPdm;NJm?nx;CJ-AS(U4tz48EAEdSmgag&+DU4Dtc5j{X zt2<1SrZOzUlL;X5~vc1b|hWcLV>~g8L^U@GkPgd)cM`uGq9Mk&~I6FYV zfC3oP7qXOI_`cAn+#rO@^{R@#{x2d3zSg+=@ndNtF!kf3&Gq##t4Mybpj=!b7v=uJ zfxWM5F=|ztiMSO3W1^V-`z>Sn=RLI~{23V2{_9=~-aL!3+T~A4MMM5%ojp~8Wv)?k z-RxIct?P`}d!iUO+mB!FPaMB}c3DZAHT=t7$d=~N66V*hP7VfO3nF#KasHY$ix;Dj zs!2>rBNGz_v!R&XRzUjTg15LRi#Z-PU#CBm2-mqu2{@cn8-eggU+;IbF)D<}8Q$Zc z6@K$?$M)|oM+S|+sCE4Xd}5IR-Mrke@84rgoM&;HPNa=OMtY`F zb|>E9i@t6hE5jDBGUgx_UT|d81v64n%#DL99<}U`8<#Uz=Ej{qZ2ZNVWuhCW$aVwgEg3MZwMge{dSYL&=sHOnv>(S3~$)&k;_Qj;HU#UtSM7u*E zjlmRhmaO5tYSTa0nwb(ki*l%ARD0F_EK~l2PQV=#I;In!%XUn%x<+~0`Ixg-4m z&0fKSY*g;xRg1*g1LN)z0SkISRL+wdOchhFac-G>kwiD(_MvM1D<$Mh)HrYS9AG1e zX|=Exrsn?u)YBh-?xqa_zZ3cocG*d=ENJ*-sW{(l+}a&mpx6|-iO-*!zNFOvK`a{^+FvZj%yO)N8)la&)Byj_T@1M%LMQ}1M zfq`w4FF5q^{7aFnqI4WGK6~SaqSKGsOtzXUx|cR7zF}IJ7q}1%rDSJBc%G)-P-Boh zaD2-pZe(Gh8Jgl4@YKjCxAhOcg~3?$uR|91n}RF(2sM)?H%`P&9%NNr-8f3w&BNM5 z5w{V>3QM59nKroHniR^um!%}nnS{>BWQwgaKjY3Ml&iuPcdqlNLf);WBW%BFFo-Ix5P}nGn z2@dWAA{-qZfq4q5D0Y}fZhIWzmhH2A|Beh#zXRC%^Y}ZP4bD1mh9cnT z*JJH?jV57=Asxybgyw4qd-PuujYt0=H$OSqBV&B5T&Z4}0<6XJA zR?~IT=hD=_@6`(%rFyx3>0}U1(^To9xTG{v{ZTn@dpiNqYF>$0v*ygZ(&4j>q6*rNgRP~MNr6rEUG|f5`!ta@j2|V2udt& z*t*f|T0_ME)yfL=TH%)F*GqSRDxA)O5EZ%0EK(xvW z7Ea)7C&%Vo0WP;!kfbxshJGPY15?pv>K36rdcCo4iPNh4T`yc`T#obbT^=nke~|iD z7DzW13>3MTi7&U=%F}{4-mN8PIZ|{@kY|`=-3|<-S@nmc* zJz2hJB!qUI|K34h_Qb2g8o3fJ#4GK1jo8l{Dss84ce1qDLlKeKm4(4HFsw_kUoGzb zSt?PwNxV%J_cQ!ESB&28mjfFlo3}2zFMW*wVBh=sSPLR96_EiTCtK= zDoHCL=jld;ZA}Q;Oh%~-Qt{B}a4XAx56TmKBDjC{{Tafyy1)9ue`mn%Xc%Ax>>EW zkEa(a%vzN9gB;Vx(v{E63jdM`YSjc=X|FyuJvD{=I_m6fYKB_fla3`R);L7u+yv+l z`ohU*Vr50(M$~COkWHB9M-ciEAA*Uk4m9|FKPs>F!hH@{&)9N4TG8LFtrVTFOnuYHk`H}z%9b(rTz~ylg z9cm_dJx*r3BQo_c_0GklCnkheOOk5#7oKy8ylTiWr`qTFNNBxtKP)*Gj;^1bJ_i}% zI(_TU%Hh!+M3bpjoIPUSUP(Iuzhxyoj%d8HdRbl$q!^q=sUcjf!_rf^OI3xA(a4`p z;&+KRRb*e{z#?%`emZj*OoT$t729`Fz(zww$gJ5DWV7CwKuK;wnvfa0LAkf_Eh&z( zHw~`KSC|n4=#Y0rK0e|KNZMs>ou9rI*N#%twgm4btU0BN+!x_{IIjhFK? z^uy#lDj3~wQ+Q9Aw{h{^vpe14IAkJE%tu>`Q6!L_;g$! z>#mH8;_4DuCib@ULg2FAnIc}RiA?N_KY;{XtpApWL9#N2hh3|Hcj*bFok&;{R z(TgT9Gb_R+0DC3eN{jJcyvM@VlontC!0Z;%EbL;}*VZV5U3caOK!R0}@uoU2{TpA( zo~fy+9@E(plFFHuz}UCI(a2Gh-WzwS^C_7kv&&n)4cyM$@nkJ=2rsvh;8bV4eNoh5 ztoOBjznNt5pWWCP7GWJ8nyN7BD!TKi4&}T{u<15$%n8wSXNne`AKQ}hgPXIL*Tp0ZQJ24vR17a{Wi%=rBNr@f@g5V}_ zGhrm51-CifR8v*ucDQ4!uirsjcexV2ow*qhfhM;gt{9$`YxLW5>WZ|FE&up2QCBzB zwp%0s<878UqiW(0h^AT*L6T_(DPYA9nA~T;0i|3Cmvf|Od&hSn1|t%rvsJwVB4sM}WqtO8M$UwSM7jcQuXR z;N0;jRqz(w$rc?Eu9`PeJnjU{r|+us<0%*7N?l!@d(#jr&zMD0exAzUjNfW`Mbqm! z?%EDfw}Xw}N8B*t7>qd2Ep}stI+~Wbi&7)s`v)c(Ng>zCO5L5f_%QKt_$J&)D-Yqc z;SKnKRV=YYGjOF2Wgf%ni;a^k?yzWdeVw(%v2HEkasL{2Cc~xukaWBW^V#|{*BfJT z_|4hV2c#H`(TdbTTJ@c2?HcE3xqER06>zb)2J+=AZ+P(OET{Xxv!eN)&2jHTClQ5(Y#SnjE)>qra z8sbz|OB%-^pWW5R0x(4ZB(Pw47(-zNk?d{W>D*M?Yd3yZLKoT^wW}K%(mn+ZF1SmR z)Y;NJd83g|b)}_FuY>!?JUiNkqR>LJMK_ko8;`Pm=AxEax0fUXcfUzy`>mAJdp+(r zA#V1!1<5pPYKE1TPIl7OPt&ubVoiP z8E~XqnVJIi7V%h`-eC;26)V6kH3=ND$dglN6mBKP9*43(WD4)R_fw@F=)BmSh`e1b znCzyY0+3UftJ~&{iwLpxJdukVk}ix#KB<%7g}goJB$0utw8-9GzsJnXTu2%oC_th)Mjr+{*2BmwRzl9?A8FecWtuN!8$;RupI zKu(xf>Y}dGky*&Hs-)VgB{|hi`8>0B8}IO3PFigS0sh3KCkn7VcBMX;!Q`-TPvSphq0?#lpx2il3cE%iB` z*SYI4u&iDqT^-nDJ5=<%rKRQZggcOXXb`Zr$p#oKxSF&p4p2K2mug^n+)kSxCPUdlr8VFQ%5n z9(S$*50o7Sqz3i*FWYsWJ|!;EZ9loOnpiJ<5jB3A3rPrjpW}Wk8IAlyUx6^abSRwR zI@*H(>m03f)^}wy&F=O$JI2013E_79T-|e&;2<(;gj%>&s}gmx&3Upd3<#@|WsU~{ zK8HICtEkGcLqOZ(R8Zes{>(@8X5WVv3#*NS6&dJybya1omWP({>-k#og&S%}!`+;^ z+Q>_#*&@NtXuH_nvB@tNn8K~l9}0WOb#wMVY#bMq-qj+S7X}T(PFk^2yz`Z^5ar+!e{jdp(Hl3XgW636O;%o^0i`7dlSQl#e^64G)MY8>_9XU^x5doX$2CKh+xTnB|0PPlrAn*~0FSmXCWU3pM z^P=R@VtkNUPRZ>YK>ORIB{yl-OQk!c-iB`!&62rne3?}+pN`BA{>qy;%YF7y(!09j zvu&(I+~7=`|G@!#^I-1fU<4g!`&JhwWdj`;iD{s9RjYU|G}TKo@$5Q8AsF!z}aGc@-NV-`skSCY%_ zS^H5fSo|Oq@eB}|C3;9GGH`c_T^0$Ytq%ay4%OFzWd3aU(K6r!r{Y(K-isok?iaE% zGI%^9CjySzPXOqb1P*L!a`G_D0w30K+|rR}D+el%K-9WP`E@qp)>0%O?p)IUR=JIM z9h7T$QmuG6T&k&9_0b`N)bu5gmJ()~y<+NbxBDNwlqnwez~7`r79V7sXrK_S|p?^o%8j=1=b}akO00pSYbYot5{cP#sKU8@$_e|r_dD8fC zrWNk2^%@Gkj(Uk{|1D9>nLR2UpSLP~CKVE7G(iIEwy?7>h^jDKT?kMgvoQjta7=$- zU+3Zj1isjOc5j!1TaSq$=J?@r`|IfKlq0s`=}tF_A3WBs3l~2T@37vKqmP~SR=H32 zrp&OC&p^V~^Ojl?2vnKe<>s;E%jrKGB`|yl>`T_;yM6Ty4Tpsz zmQ1l8mxTpF{+>Th+zw!w^Xw+hDr)EV0&!p!(5xv6i34=AoMkhJy)ugh{{co|SOJF% zX*C#=vT;^xkpmT~1tRjA-Nn+~kXO*_skWdrBwgcovsc;kJGw@xj%Gl#lcR5}h*^il zQ<@U5?X?@1lYS)-pBCu2UFFX`3g!w}xMi_?#2t_AS=9i%UyeS0{!^11RykqUA6;*T z`i;RPY_0%NK_Qv&s6f7Euy%&A?Zv=?6ew5GkcU7V%7`7f#~rw-=aqX|bd`?Ac_Uh< z*+^Mww5%^TK|x_m4CR&a?JrQh%EosBVQ=Ex{|>!3pZi)o0yJhe(>A|4%*SKLulQY= zVHsXG8Ok%#S>FTJJg)-HSy9{54CVF#&@DS5h#h{!sCWx6B`}#!Wvmd0O>twzDpS+u zDySY7XO9D~%cBCFas$TuL?%#eQJiD-OgzUK?I}|=d=_@HbU;13Nol>tn2is`N`d_2 zXcuq0*K0oiK}WO7nF+M`aDM~hoBnoNWjEZt$+R&AH*fYg{8`t_p$Bl!WQHzG zmIw{vuRZ)Pz*eF(*rUBwnRh=QYgn*`|C@^}J5dkSiaB7G#U^Yj_BL>H9a_e|d8HV! zmBt>|r=b5E2a0d|7C2-Xw;0A~F8j!i#8aK4_fX>xtjoa0P zI`natNLk0O3jX6TYcp(fz%+m(I5MF~~dep0fjsPdv1*qXu^TW6+N4!05Y*gKe`y%=3lo31xMjBXb zwm-nO?SUD=N6rVXYi^)sRTdD{H}ha`wozD(pyV*coD zjP!=k!Ri2vg|RVvHBht(BJc;eT)Nx0Z(qA}ZTPhGKuZy%%;!XpA{rM{dDU!k{v!Ii zZ~Iq#!ZDyG0iBKC=Kj^(_C&ewDjW)}{mtuy1H07|M>-=n>WG%G7JLaC>j|Ukrtvn} z5<>3?2@CsYp>7iBm9*(8(hKV@!kDVwj&@SW(SyiL2+|0(lDf9>8d&kGOF`Uo%@v!+ zqEb>XPP2b&qlqUAD!O`^OrmhCja@_p4QdWHuIz7Gl$ zYqFo&MMpTYtFKhnp7%Ak5cdObr`Y<H&j6Gw5ti$ zM>zS7U)ziXK?|H5fFWMR9*fP-EDgG3YlKpq7Pizl;`jw-`}Pk0{NcjJg@lDRyNo7Z zgF76LK9N)M7gG;O1hUS^&-`Gliujv^hlrwXI$XbNH(f8%pv>wq5O`TDMqnY! z#Cr^+#?AzOf|isAr??*|K2SzzOU(?bS<}Lpz|jQfKGynqGDuMF6=DtZ@^D%&owSt^ z^X1Hky1;qG!YNPQVtvh-lp;4kkCCxJGvFp1p82mGSSZ3E$JfC^$sgR>xfHR*a`}-@ zXf_CmmN*4`#8B19*rw;*Q{BVUOl1U&=u6E=w!+!>t)+IA{TYH*tfLQltRL!+&#=US zB(vWQq}HeBT$HJ-*QT310Qe+U&~X+8gQDYp$-;;&>`kyXB2qvQU-iNqR23qiTzwcx zAjRB~xMs3U4tYqD%jfKZS;-71^Il-o1PeeXGpgy2IzxH#6!_-o>+g=i^YA=Q>rE~E z8)5i%j6>Lrk(rq+U_=0l`~f_chy>8Az)6+%{kqD`dmemMnu0;SF!|FejpBzT8Q=z6 zW!QBhJ*n!MQ7SYVgm4F-!?KDpEf?JtaEQwbS_o)k2)Al7o$&fy?_3L{+*nRy^|nu+ zfSx+S4rEB$Wm%oaV6~KCf}~~sXrxXFg~z}3>YmHy_UOX!Ih$kXeair*g6> zlalqVcQy6(e1q=BduQdInws{6fb&y_Vn=1;rKgIirk0$ZzCP1g<`ep2J%OynxWyJG zrP)Cowp5rhc=Pz|kow>G|AT2%}rbTt&fcX%4;>Twt@2@{=8nGTG9_GuoHHtTHWo93Q*re zH+WhO)fY$7cmq#7H+V=&LmW3Cc*X%i6wu{D4vtU-ZI}HMs@eqy!>#b`o6SOM{!B_K zkHQRfP+pmZXB;Fc5uBwbr+L=@$uNoEBr!4`e}@Vlr~SAK`nv?`TUf)9(4%&Y1JF-& zn3TrE(DfPL_}<&cW49<=KQ2dX9muhUgIuFy=T-+W5W&H$#oyp4kOyExffgUp2VS#n zKzRA?fzwj&H8PWVqlD|{_I_G{x8?X@>H?^P*gYYg^3ViAEBp(A5I_KeJj6O^0<+r8 zp*i>XVG`%594P|c!4O7@&p^Pif3V9|I&ut{MKoaLRBC`N`~pF;sj}3m50HpjXC{C) zc_|J)ygQ^YdRMXEm(Mz&Vr*Y3^hA}=Ye5L(qLi_s@CQ|7o$n^m=seinp}^o!STwRx zjJbCy0a*;EJa|f9?>y#)@Uej36x99 zcBT1|PO7@RD*LroIVm3+t?CpdLc?Z_gx>8#D&W@uS3;w-S0hNe;K$9^ygN(lMqfPeYKE>Um=XFypo9!l)d9at0V}RAs$}%!V00s@%uemu#?v~~c z>N}U(585_m0Ran2V$S$o1GOIXF9UB3eHY7g2Q-^aEQq7`7rh?|d}qaaKbkn%+1UY{ z^oj5RMeOt(zpx@925U~4?$m2`18YShB+k3M1xKtc=qo$nr5 z_^;Q=KP7AjD8JkWY;tYEH?Q~ejXEWuegH5An-cXN_i*yDK)u>%J(%$q0b!SmPOCzn zo+6Vs?vC=IeP;A-M&WyeAq-_y{h`V6_Tb%bc8$53%u?A=|q!(yrr7E2g&xNKCVn549qr() zoNi3$=m}v5vTm)9oHkSGJ+vXm(KbgL{k)IbQ^KTqBv6OLbHO>c(!T-nt^g>hByi-` zoK6($az zjjpb(?YW3LE#Vk-4b83?*ln$432r%BRy7e0tymp*8b2U{OD`lm+PUCxIEO5#8F2^r zG)@_uMlm)`RmJ7W;%l6b^>4R6i%)(mk7_>`=Y$wH%&wo91#^yM(r>w+eOqf1Z+hg3r zI^MKK#+q*qE&QMceh-{3bVK+~u_ZnLMeFwtOAph~B|{16lpidUDjF%r@my!yO_ i{$G>211BJdI5}xKZ;+t4(cKAdgs3TLC|1Z@2LE69P=j&+ literal 0 HcmV?d00001 diff --git a/5-NLP/14-Embeddings/images/offset-sequence-representation.png b/5-NLP/14-Embeddings/images/offset-sequence-representation.png new file mode 100644 index 0000000000000000000000000000000000000000..2eb982e814aad86a0e0df589ecfd6797dc624aee GIT binary patch literal 38594 zcmeFYcTiJZ)GrK(0*W943evlDDWUf&LZtU5RS784rACS%f?%P@q4yAa@1P)pU<6D; zF9H&J4aHDzz~_11_r2fq&V2XY`TqE3t}~1$`>eh8Z~fM9t-Vj?BwFvj=GDu$E)x(C zT-Ca(W)C(A50$~n1O%1wSB~s1;y)958EC2ylznGh!haxp zde_{GfZzt#*&m^n0p})uP)tw9Nd5Hmw4${1_wV0fs&Cb#h3F_xZo^@8`6Mvr~lVK4YIe&(s`2h20OgS2;x- zhC24m3*xUgG8Rk&CcXj$6)BwjO=Yg^)bsK;wy*VnsfD6gg z^hH`rPFKUo2vX_CLTq=zw-O67bDXJ$!IIVrT`NNH)F-dX#{6F34aSW@g|N{DRV$vK zpphDO1+hM%l@n(pUt z6S&=`0&%c>N76SxXCgCIsCb*z*9Fwb2hd~riO~;@?APx@?Q=AJL><1rT;TJD+P0$-z7NJ z86;`-Oca`8jjMjj(e&D?&gcuro@M^nyj>ZkGiIHBLp>R~{3@r+@A1cW2um7Syn z!t=H`qJe=K#JCF~azDbg!p9hkN37b2R7jaXg&vCCF*z#-5UAenWWpWHbN$GGo5ZgG zMc4X5`P1Xm7B5g?0%2XoOML(zv z*doFp^p@9%(SP|MdgdcoM$)w7KAMien%jQe9%W*cB#Q8K4&<;IRN4OA! zpw3mZkpaKDHxB78qvNyRNbaigr3QdP{fv{=p4h;lI_drjrfx&KK%c1+9b2xmXje?l zS5scBWk#VxR)_N⪼=z!?7e%q3?&{U9EO%&( z@E*Z{LwY$}0x22{gt<2nb)(@x4}S1NH-#)^VXk=Gho)lfQ-)*((mCQQM{JL4hu zjj;|whXE(SyZtqc$fF6p5B4h;X6tRhKh+fjp^u|mN|UR;En|jh#)M4|lgpp$b*Llq z+`4&Z)6ulb7FuCy2;bq2^4Ea+fv^U`rHs+U%N)9fDHGKeHVP_k{1UuzB@a&4;{1bg ztWqdx+`)aEMYen!+MnGQUyBd-;}-L6$bG1xU*k8)i{S)52-6sVjn6$k9%wH{4Qn78 zV|Hit7O~AMcnOCNizSQsHPo2O*9Y zY_h^xx=MtUtg-WOAT$S~8VFhtLX@}8Nh0j$H*;fZWn^w(pliXxZ{ggVp2}WROyj{% zqOI~jbr$rj1J-ZNn~$W{xc%ER!h~%JKSjVH%n)D&tfW4e=OV)bZ+J2y02FFopd4V? zTo7y)cYg&S?E{Y)nx8O}VfD;T12U&^x9}xpvnp0Ew;H#myJ6m7Z|W!-6x?Ej<&MH> z9no;zT-(NT`CLUEVGPDTt6giz`J=UIfttSgs%Q-Gl0;!U8 zwaqmr2b(QH7l|47@v@YrncTN@4Pil~t|@$*Yd2#KL=N{;*n}1XShn4VFGk3Bdtm;{ z0->%s7@E_hm^kjoQRUfs&Z_A`nFqfnt+6?HRU{^iez2*0$fmne&2yS)yRr*UDLYC_ zit_l@Ta1sWLk&@4y(+%})hKp*^s4RhvmM=^Fx=M8Y^?LMkm}V7fuMNrvsk)oEaceB zhsA8G7?Po$nbSU`Y4}*~X<-4Tw=8zmcl#qcR{KdHiBJ|Bah6Q)PTFhtKfp0G!UyX4 zaHgS^TAat~b~a>MzVQo-1mt`3a`pBql(6l?o`*5Q!5T2tM0lF1edRl@{VX&*-o1XM z`%FH#hT}$#?F!08WPka9?0yCR*Atz0g$e?~wp0SvB!_$Ruh+|uytv_nA@@JP8W&e& zL(&FYj2$bp0=IA8B3?(YzPkB4B$7lOF>MXhrx!O)-0)E2U(y0_=iZ>-Sff;uW`iTK^y4z+j*Ybv{Q=0PYzx^&=;dLR2*A1J|-7ZcqHT)MWF9x{uP+*iXP5&>vmFHFg#Tc zt-sg`HugHP*+*Z8IQ#G2cO_W)Ri}mFNl2sSL<;cEcMqlZN^svmw@F!|C*4liTu58mSs-c_v=Nx``ffRm~b0YW5kM_ zB81zPBnY8z{sC1qA3C;Wru^_P6AH zulmG&GBS>PC+*dzm@HkAZ-zVaLL!?K^|4J#J3BDxrJJaI=onQ!9}RJ=8p6mYGgS<7Z}sDXuAtrCMbx*a1_R8tKba!) zq!&-^N-!y~qGE3Wz>x+L_GAs0ItOb`kUGk`i{|9uXuj~~=Y*MU0mt0!)=xJ|wP60a zex$F^mt-?;|LbueZ3m~YZNp+7cY}$|<;`yv{U+n?=F*ck>y%KPAbo5gK z8_gp!Uw)K(a1ZZ*M;q?^Azu@G6jo=;@wU?>aLknW6Bcp_>{q_515}NFNZ*)+w)%tC%@e?XW$OwJcv3})e>GM9LMUlozlYkpg$ze-RR7A^mhvf$*^~K;f&;E)A#Ss!+weY%_4efL z#Q0!DU>~+>;wovw4Uj|Y3qn+SD&Iv_tY@r)TM6dZptoIiheoi^i!JYbc-0e@4q(X` zT;r7$M0?$%kz=qoPeH!~BVupeuJIzbAT&N>P_km@=_|Uf^Tpf0so-)nE(*G7ST{PA zrR`Su5)P_3Tz+JA96iM`5Y^KpWP9&5(@*c@8B_CNtM#HR>NLJZ@pMyjgygaX^w%S8 z;XSz9Rgr#K;uzn0hQzL<+z}7=`pn7Yj2+1Ggwak(%8ExscIaWM<#u%@2oFC7r*7bHY>@TI@Ww{*b*3hiD@Lzbce*0 zD|uaA(DWc-SK9Ec(_D{M{e`bt*LqrzRRYDh9(k!@m?U$Zc~D2IHFSr-t8;+nx7x)Q zrU(7$K3WY;cb*L7*i<(eGQ>M)TJ>UrqUK&|O)1XIAI<)r{k=O}ih}X#^Sv}p7k#h% zBb&7gViO8zgnf9pnIe~ai}0MmE2~4Y$fT09CdW~_3BQ%9yP*k>aNw%E6-;~ ze`<0fYr%7l%s?}uGVKCVV!De#9*v7P;~ohb{cO#Il1eZw6_eQLcwusBnO;72t%Uv+ z&@7#&7aJIC6kCt!^o)xwyDh)u^lTilMdkXDZnvl2%S^8lLS0aONu$K}%hlXv*H z&=xhB5;#`S`j`XGosGdH%CRoRsLt?vcpu$1Im_L)@eM8kV*cAfafE8)p^pq|!a4U@ zDg};U$b)fC*26F^OQY3iQk_`^7&IGwe+;nk6g6R4r+4t4j9h;TsE-aDuoR!3kxSG0 zsRG7omG9oY5@Z=1rn{`E)6OcDvA8A(f(&VdGN>T5^|51S;DjAx7%6IJqgE=I19jG! zMmC+FG07y4&dTnC`u3j#Z5$7zmF&}(uRdNIyQ7yxJB0K;sTM^ks`4H&g4hxv?u0MY zyBJ5M+^_!FA)I+?BMzY$-X)Jl^TYxT3#jxQY===d3N{Lwv7G(Am{3b~+XQ<4o%UyZrR9Xe+}Kc|y%s%*8=4vMaEsH-1Il3GbjAz3YB;?W(br049Mi zXsSYAai^$0R{RLKs(zKWvHj4w$f0U%XhBX3{X@^G~ubia+LawXVOLiCh-gAjFs?}N^z-l)i%Vpt3wHHcB}N-Ev5cXMLL619a-UHheNzd$c5!8 zuEbleq+O<++`fJFn(J=G2fI;$USwu?Zx_c;2PBVBQ?AgA z&H>3n55l zsr*|t1-qLhluhcoyOfS*Oq^wIQ%`G9@3ICQbJNBtCg;(Y(}HO>J=OWQ$1nv=L^BN7 z5H12}inTN}e0fmh8L=g>#>UyoFA05tM|FO0%Z?kuWiRNJ%CGgB^P(#4<6IO&wv<_y zq;ArxsX%^uxI3Au$PV~i!Wor3DjkjxFHnnd+_aG=-fVD`;w z6_;tx*whzAW=$}Z>Sh|v=lh#C(2BKOi{xA*T@X1g7*OVPukPA+y~J@E9_PH8lDYxY z#`W+5+~;zJBEVJ;#8iUY^CmoO*T&Qo9kYPowjUVDuPD&8?(8O!S+kD&-wk^Q35gu&`zO>3= zXIw4)n_Uy8rI9psO9e4mRAQj1M-BckoP!}C>#N&tJ85X0C4ogt);!iG5U9MHEVC{s zbc=<8CB&Gzd%3ABM#;q|ofe|tcYzHAnQa?BS+1o6p*!`fuc{!}e$QSJ=1U$%J66RR zlvKj~cg(5q=MO4~O(_HurAMvKXdMGAT9nyn#+)=**C{F^jxh<>okqtGXu1Miy{(fg zb%$RAu-v!U82boqx-7@60`13K#$uw4oq8Z@XA0;=(BMf_6v$e>D&!ve^K||-MRH^8 z&tk^&zK?1#L$?S+E~yt0{%Az!s6!_RhBaYi0a;y!N~%_D*oJyogTZH(8e25+W0QvL z3m{=o_(gJ*P6PVT+xR^+3|MM8(6}U1jy4*3wEhGH%7I9f>M_Yz-Pc6$JAfPv+kTP# zX`R{JML;Tl|0#^~xd8A@7IuCa~4mN&&b_CLU zZv^fmlKP}Q8F2VgfPOMCkz*lH;Bz!Ax~AGE}!xr#0H&|Hr$}9ZW zCubcs0?31KD}fWJ-FZFNw;hBHZ$5U;j0T=z{qw*4Jn~9&3K+5cICeX+?`HAWRfJgm zum?>^_d+|f)kle1IN4V}S^Xy8dH$TSHL$*+Uz~mT%&Fww`(7C*8fP$bgh#O%Vv&5u z;fY}he&@-?s2U@@=@&KPH%l~vulpVM6erD1#Vi(~d+N zesB()OxM0X;@y!4Dvx<_7_3y9?aG}|HVU1%aQt;zSF~!JnkE}_qSPAaedXThhpGiL zqs;(432ZWl7GpjNJG|QN8pn?5?RG)N(8H}B63GGvAj2%~ z!%RKHc==buI!rG;cu+ymY7&|3bMjwY@F^jY5%TnsfGk6A(lrLOEQx*>bm0koym39d zK#108#B)HO-nZogTftb5&AO{pzv)11BBa&~>Pojn3c6d5jnD%34MA57 zK<{(8Ph>@GO+siuuN9WqUHG4qbVDwmKek2yluWRTNjZAd>~dI6n6$}+>#b_AXLi%K zmm{A=a$mF)nT)2#0(Q%1hx8mq^TTDdCnj%06w5T9AiI=aG~eD9oe zPc`1rcI|lZ`{mED{nJy|E6><&v*t8BY0pc)|1hqEik8{--P0k!DI5KpzHupa1BEpvRsibQK6QQ9@o;J`JIt;AO%R4ntH^8R|{TB@EG zQA&Y*t1-dd^!SNLnPh$H^^vw6X8NQf?CaUo^~XZS?;T1oj*Z3$LztF;8~w>eR0%7@ znXzJ8&$L?R1>h=#mB={U&=h6yel~_>A+1i`{FBelRPSzE zgUGB8YJ%qyVWx!BAhH!fTkNuj`C^7>fn!qCSgu%(1x+4sqgFq2z;Kt0u|_$RciClX z{lj;6>`J(A-O9n#2ljr}>e7{6sjAFHyoMEZkJ$;xUV4BWTNd%Sna-e(G%85TUA|j+ z+iYA@cying?rdmB=69_0Py;dZtqk*&p$`>Y4!r`x<>J3uF#`vhVsk63D_%Q!dITDs zGt*vvo9=~ZgY=BXg)t$91r>y12W}utCiSWLSd>zKb^&4EMV4SKWz!ZgT9wiP1P(CW zoI3R7S1}gb-v2cAX{(jw4m%y<5R(hXNkPIYS7H05x(WExq#9UUTn(|2%PlMYA#`4; zdnZT|0>4DF&O%&4w#t`5BM!X)A$9hXDI8Zq^_%sJ# zClDSyriOhz79={O zJwpjWq%U@aaJ^hv@nI&mj+?jV)0V#;^?D2iiM5&A;*nVZ2_)<@t3T*cn-!%^RXwZx zPo149{gMLV>WR>j%*_)`803rc=~jpo>-7T_z8?4N0^?N}0>>I`i>`rFF-CMgr!Az; zUp!NPh9+2_0}@$iPDe?@MrfHR#YAn)jf&{K^~i}GY$#D;(Wcc_lO_wXUmbX<$1b$CDltiIx41yU6?`9NYRKngWs=~ zro`&b;I?o5zS9fKXD4W{aa!-PH3IcR-!KZ+jBNQ!D{m$;7*?hE&0K|ZTV+c%S@1(7 zgCjc3&Xsl3JL#+P!^yV=AESIkjHTXNZf85ss33fB7Mv4{w4i2h)DeRPLCF=Jrd$zZ z>qYGm9>)V?^bUlb+q#QIj!<+#()0z~c?Wq#k27dvjcEOp7-DcubdvxQB58uWN43&i z@$qqvuW%4~E<8Fi>G39{=?d|7c{awZoWlZIksuIdWEK)0!~lxca?|>#9uC~RWMADP zz(Y~Zo6molf6|hmh{pmD7Gi;f-KE}0C!r;g=+1)ELoZdh-D(McAdHRtIxzLlS}ypyP9n=VL}oj_gxaEeq{51Iv$Yj@?NaNA`y0_lD% zF_+k#{IsN0{Rl9h3A_p+(m8YlM+A^T!W=X8Sg)38iOT%_F2EZCAzA}ve1q%5pZ*~l zJ4q>19ZlTR2?jh`tp3u1u!}sH2k*Zj=^rHEv$JGeK zq^|cH2>kwb?b~s4!8fBvryNEg%KL=9OfSM{v)(1rH`{~(o;!rNg%nq0D-kIB6@y+w zXq%wHJLchsmmTn`t;*$w10wIg-y)M*0X}yzMo?y4Wp`HtfA<@kSe*-a5;$^^!BwJG zSMXh%Ha555!A)nY27xyCOXeEwy&Fj%-o@`IY-QeX=$3eYTM69X`sR7BLiRLQ*0Yn4 zGbp;xt;zMwd)_tbPKN@-a-(nOhlK8vPsgEuJ*E9x{#2DPYeVSQ{CVnGK>Fxd`t_Ng zJO^c-&#%*u!B;N+{?2?I)HgsCRMOllLe*CtZtedwQ~62xgErnY3J2?T5c?Z9FJ*9O zU`I^&?PhOx-U+meEZa({>><<1VXnX@9d`vWvwAFP()a1pOK^ZElJiaYI=ag3MLA-5 zKpWo&$e76m8y?&Lu7l%lUL5x1j4f-&pN}K=9;4KZ#h$<~hG)L%(bwyh=Bf6`2l1iF zndfld+g<`boo3P$M(GSvJy*BAQw1K~;oP6UEBUHqFQZOZ>RWO%+urP?) zs{Cdv&NpevU~mO#0HU{$w{xeur1RWZ6&G~LKAl!Aj?CFVFF4fjQEO@ro&jjLl1ln3 z)V$_~U1n{^FKSp(Zd5uy=g%%?rFY-*?_JYpK1-ub(LLKn6A#iC^OZvWv3SHuXY|R* z8-+wH9+vU1sl}+Kfm?1q|9E_1~+c`}R%OBD~T5iku$VvN_tUT`Mx1 zF7?%}iqz#3vsBMz{&(H+K3)c@tR@61bS>{%jkPBURDl4KM6QgU=FT0&=>6L!oX+xm zBYCR&;;+&57eDORXi_IEbMQC#?NbEnq2>0K?QK{es@1%II^fyIm;h1*fHoXiCK{}#*l6M}T?K<0MB~{tW$T(3`!+X&6>$J0dcQWK@3mGVRTW#MB?v-##9rx^~eRK^=Y_Ri$Q|m^4+$Pil6YKAOG?E6U{GMQKe4 z?2Ta(dM+`G=qHN%{aloXa|?fYjZR{=9vWiVcc+-VBw#+571we@#Fg_U@aPc?ax&sQ z9+%{~1kxQF0K>UaBxsc;(Vq4)Hp8Xh8{E*QC*wa_9pXU0E@Q(saHcEF5W939KIX)ls)>z`ozrx7shf?2# zsf;7I0{iNFu@A?cqZ%w*(W4s-v|Y!2;nf&?k_zuT5JoPVo4OE|n81E?YQA)ew+NyD z6<)kPz zz8li?DXBvX4?m$g6LdBlS4MU-NzBu;K#z>~6Yr>?|0qtJ%VUWctS@FTQ> zcxAz<%=h5s<>XzC6)F^`$tG;(u$XvD;(rkBG) z1;uAuX*vc$emhsrjy56qwu_zE8Mv2Vcl<>3!@CGGV`jKufWKLagR~@uH0MFDW4C}; zf`$Dy;RxzUiL7(ISV(b$l+Pkc^{Cw zG9nm3*}t&68}iQtI zEO$q+64)YTy4)ij_hv{MH+*>UY)?OLKxD|7J0$DE+{)Dg$v>uapJlCRmr>lEb;}8Nu= zhgc-UsYE^!fCfEx+R`2N`^}3ox>aT@(nozVDPI^peZ6=zBNn|wqkBd3Zn5S}Mq1E6 zt&q%NsfL*hI)&kAX8KEyq%6K*^5*rkJ% z9ijto>O>XN%pL`1Q7f3y@=U({xQ$$>ten4Lj5K5yyf(8w+9`RBtrV^>vEvc{_x^`( z5m(0F!-{ft>IQoX*!7OBxS=5^)a9(p_fvdBM{G8Xh+G5~y*PFReL%*>-^4KGcV}L` zxq;2iis_y7M4Yw3QGIG-{JFR0&Nca00H44RePQvOeI`RG{z;;}h|^yL&aP15t5O)2 zv%*f7d}M+Wr|R{%(NO3bQ4dD(R%*=894K9>BmhRzkZzAKv)>96dHU8o4(W6Ipn}Lj=bdM&=?k=P~;2w*L_>HQ85ItTg#E_VYfD{Xs6+}Rt!m34NGes_F;<1bDpexqoXai(5 za4i!s0cU|BH=5{f6mt(qgGvFV_r}#fx_(~p&hmp;fI=mTctQExo4#k)+{Fc`iLK^w z{Je~R=E)V4?c4|BCp)4`yhVN!FfFG4bq|-J0wJRMPfKQ?i317?jHw^J2VMwW>W2`u z8Y1Cj;^rWJYjOo8lE3d67~)dZ6p4Ir#4&%opu9OSlJQ|8Ic*-)Hbp66C&M@T45o zG8BMF#=Qj;yYH3bo+;`WP}`p;ywwEZB7KzLNmqdM=H1I#nAzzmvncluvWmn$+c}js z1BD_Cuq!FRTz-Owe;&qcMLj+qC5P}TRb2E>0v@1=CBTT4VE+u-M1lBBv<-GnJ+rqJ zv$RWMy{_e00U^pv0+5=VJxl$=e1ui{zNI+!(|uFnWB|zu`UEF)8v*M5=`bE%SUEYU zT$>3F9cef6%lO3qEtb2zv(xPR9V?15WxN{(X1e%G^ukf42<$Xz%xQe1+YTUHZu`z*ibs6)A!4p`4@%oFrQ6?M!!pe_UA}cH&CrS&e|oTi#o8UW z+blm*NIFQ3J2}$IoD9Q9(JR|#fc0?;w)Q(TEoO;ALNE1G5;fn&*V3E0k#_(A*%?{C zUfwApdvgb`+vlhRfI#_7>v;tzV$ys6ot;vggEoYSQ|IhHM@JsFQpZ`foD~&uKEC+g z00z{ox6kU}U+tvh1piu3DUS60Um)&+i^#4e;0sF$au57G>&)N3+WnHM`+GfUcAdXK zu;TlNk{-T)5aL@2VVJ>LQ2%E4zZNvTA@*r9pcM88qt{zN>D5`-zoe@vO~9aj@ohPFuo)^2bHpfWC4I;c=@SnN*5AOffuE)de@AW*@ zKK$kUKe+mb`+v1#?XKkdYrP&SyT3sE2Uq`a|F3rc9?U-i`TuRu2wfe-IBq3FmK zj?;M|NY%Z{$9SIF>waIifS+Z@?9)E(w|2dUg7H1X(q?DeMEu`3<_72&_Mvj^7=QD7 z`4|@q?mdTwskZbBsI#J|&`?2ij$;;v<#Y^7QI4jO!bMu>GfA&-qME%1<}7E2_q(9N zkBhVvVo-T*XNqtn;{sJeR5Jc5Bzttkc@5z6`fr!m@Q}=-TBT-twA#7qamrCxCr3?; zQx(KO0bh6@n~EBbhAI|Na#kbn?fxEw7vYZTTg%%In$~fdzh$d=>isMJvG;F19`Zje z9IDT3{_TI1ikI&4IRhY2)cC$TZjH0=a0(uM29LiGGq`cdNyxI8`>fLLf=L6WEkS3S zq7p=9z{&M5Ok~O%f&W-Exs?KEZHE=awixXGk418+G4*)~aTB+0{D(zVErk8N9YtjA zxBg>cU1tK$J1?1QJ8kvJMrp7=hScdm35X6uX9@}|*7W)Nw^oHm0{ompLT~2<1 z5r(G<=v7?<>X%iAwaq7D&;rcSiI|^_1hg82*Tg+A>%ht6oz_B(c+kV7*v*0i9mMip?r3Q;V%iOiPp$zq=>+7UE@g^QVIpLprWFzOTI`P(2~cw3lmck4V@p?%Blor!{ABE7l`WDD>HjDyzIpuR-8w1vUSAd!; zC!4WVxLx9N-KlsEJN}<~dtZV&y1}jn@$L&)*!bC7=1M z!r`Ag5JAa6=OJZjh(zmwdmmJBIJ0O}IMF=4zpxrb9A<%CuPJyBxSiNo8W&3LcB`5U z57%clj#e$96)6jw$`0Eer(zt&HwucK(@H~FKmYN&yK9r4EW4&I8=U7n-iCiyM+;n+ z2B_8=VeA=Gm&S)bYk^r;rT4Mtu0vW)7YdqjEq!e%UqUu6SI+a4HsY$bHw=-ogQ0ga z3qk29$HOA&v_gF141?=30magj1Gn7^cBTHZXH?Q#crimEV8dXEX&}BW_V?p~J9Od% zTdDCW6!grc3RC_en_zRt@ipD!IiAudjA_h-9W9wd;z+ixfb#puxxmsQg>mSQTg>-& z@c{&28`GvmkZhWe6K@NoQRZ6np`3X!%Wp}`Y9?9TT1OW|cOeM8t2GEv79N-=UTei! z-FFRubM@t-a~tD|6r?k1BC+A~?zPC2L~iq;!+j)BaPZ6g1pRAnJQNDW7V$}1 z{P=yI`0IkFGxZzf!2LbwPuoXe*{dncrEM8Q-Ujb=`k>QL9RE()*47%q=8wR&jWhN* z-#Q%6ukaRn)EyE2%rC|5p3$CH#j>6R9Ui(zO<)sL-l}-UWFzB3GZL7hAtn&^gPu#Q z7E^Tk2;s7sb&;U_KH~6Ph=L$Rmf99xQUZ1h?0E)%xgz#=-3urQ*c9%c$q;k?foe+E z6$Ss3NP)~RtJ6c@RSLwT0?MPC z-N%)84^?av`{td3Ka>}u0qjqRzd4!h(1fj_*Gcg)N}ld|g7b9Uagm(Q_>JH2`;RS` zSK!vAij5Y+O;*E2_p!jK*>rM1?E#VB%cgqOqtkt4->_laoUOHBlrC$xZD_26ZAM*> zDLUSvsJz&x_T(3&w%np185K18PS^)=v!TI4R*md%2?RQQ@)&pb{z#Au#XV zkCb=G5#`nk;g!t}T94a42un+b1`B=R)lJiOg5)`#>Pl|69?xqS z{6ycPv=&R+Zz_uy={jbc`fl3h25wnatodx{=ig~yE<{OgXgDWx(2ywpB>$GIV46^$$biv8YeB?Nmj) z_@E8EWQ_BzOIN0^gI=r`7Iyf)oGklxOhcLixd8yR_*N}aY_=8xpML|d-6EVe zMCEz`U?64(@ip+8FEYm%VUzKeA3F=58iI>bZiD>rZC7Kx5P$QK_u3ZC6F4u3I=`T( z07X8jn~%Xd34z!y;4SgrJYWtCkmiW1c?Ct>ochj?bsbQC0p8W=X5k}v;|WR}9l?kn za;`&7_>(lDVkhv^d-n}lXL%>g&sPHDZ!U)Lfii`?ipc0c;CC^+*WqN~JO1i{;9uT~ zwZVpn-NFL*D`A{}yzMmN;d?740{yl^MP$7Ac?d`Raf276x^H;4tGMgVoCRnO2_QKb zGyH?lFH7~Wr~ycsJO~Vud;=5!Z^H3meM>r95HWzyNLi5LMfQx2Gh8`%_z6JvVG$~* z?OQ&#;{&{Zd~{Z}5Sik}(6iXi(42*P4)C0WUuQ%fitk(a{>OZks;~!-_ z@8EsoX)Y56rK7E&5CEcb0}t5{4+DKpHL*#Dks-uc8=P7EC#=TpW15%GE=%I)MH?XT z_agBb!s0}efj6ONF?;}~LfiIG-X>lb&Q4xhXQbdCvEYq+c&HOfXDd*gkz$F5|NqJ( z9-mim(NYP1{~S8Qg3nE36W$jd?Li&u-M`>XhYFpujysyiS4tQjiB1xp*FWhop#Hsg zE}t{rjQ{YqL*acs_iuOq1=@iXxQMD31+_!5 zQ(IV24KSRYEzcct>)>UAA1lIK!t?Wo;ZeLi|9Hfogn|P9XZA*Gxj<|r|5?g-nH6z6 z+WnPkYdmXsA#(ktI;O=xVewDvZk)))#~c^Vc@)OXK!g5gzc^808BqLTHhiV-@sqJV zZ@%@LG(#W%;;`*cj?6cWyy|6{7iNMW;~sJou@2*z#W>ON1d7e>bM>A{!5A+Ed^FWR zi!dPyy9jZx^)Qoyz^N!0VV1eyZF>~e>N@_8CL)yM)m6U5Q|MZlWLOT7O;TOwx6X`e z2fqlqSE5BfKTjiFQqqtI$Fe^)tm7;YZ_1DY>Di4b8Aad@Aoy9K9&$4^rE z7aYaWdSb?~qG^XuEk|sUr$)XCRRAf-{dFNJ==iNap!eQ~HgV#$IH;btXz|WNYLd8* zhEiu1$Oxe1(Sdi7fgZ#YyVA4_{8sCR%>z{`8gb;Q2aHN?;Z8 z`d^1XBWC6o27Yi3=Sp;pW}m-c*yx)KQ%o#d9Qg#!-%~ zI>i-lA0h*B6e`wv=(X?T*3NfM=^!faepxew%}a%V-+>`_P(geupH&d{p`3fR3BIAz z*GNF95?`pMFVV#k7jF2B0Oh!QRN{XLwaJeep!)=QCy_e=?Y@domc0m?bDn(yBXf@J zi>Y>S$S&CKZJPMnN3w*Y;x*iO4tz|tkE~yaQb825tGShNuWS>%WIizJssu`=!o7k9 zM&26TRM50fHG`#i%C$aJ6}Y85<8Sh^rMIep#oFlwS4zH}v}=>yTeTFMcTYR&tmlP; z<=(PO7bR{ugRVR+!CryQn5Nl*&}{cYu3vraA?6{AA{P6Cd=6N+e17jgd%VZ6cM*?5 zJ%t+;BydJc?kl{=#NyC9mfG{x)+V319&4x~0_R);N(zA6F0`jL!8xl^AJ$pKGp`sF zS%L}+iB|@%*R9hwLk zEx9BzvLv8Z`8yxbe=^ZT9twnpTB9Di?()me z6WCz6N}S8OP8(MI)Je*rjgq&YF%`SeQn6gM@QJSgv;~Ewl0Ij@{0R~c1(LgwC~uIR z*1)>>zE6x5_YV=$Y;F~>Bj}W;_-kL)_kB&6A8WC{7|O`ce(vr)EZk+xrn_meD{s7@ zIKu352eDgw`5;>0Mc_yJiQm-vPLk0~T$#mW@3-P+RWs_t z5T`vV{30w&ovG{rs64(9PZaC@k4wt(mET9)%x*<;pwvvL%@Ld-4Ij{b{O_lA zs)k!1^?r*g9|&$=;3~nWhU__;U?V3t1d5k>vjxY>swhlm`j@!$um{iZKPf2+lO;hv z=+?*&6S#J!fv}VSvEmdud<58@o2B15n@)FpTX;(Pzr|CHa|=fStL-WX_g-{M$bR#4 zw3TgferI6;zlXt6kqRQu43U#^VdiU{OY4X%|3^}upupB=?JMCkX~&~8Lz6D!^SXJh z*IOB(pBW$9+`J6xKfF{CTPdtk-vd#&aSIRsr;s=dr&RiOQtL+w?)fk(wPyMzTd_aS z85mEPw31n|t^Ff%6%*tNF{CLqb2~N%(bU8o;IMz(-naRxC}(YZiMqFH**DDYcA6nu zdvD(lGgD)`oJ*2pzPpju;^t3GR1$VYccF*hNHLENn;w{8Tih49i41LVLWj3XqDwGd zdib=dUL){N?Tmc?x%Zyj_g|;9FdRdP^VU5RFT<2~BP5jrvkxNxwt7Bw1}3I}Xuqq5 z5z-8#`boFoZur@{pcitJsgjvpB&glXad+yrqi1q_g&rDwx~s- zz<2F--=#Qp%U$^vyqXoiL%81;Ygr6r#0QcYi}BYZ>n}5z<7=z^HF{N%zC~pr)LR44 z6XUf8&(X=LfO}`tU@N;_h@(TwQt2kU=^^ac8j7Q^^i;ecAHj- z?{i2pZ<|73HG474TQXqm?rZ2b2~8=v@3HKoA3nFhudE6B_yw)m?NY>i@X?;ZEH(MZ zxg|Ns{|XRm>X0N2FEulz#TQRQt zqXO53Oy39~%5`9X?yU_;_ZOV6>0p&_TS>}y#bAOVf90~1<}{%jo3P9!nY5L%_U&rB z%)IZ1A-#2iFx;rqI{7b4C4`{1rs&haaJE~q*xZ%4gYh!S5Bx!@rr*a%UThQv;tzD@ z2=Od#xl3Lf8t-o??S8{_TOlMX0L7$Y>4UYs`+JFcV%e8>hESQBV(M;RrKKBGc|KRp zs8x57Ce?N^UFwu>UJ*Wh%HY%6;`Mns%58k~+8$$n8spl}_LAJg5hk0)F!Z`$Bn`#T zP*a?MZV7{4b;FdT*^uDvCE<5wDkj)&bA*~Fs8k0GB~@O9`v?osoJ;5}n)5;C)`net ziSr}YnaWE={ZHG|>eya}H}jFF-5z^_Y&^BMV~jvYcR6Xfpq8*1NjS8BNRI5 zLNU&(5!e&krhuhT6q8*(YJBN+#olLYHAenR2NmoqCPFH*5e|^uj~&(?NkrMG(Apes zRgN~}{IV%7H=pb!-VTnjAZ%BbvEh-oAOBL|`yoQnHG7OQuFC~t*&Wx461eDBaOK)S zT8*uWbZM48&tiKatm|WfN)p{f7LAF^sCt_SnXYT+rCd%7guNJt-{jc!s6{1|iO_Ca z|AV#nj%up=+C>pjL=;r8P*srLLTyA6q4jr7s(U-{UtZEn4I8C` znO`}oE{8WI6)5Y!)!Y_B+KE*B>?3-waaW{^?WKEvRc-djSG~xkw&WVP;3Lq&0|b6e z1(ugn(8i%??b`F@U27~Kv0vKSK}zzkuBy)Nky?-RrOQ>0X`>4ryRKu4>ALKX9LcEp zgYkP14XWphni}uNs)(^8?D%4wKsT8nC8d`scw%yrJBEd%UV7QA=7;c-pVFlVgeJQ( zlf&}Udq{w4OUmJPvoHf(<#Qt9d#VQwx__g>*%E~NAU}BpMAeTIWV^-4=JCT|AUJy+ zCEo47tPrAmf868!@k;y*-AgN3ag(M66waTLt}_Jzz@q;{^s}ZWxuDSohn1YL!TN&9uT_rp~lVPMGq) zRYFaP*UP$AI%|r(I5VIV0)6}8^!z6f4>Uv|@-HqlxhkqMi^$1(^I`&>KIY*s+`IJ1 zLP8s5Mljsv`7NcBEe{Rjf-qh$vUE39&!4UiU%VH}dh!pM-!QB$9H`d9%ADE?#+~sd zGGiW+#8+ajUuFY8`(aR+b{9XVkSU~-csSjqRcO2U+82bVZx>$p$mptnFeeYZregHl)&AHWMzRy{ildjZ?lfG;rbG^fyA?SMWn&hnhcLa4o*Omp+ zJInb-DnXA47)t<_Jl7nUwilvutKzMCpcNG&@lUS4t${!Nql!fqc=H#Kp})?&*m&1I zCh759>`2AD)tmyQKd%bse*E$6n`Y5xlNr=KarLE6hmb{2eAPqX_>r;7nq&V2PlI#pxd2EQK4 zyv%-mse$Zgy2H;#ZMzfLx&2z0|6ZU{}gVNpdu;umiBV~y0CKEdwb z>a8euzE`VU*24URP>K9ba|MOP+o_Rp_&^{|Aj29_BI!gonfPSlKS?6F9u5jCr6W!S z#wVz;H$}f7+tLy3)S*{wImUS7zb33n%0;4p&c62PX2az`p|Dy_aI^i!)+JS6V1GRs zbbH<$wA-lfr~wgpjH>w|G#q04Tw=EC<#ub1p@)092a*D!ok0JLv6;8b$tt9@gRp1v zZF^kh6YTD@BqVe@r()+p$jF#t$L<6~;EQ9ucI{vF*>Gx)b+x&sq%!kZV?*!Yo}j&@ z(PY<|i=X!R=UJaoqfoi2JRzhc_~OKuG*D3a&88$l; z9X^T=ulRDzqMxSvjCOBPV|(JQ4G9*rTh1 zw%|We%Rgp`e$APLC(uKU(4$wIZwq}V{_CgLTv}l0wjY`Up3V46Z&6z^0%`4z9@%oX zo_h2cMS4VqwEmHw3Lm_FaT1QR5AbI1+bJWPkczBD0?4H*L zxTzMVO0bwkAJ>f}6rh1aVQxr!n5M7La_DY9xIR2R$dE_)lg5Jfrp1LXJ|@-#n{86c z8y^SN_Cej8JdD&OH-q|r+#e!;3}QYQ=+FeqHy@tuTAS-$9Ee#NyytT` zNh5MOiFkv$!22F9dJHQV=PV8bK1814> zccZBeU(RYWaFTb5&9u#OiK59~<^=pA+U{*J)%DmWDgUM8bWG-=+R1t!;>HcK7-`=| z5enRDgOVhYVFr99XBN~2$=TSIP#8iPl;iV>2s#kpt#j}2jHrR91y??M-qIRA425_6?M+TXI&SjC*2;%MGT1RzJOBwFoAJ^GtXnJ2iLCF z#lSxhg{eulRBtjaM~AT=V4G*VDNB1 z4p`zv6%+38MrK?S$yA6g*t>%c4ZK#82hS|Z3DBf~tHQqDU;$O?gZDl^25EeaFwKO= zRFUQ=dU5!2ssci zpVK*_w5=TRRRxp5W)77xIrpH@ug`xUK#(=#x8(J9>hqS)K$OJ4P zz2e#5&gaiXyaNBOO81!4h=J!H4=6zg^8O4rAX#5;zWCZ&!1zW5s%^h=a9+Z9IG-{h z*7Yh+A{9p3W^aj7hHke&2VnMTl;xTF80tK0bwD&?=%6q~YhIgQG#gv!dv5*sd9L-F zKzQG!BPLrT^VfcJKX6cOH3fHfm$RfhxObt?yYJ<1_39bkd(a)t(0f-d$A|~8av}-F zI1JbLMWqGEWnKSW3h?&2I!N{htZT5A^6+SYopiyR^Q}7ytA@$=kO0qQwD{Q2TOEqq z8HMmA!e{V!gi&ZQRSVd!u|%{rUwJug5ye{cbc;I3bM=k37} zo!%9iM>q*tSglPau-8KxC-I6nqu@Q5v^NhPsthMpy^&N z8)ya(q$2|V>(ENFFBB$b90gNbe8Ck8OP9aMMH>is5ZC$!e2!8qA`^%7@Nh`hSOd>D zvTETnci!xf8!)}ewcmd3-=D7qrmXf7#bM#vjpt`fQi_L(bd8jC$*3;$joZ2Y!#wO! z&KKO{B*1qH#4jcz3!VvjEJh=n1n*eP`Im7Rl|hPiD>b!U0qkG;;EkEJ{8UeHaT@kIE={q`ls?Cp>4BrAG<1bk*nm zk@zQ%unVP9nj2}DSIv$dtEc|8*2Tq(!@fOHMSF)h-U3ENW$1tw?E8V#j`j7{LZcIM z9L|pz$DN@9(kr#8cNX9KUavB)1lx+wMLfIVPEqTlSB0SF`G`t^MZ3(=3;VB2B3aoT z1*I8fCLE{vgRyc>pAa`TO7g7T@s9x6+Z0fN_xfO-1d;~WBX6wT)r-0Up#t&>qFpyr zKW7yI#w>O=G~)Azj=VUv;6+gkiov@|bj&Lo9wgE`Ln=QmJPmFeGasoqD){2q_Z+5B zee<8<#77fzsSY*@7ym@L6nsX4KK!MRX%NIM*5ATL(fV<}L$WSKc+_{M%cE2TXUh_niP4~e*aT#jR@Fu9s;&w z<`2K1llsTF)Oreu7bX7bTG{Xp3y$FIjI#2RPhZ+@zExXF`1673BjOo2GfHH%p`DSg z=SKb}ADF`{N13Qaz7yuL6S;W*if!)WWbVfQLj6VQC&tTr2WbFh$$$c%=(?5aYf^Mt zV4r}DQ+-J}U((m{^f8j%Dn;`Ld=V`xt+@8hXk$Y8DRDqM$A-P|jhFWj_ zu|)U?mY%_1z2I#*(0DE)arkXdDNcn&wD9KN;5kXt$9U6`FeLb8UlfsGUGG?QDMQx0 zC?CFx4_8*6Q*B)Q<3{0J_ME|IKG>5q*c-MBLrq}DsL;jxb z{wJWDQF{O7l$$8g%%ju7YOeQk#%nYp;aq|4t?Bob* zl5rN(i7O<_M?7lX&%LpZP*ZkSThFPb1)KOj)+k_M%I(Yx@LA&3pUu5*)ab6{ej=zB zZyUK$QH~q7rtUjxDUTcNs;gb4rXBg$cPFYZfNkeL5J@`1OiQi@>K9ZnjwF1`;qObo zV{8a0h<7(TONchElxil|!TV@-Z-e2U;&^BHStNZ(wd2g&>(iQ71Q;lj zPBrhFg%)%VM{_a1lgyw~Ncq~+49eHx)-6pBdZVURH@mF(btzY$jSj^)-yXg{{Fi-z z(ODRUVhCX#0<-AVMrQmIs+_uRN3D+}X(!WTk%$5TtYHu3%a`gq%Wh!D#crYFTS8#i z9u0Q_cF*r|_CUw0ZnnW)v(CrSN&=wn^FXdwWrsxQ5BFqirYpC{+LP~M2oFTghEut( zygJSK!6PqY^Rgjrx#IIkA9T6a>}gOoY}D>sO|AIIQ|XZ9H{Ev>%(whLQgl@P<$-k> zP+T zs$l%VI+=x=kHV{mm+M4d*s(pahegOnF7sn`gnM+HC|ZcBZ)MY&3QmNYT4h)bjE|g} z=ol&cu^-e=?|T1>rXGsy3Nwt=XeLWtYYM7&F|DdL2Q~3rQ-BPlr7y;Qh$t&$EIUMp z$-j0OpL-K#My;uB>1Y}`#66u`8|E-C6rf0vMMZzp_Gls?zD*~uA?9^albxQ;z{0_a zz^RY1-iRuSWySN`+VS}l!L)d;`YyG=x0wU-caWo z+x%oA9S<7jR-fc5zQXjQ@?TWX`=j9-55|D$dbwVLL43Zp@3o6`cUA(KJ$PLE=m8Gr zp6)_cLkO>acVc9o9U!w}OGsKKwmG6sDS01q=2d!Vs*fgf%RdiXmoUJj*>`*^Ijlza z?w;-TvbRFO)up16Pl6QaE7E@Le!(vq*7ZW`_I*B`egvDgdXJCTG=GO`XoNAHo*!$c z@=169O^PSzMNW{JD$xkQS76DX*T@9TbqKf&8PNh zCYJ}q)sn_EIuB#%VQurKSC_=frB17Bd-HnH^{*TvS#Nx_b5qv-RaEsMXUEs{##!$w z>+AXt=BEvK_I?H$I^4gBs3r&d9xJ~l`D?4J`C&cPKNHj%in}fTSCqj&QV-cK&C1w1 zeZCs8PNQRUuExLO)9T!rrOthGP2yo%A#0_zIhSq*{=pQ7g&PxzWM z9vyxcoEj@$rB;agvgv02c}~E6`_MW5JJWpboMinNl@gto$sXx8?MK@lnJ96gq=@VH zhNa)^bbwaA+^RyU=&pwc8?MCweN&WS(_lBPU;y<3n>NSy_9Kcg3(keoDFdgf z_lJ4tb(=?9=|HO6)SQ=M(#cCE6VlVmzGfcGV0qI6(v(b`x#WB9VyhR`+{j&joCYV@ zci@)Y3{DV^bUi9?2}IwH>8-_iHxY*ML%l0?>^y{wd-r=z`)N{DU)VH878%p!!Y_V_ zaVuS&pN?Luj-Gne7YOgsFu;r#lPjtmn;P2UO3$}_iHNqFn8eoA)i^#_?OCVeF6w>{)*)p|k(S9=;~<@JnrY#!IocvUkKJn;i(&-RX)qzVdN?d*?Cib0Nv8&yF+02gb~%AC8R*HxJLlU&P9Bz<`|bM>D!!9*vxs&= zdYzEBsQBLc>9?JDnubhma&-8PA|6fynV+a8q=)d$0cBJQjIiQycJOj{*8?HJEX^1N+CN zaq$K_6HuaqhM)rKrb#aM*HQ0bU8)v*^oNql17VkSbnaJXjQSprooZtaIF_BI!^}ui z@skedIr;u`#-w|wXJw<`F^&RfUd!y>lJ|222Qw-hZJlv~Q37A&mNPR#BPRjY9DjmDkE;K^;xS@v)H2W5s1SLemn(9yT{bOX#Vsss@lt!S zF-A1+ z`sK%bROx4_ZL_dUzP6(c^q^bk0Mm^sOen;=1-QD3z&haa1`Ru=QE7>T}!ey zSck=`TAgR&6HXJ2+S1`n7J@@C-4^G?@2$z?l@RcOa<3HRfshib6CP=gP|euw;5{4# z<-%JVn8yN7>Cm0}7>w7S{G&dPqLo4eh62=((HrmPE|TtVDVpG+B@Uv4)=+Ism~K_B z!J%@k(u0R^70^T95T58T)`eu+0#UU-!4Y|6Rwt;Mi=C3FmyAzVH0fP;VY0i?`c6QFKmp3AjcImHnU zd}bCXLXk_DC6fz(A}qjmNatHc5)^dawkSAMKXy;hp3%-;s4(91FvU&UIBE0MU=wFghM6q(VB`jS_O0!7@=HNp0$T)2}y-Vq2Z0)o-F zCd_@S0C6v+CW5v&s=@*vXb$1%L+4I!0gM%y)} z$6i2>(#C*Q*bK=I`^a~?2t$`LVL^|sfCsz7_h{@L&~HFEPp z1gcK6ar5h3DSW_Q9a7zt2QRyDBFXm=2qtMx<`>s^3)Ia@bFY$R!8I=lJ2OfoeR zR}Oc{cLH0dSF|P^bS9DaO0Lr5H8J5HpeCcoduYTI?@0jqaFZfs<-wET$M*R!gW`N7 ztIQ)MZOr^K8%qy&@t!v3Tg01 zC3rZ-?C<8b@92Z)5HR$^T=*pVmbi|#>-{bhi1gAT1rD0B{x!4% z9RJq$i%^r?aCKZNM%hMfZKRG>ovMYr?2HQvB>*d#2F_~BM~!!GmPjMuZ7T~uN36-3 z3O&&XlN@8`-3)fzuP=iz< z=DU#=dHjQRtNC@^hK9be z;QVT4Za4^X>SCOAKBb{KYRCl+C-&-~+WXDJmoj`GIjx`Ky*3DHfz`0lqQ4RU2n+=D zrd!HEL?t>pV0tnby1l#ZD04tcA@#lkpTY{3r$8byJKoV*ujzr)9PT+Xcqr$AW}m&Z zY*eX5g2N<3318}HAee<3aAwS+#0RL`B!i6#s7d$2!Lz*7z=9T38+*C;?5^d}vChDp z&vl~4M>-h3QfXWTo?BRT<~gpS{SuCzf~#n`_}6y5HwL_grgjbzKCMJR%egXg?ko>7 z-~^AKa_V55jkoJ`27DwzPhSkGOY#^O+`7owTXltSGBiyEEONwplTmOLBY;5@+<3p< zaeDF@M*FQQ6^eO)lLK9Pwa1YPrNX&ugI0GEX>j2g$66_izEExVd&-~&Jzxcf6T>36_%{Wxk3UnrDr?joUE=DTCgRzXx*U5{eZKyg>|C28GQlf&chiLK3`n;sBW> zL5KOTfwok*vBw{(N9L_8bLX|dTFx=BqZ0Tp8PdOkfr~TL`!R4%f!vQs!~!=k{+0lQ zL}Vs|4uyR{z*z@zevw!~fn$1{#7AIsPR~jw4vP3ISKtv&Fa{>22z+wF3px-8jLOt- zU;+3RaV{KKN=JLUUp=zOMRYM~CtXH!Eb&0V=ND={x}^lt!Vu|fzd|3o>xA{Y&wML$UPW8YL@b5|h zjCU7cgu*@t%HqP~Lf~Ull|iu2K4H)SUqCyOYghiFK^^dX4=&Is-OKgXKN9x22N;=w zPv!*t$6&>MnZR(YpveE;Uco}8k*q`q7yZy=Uer(*gUVR} zYu%Mge>@A3(W!&fdPox$3){&bGx?4MfVpaTsB;HnFC!j z{!0rSbnD_~0Gk16oMj<_4Ek>fT*T#$I$(`P=KmURg$1E7gCc-%0ha)8vtNXg41mTz z|0E3BU+8}MLU%cw<<;7CV2u7xSW*8IR&me)ufJAZYNq-pk}j}8;Qs&BawRzOua=1U zU1}l+iT81U$w24<_S;onK#ZXW)V%Z`G*7hwG_&DEQ{?|El>blflokKm^#9#EfXn`4 zHiinxD$sX{*bIuPdL3Rxt75UFU*6wX&HHlIz&Y%9*r3W$N`Lmlm?A4~qtJjJJh_xIs>sg8qv=rDV>&7~z}sFC5cr?GKe6?Fp( z2lBoRm)_t#zuT9ACHJn**i~2o?@nT-R`C+iP~!P`o>4;;@>cBH3(%z(Z{AK=zKj!B z91!O6isp7JGj>Xs)AKEG)@uc6IFHm47J0Os^ZagJPEF4)dEuS#O#vs|-Jr{&JY(&w z<3pbq$Hwoq%}uCz+c38RT7TutB6|0Qfd-eb??q%j8dUSWPU=#r)3c|m;_Tvh1g#WRQ%r}DKbbgp?0rIY?a5cRRjth=cb-+Ww6Oe# zFO(J*|7Q{Z->w{lh(D5$<5A_0I(Pe${eg-mW7Bp_SdMLda_^Su`FzS7GNLyRTJj&v z*#9~8ReAg~G%h;vQX}bnAx{FWswfLx(Ez(-R7P-PN6pQC(wp7>?QY-Q{fdvc!Gr;C z>91iy9%$!h@y=U!@8AEEwmRCAk=d!==&R!IOi4ufLuYMsCZPVl_)_gHsmtNNUFp1? z&l*dyc<}N&PfnUbu_(NzHn(}3{ilQ&cp06kmlRtVafXR>_9IZ5VN>T(j+*tuIp6*z zQ2~+gfZcfEjXk0bxx8fcEb^J3GV}C}q+Y4b>@e+6{jNMLP3o}3J@LfbD&f5mED0kA zDznvT8p7US{}}jQ`Ov0`=kKlbkDpu+!zW7|&Qq(Grb1N&ex#SHi?JE&Io5rExJ6ub z|DhE76sj~{r=I7{P2ST~kxQWbEzZS1_rmmroig}%WZ9Q5<#Hw+Jtd)YVm%!)iND;; zoM1hdV(xJ}&#m84SV92xa(=s^h`RFG;Ag+ERYBfu_PXiWY}~B`t3QL$Q=BT5#~iNY~=+Zh{ZCf|N%* zsTGFGXF7aL*wHoDR71U9?XlaV6=zRZnDSbv6uljjOA6_`OID~o@nA;d#D-G8_j7cq zE}6Ac9CHnlL=y3Sb7kVKcRW55N4*lPQLOrI7st$@s2URg#ZU-N7>Qki=&QmI+@QH^ z__IXLp;X604w4ufyQY`cTH&7er!JV1t6M668Z1fDaP=M zJm#pFibOcC!|$LiO5Z)ij~u?No8MRO<5VDSs^1a^Nr_l!#52)^ z>`bQET8O}?5=Jl@9_XEBx*5MT!AO+exp9&btMcB?E(KBE;xSDU3n>p*w{ggb@QN4` zZM*J%6Gh_zth#KHDg6N+e2yq#0?yDWaYH^W+Fcj6RAJRx3+~z5-BHo0oE@P<9r`kN zK?V}oZ`Hu(oqlG!o!%yVE3F zul;ei!sEhW3ZO7Yo%!WN!W%ueuZVh2f|1HIjesb|2W9#%+acQ_`So$tq(P)Am~w4T z*E=fZ-0YtAVxX(}Zofyay>+~bl6hE!dcp(BTS}0fAK6(mOaT(Sc9K>Rv_1o5f-+?L zJ=PD6b$4B#tE@2(sezZ&^5+F5Jb_!LV~wfNQIo#ov$0>iqY#rCu*PUOQ4qHN$#QP4 zbvU{*C6BpG%{$b-+n9u_FR8;^s6H9@8GEB8e~d$_QGNBY15aM;kY4+?_W~P9xsG(9 zOvYOEhx-dkU?LgV*~rUrXs>)J?+gpkJ(ccT$se0lYr^cStn-kB{^ZlvY>?yhCwX95 z4dTR8D@0$Il5Jq9LLiD0=kibI+t1^e*~!k+s=clSCOYUyuoeN%IJR-mvdP`eCFnM1 zS+Lx9)#O!;M{Zk2az$0o-$m#JPu$8;j#`f;Km0s@JGmDzHXh=3s98B$INX_xNkL=l z695qBZ++i*rlt$ylpN2|4Qw=e=oF99V-4G3=eg?!it6X3(eG_LpuJr*a5ME4RckC~ zHotQRA1~49<8T$?RES^Nb;yOsM=EIPhN87(=YEHh6JvtPc5;YJi-mM~rRIq3Th z!3kU9RmWtHg{}`D5T^kZ`tVeTzcf4fkx*(`%c5Z3&KY$=IkJtxxpk8pWVL0~z&CQW zU@%-^s8k+?AVh0Y2ql{4PS2R=%)Xv?WEtSb%XU>jpxcZWNhw=Hv5b4-8XVB+$uF8$ zQ5hk`Ys2`qi{Uk#TZjj>IGAx_qnA#c5fIOAY9elLtr3?&c>@w_*(71gC_Ap#@cxPP zefEZ~-@PWVi3`Xt=OLPUEfi^Q->tT>X6a0i*D+dS*_P{h4udt#&4(8kDw?MI%WXC+ z;68^8#%4-RzVZ&mZUoJ}(1X!RdhXCFgMrpWPjSKlZc-9|q{2;+lC17yZwtPq?%wOK zkxB}KMN*o_6$yWSIyd%87M1)X-PyUZb=Uv6mUjHbt-99oV^yx8#oMjYZ@x=zy%Q(W zR|3~wITgIO_=Uh1F+6N7ni(D2=I)Yt3Emc#N*K7oO6g%)c*cYF#wq$y_k zLBKcZJQhmp8=E}eTwlBjiF0~Rv@w*2cqsNzgL0?wS}1HYEu+JZ3skL!X=pv@nuud# zQ5GMyRHVp(=M}a(FycnPt}*z)HVth?3{nds^_4MNz}aYT01ivXN@Q3w0PijcemdGc zG~=oYh~V!X3r6b>y`VO_|BO}(?970R=eq9dZzvl2F^eKc`8)6=4N66H7LYo=fLj2%_gC0ztH8vVPs0D7xek8pGQMrptiAuArjQJ0DWf`ub%QiDU zx@3H@z8e~w1dr8Fke5Xr(#iwVyrwWM9&NXU+qtoh2%F)^9QghwFbv9|Xky?))sgrg zg3GEJ1KfpJTa3CQb5;`Vm0atlla8a58$R;Sflb3T3IPPAMrF(=ED2HQwa>5iuZ5ED z7`6!(yfJ?ou*y!4c75A@=&nls95)o{P;BNeefhVpl--Yx5+9PssAPg~f0ka;g-MFa z7$;LBHg1q(brjItDwiPhGgSJIla5iWz&yoA8oV>)iB!OEW)d0{O?EPtUXkx_X{EfA zl8}JuVFpo7cbm=d3qv}5zEC%)wC8SS$;NzFdUqqUVt@$mQn?Fvx|SjB<_cjAS& zE~-1!p=VbyRg2~wD3H`${Fok41-^re1N44EO2YI{UD*U8@XE3FVsX0a0J%d$c4oUq zO|gCkZ}&%gTXfR{pDsFYAOxsc3`(1h6YH*Zu+RDa%7ORugZI6!HH7MtbUXFw>7@r@ z-BmM|T1CbtFh6OEc1DG)cq1r86;7v`A~@?+NOz9Jc}hAz)SR^(SJ`Ws$i_7_J?zSJ zx)?cLLIhS_EenP>lu=wiO_XC*sA_+$MDQr4_=!|!lsBRcbFEc&o9lZ$CSsGYFYYgN ze~yY+MXQf?%D@l}zipw;-72Y=7UF^l4SDT(ZqPkmR8Y!gOCcn>Co z?X0*^>CI4 z-|6d0uw`qc`VRNiF+OXj4vF)B>_5JhT{#ZJ<0hL8nct}5Yx)TIViwaTwG8K^aQ`B= zAtYOY$NJdj@p1g3owthB#q>olC1Bjvf|ol6C2%U|HC<}Acf?n2!M(^<#PRheA1;?8 z-?UASdQxtzmM`pQQt>MBq$N$L9d!3IdoL3oZL#lqH!WPsrs)rd)rFeI_9_iy1GSpf zr}RFy8V~1c;;P{oPCUs-&vb;OM@Vsgn-%p7=n3m?{R=O@mNDH9vtJFKfQ=qN@Wml79_~c_Ls+iBj`avrfB}PoMqw<=>L2E2nr}1!sg#dXaNxh*w>oPADz zmd&~`uT_p$gzr;ML09Lsv`z11^#?!Ia(mW4)gce|4@Ht0J%Ncy24~y{?XqOEzK%gI z?vONE5oxG{LX8$>-XW9oS9dH0`=`b(NvvHy_|pJCiY2-b-fZe3e)TmQ>`u+U@@5C= z@@?{g>jipZw^(7U=NjbswUx(20j_8iUyWzpWwUI8Wo}Dn(S8>S(KtK$zp5y}z>JZe zY6*4fAX1OYMl?Qok#{*b8zF7@L7Q}={^r!0G8UU;RM#IYqE-I%LK!0oX&msM`u<=d z1Lm8J&P4O)jLk=}4-zOyG;!V!?Q&wS+E7ewK4|s;;5J;82}h35*lo-JqUEq8#hH={ zqvil|m*0$-B>wa$xVQyC>m;p*3Z%6t^U3iQvN~q)z5;FPg67DBSU?r^q zAtD>k38|nI8vshzMHNyehs$$9{k5*8H=C2ws^bmkaaMn&$$UYI{#S=*=Xe=u2YhoM z{_0eRdwS!K@Jq$J5rD8s{-B4chD)mEP_bKPbl;j77XDmK9rd2G;XA9C7@gxxSi38H z3)H}HZ4wkzM><5a`1j7{V`@a$B28%O$j`@pnkmxZ5` z;nOY9yo;}9QrTZH>Om8NJ;Hwj<;BWni_yJcLEUD%_6nQM(*7j5V)sXKgKEa*lIY9o z*cC^;@g0^dE{1E{Z!U`%?;l*M`5SYW==HRS@{eb$zx?eL2-v)stCMcn?Sq?c{>tGh zq{QDa^6GmpM9y<kHyB4PH8Fs(8v-K;rj=hcvED6T_;rK&& zv5|Do9$nBay1??q!FuR^w}WlKV|2BfDWcd8!3}Fa?CaPiH|X&l5*!#6n!A^5iI_hI zi$|%N&ZF?Uym>y9i}xs|zKfqj5U6E%am{kAT~}q{;TCa^tbaI$H59HDOZYb1;5dlmw*l`g7O;Ue%skm-i&uYBkW;f>4Vc?e^zgxw&{$y{J9Oh|;Pvy$| z@VcX0;FG7i>htNV!l%(|-Md_lgwd``MxOJ5_ycXmt};L0=m|0~pLJyE&qaSrgG%;{ zqu7auz)?bOfzeWT13V5an_eDOdv+EGj+YalNs*vS#i9ZQX3_&I2&4V(vXv52N6PAV zh@`}lqR;;Bp4rxg&CrQD)+Ap%c6S1lO&~XQj7ms>G<5BoIylY4f%d#K2?TR#i-j($ z30~3Bl1C#ZceaR5^J-ou?7HyA#Ug1I+%opd&x;u+Y7s?*QuBnozYNX#RvGhrHlY}J7Avzs5>2Fo@d6`m9Z_Div9P{ef8#%O5e73Ta0SBX{ zJ0tebOXM)Tb5`|DS1X!KJWdW?xwO$X-xWSPbj4rT25iH$#o&(VA%)U>X z$h~aq)MAb!X4p|hQ=_M<^hrSJZ^>`xtrL~(qOi*?mr3sNwVVe!MqBU5J@CF?#fcEF zfjV?w^9#=(?^l~KyhcPcrP>+$lITPFMyo~%h|JX>mjvfWZQL(r6a{9LKHBhXtai;> zy|yq%PDHekFnKHoO4&V8m3rnUwc{I^8iA?XIX||N_u=*$`0cyb9TlTf_NzBBJuQA- zFM8Wp%2!er_lWmlLoq`UIiS)jZ7w8X8%bEMr<#@5ve?-UFCt#8Q0J^(_BkE_ zy>9ACzvh&?ty(S&(eYtiop!Qe6rfo1lEeI|z6H#miJJ9TF~0>DmF?}^Izd~?!n}*2 zA!Zd1Y#bfvvucdm3lMUhdv#@Ub^8V*b!r82Em_pK&$YBCBfAI&e;>)we2IEu|F1|L z@QIHpp?l@tjpv|;N+8qK&<|cZ*x>`E_2M!3JG|7)A}(NTWmuS~ec3-H|q0}Vg6KgRb!ico>_0fUFm zx{GNN6fF_IpxV9W75$GKVtW$36v4MT*vWy;4-C+q%H@;GL4S}x+5A_2O&dLI>Ocyt zSdT?m3~>y&xY8eIXkmt}d{)Hjl2jEWsswWc$y$3SM>;+adK=54lDYlBg^w!Oi`joN-2#l_-LoK_?QM)Xpyw|ob6dT z=Y!Q(Inez1uQOs3Irr9cs8aht40&9(I_vWVhr1JvJwNs?n_6pP*n-DH!YSA((8YY= zfGTl;nY(`VS%2PW&P?s-MRLPi!v!`srrnOak)pZHljx z-F0(}XTG|3y{f!?yVDF>|IN;{WCfIzJog;uH>@F8Cs@4C%R5a#a_E5$X^f{Wibjkj zK7D~LiSG1hNeLfyqRe-<3CK_j#iF}~i2yi_&+N9Ch7%vDz27&sg z@F^E=ff(^{#9Yl;F)3y3vv924Rn|VT3E_(Fe%dz@7XsVMJRFjCdv^)vM>{k-=VF-9 zfG_v9!p5WF!NR`!Cc^FAn~AVJxb0vk`{aie%Iptp*}M;0G?V~9rn+MZ-;Zkw%S786 zAA6{5unWXopH7_P#(jqH@|2#wgCQ3ARVJOXz{4-z+qOh19$KkL=_L#f>0+6q$zP7E{)4yr<2=EssLorW9lYCUp_0N?v+R=`8rlXZrn-oyu|{`J=(!-4yInWx5>5 zOaY^K6aHha-1R9Fakm5C##gdu#b3Su>gHxb*HZ9P3-4LsXTaPD%7a! z7D^~mu9oRq^5a~Swk$6OP9DEC-*UCM4Lli?>gQRSUb}g7l}bk4E1Hv0B!>>DXendG z8`!&YQu7q2=0KOK!g0n6J7KGAuxL{UU!;LL+%ogNl3*kCBNiuK`5R{IP0Vk*Po2t` z&^2FK;|0OW4Ij432R&)D+Lc`-eg?S5`=1u*WU5p}2g9T`+2H)@9$f@g?^<_9EOh~# zTE3G~(1~@{foJ=e!=7K>Yr|9yBR$%<&H3yQFp=nNz7MQL99Tbin0Y|VK=WKqIgUy# zjyI$;Lkn!Om6G&Eeem;Gom;z~g5zfehDE;TuGpl)@f2q6-{$jVGN6abLTzFsT?Ehb zF!5}JCT{jDel=c=GihObqD7M8tpu(f1Us<~mQ?`;ULrs8C9!_n`iEtBO3}jGAJ{RQ z{o}kXCu%r%Rlmhluu<)!ZKy!9EWNGb=9nP_Y$N^+vC=_`KGZDZXv?n7cv4HK)Y3-4 zwhV#-QiB-qlW2G(^2`kT?JV~+IYk2KV1vOSPLOu_3j*b&Kl9JckaUb26H7Frpk;YxF38KEE@;^*rOu_OT9eIh9>xH zQQ5*5kYs>EMP7fDR0-aCGIw&kA1W0GpKZv2A6tG)*}bf@Eoi6P)UPW8x-Ask(OaoC zWBaNsPs=)mA#MC^1`=K^e^c-xofO}Sukhn?b$dRZ?$R#`NVV(481~uQcqLW6W;>q~ zK)*!5x;ArVUrqF^mq*=zZf{;&Z@14O{3KA@T)GJSf^V7s)i@&nfj)X46paBLJxkF1i|se5lRVd{|3VV}dgtW@0KIqi~I z8@WY{_BQ;F6}K!w6^8xP{ah)v@5O!JZ1g7^*={-&&8knnUXOFh&eKb>b22SCfn@T% zA7-tXAx7J27ep_n_Dcg?AQs_%t@xFAB$87Fu8o6yH@)+|nxgm?lxnJJc{Ko*VipExzRgZ93qApVHWL(XSeGwSHFo$f;k=Qy#S?62s{yVeL2V}I>>=5M7=Xh5oqj5 z1m(j)*qX4LTe-u$Uw=KLgl<2FYCAt&#Eo6?YWmt9@5FyHjeP7gC{C^@8F|BUCao{` zzEiy5b+S?@j!7y9F8rUs1-4?Uh+#jybP7a#_E(qa6nKY~D36G}XRd0xOv}3&^RtwN z)yE%A*h^4`3thpx;9$wtN+33McrBTy$UGGER{Q039Wbxiu)0&)!7KN);wUx-#R1~S z-#Z6){X^dm>XYea7_3k!<(Ps&yW`Ia!zgIo@&zqb7gha=BWUO4E!SVsCh=B;bq+MW z?zf}8s(5}KMF4AAq%~tT$&PkH0&{I=um?BrB5x2)Ou{{B`v!2HJOE8qWU?(F}WK;Sq& zxl1nzJsc6z(@{zfwi)K(Zpx+R?BVueGjh^uVM?YO+Z;tNBq3^MTar925A(dTy5wQb zRGWvJhq<&3YdJZ_m>vJceSiCWU!T|a_4(=ZAB!a*l`Klk0o6}+dn63vSUIN)(OE#-8=I&busIO%8<$bf;sI8YJQ>^lD_)69JC6jyRN4wZE;{ zdb`J(Bn#>Xq45JuP=7cy@;qS~TeGjCG#&>_{RDo>5zk;JCBwsx&rwcIZzr1Yve243 zeEad20Vs7ZysM@lHMiZP(rt8JJ4I4ml{mZ;@V8~IvGQW-cII&RhS5VR7uoN&~rs`4EY*qNBpsQnm^OQ=FhKB}K?%$U>&RD#Ph4Br-Z4mz zg~K-^l`S!a%5=3yAz4~)UM7(CKIzu9ru7*YHqfap0h$)JaP07xuV`qej;E{@R(U|z_S4dVi(Uc<97sDcHm&J75c zh5g|$7t<5zIcqVFVdN?th8U5<<6_a=yR7$tcGjTB_!)M1q|)5zii;GbH`{JY(YAO~ zFMHg&k$|UsU6Ky5g1xNuD+m`jM^FyL?VJLzgQ5mu##?blB~4sA4aL#&W%{1Yi|Aq&fDAK3m(++hWwDnoSyagVNAhCDj4CIX<^jE5N^NM{7?a`UuiM59X*3sN+U2RvH7ilp z?vEUkm3OW>x^t|Oz=vl1a{ifMAWHUPylZfw4uR72UG+|kumsRabygzBN4MfALfw64 zJkL^%iT>E#nR)n&94oH#nO0p_vX*)B$wSjcMJMx5foPhzFP>C#jVrsueY@Zwfkxp5 zEAaD6<-v`6COJnFR`g8{1lMC>9e%{58xjc#@_9?LUiEg##GVM4covZR{T*unByG;h z90Z6ppLE&S+IJFsB!E{xWc4SP8ziUhW*Z<30Y)di3`;fx4#L-s|J*y&H`biLp=m52 tzgL74Zde34AdL)Zx_S=_iscxdpf}P`HvS;{sSA{Gsge` literal 0 HcmV?d00001 diff --git a/5-NLP/15-LanguageModeling/README.md b/5-NLP/15-LanguageModeling/README.md new file mode 100644 index 00000000..7541e1f5 --- /dev/null +++ b/5-NLP/15-LanguageModeling/README.md @@ -0,0 +1,20 @@ + +# Language Modeling + +Semantic embeddings, such as Word2Vec and GloVe, are in fact a first step towards **language modeling** - creating models that somehow *understand* (or *represent*) the nature of the language. + +The main idea behind language modeling is training them on unlabeled datesets in unsupervised manner. It is important, because we have huge amounts of unlabeled text available, while the amount of labeled text would always be limited by the amount of effort we can spend on labeling. Most often, we build language models that can **predict missing words** in the text, because it is easy to mask out a random word in text and use it as a training sample. + +## Training embeddings + +In our previous examples, we have been using pre-trained semantic embeddings, but it is interesting to see how those embeddings can be trained using either CBoW, or Skip-gram architectures. + +![](../14-Embeddings/images/example-algorithms-for-converting-words-to-vectors.png) + +The idea of CBoW is exactly predicting a missing word, however, to do this we take a small sliding window of text tokens (we can denote them from W-2 to W2), and train a model to predict the central word W0 from few surrounding words. + +## More Info + +* [Official PyTorch tutorial on Language Modeling](https://pytorch.org/tutorials/beginner/nlp/word_embeddings_tutorial.html). +* [Official TensorFlow tutorial on training Word2Vec model](https://www.tensorflow.org/tutorials/text/word2vec). +* Using **gensim** framework to train most commonly used embeddings in a few lines of code is as described [in this documentation](https://pytorch.org/tutorials/beginner/nlp/word_embeddings_tutorial.html). diff --git a/5-NLP/README.md b/5-NLP/README.md index 5ea06aa8..fa9f7b3f 100644 --- a/5-NLP/README.md +++ b/5-NLP/README.md @@ -33,7 +33,23 @@ pip install -r requirements-torch.txt pip install -r requirements-tf.txt ``` +## GPU Warning + +In this section, in some of the examples we will be training quite large models. It is advisable to run notebooks on GPU-enabled compute to minimize waiting time. + +When running on GPU, you may experience situations when you run out of GPU memory. During training, the amount of GPU memory consumed depends on many factors, including minibatch size. If you experience any memory problems - you may try to minimize the minibatch size in the code. + +Also, some older versions of Tensorflow do not release GPU memory correctly if we are training multiple models in one Python kernel. In order to use GPU memory cautiously, you may set tensorflow option to grow GPU memory allocation only when required. You would need to include the following code in your notebooks: + +```python +physical_devices = tf.config.list_physical_devices('GPU') +if len(physical_devices)>0: + tf.config.experimental.set_memory_growth(physical_devices[0], True) +``` + ## Contents * [Representing text as tensors](13-TextRep/README.md) * [Word Embeddings](14-Emdeddings/README.md) +* [Language Modeling](15-LanguageModeling/README.md) + diff --git a/README.md b/README.md index 0080b20a..42d2a503 100644 --- a/README.md +++ b/README.md @@ -62,8 +62,8 @@ For a gentle introduction to *AI in the Cloud* topic you may consider taking [Ge MS Learn PAT 13Text Representation. Bow/TF-IDFTextPyTorchTensorflow -14Semantic word embeddingsTextPyTorchTensorflow -15Training your own embeddingsTextPyTorchTensorflow +14Semantic word embeddings. Word2Vec and GloVeTextPyTorchTensorflow +15Language Modeling. Training your own embeddingsTextPyTorchTensorflow 16Recurrent Neural NetworksTextPyTorchTensorflow 17Generative Recurrent NetworksTextPyTorchTensorflow 18Language Modelling. Transformers. BERT.TextPyTorchTensorflow