From 92e8f9ae061d058d06ea7a4024f8b32f767318ee Mon Sep 17 00:00:00 2001 From: Dmitri Soshnikov Date: Fri, 14 Jan 2022 10:48:30 +0300 Subject: [PATCH] Add RNNs and Embedding readme --- 5-NLP/14-Embeddings/EmbeddingsPyTorch.ipynb | 5 + 5-NLP/14-Embeddings/README.md | 41 + 5-NLP/16-RNN/README.md | 58 + 5-NLP/16-RNN/RNNPyTorch.ipynb | 486 ++++++ 5-NLP/16-RNN/RNNTF.ipynb | 443 ++++++ .../images/long-short-term-memory-cell.svg | 1334 +++++++++++++++++ 5-NLP/16-RNN/images/multi-layer-lstm.jpg | Bin 0 -> 29122 bytes 5-NLP/16-RNN/images/rnn-anatomy.png | Bin 0 -> 16735 bytes 5-NLP/16-RNN/images/rnn.png | Bin 0 -> 18140 bytes 5-NLP/16-RNN/torchnlp.py | 104 ++ 5-NLP/README.md | 4 +- README.md | 6 +- 12 files changed, 2477 insertions(+), 4 deletions(-) create mode 100644 5-NLP/16-RNN/README.md create mode 100644 5-NLP/16-RNN/RNNPyTorch.ipynb create mode 100644 5-NLP/16-RNN/RNNTF.ipynb create mode 100644 5-NLP/16-RNN/images/long-short-term-memory-cell.svg create mode 100644 5-NLP/16-RNN/images/multi-layer-lstm.jpg create mode 100644 5-NLP/16-RNN/images/rnn-anatomy.png create mode 100644 5-NLP/16-RNN/images/rnn.png create mode 100644 5-NLP/16-RNN/torchnlp.py diff --git a/5-NLP/14-Embeddings/EmbeddingsPyTorch.ipynb b/5-NLP/14-Embeddings/EmbeddingsPyTorch.ipynb index c77795cf..c70f64af 100644 --- a/5-NLP/14-Embeddings/EmbeddingsPyTorch.ipynb +++ b/5-NLP/14-Embeddings/EmbeddingsPyTorch.ipynb @@ -686,6 +686,11 @@ "\n", "The pretrained embeddings above represent both of these meanings of the word 'play' in the same embedding. To overcome this limitation, we need to build embeddings based on the **language model**, which is trained on a large corpus of text, and *knows* how words can be put together in different contexts. Discussing contextual embeddings is out of scope for this tutorial, but we will come back to them when talking about language models in the next unit.\n" ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [] } ], "metadata": { diff --git a/5-NLP/14-Embeddings/README.md b/5-NLP/14-Embeddings/README.md index e69de29b..b94abb0f 100644 --- a/5-NLP/14-Embeddings/README.md +++ b/5-NLP/14-Embeddings/README.md @@ -0,0 +1,41 @@ +# Embeddings + +When training classifiers based on BoW or TF/IDF, we operated on high-dimensional bag-of-words vectors with length `vocab_size`, and we were explicitly converting from low-dimensional positional representation vectors into sparse one-hot representation. This one-hot representation is not memory-efficient, in addition, each word is treated independently from each other, i.e. one-hot encoded vectors do not express any semantic similarity between words. + +The idea of **embedding** is to represent words by lower-dimensional dense vectors, which somehow reflect semantic meaning of a word. We will later discuss how to build meaningful word embeddings, but for now let's just think of embeddings as a way to lower dimensionality of a word vector. + +So, embedding layer would take a word as an input, and produce an output vector of specified `embedding_size`. In a sense, it is very similar to `Linear` layer, but instead of taking one-hot encoded vector, it will be able to take a word number as an input, allowing us to avoid creating large one-hot-encoded vectors. + +By using embedding layer as a first layer in our classifier network, we can switch from bag-or-words to **embedding bag** model, where we first convert each word in our text into corresponding embedding, and then compute some aggregate function over all those embeddings, such as `sum`, `average` or `max`. + +![Image showing an embedding classifier for five sequence words.](images/embedding-classifier-example.png) + +## Continue in Notebooks + +* [Embeddings with PyTorch](EmbeddingsPyTorch.ipynb) +* [Embeddings Tensorflow](EmbeddingsTF.ipynb) + +## Semantic Embeddings: Word2Vec + +While embedding layer learnt to map words to vector representation, however, this representation did not necessarily have much semantical meaning. It would be nice to learn such vector representation that similar words or symonims correspond to vectors that are close to each other in terms of some vector distance (eg. Euclidean distance). + +To do that, we need to pre-train our embedding model on a large collection of text in a specific way. One of the first ways to train semantic embeddings is called [Word2Vec](https://en.wikipedia.org/wiki/Word2vec). It is based on two main architectures that are used to produce a distributed representation of words: + + - **Continuous bag-of-words** (CBoW) — in this architecture, we train the model to predict a word from surrounding context. Given the ngram $(W_{-2},W_{-1},W_0,W_1,W_2)$, the goal of the model is to predict $W_0$ from $(W_{-2},W_{-1},W_1,W_2)$. + - **Continuous skip-gram** is opposite to CBoW. The model uses surrounding window of context words to predict the current word. + +CBoW is faster, while skip-gram is slower, but does a better job of representing infrequent words. + +![Image showing both CBoW and Skip-Gram algorithms to convert words to vectors.](./images/example-algorithms-for-converting-words-to-vectors.png) + +Word2Vec pre-trained embeddings (as well as other similar models, such as GloVe) can also be used in place of embedding layer in neural networks. However, we need to deal with vocabularies, because the vocabulary used to pre-train Word2Vec/GloVe is likely to differ from the vocabulary in our text corpus. Have a look into Notebooks to see how this problem can be resolved. + +## Contextual Embeddings + +One key limitation of tradition pretrained embedding representations such as Word2Vec is the problem of word sense disambiguation. While pretrained embeddings can capture some of the meaning of words in context, every possible meaning of a word is encoded into the same embedding. This can cause problems in downstream models, since many words such as the word 'play' have different meanings depending on the context they are used in. + +For example word 'play' in those two different sentences have quite different meaning: +- I went to a **play** at the theature. +- John wants to **play** with his friends. + +The pretrained embeddings above represent both of these meanings of the word 'play' in the same embedding. To overcome this limitation, we need to build embeddings based on the **language model**, which is trained on a large corpus of text, and *knows* how words can be put together in different contexts. Discussing contextual embeddings is out of scope for this tutorial, but we will come back to them when talking about language models later in the course. diff --git a/5-NLP/16-RNN/README.md b/5-NLP/16-RNN/README.md new file mode 100644 index 00000000..0f403d5f --- /dev/null +++ b/5-NLP/16-RNN/README.md @@ -0,0 +1,58 @@ +# Recurrent Neural Networks + +In the previous sections, we have been using rich semantic representations of text, and a simple linear classifier on top of the embeddings. What this architecture does is to capture aggregated meaning of words in a sentence, but it does not take into account the **order** of words, because aggregation operation on top of embeddings removed this information from the original text. Because these models are unable to model word ordering, they cannot solve more complex or ambiguous tasks such as text generation or question answering. + +To capture the meaning of text sequence, we need to use another neural network architecture, which is called a **recurrent neural network**, or RNN. In RNN, we pass our sentence through the network one symbol at a time, and the network produces some **state**, which we then pass to the network again with the next symbol. + +![RNN](./images/rnn.png) + +Given the input sequence of tokens X0,...,Xn, RNN creates a sequence of neural network blocks, and trains this sequence end-to-end using back propagation. Each network block takes a pair (Xi,Si) as an input, and produces Si+1 as a result. Final state Sn or (output Yn) goes into a linear classifier to produce the result. All network blocks share the same weights, and are trained end-to-end using one backpropagation pass. + +Because state vectors S0,...,Sn are passed through the network, it is able to learn the sequential dependencies between words. For example, when the word *not* appears somewhere in the sequence, it can learn to negate certain elements within the state vector, resulting in negation. + +> Since weights of all RNN blocks on the picture are shared, the same picture can be represented as one block (on the right) with a recurrent feedback loop, which passes output state of the network back to the input. + +## Anatomy of RNN Cell + +Let's see how simple RNN cell is organized. It accepts previous state Si-1 and current symbol Xi as inputs, and has to produce output state Si (and, sometimes, we are also interested in some other output Yi, as in case with generative networks). + +Simple RNN cell has two weight matrices inside: one transforms input symbol (let call it W), and another one transforms input state (H). In this case the output of the network is calculated as σ(W×Xi+H×Si-1+b), where σ is the activation function, b is additional bias. + +![RNN Cell Anatomy](images/rnn-anatomy.png) + +In many cases, input tokens are passed through the embedding layer before entering the RNN to lower the dimensionality. In this case, if the dimension of the input vectors is *emb_size*, and state vector is *hid_size* - the size of W is *emb_size*×*hid_size*, and the size of H is *hid_size*×*hid_size*. + +## Long Short Term Memory (LSTM) + +One of the main problems of classical RNNs is so-called **vanishing gradients** problem. Because RNNs are trained end-to-end in one back-propagation pass, it is having hard times propagating error to the first layers of the network, and thus the network cannot learn relationships between distant tokens. One of the ways to avoid this problem is to introduce **explicit state management** by using so called **gates**. There are two most known architectures of this kind: **Long Short Term Memory** (LSTM) and **Gated Relay Unit** (GRU). + +![Image showing an example long short term memory cell](./images/long-short-term-memory-cell.svg) + +LSTM Network is organized in a manner similar to RNN, but there are two states that are being passed from layer to layer: actual state C, and hidden vector H. At each unit, hidden vector Hi is concatenated with input Xi, and they control what happens to the state C via **gates**. Each gate is a neural network with sigmoid activation (output in the range [0,1]), which can be thought of as bitwise mask when multiplied by the state vector. There are the following gates (from left to right on the picture above): +* **forget gate** takes hidden vector and determines, which components of the vector C we need to forget, and which to pass through. +* **input gate** takes some information from the input and hidden vector, and inserts it into state. +* **output gate** transforms state via some linear layer with *tanh* activation, then selects some of its components using hidden vector Hi to produce new state Ci+1. + +Components of the state C can be thought of as some flags that can be switched on and off. For example, when we encounter a name *Alice* in the sequence, we may want to assume that it refers to female character, and raise the flag in the state that we have female noun in the sentence. When we further encounter phrases *and Tom*, we will raise the flag that we have plural noun. Thus by manipulating state we can supposedly keep track of grammatical properties of sentence parts. + +> **Note**: A great resource for understanding internals of LSTM is this great article [Understanding LSTM Networks](https://colah.github.io/posts/2015-08-Understanding-LSTMs/) by Christopher Olah. + +## Bidirectional and multilayer RNNs + +We have discussed recurrent networks that operate in one direction, from beginning of a sequence to the end. It looks natural, because it resembles the way we read and listen to speech. However, since in many practical cases we have random access to the input sequence, it might make sense to run recurrent computation in both directions. Such networks are call **bidirectional** RNNs. When dealing with bidirectional network, we would need two hidden state vectors, one for each direction. + +Recurrent network, one-directional or bidirectional, captures certain patterns within a sequence, and can store them into state vector or pass into output. As with convolutional networks, we can build another recurrent layer on top of the first one to capture higher level patterns, build from low-level patterns extracted by the first layer. This leads us to the notion of **multi-layer RNN**, which consists of two or more recurrent networks, where output of the previous layer is passed to the next layer as input. + +![Image showing a Multilayer long-short-term-memory- RNN](./images/multi-layer-lstm.jpg) + +*Picture from [this wonderful post](https://towardsdatascience.com/from-a-lstm-cell-to-a-multilayer-lstm-network-with-pytorch-2899eb5696f3) by Fernando López* + +## Continue to Notebooks + +* [RNNs with PyTorch](RNNPyTorch.ipynb) +* [RNNs with Tensorflow](RNNTF.ipynb) + +## RNNs for other tasks + +In this unit, we have seen that RNNs can be used for sequence classification, but in fact, they can handle many more tasks, such as text generation, machine translation, and more. We will consider those tasks in the next unit. + diff --git a/5-NLP/16-RNN/RNNPyTorch.ipynb b/5-NLP/16-RNN/RNNPyTorch.ipynb new file mode 100644 index 00000000..2f7f6d07 --- /dev/null +++ b/5-NLP/16-RNN/RNNPyTorch.ipynb @@ -0,0 +1,486 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Recurrent neural networks\n", + "\n", + "In the previous module, we have been using rich semantic representations of text, and a simple linear classifier on top of the embeddings. What this architecture does is to capture aggregated meaning of words in a sentence, but it does not take into account the **order** of words, because aggregation operation on top of embeddings removed this information from the original text. Because these models are unable to model word ordering, they cannot solve more complex or ambiguous tasks such as text generation or question answering.\n", + "\n", + "To capture the meaning of text sequence, we need to use another neural network architecture, which is called a **recurrent neural network**, or RNN. In RNN, we pass our sentence through the network one symbol at a time, and the network produces some **state**, which we then pass to the network again with the next symbol.\n", + "\n", + "\"RNN\"\n", + "\n", + "Given the input sequence of tokens $X_0,\\dots,X_n$, RNN creates a sequence of neural network blocks, and trains this sequence end-to-end using back propagation. Each network block takes a pair $(X_i,S_i)$ as an input, and produces $S_{i+1}$ as a result. Final state $S_n$ or output $X_n$ goes into a linear classifier to produce the result. All network blocks share the same weights, and are trained end-to-end using one backpropagation pass.\n", + "\n", + "Because state vectors $S_0,\\dots,S_n$ are passed through the network, it is able to learn the sequential dependencies between words. For example, when the word *not* appears somewhere in the sequence, it can learn to negate certain elements within the state vector, resulting in negation. \n", + "\n", + "> Since weights of all RNN blocks on the picture are shared, the same picture can be represented as one block (on the right) with a recurrent feedback loop, which passes output state of the network back to the input.\n", + "\n", + "Let's see how recurrent neural networks can help us classify our news dataset." + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Loading dataset...\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "d:\\WORK\\ai-for-beginners\\5-NLP\\16-RNN\\data\\train.csv: 29.5MB [00:01, 28.3MB/s] \n", + "d:\\WORK\\ai-for-beginners\\5-NLP\\16-RNN\\data\\test.csv: 1.86MB [00:00, 9.72MB/s] \n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Building vocab...\n" + ] + } + ], + "source": [ + "import torch\n", + "import torchtext\n", + "from torchnlp import *\n", + "train_dataset, test_dataset, classes, vocab = load_dataset()\n", + "vocab_size = len(vocab)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Simple RNN classifier\n", + "\n", + "In case of simple RNN, each recurrent unit is a simple linear network, which takes concatenated input vector and state vector, and produce a new state vector. PyTorch represents this unit with `RNNCell` class, and a networks of such cells - as `RNN` layer.\n", + "\n", + "To define an RNN classifier, we will first apply an embedding layer to lower the dimensionality of input vocabulary, and then have RNN layer on top of it: " + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "metadata": {}, + "outputs": [], + "source": [ + "class RNNClassifier(torch.nn.Module):\n", + " def __init__(self, vocab_size, embed_dim, hidden_dim, num_class):\n", + " super().__init__()\n", + " self.hidden_dim = hidden_dim\n", + " self.embedding = torch.nn.Embedding(vocab_size, embed_dim)\n", + " self.rnn = torch.nn.RNN(embed_dim,hidden_dim,batch_first=True)\n", + " self.fc = torch.nn.Linear(hidden_dim, num_class)\n", + "\n", + " def forward(self, x):\n", + " batch_size = x.size(0)\n", + " x = self.embedding(x)\n", + " x,h = self.rnn(x)\n", + " return self.fc(x.mean(dim=1))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "> **Note:** We use untrained embedding layer here for simplicity, but for even better results we can use pre-trained embedding layer with Word2Vec or GloVe embeddings, as described in the previous unit. For better understanding, you might want to adapt this code to work with pre-trained embeddings.\n", + "\n", + "In our case, we will use padded data loader, so each batch will have a number of padded sequences of the same length. RNN layer will take the sequence of embedding tensors, and produce two outputs: \n", + "* $x$ is a sequence of RNN cell outputs at each step\n", + "* $h$ is a final hidden state for the last element of the sequence\n", + "\n", + "We then apply a fully-connected linear classifier to get the number of class.\n", + "\n", + "> **Note:** RNNs are quite difficult to train, because once the RNN cells are unrolled along the sequence length, the resulting number of layers involved in back propagation is quite large. Thus we need to select small learning rate, and train the network on larger dataset to produce good results. It can take quite a long time, so using GPU is preferred." + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": { + "scrolled": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "3200: acc=0.3090625\n", + "6400: acc=0.38921875\n", + "9600: acc=0.4590625\n", + "12800: acc=0.511953125\n", + "16000: acc=0.5506875\n", + "19200: acc=0.57921875\n", + "22400: acc=0.6070089285714285\n", + "25600: acc=0.6304296875\n", + "28800: acc=0.6484027777777778\n", + "32000: acc=0.66509375\n", + "35200: acc=0.6790056818181818\n", + "38400: acc=0.6929166666666666\n", + "41600: acc=0.7035817307692308\n", + "44800: acc=0.7137276785714286\n", + "48000: acc=0.72225\n", + "51200: acc=0.73001953125\n", + "54400: acc=0.7372794117647059\n", + "57600: acc=0.7436631944444444\n", + "60800: acc=0.7503947368421052\n", + "64000: acc=0.75634375\n", + "67200: acc=0.7615773809523809\n", + "70400: acc=0.7662642045454545\n", + "73600: acc=0.7708423913043478\n", + "76800: acc=0.7751822916666666\n", + "80000: acc=0.7790625\n", + "83200: acc=0.7825\n", + "86400: acc=0.7858564814814815\n", + "89600: acc=0.7890513392857142\n", + "92800: acc=0.7920474137931034\n", + "96000: acc=0.7952708333333334\n", + "99200: acc=0.7982258064516129\n", + "102400: acc=0.80099609375\n", + "105600: acc=0.8037594696969697\n", + "108800: acc=0.8060569852941176\n" + ] + } + ], + "source": [ + "train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=16, collate_fn=padify, shuffle=True)\n", + "net = RNNClassifier(vocab_size,64,32,len(classes)).to(device)\n", + "train_epoch(net,train_loader, lr=0.001)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Long Short Term Memory (LSTM)\n", + "\n", + "One of the main problems of classical RNNs is so-called **vanishing gradients** problem. Because RNNs are trained end-to-end in one back-propagation pass, it is having hard times propagating error to the first layers of the network, and thus the network cannot learn relationships between distant tokens. One of the ways to avoid this problem is to introduce **explicit state management** by using so called **gates**. There are two most known architectures of this kind: **Long Short Term Memory** (LSTM) and **Gated Relay Unit** (GRU).\n", + "\n", + "![Image showing an example long short term memory cell](./images/long-short-term-memory-cell.svg)\n", + "\n", + "LSTM Network is organized in a manner similar to RNN, but there are two states that are being passed from layer to layer: actual state $c$, and hidden vector $h$. At each unit, hidden vector $h_i$ is concatenated with input $x_i$, and they control what happens to the state $c$ via **gates**. Each gate is a neural network with sigmoid activation (output in the range $[0,1]$), which can be thought of as bitwise mask when multiplied by the state vector. There are the following gates (from left to right on the picture above):\n", + "* **forget gate** takes hidden vector and determines, which components of the vector $c$ we need to forget, and which to pass through. \n", + "* **input gate** takes some information from the input and hidden vector, and inserts it into state.\n", + "* **output gate** transforms state via some linear layer with $\\tanh$ activation, then selects some of its components using hidden vector $h_i$ to produce new state $c_{i+1}$.\n", + "\n", + "Components of the state $c$ can be thought of as some flags that can be switched on and off. For example, when we encounter a name *Alice* in the sequence, we may want to assume that it refers to female character, and raise the flag in the state that we have female noun in the sentence. When we further encounter phrases *and Tom*, we will raise the flag that we have plural noun. Thus by manipulating state we can supposedly keep track of grammatical properties of sentence parts.\n", + "\n", + "> **Note**: A great resource for understanding internals of LSTM is this great article [Understanding LSTM Networks](https://colah.github.io/posts/2015-08-Understanding-LSTMs/) by Christopher Olah.\n", + "\n", + "While internal structure of LSTM cell may look complex, PyTorch hides this implementation inside `LSTMCell` class, and provides `LSTM` object to represent the whole LSTM layer. Thus, implementation of LSTM classifier will be pretty similar to the simple RNN which we have seen above:" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "metadata": {}, + "outputs": [], + "source": [ + "class LSTMClassifier(torch.nn.Module):\n", + " def __init__(self, vocab_size, embed_dim, hidden_dim, num_class):\n", + " super().__init__()\n", + " self.hidden_dim = hidden_dim\n", + " self.embedding = torch.nn.Embedding(vocab_size, embed_dim)\n", + " self.embedding.weight.data = torch.randn_like(self.embedding.weight.data)-0.5\n", + " self.rnn = torch.nn.LSTM(embed_dim,hidden_dim,batch_first=True)\n", + " self.fc = torch.nn.Linear(hidden_dim, num_class)\n", + "\n", + " def forward(self, x):\n", + " batch_size = x.size(0)\n", + " x = self.embedding(x)\n", + " x,(h,c) = self.rnn(x)\n", + " return self.fc(h[-1])" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Now let's train our network. Note that training LSTM is also quite slow, and you may not seem much raise in accuracy in the beginning of training. Also, you may need to play with `lr` learning rate parameter to find the learning rate that results in reasonable training speed, and yet does not cause " + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "3200: acc=0.259375\n", + "6400: acc=0.25859375\n", + "9600: acc=0.26177083333333334\n", + "12800: acc=0.2784375\n", + "16000: acc=0.313\n", + "19200: acc=0.3528645833333333\n", + "22400: acc=0.3965625\n", + "25600: acc=0.4385546875\n", + "28800: acc=0.4752777777777778\n", + "32000: acc=0.505375\n", + "35200: acc=0.5326704545454546\n", + "38400: acc=0.5557552083333334\n", + "41600: acc=0.5760817307692307\n", + "44800: acc=0.5954910714285714\n", + "48000: acc=0.6118333333333333\n", + "51200: acc=0.62681640625\n", + "54400: acc=0.6404779411764706\n", + "57600: acc=0.6520138888888889\n", + "60800: acc=0.662828947368421\n", + "64000: acc=0.673546875\n", + "67200: acc=0.6831547619047619\n", + "70400: acc=0.6917897727272727\n", + "73600: acc=0.6997146739130434\n", + "76800: acc=0.707109375\n", + "80000: acc=0.714075\n", + "83200: acc=0.7209134615384616\n", + "86400: acc=0.727037037037037\n", + "89600: acc=0.7326674107142858\n", + "92800: acc=0.7379633620689655\n", + "96000: acc=0.7433645833333333\n", + "99200: acc=0.7479032258064516\n", + "102400: acc=0.752119140625\n", + "105600: acc=0.7562405303030303\n", + "108800: acc=0.76015625\n", + "112000: acc=0.7641339285714286\n", + "115200: acc=0.7677777777777778\n", + "118400: acc=0.7711233108108108\n" + ] + }, + { + "data": { + "text/plain": [ + "(0.03487814127604167, 0.7728)" + ] + }, + "execution_count": 5, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "net = LSTMClassifier(vocab_size,64,32,len(classes)).to(device)\n", + "train_epoch(net,train_loader, lr=0.001)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Packed sequences\n", + "\n", + "In our example, we had to pad all sequences in the minibatch with zero vectors. While it results in some memory waste, with RNNs it is more critical that additional RNN cells are created for the padded input items, which take part in training, yet do not carry any important input information. It would be much better to train RNN only to the actual sequence size.\n", + "\n", + "To do that, a special format of padded sequence storage is introduced in PyTorch. Suppose we have input padded minibatch which looks like this:\n", + "```\n", + "[[1,2,3,4,5],\n", + " [6,7,8,0,0],\n", + " [9,0,0,0,0]]\n", + "```\n", + "Here 0 represents padded values, and the actual length vector of input sequences is `[5,3,1]`.\n", + "\n", + "In order to effectively train RNN with padded sequence, we want to begin training first group of RNN cells with large minibatch (`[1,6,9]`), but then end processing of third sequence, and continue training with shorted minibatches (`[2,7]`, `[3,8]`), and so on. Thus, packed sequence is represented as one vector - in our case `[1,6,9,2,7,3,8,4,5]`, and length vector (`[5,3,1]`), from which we can easily reconstruct the original padded minibatch.\n", + "\n", + "To produce packed sequence, we can use `torch.nn.utils.rnn.pack_padded_sequence` function. All recurrent layers, including RNN, LSTM and GRU, support packed sequences as input, and produce packed output, which can be decoded using `torch.nn.utils.rnn.pad_packed_sequence`.\n", + "\n", + "To be able to produce packed sequence, we need to pass length vector to the network, and thus we need a different function to prepare minibatches:" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "metadata": {}, + "outputs": [], + "source": [ + "def pad_length(b):\n", + " # build vectorized sequence\n", + " v = [encode(x[1]) for x in b]\n", + " # compute max length of a sequence in this minibatch and length sequence itself\n", + " len_seq = list(map(len,v))\n", + " l = max(len_seq)\n", + " return ( # tuple of three tensors - labels, padded features, length sequence\n", + " torch.LongTensor([t[0]-1 for t in b]),\n", + " torch.stack([torch.nn.functional.pad(torch.tensor(t),(0,l-len(t)),mode='constant',value=0) for t in v]),\n", + " torch.tensor(len_seq)\n", + " )\n", + "\n", + "train_loader_len = torch.utils.data.DataLoader(train_dataset, batch_size=16, collate_fn=pad_length, shuffle=True)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Actual network would be very similar to `LSTMClassifier` above, but `forward` pass will receive both padded minibatch and the vector of sequence lengths. After computing the embedding, we compute packed sequence, pass it to LSTM layer, and then unpack the result back.\n", + "\n", + "> **Note**: We actually do not use unpacked result `x`, because we use output from the hidden layers in the following computations. Thus, we can remove the unpacking altogether from this code. The reason we place it here is for you to be able to modify this code easily, in case you should need to use network output in further computations." + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "metadata": {}, + "outputs": [], + "source": [ + "class LSTMPackClassifier(torch.nn.Module):\n", + " def __init__(self, vocab_size, embed_dim, hidden_dim, num_class):\n", + " super().__init__()\n", + " self.hidden_dim = hidden_dim\n", + " self.embedding = torch.nn.Embedding(vocab_size, embed_dim)\n", + " self.embedding.weight.data = torch.randn_like(self.embedding.weight.data)-0.5\n", + " self.rnn = torch.nn.LSTM(embed_dim,hidden_dim,batch_first=True)\n", + " self.fc = torch.nn.Linear(hidden_dim, num_class)\n", + "\n", + " def forward(self, x, lengths):\n", + " batch_size = x.size(0)\n", + " x = self.embedding(x)\n", + " pad_x = torch.nn.utils.rnn.pack_padded_sequence(x,lengths,batch_first=True,enforce_sorted=False)\n", + " pad_x,(h,c) = self.rnn(pad_x)\n", + " x, _ = torch.nn.utils.rnn.pad_packed_sequence(pad_x,batch_first=True)\n", + " return self.fc(h[-1])" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Now let's do the training:" + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "metadata": { + "scrolled": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "3200: acc=0.285625\n", + "6400: acc=0.33359375\n", + "9600: acc=0.3876041666666667\n", + "12800: acc=0.44078125\n", + "16000: acc=0.4825\n", + "19200: acc=0.5235416666666667\n", + "22400: acc=0.5559821428571429\n", + "25600: acc=0.58609375\n", + "28800: acc=0.6116666666666667\n", + "32000: acc=0.63340625\n", + "35200: acc=0.6525284090909091\n", + "38400: acc=0.668515625\n", + "41600: acc=0.6822596153846154\n", + "44800: acc=0.6948214285714286\n", + "48000: acc=0.7052708333333333\n", + "51200: acc=0.71521484375\n", + "54400: acc=0.7239889705882353\n", + "57600: acc=0.7315277777777778\n", + "60800: acc=0.7388486842105263\n", + "64000: acc=0.74571875\n", + "67200: acc=0.7518303571428572\n", + "70400: acc=0.7576988636363636\n", + "73600: acc=0.7628940217391305\n", + "76800: acc=0.7681510416666667\n", + "80000: acc=0.7728125\n", + "83200: acc=0.7772235576923077\n", + "86400: acc=0.7815393518518519\n", + "89600: acc=0.7857700892857142\n", + "92800: acc=0.7895043103448276\n", + "96000: acc=0.7930520833333333\n", + "99200: acc=0.7959072580645161\n", + "102400: acc=0.798994140625\n", + "105600: acc=0.802064393939394\n", + "108800: acc=0.8051378676470589\n", + "112000: acc=0.8077857142857143\n", + "115200: acc=0.8104600694444445\n", + "118400: acc=0.8128293918918919\n" + ] + }, + { + "data": { + "text/plain": [ + "(0.029785829671223958, 0.8138166666666666)" + ] + }, + "execution_count": 8, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "net = LSTMPackClassifier(vocab_size,64,32,len(classes)).to(device)\n", + "train_epoch_emb(net,train_loader_len, lr=0.001,use_pack_sequence=True)\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "> **Note:** You may have noticed the parameter `use_pack_sequence` that we pass to the training function. Currently, `pack_padded_sequence` function requires length sequence tensor to be on CPU device, and thus training function needs to avoid moving the length sequence data to GPU when training. You can look into implementation of `train_emb` function in the [`torchnlp.py`](torchnlp.py) file." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Bidirectional and multilayer RNNs\n", + "\n", + "In our examples, all recurrent networks operated in one direction, from beginning of a sequence to the end. It looks natural, because it resembles the way we read and listen to speech. However, since in many practical cases we have random access to the input sequence, it might make sense to run recurrent computation in both directions. Such networks are call **bidirectional** RNNs, and they can be created by passing `bidirectional=True` parameter to RNN/LSTM/GRU constructor.\n", + "\n", + "When dealing with bidirectional network, we would need two hidden state vectors, one for each direction. PyTorch encodes those vectors as one vector of twice larger size, which is quite convenient, because you would normally pass the resulting hidden state to fully-connected linear layer, and you would just need to take this increase in size into account when creating the layer.\n", + "\n", + "Recurrent network, one-directional or bidirectional, captures certain patterns within a sequence, and can store them into state vector or pass into output. As with convolutional networks, we can build another recurrent layer on top of the first one to capture higher level patterns, build from low-level patterns extracted by the first layer. This leads us to the notion of **multi-layer RNN**, which consists of two or more recurrent networks, where output of the previous layer is passed to the next layer as input.\n", + "\n", + "![Image showing a Multilayer long-short-term-memory- RNN](images/multi-layer-lstm.jpg)\n", + "\n", + "*Picture from [this wonderful post](https://towardsdatascience.com/from-a-lstm-cell-to-a-multilayer-lstm-network-with-pytorch-2899eb5696f3) by Fernando López*\n", + "\n", + "PyTorch makes constructing such networks an easy task, because you just need to pass `num_layers` parameter to RNN/LSTM/GRU constructor to build several layers of recurrence automatically. This would also mean that the size of hidden/state vector would increase proportionally, and you would need to take this into account when handling the output of recurrent layers." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## RNNs for other tasks\n", + "\n", + "In this unit, we have seen that RNNs can be used for sequence classification, but in fact, they can handle many more tasks, such as text generation, machine translation, and more. We will consider those tasks in the next unit." + ] + } + ], + "metadata": { + "interpreter": { + "hash": "0cb620c6d4b9f7a635928804c26cf22403d89d98d79684e4529119355ee6d5a5" + }, + "kernelspec": { + "display_name": "py37_pytorch", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.8.12" + } + }, + "nbformat": 4, + "nbformat_minor": 2 +} diff --git a/5-NLP/16-RNN/RNNTF.ipynb b/5-NLP/16-RNN/RNNTF.ipynb new file mode 100644 index 00000000..c461cedc --- /dev/null +++ b/5-NLP/16-RNN/RNNTF.ipynb @@ -0,0 +1,443 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "source": [ + "# Recurrent neural networks\n", + "\n", + "In the previous module, we covered rich semantic representations of text. The architecture we've been using captures the aggregated meaning of words in a sentence, but it does not take into account the **order** of the words, because the aggregation operation that follows the embeddings removes this information from the original text. Because these models are unable to represent word ordering, they cannot solve more complex or ambiguous tasks such as text generation or question answering.\n", + "\n", + "To capture the meaning of a text sequence, we'll use a neural network architecture called **recurrent neural network**, or RNN. When using an RNN, we pass our sentence through the network one token at a time, and the network produces some **state**, which we then pass to the network again with the next token.\n", + "\n", + "![Image showing an example recurrent neural network generation.](images/rnn.png)\n", + "\n", + "Given the input sequence of tokens $X_0,\\dots,X_n$, the RNN creates a sequence of neural network blocks, and trains this sequence end-to-end using backpropagation. Each network block takes a pair $(X_i,S_i)$ as an input, and produces $S_{i+1}$ as a result. The final state $S_n$ or output $Y_n$ goes into a linear classifier to produce the result. All network blocks share the same weights, and are trained end-to-end using one backpropagation pass.\n", + "\n", + "> The figure above shows recurrent neural network in the unrolled form (on the left), and in more compact recurrent representation (on the right). It is important to realize that all RNN Cells have the same **shareable weights**.\n", + "\n", + "Because state vectors $S_0,\\dots,S_n$ are passed through the network, the RNN is able to learn sequential dependencies between words. For example, when the word *not* appears somewhere in the sequence, it can learn to negate certain elements within the state vector.\n", + "\n", + "Inside, each RNN cell contains two weight matrices: $W_H$ and $W_I$, and bias $b$. At each RNN step, given input $X_i$ and input state $S_i$, output state is calculated as $S_{i+1} = f(W_H\\times S_i + W_I\\times X_i+b)$, where $f$ is an activation function (often $\\tanh$).\n", + "\n", + "> For problems like text generation (that we will cover in the next unit) or machine translation we also want to get some output value at each RNN step. In this case, there is also another matrix $W_O$, and output is caluclated as $Y_i=f(W_O\\times S_i+b_O)$.\n", + "\n", + "Let's see how recurrent neural networks can help us classify our news dataset.\n", + "\n", + "> For the sandbox environment, we need to run the following cell to make sure the required library is installed, and data is prefetched. If you are running locally, you can skip the following cell." + ], + "metadata": {} + }, + { + "cell_type": "code", + "source": [ + "import sys\n", + "!{sys.executable} -m pip install --quiet tensorflow_datasets==4.4.0\n", + "!cd ~ && wget -q -O - https://mslearntensorflowlp.blob.core.windows.net/data/tfds-ag-news.tgz | tar xz" + ], + "outputs": [], + "execution_count": 1, + "metadata": {} + }, + { + "cell_type": "code", + "source": [ + "import tensorflow as tf\n", + "from tensorflow import keras\n", + "import tensorflow_datasets as tfds\n", + "import numpy as np\n", + "\n", + "# We are going to be training pretty large models. In order not to face errors, we need\n", + "# to set tensorflow option to grow GPU memory allocation when required\n", + "physical_devices = tf.config.list_physical_devices('GPU') \n", + "if len(physical_devices)>0:\n", + " tf.config.experimental.set_memory_growth(physical_devices[0], True)\n", + "\n", + "ds_train, ds_test = tfds.load('ag_news_subset').values()" + ], + "outputs": [], + "execution_count": 2, + "metadata": {} + }, + { + "cell_type": "markdown", + "source": [ + "When training large models, GPU memory allocation may become a problem. We also may need to experiment with different minibatch sizes, so that the data fits into our GPU memory, yet the training is fast enough. If you are running this code on your own GPU machine, you may experiment with adjusting minibatch size to speed up training.\r\n", + "\r\n", + "> **Note**: Certain versions of NVidia drivers are known not to release the memory after training the model. We are running several examples in this notebooks, and it might cause memory to be exhausted in certain setups, especially if you are doing your own experiments as part of the same notebook. If you encounter some weird errors when starting to train the model, you may want to restart notebook kernel." + ], + "metadata": { + "nteract": { + "transient": { + "deleting": false + } + } + } + }, + { + "cell_type": "code", + "source": [ + "batch_size = 16\r\n", + "embed_size = 64" + ], + "outputs": [], + "execution_count": 3, + "metadata": { + "collapsed": true, + "jupyter": { + "source_hidden": false, + "outputs_hidden": false + }, + "nteract": { + "transient": { + "deleting": false + } + } + } + }, + { + "cell_type": "markdown", + "source": [ + "## Simple RNN classifier\n", + "\n", + "In the case of a simple RNN, each recurrent unit is a simple linear network, which takes in an input vector and state vector, and produces a new state vector. In Keras, this can be represented by the `SimpleRNN` layer.\n", + "\n", + "While we can pass one-hot encoded tokens to the RNN layer directly, this is not a good idea because of their high dimensionality. Therefore, we will use an embedding layer to lower the dimensionality of word vectors, followed by an RNN layer, and finally a `Dense` classifier.\n", + "\n", + "> **Note**: In cases where the dimensionality isn't so high, for example when using character-level tokenization, it might make sense to pass one-hot encoded tokens directly into the RNN cell." + ], + "metadata": {} + }, + { + "cell_type": "code", + "source": [ + "vocab_size = 20000\n", + "\n", + "vectorizer = keras.layers.experimental.preprocessing.TextVectorization(\n", + " max_tokens=vocab_size,\n", + " input_shape=(1,))\n", + "\n", + "model = keras.models.Sequential([\n", + " vectorizer,\n", + " keras.layers.Embedding(vocab_size, embed_size),\n", + " keras.layers.SimpleRNN(16),\n", + " keras.layers.Dense(4,activation='softmax')\n", + "])\n", + "\n", + "model.summary()" + ], + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "Model: \"sequential\"\n", + "_________________________________________________________________\n", + "Layer (type) Output Shape Param # \n", + "=================================================================\n", + "text_vectorization (TextVect (None, None) 0 \n", + "_________________________________________________________________\n", + "embedding (Embedding) (None, None, 64) 1280000 \n", + "_________________________________________________________________\n", + "simple_rnn (SimpleRNN) (None, 16) 1296 \n", + "_________________________________________________________________\n", + "dense (Dense) (None, 4) 68 \n", + "=================================================================\n", + "Total params: 1,281,364\n", + "Trainable params: 1,281,364\n", + "Non-trainable params: 0\n", + "_________________________________________________________________\n" + ] + } + ], + "execution_count": 4, + "metadata": {} + }, + { + "cell_type": "markdown", + "source": [ + "> **Note:** We use an untrained embedding layer here for simplicity, but for better results we can use a pretrained embedding layer using Word2Vec, as described in the previous unit. It would be a good exercise for you to adapt this code to work with pretrained embeddings.\n", + "\n", + "Now let's train our RNN. RNNs in general are quite difficult to train, because once the RNN cells are unrolled along the sequence length, the resulting number of layers involved in backpropagation is quite large. Thus we need to select a smaller learning rate, and train the network on a larger dataset to produce good results. This can take quite a long time, so using a GPU is preferred.\n", + "\n", + "To speed things up, we will only train the RNN model on news titles, omitting the description. You can try training with description and see if you can get the model to train." + ], + "metadata": {} + }, + { + "cell_type": "code", + "source": [ + "def extract_title(x):\n", + " return x['title']\n", + "\n", + "def tupelize_title(x):\n", + " return (extract_title(x),x['label'])\n", + "\n", + "print('Training vectorizer')\n", + "vectorizer.adapt(ds_train.take(2000).map(extract_title))" + ], + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "Training vectorizer\n" + ] + } + ], + "execution_count": 5, + "metadata": { + "scrolled": true + } + }, + { + "cell_type": "code", + "source": [ + "model.compile(loss='sparse_categorical_crossentropy',metrics=['acc'], optimizer='adam')\n", + "model.fit(ds_train.map(tupelize_title).batch(batch_size),validation_data=ds_test.map(tupelize_title).batch(batch_size))" + ], + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "7500/7500 [==============================] - 82s 11ms/step - loss: 0.6629 - acc: 0.7623 - val_loss: 0.5559 - val_acc: 0.7995\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n" + ] + }, + { + "output_type": "execute_result", + "execution_count": 6, + "data": { + "text/plain": "" + }, + "metadata": {} + } + ], + "execution_count": 6, + "metadata": {} + }, + { + "cell_type": "markdown", + "source": [ + "> **Note** that accuracy is likely to be lower here, because we are training only on news titles." + ], + "metadata": { + "nteract": { + "transient": { + "deleting": false + } + } + } + }, + { + "cell_type": "markdown", + "source": [ + "## Revisiting variable sequences \n", + "\n", + "Remember that the `TextVectorization` layer will automatically pad sequences of variable length in a minibatch with pad tokens. It turns out that those tokens also take part in training, and they can complicate convergence of the model.\n", + "\n", + "There are several approaches we can take to minimize the amount of padding. One of them is to reorder the dataset by sequence length and group all sequences by size. This can be done using the `tf.data.experimental.bucket_by_sequence_length` function (see [documentation](https://www.tensorflow.org/api_docs/python/tf/data/experimental/bucket_by_sequence_length)). \n", + "\n", + "Another approach is to use **masking**. In Keras, some layers support additional input that shows which tokens should be taken into account when training. To incorporate masking into our model, we can either include a separate `Masking` layer ([docs](https://keras.io/api/layers/core_layers/masking/)), or we can specify the `mask_zero=True` parameter of our `Embedding` layer.\n", + "\n", + "> **Note**: This training will take around 5 minutes to complete one epoch on the whole dataset. Feel free to interrupt training at any time if you run out of patience. What you can also do is limit the amount of data used for training, by adding `.take(...)` clause after `ds_train` and `ds_test` datasets." + ], + "metadata": {} + }, + { + "cell_type": "code", + "source": [ + "def extract_text(x):\n", + " return x['title']+' '+x['description']\n", + "\n", + "def tupelize(x):\n", + " return (extract_text(x),x['label'])\n", + "\n", + "model = keras.models.Sequential([\n", + " vectorizer,\n", + " keras.layers.Embedding(vocab_size,embed_size,mask_zero=True),\n", + " keras.layers.SimpleRNN(16),\n", + " keras.layers.Dense(4,activation='softmax')\n", + "])\n", + "\n", + "model.compile(loss='sparse_categorical_crossentropy',metrics=['acc'], optimizer='adam')\n", + "model.fit(ds_train.map(tupelize).batch(batch_size),validation_data=ds_test.map(tupelize).batch(batch_size))" + ], + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "7500/7500 [==============================] - 371s 49ms/step - loss: 0.5401 - acc: 0.8079 - val_loss: 0.3780 - val_acc: 0.8822\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n" + ] + }, + { + "output_type": "execute_result", + "execution_count": 7, + "data": { + "text/plain": "" + }, + "metadata": {} + } + ], + "execution_count": 7, + "metadata": {} + }, + { + "cell_type": "markdown", + "source": [ + "Now that we're using masking, we can train the model on the whole dataset of titles and descriptions.\r\n", + "\r\n", + "> **Note**: Have you noticed that we have been using vectorizer trained on the news titles, and not the whole body of the article? Potentially, this can cause some of the the tokens to be ignored, so it is better to re-train the vectorizer. However, it might only have very small effect, so we will stick to the previous pre-trained vectorizer for the sake of simplicity." + ], + "metadata": {} + }, + { + "cell_type": "markdown", + "source": [ + "## LSTM: Long short-term memory\n", + "\n", + "One of the main problems of RNNs is **vanishing gradients**. RNNs can be pretty long, and may have a hard time propagating the gradients all the way back to the first layer of the network during backpropagation. When this happens, the network cannot learn relationships between distant tokens. One way to avoid this problem is to introduce **explicit state management** by using **gates**. The two most common architectures that introduce gates are **long short-term memory** (LSTM) and **gated relay unit** (GRU). We'll cover LSTMs here.\n", + "\n", + "![Image showing an example long short term memory cell](images/long-short-term-memory-cell.svg)\n", + "\n", + "An LSTM network is organized in a manner similar to an RNN, but there are two states that are passed from layer to layer: the actual state $c$, and the hidden vector $h$. At each unit, the hidden vector $h_{t-1}$ is combined with input $x_t$, and together they control what happens to the state $c_t$ and output $h_{t}$ through **gates**. Each gate has sigmoid activation (output in the range $[0,1]$), which can be thought of as a bitwise mask when multiplied by the state vector. LSTMs have the following gates (from left to right on the picture above):\n", + "* **forget gate** which determines which components of the vector $c_{t-1}$ we need to forget, and which to pass through. \n", + "* **input gate** which determines how much information from the input vector and previous hidden vector should be incorporated into the state vector.\n", + "* **output gate** which takes the new state vector and decides which of its components will be used to produce the new hidden vector $h_t$.\n", + "\n", + "The components of the state $c$ can be thought of as flags that can be switched on and off. For example, when we encounter the name *Alice* in the sequence, we guess that it refers to a woman, and raise the flag in the state that says we have a female noun in the sentence. When we further encounter the words *and Tom*, we will raise the flag that says we have a plural noun. Thus by manipulating state we can keep track of the grammatical properties of the sentence.\n", + "\n", + "> **Note**: Here's a great resource for understanding the internals of LSTMs: [Understanding LSTM Networks](https://colah.github.io/posts/2015-08-Understanding-LSTMs/) by Christopher Olah.\n", + "\n", + "While the internal structure of an LSTM cell may look complex, Keras hides this implementation inside the `LSTM` layer, so the only thing we need to do in the example above is to replace the recurrent layer:" + ], + "metadata": {} + }, + { + "cell_type": "code", + "source": [ + "model = keras.models.Sequential([\n", + " vectorizer,\n", + " keras.layers.Embedding(vocab_size, embed_size),\n", + " keras.layers.LSTM(8),\n", + " keras.layers.Dense(4,activation='softmax')\n", + "])\n", + "\n", + "model.compile(loss='sparse_categorical_crossentropy',metrics=['acc'], optimizer='adam')\n", + "model.fit(ds_train.map(tupelize).batch(8),validation_data=ds_test.map(tupelize).batch(8))" + ], + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "15000/15000 [==============================] - 188s 13ms/step - loss: 0.5692 - acc: 0.7916 - val_loss: 0.3441 - val_acc: 0.8870\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n" + ] + }, + { + "output_type": "execute_result", + "execution_count": 8, + "data": { + "text/plain": "" + }, + "metadata": {} + } + ], + "execution_count": 8, + "metadata": {} + }, + { + "cell_type": "markdown", + "source": [ + "> **Note** that training LSTMs is also quite slow, and you may not seem much increase in accuracy in the beginning of training. You may need to continue training for some time to achieve good accuracy." + ], + "metadata": {} + }, + { + "cell_type": "markdown", + "source": [ + "## Bidirectional and multilayer RNNs\n", + "\n", + "In our examples so far, the recurrent networks operate from the beginning of a sequence until the end. This feels natural to us because it follows the same direction in which we read or listen to speech. However, for scenarios which require random access of the input sequence, it makes more sense to run the recurrent computation in both directions. RNNs that allow computations in both directions are called **bidirectional** RNNs, and they can be created by wrapping the recurrent layer with a special `Bidirectonal` layer.\n", + "\n", + "> **Note**: The `Bidirectional` layer makes two copies of the layer within it, and sets the `go_backwards` property of one of those copies to `True`, making it go in the opposite direction along the sequence.\n", + "\n", + "Recurrent networks, unidirectional or bidirectional, capture patterns within a sequence, and store them into state vectors or return them as output. As with convolutional networks, we can build another recurrent layer following the first one to capture higher level patterns, built from lower level patterns extracted by the first layer. This leads us to the notion of a **multi-layer RNN**, which consists of two or more recurrent networks, where the output of the previous layer is passed to the next layer as input.\n", + "\n", + "![Image showing a Multilayer long-short-term-memory- RNN](images/multi-layer-lstm.jpg)\n", + "\n", + "*Picture from [this wonderful post](https://towardsdatascience.com/from-a-lstm-cell-to-a-multilayer-lstm-network-with-pytorch-2899eb5696f3) by Fernando López.*\n", + "\n", + "Keras makes constructing these networks an easy task, because you just need to add more recurrent layers to the model. For all layers except the last one, we need to specify `return_sequences=True` parameter, because we need the layer to return all intermediate states, and not just the final state of the recurrent computation.\n", + "\n", + "Let's build a two-layer bidirectional LSTM for our classification problem.\n", + "\n", + "> **Note** this code again takes quite a long time to complete, but it gives us highest accuracy we have seen so far. So maybe it is worth waiting and seeing the result." + ], + "metadata": {} + }, + { + "cell_type": "code", + "source": [ + "model = keras.models.Sequential([\n", + " vectorizer,\n", + " keras.layers.Embedding(vocab_size, 128, mask_zero=True),\n", + " keras.layers.Bidirectional(keras.layers.LSTM(64,return_sequences=True)),\n", + " keras.layers.Bidirectional(keras.layers.LSTM(64)), \n", + " keras.layers.Dense(4,activation='softmax')\n", + "])\n", + "\n", + "model.compile(loss='sparse_categorical_crossentropy',metrics=['acc'], optimizer='adam')\n", + "model.fit(ds_train.map(tupelize).batch(batch_size),\n", + " validation_data=ds_test.map(tupelize).batch(batch_size))" + ], + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "5044/7500 [===================>..........] - ETA: 2:33 - loss: 0.3709 - acc: 0.8706\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r5045/7500 [===================>..........] - ETA: 2:33 - loss: 0.3709 - acc: 0.8706" + ] + } + ], + "execution_count": 9, + "metadata": {} + }, + { + "cell_type": "markdown", + "source": [ + "## RNNs for other tasks\n", + "\n", + "Up until now, we've focused on using RNNs to classify sequences of text. But they can handle many more tasks, such as text generation and machine translation — we'll consider those tasks in the next unit." + ], + "metadata": {} + } + ], + "metadata": { + "kernelspec": { + "name": "conda-env-py37_tensorflow-py", + "language": "python", + "display_name": "py37_tensorflow" + }, + "language_info": { + "name": "python", + "version": "3.7.9", + "mimetype": "text/x-python", + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "pygments_lexer": "ipython3", + "nbconvert_exporter": "python", + "file_extension": ".py" + }, + "kernel_info": { + "name": "conda-env-py37_tensorflow-py" + }, + "nteract": { + "version": "nteract-front-end@1.0.0" + } + }, + "nbformat": 4, + "nbformat_minor": 4 +} \ No newline at end of file diff --git a/5-NLP/16-RNN/images/long-short-term-memory-cell.svg b/5-NLP/16-RNN/images/long-short-term-memory-cell.svg new file mode 100644 index 00000000..7b66a2c3 --- /dev/null +++ b/5-NLP/16-RNN/images/long-short-term-memory-cell.svg @@ -0,0 +1,1334 @@ + + + + + LSTM Cell + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + image/svg+xml + + LSTM Cell + + + + Guillaume Chevalier + + + 13 May 2017 + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + tanh + + + + x + + + + x + + + + x + + + + x + + + + + + σ + + + + σ + + + + σ + + + + tanh + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + σ + + + + σ + + + + σ + + + + tanh + + + + + + + + + + + + + + + + + + + + + + + + + + + + + ht + ct + ht + + + xt + + ct-1 + + + + tanh + + + + x + + + + + + + + + x + + + + x + + + Legend: + + Layer + + Pointwize op + + + + + + + + + Copy + + + + ht-1 + + diff --git a/5-NLP/16-RNN/images/multi-layer-lstm.jpg b/5-NLP/16-RNN/images/multi-layer-lstm.jpg new file mode 100644 index 0000000000000000000000000000000000000000..96e8f18548e37d4f415ff46db0b662d6a8a2f155 GIT binary patch literal 29122 zcmeFZ2UL^owk{f^DOGwW2q-8ZD!oQPnurKUFF{dh(m^^AP(h?4AV`r8(t9UJM|y|Q zdxwM?ASCzmUu*64yY|@oU;B(R?mgpZ{}wR1~D7SLv@((a_S-(UDOyFw)a9Qq$7W z;^70w6A)e@B)mdPPD)PuKYrnwKvb9UTrb_n$KwQDqQb+c!o#)VrGY?rgm{0vK!1PX zT>_qui1_jq64I-nOL+MBmk98GJuPsg53mnJKt)J>?XDaVjpj3APDk2%evzq{x#UZJ z&}j`KxrLrP`ClQSXJBMvzJ7y;_vS5O5m7O52}y+qib~2Vst>iF>gej}8yK3uc=^i0 z(#qP|#r2JwyN74MyZ0XggFXgFMSqTojr$Uxkd~g2nU$TB`@OWRyrQzIx~8_drM0cS zqqFPh(D2CU*!aXGVqtM}Z8*U1zYXY5bt?ix`ba-C~bNxnIt4{bn;3aTmD%=O!@m8_5M6;Ix9MmND+qP)vq{^ z94Yi1179+gO?a7MO=tT|hwa_-xNUm)!4BmGl5b&-QT|L52E*Wabrj)1c2@Ais}(p9 zyc;VF`_mhdWw06Waf&5u&rf+m0JeqzpU-RIK*rZ_AXzY$5C;OuEREwpRl;*;mkn{C zxfL9U8S>k^ead2BM}Ilwq$&L$x99!8SNLNY{{J}-6X|2Zr^cL6BPsUa(p;XSJl92m zY(vQi+h-g|xez_Yd7oE4VEq7fk7N_ky_;lq&c_~?qAn%Z>0i1#vBcO_q%F8;D89M* z?%U!lV#bqX$IkO5ttRxjzNR8ANK?@{S!z6cQ24!(7}cCf*RsMjlV-usQRuz56vU4r zllg8{TvMXX6?+jg*Qq`+Q`1;k)`7Uq86YFwL7LK+_nV}PEa5p zVzmJ;Kxf@(3Zs!xEG6Mh`}R!Hy&-%j8fl@0ZV_zWg0W4gE?Mp}_e?OZPKLK;W*kB(kEIK=O zqp|m^z)0}LWj#uhj*pOKQb!yJ573nN4o5rl@DCTT&cz}Wp~aT$wc-KcQ$c%+{hf4| z(0HnhYe~(#&Hf&*@KO2jk4`ob7#S1`+FIy#VtrouU6vcp0yYsLN%G8xUX1*+S z%Gkilura36OFS#e=BAcSpx9l>jAXN(@CMr0WxqP&MD}q$OS8yL&;Cy?Fp7S2`pg^! zi|gBIq>fy$d?$Iooo?Q>I=tD6ytUq!^#Thu+mO}|KsXS&5AlPSUIDr!`*8Xcbbf{4*%~F*-)1=WcxHEY(2i3~#b4>I_-u|UUhvTm zj*xTa128=f)VY$21C5Kr4r!owTMMKs-|WKgaVJ=>x&k&VePvPZ0f9YCLfh&M<)X(i$mmxN)&V zJ+`RErt6MZUhTf|gB?9L^v3J>@oiWt3_*F}_C)mmduRfIz5|{)X7yiK+Xp zb&t3*&l$~

->*N{a?5BsJ?48V`^3xrG#1-x2bs1`&&Q9u@JTj^={1k=QD_+3+^%cgYd#eEl z!njWZ>m-U*D$hoBjA)oo2n!|A0CQdBjsqQ1J+W@QIatp3Mc{}{tFPSy2O3p$luU;V z(DbSUM!xf(2b#K?r&-6!ilN?)*54L6H?iL3vdRezYL88RU}+X1F`a7i@-;|qF1e;? zJr*(HII+c#<+41v1FicxHe_gQY~+)ym1UK{@WPz==5@c9X2e=bcM}GY@Q(%3Mj4^< zy*$AXQGIs4FCk9K+;*CypU>_^nktDvh#r^Z4()P{WF>4H%&;s!0+;tlH)GAF;>8ko zqagW9lO%*vcSpDSFYgxZ^kw-ip z?HfI(G(~4Fz{xQ3$RdANmbHLZU!LGylUpN3om=9SiXNq#ncCLXkh@Il_uLCh-wOR1 zRQDWB$(Zi*eu&OB+9=G3XI|`?CSrSg=|RVqY5bzb>7{8{6YTJsv5Xg^O75Ve&-ql9 zc~pMljSP7Ifdph5#!QjSGo*2j;KG5_BA-@3FQbbL!)B#syyU9Yyn=SqDO1(=J42XQ zKK&SYYg3a}E1RcL4@cgFeF0ESIso%KFyZ{wXb8+%pgvUaozBi&j{-v=PdLv-K~=g# zCowS>2de{LHntzti35E(3H*wkFAKk@D=ZC$>SR^g<3N|-N{njK%RxUMZ>M)M?pU(PQ){qPrdX>m zmERGkxKCm;6v<}cv8fNvd4rOrd&7&0&YPBY0#(1%FaUdQHOhaIjJaXD_d{1 zUybFf8pDUo4=#(J>56Q9v7fq_Ml=_Wm%6H|`k!r<#BNEC1!22khs z%E#HE9Z+&hD3Z<79;*P%s7weBw6+c3hhiVdltb1=VEgu1l~b1E98^)Lx6C4pc;ViRw{k>NYy7&b79;bvwBTY2J33u?a23@Uz#hBVf9#cho@+V^U%0Vy}m=*Hvkn+}a>6 zyx9GI0$bLDJ9#O{FIxdcey-`*ol!t0xOI<#-yG@?Yy&Xu1B#|a>w@Nx({n)#V1y$j zV=Rr(nmAAw{FEz*`3N!Dkd5K9HQz2|6z03}wfF;fG1rd%8joy^aKhs%Q9r9=6RCyD zvTP$efn@{^WWo93NZcs@Rb4ReF%1w;7VU=PK%5X;%F{#muSG%b0q<$N=HTu>Lr;@!0k5?juGN}dM*t&x)n--yC0YP zxA$EOny(5`3YK}hSpU)NZV`J4RjY_PcS5P#=wYeaxbt`zgs7O3l3&u5#a61qmcgj< zLR3Igk?v5r|KB#4UC^q%4-XB{_#GWf?D0q__Uidm{KDAn!$_E;@Bt@h2T4G~hqlks z4;jUu)u`NmfCH6gf5g}?hO?C?%vfWtWti5u*V36aA=;xXJ zh`Xz|=P7OpPI)yo-mXK5y_NP@fD?D2sE1oBJ$pmv0t-xh$Eg~sRMH##ST4*T=n;B| zJBB#51YU%AUs)=;ie6eM`dG0Sn2^%aAlL+>9Bvje-g3-ms2v^LzefOZmji$9$waF9 z5-`T-*cz;0w4bPT%K-8H z#2Dc~T(Q{z5M2P|N!>Hk1NsTJ>@(Im zs~4$1XPkl<)+-aEVG%)^WHe8AdKa&r&C`<(JA%lT#4nY3 zB8^o!rXP!vJug3mGHsfQUlh8?SPyH(V>kw_6K*FFXoERd3OGMzS0gCKda5WH?NzKU zRIATby4A!^T#hQ?$RAyq{kfJ!B&g%W6dH)-C&5cwaa9B8=iMzlhRm)p+m$JHHD1va zFWln|Vnv?FFFSMah{P>Iz*S8e6jrm$$gu{;C0peu<5|UiM_?(PXcalC_=QM;7LA*V z&=|`8z%H{q1Ad{X;7=#Xl~o#=PSa;Mf=^#yG*Ogc-a5!;b-l-p1Gm%0jy^FBOqm!c z>Hc*7;aePev)2xA2~PFz>fZ8N)cxvP?Q`d>b~Fo-2$4ol;uP9i4-Ziu7*c zK(JKo_0wp9@MnNgJSp}6b;568aChq+Itd2?{{t%)fyDqp^noEuUzYqFRHQ>o$o<`I z5jgj=<=;~m{z#^oYZ0YH@|{>qb^YdGXS7l&S77Lem6s9(?C+T8A(qc@AbQG0<%>@V z;n-+ZV|Y-7Af&DtRHjK0V153h1qL`Ydgj@3R>PQGO_ z6Z~-P`xH7)v9_|hCP1$^VRq8LU)YVei;4$ymzc!4LpTCmrBD;|JkVfxC9;;&4pR^i{ggcUitA>yf2jj>Fa*7}f5ndG_{`>kmCX!RP&*YuJ1%gV(KnCFcvI!WQr z9^GBgBcKr^TAt$?bQ^{WoD=hyZ>oscu*7}5R$mpfGHw1eM!#|RLp!CoH27f!l-;-g z8?4zD`4!uM0YoEPv8Z^9Or7$XtkE3oN5vV<^`FgvENlHC%XxD+P)BJ0g|q##b( z2MT9NIa{6U9Q%wFKl`}tiT&NdewPGe`~ZnRI|00(9GxdT?Z;~%IK2Ch(Ei`{G2QSg zfw*Q;AFk%JlhzdQ7KEZ+F7s;>jD-qFEMI9l(Fw0IPG8Fiz^!afxpWJt}hL;mXevs;DAI$HA|re$p#$IqX+ zx3nSbTfrb;bt#24uBb0P9jZvO6Yy%&Rd+)86qFjpo0pSz#_C%7aU6!HXm()22 z7*(|2LMY|p(Mf>o9-;f(58_sYPGx*n)NNlITVxM+CplxHoiUx1A2G*Uyknv=SGtoA zh*e^0xqCp=BY7_>1Kc^zWKtExrC)wCl9rSA{`Yhcd5fR1x|h4)0vx(8o`;cBr0Y## z(@%14Wmcs&GPXQMPOCXehF@>Z@t`%c*}s)|)Xaz5;*|>IUeSLI;5LztIT&2wygW=$uBJDE&?Y(5*F! zqNx@H!RSJJY5jh5${d*J(AhN@818#|gl(@?z(AIUCqYCNu$RYQ zQCa;Nu~sb4D&k`@L$cl^el~SXmK=iy%2K$iHLES?47rq9HZu6HPCe8~mcOQYd^$gTVd2^)%@6?I2f#;zCKh z-&~T~5#?e}T{z{bP}%$Zx;lwViRDe@LmJN8?i|q_bmzlXcDid`iXK@Ns3_6mk?58& z6m%2eR5q=c!z7T3$hNU{2r>9%tMi0Hjbe1&&}5_v@iMdLcy-7lRh7Hy71{IW-(t93 ze=Nz4L~EFpy!!8bBF}&26A4Z9)XK&Vy9=bAR0N+j@80lVM7U70mK3+}-JJKoLcqLS zuHgr0y0nnzxTl7{FuCMLn!BNmttjtgy6dfV6;mBQj<9#kJL`ENYjl=%9CZ>uPJ0`sOo^-ZKcVq;q9gOGtNmFIepUyVKGH77ecuno2c8mbgQ|Jta^P~puLs0 zs`Mm(-^{z`DM)n_R3ORg{5ff=+T_caDd~XlROoFlb^2z6?X8JPeO0*TgldnXubJmlm|)+4OL<1=5ZYL?EoXh z%vq|Iq)OU%tCvJZ26c8`9j*FQtxJ`WK0@*Y3LAam`$SIqxHP%*u|S6ZBvmSn=&Aoz zS(BXuQORU=yFo>T=-7f^$-Vo(Xg|8YVxh(lA*YuBr+%~t2B_R4%>e)$T>&8Ucs=aw z@^{Jpki!P>{c#aT$vLC-E-urKS6keX7IIORh5>jAcE)+GTpHhAEMl6wqDj)+ln3=VJ*DS&L%qPGk@ON zi5G{Oig$j+o-)Lb0P@i``@&JwsRC7DCaqHeN*$ zl~{3Y$YyjB?9>-{NZWd0#9=afp0&!a>SPm*&d%a&8GLm zO)@j7Jv+amtuoa1c!lz%g`M_^W7#_=wdi&k=Zai+vE!dU4|no?IlS+LwPiWmMQ^wX zaCTI0lgXP!+)Z%%0e=f&5rA2#*#|@ErL`B>oVOd9Q(BM;3sWBRu9ObZM7P%QR-lP% zk!>I!u|IP_ZtUE^00lZQ#5j;V-OB7WB(+R|0&Z zjwgZAdI0QgCe8~>xhnk-vJ(ry)y~ggye$HAEIq?V$33D1*l}Nj!INykQ}uVt>6sQdUcT^THB?{_e$m6DLGpLJ%%Up z0#Eaul-XK?qIhNSZ+*a63(X6>NYzGBw!T+sAGXb0VoyP>$-C1ShSF&-G6d<|Zvwv@ zc7W__MYa5pI@kHdH$@WjVBnG88>Nv4?uAe?9LNTd1*hiE)^MA;smnwt*S-s>OLQyb zYJ=}fSi_F`z!wDLeK-&k!~z1w$fH->UAhDXceGfXycO~;?<^5;tUnLMr<3*$R@vJ! ztyn0!Y>g%M3gt6GUGEPTObcThuTuMfZ$C{O%i&qVvoTaZE8Mi;Fez+3RG4|wsfzvk z@z$A?Py!u~1VXb^!}(1xZ_!#={;$yTueMtNhoOKFyYTG*_;Y_t`)v47`mo)%vU;(| z?c~+zzE9)diREE?siuujqI-C6?Cc5le0=To(tGl|x8SHZzHHeNxq`;<(Z+275u#+$ z`1tW1F}J~T>#K#*o zcKnc__z*~j3+iFjf%OhmFHDI$O%!ZRp`NsEq(B)vx!WNo4?CH+`bGrCR9?6*Gd3}H z&G=Iml6wifdm?9U&-OjdFvGg!me0Uj2jWV;=x8O>cBhsGwfn0FGCK8Q4srk$bEcVz zzPOLy>dV|TkGHS|Sn8^gTUQ@r#~?27)2mKCaIWt)BLxDt1q#+`KU~?>&!5qGr{P^@U|_TfU;8d+d>{ z1q0m;o|{-?cSD5?5wMJ_0sr1I{+XrE*RfUR(x^q?$!eR?gth-7?MJU{ zdlCJHsJ;8HkM0!J>xq4QHB#PxW$UhWLVmHq#D4E_wONH;MDE3s;+1ApB**+wk=0-Y zC77JO%X5m{E^_AJIjk2YuXraYuk5tQJ`sNAynzHd5vHCcWd;RyXPrGm+KavB=Nux!L zb{Wsbp)p$tjN-zbA7Jg#FRcSv4v(Cv;#IJ_tcOgRy##@F%e}fcxqjUwe2qT3{ zP9+@oHNG+C+MKc4n56~Z3@OOjsPt_6TIez_j*ImbCqce@mori98Xd05$%!f zpO`sFfItjy0Pk^H(oDesz_e&FtxH;90&F2$!LW7C0v!_}tXyTvQ5)otN;hw14J9;U z#E}+!i|7{?dgcD_C5~w|P5t`*08ivQF^!ov*DFSDI@)uzba47-BXe^yxT0YnJmu_X zx_n4*AjjKZ05jI0Taf(VO_%=gEvaW8*|#=K*=55R$mLyi7;9WkboZ9MPALq? z#_wjn|60ycpR}`E0G%s6cnO|#ZWG0UsH2|_R$R!Vj`E!BzPYB@f4u^^)A9W-IDqou z89tUSEDi^P7zSbqU`KiLun+W&LP&VX>!TUs7DUBSUyN(1`_6WzuQIj#`O}^2ss!vx zHi!N{*LEr$j6JmER@+NGv~Q0;3RwtzE=gtjN=<_@D78yBcr{#)?)!Oo&Qju6%D7h~ zjr7woBBfQeW1HPC4sAvg;TS$PL9A~DCFU}~x&k(fqyupDDFq5Zo(Xa`I<5&-I3nZ0 zoP}dA!8u@$Z(n*Y4M zgk~oIgnru(M&<*q>TCpp1g=zIWWT5`387Z_R=#XqijdiqTd8EGb5*jPm}-#>v5-D2 zbe2b#q;E!E-p%&!*7nN#WXVsku$Oh`Jc+|MxO-}dIF04Ax`)5$!Vq3&h-lx-wlnQ9k`5cEPi;?4jTVIpsm_E8KtvxzPSnqhFgd1B^o0 zDBe&dJ)yIt%g7*}Mz;Gzd7jYw=v;sE4V>~&T})~4h!kDnBZt{wHBM~1#9nXa-}{y_ zt-{9uijVO?7KMA8FNIUNC!e=4CpE^{^^~ac^Ssa2nXN zz)r6CZjAGHt_7BvSGd7{6BDSoZC(GQTk3TZ*$?={n3jVg&<+U^*a%3HIOINY6wO^8cQw z|HZP%{%KVJ6{SLUsP$vKH^Jzuu;xl(4U&yyu!WbD_v0l>>B1M0fmy5Ee%s>51=f6q z?uN8b5lJfucZ3t*X!&yf>}~M}T>4Lm5U_FDUujUo@3nvOXnNiKixxSRfUA2-_A3nO z`7JF5qmB|*OC-CP18SkjYv9!|$>TfDAy(yc+OY)^KlOsEH*eN`Jr6uccZNA&U1xT# zVLCe|MzFe)!FH?I1Ke7ON)w&WncNgRX%dn zj7Wx7rGEM3SlhJMz?U1jo3yw?y`?(3t#z(ZV^Z33!@=h=SA^UT0;kK1^_9adbJVi5 zc(w~0oT#}E>5FV)+Fv^xt4uqi!7(GSifs_x+O0fNo-y({bwhg}$-J$sM&+TO!M(XZ zDpO5wDjrxdWfU;<9BMu6IxrB9ZrBZfl~(`Rcfec&&vovSd!5o}pH)e#1XdiVjFTO9 z3$l=f;Sjg6b!4fnj1g7FN*U&N00Ix8J{mrdNTGbyaA)5)p{T)xy+YB02P3=}063Np zKoOFEE*zZzS;+PV<3Q3PMHo6D!fGAH4m891CoUFoAiXmCGr|=i%!?BJY5p7fio?bxojod2@);_6(#uGPTOIzcaDo{l%SNR@nS#S%)nl-@a0oHt z1yzZ6;)|7SyRwQoRAr4*< zJ{yc}>ODTo$AOUaZy-e8MdJYI|NW*UEqR7!P3wm?63j5A(tjFw z){p?}Picp&dX75H(eCcO_GLjQsEf|*9&u&&vgK11s`Yg0-UYz;9SmO>0C51#zMv5V z!{&kJgtGL1xkYr3LjjEDfbz%-$cLpsLSQ`$7{2Tln`0;?dhJQ$-3`S1_PDI3N|;6E zIlrNc6_>x>4VmUgqN?3*HLN!a1iU4_Z8cuSf!?|83|mE7YkYi-se0Eh{ED-vZk2<# zYeH7DYcX{c%a5?BdJ_kXo)&&s$6zs>ia5xe$VUz!9g_MtDX-;vVJHJet#_0jN1M_QjzW5IK zLWM9AsBO}V)xmoZHV^jqg0ebfOwU6_26Q z|4CimZ(GCo*c8Qn9qICif`(gBuFC!I`BaATcg8vMAj|L{j=1P+N^9RyF;IWQ^^2Sq z@&*twht1zQX+9QCwZis^s{hevldx2P0?Ppr*OfEuFZR9Zu=!W*z#m(4EvnDYLa6OV zp6%*2L^q2)!iQpd{`@HfS&Wea45V0}$@3x>W1Ree;S~c)dN3}~~FycJ#QueeC?6<}7aN9m#Vxv(U!02Ul<#3CR`Cf+y z_mWgywbN>7P}7rK$resKw5X7N>R7EhQoy~Y-Z9%)czd(=sO(|yR_(S|CS6~6kSQAv z7!gUC^X!7py5kAJ?#C|nE%N2CWE5AsLV^l2h9^+w;;IONdu_Cua^L>enF4gD7)l9ax*c#=FY4QibE|I_dU95lC zh9@fw=`WS3E{pK?@@#te9LEc8uPw4ko~u>IW7#(2)U3R+`ngnUjSGEM)3~F!cs>vx zahp$Xb@?Vj7^JkwTJ@H$%kbaU9}b=c3}Rgx|{bL-5m z3ck%SU~;Lh3mV=eWm*Zy9N^NK3gLjB4&>lj0IbJAHSkckiI8L%M(ZILMYBTzf2o^s&-=!ft&Hs6aq(95dSqsM>`) zrUf4~khV2)qv8-ENo+2I3+wi)WolNylr}2}DG2-hz@u39$qPs%i%fOeHOLxpnY=A_ky?;`&m+a_ z&0ymB>Q`NAH(@@?_Jke;L*QC7n`?aZ($dig3JSW%IaL-5KuCnfTf$A8Tvgq7EL3r_qTsQ?joC zNs9R8S^wMZgMOkKU2lDXtrx!i2lqZEjnpIUf!z7vgy5kVgTaJc^~rBC<}GL0$Fm>J zVP)MmK{}IoePdjaXx88XZoDX)IHY1zIg4fHvikE{4oZaq z9QPk|*uSAS7)C1N2hki}p(=OC@R(T8(&M$5%MIu%uZ0=K#yk4%dK%;O z$btlS)itj-l@niXzj{7(vbW*)mKVSFpE_M6GG4Ep;chCPF?z&eD{=k0xy{E*L|lbJ zrZJ0NQ-<^rvPVJ|+y&p+)3le_iyW*Y$EJP%sBV+OuKDJRyF**Ay~6M9y|~CGUFtvT{xwQVuCbRv+^WQf95%?<;+qCAc_tS zmH{*UcOd!S{LR_YbC|H|cXYhfLSPrNFVUn~?__Q0-5eG5>nB{>s*Jpx9!EdlH zPd_AS0BP9wRr8Hl-l4#;4dSChdD6M}XX>Lz$1#gZx0-hsRqY-q7)-ss4ZfbaU$sM~ zzZ@aCd{ktd1~;sLu0nwXkkB?u2DbYg2+7&;$Y%rWzX?R5zbA${t5lu!8JS;BX# zo@yTfESUH23x=8t0hfOWM;@pS?Uk!F@U^bm{2=cwC^!@TMBjehPu{@^)J5Q|16lBN z*I4Wb6fhqWd?Xn)^I+eq%KXLVJ*WHYEi8M71C2 z*cMs*#DQ+7Ak|AWt{l-v`3^%j0Op<#2o_EwQ2<@{*X=97F~nZTlPBd+Vyhs#XJlL) zxe*q7%-Q+gV#pb6VE)9$n={{V=iFvwmxmRB*IV;g@x5ehg?7^%WxasUf`Ft=3c2M8 zjU@Gu+A|sD%7ncI)+&`AlwWVNY5r)WMuCY*nI) z&qO-m^``JO<_1#A6=@OJKduJwV?h9=)trtLvnN_7#qP!fNuNpU(>zFNBhVB=^qn@g z_hJsI4q-yaNV6=+Kuw~EY5*&x7EMw)bpAA{W+zdP2id>0_D=SuGOMB8Hv!jsvbLfp zRQ<{4N)AgGLq4cSsPQJCG14ZPV$9_N+>#*hA=lZ4TX!7%LZ#J-YQJ(tONr8fF9>aN z4tBTyQD$TII+W)CLlsSAoj;@2l07Y{aJJe{{tQILN%MBEt|RJzB7j( zwbahhtBnb;A61Lt-vD29q;Y5O@NY~kH&#A>)b?-5#4bogp2&tkcz8FGQxSizri@#YZ<5+vv=x-EZy*7G_fdo`rki~@0{FmK-~$l|StJWlmmO8GDjCaYk)-E-BdN;2Qjulm#T zszmndcuK1;r@Bs^XWvTw3(%_kr^PP3QN87@Jn#}&4qtd0;82TYer3NwzayK58}(C* zJeW{X*wHfVP;%ed{)D+P<-52e{KskfC7@E`XaD72@a}J08bbt2ne~m3>37tO3ZV{&AG9K5lbnuE3@bt_n$2Cb zjngETU%a46j(K>`i$i$*JAExS-W!=&nQrB9V0SlPJC9-Nq;mcs44p>_vt^rp6Y5Ak zqmmg#4Tjw*Wr3ei%r%Vwz5vdDGxf z**~gD_#)6L0751ON;E&6CTwl}_I9lmNO)$Wq07$CVF7{)9{@{^8eaRSCC|>cp*IB& z*}KXpElSDA_AyjfSASmM43&HJ+zW&!Jh#&zZ8?Y7@gneqPGZOGxg+@H+3M<>JKolL zbnR%wRCwWe1|JlN9S5T=@-`B(QZ1HDZg{B+&>MBpdwcNqQ5gqxY4xN#Fm|yXXj;O{ z;{f7^%G%ND%p&yg)7E4aO~?uZE*oS|l&VQf#wrbmO8oFD(>k@pOH;k2PNLc?0?BL+;{|$s~t( z*aSo?D*x!-VgH0Yz&_{|#MY!-Tpm|8UTf(;r4fdot-=nuOxSz}!SBIp%4o+!z$ojm z{Np4x?4QG%g1;3S*6{4py=u5#f;vFYZ;S41bD(dXs>MminkGl0xW z%~jbik6jfk`> zy!D8W%!Z?){tM$rx6qW6`jtw*ro!lBy^C5bEp`AtnaqU5=&@6xXbj5`?)F-h`UE9P zmtm!~uOu|(t4I`{>Fn)A3X60a|YB1;G?stTTa z>WSs#tA#UJIoKe8jGypt89y-Z9~r-bS`c4vBPXnL7)Sw_0=lJ8r24ykst=jEXfMj% z4CA*)`dT&tTC-o){2$Q)N@PSpcn@)|=T-MbK&PJa6}~!9Lo!t;qwe+6$&?SRJRi=s zshWchZ#ERI^NqYrzouCd=0RWI!s; zS{tZ)Q(22`SIysd@z_5~jv0cw8!3%!zveGI%#->gb>(^T(5wn=hdnoRGoFD_>kEs@ z2BP75cnB@6(TT62QP&i5*ruVYWWS>Wc13PpFod0A)SDs0$V#D^9o&c_@O%|#7ojYy z>h`)y^WA+S(<>J165QGta6UK)USaWG_F9XOTAKC17PINc2>FJXM8}(7EXt$19hAYY zx{WzCK(UzD_Zyjx*@pc&*Xs}hBawV6*PKW@szY|3e^*&|xRsSMA5LkdtSoAH{eAYa zw3b}9hiGStd1w2IlqIUHSeeCCCrJ6N)BWd4lH+$5DOoolz!we;yp zrK-P}dUrKjHd~vVGjh0DFPR!j+PhG29V1>nmB}LBRvO`SIh>JLdYNo3p^vfF`i{1G zW@cl*iS4*g`FLTf`Dz}N0Ug&idk6hoT8GwLbJ$z4P@j9zNH^AU{9(-Jp4g!-R;|z# z13c_i2Qj@8>`&Of9h4HbO+$H9*#8J^vmnDftX5HsoQ<=v%@htdr=<@{o6OKEy=Uf~ z856|qBqvee!=PA<+>No#vU>Bx3gPGa!7!!loo!XtKJ45< zzYgeEjQzu*e;)@buy%niU)ULAo(Par!OC=8xB&e;3UuQDAoIuu`q%t+g@G>WMxQ^v z!REIheSG}SA$$KvETIMx-nj~RDzg4vh96y=kT| zA#Hmf*u~IJ5f{;-q^?mSOgR ziQuW`q*3$2l0UcOU<^oP>S6uOA4-~Tk2m|~(y0!3en40hzk8bjUw)ttYk|?`a}IF3 z*Q-9uy54||rY?WR5kC-vI0f@wp4566;ymQdyWf|DLNEA2>8z#CVBe;I()F!bf$g0g zvD2%H6vxxEhJ@2Tk0ld4TZ36Hf4kdl<|=zs@#w{?IeuFgII-_X!5WSdWL5Jyxz>17 zBQ;vamUSl`5TBu=jJiiVm8GJahHlAt~LMowt|= z`NyCg&Nl`Ryfhf4f`05k@8@f#n&jk_KXZ_zlv88oSD~Zj3=2*N>Bbxo8(6V;{e+6Z zdX07uNWsG15OgXs0SFUPNTr3?Q|QYUStMTX?07@0teMZ%8ba?!S(#sdBri(Qf9(V! z;xJ2(j$Q(AN$@fJHOeF=tP#r}+q~8??zz$enRwUJ_;BN+_0PQJw_A>@=IKK75wg8% z`sPWiD&__mAZfq*ukf=pJmsuFwPLalV3vGkeG_6khyBK0PfE(uR?h2Y&+8B;`xE@$ z0p}1&05;kpzr3*6#sbNo;k5CI80M+gcXXk&g^us>nYOg_ZtzhZ#UCK1umbQApg)DK z{SRRT5GMYhFFUL=9w;{&7Xr`{^ZHgI+5_mdeg))0jQg?gY|@JmT*&kdOB{$o_E%;k z-Ru`G{dFe%A9bA3rzB-M<^M$9;~AB*vM`n-!1BivuR(Ws7B$`>ly_7MH+`?6p-b@4 zvz?^Xui2z}&t(15XEp4v`VY+#n}za!;}6MawEMi2xHOBs>>dDso(crot3!rov+j+|^$_TZZN_}k?D*6a1 z-d?LLE~^VhTz+u%V~pC{$ZNNSqGlf7yDuB?XSn`5X?g&@itf0N3~q9raaJEx{PFC{ z$nB+o9xkS-`j90q@gur^@yD5pjC_rqC&l+P)25Svlv-ymI@it1968lGWL3H6mdhq< zxnmaUWco@z2eOw1e(zrN-UV&z+r6;Ix+%6QBlV8$n<)_D?+zt^yCk51 z)rnw?&i~Wil}AI}_We=OB4kU*mZGdhmTZGULsF8lD|ZOV&d4$%dyFL%WeEwHLB>Rk z7<*;kRb$4!WhToEGsgStzMuPf>OSXr-upT4bDnoO?;pQ2bI$y(>&)-^?w{}Hx_(AY z4)EDe*csKq7XUWYRq|5>1GI@_z||75BpUIOnxb_hlIm}CaoXMJVh!Y0oNm{VL+nE! z=1I|qlrCxiMWRSF_V|LbRZ+CkML%hiN?y3U81Xxr<7r0YvZZ>Ie9l7Q}#Z9bd#mpr$hFmMoa?+wle&yR4ueb z7i+uFK}3!NF3`*`@gYU?D=tF4DM#XJKZ;)pOWe=Km5dnaPG1an!tdfVPP zJOt&m{>VO3_PLinG8vM4dXJnp3&9nyhn}HdM^n!cc*xAl562=gW5jA$t|xN0vy5V6tJ*E+0q7|&GNQ?z zx9*g?O0*vg>1_SR%K8*Iz>80EML)3Z&O^j`XzV+OZ2N0RiZPe_Y(C3n*x?mZx$V*l z^*Vg49n^q|jYC8GFFxnH-}y(^AH5lY4k166ry2zNd3gAE6uOO>WtflM+uvs?`7K>+ zc1gM^R>JItPOI9gZQBo3huzhebdf=Za9c4oN4EykFf4(tO9XT*1%kZ0VhVkHbWLyH zt|Zt)U({IfSm$i7E~8UhjHnPwkS^wUCkv2Njub}8(F1bB9I4m&+RzA-eAG9o(h@*IWdyN(6->x^7I*9_4INIPW!56VPCVl8z zi0k17v1^Ns)0YiX_FtErJ?cCyDMYsJ$C;PyA1Gp(6dHq~X&XExx?ERNUk3$d-v<(h zGkN&mdH4`|q-Dw2-&#p-ebyo(;DoJ6A23(2ZQulW*P;L;hp>FUB_RFKX2tC>)4R2A z)x$=Qd=2a-*eda~YZ-`OH~c zSj7f&x=S?uh-cstRk0`oeim%;ANuq$;^NtEm>(VQk2r~d!}&oT1`p1iTiD_J`c>0j zi_h7~OqCeT8!4uT62~GQt*UCaKGlH@&OS0)dI$_FbXuPoX7$|o`6#{Sm z8C1~>opsTpYE$;>%|zlC^9w1b%oJ+V`S%F#D@Fg5ERKe|I}|I?->>Qh3CDyUq8?f& zL8DEE4Sxmoaw4!yn!y|d+G7Fu-}BoE$@K4?&OdW`P>l#fh%%UW>&A?&t#g_})3|m0ebwWh$sbDU*x#8q z(52j2(cLlH8)Ufx@CFe@*8pUy2=LvJKz*yL)I0q35&7}sMt!AQDYp2{>}{#D&r~tI zm2+MVq+(#(C7mp~1(hk_S zW2b1BG&PXrqmoe+-tX;Xt)_F$a%7+jj$OqZ7gn#(C=H8IhGRm zygJswQqIxE@dk@RS97{n!C>u4RQ98%S*f41;~>)Fe?M=_t=7{0OWwu=C@YHxU`A7H zKm!5@<~@;KO5-pDLwoqQv-`Mh{IKG43OgVI*uEElWh{grf2TD-S;;Dw()35QB(zBC+UVoK7bT62+0sYjrS|HxKe(mvG~zPK zsFtL4KS0q)BF%l~+@}w*uf5h)TTSW#*2@73z>;B&XKZ&&(MVl1@c~AUee_k^Yvstv}u5tz>%Rop&dX*iOe7iZZwkh z(~#rG?YPMg@yaph_s>I-lnAE1}j$X#R0) z9#kmyJFCaWp3)Cb5Fw$00sfEnn}!ttMiSBk%zrWhJXizo z#^V_KLJU!U+rfx)&+7#lyej|!h>vf34L=D`w(;LdFmm#=`G!`+U*{JwE&U9)7w@`_ z(gzGS2%rTqw*b^F6>9HK%ap%uok8EGgD0W~N+|Zj?W>v6y$u9Afb(LA@3!52N;w+ zXztJ7%TGC{CJk`WfA~3{!6|LBxd`4~12Ig9gl z3(7-5-dGe+ zijMIutyjEWQ9ZCW{$@-tUcKGi!b$h^)!zFD3;gyt)Nv5-@4!*!4FU|&$Dv25m$?f& z3P0ZXrDqwsloH5HX?d_I=*gDW*v^8j`e_AFveXbl5R^Jic08DR6I|)lzH;ZwpyAVA zuGTB5#tZ{vJ-?hER)BXZr|Ly2(Stcyjto!GkR>J;im*|Gt_*@-fAyXFYQA70JC(-d zt#kDeXu^zkif?^E%h~?tDF90634jx$2_!SvXzJCa0uj6XPg8Tq*FVStLA_80o&{t@ z=>bBA=A($kfuN_)c!z!-x&zvmRzzRVVQh&I3@VdC?jd|qpVxN(G!{(Y0v}+~Ut8e_FKs(TiIDLQVN;h5xmoQUYVP)WI4K_{;49wi__fpVSw*g+-dcbsPcv%g&_StRD)_aBu|CH zQarL+)x4i3<3DVam`y$5ZoQKEnjTAMo6y`%Kt;I+YZ4OVgaq}_ayKe5lJ5~dhuw2i zAHL>Zm%uFkY{w-6JD>=;yJy|rtQqS{vstDd9#)I2s0~+Hhb0r8g&T(#jYcn`6LPEU zlOy$R(vPa)W8~@Zd@7P+f*lAxgEA;6oJqK~aiv>uK3yg2p{em1b2iu_rCkgf=3@td zEq~J$V4nXYqW@XH|F>F3|Bqnr)y938aMedCFY|OXJ((f4@xRO#tVr|#QKRFp-M$}q zff}V706{**uK+$WO8{?Q^j&Bs4@YJYC6~E>$#cz|jg>vCZgQX_NH`}C7EZaS|Hch{ zUM9gs#zgEup0>Y~!U!nE&Et<#m~7Spg^7bR1Ak@L4hVOaYQPBJ0UZL9CaKXopwyB= z%u>;Wx;pLsoE+VYd3mP0>Qft;wVCxejfFdib*Z}dUAInOF4cwDAhejn$q=8%9vi4) zA#J3hkQpAll^whTu3L_2&9@Vh*o_+^};59z#{T)!WNHduJ%+8p2 z)yJ78#OSlSFqU0dGB&r7EmdBb0mY_oP0O|vBvw?dDndj}mzew=SXexn0{}7tLkdP~ zwG$8GI6Rl$Gfz=7b@k6oD@?UKR^w+LCjRMCKT^oj**0t^@1)75QqCBPG1IUu z<_g+E{u#{k_j|B6tB29{Y>ZA}a6_*We^tWOU~6Jb9u{M!&^QCj-cXpYw+sNy?um$q zh}*h{H3O(ZsprX*?Zi-^FFk<#8U{Qt5689{JVu17xdU2~RGszj4>V$o$Pwh^Y`-`Z@j z)-MrhTtZs zf^QPDo#>C>Lfjr)r~+O(@%_aU|K$rff~X2ZPc0K?w1M~UfELadL6s&^v7ua4dz$gc z7ufseE%1XSRGrvyWsKGInz?89F}q@4gS0fuFuxNzUF=?Q^KciBO-U;QKsW$uI}m;O zWkQwYMme9bX-F>YcS%_ClbL9C0}|XMGC1Fa>x~~by*S3<=@S5`S~RgGJ_wS^@An{CJx{;?lH;|6bRW&AOuH~_0u1p;a*VdpkZbiq* zfp2?xsglzeX~LLNJ=L0c^9l?GbD@S)5hofXm=#>$#`?}wAJ3iC3<1*N*OX|#dASu| z6TYH^vyaiA6_$PX`!)}}lJTV}*F1aTtj+6=`qrd#{H;a4{!5-xmJJT7O}5+)=s7nT z@Or!J0H6Bas;ro9N|t+FWNTqZ-%Va`c{X5tuou7*qRZ!S2icYGPW0?mr=1z4W?0wM z57G64N}aXQ!fY3329a{IXz9{(1xEE3#oIeCrAR%09ZC)^-p!qT&RziC@rp4BrvUTJ zvqA$29Q>}KyLyPdb(%DF>MW#>7~qS>)RRbh&JG{oQ?j39B|i;wPRa^43wSIA(Dd!8 z2;B^v%Dtf+VuC4~+`fj^1h{))$ZE2Pjf zMG%0`C5tc$1N_!)gInD5Ttkm0(*&N#VD&Fxalc@q9}{h|&|6w-d|Ix~^r zC>z&PRrj_ZKiSd{cGlPp_vKhUL`hb_m(;wGWNTL#cgintxtU9`>&>~2;1tE7w-scX zIGBXg5`|9yh;H>&U}GMIQQX$dLFe27ZFGVr(D*nz@BmO79kqp-b~nGaY$aw-R zt&V(jca%b)1#h_>4)j+@I)7dfNgTQP`V0(gO%|@7#6N%vOQt|pB4=uoK57;W^>u(S zrcrw7VNtLNe>S`NBDha(d=7EUI9k z2NL}1rmW-3al1S2b%e+oo5ZZrpe7L#P!9YePd@=3g^!Jo2W)_!{93K&D2KS+{f*%& z{V`OTE;FnpM}@uT)gTkAXG!yAPqXCA^~~Qo zldrZzh0G`Q-y$x&eO*5IkpjD3z^DD6%Knmr2cx zBqUpisWbL%I@W_R^E;mBvpnDD^Z9&#fBas*UWYm7zLxvGuKT**_jO-$6l-Z=v~RcA z?k!uk>@zVwcVWwx?Ew7qZW|Ze@=;w{8~)hlrDv|UWlI@x59{)F`1!ZM3r1(Rl(Zk1 zgdcWYH?|MlvgP=@&7a>Ct~du5&EVTV|8Y1R_$D?s*4Wti!-o&XIfx|q zd0Ws03+pY8Z-u{XF?qdZi_{jAb9y$|ftjJ!0N|U%yUVh1I$vJuc%Bdco*x;NpLQfc zGw+Z}l990-Ptti8EmxTvvU4=9S{&V%mH_T$gG`yMR_|^9=<$DlXq)bT4}6L~Bwdef znur9al@RIodFx$+$Pg(S{Ve)eLltyH(8XoI6NjTkyQ3S5p*FFJsU?2xP?|j2l67Ta zLZejWSt?i>=XG7J5xN8Pxu8E|y+;Z7x^ieXC(i|~smd@~ury;HiG2I^RhE((d$D`@ zPG_rVDfTm#>Q0LijJuIxfc9%HmB2#?w=@D5K&VCEYvajWl~Crm`K z`-RA-)-16z1k~@I*KKev?Ecn!0KS{#;l%zxi~?44JDT5UyVJbdL=OldicQA6;!$FL zDLHEFYu&6*pGt2TbwF;8!^eqsqU(U}l+%;jnGw@>mu7;?cYRLWa%1cFaJD!H!!)PE zk>o?4rqvoB8R~jfB_9err;17grw_MO8G`mg0XG|=7oyB2;WrZv$HGB65}p_gvlz|okYODI89p%<&eFCO!BtGd5`ma-tK)RQ&W z-88TGd&nqC2!6snX0`q#&{ceYj|+YDoAWbpNk5Juil9q-0qz$abQl=L$WY*8_GyNE z0)Dx+izrGqjU0LkhHXS&s_g5HiPA6C zj4wBJ4TvHzffgl7k+@@w^0fyGX=)qZvjjv>sdb1sSlT8^uOHo@IP*AA%x8?F~ z00$zBl03LcHU53Gqc7zPqj3kUpI*@e-}=(d zGyC<^HYcal-Zp5$fk~ZsPtd(VOd2R_S`W3_csT#%A?!Zg5hu`R!BC3PMa{F=nN^_{ zxfp>TqfrY(DS22~(4cmjnv$1zq9k$)4==q3Qyx~UOy^!7{N-WWyjsI!+8TbiD6hxS zRp(k!XaTeIA!&8rmqrbPNawWj(VnV63AsUe=P8Ll(Mja#kplq*CpUs+v3b5Z*ohn9 zMfF9y66}A)u3NhH{Li-!OWNY5Dm*Qz8Q|UmsQ;^9TTrb_tL&uE{`vFq zzmNZ~THyQtUlmPyfTiXAGZ}!WF<7pBRY;Ahb_-};uyiODL^0z#-n*1t(AH65hsP8P zp~&&5vqLuEVl7l;#mWFC1Hm>vMC9R2q_FPl1j}PNrQZo$Btt|UkbaV9`13maR1bmS zzsQd`Pn0qvg-MOfyL7yff9qoxVNN3feUAQ6MG!@d??|uv(Tu#E-9aT|sx=)_^@Pw0 znR-Ixi`r3&$W1?Ix(H(=h)3m@L3*TCs?~3|hsgU9mvd zF9UrSFen=uDe0;aLiNSbpK;k133eQd>zHd0NdqQrKocl^LLfm~;h0owCiqpfU(SYD6^vlWw5ErnOxK&}u@o zxI(7CFn$^-PVMRksvJvNiXC6a<#kcXhclaI3RM}RW?+x`w7@j>z9oxYX=#6Hz)qhT zheA&^@6if*2)n;WrSCx``>igu>mIPM=BpEGxU)e8i7RAZi)^Ou6@(4u8FgTq;Q#tjJAT3B9M&YSlbwHQ3j0?E<28xpAv>JnpHPD3S zz*5^qYF8^@#K3+s3Z+_U>l}?-sj;qihZ(D&*$+kZua zs^kL3+0_F=rPx1jSu*GbdCppGb>X@HlkgXw|G3S;QsL|Gs{YM}cLn?UFXay zh9QT_^Ah6R$5%mdx48+VQJ9~2^R zZwcAqD(AI@*791H^GoJ6??XPrZ_Gxm)7uWL>RpXZIHL{l5wR2*NifDp8{p!@4@5|E zSLtA`$g$rJZXlr>lv$A&Bi0Mxo>{lAi&Z1%Mi=dp{|Q<4c&+o-cRjQI#r2*SYSe>k zSz2vobt>s-1;g$sDRwcHPRR8HM-7{Q19gQ)bmxFpy198u#;eum?uUhWIqg9F!{H{` z#sP1?{Rs7koa#8?bQiQnqJr)#Hv||H%8S+?6p85-m-7T`&i32Z7vZl=)%NvZ1l_w2 zvJ#Mg0n9?9TA@w zS00h15KDwz0R0koMSvnm{E;%e-|0#+<}9|{**B;;Bnnfc;4#)1N#FD zKGH4)MmFca6b!zjz4dh6Ly=*h2K?p+^}yM1dkY}v$`vsf7o4+r0j4wz`m*ioWo8;=|wPNz1{=C6Ou>PvLwc`YRgmdBGl7xL|K zN};B-@PWn>o~je_N;Yb zAIibfDJ-2T7bNk-q&Sx?E9pANe*~^{pSOa|u$+C71HFG+w`{9_q@P46YSn@u3^%WynU?WzzM}qXN{${jIrnK4K1+ezf3;Vp>Zv!t(i@w~&q5y-M z7hfxGYWUSItBtkJo9qB@*s6=>n?4<4&ebMa+lsULN5-pAmLiU&lINO>-bZ6~0Y^FI z!CEi#heK?_5sg!I{uTYRjXj6Aw9N$>RqKp{5vMiG+%JtTGgCK5rV(G-_e@)G2bgWI z`KI63m}NAL6Fn?zJ$5VLVwWN73ZRrYSs-$=y&}HX0axBVh3KL?5HZa!)M2;Wf&zMe z%sllUEXpmeUv@QS{hZ^!j5RkdJazfpZ*lz@Xm(LQ;qm8kEHdbA9~C4l_x>J z6QTfmdD_thp14ia_zQsK?I*YM5}_*o(%~mh8PfuF7BEj&oZZI`a4EPqMRetjn91sNyJX?eA#mE!$h$})IkCGiArl(m~>~k|2 z!r>MRB3N4~UP_b>nM@?S($=sP)KR3PbS9FWK=CTfi=z`~`f`UPB3RS$9T!nTh^3Oz z@WnbUn9q;uRp>)uj(S3=zA>^o1eU-AgHQLPoTIsHBQpXtKsO(fycw*gT1$FC#4CaQ zyTL^eBa3v@ajusx3_HjAS}wK*<;AV6F7?L=N9eaTQ{wLtohBJ*7uIQ>(^x_JAH8rN z9RA{nv1{E8_XrOWc5faLcJL&F4~2kz($x%mViAntv8z8di;ONNg`_1w3n6ukD1^~W z2pOi399Y2-&{(E-k@KM!-@W0je67yM^Sg5i-gj-SE<+|l8;}=L@t?0Wgt0k88eAQ^ zBfJc>lKeNV$;bAGEks|$RdgcCA=0|KOA)m>viH+-qEWp)?z{7hnbF%}1@(A4S6+kZKcy8p#{Tx2W#Hqo^e{k zr~_H=0*i<1vQ>@-|lFNTy`6 zk=a3Mhx z4wYHqzV?aS1hb!YADD{rSkOrO=GRLy)8!-Mz=ZNhFf8Gi7%!GBIaQYosNqGZ$_xt+0eQ*mKKmJRB)@57p>X_vn|8!G0=@mX#MZK8_3Fr$qBl z%Gfub-APfgV)dS9t)0yCS9EDONI42l63j1poTw!;?2r%pr6O+t?vCIM1otzj+PX2s za*e&3UQ{o^>gh@#z0eAys93OSSDW9xzOWob)C3L2c{8RAi=JBHkg1idQ)IE-pvn>C zRLzthr7#(sD#Vwl+9i+f*Y65&&5)J^_pSQ-5DanaE*C|Bz&e5l(@+vDJ;HWK4q@ua zxKvuiOHnx89)^u4&%tYfq8e$!p7@ra<;X7>S&3bew z9^oJ_|EATCm#ht5h_Lyje^F*G?|Xd9A-^N(X0k2%@=li@_en*D0e<7}SwfrxvxX(< z3x;X+e9c}O{U0#&M<=~@11a*zlEzDcd55~ac53VZ=aC$teVSp|fMf%n(haYrj2Ip# zoQ-mRQ1P%%?*WaTYbnJy_9F|&3JN>&`-eui6`YR>O**g?7C=~`5vX=HVSKcW>UF5E=5$)8 zACav3I{It%4kQ;x)`0>;o;nui7Ta9e7pohh!soO6{ZKOF;vP|#a1rpg*iguAWrL_y z7&hzSoU+)oT}(yk_DeL%|4pYK_qe{-UO-qevPSjGmDO+Cvv#0FDKbbYkdYFN(Ds4> z`4&Z((Z`D4F=ow2gsxNeAguLQzZBkqDVFJ1XLHV6z-sPd>4l=g5evyMKQAEAQMh5S!F^U67{?rEGJo<#wNWVnDeJJfN)@}LYH3at!VUl#_PL&t4)a#@WT|gs7{;VL$Tn1^R&F4xHh(qzA?kccPB|uxjbGOPW*Hp*7 z;9`GE5It6Qm4lyQ&|~DE$g!`}XSwN^KTb|?^3s<#TG??1Hqn~pc@X4-hay9|fV=;P z{Fo5QLKbNV8zTr0RA%a-v(CwTLCOifsoLZi`9lK#wE1prq&5)SLP6y6 zsqmq;iDWf$?*=L79>^k+V-Tw)x)cF{d%=qRf%Ne>tXuZZ#b-mXbe!8ku-35zwwqWn zdsM*)9GKru5h#yd`s4oUE_hZh>qs*dT(k}v!qd2OaMA?kk#piheMZENmJj&@OY-oX zuIf8j^j60b|Cr!anJU%*oNiZexlsgK(=vkuO7WGu!E?M_$=n=ZsRrgBHG{ePh|%yk zRqe~c#tvh=80-eOiS(QFar(@VpIgr1E%$%6$$i>-{`1!Cwykbzd%onhW@4Mg-K37j zs^Ht7CfXVyKrTM7%#nkcfL==idalw%9#B;24sy2k)Li95fzP*n#Tw2E760S{6dRv( zhb58fu1RGAMqNF@NgZDIrDQSz{eCT;`N_T8Db~ZsKXy>D6l<132}!R_8-G@N)I5si zL@?4G)%o+2njwXU9`{JY8cd#L)~-7402Cu1efKFc@ygg9Q#I}SyhyprS%Io{A#@}8 zY&-IU0i5acl9PYu7Q0*92FH#|3^74h$gR6C3|{FQ`0}*`ayOhsXeQa)+K!(1X588T{9-zn! zi-Wf_l*1x4VZ{x6=(ndN-a0S&Z61dB$e~s((Q7UiFcSha*s9y_FWp)WS75Zr zjO7{0NwCYIpVJ0(1kk-P@7CmZuNykEjSrCXt-*V-k=tPrBgaQFZ~P&M#ifI zw$*~_WhZa}_aI$ zEnSwioRhyW2i_cij3Bh&O&KFF;I1n))}I#boA=m<|0c~AqE26E;{wp?J>6Er1l%s< zfHYf<+JPSe%6Mw>RQdyE4|x49e6BtP_)sG_p9*~`wJaY?qP`rVj!on_^^`@yqxAEg z_Fh@@Xp>fIQ8{7UdT1^^(+r*;M^HV?Ii=0KWEo47BlCVVR;X>0QQoE#_5TI$ z$vpjeV`{$B{QCfo3oA6hrBRVCe4L;2Ojeg?F7Z|6&v9viz_FYK8Le9oSpYp$<~W~i zk}FWPbk&MgT-z18GxuVZUHt)WAA2%&|eouqg_4`xsK@G^K!@Qgnz|os~UuTGPzA!@RQC}QW6B%S1x-LEQ9Uxs> zA4uKUwh+V70>d6WH7LSWDkz>Od8>La7w#HJUn`9~zM9j1NT;k;D&KT~Tyfyd{O$Hm z+1^Y0n$NNBQ|-q67neGbot}rN_i7a1FJmxKcG-F8$CgzCC(fmOJwGt_>oUCY>q<55 z+m#oVUHsW;`Uc&$Kg?4KX)_fVEFd3Iv^s&=4aT5lkYbl51ZJoFvxE>GXH)mndDNRj z+-vcu2JW&RtwS;N*HC+ZJrq$DE4O*Rn;Ok=y0Ith<=$T*Svx?-{Up&CVjbB`BF>H^ zJvvmX9JHDe?knue+fgmn0KL^}a!6b)`d|Wn^6lY6?um-{q!s1-3SQPrz~hqEpsZ1` zw0NGLmyfqE?}=tcX0a}{?|_%4N(_-e;V$Q*=1~emw{;4Wj#APK{}8JH?&V`uMGSI< zRCavmON^W7I$QMMi4nZ%_r>x&;&f^#E!3>sK17L*GC$WA^$iG&-7$Iw%o zC@h9z$0cv1AK46aNv7KZJ~ao2<-w)rK#mLU8?%<@V)sDSqld#LlyKQbvxNsPtRLWv z7`TFN=-;Ng3I-v8B)s7f^0k+11W#Jn=$fL<9+-^p9IJ$De zOI8#h$-8XWkg_1G9!^ux54m7{I_Aiui)sdNa$@Q(Zx`1QTsTcv7i^|w(U!y0V7y8O zK^o9pt|ZEXEfs#74ws5z`r|Q#(NDq=dbYiwBpQ#a22G>k@+J*zomjXqfes(Ak zaO!E#=n8lxiunz$JIcG9K;h1A%t++V|DM&o*o-oG=%1O+^5#!T&DsSfQsQr(e;Kei zmtQzgz7{U>pi-FqJ;uA!NEY$WcLDNFiIhgtKhC#o~THe_(k;msm~gW zW$wFix>CLHQOmf5ebJ4J%E873{l+SF{XOQ_vTk-T9Gt{!PZrHXeTFbdlnT?^wZdA| zfyX>PL8eDzITOw*rd%zLkk6q$FSxQ3LeV&o;+7#E6;D>|j)jRMkNEJCOH0buPG zh&ErEOp=b|0_cVRS=!mG@{k{6AMD)?16VH*^)s(f?Vokye^h`Ld;TJ34MWwZA4wAU zBcC*2316Ui&tEG=e<>P0lXx@XYY(U1Df}F;PwUpd%F%Q~R-Olg2aMs29xRH~dcBU6 zgEh6v8P85lGz9Y?;D-zk%&wCuA*3OAOWjntu9RsL1zn33e(S_XRs>fr1;T-+A^^=u=x~q&_%G-z%*62)h;V%?g?)%#;9;zLJ4Dz-a!uk2-_l`iF|PJ7dsd*&1nvR z>}Jb&(JIx)9Hb;0*cjzT_9Pr_cwn-X8^Nx2G*{~1O+Jf5mLxZip3DT_U&1MvSHf;^ zeEg)p;XSJvxRj~C5b<-hhd$7z+qXA7(xRbvF8+1n0s&*CjApefW@ z-%u$GM(74-!Mka(^xpY-ST<-Hd~Gb=#e_RQg_(FW*;rgi-RT zac(bV+&N;nf%QxinGGLch)%ts7n=%Iwq(V0h^1Z-XXJ6*9O@^c(@pD8@~{_A=arfY zwYm2c92f2h5d@omUutm;HU#wL-UU@fzgs2qficX({fy^{&jj#OK&2X(|JB%h5G|A# z1y8N+_OeyRL+Dc9v#9m-!YUeWpa7sdYCYA`#jaO`kCZ1aW%u@Df3>xqgLhrd zqpnbtG`A>7+E5S$Y>~4 zjhWO3f4*uS4@2|8iz2xF38l5>x28f$UXGT*)k+sQZ7CpdzwL-6n{h8TUq}!smBTxD z!AYcUzmntx)hKyrA}XMD6Egj*G%@Gy!<*{l;=OCee6u+9E}&31#-@rZ+d%8-ysK0s z(4*I>^Y%QPMzYE+&}x!rPAFvxp%bG&SEwQ}hUUYMe6HQOc|SMc*!5l)ueL8g&iU}$ z6|GVTrA?Tk~^h|iu-vmw= zWB+SUe{1C-N_qstbNRPsYlWCROI{RAP=tpE$7UXk%Ytx4`QO?%i3#yPc%>q(JvyY% zuyW7i01^0Md@0KP7PRm28Oc*SW9SzMK1ef~-j3#?d}}Yy)1rF8v06Z} zrpZL=0BxqoB{;fL_z|ggGp%r>tH`l!IVb-6{-Gkn_RXI7>UUyqmjY4?Sp|H^@hf*p ztZG2yK9iyrjKfyIYyZ-U+PXhkK{q-ZQ>K0oEr&cYT9pUp^!lk6|1!t?v%@^O-lb1Q_HAqY28ypdUb$ zCzC5Zrp?`*dVaJ9R4D*tZRR&y4d0TwqQ+e!$g$k~@$o0nuM2srwvpxQZ!}48N3AhD z1BU6Zk6zm-89BNr3Wlj%j2S%N`K!Tiz~#39XJX3Z0_(uy{wm!`_?W38gJi}{)GL12 zm^*orH}s)+7++F?ihdib$3EZ*$)ExpzA1bvLEHC5qHWr#2`}h#@A|50 zWP0X?x4+wup;Q1TUjGgwR|aJH@G5dW`{mad+S#MX^3T6ZsEG^E&+<@aZ*hJ-xE+H| z!1n0p=HG#`H{^sIMeRFLBV0xJ{fHAvS1DlvNLH^9Sjb0y;AGu-^{O_`af>%N)SsP{Q8otW7m+4llPnk|7vNtt0=FM|1V^GiMOY&TpTWw2 zuAAXpS46)6gXdAJPE{MyF!2sx%eG>L%sjUZ-IfBCTo7;|bHPDoh=bn<1v~H7^e~)b zJ7zOAVb;3%s{3brz_pg6@Y=wFE#CphR(mw=dHDCjFPgDXbPT;ixpCIvyRgd5vVMTE z8LP&uUY{T&XioK{!}Y(i)sBdaQ?M)T9L;MC2bL-Xfe61MxPgixNEft2WmW^!f4Ohs=|$X>7*3T8&DmR*+>a(5qVI*FC@_}D6eOp8&2awR zYpA9&UK0702P_mM-`F>vDjUi4c0Zt7E$2dfcAm5rsX!?;tS=8-mjzoUJl+R>KjsZy zY}3I{X>H+y-P&H#m`u^|yW4fhW-7Mouzp6K#g_!ACuYUTB`+3LTSy7{+Ejt{J5$Z+ zcC@3E8r5fRnVA*_s5tc~b9Fstny;38+LGT{^(NnH7=lyy$b9#3sfJ`e`X`B!#H$0M zEi>|#4dO92pIw>aV}*LGvw&%R&!BEe!2R4^Kz!RLBhwGs8cW_u&yO}`LvQ`wy$$`#Sf0R zu8QKtD5dAM;opB$JO5+9`AaKohIu!NfTiGzpl)e)tbPR_YIEyTIhyIP4}inLLPrvO z``2^AUxfdC3vX0^Jg~&$%d5ulQYX;g%QWBE_oXrwH7^p5;aD1a|3ZOUKA> z1Mf;@^8UN=561sR{6ivUy!6cfmj9FGKh*xMum3%S$YhSeT~-oDCKH~&ZEz)d(*Ra7 zHT*7N+rOUwVGt1uTPpzX&VOOuwCOK4|5uiOxC6Fw)7t+F%Rfl4$Nt_COMD#lwiV8Z zdt?dO3b2PsVw~%-%>GCL5_t!ELxsEWOa}(o+>-N6awd?V2BrTUne6;L!~41i+-OZ@ z{v-1sDInqfa(tzbpZo~Yu3d12p@0VnRguf=HKCN6)`uo2OTsb!92%*h;IX_M9Owwk zGpOiu7BFXCJTYY$&YAz21xbRTqwro(YFv-uhv88NAX-c2!8-{;n$7$v((u8RpF)0u zrFZ9;4e|uU=ft`-yXO8{jtB}#@^56uuBcd5r>ped|_~LK$7Jm9T1AhTC3AF0vR&jbS zp?u@JhZa5~-uD(Eyq-WI@l15AMmr+|y^;o}^?u;s>&at?iOhw3$l~cv+sC;wd8Ty# z4GKT~+5>YTM9a&0sAj02bR2#Uf~tW#o})w^-A~iGD~p)Ar?{K$6w5=|(M9EM|8N(W zNNFAYwKR{9ppYLzX9MkJs-_gF`(K{wQo7<7kQ+KE@@jj4m9qqMT|x*$8NNB5k=EM* zzbs`3Fy_Z9$|OVPZqe=xYfr3-znoveQuMj~jdd1p?r%7U*;0!T6?+1XBCq+evI5VXpAf{P zH{=Vrbq(o^V2-!#1JEY8?_Xl4mtH^)7oYF$KVvhUaH6Vge~>AqLMm0cacIy``6N*5 zHkdQzx+-40*kRz{0USMNbZ>!fo}b;?+ZaA}Go4yHZK zp*9Cl&m|?u+4@)sPpA1Uv;~2&m+BV|tZl5Mj_&|VgT@PVy}|Rn7MCtwJE~=8QWA%@ zhqUshM0izDq-WJw!rY$CDUZz4eRN)}2MHXsd-Bb#sS>~OO6mAK9iI(-3yh(^a@)`m zQ{dwWf^JtNGFkeY9Sk)^tiV@ z5g3W1U43$31P~YuEgsEf>a*5kJC;6N%ELN`@&UoY<7i5r%;u7Tzh29xx!KBU7Qzwk z6-(D_D%f6TU0?E};}0kl=A{dDowQhR1sk;B?=>S!dRs1WnGK+xKrdEOjaxr21!k{P zyWlOlIgUu(`^5LQBTh398%$f(zI~P@L($3`i%h~UrK)42r13GFD`3lW=&CYUQ0Hku z7Ng+nq59dorJA}hq#zC>XKrf0;M-vEakhbj(Q&0Ty9f^#?2P#vxWu3`teJl??A^1% z%+?<|E^)EYihwR_+j%|QneckMgp%NP_~o_Dq%5oiYgM+k3`2z+B|dR$RA@YL(&I-< z*ztu;t}s0tUsn z36X5R0JUSQv<;xmQQOOa6&^1NaOUL@GSz4KTx#{w#H*MMiOV2K4?q2C-{E_vX!R2UM@}v}**i;pi z9pamQm3A?MquO~OLVkKRw;c`ZehN^ZGGj{`y8TuSXF|I&WtZZ!kK3==IkIB2MIGd+ zQcEz{AiR~xAFm`+Qbr0j>@$PZt{BXUEP3V~alzVOa;F0<&%qj9`#v z3X@WAyVR_{TaOy8TRq#H(NR!f5t)Y#w>%FYRXSwYqK* z!^q*`cV!KN>Us~k%;bb$=j@>*AJF=syQTKGq`T+7N^T*0xu_O%Dej!p}Ck@teOeRoNRQ}W*D`iFc?g+ zR0ozu?1OTDT-e7j!oOTY(9|L?(D1R>E)-s7rTQ!iaXw>dr~$8x@wDbV2@>j@Zg%Tv?!3k%@F0G zsYw#N{zZ8A|3d#$aLfOc|0{v(Z~ECCCnQhp`#(7T_4gPe(y_!X$!TtUVjJ;~%a@SYUWhlE1mO|YViA>(?hdLhNzHHF>ZORW{})C)LyA;q`EEssfjKZjZeTZ{#OYQ9@{J@; zHE1%&&s7xzR_&WdXhdcO#AQ&R!JA7r7-}@eAysyz9{U5WLzF1~4Vvd>CGMeNg&R@{ zAPsAFov7%}^rz1zFBaj*GR&Fy~b(IW>Wex)|A zHipOGjY(ZkK>IFesBU#pv8|YLt2LoxDe~sA? z@YhDy+I6=!ovuMjJi4_&i%u!=c;CXc`sUYKnf5UgqCw8F1`=T?er)o~iO7qYnoE=2 zNTpju5psU?REj6|6-BF}*3h|`N9nQrIQUGxaj&}7hD`*2iWtJBR`P1Xfx`RCGv~5n z2>99EX3nTp+*j!J1u>d@N=fxM=wM?i!MttyRrzVYPw}U4MlVT30{picRdTp;l}2Ny zE4bOvNhj^*+5>|N0&fgx^ib!6PwU<0jwAPcTIk7F?GW_4o(~55?mVxaQF)VEm!P}) z=tUS89286PayK;y&9(%5Y*1W9K_ypnvpJ_}!u zA$?IF4MLszp@z>(f(iA7t%YgwJlau97Nw{0MTe#u<3aCv4&W8rm8Kb)fTMx@%t zb!SHCg}&sYfd5aRfkqCpbxJJv7S)Y!e{?S`vN{&5$c%RneD0uD%B&+WHMpmgPpOE} znse67_DS}gj4brzN1tkZ`!>O|7)en{xks}b*BkZ=@)t*N)J=|Di1HIBKlT+W8WI1! z`*CHw0`>d7;z6V~ueTb+un=uxaG-J4~?NFz940)+^ z44a?pFmuSS*Z0k`AXBRMMWd0^RhF}R{#4OUzlG7{DgF5&=Wj<`cBx}uZrt=-Z8>$J z*toZrtEYbS?am#mvVlVJN+KkxjzjsIK@aj-*6~vc#oRxsmvg5si?06q-hU+LHs@o8 zNzTH-s~016j_A}33d!cDWTSenT{mDbq^tyq#Llg~%S?dwmB?3e~Y>Gf?06#$~5g$iRJa+TJVNf7k;+^4z1zH)w`CN-&!fNWnpJTqz=? z6Wk;9e#ATW?##rZ<>M!iIMylqK$)bk(D?Eq@%vp~r#`4I8^oC8U2dCAo%dBB5>WG! z$r@ul7;Lqc8ZA1x*d+z_Koex!_42|SbI3cewc&$aYw@~m)_mE+<+o2w+{53#V5wp- zDz=JbY1WCflwd*5PE3L+>q3G%tLZnYoB1G$+Y~^*wsgb<~c~?Aj7n60)Qw98ZAXp=4zxNo;%8 zr$uvr^x3h@Ga}N+kLF%I@+{OOh0ttluW{DwYtHyxP(-GSb6&cehw8GOd4d60wh;kCrIa@8-i!id+KLjSD(N1h9^scvX#S!&ogBZAtS3TLz5WX2sl#+T zF3B}+Nt&tKqR3E{|2GwJQv(~|DLt4ABI znyatruHlZI?6p#BD_VKW>zaBZL`&&=2=$eT*pPeLV2Ox&8*S@GvWSMg3{`ebUe$A~ zSgEJ}^(k(esTQSZ+9=?*WX@i7lH>Th$nK7|1=_cdj`i{H5Jh`-^P3wBWzey8rJtznVe(t413bPexB|YBMpg LI9GDk_0InTf@)FE literal 0 HcmV?d00001 diff --git a/5-NLP/16-RNN/images/rnn.png b/5-NLP/16-RNN/images/rnn.png new file mode 100644 index 0000000000000000000000000000000000000000..160b03036d0872cc48f35733708862a2226ffba4 GIT binary patch literal 18140 zcmdVC2|Sc<+c!>18z~hbBuSE;QQ2C_UWDvRT(&_HVnU^;?E5U1?Ago3C}T~s&5*sz zSlSQ<6~;1{|269OtNXb>_y4&+&-*^_^S(aK^*PVuJdW@2J(lyF%Z#hXm-RH*cku3D zVq#*~ymY~UiD@Um#I$*8GYc%Kd0`1(nKnC~)ji9^^df5Ks_hnd&g^BNagM3D{on*V z*?!}asTUKIY#rmpe9YdA3=8=#>s-C4aYF|l{`{fS>G0t3_` zSnAlT_SIGdvfkypEw+xVT@hk(|M1Ysyh-c>t5vJ*06x+X)<8Fuc_(jj|Im0ar|9#( z8!S>|(i`*9gE<@PPaAqXcf7opZLBr?a>!_y_PYpj=9 zVZmSPHavf?Gq*9dGN#Yjp0Y8t;OX7JQ8tYJLitMz24xsAwEi;s&!H2U|Av0f#vu3? zpACZl=lT2*>htj5k%ZWliAe8tLSWgQ@PI2~mGf}|D+$5ogqL?hJcZ~l{W|3S+El9u&bv~ z*Hh>&;EK0B17@EE75L=MdFA=uA%92k59`dNK4b&a-xPl$!J+ZNqrdFnAb&-422>zR zZ+Q3_(Qn892NeEGvHg)7MmhOd|I7p&*6)aZ5uyS>db$RhKVEzLeq`wv@p0C6bR8k?Z81cMn7iD6WoFhu*%c2YA&RcgVdaS z@Fc5~16cM~3bZe|fFqegwK?#P@)B~Z=%plMho%>4>y;5Ohz7~(^Z=G$5hb@8l{EC6 z6#P?0?M((mah66M+WMJLv?16azNAQMFM)cyU+b4C@ZEHwzBN{+T2Tr5e-z(i>b*AAlt^{f*6bLoTGsQ+ zcl19@tE`weoG~R0M+iD8=2kGxbO$Bj`c2%2=C!pV7aJ;6Mc3=JS2&!gqS)%N*p)jw zzg<@b4vgPu8UL1Pc!h*pr^e(5+dh2;lxPH{<19Z-Wg3rb6nIbe4wBO@-PyIi)VcMG z0;&emw++V4S7@-4tVRMy$zLI@A13XB@BQe)Mp$+Xf)FX_G-7pMMGrfa6)1M66S1CJ zHy&k0a;7SgtR9|Ekt0VaDaoS_d?uX!!q3%6C; zYr73Wp?+Y|Hn@MnBZ?QxKln`WQ)yX?b>0ewF3Fze;=?bGTJtgG(Q9{GLaj_)fkNB* z>LLHuWuf8G@6(6NWZ3rgQog2Du6$En+pR#@F4=?AM3{7%SB?8p-*%`g zN+m7E>sYn3doa~3zMoe^5Vq@At1h{DFFiYc$5wUoD)niW-%@BbLDOZ>g+==G)At;E zecs{avyQ(9m=VvJHdiy@+Dlnj3u&3IZN>&7+3j*GOR+QehHsp()-)NtSHiB2jfKDu zCbxZU>;zhrX{TeX1mmrI-5loYmyf^dDp&n}00FiEbQcb{{ee@^mbANb7fZkLyNwO3 z4drZKEz$Okls4xrfQ)EfFQu;B%7so;SqhoyJodQcD)6cMRcK`Q1^HxKYKF=>`NrL; zNTaDGsrUOxk=(jFb`B5|kJ|Y7=e`cGr{;ftqnvJAGgE^%Ri3!+BL(8V%1NPh?&Jbe zV8Ogr<7=BNF${m<{g9DLVNoE=o_)T?)l(s`>!@-{ffh`GkOY0-@iu48S%CljQ5nK5 zZyCR1j;_*OZ-`a`fgL6(jy9dFtDV-Dt4vSyWtBZ8QG@|SwY`_r@vijhQi#yiWPfLq z6}RL$S##c6$ea3TsM6K?T_B(_PD#|=id*YypnN;lp=H9Y7&fmZ17HAHpF!*1CcI6h24CB+_8&4GdIphd{nq z*?6gi;(z5fpDFK#&stFj?=DJ1-%8YS9N%L|%IRbq#s4Y9+@TF<)AhkP50VN|HJQ}7 z^9Z;)r3`i`qE5Y`P>VcT+!o<7yHgu%*W`;sp?YnO*ov5J8JaRU9!yfLik4eSLQCil zQe2)w^|b0BIhwUR;WR)k2?T;>T~aB^(O<08oLzts4mb@8+aR19T4j(z+e^z&m8U&F zQz{WY5_UTKF-CGAO8zW(OFdFl*i# zuh;SvyPu}#!W*RALA8hSEn9&P&^AO6-1Wy14a8l;1F6%B_d`HZYPnis316x$^)x!R z{!*NRQegReyGPK+oeo0=2-vQ(^AqnaebBc>DHIR$=%rXr)gzxmo!YKw6(Z$$Rd3Fc zv;RG4O;r~o5z#}@rCndHv<(8eADUcC|EL4vzOmde!4kJjLEa@{(Zs~J&OVldRi8G2 zxH`&@PbYc((UuOr1hr&GpK1xVqm_@}*knD%RBO8rS+1J{i-fZ{Xb6}U41R1~&qMs- z%K;V`Ej|qSCX~NcTcL2k`83eqIbmUFhN!ktA(SFd*HMl_9wpm@22c2-hc*Y@Nh*o2 zAV&@Ih0F7UzMbla8o7TYl2U!$my%P+GPHx>2<~F(-4{EHFR6+s75HE+B|gzpBts=Q zIL;NZsbj;Txvze6U%A&l`_i&QAwuXiYG|ZJ$s!+KmJkJn%`TqLBFFTHax1lG8qQ=u zv|9V+!I`(%KqiWH^ZHc_GJuyq2Ts}7R-ayd;hvaFDsE64wih2Q@QIpM0BPuy*}MZM zY2Sm6(nPsP!)vP&y6aZKX(l@rMok3Zvn>r;KmiTwtl+fm)(&`=`PMIgkQ}Zm+0nj# zM7!gGG{K&FTy?ETY0>-y&eFHG|0rv9)yQEm`wFId#ee8JRotB4in1h-axaOv9{X`& zj$^fYVuk5_ZFE5GJWBwA&Qa|NBI+b;9kuc~54md6T*=#QYT10=~Bw&#y)6iW0` z9qFq-YQ3xapFXK=uYHCGaODiudx*|ar8cd`T&XHC)j``nA~5toOt+OXKC^h)VZ-f2bJy22Lwh+?p^Q0i z3gi)$B5I=>I^L8Tj2x*<4cfH&w0bb(9uzAnLz`EetcWW45Jg@O`tW^j#}~5KxM*E> z=+}}CCmVbgy%{Mlje|JB&N`$f5y019BC4~1A~JmJ#Kbp9fG_+Fyl&13V%(%-(IjMW z3_5}p(6&Q>?CKlBKMXg7b_iay#AAqvH~?RM!{H!SVFo5c47)u7$dPT10GaoGiGREL zCI0Q}PX-&#{-nV0<|h3I2L^d|^*>2C*&sj>hNs_(WJbU(Nmjy08N*On4|NGyyklj8DXsqIR1Jai_c zGcfb!z9XZEldRV%)CLM)=vf1Rh3D=2aqoLdKga6tdvl@0B3LKd_JJjVS;L^7Rm1kq z&FAGmRV~=b^SF#gpJWp;A(Rn!ASLO9{e8KV|V2+QC zC!Yz~<2@}xjn#Yq!72R4$;kUzhmtDM63(Yz^)wk$?31(YBwW9m0p5cKWbPwt>Faw`8f;3`iU%0toQp1IF>S4J@ilSCs(5p4RV>How5wH(|Tm z7{hlqThPBWA2~3FPR}>}1^lLzF_gc?48W*x^X;)SWei^wGW`Ynd-#)(P~v_Y{GZ%@ zLTu1{&1m8W*NpE%hkp)XVlcm-UjG^@!yJDS`-|gkGa}a?u<-C(8Cv4p3Gq7Q9$N&6 zVGKVw%b0Iw|BZajxUSm{zm4}FiDD$~v;JuX1Dp;1o9|B_|CFDf^a{F_PMxyF=XyT^ zI+AWLUgs$-!|CH>a1dfISk42gNH8KC2#1Jx41@QqKqShF=P%jt&Cy(?)HZkG@Pfmo zA*9<|BtX_7_5qBT;jI3e!xP^QNm}2cSbO(mCa3R%lS*Nc zB`6-pq@rupU+F89B#k-1kM~)S~o9=MOA2^r-r_=iGP3S{G$KBUoQgPg0$I`dLito=;QENrfh- zu0>*o&az(Ini_Wx62q#n4OA#P)07F;o|7wgwGl)+{#vvp{e+UmcMf|SpZh?R)?Bn^ z8xV!Ao|MNymiA>W3qJ7KA4&|wNQEXf4#=ZqNb9!+oOj3FY-|7w+5fr7l!>GgRsA~Pd$axd?Y0RNAg5#sI$G|}7o=5x-=2g#;CZZ( zq9B?3W%e^Ke5S!!nCzl_ZJ2wY%{8}W7dYi$pz3(wC2I>3pr7gt9p9dGV(hgXD#Fv} zT@cIS3S4pz+*%~m_OKT4$%W?mnc*nXt%8$bC~})kx346uK>(;|C?)B{b*eE@RhR`_ ztfR;QI7he(hk!V`~pU9M%TI0QZ2?oL>Gl%K&yZg_dt8uGqbp zi0D4DYXFB^OUC6sffU6_HTuJpPvIXMnBj9{v5R7tqcXE=da4M^&pTg=5ioYkA>&~a zGNsr50tk|GOQFVzTfG#*7t%1YKF$L^OtPU<+0#LiuAVJ(NCLo3m~fB_ur9}x;t_^Y zt(WsL5c5FNs8Jp^Ag%^#T!>eAr&mKlr;PPXk~-2YG6Dwq2S5p2fcdbFBFrzqTx6>8 zfR=(h;bc(F37mm~lJ*G2=ClF*7$y)WqtueuOHSigZaEzEpqu_^baOaHMTgv8KV+GaOfF3BRFZu1M z&fzTkoiTwbA>}WR;aQb>)(unR#rgSvAG4!$SHxV)zJ);=1plg?!HfN&M-mLt`8^}{Rm zilosUTq3A#k6d%@gMp?J00@$_+Sg86zarqly>n6g>%<#UyT>xk=Q0fQ28NLq@NuMm zVnMwn%b`}#CO{2FmHVg>qWk*yTC?NOv(ASP0AOnlk+g0q?*vikTj=86V3OQ4WD9p$ zH`c88^~;fTsCvworZA|ZriBxWE-b|a?h7x!S{j)7*+dS^bpWMy8nyyzTf&m6pf>&2 z{DN2E`WNLH1%S2QS1Yc3-s<`iBRMxMH=@(sbwzPd=boioLSg6kf^ogk?h z0&J~CtpHNRxyk&-uytOZkp#H)RAw%JKq3`ST2JkFGCSzlCPG_(HGf@P1c!tuwAGpz zsfS_?;A~K0pw1BG?Akr*dKnKGhczp5_Sw1^HSbHLw0%o3c!|k*=;@X~dDe3Ics|rE z_|ff1I%INDfA$_E71}b@;35ZzDwOAYQ2QGN+s%r}PvOn{Hanm4Q zqNLEW2=SXe#*)*raMtTG=xZT zXR`0B<+@Z9l)?Y8^t<76%#Y8M0#dx7qj|N*mqK~xX(HuB)z{lf5^&x~aPnrNpwM2R z%~dAiufGLgP+5oqINC;@%Rl-)DpS7K!WZX}HupIxo$6^KB0ge3uZpYjL$3i<;&F^b^&O#}G@?)|eDJl_uArS_q#M28IPe^0dWex8*k#)#Qu*f+nYW zJ09~oxD88Rk%T%v!aQJ**caMz+{uf&WmHu|+sk;;@6o*0#C`W@SU$D*%u_TCjoAgDM{ z@Jrdrlt2fcK-L_AN5jNS3~*64&vM2Fk6(i zp-xH8s_a{QVp(~QRgzz<><*Jy3p4TL*61HMDWdmTSnroTyLUlx-Qrd1r&;jbklKCr z==h+q=MSUTXtt=gevziH&_CuVdFQyCM&`m~f0$$_^~OR$K?7Dy7^!oKM9H^NS&)7( zCm(Ii=vFry5I19Hx_3VR0`*+R%|wN!rotPN38Xy%B6rWmg{){=e|#0ol@KO2J{1pe zi$3RD;Cd(iJ*~qX?_U++4bGKVBb4YX`$#hyb7v=b=OTTv^~f$u@dEe`(wPbo+j(U= zCW|JMv^0R8tjZ{foCs4L$Pa(8bX6JkdiW6(-zwQHHG`7CRUhO7#wDUlA;10y8Bmh{ zNJ;(C+m+6&pr&sTh5J6>^~HSl4v-|e0LhPQEz_%AIw@+`L3HUH@|69uAGHiH??|kM zziGt)d-)VT_HWB40Ps@1IQ+3j_>e!WQW%krG z&pb3BdrMuW@#0s#GLNi$vEE9kz_W!?L~)%0LQ6|@yC_boJwMhT`fbEkkk1xiz|NoC zO$lt2H0`kyik@HQ$DG((pD(Zc*D3?quY9QP$ zmi$3)uPk`VAn;q8AIWMaq#o^{4JOQ_+TtlQdn|$rEAbOsTWq!_C|PcYL(Lg1&ws^P z<@Y{d>1l^dyDc7(YXU4x0;6q56?+d3v}0YwJzT=O<_fdn^if)#9HKIAf-8hojXL{>^HjbkE*0H~f!gr%ZI0 zP7`bC63Mm?47*l! zLt755i+3Fjft2!6wIU8QP`<7LIFq7_%{h(186-Qi@s31Q+h(M09)g6x2|tA5KE<30 zfuL=%@??vpYQuF}@Oy(Fjl-Qesem-jslRnUh*~(aU@l{BQC@sPNZYe?zu}~*xsYoU za#ZM4B{XsSJyEYII0tGqvRanekzwTPN>Z(H9xH$@R2oa0OXL&Su@_YiF%u~=m9vH^zX1Z0?)**o5jB@2_UXD;PnDuC{&g5oR;FDq+O?_QH4Vg?b zWLMm_f`7%&;gn^@mZM|mPAGxpV~^~7etcHLL3V7mh$#8F_sAU#&h87Qg%@S=Op3Ic zH|x8a%ZP;QzQPn`7u(?v>?wd+9_iR1!sO>5Q`{D{4<5ypcD}jzN!@GIJx4S5y z5+gZ?Ks2C=(|zhX`W(vduFzd4W}ZVB3p79D>efN&W~_wIUAt39ME1g8@4KC5a{wFW zSuLn2T8^_;l0;Yx?HFS08mq|gyd7pfaY#p zSDiZV-+mo0+i>+fqWDD4cKE~?d*nhb>!K(S$6^61+;m`nt1XzYcyPYsNrIJ5t5Jcm zj;E0f=^C~utJuZonlqIP@cIp@GVPr|hd+1pky&Dxv-$qg=xoStc-2@|N&9jbB!?!6 zkX8}o<>G`kU~A)u580mF{Gw4lq9=E(_*NXewRo;OQjZu|iy``SU!eJ4BPE^3Dxh}X zA`1k~IACnm(iLb6%gpoBkB6f=K}yV1k`X-MpnzfnUu9uN0Hu-gn-C=|d6KMd%x|gF zXjH&e>ggb(dkrgr6UIW_T4294P7~|rj-Rp=Pa@6a&R(4!xu82^p?1iAS}@{#pNTd5 zdPJM_k^|x%d<1Q-G-fB&xRr9$;|yqYz`n-Ppa;d$wumR1isnxl zh^ZLe@f~}COf(yqslkBkpzMR8?G9hjS(_RX8gT&c!nEl`E~f@0XH#9Soz|_I8oyzF zeU6qrP~;XksC7&VjHG$6<9upk8Fxp*R#?_UtWl*LBMztQ3d{C~3Rn;YBtm+^CQhcp zQ3JNr?@iI__K~M!`pS_jC0%wd)j8vD#g+QEc>$y-saof;?)D9n9_V2}_<50{cToeOytZB-TgHaO)g1AjUpu(2X4uXG+)4c#9P&Ej7- zI$^QviS4!{kPa#G^hKp>b;xn}$l3n@YNYDiLx_S8+oNK~J9JtC-QHby^~rVG+PMix zn|*ov|3z5t^r+dO-B-`%2hgI*7~mV~vs<%F zC^$HTxb+d63Ygj0JQ5vo1WG2&uraXDvo5mju@{#8LR9TAPSYHvzg+!5*a?iJm5)?P z)=&!8>#y!=m|O*XA2M)CR(0h}*fm=iO&Jg4OA%+b{bbnB*3!(7#zL`3*lV!QAwm6J zprzaqQ9d9}j?kI7A}B~0Sz0Q$HPD&d70-&{TYr@4wEVpw0F3gxL+;G<_9NzN3iF_| zF>xXL>^~#5NR*@IlewPj=Cw-?k#yT7T+qGIC(%$-WdmqLP>tTz3xCdG|g<>?YQ;Sc+=_iUS6MQ(Ins@1|T9 zd?+FW&T0}t_%_fP>@1%gq^O%uIznffbXk+rAoJ@7jO~f4iImU8Vc$F)3VQ{?XIaor za>r+@O7gy>ziAyHBmueIlzw~aqT(=0;Ol)zBQ78wLP>dh>C3$aGF^LLK_q-nM-PdE z9Dv(IA#!z7)Vh)?d{Dwt?6p0O+gnsWY~z*k27hUnL+lkjO=fdGgpt+I6vz&{mc!N> z@(4g*W&Qdcsyho zlY$+x|4TvKP6#LPXuljN+F4T&DTx{4fnHONY7Yry@fg8OGPp`xYSzSuff5cJtBYTw{tWjLwVeX|iQ-@QgO(X6 zqlhva;axUg{_t!UX(pCp_3lO9YSenkO8u1~%#avV z9m?FrM!Iq#CoGyY$$IToc6ru*DX_x(_3;iG%KI%Pup&?43^x7Lljd89`a30YC)9?o z;Vf}4CsV3tiy7x8n^UAePpLP%M7od0R0s%pckE5cYk2+uKEC8N_Fy8z@_g3kl&tTg z9Nnp|)AC9f$JH|U{vrl&iaadq7ILEamF$zStwU=jh=>HpmV}SF+%DTUTEYrzd^Ho2 zmC%Fe;#D4D`D#7`mT&KKNnRuhmHTE%;WRmnU{BfwlcQEQ$%%yzuG!uKb+6 znfs2a8BY~+cY)`!gq*zGfH_k!Z^qk$rTB%nF*$dsode-K;-D+m-*7s!%QqONd#h+S zkKz-}G($b&wq-l2v36gq7<_hz6z@O$SzXvS!&X6|1EdhGjHtB40W}Lw83KKGGaoVS zerWo%V-^oK^Q+kl8FTC8ZMygG$e4FbV3v!96qNihbBAfm2)_YuDtP-FmC2cv>0P;< z6O|K~daT_-x%9%c$_O|n5j7JHKhj_+?hf$rV(9Pof)O5a3iNakuvUN49rLN|8*R&m z8p80p=?i}sanj=F7) z^J5>aYBeE>FDGDq6+JZDiljP+j2%1N9Uvp2_GQnqD_(Yo3=fzx74ckuqi~3GTzy<| zp5GKu#;$nJ%d)6zw$zPpR8- z_E}nlvJ@zuwyMggAnb*Ihv}$t=}b9(l~|m-wSCgcbH}ea$fot{$Q8?V$gV0m5Euf?hwcBx~9}yPP)0YTjxwC z#*&jghE|P_0fGM0R#3p1qlX=aQ0*bsAvHmy8H?Ns-C}@GF4vgImRCrt-Kt2+gEDn> zJCWDNdY)jM+;g1!X~^8T0HlwvA%MeWKyQh8`{ja%by8{tCeHU;;^CVsWNR6%R=eoe z(uhtx*mlw6ekIM1x~vT z%6Erh<9o~1aI?PPY#&)UH6X9?PPw$26JEy4B^&DSmy^hxFAww}vG_vI^3us)%Wiko zB1(a%{vd~uq1k;UjjQQ?!<_+x)uZMoUn!vIO)wF|f^_#Bjs}o+sM}G&uO#f;+oU`Byt2+Z9&%l9i>_jPG{=fkV;IV zqORw6aP_nI74q@f#F8YdVwRP_fE%~g6mED0ny-FzLFmI+2yVqu`e<6WzctX--&P>! zmgW`}81OA{rFu<3+BBTL>}iTAQ8Hl+c5s{E#l!h6#E?Gw9~ve$iw zYNvv{cjC8@Ue|6rK?R~5U9D&Ahu8i4tlX_KZx}uiRS*dUW@$go21)W(1ZVe#{e*xK zzYZe79V`)3Af$$t`-CHoWcy4Xoh7)eTY_>2;NbO&LI{|?nH=(%vs+X3RoVxK{&U_j zSHb=aOFC8nQAFghM144Z(OO<97%tOjk4qhH;yc;(;({eu$1b>vhP)@++*@9*7O>%G=3IfhNC|H-_3Vmm%&&@LgN)R_-*pSKEgj z`BcR}0a60vt(Kq1%xU5&53P!yfJ@0OM$+T(QWG${@DqSSn%8;LM{zui17^Q(h(>`f z)IVPf!uk3{KADBc+_W5^2pj9`P`|DT+tRi2D?M##ZdaC`tdT?bS!=Rq?w z(B*MI(Rt(;(Z1E;L3bh<(dR%L(Yq1Rws1NSKx>D;`{*|7$Yz(~LEH>4#IlxJ5 zt^0pP9u64(449D)GFqnL7yo}HXM?kSJ%{a&v?QGc6YlYBqz%OUfwECEQjQL2X-xut zHq-BH!KH+~ZD;zk8W9-TpS6|Q%k*ba$CUsthuwEAa4AfZ=UX(cCoZAk5)v?sa>CD3 zG5*QGP3$Abm?8eIV13%Id#;ybZ`yqxe}1gpatrCL|ffS{DmUz3>$iPc*gyJD3;WwY@GtGn{w@2Qj0#uE z@e5TO`8c}r@*~d`aBk4pf}NwtS=IfvdJjdpVt?BcoLKdZwsM&^Uvrr?_oJi`nK0ha z%#DGz_LGJyig7>#T4rO9{%odjMoUQp#Jh}r2k$nMvtR^|+*2LUmiTkeGIrYZ@dI+e z8O$H1t@3b!5du&?X7J|7Q9T4BAd<#R>%g0wsif}q|eizv^`XJuk+t#e=i=%iz-i>4@;tQEMKiDt>R7hNEBRRJ^cf{YPM}P z%lb>lg{J9Cqki$;yKnDAeH7b+IpNWSOXPktuXAfS@@h`aCJ$SU^f6}@=l;L@6P_CnePt=Id!vxuW#lXc0rnzRFsWQ#*t z9B$oe<&p%eH4!5Gxl}ShUXOXZtYrUCXJwOKk%o{$%>yI%O1hD3G=-;cQoLJQJ{;Zf3-h!vu|F0RDQ~%Dgyat$Hct&R8Qr+ zwpi`=d_lM4Ywl{oe28s#iiD~Pjb3NROW2kNmYL>Y^lIz&%TBbN+?(>C=NjJ0AFER$ ztdVQ|eg|gWA;qj$-17B&rmpC0vrOrgV4Er2rY)DuL@cx*&YTJ7NoSu+ChEd`GTY*F zm|B!vCKDuVv8nKPFHssI?OwLx_FddwE<-!=>N&>(o^N*;3RFiRcgt%T-YsSjkhH}* z0a!pCapPfp4mxF&pq-Z{QEu229~*g4n6KiTmSIm`FN5dsR$L%<$o(`ARCo5N{1zv3 z*#87)&SN0Q8L&TZa>@D0slEJ`T7Gz^49qoh%xBz6jQFjBJuikv?s6JHocR7IL=m2> zpKJZ)&}A&=pt;C{tM~`b85sEfrA%~7Np#mfucN1*nVj~jRr1!E9*%o1Q+n2j3c=Cq zOJ?8N?ke92OQ&;H#BSYta7J^gKwHf8wY&;Ro>V5zlE)iyVJxwYFzw0<<)n$}SsCs%sK_qfKILH2v4z3kwq(;)x@w0j9z&TexLxc2m! zKjf(Pp1u8ZXUz30lLQqeCf37@{}E1y+% z;NND(uSF4cf(I4z5Xr@#FcP@4#5QFAQv# z2#|ApP0}moI5PK^S@Gre;if)*JD)bGBU~UXigb2&$Sd99_Tb!Lw3&0lHE%Df({`M* zy)O7NZk;-7`p~6f8%w`yiP5LP166vakZecuq{vR?2||!sD`nH9(W!DDa(KO)N)8WX zm@O>&^$IQd%`I~w4F9RDDrWD6rB~;)-zy6jB%PSNLK80P&ODc(^x(>rH+0$M$E(}& zDlLf^^*;G9r3sJH9SSSzoBG@ME>AzOG<^;5nev}Hs|}Jk>!%AiC$o+05~?@=2D5-u zl7O?`atoP1hVz_Dq0y)D+Q;jqS$vzlY3~L(5#7y`&D%|}GhadDbc1s)?~lK5=33n4 z+chtb@Cq963v#iq5Vwxp>E9DLozMzHpL2P4yz|6Nb$Qu@a~63P*A$z^3w3{$7jDm* zrFtH}daing_G5-l#XzU zk3DuzhJ5I%hucGYpZRDlbCF4;b=f1!E4341q{Nl)4~;Qvl;+g8IXt!G_bK_Om;Ekb z_?HY?NACJhag9{ml3L$d;7_X^=JdJNLJ=MH7;vt8jQ5xl^hdxu)k3SF&?sX9(NBAr zY){kGpWrmMMx|aL9T6@dOLe_1Z#ntTzXYtJxP4&K{&-RGD53lwgzam0+6jG~EM*8y( z-cEozq9~q7tPr?-^^$iHr!!EJ6a$f-yS+@4kVRo*_;1mQM%r_^Swz6)5uskZR!@sU zc{ou3rBVVb&?aI?{{lEVOz~fqHr%lijuUC2=?^OPwxOL=1fUt-(`Mqg@1C45M8p5XUA0sIIe7 zBQrr30FZ_k_Pn09HZc__?M|EbRrH&T|)h;8DdV2F5_LN-rz<3^TIHwVwQ~muumY4|1;@q1N2Coa z-;am6+&5PXRQtj?y!EpXZ2h3m;v9;wGJM->t>!`bK_z>{1H+bQ-#c%htby}A5!478 zGfd#TjLbsAwPqxS{o}O9_655mgB^*`<-2WjO|=&4%}6GD#F;DkGkbhTofZG{ySD#D djc<;$tqbNF!gZT$1Q<}77xgX_pR>9D{{fsr-1Gnd literal 0 HcmV?d00001 diff --git a/5-NLP/16-RNN/torchnlp.py b/5-NLP/16-RNN/torchnlp.py new file mode 100644 index 00000000..d6ca5e0c --- /dev/null +++ b/5-NLP/16-RNN/torchnlp.py @@ -0,0 +1,104 @@ +import builtins +import torch +import torchtext +import collections +import os + +device = torch.device("cuda" if torch.cuda.is_available() else "cpu") + +vocab = None +tokenizer = torchtext.data.utils.get_tokenizer('basic_english') + +def load_dataset(ngrams=1,min_freq=1): + global vocab, tokenizer + print("Loading dataset...") + train_dataset, test_dataset = torchtext.datasets.AG_NEWS(root='./data') + train_dataset = list(train_dataset) + test_dataset = list(test_dataset) + classes = ['World', 'Sports', 'Business', 'Sci/Tech'] + print('Building vocab...') + counter = collections.Counter() + for (label, line) in train_dataset: + counter.update(torchtext.data.utils.ngrams_iterator(tokenizer(line),ngrams=ngrams)) + vocab = torchtext.vocab.Vocab(counter, min_freq=min_freq) + return train_dataset,test_dataset,classes,vocab + +def encode(x,voc=None,unk=0,tokenizer=tokenizer): + v = vocab if voc is None else voc + return [v.stoi.get(s,unk) for s in tokenizer(x)] + +def train_epoch(net,dataloader,lr=0.01,optimizer=None,loss_fn = torch.nn.CrossEntropyLoss(),epoch_size=None, report_freq=200): + optimizer = optimizer or torch.optim.Adam(net.parameters(),lr=lr) + loss_fn = loss_fn.to(device) + net.train() + total_loss,acc,count,i = 0,0,0,0 + for labels,features in dataloader: + optimizer.zero_grad() + features, labels = features.to(device), labels.to(device) + out = net(features) + loss = loss_fn(out,labels) #cross_entropy(out,labels) + loss.backward() + optimizer.step() + total_loss+=loss + _,predicted = torch.max(out,1) + acc+=(predicted==labels).sum() + count+=len(labels) + i+=1 + if i%report_freq==0: + print(f"{count}: acc={acc.item()/count}") + if epoch_size and count>epoch_size: + break + return total_loss.item()/count, acc.item()/count + +def padify(b,voc=None,tokenizer=tokenizer): + # b is the list of tuples of length batch_size + # - first element of a tuple = label, + # - second = feature (text sequence) + # build vectorized sequence + v = [encode(x[1],voc=voc,tokenizer=tokenizer) for x in b] + # compute max length of a sequence in this minibatch + l = max(map(len,v)) + return ( # tuple of two tensors - labels and features + torch.LongTensor([t[0]-1 for t in b]), + torch.stack([torch.nn.functional.pad(torch.tensor(t),(0,l-len(t)),mode='constant',value=0) for t in v]) + ) + +def offsetify(b,voc=None): + # first, compute data tensor from all sequences + x = [torch.tensor(encode(t[1],voc=voc)) for t in b] + # now, compute the offsets by accumulating the tensor of sequence lengths + o = [0] + [len(t) for t in x] + o = torch.tensor(o[:-1]).cumsum(dim=0) + return ( + torch.LongTensor([t[0]-1 for t in b]), # labels + torch.cat(x), # text + o + ) + +def train_epoch_emb(net,dataloader,lr=0.01,optimizer=None,loss_fn = torch.nn.CrossEntropyLoss(),epoch_size=None, report_freq=200,use_pack_sequence=False): + optimizer = optimizer or torch.optim.Adam(net.parameters(),lr=lr) + loss_fn = loss_fn.to(device) + net.train() + total_loss,acc,count,i = 0,0,0,0 + for labels,text,off in dataloader: + optimizer.zero_grad() + labels,text = labels.to(device), text.to(device) + if use_pack_sequence: + off = off.to('cpu') + else: + off = off.to(device) + out = net(text, off) + loss = loss_fn(out,labels) #cross_entropy(out,labels) + loss.backward() + optimizer.step() + total_loss+=loss + _,predicted = torch.max(out,1) + acc+=(predicted==labels).sum() + count+=len(labels) + i+=1 + if i%report_freq==0: + print(f"{count}: acc={acc.item()/count}") + if epoch_size and count>epoch_size: + break + return total_loss.item()/count, acc.item()/count + diff --git a/5-NLP/README.md b/5-NLP/README.md index fa9f7b3f..89b9f55a 100644 --- a/5-NLP/README.md +++ b/5-NLP/README.md @@ -52,4 +52,6 @@ if len(physical_devices)>0: * [Representing text as tensors](13-TextRep/README.md) * [Word Embeddings](14-Emdeddings/README.md) * [Language Modeling](15-LanguageModeling/README.md) - +* [Recurrent Neural Networks](16-RNN/README.md) +* [Generative Networks](17-GenerativeNetworks/README.md) +* [Transformers](18-Transformers/README.md) diff --git a/README.md b/README.md index 42d2a503..ad51a05f 100644 --- a/README.md +++ b/README.md @@ -64,9 +64,9 @@ For a gentle introduction to *AI in the Cloud* topic you may consider taking [Ge 13Text Representation. Bow/TF-IDFTextPyTorchTensorflow 14Semantic word embeddings. Word2Vec and GloVeTextPyTorchTensorflow 15Language Modeling. Training your own embeddingsTextPyTorchTensorflow -16Recurrent Neural NetworksTextPyTorchTensorflow -17Generative Recurrent NetworksTextPyTorchTensorflow -18Language Modelling. Transformers. BERT.TextPyTorchTensorflow +16Recurrent Neural NetworksTextPyTorchTensorflow +17Generative Recurrent NetworksTextPyTorchTensorflow +18Transformers. BERT.TextPyTorchTensorflow 19Named Entity RecognitionTextPyTorchTensorflow 20Text Generation using GPTTextPyTorchTensorflow VIOther AI TechniquesPAT