485 lines
24 KiB
Plaintext
485 lines
24 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"# Recurrent neural networks\n",
|
|
"\n",
|
|
"For di previous module, we don dey use beta semantic representations of text, and one simple linear classifier wey dey on top di embeddings. Wetin dis architecture dey do na to capture di overall meaning of words for one sentence, but e no dey consider di **order** of di words, because di aggregation operation wey dey on top di embeddings don remove dis information from di original text. Since dis models no fit model word ordering, dem no fit solve more complex or ambiguous tasks like text generation or question answering.\n",
|
|
"\n",
|
|
"To fit capture di meaning of text sequence, we go need use another neural network architecture wey dem dey call **recurrent neural network**, or RNN. For RNN, we go dey pass our sentence through di network one symbol at a time, and di network go produce one **state**, wey we go then pass back to di network with di next symbol.\n",
|
|
"\n",
|
|
"<img alt=\"RNN\" src=\"../../../../../translated_images/pcm/rnn.27f5c29c53d727b5.webp\" width=\"60%\"/>\n",
|
|
"\n",
|
|
"If we get input sequence of tokens $X_0,\\dots,X_n$, RNN go create one sequence of neural network blocks, and e go train dis sequence end-to-end using back propagation. Each network block dey take one pair $(X_i,S_i)$ as input, and e go produce $S_{i+1}$ as result. Final state $S_n$ or output $X_n$ go enter one linear classifier to produce di result. All di network blocks dey share di same weights, and dem dey train end-to-end using one back propagation pass.\n",
|
|
"\n",
|
|
"Because state vectors $S_0,\\dots,S_n$ dey pass through di network, e fit learn di sequential dependencies between words. For example, if di word *not* show somewhere for di sequence, e fit learn how to negate some elements inside di state vector, wey go result in negation.\n",
|
|
"\n",
|
|
"> Since di weights of all RNN blocks for di picture dey shared, di same picture fit represent as one block (for di right) with one recurrent feedback loop, wey dey pass di output state of di network back to di input.\n",
|
|
"\n",
|
|
"Make we see how recurrent neural networks fit help us classify our news dataset.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 1,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Loading dataset...\n",
|
|
"Building vocab...\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"import torch\n",
|
|
"import torchtext\n",
|
|
"from torchnlp import *\n",
|
|
"train_dataset, test_dataset, classes, vocab = load_dataset()\n",
|
|
"vocab_size = len(vocab)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Simple RNN classifier\n",
|
|
"\n",
|
|
"For simple RNN, each recurrent unit na just simple linear network wey go take join input vector and state vector together, come produce new state vector. PyTorch dey represent dis unit wit `RNNCell` class, and network of dis kain cells - as `RNN` layer.\n",
|
|
"\n",
|
|
"To define RNN classifier, we go first use embedding layer to reduce di size of di input vocabulary, den we go put RNN layer on top:\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 2,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"class RNNClassifier(torch.nn.Module):\n",
|
|
" def __init__(self, vocab_size, embed_dim, hidden_dim, num_class):\n",
|
|
" super().__init__()\n",
|
|
" self.hidden_dim = hidden_dim\n",
|
|
" self.embedding = torch.nn.Embedding(vocab_size, embed_dim)\n",
|
|
" self.rnn = torch.nn.RNN(embed_dim,hidden_dim,batch_first=True)\n",
|
|
" self.fc = torch.nn.Linear(hidden_dim, num_class)\n",
|
|
"\n",
|
|
" def forward(self, x):\n",
|
|
" batch_size = x.size(0)\n",
|
|
" x = self.embedding(x)\n",
|
|
" x,h = self.rnn(x)\n",
|
|
" return self.fc(x.mean(dim=1))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"> **Note:** We dey use untrained embedding layer here make e simple, but if we wan get beta result, we fit use pre-trained embedding layer wey dey use Word2Vec or GloVe embeddings, as dem explain for di previous unit. To sabi am well, you fit try change dis code make e work with pre-trained embeddings.\n",
|
|
"\n",
|
|
"For our own case, we go use padded data loader, so each batch go get some padded sequences wey get di same length. Di RNN layer go collect di sequence of embedding tensors, and e go produce two outputs: \n",
|
|
"* $x$ na di sequence of RNN cell outputs for each step\n",
|
|
"* $h$ na di final hidden state for di last element of di sequence\n",
|
|
"\n",
|
|
"After dat, we go apply fully-connected linear classifier to get di number of class.\n",
|
|
"\n",
|
|
"> **Note:** RNNs no easy to train, because once di RNN cells don unroll for di sequence length, di number of layers wey dey involved for back propagation go plenty well well. So, we need to choose small learning rate, and train di network for bigger dataset to get beta result. E fit take plenty time, so e better make we use GPU.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 3,
|
|
"metadata": {
|
|
"scrolled": true
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"3200: acc=0.3090625\n",
|
|
"6400: acc=0.38921875\n",
|
|
"9600: acc=0.4590625\n",
|
|
"12800: acc=0.511953125\n",
|
|
"16000: acc=0.5506875\n",
|
|
"19200: acc=0.57921875\n",
|
|
"22400: acc=0.6070089285714285\n",
|
|
"25600: acc=0.6304296875\n",
|
|
"28800: acc=0.6484027777777778\n",
|
|
"32000: acc=0.66509375\n",
|
|
"35200: acc=0.6790056818181818\n",
|
|
"38400: acc=0.6929166666666666\n",
|
|
"41600: acc=0.7035817307692308\n",
|
|
"44800: acc=0.7137276785714286\n",
|
|
"48000: acc=0.72225\n",
|
|
"51200: acc=0.73001953125\n",
|
|
"54400: acc=0.7372794117647059\n",
|
|
"57600: acc=0.7436631944444444\n",
|
|
"60800: acc=0.7503947368421052\n",
|
|
"64000: acc=0.75634375\n",
|
|
"67200: acc=0.7615773809523809\n",
|
|
"70400: acc=0.7662642045454545\n",
|
|
"73600: acc=0.7708423913043478\n",
|
|
"76800: acc=0.7751822916666666\n",
|
|
"80000: acc=0.7790625\n",
|
|
"83200: acc=0.7825\n",
|
|
"86400: acc=0.7858564814814815\n",
|
|
"89600: acc=0.7890513392857142\n",
|
|
"92800: acc=0.7920474137931034\n",
|
|
"96000: acc=0.7952708333333334\n",
|
|
"99200: acc=0.7982258064516129\n",
|
|
"102400: acc=0.80099609375\n",
|
|
"105600: acc=0.8037594696969697\n",
|
|
"108800: acc=0.8060569852941176\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=16, collate_fn=padify, shuffle=True)\n",
|
|
"net = RNNClassifier(vocab_size,64,32,len(classes)).to(device)\n",
|
|
"train_epoch(net,train_loader, lr=0.001)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Long Short Term Memory (LSTM)\n",
|
|
"\n",
|
|
"One big wahala wey classical RNNs get na di **vanishing gradients** problem. Because RNNs dey train end-to-end for one back-propagation pass, e dey hard for am to carry error go di first layers of di network, and so di network no fit sabi di relationship wey dey between far tokens. One way wey dem dey take avoid dis wahala na to use **explicit state management** wit di help of wetin dem dey call **gates**. Two di most popular architectures wey dey dis kind na **Long Short Term Memory** (LSTM) and **Gated Relay Unit** (GRU).\n",
|
|
"\n",
|
|
"\n",
|
|
"\n",
|
|
"LSTM Network dey arrange like RNN, but e get two states wey dey pass from layer to layer: di real state $c$, and hidden vector $h$. For each unit, hidden vector $h_i$ go join body wit input $x_i$, and dem go control wetin go happen to di state $c$ through **gates**. Each gate na neural network wey get sigmoid activation (output dey between $[0,1]$), wey you fit think of as bitwise mask when e multiply di state vector. Di gates wey dey (from left to right for di picture above) na:\n",
|
|
"* **forget gate** dey use hidden vector take decide which parts of di vector $c$ we go forget, and which ones we go allow pass.\n",
|
|
"* **input gate** dey collect some information from di input and hidden vector, then e go put am inside di state.\n",
|
|
"* **output gate** dey change di state through some linear layer wey get $\\tanh$ activation, then e go use hidden vector $h_i$ select some parts of di state to produce new state $c_{i+1}$.\n",
|
|
"\n",
|
|
"Di parts of di state $c$ fit be like flags wey you fit switch on and off. For example, if we see name *Alice* for di sequence, we fit assume say e dey talk about female character, and we go raise di flag for di state say we get female noun for di sentence. If later we see phrase *and Tom*, we go raise di flag say we get plural noun. So, by di way we dey manipulate di state, we fit dey track di grammatical properties of di sentence parts.\n",
|
|
"\n",
|
|
"> **Note**: One better resource wey fit help you understand di inside of LSTM na dis fine article [Understanding LSTM Networks](https://colah.github.io/posts/2015-08-Understanding-LSTMs/) by Christopher Olah.\n",
|
|
"\n",
|
|
"Even though di inside structure of LSTM cell fit look complex, PyTorch dey hide di implementation inside `LSTMCell` class, and e dey provide `LSTM` object to represent di whole LSTM layer. So, di way we go take implement LSTM classifier go dey almost di same as di simple RNN wey we don see before:\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 4,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"class LSTMClassifier(torch.nn.Module):\n",
|
|
" def __init__(self, vocab_size, embed_dim, hidden_dim, num_class):\n",
|
|
" super().__init__()\n",
|
|
" self.hidden_dim = hidden_dim\n",
|
|
" self.embedding = torch.nn.Embedding(vocab_size, embed_dim)\n",
|
|
" self.embedding.weight.data = torch.randn_like(self.embedding.weight.data)-0.5\n",
|
|
" self.rnn = torch.nn.LSTM(embed_dim,hidden_dim,batch_first=True)\n",
|
|
" self.fc = torch.nn.Linear(hidden_dim, num_class)\n",
|
|
"\n",
|
|
" def forward(self, x):\n",
|
|
" batch_size = x.size(0)\n",
|
|
" x = self.embedding(x)\n",
|
|
" x,(h,c) = self.rnn(x)\n",
|
|
" return self.fc(h[-1])"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Make we train our network now. Note say to train LSTM dey slow well, and you fit no see plenty increase for accuracy for di beginning of di training. Plus, you go need try adjust di `lr` learning rate parameter to find di learning rate wey go make di training speed dey okay, and still no go waste memory.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 5,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"3200: acc=0.259375\n",
|
|
"6400: acc=0.25859375\n",
|
|
"9600: acc=0.26177083333333334\n",
|
|
"12800: acc=0.2784375\n",
|
|
"16000: acc=0.313\n",
|
|
"19200: acc=0.3528645833333333\n",
|
|
"22400: acc=0.3965625\n",
|
|
"25600: acc=0.4385546875\n",
|
|
"28800: acc=0.4752777777777778\n",
|
|
"32000: acc=0.505375\n",
|
|
"35200: acc=0.5326704545454546\n",
|
|
"38400: acc=0.5557552083333334\n",
|
|
"41600: acc=0.5760817307692307\n",
|
|
"44800: acc=0.5954910714285714\n",
|
|
"48000: acc=0.6118333333333333\n",
|
|
"51200: acc=0.62681640625\n",
|
|
"54400: acc=0.6404779411764706\n",
|
|
"57600: acc=0.6520138888888889\n",
|
|
"60800: acc=0.662828947368421\n",
|
|
"64000: acc=0.673546875\n",
|
|
"67200: acc=0.6831547619047619\n",
|
|
"70400: acc=0.6917897727272727\n",
|
|
"73600: acc=0.6997146739130434\n",
|
|
"76800: acc=0.707109375\n",
|
|
"80000: acc=0.714075\n",
|
|
"83200: acc=0.7209134615384616\n",
|
|
"86400: acc=0.727037037037037\n",
|
|
"89600: acc=0.7326674107142858\n",
|
|
"92800: acc=0.7379633620689655\n",
|
|
"96000: acc=0.7433645833333333\n",
|
|
"99200: acc=0.7479032258064516\n",
|
|
"102400: acc=0.752119140625\n",
|
|
"105600: acc=0.7562405303030303\n",
|
|
"108800: acc=0.76015625\n",
|
|
"112000: acc=0.7641339285714286\n",
|
|
"115200: acc=0.7677777777777778\n",
|
|
"118400: acc=0.7711233108108108\n"
|
|
]
|
|
},
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"(0.03487814127604167, 0.7728)"
|
|
]
|
|
},
|
|
"execution_count": 5,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"net = LSTMClassifier(vocab_size,64,32,len(classes)).to(device)\n",
|
|
"train_epoch(net,train_loader, lr=0.001)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Packed sequences\n",
|
|
"\n",
|
|
"For di example wey we show, we gatz pad all di sequences for di minibatch wit zero vectors. Dis one dey waste memory small, but for RNNs, di main wahala na say extra RNN cells go dey created for di padded input items. Dis cells go dey train, but dem no go carry any important input information. E go better if we fit train RNN only for di real sequence size.\n",
|
|
"\n",
|
|
"To do dis one, PyTorch don bring one special format wey go store padded sequence. Imagine say we get input padded minibatch wey be like dis:\n",
|
|
"```\n",
|
|
"[[1,2,3,4,5],\n",
|
|
" [6,7,8,0,0],\n",
|
|
" [9,0,0,0,0]]\n",
|
|
"```\n",
|
|
"Here 0 na di padded values, and di real length vector for di input sequences na `[5,3,1]`.\n",
|
|
"\n",
|
|
"To train RNN well wit padded sequence, we go wan start di training for di first group of RNN cells wit big minibatch (`[1,6,9]`), but later we go stop di processing for di third sequence, and continue di training wit smaller minibatches (`[2,7]`, `[3,8]`), and so on. So, packed sequence go dey represented as one vector - for dis case `[1,6,9,2,7,3,8,4,5]`, and length vector (`[5,3,1]`), wey we fit use to reconstruct di original padded minibatch.\n",
|
|
"\n",
|
|
"To create packed sequence, we fit use `torch.nn.utils.rnn.pack_padded_sequence` function. All di recurrent layers, like RNN, LSTM and GRU, dey support packed sequences as input, and dem go produce packed output, wey we fit decode wit `torch.nn.utils.rnn.pad_packed_sequence`.\n",
|
|
"\n",
|
|
"To fit create packed sequence, we gatz pass di length vector to di network, and because of dis, we go need different function to prepare minibatches:\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 6,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"def pad_length(b):\n",
|
|
" # build vectorized sequence\n",
|
|
" v = [encode(x[1]) for x in b]\n",
|
|
" # compute max length of a sequence in this minibatch and length sequence itself\n",
|
|
" len_seq = list(map(len,v))\n",
|
|
" l = max(len_seq)\n",
|
|
" return ( # tuple of three tensors - labels, padded features, length sequence\n",
|
|
" torch.LongTensor([t[0]-1 for t in b]),\n",
|
|
" torch.stack([torch.nn.functional.pad(torch.tensor(t),(0,l-len(t)),mode='constant',value=0) for t in v]),\n",
|
|
" torch.tensor(len_seq)\n",
|
|
" )\n",
|
|
"\n",
|
|
"train_loader_len = torch.utils.data.DataLoader(train_dataset, batch_size=16, collate_fn=pad_length, shuffle=True)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Di network go dey almost like `LSTMClassifier` wey dey up, but di `forward` pass go collect both padded minibatch and di vector of sequence lengths. After we don compute di embedding, we go compute packed sequence, pass am go di LSTM layer, and then unpack di result back.\n",
|
|
"\n",
|
|
"> **Note**: We no dey actually use di unpacked result `x`, because na di output from di hidden layers we dey use for di next computations. So, we fit remove di unpacking completely from dis code. Di reason why we put am here na so you fit modify di code easily if you need use di network output for other computations.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 7,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"class LSTMPackClassifier(torch.nn.Module):\n",
|
|
" def __init__(self, vocab_size, embed_dim, hidden_dim, num_class):\n",
|
|
" super().__init__()\n",
|
|
" self.hidden_dim = hidden_dim\n",
|
|
" self.embedding = torch.nn.Embedding(vocab_size, embed_dim)\n",
|
|
" self.embedding.weight.data = torch.randn_like(self.embedding.weight.data)-0.5\n",
|
|
" self.rnn = torch.nn.LSTM(embed_dim,hidden_dim,batch_first=True)\n",
|
|
" self.fc = torch.nn.Linear(hidden_dim, num_class)\n",
|
|
"\n",
|
|
" def forward(self, x, lengths):\n",
|
|
" batch_size = x.size(0)\n",
|
|
" x = self.embedding(x)\n",
|
|
" pad_x = torch.nn.utils.rnn.pack_padded_sequence(x,lengths,batch_first=True,enforce_sorted=False)\n",
|
|
" pad_x,(h,c) = self.rnn(pad_x)\n",
|
|
" x, _ = torch.nn.utils.rnn.pad_packed_sequence(pad_x,batch_first=True)\n",
|
|
" return self.fc(h[-1])"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Make we start di training:\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 8,
|
|
"metadata": {
|
|
"scrolled": true
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"3200: acc=0.285625\n",
|
|
"6400: acc=0.33359375\n",
|
|
"9600: acc=0.3876041666666667\n",
|
|
"12800: acc=0.44078125\n",
|
|
"16000: acc=0.4825\n",
|
|
"19200: acc=0.5235416666666667\n",
|
|
"22400: acc=0.5559821428571429\n",
|
|
"25600: acc=0.58609375\n",
|
|
"28800: acc=0.6116666666666667\n",
|
|
"32000: acc=0.63340625\n",
|
|
"35200: acc=0.6525284090909091\n",
|
|
"38400: acc=0.668515625\n",
|
|
"41600: acc=0.6822596153846154\n",
|
|
"44800: acc=0.6948214285714286\n",
|
|
"48000: acc=0.7052708333333333\n",
|
|
"51200: acc=0.71521484375\n",
|
|
"54400: acc=0.7239889705882353\n",
|
|
"57600: acc=0.7315277777777778\n",
|
|
"60800: acc=0.7388486842105263\n",
|
|
"64000: acc=0.74571875\n",
|
|
"67200: acc=0.7518303571428572\n",
|
|
"70400: acc=0.7576988636363636\n",
|
|
"73600: acc=0.7628940217391305\n",
|
|
"76800: acc=0.7681510416666667\n",
|
|
"80000: acc=0.7728125\n",
|
|
"83200: acc=0.7772235576923077\n",
|
|
"86400: acc=0.7815393518518519\n",
|
|
"89600: acc=0.7857700892857142\n",
|
|
"92800: acc=0.7895043103448276\n",
|
|
"96000: acc=0.7930520833333333\n",
|
|
"99200: acc=0.7959072580645161\n",
|
|
"102400: acc=0.798994140625\n",
|
|
"105600: acc=0.802064393939394\n",
|
|
"108800: acc=0.8051378676470589\n",
|
|
"112000: acc=0.8077857142857143\n",
|
|
"115200: acc=0.8104600694444445\n",
|
|
"118400: acc=0.8128293918918919\n"
|
|
]
|
|
},
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"(0.029785829671223958, 0.8138166666666666)"
|
|
]
|
|
},
|
|
"execution_count": 8,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"net = LSTMPackClassifier(vocab_size,64,32,len(classes)).to(device)\n",
|
|
"train_epoch_emb(net,train_loader_len, lr=0.001,use_pack_sequence=True)\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"> **Note:** You fit don notice di parameter `use_pack_sequence` wey we dey pass to di training function. Right now, di `pack_padded_sequence` function need di length sequence tensor to dey for CPU device, so di training function need to make sure say e no carry di length sequence data go GPU when e dey train. You fit check di implementation of `train_emb` function for di [`torchnlp.py`](../../../../../lessons/5-NLP/16-RNN/torchnlp.py) file.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Bidirectional and multilayer RNNs\n",
|
|
"\n",
|
|
"For di examples wey we don show, all di recurrent networks dey work for one direction, from di start of di sequence go di end. E dey look normal like dat, because e resemble how we dey read and hear speech. But, for many cases wey we fit get random access to di input sequence, e go make sense to run di recurrent computation for both directions. Dis kain networks na wetin dem dey call **bidirectional** RNNs, and you fit create am by passing `bidirectional=True` parameter to RNN/LSTM/GRU constructor.\n",
|
|
"\n",
|
|
"If you dey use bidirectional network, you go need two hidden state vectors, one for each direction. PyTorch dey encode di two vectors as one vector wey big pass di normal size by two times, and e dey very convenient because normally you go pass di hidden state wey you get to fully-connected linear layer, and you go just need to adjust di size when you dey create di layer.\n",
|
|
"\n",
|
|
"Recurrent network, whether e dey go one direction or e dey bidirectional, dey capture some patterns inside sequence, and e fit store dem for state vector or pass dem go output. Just like convolutional networks, we fit build another recurrent layer on top di first one to capture higher level patterns, wey di low-level patterns wey di first layer extract go help build. Dis one na wetin dem dey call **multi-layer RNN**, wey get two or more recurrent networks, and di output of di previous layer go dey pass go di next layer as input.\n",
|
|
"\n",
|
|
"\n",
|
|
"\n",
|
|
"*Di picture dey from [dis fine post](https://towardsdatascience.com/from-a-lstm-cell-to-a-multilayer-lstm-network-with-pytorch-2899eb5696f3) by Fernando López*\n",
|
|
"\n",
|
|
"PyTorch dey make am easy to construct dis kain networks, because you just need to pass `num_layers` parameter to RNN/LSTM/GRU constructor to build plenty layers of recurrence automatically. Dis one go also mean say di size of di hidden/state vector go increase well well, and you go need to adjust di way you dey handle di output of di recurrent layers.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## RNNs for oda work dem\n",
|
|
"\n",
|
|
"For dis unit, we don see say RNNs fit dey use for sequence classification, but true true, dem fit handle plenty oda work dem, like text generation, machine translation, and plenty more. We go look dose work dem for di next unit.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"---\n\n<!-- CO-OP TRANSLATOR DISCLAIMER START -->\n**Disclaimer**: \nDis dokyument don use AI transle-shun service [Co-op Translator](https://github.com/Azure/co-op-translator) do di transle-shun. Even as we dey try make am correct, abeg make you sabi say AI transle-shun fit get mistake or no dey accurate well. Di original dokyument wey dey for im native language na di one wey you go take as di correct source. For important mata, e good make you use professional human transle-shun. We no go fit take blame for any misunderstanding or wrong interpretation wey fit happen because you use dis transle-shun.\n<!-- CO-OP TRANSLATOR DISCLAIMER END -->\n"
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"interpreter": {
|
|
"hash": "16af2a8bbb083ea23e5e41c7f5787656b2ce26968575d8763f2c4b17f9cd711f"
|
|
},
|
|
"kernelspec": {
|
|
"display_name": "Python 3.8.12 ('py38')",
|
|
"language": "python",
|
|
"name": "python3"
|
|
},
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3",
|
|
"version": "3.8.12"
|
|
},
|
|
"coopTranslator": {
|
|
"original_hash": "522ee52ae3d5ae933e283286254e9a55",
|
|
"translation_date": "2025-11-18T19:35:26+00:00",
|
|
"source_file": "lessons/5-NLP/16-RNN/RNNPyTorch.ipynb",
|
|
"language_code": "pcm"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 2
|
|
} |