479 lines
24 KiB
Plaintext
479 lines
24 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"# Mitandao ya neva ya kurudia\n",
|
|
"\n",
|
|
"Katika moduli iliyopita, tumekuwa tukitumia uwakilishi wa semantiki tajiri wa maandishi, na kionyeshi rahisi cha mstari juu ya embeddings. Kile usanifu huu unafanya ni kunasa maana iliyojumlishwa ya maneno katika sentensi, lakini hauzingatii **mpangilio** wa maneno, kwa sababu operesheni ya kujumlisha juu ya embeddings huondoa taarifa hii kutoka kwa maandishi ya asili. Kwa kuwa mifano hii haiwezi kuiga mpangilio wa maneno, haiwezi kutatua kazi ngumu zaidi au zenye utata kama vile uzalishaji wa maandishi au kujibu maswali.\n",
|
|
"\n",
|
|
"Ili kunasa maana ya mlolongo wa maandishi, tunahitaji kutumia usanifu mwingine wa mtandao wa neva, unaoitwa **mtandao wa neva wa kurudia**, au RNN. Katika RNN, tunapitisha sentensi yetu kupitia mtandao moja kwa moja, ishara moja kwa wakati, na mtandao huzalisha hali fulani (**state**), ambayo tunapitisha tena kwenye mtandao pamoja na ishara inayofuata.\n",
|
|
"\n",
|
|
"Kwa kuzingatia mlolongo wa ishara za pembejeo $X_0,\\dots,X_n$, RNN huunda mlolongo wa vizuizi vya mtandao wa neva, na hufundisha mlolongo huu kutoka mwanzo hadi mwisho kwa kutumia kurudisha nyuma. Kila kizuizi cha mtandao huchukua jozi $(X_i,S_i)$ kama pembejeo, na huzalisha $S_{i+1}$ kama matokeo. Hali ya mwisho $S_n$ au matokeo $X_n$ huingia kwenye kionyeshi cha mstari ili kutoa matokeo. Vizuizi vyote vya mtandao vinashiriki uzito sawa, na hufundishwa kutoka mwanzo hadi mwisho kwa kutumia mchakato mmoja wa kurudisha nyuma.\n",
|
|
"\n",
|
|
"Kwa sababu vekta za hali $S_0,\\dots,S_n$ zinapitishwa kupitia mtandao, inaweza kujifunza utegemezi wa mlolongo kati ya maneno. Kwa mfano, wakati neno *si* linatokea mahali fulani katika mlolongo, linaweza kujifunza kukanusha vipengele fulani ndani ya vekta ya hali, na kusababisha kukanusha.\n",
|
|
"\n",
|
|
"> Kwa kuwa uzito wa vizuizi vyote vya RNN kwenye picha vinashirikiana, picha hiyo hiyo inaweza kuwakilishwa kama kizuizi kimoja (kwenye upande wa kulia) na kitanzi cha maoni ya kurudia, ambacho kinapitisha hali ya matokeo ya mtandao kurudi kwenye pembejeo.\n",
|
|
"\n",
|
|
"Hebu tuone jinsi mitandao ya neva ya kurudia inaweza kutusaidia kuainisha seti yetu ya data ya habari.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 1,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Loading dataset...\n",
|
|
"Building vocab...\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"import torch\n",
|
|
"import torchtext\n",
|
|
"from torchnlp import *\n",
|
|
"train_dataset, test_dataset, classes, vocab = load_dataset()\n",
|
|
"vocab_size = len(vocab)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Kainishi cha RNN\n",
|
|
"\n",
|
|
"Katika hali ya RNN rahisi, kila kitengo cha kurudia ni mtandao rahisi wa mstari, ambao huchukua vector ya pembejeo iliyounganishwa na vector ya hali, na kutoa vector mpya ya hali. PyTorch inawakilisha kitengo hiki kwa darasa la `RNNCell`, na mitandao ya seli kama hizo - kama safu ya `RNN`.\n",
|
|
"\n",
|
|
"Ili kufafanua classifier ya RNN, tutatumia kwanza safu ya embedding kupunguza ukubwa wa msamiati wa pembejeo, kisha kuweka safu ya RNN juu yake:\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 2,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"class RNNClassifier(torch.nn.Module):\n",
|
|
" def __init__(self, vocab_size, embed_dim, hidden_dim, num_class):\n",
|
|
" super().__init__()\n",
|
|
" self.hidden_dim = hidden_dim\n",
|
|
" self.embedding = torch.nn.Embedding(vocab_size, embed_dim)\n",
|
|
" self.rnn = torch.nn.RNN(embed_dim,hidden_dim,batch_first=True)\n",
|
|
" self.fc = torch.nn.Linear(hidden_dim, num_class)\n",
|
|
"\n",
|
|
" def forward(self, x):\n",
|
|
" batch_size = x.size(0)\n",
|
|
" x = self.embedding(x)\n",
|
|
" x,h = self.rnn(x)\n",
|
|
" return self.fc(x.mean(dim=1))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"> **Note:** Tunatumia safu ya embedding isiyofunzwa hapa kwa urahisi, lakini kwa matokeo bora zaidi tunaweza kutumia safu ya embedding iliyofunzwa awali na Word2Vec au GloVe embeddings, kama ilivyoelezwa katika sehemu ya awali. Ili kuelewa vyema, unaweza kurekebisha msimbo huu ili ufanye kazi na embeddings zilizofunzwa awali.\n",
|
|
"\n",
|
|
"Katika hali yetu, tutatumia data loader yenye padding, hivyo kila kundi litakuwa na idadi ya mfuatano uliowekwa padding wa urefu sawa. Safu ya RNN itachukua mfuatano wa tensors za embedding, na kutoa matokeo mawili:\n",
|
|
"* $x$ ni mfuatano wa matokeo ya seli za RNN katika kila hatua\n",
|
|
"* $h$ ni hali ya mwisho ya siri kwa kipengele cha mwisho cha mfuatano\n",
|
|
"\n",
|
|
"Kisha tunatumia classifier ya mstari iliyounganishwa kikamilifu ili kupata idadi ya darasa.\n",
|
|
"\n",
|
|
"> **Note:** RNNs ni ngumu sana kufunza, kwa sababu mara seli za RNN zinapopanuliwa kulingana na urefu wa mfuatano, idadi ya safu zinazohusika katika back propagation huwa kubwa sana. Kwa hivyo tunahitaji kuchagua kiwango kidogo cha kujifunza, na kufunza mtandao kwenye seti kubwa ya data ili kupata matokeo mazuri. Inaweza kuchukua muda mrefu, hivyo kutumia GPU inapendekezwa.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 3,
|
|
"metadata": {
|
|
"scrolled": true
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"3200: acc=0.3090625\n",
|
|
"6400: acc=0.38921875\n",
|
|
"9600: acc=0.4590625\n",
|
|
"12800: acc=0.511953125\n",
|
|
"16000: acc=0.5506875\n",
|
|
"19200: acc=0.57921875\n",
|
|
"22400: acc=0.6070089285714285\n",
|
|
"25600: acc=0.6304296875\n",
|
|
"28800: acc=0.6484027777777778\n",
|
|
"32000: acc=0.66509375\n",
|
|
"35200: acc=0.6790056818181818\n",
|
|
"38400: acc=0.6929166666666666\n",
|
|
"41600: acc=0.7035817307692308\n",
|
|
"44800: acc=0.7137276785714286\n",
|
|
"48000: acc=0.72225\n",
|
|
"51200: acc=0.73001953125\n",
|
|
"54400: acc=0.7372794117647059\n",
|
|
"57600: acc=0.7436631944444444\n",
|
|
"60800: acc=0.7503947368421052\n",
|
|
"64000: acc=0.75634375\n",
|
|
"67200: acc=0.7615773809523809\n",
|
|
"70400: acc=0.7662642045454545\n",
|
|
"73600: acc=0.7708423913043478\n",
|
|
"76800: acc=0.7751822916666666\n",
|
|
"80000: acc=0.7790625\n",
|
|
"83200: acc=0.7825\n",
|
|
"86400: acc=0.7858564814814815\n",
|
|
"89600: acc=0.7890513392857142\n",
|
|
"92800: acc=0.7920474137931034\n",
|
|
"96000: acc=0.7952708333333334\n",
|
|
"99200: acc=0.7982258064516129\n",
|
|
"102400: acc=0.80099609375\n",
|
|
"105600: acc=0.8037594696969697\n",
|
|
"108800: acc=0.8060569852941176\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=16, collate_fn=padify, shuffle=True)\n",
|
|
"net = RNNClassifier(vocab_size,64,32,len(classes)).to(device)\n",
|
|
"train_epoch(net,train_loader, lr=0.001)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Long Short Term Memory (LSTM)\n",
|
|
"\n",
|
|
"Moja ya matatizo makubwa ya RNN za kawaida ni tatizo linalojulikana kama **vanishing gradients**. Kwa sababu RNN hufunzwa kutoka mwanzo hadi mwisho kwa kutumia mchakato mmoja wa back-propagation, huwa na ugumu wa kusambaza makosa hadi kwenye tabaka za mwanzo za mtandao, na hivyo mtandao hauwezi kujifunza uhusiano kati ya tokeni za mbali. Njia moja ya kuepuka tatizo hili ni kuanzisha **usimamizi wa hali wazi** kwa kutumia kile kinachoitwa **milango**. Kuna usanifu mawili yanayojulikana zaidi wa aina hii: **Long Short Term Memory** (LSTM) na **Gated Relay Unit** (GRU).\n",
|
|
"\n",
|
|
"\n",
|
|
"\n",
|
|
"Mtandao wa LSTM umeandaliwa kwa namna inayofanana na RNN, lakini kuna hali mbili zinazopitishwa kutoka tabaka moja hadi nyingine: hali halisi $c$, na vekta iliyofichwa $h$. Katika kila kitengo, vekta iliyofichwa $h_i$ inaunganishwa na ingizo $x_i$, na vinaamua nini kitafanyika kwa hali $c$ kupitia **milango**. Kila mlango ni mtandao wa neva wenye uanzishaji wa sigmoid (matokeo katika safu $[0,1]$), ambao unaweza kufikiriwa kama maski ya bitwise inapozidishwa na vekta ya hali. Kuna milango ifuatayo (kutoka kushoto kwenda kulia kwenye picha hapo juu):\n",
|
|
"* **mlango wa kusahau** huchukua vekta iliyofichwa na kuamua ni vipengele vipi vya vekta $c$ tunavyohitaji kusahau, na vipi kupitisha.\n",
|
|
"* **mlango wa kuingiza** huchukua baadhi ya taarifa kutoka kwa ingizo na vekta iliyofichwa, na kuziingiza kwenye hali.\n",
|
|
"* **mlango wa kutoa** hubadilisha hali kupitia safu ya mstari yenye uanzishaji wa $\\tanh$, kisha huchagua baadhi ya vipengele vyake kwa kutumia vekta iliyofichwa $h_i$ ili kutoa hali mpya $c_{i+1}$.\n",
|
|
"\n",
|
|
"Vipengele vya hali $c$ vinaweza kufikiriwa kama bendera fulani zinazoweza kuwashwa na kuzimwa. Kwa mfano, tunapokutana na jina *Alice* katika mlolongo, tunaweza kudhani kuwa linahusu mhusika wa kike, na kuinua bendera katika hali inayoonyesha kuwa tuna nomino ya kike katika sentensi. Tunapokutana baadaye na maneno *and Tom*, tutainua bendera inayoonyesha kuwa tuna nomino ya wingi. Hivyo, kwa kudhibiti hali, tunaweza kudumisha ufuatiliaji wa mali za kisarufi za sehemu za sentensi.\n",
|
|
"\n",
|
|
"> **Note**: Chanzo kizuri cha kuelewa undani wa LSTM ni makala hii nzuri [Understanding LSTM Networks](https://colah.github.io/posts/2015-08-Understanding-LSTMs/) iliyoandikwa na Christopher Olah.\n",
|
|
"\n",
|
|
"Ingawa muundo wa ndani wa seli ya LSTM unaweza kuonekana mgumu, PyTorch huficha utekelezaji huu ndani ya darasa `LSTMCell`, na hutoa kitu `LSTM` kuwakilisha safu nzima ya LSTM. Hivyo, utekelezaji wa classifier ya LSTM utakuwa karibu sawa na RNN rahisi ambayo tumeiona hapo juu:\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 4,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"class LSTMClassifier(torch.nn.Module):\n",
|
|
" def __init__(self, vocab_size, embed_dim, hidden_dim, num_class):\n",
|
|
" super().__init__()\n",
|
|
" self.hidden_dim = hidden_dim\n",
|
|
" self.embedding = torch.nn.Embedding(vocab_size, embed_dim)\n",
|
|
" self.embedding.weight.data = torch.randn_like(self.embedding.weight.data)-0.5\n",
|
|
" self.rnn = torch.nn.LSTM(embed_dim,hidden_dim,batch_first=True)\n",
|
|
" self.fc = torch.nn.Linear(hidden_dim, num_class)\n",
|
|
"\n",
|
|
" def forward(self, x):\n",
|
|
" batch_size = x.size(0)\n",
|
|
" x = self.embedding(x)\n",
|
|
" x,(h,c) = self.rnn(x)\n",
|
|
" return self.fc(h[-1])"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": []
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 5,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"3200: acc=0.259375\n",
|
|
"6400: acc=0.25859375\n",
|
|
"9600: acc=0.26177083333333334\n",
|
|
"12800: acc=0.2784375\n",
|
|
"16000: acc=0.313\n",
|
|
"19200: acc=0.3528645833333333\n",
|
|
"22400: acc=0.3965625\n",
|
|
"25600: acc=0.4385546875\n",
|
|
"28800: acc=0.4752777777777778\n",
|
|
"32000: acc=0.505375\n",
|
|
"35200: acc=0.5326704545454546\n",
|
|
"38400: acc=0.5557552083333334\n",
|
|
"41600: acc=0.5760817307692307\n",
|
|
"44800: acc=0.5954910714285714\n",
|
|
"48000: acc=0.6118333333333333\n",
|
|
"51200: acc=0.62681640625\n",
|
|
"54400: acc=0.6404779411764706\n",
|
|
"57600: acc=0.6520138888888889\n",
|
|
"60800: acc=0.662828947368421\n",
|
|
"64000: acc=0.673546875\n",
|
|
"67200: acc=0.6831547619047619\n",
|
|
"70400: acc=0.6917897727272727\n",
|
|
"73600: acc=0.6997146739130434\n",
|
|
"76800: acc=0.707109375\n",
|
|
"80000: acc=0.714075\n",
|
|
"83200: acc=0.7209134615384616\n",
|
|
"86400: acc=0.727037037037037\n",
|
|
"89600: acc=0.7326674107142858\n",
|
|
"92800: acc=0.7379633620689655\n",
|
|
"96000: acc=0.7433645833333333\n",
|
|
"99200: acc=0.7479032258064516\n",
|
|
"102400: acc=0.752119140625\n",
|
|
"105600: acc=0.7562405303030303\n",
|
|
"108800: acc=0.76015625\n",
|
|
"112000: acc=0.7641339285714286\n",
|
|
"115200: acc=0.7677777777777778\n",
|
|
"118400: acc=0.7711233108108108\n"
|
|
]
|
|
},
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"(0.03487814127604167, 0.7728)"
|
|
]
|
|
},
|
|
"execution_count": 5,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"net = LSTMClassifier(vocab_size,64,32,len(classes)).to(device)\n",
|
|
"train_epoch(net,train_loader, lr=0.001)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Sehemu zilizojazwa\n",
|
|
"\n",
|
|
"Katika mfano wetu, tulilazimika kujaza mfuatano wote kwenye kundi dogo kwa kutumia vekta za sifuri. Ingawa hii husababisha upotevu wa kumbukumbu, kwa RNN ni muhimu zaidi kwamba seli za ziada za RNN zinaundwa kwa ajili ya vipengele vya pembejeo vilivyojazwa, ambavyo vinashiriki katika mafunzo lakini havibebi taarifa muhimu za pembejeo. Ingekuwa bora zaidi kufundisha RNN kwa ukubwa halisi wa mfuatano.\n",
|
|
"\n",
|
|
"Ili kufanya hivyo, muundo maalum wa uhifadhi wa mfuatano uliojazwa umetambulishwa katika PyTorch. Tuseme tuna kundi dogo la pembejeo lililojazwa ambalo linaonekana kama hili:\n",
|
|
"```\n",
|
|
"[[1,2,3,4,5],\n",
|
|
" [6,7,8,0,0],\n",
|
|
" [9,0,0,0,0]]\n",
|
|
"```\n",
|
|
"Hapa 0 inawakilisha thamani zilizojazwa, na vekta halisi ya urefu wa mfuatano wa pembejeo ni `[5,3,1]`.\n",
|
|
"\n",
|
|
"Ili kufundisha RNN kwa ufanisi na mfuatano uliojazwa, tunataka kuanza mafunzo ya kundi la kwanza la seli za RNN na kundi kubwa (`[1,6,9]`), lakini kisha kumaliza uchakataji wa mfuatano wa tatu, na kuendelea na mafunzo kwa makundi madogo (`[2,7]`, `[3,8]`), na kadhalika. Kwa hivyo, mfuatano uliojazwa unawakilishwa kama vekta moja - katika kesi yetu `[1,6,9,2,7,3,8,4,5]`, na vekta ya urefu (`[5,3,1]`), ambayo tunaweza kuunda upya kundi dogo la awali lililojazwa kwa urahisi.\n",
|
|
"\n",
|
|
"Ili kuzalisha mfuatano uliojazwa, tunaweza kutumia kazi ya `torch.nn.utils.rnn.pack_padded_sequence`. Tabaka zote za kurudia, ikiwa ni pamoja na RNN, LSTM na GRU, zinaunga mkono mfuatano uliojazwa kama pembejeo, na huzalisha matokeo yaliyofungashwa, ambayo yanaweza kufasiriwa kwa kutumia `torch.nn.utils.rnn.pad_packed_sequence`.\n",
|
|
"\n",
|
|
"Ili kuweza kuzalisha mfuatano uliojazwa, tunahitaji kupitisha vekta ya urefu kwa mtandao, na kwa hivyo tunahitaji kazi tofauti ya kuandaa makundi madogo:\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 6,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"def pad_length(b):\n",
|
|
" # build vectorized sequence\n",
|
|
" v = [encode(x[1]) for x in b]\n",
|
|
" # compute max length of a sequence in this minibatch and length sequence itself\n",
|
|
" len_seq = list(map(len,v))\n",
|
|
" l = max(len_seq)\n",
|
|
" return ( # tuple of three tensors - labels, padded features, length sequence\n",
|
|
" torch.LongTensor([t[0]-1 for t in b]),\n",
|
|
" torch.stack([torch.nn.functional.pad(torch.tensor(t),(0,l-len(t)),mode='constant',value=0) for t in v]),\n",
|
|
" torch.tensor(len_seq)\n",
|
|
" )\n",
|
|
"\n",
|
|
"train_loader_len = torch.utils.data.DataLoader(train_dataset, batch_size=16, collate_fn=pad_length, shuffle=True)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Mtandao halisi utakuwa sawa sana na `LSTMClassifier` hapo juu, lakini mchakato wa `forward` utapokea minibatch iliyopangwa pamoja na vector ya urefu wa mfuatano. Baada ya kuhesabu embedding, tunahesabu mfuatano uliopakiwa, kuupitisha kwenye safu ya LSTM, na kisha kufungua tena matokeo.\n",
|
|
"\n",
|
|
"> **Note**: Kwa kweli hatutumii matokeo yaliyofunguliwa `x`, kwa sababu tunatumia matokeo kutoka kwa tabaka zilizofichwa katika mahesabu yanayofuata. Hivyo basi, tunaweza kuondoa kabisa hatua ya kufungua kutoka kwenye msimbo huu. Sababu ya kuiweka hapa ni ili uweze kurekebisha msimbo huu kwa urahisi, iwapo utahitaji kutumia matokeo ya mtandao katika mahesabu zaidi.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 7,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"class LSTMPackClassifier(torch.nn.Module):\n",
|
|
" def __init__(self, vocab_size, embed_dim, hidden_dim, num_class):\n",
|
|
" super().__init__()\n",
|
|
" self.hidden_dim = hidden_dim\n",
|
|
" self.embedding = torch.nn.Embedding(vocab_size, embed_dim)\n",
|
|
" self.embedding.weight.data = torch.randn_like(self.embedding.weight.data)-0.5\n",
|
|
" self.rnn = torch.nn.LSTM(embed_dim,hidden_dim,batch_first=True)\n",
|
|
" self.fc = torch.nn.Linear(hidden_dim, num_class)\n",
|
|
"\n",
|
|
" def forward(self, x, lengths):\n",
|
|
" batch_size = x.size(0)\n",
|
|
" x = self.embedding(x)\n",
|
|
" pad_x = torch.nn.utils.rnn.pack_padded_sequence(x,lengths,batch_first=True,enforce_sorted=False)\n",
|
|
" pad_x,(h,c) = self.rnn(pad_x)\n",
|
|
" x, _ = torch.nn.utils.rnn.pad_packed_sequence(pad_x,batch_first=True)\n",
|
|
" return self.fc(h[-1])"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": []
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 8,
|
|
"metadata": {
|
|
"scrolled": true
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"3200: acc=0.285625\n",
|
|
"6400: acc=0.33359375\n",
|
|
"9600: acc=0.3876041666666667\n",
|
|
"12800: acc=0.44078125\n",
|
|
"16000: acc=0.4825\n",
|
|
"19200: acc=0.5235416666666667\n",
|
|
"22400: acc=0.5559821428571429\n",
|
|
"25600: acc=0.58609375\n",
|
|
"28800: acc=0.6116666666666667\n",
|
|
"32000: acc=0.63340625\n",
|
|
"35200: acc=0.6525284090909091\n",
|
|
"38400: acc=0.668515625\n",
|
|
"41600: acc=0.6822596153846154\n",
|
|
"44800: acc=0.6948214285714286\n",
|
|
"48000: acc=0.7052708333333333\n",
|
|
"51200: acc=0.71521484375\n",
|
|
"54400: acc=0.7239889705882353\n",
|
|
"57600: acc=0.7315277777777778\n",
|
|
"60800: acc=0.7388486842105263\n",
|
|
"64000: acc=0.74571875\n",
|
|
"67200: acc=0.7518303571428572\n",
|
|
"70400: acc=0.7576988636363636\n",
|
|
"73600: acc=0.7628940217391305\n",
|
|
"76800: acc=0.7681510416666667\n",
|
|
"80000: acc=0.7728125\n",
|
|
"83200: acc=0.7772235576923077\n",
|
|
"86400: acc=0.7815393518518519\n",
|
|
"89600: acc=0.7857700892857142\n",
|
|
"92800: acc=0.7895043103448276\n",
|
|
"96000: acc=0.7930520833333333\n",
|
|
"99200: acc=0.7959072580645161\n",
|
|
"102400: acc=0.798994140625\n",
|
|
"105600: acc=0.802064393939394\n",
|
|
"108800: acc=0.8051378676470589\n",
|
|
"112000: acc=0.8077857142857143\n",
|
|
"115200: acc=0.8104600694444445\n",
|
|
"118400: acc=0.8128293918918919\n"
|
|
]
|
|
},
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"(0.029785829671223958, 0.8138166666666666)"
|
|
]
|
|
},
|
|
"execution_count": 8,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"net = LSTMPackClassifier(vocab_size,64,32,len(classes)).to(device)\n",
|
|
"train_epoch_emb(net,train_loader_len, lr=0.001,use_pack_sequence=True)\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"> **Kumbuka:** Unaweza kuwa umeona kipengele `use_pack_sequence` tunachopitisha kwa kazi ya mafunzo. Kwa sasa, kazi ya `pack_padded_sequence` inahitaji tensor ya urefu wa mlolongo kuwa kwenye kifaa cha CPU, na hivyo kazi ya mafunzo inahitaji kuepuka kuhamisha data ya urefu wa mlolongo kwenda GPU wakati wa mafunzo. Unaweza kuangalia utekelezaji wa kazi ya `train_emb` katika faili [`torchnlp.py`](../../../../../lessons/5-NLP/16-RNN/torchnlp.py).\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## RNN za Mwelekeo Mbili na Tabaka Nyingi\n",
|
|
"\n",
|
|
"Katika mifano yetu, mitandao yote ya kurudiarudia (recurrent networks) ilifanya kazi kwa mwelekeo mmoja, kutoka mwanzo wa mlolongo hadi mwisho. Inaonekana kuwa ya kawaida, kwa sababu inafanana na jinsi tunavyosoma na kusikiliza hotuba. Hata hivyo, kwa kuwa katika hali nyingi za vitendo tunaweza kufikia mlolongo wa pembejeo kwa nasibu, inaweza kuwa na maana kuendesha hesabu ya kurudiarudia katika pande zote mbili. Mitandao kama hiyo inaitwa **RNN za mwelekeo mbili**, na zinaweza kuundwa kwa kupitisha kipengele `bidirectional=True` kwenye mjengaji wa RNN/LSTM/GRU.\n",
|
|
"\n",
|
|
"Unaposhughulika na mtandao wa mwelekeo mbili, tunahitaji vekta mbili za hali ya siri, moja kwa kila mwelekeo. PyTorch inahifadhi vekta hizo kama vekta moja yenye ukubwa mara mbili, jambo ambalo ni rahisi sana, kwa sababu kawaida ungetuma hali ya siri inayotokana kwa safu ya mstari iliyounganishwa kikamilifu, na unahitaji tu kuzingatia ongezeko hili la ukubwa wakati wa kuunda safu hiyo.\n",
|
|
"\n",
|
|
"Mtandao wa kurudiarudia, wa mwelekeo mmoja au wa mwelekeo mbili, unakamata mifumo fulani ndani ya mlolongo, na inaweza kuihifadhi kwenye vekta ya hali au kuipitisha kwenye pato. Kama ilivyo kwa mitandao ya convolutional, tunaweza kujenga safu nyingine ya kurudiarudia juu ya safu ya kwanza ili kukamata mifumo ya kiwango cha juu, iliyojengwa kutoka kwa mifumo ya kiwango cha chini iliyotolewa na safu ya kwanza. Hii inatupeleka kwenye dhana ya **RNN ya tabaka nyingi**, ambayo inajumuisha mitandao miwili au zaidi ya kurudiarudia, ambapo pato la safu ya awali linapitishwa kwa safu inayofuata kama pembejeo.\n",
|
|
"\n",
|
|
"\n",
|
|
"\n",
|
|
"*Picha kutoka [makala hii nzuri](https://towardsdatascience.com/from-a-lstm-cell-to-a-multilayer-lstm-network-with-pytorch-2899eb5696f3) ya Fernando López*\n",
|
|
"\n",
|
|
"PyTorch inafanya ujenzi wa mitandao kama hiyo kuwa kazi rahisi, kwa sababu unahitaji tu kupitisha kipengele `num_layers` kwenye mjengaji wa RNN/LSTM/GRU ili kujenga tabaka kadhaa za kurudiarudia moja kwa moja. Hii pia inamaanisha kuwa ukubwa wa vekta ya hali/siri utaongezeka kwa uwiano, na unahitaji kuzingatia hili wakati wa kushughulikia pato la tabaka za kurudiarudia.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## RNNs kwa kazi nyingine\n",
|
|
"\n",
|
|
"Katika sehemu hii, tumeona kwamba RNNs zinaweza kutumika kwa uainishaji wa mfuatano, lakini kwa kweli, zinaweza kushughulikia kazi nyingi zaidi, kama vile uzalishaji wa maandishi, tafsiri ya mashine, na mengine. Tutazingatia kazi hizo katika sehemu inayofuata.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"\n---\n\n**Kanusho**: \nHati hii imetafsiriwa kwa kutumia huduma ya tafsiri ya AI [Co-op Translator](https://github.com/Azure/co-op-translator). Ingawa tunajitahidi kuhakikisha usahihi, tafsiri za kiotomatiki zinaweza kuwa na makosa au kutokuwa sahihi. Hati ya asili katika lugha yake ya awali inapaswa kuchukuliwa kama chanzo cha mamlaka. Kwa taarifa muhimu, tafsiri ya kitaalamu ya binadamu inapendekezwa. Hatutawajibika kwa kutoelewana au tafsiri zisizo sahihi zinazotokana na matumizi ya tafsiri hii.\n"
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"interpreter": {
|
|
"hash": "16af2a8bbb083ea23e5e41c7f5787656b2ce26968575d8763f2c4b17f9cd711f"
|
|
},
|
|
"kernelspec": {
|
|
"display_name": "Python 3.8.12 ('py38')",
|
|
"language": "python",
|
|
"name": "python3"
|
|
},
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3",
|
|
"version": "3.8.12"
|
|
},
|
|
"coopTranslator": {
|
|
"original_hash": "522ee52ae3d5ae933e283286254e9a55",
|
|
"translation_date": "2025-08-29T16:19:29+00:00",
|
|
"source_file": "lessons/5-NLP/16-RNN/RNNPyTorch.ipynb",
|
|
"language_code": "sw"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 2
|
|
} |