460 lines
23 KiB
Plaintext
460 lines
23 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"# Mitandao ya neva ya kurudia\n",
|
|
"\n",
|
|
"Katika moduli iliyopita, tulijifunza uwakilishi wa kina wa semantiki ya maandishi. Usanifu ambao tumekuwa tukitumia unakamata maana ya jumla ya maneno katika sentensi, lakini hauzingatii **mpangilio** wa maneno, kwa sababu operesheni ya kujumlisha inayofuata embeddings huondoa taarifa hii kutoka kwa maandishi ya asili. Kwa kuwa mifano hii haiwezi kuwakilisha mpangilio wa maneno, haiwezi kutatua kazi ngumu zaidi au zenye utata kama vile uzalishaji wa maandishi au kujibu maswali.\n",
|
|
"\n",
|
|
"Ili kukamata maana ya mlolongo wa maandishi, tutatumia usanifu wa mtandao wa neva unaoitwa **mtandao wa neva wa kurudia**, au RNN. Tunapotumia RNN, tunapitisha sentensi yetu kupitia mtandao tokeni moja kwa wakati, na mtandao huzalisha **hali fulani**, ambayo tunapitisha tena kwa mtandao pamoja na tokeni inayofuata.\n",
|
|
"\n",
|
|
"\n",
|
|
"\n",
|
|
"Kwa kuzingatia mlolongo wa tokeni za ingizo $X_0,\\dots,X_n$, RNN huunda mlolongo wa vizuizi vya mtandao wa neva, na hufundisha mlolongo huu kutoka mwanzo hadi mwisho kwa kutumia backpropagation. Kila kizuizi cha mtandao huchukua jozi $(X_i,S_i)$ kama ingizo, na huzalisha $S_{i+1}$ kama matokeo. Hali ya mwisho $S_n$ au matokeo $Y_n$ huingia kwenye classifier ya mstari ili kutoa matokeo. Vizuizi vyote vya mtandao vinashiriki uzito sawa, na hufundishwa kutoka mwanzo hadi mwisho kwa kutumia mchakato mmoja wa backpropagation.\n",
|
|
"\n",
|
|
"> Mchoro hapo juu unaonyesha mtandao wa neva wa kurudia katika umbo lililofunguliwa (kushoto), na katika uwakilishi wa kurudia ulio kompakt (kulia). Ni muhimu kutambua kwamba Seli zote za RNN zina **uzito unaoweza kushirikishwa**.\n",
|
|
"\n",
|
|
"Kwa sababu vekta za hali $S_0,\\dots,S_n$ zinapitishwa kupitia mtandao, RNN inaweza kujifunza utegemezi wa mlolongo kati ya maneno. Kwa mfano, wakati neno *not* linapojitokeza mahali fulani katika mlolongo, linaweza kujifunza kukanusha vipengele fulani ndani ya vekta ya hali.\n",
|
|
"\n",
|
|
"Ndani, kila seli ya RNN ina matriksi mawili ya uzito: $W_H$ na $W_I$, na upendeleo $b$. Katika kila hatua ya RNN, kwa kuzingatia ingizo $X_i$ na hali ya ingizo $S_i$, hali ya matokeo huhesabiwa kama $S_{i+1} = f(W_H\\times S_i + W_I\\times X_i+b)$, ambapo $f$ ni kazi ya uanzishaji (mara nyingi $\\tanh$).\n",
|
|
"\n",
|
|
"> Kwa matatizo kama uzalishaji wa maandishi (ambayo tutashughulikia katika kitengo kinachofuata) au tafsiri ya mashine, tunataka pia kupata thamani ya matokeo katika kila hatua ya RNN. Katika kesi hii, kuna pia matriksi mengine $W_O$, na matokeo huhesabiwa kama $Y_i=f(W_O\\times S_i+b_O)$.\n",
|
|
"\n",
|
|
"Hebu tuone jinsi mitandao ya neva ya kurudia inaweza kutusaidia kuainisha seti yetu ya data ya habari.\n",
|
|
"\n",
|
|
"> Kwa mazingira ya sandbox, tunahitaji kuendesha seli ifuatayo ili kuhakikisha maktaba inayohitajika imewekwa, na data imepakiwa mapema. Ikiwa unafanya kazi kwa ndani, unaweza kuruka seli ifuatayo.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 1,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"import sys\n",
|
|
"!{sys.executable} -m pip install --quiet tensorflow_datasets==4.4.0\n",
|
|
"!cd ~ && wget -q -O - https://mslearntensorflowlp.blob.core.windows.net/data/tfds-ag-news.tgz | tar xz"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 2,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"import tensorflow as tf\n",
|
|
"from tensorflow import keras\n",
|
|
"import tensorflow_datasets as tfds\n",
|
|
"import numpy as np\n",
|
|
"\n",
|
|
"# We are going to be training pretty large models. In order not to face errors, we need\n",
|
|
"# to set tensorflow option to grow GPU memory allocation when required\n",
|
|
"physical_devices = tf.config.list_physical_devices('GPU') \n",
|
|
"if len(physical_devices)>0:\n",
|
|
" tf.config.experimental.set_memory_growth(physical_devices[0], True)\n",
|
|
"\n",
|
|
"ds_train, ds_test = tfds.load('ag_news_subset').values()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"nteract": {
|
|
"transient": {
|
|
"deleting": false
|
|
}
|
|
}
|
|
},
|
|
"source": [
|
|
"Wakati wa kufundisha mifano mikubwa, ugawaji wa kumbukumbu ya GPU unaweza kuwa tatizo. Pia tunaweza kuhitaji kujaribu ukubwa tofauti wa minibatch, ili data iweze kutoshea kwenye kumbukumbu ya GPU yetu, lakini mafunzo yawe ya haraka vya kutosha. Ikiwa unaendesha msimbo huu kwenye mashine yako ya GPU, unaweza kujaribu kurekebisha ukubwa wa minibatch ili kuharakisha mafunzo.\n",
|
|
"\n",
|
|
"> **Note**: Toleo fulani za madereva ya NVidia zinajulikana kwa kutotoa kumbukumbu baada ya kufundisha mfano. Tunaendesha mifano kadhaa katika daftari hili, na inaweza kusababisha kumbukumbu kuisha katika mipangilio fulani, hasa ikiwa unafanya majaribio yako mwenyewe kama sehemu ya daftari hilo hilo. Ikiwa utakutana na makosa ya ajabu unapoanza kufundisha mfano, unaweza kutaka kuwasha upya kernel ya daftari.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 3,
|
|
"metadata": {
|
|
"collapsed": true,
|
|
"jupyter": {
|
|
"outputs_hidden": false,
|
|
"source_hidden": false
|
|
},
|
|
"nteract": {
|
|
"transient": {
|
|
"deleting": false
|
|
}
|
|
}
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"batch_size = 16\n",
|
|
"embed_size = 64"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Kainishi Rahisi ya RNN\n",
|
|
"\n",
|
|
"Katika hali ya RNN rahisi, kila kitengo cha kurudia ni mtandao rahisi wa mstari, ambao unachukua vekta ya pembejeo na vekta ya hali, na kutoa vekta mpya ya hali. Katika Keras, hii inaweza kuwakilishwa na safu ya `SimpleRNN`.\n",
|
|
"\n",
|
|
"Ingawa tunaweza kupitisha tokeni zilizokodishwa moja kwa moja kwa safu ya RNN, hii si wazo zuri kwa sababu ya ukubwa wao wa juu. Kwa hivyo, tutatumia safu ya embedding kupunguza ukubwa wa vekta za maneno, ikifuatiwa na safu ya RNN, na hatimaye classifier ya `Dense`.\n",
|
|
"\n",
|
|
"> **Note**: Katika hali ambapo ukubwa si mkubwa sana, kwa mfano wakati wa kutumia tokeni za kiwango cha herufi, inaweza kuwa na maana kupitisha tokeni zilizokodishwa moja kwa moja kwenye seli ya RNN.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 4,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Model: \"sequential\"\n",
|
|
"_________________________________________________________________\n",
|
|
"Layer (type) Output Shape Param # \n",
|
|
"=================================================================\n",
|
|
"text_vectorization (TextVect (None, None) 0 \n",
|
|
"_________________________________________________________________\n",
|
|
"embedding (Embedding) (None, None, 64) 1280000 \n",
|
|
"_________________________________________________________________\n",
|
|
"simple_rnn (SimpleRNN) (None, 16) 1296 \n",
|
|
"_________________________________________________________________\n",
|
|
"dense (Dense) (None, 4) 68 \n",
|
|
"=================================================================\n",
|
|
"Total params: 1,281,364\n",
|
|
"Trainable params: 1,281,364\n",
|
|
"Non-trainable params: 0\n",
|
|
"_________________________________________________________________\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"vocab_size = 20000\n",
|
|
"\n",
|
|
"vectorizer = keras.layers.experimental.preprocessing.TextVectorization(\n",
|
|
" max_tokens=vocab_size,\n",
|
|
" input_shape=(1,))\n",
|
|
"\n",
|
|
"model = keras.models.Sequential([\n",
|
|
" vectorizer,\n",
|
|
" keras.layers.Embedding(vocab_size, embed_size),\n",
|
|
" keras.layers.SimpleRNN(16),\n",
|
|
" keras.layers.Dense(4,activation='softmax')\n",
|
|
"])\n",
|
|
"\n",
|
|
"model.summary()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"> **Note:** Tunatumia safu ya embedding isiyofunzwa hapa kwa urahisi, lakini kwa matokeo bora tunaweza kutumia safu ya embedding iliyofunzwa awali kwa kutumia Word2Vec, kama ilivyoelezwa katika sehemu iliyopita. Itakuwa zoezi zuri kwako kubadilisha msimbo huu ili ufanye kazi na embeddings zilizofunzwa awali.\n",
|
|
"\n",
|
|
"Sasa hebu tufunze RNN yetu. Kwa ujumla, RNN ni ngumu sana kufunza, kwa sababu mara seli za RNN zinapopanuliwa kulingana na urefu wa mlolongo, idadi ya tabaka zinazohusika katika backpropagation huwa kubwa sana. Kwa hivyo tunahitaji kuchagua kiwango kidogo cha kujifunza, na kufunza mtandao kwenye seti kubwa ya data ili kupata matokeo mazuri. Hii inaweza kuchukua muda mrefu sana, kwa hivyo kutumia GPU inapendekezwa.\n",
|
|
"\n",
|
|
"Ili kuharakisha mambo, tutafunza tu mfano wa RNN kwenye vichwa vya habari vya habari, tukiacha maelezo. Unaweza kujaribu kufunza na maelezo na kuona kama unaweza kufanya mfano ufanye kazi.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 5,
|
|
"metadata": {
|
|
"scrolled": true
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Training vectorizer\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"def extract_title(x):\n",
|
|
" return x['title']\n",
|
|
"\n",
|
|
"def tupelize_title(x):\n",
|
|
" return (extract_title(x),x['label'])\n",
|
|
"\n",
|
|
"print('Training vectorizer')\n",
|
|
"vectorizer.adapt(ds_train.take(2000).map(extract_title))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 6,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"7500/7500 [==============================] - 82s 11ms/step - loss: 0.6629 - acc: 0.7623 - val_loss: 0.5559 - val_acc: 0.7995\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n"
|
|
]
|
|
},
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"<tensorflow.python.keras.callbacks.History at 0x7f3e0030d350>"
|
|
]
|
|
},
|
|
"execution_count": 6,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"model.compile(loss='sparse_categorical_crossentropy',metrics=['acc'], optimizer='adam')\n",
|
|
"model.fit(ds_train.map(tupelize_title).batch(batch_size),validation_data=ds_test.map(tupelize_title).batch(batch_size))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"nteract": {
|
|
"transient": {
|
|
"deleting": false
|
|
}
|
|
}
|
|
},
|
|
"source": [
|
|
"> **Kumbuka** kwamba usahihi unaweza kuwa wa chini hapa, kwa sababu tunajifunza tu kwa vichwa vya habari vya habari.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Kurudia mfuatano wa vigezo\n",
|
|
"\n",
|
|
"Kumbuka kwamba safu ya `TextVectorization` itaongeza kiotomatiki alama za kujaza (pad tokens) kwenye mfuatano wa urefu tofauti katika kundi dogo. Inageuka kuwa alama hizo pia zinashiriki katika mafunzo, na zinaweza kufanya mchakato wa muundo kufikia matokeo kuwa mgumu zaidi.\n",
|
|
"\n",
|
|
"Kuna mbinu kadhaa tunazoweza kutumia kupunguza kiasi cha kujaza. Mojawapo ni kupanga upya seti ya data kulingana na urefu wa mfuatano na kuunganisha mfuatano wote kulingana na ukubwa. Hii inaweza kufanyika kwa kutumia kazi ya `tf.data.experimental.bucket_by_sequence_length` (tazama [hati](https://www.tensorflow.org/api_docs/python/tf/data/experimental/bucket_by_sequence_length)).\n",
|
|
"\n",
|
|
"Mbinu nyingine ni kutumia **masking**. Katika Keras, baadhi ya safu zinasaidia pembejeo ya ziada inayoonyesha ni alama zipi zinapaswa kuzingatiwa wakati wa mafunzo. Ili kuingiza masking kwenye muundo wetu, tunaweza ama kujumuisha safu tofauti ya `Masking` ([hati](https://keras.io/api/layers/core_layers/masking/)), au tunaweza kubainisha kipengele cha `mask_zero=True` kwenye safu yetu ya `Embedding`.\n",
|
|
"\n",
|
|
"> **Note**: Mafunzo haya yatachukua takriban dakika 5 kukamilisha kipindi kimoja kwenye seti nzima ya data. Usisite kusitisha mafunzo wakati wowote ikiwa utapoteza uvumilivu. Kile unachoweza pia kufanya ni kupunguza kiasi cha data kinachotumika kwa mafunzo, kwa kuongeza kipengele `.take(...)` baada ya seti za data `ds_train` na `ds_test`.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 7,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"7500/7500 [==============================] - 371s 49ms/step - loss: 0.5401 - acc: 0.8079 - val_loss: 0.3780 - val_acc: 0.8822\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n"
|
|
]
|
|
},
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"<tensorflow.python.keras.callbacks.History at 0x7f3dec118850>"
|
|
]
|
|
},
|
|
"execution_count": 7,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"def extract_text(x):\n",
|
|
" return x['title']+' '+x['description']\n",
|
|
"\n",
|
|
"def tupelize(x):\n",
|
|
" return (extract_text(x),x['label'])\n",
|
|
"\n",
|
|
"model = keras.models.Sequential([\n",
|
|
" vectorizer,\n",
|
|
" keras.layers.Embedding(vocab_size,embed_size,mask_zero=True),\n",
|
|
" keras.layers.SimpleRNN(16),\n",
|
|
" keras.layers.Dense(4,activation='softmax')\n",
|
|
"])\n",
|
|
"\n",
|
|
"model.compile(loss='sparse_categorical_crossentropy',metrics=['acc'], optimizer='adam')\n",
|
|
"model.fit(ds_train.map(tupelize).batch(batch_size),validation_data=ds_test.map(tupelize).batch(batch_size))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Sasa kwa kuwa tunatumia masking, tunaweza kufundisha modeli kwa seti nzima ya data ya vichwa vya habari na maelezo.\n",
|
|
"\n",
|
|
"> **Note**: Je, umeona kwamba tumekuwa tukitumia vectorizer iliyofundishwa kwa vichwa vya habari vya habari, na si mwili mzima wa makala? Inawezekana, hili linaweza kusababisha baadhi ya token kupuuzwa, kwa hivyo ni bora kufundisha upya vectorizer. Hata hivyo, huenda hili likawa na athari ndogo sana, kwa hivyo tutatumia vectorizer iliyofundishwa awali kwa urahisi.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## LSTM: Kumbukumbu ya muda mrefu na mfupi\n",
|
|
"\n",
|
|
"Moja ya matatizo makuu ya RNNs ni **kupotea kwa gradients**. RNNs zinaweza kuwa ndefu sana, na zinaweza kuwa na ugumu wa kusambaza gradients hadi kwenye safu ya kwanza ya mtandao wakati wa backpropagation. Hali hii ikitokea, mtandao hauwezi kujifunza uhusiano kati ya tokeni za mbali. Njia moja ya kuepuka tatizo hili ni kuanzisha **usimamizi wa hali wazi** kwa kutumia **milango**. Miundo miwili maarufu inayotumia milango ni **kumbukumbu ya muda mrefu na mfupi** (LSTM) na **kitengo cha relay chenye milango** (GRU). Tutazungumzia LSTMs hapa.\n",
|
|
"\n",
|
|
"\n",
|
|
"\n",
|
|
"Mtandao wa LSTM umeandaliwa kwa namna inayofanana na RNN, lakini kuna hali mbili zinazopitishwa kutoka safu moja hadi nyingine: hali halisi $c$, na vector iliyofichwa $h$. Kwenye kila kitengo, vector iliyofichwa $h_{t-1}$ inaunganishwa na ingizo $x_t$, na pamoja zinadhibiti kinachotokea kwa hali $c_t$ na matokeo $h_{t}$ kupitia **milango**. Kila mlango una uanzishaji wa sigmoid (matokeo katika safu $[0,1]$), ambayo inaweza kufikiriwa kama mask ya biti wakati inazidishwa na vector ya hali. LSTMs zina milango ifuatayo (kutoka kushoto kwenda kulia kwenye picha hapo juu):\n",
|
|
"* **mlango wa kusahau** ambao huamua ni vipengele gani vya vector $c_{t-1}$ tunahitaji kusahau, na vipi kupitisha.\n",
|
|
"* **mlango wa ingizo** ambao huamua ni kiasi gani cha taarifa kutoka kwa vector ya ingizo na vector iliyofichwa ya awali kinapaswa kuingizwa kwenye vector ya hali.\n",
|
|
"* **mlango wa matokeo** ambao huchukua vector mpya ya hali na kuamua ni vipengele gani vya vector hiyo vitatumika kuzalisha vector mpya iliyofichwa $h_t$.\n",
|
|
"\n",
|
|
"Vipengele vya hali $c$ vinaweza kufikiriwa kama bendera zinazoweza kuwashwa na kuzimwa. Kwa mfano, tunapokutana na jina *Alice* katika mlolongo, tunakisia kuwa linahusu mwanamke, na tunainua bendera katika hali inayoonyesha kuwa tuna nomino ya kike katika sentensi. Tunapokutana na maneno *na Tom*, tunainua bendera inayoonyesha kuwa tuna nomino ya wingi. Hivyo, kwa kudhibiti hali tunaweza kufuatilia mali za kisarufi za sentensi.\n",
|
|
"\n",
|
|
"> **Note**: Hapa kuna rasilimali nzuri ya kuelewa undani wa LSTMs: [Understanding LSTM Networks](https://colah.github.io/posts/2015-08-Understanding-LSTMs/) na Christopher Olah.\n",
|
|
"\n",
|
|
"Ingawa muundo wa ndani wa seli ya LSTM unaweza kuonekana kuwa mgumu, Keras huficha utekelezaji huu ndani ya safu ya `LSTM`, kwa hivyo kitu pekee tunachohitaji kufanya katika mfano hapo juu ni kubadilisha safu ya kurudia:\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 8,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"15000/15000 [==============================] - 188s 13ms/step - loss: 0.5692 - acc: 0.7916 - val_loss: 0.3441 - val_acc: 0.8870\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n"
|
|
]
|
|
},
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"<tensorflow.python.keras.callbacks.History at 0x7f3d6af5c350>"
|
|
]
|
|
},
|
|
"execution_count": 8,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"model = keras.models.Sequential([\n",
|
|
" vectorizer,\n",
|
|
" keras.layers.Embedding(vocab_size, embed_size),\n",
|
|
" keras.layers.LSTM(8),\n",
|
|
" keras.layers.Dense(4,activation='softmax')\n",
|
|
"])\n",
|
|
"\n",
|
|
"model.compile(loss='sparse_categorical_crossentropy',metrics=['acc'], optimizer='adam')\n",
|
|
"model.fit(ds_train.map(tupelize).batch(8),validation_data=ds_test.map(tupelize).batch(8))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": []
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## RNN za Mwelekeo Mbili na Tabaka Nyingi\n",
|
|
"\n",
|
|
"Katika mifano yetu hadi sasa, mitandao ya kurudia hufanya kazi kutoka mwanzo wa mlolongo hadi mwisho. Hii inahisi kuwa ya kawaida kwetu kwa sababu inafuata mwelekeo sawa ambao tunasoma au kusikiliza hotuba. Hata hivyo, kwa hali zinazohitaji ufikiaji wa nasibu wa mlolongo wa pembejeo, ina mantiki zaidi kuendesha hesabu ya kurudia katika pande zote mbili. RNNs zinazowezesha hesabu katika pande zote mbili zinaitwa **RNN za mwelekeo mbili**, na zinaweza kuundwa kwa kuzungushia tabaka ya kurudia na tabaka maalum ya `Bidirectional`.\n",
|
|
"\n",
|
|
"> **Note**: Tabaka ya `Bidirectional` hufanya nakala mbili za tabaka ndani yake, na kuweka mali ya `go_backwards` ya moja ya nakala hizo kuwa `True`, ikifanya iende katika mwelekeo kinyume na mlolongo.\n",
|
|
"\n",
|
|
"Mitandao ya kurudia, iwe ya mwelekeo mmoja au mwelekeo mbili, huchukua mifumo ndani ya mlolongo, na kuihifadhi katika vekta za hali au kuzirudisha kama matokeo. Kama ilivyo kwa mitandao ya convolutional, tunaweza kujenga tabaka nyingine ya kurudia kufuatia ya kwanza ili kuchukua mifumo ya kiwango cha juu, iliyojengwa kutoka mifumo ya kiwango cha chini iliyotolewa na tabaka ya kwanza. Hii inatupeleka kwenye dhana ya **RNN ya tabaka nyingi**, ambayo inajumuisha mitandao miwili au zaidi ya kurudia, ambapo matokeo ya tabaka ya awali hupitishwa kwa tabaka inayofuata kama pembejeo.\n",
|
|
"\n",
|
|
"\n",
|
|
"\n",
|
|
"*Picha kutoka [chapisho hili zuri](https://towardsdatascience.com/from-a-lstm-cell-to-a-multilayer-lstm-network-with-pytorch-2899eb5696f3) na Fernando López.*\n",
|
|
"\n",
|
|
"Keras inafanya ujenzi wa mitandao hii kuwa kazi rahisi, kwa sababu unahitaji tu kuongeza tabaka zaidi za kurudia kwenye modeli. Kwa tabaka zote isipokuwa ya mwisho, tunahitaji kubainisha parameter ya `return_sequences=True`, kwa sababu tunahitaji tabaka irudishe hali zote za kati, na siyo hali ya mwisho tu ya hesabu ya kurudia.\n",
|
|
"\n",
|
|
"Hebu tujenge LSTM ya tabaka mbili ya mwelekeo mbili kwa tatizo letu la uainishaji.\n",
|
|
"\n",
|
|
"> **Note**: Msimbo huu tena unachukua muda mrefu kukamilika, lakini unatupa usahihi wa juu zaidi ambao tumeona hadi sasa. Kwa hivyo labda inafaa kusubiri na kuona matokeo.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 9,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"5044/7500 [===================>..........] - ETA: 2:33 - loss: 0.3709 - acc: 0.8706\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r5045/7500 [===================>..........] - ETA: 2:33 - loss: 0.3709 - acc: 0.8706"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"model = keras.models.Sequential([\n",
|
|
" vectorizer,\n",
|
|
" keras.layers.Embedding(vocab_size, 128, mask_zero=True),\n",
|
|
" keras.layers.Bidirectional(keras.layers.LSTM(64,return_sequences=True)),\n",
|
|
" keras.layers.Bidirectional(keras.layers.LSTM(64)), \n",
|
|
" keras.layers.Dense(4,activation='softmax')\n",
|
|
"])\n",
|
|
"\n",
|
|
"model.compile(loss='sparse_categorical_crossentropy',metrics=['acc'], optimizer='adam')\n",
|
|
"model.fit(ds_train.map(tupelize).batch(batch_size),\n",
|
|
" validation_data=ds_test.map(tupelize).batch(batch_size))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## RNNs kwa kazi nyingine\n",
|
|
"\n",
|
|
"Hadi sasa, tumekuwa tukizingatia kutumia RNNs kuainisha mfuatano wa maandishi. Lakini zinaweza kushughulikia kazi nyingi zaidi, kama vile uzalishaji wa maandishi na tafsiri ya lugha — tutazingatia kazi hizo katika kitengo kijacho.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"\n---\n\n**Kanusho**: \nHati hii imetafsiriwa kwa kutumia huduma ya tafsiri ya AI [Co-op Translator](https://github.com/Azure/co-op-translator). Ingawa tunajitahidi kuhakikisha usahihi, tafadhali fahamu kuwa tafsiri za kiotomatiki zinaweza kuwa na makosa au kutokuwa sahihi. Hati ya asili katika lugha yake ya awali inapaswa kuzingatiwa kama chanzo cha mamlaka. Kwa taarifa muhimu, tafsiri ya kitaalamu ya binadamu inapendekezwa. Hatutawajibika kwa kutoelewana au tafsiri zisizo sahihi zinazotokana na matumizi ya tafsiri hii.\n"
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"kernel_info": {
|
|
"name": "conda-env-py37_tensorflow-py"
|
|
},
|
|
"kernelspec": {
|
|
"display_name": "py37_tensorflow",
|
|
"language": "python",
|
|
"name": "conda-env-py37_tensorflow-py"
|
|
},
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3",
|
|
"version": "3.7.9"
|
|
},
|
|
"nteract": {
|
|
"version": "nteract-front-end@1.0.0"
|
|
},
|
|
"coopTranslator": {
|
|
"original_hash": "81351e61f619b432ff51010a4f993194",
|
|
"translation_date": "2025-08-29T16:14:03+00:00",
|
|
"source_file": "lessons/5-NLP/16-RNN/RNNTF.ipynb",
|
|
"language_code": "sw"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 4
|
|
} |