RomanNum3ral
/
AI-For-Beginners
mirror of https://github.com/microsoft/AI-For-Beginners.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
AI-For-Beginners/translations/sw/lessons/5-NLP/14-Embeddings/EmbeddingsPyTorch.ipynb

724 lines
28 KiB
Plaintext
Raw Blame History

{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Uwekaji wa Vifurushi\n",
"\n",
"Katika mfano wetu wa awali, tulifanya kazi na vekta za maneno za hali ya juu zenye urefu wa `vocab_size`, na tulikuwa tunabadilisha wazi kutoka kwa vekta za uwakilishi wa nafasi za hali ya chini kwenda kwenye uwakilishi wa sparse wa one-hot. Uwiano huu wa one-hot si wa ufanisi wa kumbukumbu, na zaidi ya hayo, kila neno linachukuliwa kuwa huru kutoka kwa mengine, yaani, vekta za one-hot hazionyeshi uhusiano wowote wa maana kati ya maneno.\n",
"\n",
"Katika sehemu hii, tutaendelea kuchunguza seti ya data ya **News AG**. Kuanza, hebu tupakie data na tupate baadhi ya ufafanuzi kutoka kwenye daftari la awali.\n"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Loading dataset...\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"d:\\WORK\\ai-for-beginners\\5-NLP\\14-Embeddings\\data\\train.csv: 29.5MB [00:01, 18.8MB/s] \n",
"d:\\WORK\\ai-for-beginners\\5-NLP\\14-Embeddings\\data\\test.csv: 1.86MB [00:00, 11.2MB/s] \n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Building vocab...\n",
"Vocab size = 95812\n"
]
}
],
"source": [
"import torch\n",
"import torchtext\n",
"import numpy as np\n",
"from torchnlp import *\n",
"train_dataset, test_dataset, classes, vocab = load_dataset()\n",
"vocab_size = len(vocab)\n",
"print(\"Vocab size = \",vocab_size)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Je, nini maana ya embedding?\n",
"\n",
"Wazo la **embedding** ni kuwakilisha maneno kwa kutumia vekta zenye vipimo vya chini, ambazo kwa namna fulani zinaonyesha maana ya kisemantiki ya neno. Tutajadili baadaye jinsi ya kujenga embeddings za maneno zenye maana, lakini kwa sasa wacha tuzingatie embedding kama njia ya kupunguza vipimo vya vekta ya neno.\n",
"\n",
"Kwa hivyo, safu ya embedding itachukua neno kama ingizo, na kutoa vekta ya matokeo yenye `embedding_size` maalum. Kwa namna fulani, hii ni sawa na safu ya `Linear`, lakini badala ya kuchukua vekta iliyosimbwa kwa one-hot, itakuwa na uwezo wa kuchukua namba ya neno kama ingizo.\n",
"\n",
"Kwa kutumia safu ya embedding kama safu ya kwanza katika mtandao wetu, tunaweza kubadilisha kutoka mfuko wa maneno (bag-of-words) kwenda kwenye mfano wa **embedding bag**, ambapo tunabadilisha kila neno katika maandishi yetu kuwa embedding inayolingana, kisha tunahesabu kazi fulani ya jumla juu ya embeddings zote hizo, kama vile `sum`, `average` au `max`.\n",
"\n",
"![Picha inayoonyesha classifier ya embedding kwa maneno matano ya mfululizo.](../../../../../translated_images/sw/embedding-classifier-example.b77f021a7ee67eee.webp)\n",
"\n",
"Mtandao wetu wa neva wa kuainisha utaanza na safu ya embedding, kisha safu ya jumlisho, na classifier ya linear juu yake:\n"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
"class EmbedClassifier(torch.nn.Module):\n",
" def __init__(self, vocab_size, embed_dim, num_class):\n",
" super().__init__()\n",
" self.embedding = torch.nn.Embedding(vocab_size, embed_dim)\n",
" self.fc = torch.nn.Linear(embed_dim, num_class)\n",
"\n",
" def forward(self, x):\n",
" x = self.embedding(x)\n",
" x = torch.mean(x,dim=1)\n",
" return self.fc(x)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Kushughulikia Ukubwa wa Mfuatano wa Vigezo\n",
"\n",
"Kutokana na usanifu huu, minibatches kwa mtandao wetu zitahitaji kuundwa kwa njia maalum. Katika sehemu iliyopita, tulipotumia bag-of-words, tensor zote za BoW katika minibatch zilikuwa na ukubwa sawa `vocab_size`, bila kujali urefu halisi wa mfuatano wa maandishi yetu. Mara tu tunapohamia kwenye word embeddings, tutakuwa na idadi tofauti ya maneno katika kila sampuli ya maandishi, na tunapochanganya sampuli hizo katika minibatches tutalazimika kutumia padding fulani.\n",
"\n",
"Hili linaweza kufanyika kwa kutumia mbinu ile ile ya kutoa kazi ya `collate_fn` kwa chanzo cha data:\n"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
"def padify(b):\n",
" # b is the list of tuples of length batch_size\n",
" # - first element of a tuple = label, \n",
" # - second = feature (text sequence)\n",
" # build vectorized sequence\n",
" v = [encode(x[1]) for x in b]\n",
" # first, compute max length of a sequence in this minibatch\n",
" l = max(map(len,v))\n",
" return ( # tuple of two tensors - labels and features\n",
" torch.LongTensor([t[0]-1 for t in b]),\n",
" torch.stack([torch.nn.functional.pad(torch.tensor(t),(0,l-len(t)),mode='constant',value=0) for t in v])\n",
" )\n",
"\n",
"train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=16, collate_fn=padify, shuffle=True)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Kufundisha classifier ya embedding\n",
"\n",
"Sasa kwa kuwa tumefafanua dataloader sahihi, tunaweza kufundisha modeli kwa kutumia kazi ya mafunzo tuliyofafanua katika kitengo kilichopita:\n"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"3200: acc=0.6415625\n",
"6400: acc=0.6865625\n",
"9600: acc=0.7103125\n",
"12800: acc=0.726953125\n",
"16000: acc=0.739375\n",
"19200: acc=0.75046875\n",
"22400: acc=0.7572321428571429\n"
]
},
{
"data": {
"text/plain": [
"(0.889799795315499, 0.7623160588611644)"
]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"net = EmbedClassifier(vocab_size,32,len(classes)).to(device)\n",
"train_epoch(net,train_loader, lr=1, epoch_size=25000)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"> **Kumbuka**: Tunafanya mafunzo kwa rekodi 25k tu hapa (chini ya kipindi kimoja kamili) kwa sababu ya muda, lakini unaweza kuendelea kufundisha, andika kazi ya kufundisha kwa vipindi kadhaa, na jaribu na kigezo cha kiwango cha kujifunza ili kufikia usahihi wa juu. Unapaswa kuwa na uwezo wa kufikia usahihi wa takriban 90%.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Tabaka la EmbeddingBag na Uwakilishi wa Mfuatano wa Urefu Tofauti\n",
"\n",
"Katika usanifu wa awali, tulihitaji kuongeza urefu wa mfuatano wote ili kufanana na urefu mmoja kwa ajili ya kuingiza kwenye kundi dogo (minibatch). Hii si njia bora zaidi ya kuwakilisha mfuatano wa urefu tofauti - njia nyingine inaweza kuwa kutumia **offset** vector, ambayo itahifadhi nafasi za mfuatano wote uliowekwa kwenye vector moja kubwa.\n",
"\n",
"![Picha inayoonyesha uwakilishi wa mfuatano wa offset](../../../../../translated_images/sw/offset-sequence-representation.eb73fcefb29b46ee.webp)\n",
"\n",
"> **Note**: Katika picha hapo juu, tunaonyesha mfuatano wa herufi, lakini katika mfano wetu tunafanya kazi na mfuatano wa maneno. Hata hivyo, kanuni ya jumla ya kuwakilisha mfuatano kwa kutumia offset vector inabaki ile ile.\n",
"\n",
"Ili kufanya kazi na uwakilishi wa offset, tunatumia tabaka [`EmbeddingBag`](https://pytorch.org/docs/stable/generated/torch.nn.EmbeddingBag.html). Tabaka hili linafanana na `Embedding`, lakini linachukua vector ya maudhui na vector ya offset kama pembejeo, na pia linajumuisha tabaka ya wastani, ambayo inaweza kuwa `mean`, `sum` au `max`.\n",
"\n",
"Hapa kuna mtandao uliorekebishwa unaotumia `EmbeddingBag`:\n"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [],
"source": [
"class EmbedClassifier(torch.nn.Module):\n",
" def __init__(self, vocab_size, embed_dim, num_class):\n",
" super().__init__()\n",
" self.embedding = torch.nn.EmbeddingBag(vocab_size, embed_dim)\n",
" self.fc = torch.nn.Linear(embed_dim, num_class)\n",
"\n",
" def forward(self, text, off):\n",
" x = self.embedding(text, off)\n",
" return self.fc(x)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Ili kuandaa seti ya data kwa mafunzo, tunahitaji kutoa kazi ya ubadilishaji ambayo itaandaa vector ya upendeleo:\n"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [],
"source": [
"def offsetify(b):\n",
" # first, compute data tensor from all sequences\n",
" x = [torch.tensor(encode(t[1])) for t in b]\n",
" # now, compute the offsets by accumulating the tensor of sequence lengths\n",
" o = [0] + [len(t) for t in x]\n",
" o = torch.tensor(o[:-1]).cumsum(dim=0)\n",
" return ( \n",
" torch.LongTensor([t[0]-1 for t in b]), # labels\n",
" torch.cat(x), # text \n",
" o\n",
" )\n",
"\n",
"train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=16, collate_fn=offsetify, shuffle=True)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Kumbuka, kwamba tofauti na mifano yote ya awali, mtandao wetu sasa unakubali vigezo viwili: vector ya data na vector ya offset, ambazo zina ukubwa tofauti. Vivyo hivyo, kipakiaji chetu cha data pia kinatupatia thamani 3 badala ya 2: vector za maandishi na offset zinatolewa kama vipengele. Kwa hivyo, tunahitaji kurekebisha kidogo kazi yetu ya mafunzo ili kushughulikia hilo:\n"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"3200: acc=0.6153125\n",
"6400: acc=0.6615625\n",
"9600: acc=0.6932291666666667\n",
"12800: acc=0.715078125\n",
"16000: acc=0.7270625\n",
"19200: acc=0.7382291666666667\n",
"22400: acc=0.7486160714285715\n"
]
},
{
"data": {
"text/plain": [
"(22.771553103007037, 0.7551983365323096)"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"net = EmbedClassifier(vocab_size,32,len(classes)).to(device)\n",
"\n",
"def train_epoch_emb(net,dataloader,lr=0.01,optimizer=None,loss_fn = torch.nn.CrossEntropyLoss(),epoch_size=None, report_freq=200):\n",
" optimizer = optimizer or torch.optim.Adam(net.parameters(),lr=lr)\n",
" loss_fn = loss_fn.to(device)\n",
" net.train()\n",
" total_loss,acc,count,i = 0,0,0,0\n",
" for labels,text,off in dataloader:\n",
" optimizer.zero_grad()\n",
" labels,text,off = labels.to(device), text.to(device), off.to(device)\n",
" out = net(text, off)\n",
" loss = loss_fn(out,labels) #cross_entropy(out,labels)\n",
" loss.backward()\n",
" optimizer.step()\n",
" total_loss+=loss\n",
" _,predicted = torch.max(out,1)\n",
" acc+=(predicted==labels).sum()\n",
" count+=len(labels)\n",
" i+=1\n",
" if i%report_freq==0:\n",
" print(f\"{count}: acc={acc.item()/count}\")\n",
" if epoch_size and count>epoch_size:\n",
" break\n",
" return total_loss.item()/count, acc.item()/count\n",
"\n",
"\n",
"train_epoch_emb(net,train_loader, lr=4, epoch_size=25000)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Uwakilishi wa Semantiki: Word2Vec\n",
"\n",
"Katika mfano wetu wa awali, safu ya kuingiza ya modeli ilijifunza kuwakilisha maneno kwa njia ya vekta, hata hivyo, uwakilishi huu haukuwa na maana ya kisemantiki sana. Ingekuwa vizuri kujifunza uwakilishi wa vekta ambapo maneno yanayofanana au visawe vinahusiana na vekta zilizo karibu kwa mujibu wa umbali fulani wa vekta (mfano, umbali wa euclidian).\n",
"\n",
"Ili kufanikisha hilo, tunahitaji kufundisha modeli yetu ya kuingiza kwa maandishi mengi kwa njia maalum. Mojawapo ya njia za kwanza za kufundisha uwakilishi wa semantiki inaitwa [Word2Vec](https://en.wikipedia.org/wiki/Word2vec). Inategemea usanifu kuu mbili zinazotumika kuzalisha uwakilishi wa maneno ulioenea:\n",
"\n",
" - **Mfuko endelevu wa maneno** (CBoW) — katika usanifu huu, tunafundisha modeli kutabiri neno kutoka muktadha wa maneno yanayozunguka. Kwa kuzingatia ngram $(W_{-2},W_{-1},W_0,W_1,W_2)$, lengo la modeli ni kutabiri $W_0$ kutoka $(W_{-2},W_{-1},W_1,W_2)$.\n",
" - **Skip-gram endelevu** ni kinyume cha CBoW. Modeli hutumia dirisha la muktadha wa maneno yanayozunguka kutabiri neno la sasa.\n",
"\n",
"CBoW ni ya haraka, wakati skip-gram ni ya polepole, lakini inafanya kazi bora ya kuwakilisha maneno yasiyo ya kawaida.\n",
"\n",
"![Picha inayoonyesha algorithimu za CBoW na Skip-Gram za kubadilisha maneno kuwa vekta.](../../../../../translated_images/sw/example-algorithms-for-converting-words-to-vectors.fbe9207a726922f6.webp)\n",
"\n",
"Ili kujaribu kuingiza kwa Word2Vec iliyofundishwa awali kwenye seti ya data ya Google News, tunaweza kutumia maktaba ya **gensim**. Hapa chini tunapata maneno yanayofanana zaidi na 'neural'\n",
"\n",
"> **Note:** Unapounda vekta za maneno kwa mara ya kwanza, kuzidownload kunaweza kuchukua muda!\n"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [],
"source": [
"import gensim.downloader as api\n",
"w2v = api.load('word2vec-google-news-300')"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"neuronal -> 0.7804799675941467\n",
"neurons -> 0.7326500415802002\n",
"neural_circuits -> 0.7252851724624634\n",
"neuron -> 0.7174385190010071\n",
"cortical -> 0.6941086649894714\n",
"brain_circuitry -> 0.6923246383666992\n",
"synaptic -> 0.6699118614196777\n",
"neural_circuitry -> 0.6638563275337219\n",
"neurochemical -> 0.6555314064025879\n",
"neuronal_activity -> 0.6531826257705688\n"
]
}
],
"source": [
"for w,p in w2v.most_similar('neural'):\n",
" print(f\"{w} -> {p}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Tunaweza pia kuhesabu embeddings za vector kutoka kwa neno, zitakazotumika katika kufundisha modeli ya uainishaji (tunaonyesha tu sehemu 20 za kwanza za vector kwa uwazi):\n"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([ 0.01226807, 0.06225586, 0.10693359, 0.05810547, 0.23828125,\n",
" 0.03686523, 0.05151367, -0.20703125, 0.01989746, 0.10058594,\n",
" -0.03759766, -0.1015625 , -0.15820312, -0.08105469, -0.0390625 ,\n",
" -0.05053711, 0.16015625, 0.2578125 , 0.10058594, -0.25976562],\n",
" dtype=float32)"
]
},
"execution_count": 10,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"w2v.word_vec('play')[:20]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Jambo kubwa kuhusu upachikaji wa kisemantiki ni kwamba unaweza kudhibiti usimbaji wa vekta kubadilisha maana. Kwa mfano, tunaweza kuomba kutafuta neno, ambalo uwakilishi wake wa vekta ungekuwa karibu iwezekanavyo na maneno *mfalme* na *mwanamke*, na mbali na neno *mwanaume*:\n"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"('queen', 0.7118192911148071)"
]
},
"execution_count": 10,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"w2v.most_similar(positive=['king','woman'],negative=['man'])[0]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Mbinu za CBoW na Skip-Grams ni \"utabiri\" wa embeddings, kwa kuwa zinazingatia tu muktadha wa karibu. Word2Vec haichukui faida ya muktadha wa jumla.\n",
"\n",
"**FastText**, inajengwa juu ya Word2Vec kwa kujifunza uwakilishi wa vekta kwa kila neno na n-grams za herufi zinazopatikana ndani ya kila neno. Thamani za uwakilishi zinajumlishwa kuwa vekta moja katika kila hatua ya mafunzo. Ingawa hii inaongeza hesabu nyingi za ziada wakati wa mafunzo ya awali, inaruhusu embeddings za maneno kuhifadhi taarifa za sehemu za maneno.\n",
"\n",
"Njia nyingine, **GloVe**, inatumia wazo la matriki ya ushirikiano, ikitumia mbinu za neva kugawa matriki ya ushirikiano kuwa vekta za maneno zenye maelezo zaidi na zisizo za mstari.\n",
"\n",
"Unaweza kucheza na mfano kwa kubadilisha embeddings kuwa FastText na GloVe, kwa kuwa gensim inaunga mkono mifano mbalimbali ya embeddings za maneno.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Kutumia Embeddings Zilizofunzwa Awali katika PyTorch\n",
"\n",
"Tunaweza kurekebisha mfano hapo juu ili kujaza awali matriki katika safu yetu ya embedding kwa kutumia embeddings za maana, kama Word2Vec. Tunapaswa kuzingatia kwamba msamiati wa embeddings zilizofunzwa awali na msamiati wa maandishi yetu huenda usifanane, kwa hivyo tutaweka uzito wa maneno yanayokosekana kwa thamani za nasibu:\n"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {
"tags": []
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Embedding size: 300\n",
"Populating matrix, this will take some time...Done, found 41080 words, 54732 words missing\n"
]
}
],
"source": [
"embed_size = len(w2v.get_vector('hello'))\n",
"print(f'Embedding size: {embed_size}')\n",
"\n",
"net = EmbedClassifier(vocab_size,embed_size,len(classes))\n",
"\n",
"print('Populating matrix, this will take some time...',end='')\n",
"found, not_found = 0,0\n",
"for i,w in enumerate(vocab.get_itos()):\n",
" try:\n",
" net.embedding.weight[i].data = torch.tensor(w2v.get_vector(w))\n",
" found+=1\n",
" except:\n",
" net.embedding.weight[i].data = torch.normal(0.0,1.0,(embed_size,))\n",
" not_found+=1\n",
"\n",
"print(f\"Done, found {found} words, {not_found} words missing\")\n",
"net = net.to(device)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Sasa hebu tufunze modeli yetu. Kumbuka kwamba muda unaochukua kufundisha modeli ni mkubwa zaidi kuliko katika mfano wa awali, kutokana na ukubwa mkubwa wa safu ya embedding, na hivyo idadi kubwa zaidi ya vigezo. Pia, kwa sababu ya hili, huenda tukahitaji kufundisha modeli yetu kwa mifano zaidi ikiwa tunataka kuepuka overfitting.\n"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"3200: acc=0.6359375\n",
"6400: acc=0.68109375\n",
"9600: acc=0.7067708333333333\n",
"12800: acc=0.723671875\n",
"16000: acc=0.73625\n",
"19200: acc=0.7463541666666667\n",
"22400: acc=0.7560714285714286\n"
]
},
{
"data": {
"text/plain": [
"(214.1013875559821, 0.7626759436980166)"
]
},
"execution_count": 12,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"train_epoch_emb(net,train_loader, lr=4, epoch_size=25000)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Katika hali yetu hatuoni ongezeko kubwa la usahihi, jambo ambalo huenda linatokana na tofauti kubwa za msamiati. \n",
"Ili kushinda tatizo la tofauti za msamiati, tunaweza kutumia mojawapo ya suluhisho zifuatazo: \n",
"* Kufunza upya modeli ya word2vec kwa kutumia msamiati wetu \n",
"* Kupakia seti yetu ya data kwa kutumia msamiati kutoka kwenye modeli ya word2vec iliyofunzwa tayari. Msamiati unaotumika kupakia seti ya data unaweza kubainishwa wakati wa kupakia. \n",
"\n",
"Njia ya pili inaonekana rahisi zaidi, hasa kwa sababu mfumo wa PyTorch `torchtext` una msaada wa kujengwa ndani kwa embeddings. Kwa mfano, tunaweza kuanzisha msamiati unaotegemea GloVe kwa njia ifuatayo: \n"
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"100%|█████████▉| 399999/400000 [00:15<00:00, 25411.14it/s]\n"
]
}
],
"source": [
"vocab = torchtext.vocab.GloVe(name='6B', dim=50)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Msamiati uliojazwa una shughuli zifuatazo za msingi: \n",
"* Kamusi ya `vocab.stoi` inatuwezesha kubadilisha neno kuwa faharasa yake ya kamusi. \n",
"* `vocab.itos` hufanya kinyume chake - hubadilisha namba kuwa neno. \n",
"* `vocab.vectors` ni safu ya vekta za uwekaji, kwa hivyo ili kupata uwekaji wa neno `s` tunahitaji kutumia `vocab.vectors[vocab.stoi[s]]`. \n",
"\n",
"Hapa kuna mfano wa kudhibiti uwekaji ili kuonyesha usawa **kind-man+woman = queen** (nililazimika kurekebisha kidogo mgawo ili ifanye kazi): \n"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"'queen'"
]
},
"execution_count": 15,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# get the vector corresponding to kind-man+woman\n",
"qvec = vocab.vectors[vocab.stoi['king']]-vocab.vectors[vocab.stoi['man']]+1.3*vocab.vectors[vocab.stoi['woman']]\n",
"# find the index of the closest embedding vector \n",
"d = torch.sum((vocab.vectors-qvec)**2,dim=1)\n",
"min_idx = torch.argmin(d)\n",
"# find the corresponding word\n",
"vocab.itos[min_idx]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Ili kufundisha classifier kwa kutumia embeddings hizo, tunahitaji kwanza kusimba seti yetu ya data kwa kutumia msamiati wa GloVe:\n"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {},
"outputs": [],
"source": [
"def offsetify(b):\n",
" # first, compute data tensor from all sequences\n",
" x = [torch.tensor(encode(t[1],voc=vocab)) for t in b] # pass the instance of vocab to encode function!\n",
" # now, compute the offsets by accumulating the tensor of sequence lengths\n",
" o = [0] + [len(t) for t in x]\n",
" o = torch.tensor(o[:-1]).cumsum(dim=0)\n",
" return ( \n",
" torch.LongTensor([t[0]-1 for t in b]), # labels\n",
" torch.cat(x), # text \n",
" o\n",
" )"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Kama tulivyoona hapo juu, viambatisho vyote vya vector vinahifadhiwa katika matriki ya `vocab.vectors`. Hii inafanya iwe rahisi sana kupakia uzito huo kwenye uzito wa safu ya viambatisho kwa kutumia kunakili rahisi:\n"
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {},
"outputs": [],
"source": [
"net = EmbedClassifier(len(vocab),len(vocab.vectors[0]),len(classes))\n",
"net.embedding.weight.data = vocab.vectors\n",
"net = net.to(device)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": []
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"3200: acc=0.6271875\n",
"6400: acc=0.68078125\n",
"9600: acc=0.7030208333333333\n",
"12800: acc=0.71984375\n",
"16000: acc=0.7346875\n",
"19200: acc=0.7455729166666667\n",
"22400: acc=0.7529464285714286\n"
]
},
{
"data": {
"text/plain": [
"(35.53972978646833, 0.7575175943698017)"
]
},
"execution_count": 18,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=16, collate_fn=offsetify, shuffle=True)\n",
"train_epoch_emb(net,train_loader, lr=4, epoch_size=25000)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Moja ya sababu hatuoni ongezeko kubwa la usahihi ni kwa sababu baadhi ya maneno kutoka kwenye seti yetu ya data yanakosekana katika msamiati wa GloVe uliotangulizwa, na hivyo yanapuuzwa kimsingi. Ili kushinda hali hii, tunaweza kufundisha embeddings zetu wenyewe kwenye seti yetu ya data.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Uwakilishi wa Muktadha wa Maneno\n",
"\n",
"Kikwazo kimoja kikubwa cha uwakilishi wa jadi wa embeddings zilizofunzwa awali kama Word2Vec ni tatizo la kutofautisha maana ya maneno kulingana na muktadha. Ingawa embeddings zilizofunzwa awali zinaweza kunasa baadhi ya maana ya maneno katika muktadha, kila maana inayowezekana ya neno huwakilishwa katika embedding moja. Hili linaweza kusababisha changamoto katika mifano inayofuata, kwa kuwa maneno mengi kama 'play' yana maana tofauti kulingana na muktadha yanapotumika.\n",
"\n",
"Kwa mfano, neno 'play' katika sentensi hizi mbili lina maana tofauti kabisa:\n",
"- Nilikwenda kwenye **play** ukumbini.\n",
"- John anataka **play** na marafiki zake.\n",
"\n",
"Embeddings zilizofunzwa awali hapo juu zinawakilisha maana zote mbili za neno 'play' katika embedding moja. Ili kushinda kikwazo hiki, tunahitaji kujenga embeddings kulingana na **mfano wa lugha**, ambao umefunzwa kwenye mkusanyiko mkubwa wa maandishi, na *unajua* jinsi maneno yanavyoweza kuunganishwa katika muktadha tofauti. Kujadili uwakilishi wa muktadha wa maneno ni nje ya mada ya mafunzo haya, lakini tutarudi kwenye mada hii tutakapozungumzia mifano ya lugha katika sehemu inayofuata.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n---\n\n**Kanusho**: \nHati hii imetafsiriwa kwa kutumia huduma ya tafsiri ya AI [Co-op Translator](https://github.com/Azure/co-op-translator). Ingawa tunajitahidi kuhakikisha usahihi, tafadhali fahamu kuwa tafsiri za kiotomatiki zinaweza kuwa na makosa au kutokuwa sahihi. Hati asilia katika lugha yake ya awali inapaswa kuchukuliwa kama chanzo cha mamlaka. Kwa taarifa muhimu, tafsiri ya kitaalamu ya binadamu inapendekezwa. Hatutawajibika kwa kutoelewana au tafsiri zisizo sahihi zinazotokana na matumizi ya tafsiri hii.\n"
]
}
],
"metadata": {
"interpreter": {
"hash": "0cb620c6d4b9f7a635928804c26cf22403d89d98d79684e4529119355ee6d5a5"
},
"kernelspec": {
"display_name": "py37_pytorch",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.12"
},
"coopTranslator": {
"original_hash": "f50b026abce5cf36783a560ea72cb9b1",
"translation_date": "2025-08-29T16:33:11+00:00",
"source_file": "lessons/5-NLP/14-Embeddings/EmbeddingsPyTorch.ipynb",
"language_code": "sw"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
Reference in New Issue View Git Blame Copy Permalink