577 lines
21 KiB
Plaintext
577 lines
21 KiB
Plaintext
{
|
||
"cells": [
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"# Kazi ya uainishaji wa maandishi\n",
|
||
"\n",
|
||
"Kama tulivyotaja, tutazingatia kazi rahisi ya uainishaji wa maandishi kwa kutumia seti ya data ya **AG_NEWS**, ambayo ni kuainisha vichwa vya habari vya habari katika mojawapo ya makundi 4: Dunia, Michezo, Biashara, na Sayansi/Teknolojia.\n",
|
||
"\n",
|
||
"## Seti ya Data\n",
|
||
"\n",
|
||
"Seti hii ya data imejumuishwa ndani ya moduli ya [`torchtext`](https://github.com/pytorch/text), kwa hivyo tunaweza kuipata kwa urahisi.\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 1,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"import torch\n",
|
||
"import torchtext\n",
|
||
"import os\n",
|
||
"import collections\n",
|
||
"os.makedirs('./data',exist_ok=True)\n",
|
||
"train_dataset, test_dataset = torchtext.datasets.AG_NEWS(root='./data')\n",
|
||
"classes = ['World', 'Sports', 'Business', 'Sci/Tech']"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"Hapa, `train_dataset` na `test_dataset` zina makusanyo yanayorudisha jozi za lebo (nambari ya darasa) na maandishi mtawalia, kwa mfano:\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 2,
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"data": {
|
||
"text/plain": [
|
||
"(3,\n",
|
||
" \"Wall St. Bears Claw Back Into the Black (Reuters) Reuters - Short-sellers, Wall Street's dwindling\\\\band of ultra-cynics, are seeing green again.\")"
|
||
]
|
||
},
|
||
"execution_count": 2,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"list(train_dataset)[0]"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"Kwa hivyo, wacha tuchapishe vichwa vya habari 10 vya kwanza kutoka kwenye seti yetu ya data:\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 5,
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"**Sci/Tech** -> Wall St. Bears Claw Back Into the Black (Reuters) Reuters - Short-sellers, Wall Street's dwindling\\band of ultra-cynics, are seeing green again.\n",
|
||
"**Sci/Tech** -> Carlyle Looks Toward Commercial Aerospace (Reuters) Reuters - Private investment firm Carlyle Group,\\which has a reputation for making well-timed and occasionally\\controversial plays in the defense industry, has quietly placed\\its bets on another part of the market.\n",
|
||
"**Sci/Tech** -> Oil and Economy Cloud Stocks' Outlook (Reuters) Reuters - Soaring crude prices plus worries\\about the economy and the outlook for earnings are expected to\\hang over the stock market next week during the depth of the\\summer doldrums.\n",
|
||
"**Sci/Tech** -> Iraq Halts Oil Exports from Main Southern Pipeline (Reuters) Reuters - Authorities have halted oil export\\flows from the main pipeline in southern Iraq after\\intelligence showed a rebel militia could strike\\infrastructure, an oil official said on Saturday.\n",
|
||
"**Sci/Tech** -> Oil prices soar to all-time record, posing new menace to US economy (AFP) AFP - Tearaway world oil prices, toppling records and straining wallets, present a new economic menace barely three months before the US presidential elections.\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"for i,x in zip(range(5),train_dataset):\n",
|
||
" print(f\"**{classes[x[0]]}** -> {x[1]}\")\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"Kwa sababu seti za data ni iterators, ikiwa tunataka kutumia data mara nyingi tunahitaji kuibadilisha kuwa orodha:\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 3,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"train_dataset, test_dataset = torchtext.datasets.AG_NEWS(root='./data')\n",
|
||
"train_dataset = list(train_dataset)\n",
|
||
"test_dataset = list(test_dataset)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## Uwekaji Tokeni\n",
|
||
"\n",
|
||
"Sasa tunahitaji kubadilisha maandishi kuwa **nambari** ambazo zinaweza kuwakilishwa kama tensors. Ikiwa tunataka uwakilishi wa kiwango cha maneno, tunahitaji kufanya mambo mawili:\n",
|
||
"* tumia **tokenizer** kugawanya maandishi katika **tokeni**\n",
|
||
"* tengeneza **msamiati** wa tokeni hizo.\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 4,
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"data": {
|
||
"text/plain": [
|
||
"['he', 'said', 'hello']"
|
||
]
|
||
},
|
||
"execution_count": 4,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"tokenizer = torchtext.data.utils.get_tokenizer('basic_english')\n",
|
||
"tokenizer('He said: hello')"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 5,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"counter = collections.Counter()\n",
|
||
"for (label, line) in train_dataset:\n",
|
||
" counter.update(tokenizer(line))\n",
|
||
"vocab = torchtext.vocab.vocab(counter, min_freq=1)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"Kutumia msamiati, tunaweza kwa urahisi kuweka msimbo mfululizo wetu uliotenganishwa kuwa seti ya nambari:\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 19,
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Vocab size if 95810\n"
|
||
]
|
||
},
|
||
{
|
||
"data": {
|
||
"text/plain": [
|
||
"[599, 3279, 97, 1220, 329, 225, 7368]"
|
||
]
|
||
},
|
||
"execution_count": 19,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"vocab_size = len(vocab)\n",
|
||
"print(f\"Vocab size if {vocab_size}\")\n",
|
||
"\n",
|
||
"stoi = vocab.get_stoi() # dict to convert tokens to indices\n",
|
||
"\n",
|
||
"def encode(x):\n",
|
||
" return [stoi[s] for s in tokenizer(x)]\n",
|
||
"\n",
|
||
"encode('I love to play with my words')"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## Uwakilishi wa maandishi kwa kutumia Bag of Words\n",
|
||
"\n",
|
||
"Kwa sababu maneno yanawakilisha maana, wakati mwingine tunaweza kuelewa maana ya maandishi kwa kuangalia tu maneno ya kibinafsi, bila kujali mpangilio wake katika sentensi. Kwa mfano, wakati wa kuainisha habari, maneno kama *hali ya hewa*, *theluji* yanaweza kuashiria *utabiri wa hali ya hewa*, wakati maneno kama *hisa*, *dola* yanaweza kuelekea kwenye *habari za kifedha*.\n",
|
||
"\n",
|
||
"**Bag of Words** (BoW) ni uwakilishi wa vector unaotumika sana katika mbinu za jadi. Kila neno linaunganishwa na faharasa ya vector, na kipengele cha vector kinaonyesha idadi ya mara neno fulani linavyotokea katika hati fulani.\n",
|
||
"\n",
|
||
" \n",
|
||
"\n",
|
||
"> **Note**: Unaweza pia kufikiria BoW kama jumla ya vectors zote zilizowakilishwa kwa njia ya one-hot-encoding kwa maneno ya kibinafsi katika maandishi.\n",
|
||
"\n",
|
||
"Hapa chini kuna mfano wa jinsi ya kuunda uwakilishi wa Bag of Words kwa kutumia maktaba ya python ya Scikit Learn:\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 7,
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"data": {
|
||
"text/plain": [
|
||
"array([[1, 1, 0, 2, 0, 0, 0, 0, 0]], dtype=int64)"
|
||
]
|
||
},
|
||
"execution_count": 7,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"from sklearn.feature_extraction.text import CountVectorizer\n",
|
||
"vectorizer = CountVectorizer()\n",
|
||
"corpus = [\n",
|
||
" 'I like hot dogs.',\n",
|
||
" 'The dog ran fast.',\n",
|
||
" 'Its hot outside.',\n",
|
||
" ]\n",
|
||
"vectorizer.fit_transform(corpus)\n",
|
||
"vectorizer.transform(['My dog likes hot dogs on a hot day.']).toarray()"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"Ili kuhesabu vector ya bag-of-words kutoka kwa uwakilishi wa vector wa seti yetu ya data ya AG_NEWS, tunaweza kutumia kazi ifuatayo:\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 20,
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"tensor([2., 1., 2., ..., 0., 0., 0.])\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"vocab_size = len(vocab)\n",
|
||
"\n",
|
||
"def to_bow(text,bow_vocab_size=vocab_size):\n",
|
||
" res = torch.zeros(bow_vocab_size,dtype=torch.float32)\n",
|
||
" for i in encode(text):\n",
|
||
" if i<bow_vocab_size:\n",
|
||
" res[i] += 1\n",
|
||
" return res\n",
|
||
"\n",
|
||
"print(to_bow(train_dataset[0][1]))"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"> **Kumbuka:** Hapa tunatumia variable ya kimataifa `vocab_size` kuonyesha ukubwa wa kawaida wa msamiati. Kwa kuwa mara nyingi ukubwa wa msamiati ni mkubwa sana, tunaweza kupunguza ukubwa wa msamiati kwa maneno yanayotumika mara nyingi zaidi. Jaribu kupunguza thamani ya `vocab_size` na kuendesha msimbo hapa chini, na uone jinsi inavyoathiri usahihi. Unapaswa kutarajia kupungua kwa usahihi, lakini si kwa kiasi kikubwa, badala ya utendaji bora.\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## Kufundisha Kainishaji wa BoW\n",
|
||
"\n",
|
||
"Sasa kwa kuwa tumejifunza jinsi ya kujenga uwakilishi wa Bag-of-Words wa maandishi yetu, hebu tufundishe kainishaji juu yake. Kwanza, tunahitaji kubadilisha seti yetu ya data kwa mafunzo kwa njia ambayo uwakilishi wote wa vekta za nafasi hubadilishwa kuwa uwakilishi wa Bag-of-Words. Hili linaweza kufanyika kwa kupitisha kazi ya `bowify` kama parameter ya `collate_fn` kwa `DataLoader` ya kawaida ya torch:\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 21,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"from torch.utils.data import DataLoader\n",
|
||
"import numpy as np \n",
|
||
"\n",
|
||
"# this collate function gets list of batch_size tuples, and needs to \n",
|
||
"# return a pair of label-feature tensors for the whole minibatch\n",
|
||
"def bowify(b):\n",
|
||
" return (\n",
|
||
" torch.LongTensor([t[0]-1 for t in b]),\n",
|
||
" torch.stack([to_bow(t[1]) for t in b])\n",
|
||
" )\n",
|
||
"\n",
|
||
"train_loader = DataLoader(train_dataset, batch_size=16, collate_fn=bowify, shuffle=True)\n",
|
||
"test_loader = DataLoader(test_dataset, batch_size=16, collate_fn=bowify, shuffle=True)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"Sasa hebu tuelezee mtandao wa neva wa kuainisha rahisi ambao una safu moja ya mstari. Ukubwa wa vector ya ingizo ni sawa na `vocab_size`, na ukubwa wa matokeo unalingana na idadi ya madarasa (4). Kwa sababu tunatatua kazi ya uainishaji, kazi ya mwisho ya uanzishaji ni `LogSoftmax()`.\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 22,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"net = torch.nn.Sequential(torch.nn.Linear(vocab_size,4),torch.nn.LogSoftmax(dim=1))"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"Sasa tutafafanua mzunguko wa mafunzo wa kawaida wa PyTorch. Kwa sababu seti yetu ya data ni kubwa sana, kwa madhumuni ya kufundisha tutafanya mafunzo kwa kipindi kimoja tu, na wakati mwingine hata kwa chini ya kipindi kimoja (kuweka kipengele cha `epoch_size` huturuhusu kupunguza mafunzo). Pia tutaripoti usahihi wa mafunzo uliokusanywa wakati wa mafunzo; mara kwa mara ya kuripoti inabainishwa kwa kutumia kipengele cha `report_freq`.\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 24,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"def train_epoch(net,dataloader,lr=0.01,optimizer=None,loss_fn = torch.nn.NLLLoss(),epoch_size=None, report_freq=200):\n",
|
||
" optimizer = optimizer or torch.optim.Adam(net.parameters(),lr=lr)\n",
|
||
" net.train()\n",
|
||
" total_loss,acc,count,i = 0,0,0,0\n",
|
||
" for labels,features in dataloader:\n",
|
||
" optimizer.zero_grad()\n",
|
||
" out = net(features)\n",
|
||
" loss = loss_fn(out,labels) #cross_entropy(out,labels)\n",
|
||
" loss.backward()\n",
|
||
" optimizer.step()\n",
|
||
" total_loss+=loss\n",
|
||
" _,predicted = torch.max(out,1)\n",
|
||
" acc+=(predicted==labels).sum()\n",
|
||
" count+=len(labels)\n",
|
||
" i+=1\n",
|
||
" if i%report_freq==0:\n",
|
||
" print(f\"{count}: acc={acc.item()/count}\")\n",
|
||
" if epoch_size and count>epoch_size:\n",
|
||
" break\n",
|
||
" return total_loss.item()/count, acc.item()/count"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 25,
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"3200: acc=0.8028125\n",
|
||
"6400: acc=0.8371875\n",
|
||
"9600: acc=0.8534375\n",
|
||
"12800: acc=0.85765625\n"
|
||
]
|
||
},
|
||
{
|
||
"data": {
|
||
"text/plain": [
|
||
"(0.026090790722161722, 0.8620069296375267)"
|
||
]
|
||
},
|
||
"execution_count": 25,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"train_epoch(net,train_loader,epoch_size=15000)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## BiGrams, TriGrams na N-Grams\n",
|
||
"\n",
|
||
"Kikwazo kimoja cha mbinu ya mfuko wa maneno ni kwamba baadhi ya maneno ni sehemu ya misemo ya maneno mengi, kwa mfano, neno 'hot dog' lina maana tofauti kabisa kuliko maneno 'hot' na 'dog' katika muktadha mwingine. Ikiwa tutawakilisha maneno 'hot' na 'dog' kila mara kwa vekta zile zile, inaweza kuchanganya modeli yetu.\n",
|
||
"\n",
|
||
"Ili kushughulikia hili, **uwakilishi wa N-gram** mara nyingi hutumika katika mbinu za uainishaji wa nyaraka, ambapo marudio ya kila neno, maneno mawili au maneno matatu ni kipengele muhimu kwa kufundisha viainishi. Katika uwakilishi wa bigram, kwa mfano, tutaongeza jozi zote za maneno kwenye msamiati, pamoja na maneno ya awali.\n",
|
||
"\n",
|
||
"Hapa chini ni mfano wa jinsi ya kuzalisha uwakilishi wa mfuko wa maneno wa bigram kwa kutumia Scikit Learn:\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 26,
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Vocabulary:\n",
|
||
" {'i': 7, 'like': 11, 'hot': 4, 'dogs': 2, 'i like': 8, 'like hot': 12, 'hot dogs': 5, 'the': 16, 'dog': 0, 'ran': 14, 'fast': 3, 'the dog': 17, 'dog ran': 1, 'ran fast': 15, 'its': 9, 'outside': 13, 'its hot': 10, 'hot outside': 6}\n"
|
||
]
|
||
},
|
||
{
|
||
"data": {
|
||
"text/plain": [
|
||
"array([[1, 0, 1, 0, 2, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]],\n",
|
||
" dtype=int64)"
|
||
]
|
||
},
|
||
"execution_count": 26,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"bigram_vectorizer = CountVectorizer(ngram_range=(1, 2), token_pattern=r'\\b\\w+\\b', min_df=1)\n",
|
||
"corpus = [\n",
|
||
" 'I like hot dogs.',\n",
|
||
" 'The dog ran fast.',\n",
|
||
" 'Its hot outside.',\n",
|
||
" ]\n",
|
||
"bigram_vectorizer.fit_transform(corpus)\n",
|
||
"print(\"Vocabulary:\\n\",bigram_vectorizer.vocabulary_)\n",
|
||
"bigram_vectorizer.transform(['My dog likes hot dogs on a hot day.']).toarray()\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"Hasara kuu ya mbinu ya N-gram ni kwamba ukubwa wa msamiati huanza kuongezeka kwa kasi sana. Kwa vitendo, tunahitaji kuchanganya uwakilishi wa N-gram na mbinu fulani za kupunguza vipimo, kama vile *embeddings*, ambazo tutajadili katika kipengele kinachofuata.\n",
|
||
"\n",
|
||
"Ili kutumia uwakilishi wa N-gram katika seti yetu ya data ya **AG News**, tunahitaji kujenga msamiati maalum wa ngram:\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 27,
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Bigram vocabulary length = 1308842\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"counter = collections.Counter()\n",
|
||
"for (label, line) in train_dataset:\n",
|
||
" l = tokenizer(line)\n",
|
||
" counter.update(torchtext.data.utils.ngrams_iterator(l,ngrams=2))\n",
|
||
" \n",
|
||
"bi_vocab = torchtext.vocab.vocab(counter, min_freq=1)\n",
|
||
"\n",
|
||
"print(\"Bigram vocabulary length = \",len(bi_vocab))"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"Tunaweza kutumia msimbo ule ule kama hapo juu kufundisha classifier, hata hivyo, hii itakuwa na matumizi makubwa ya kumbukumbu. Katika sehemu inayofuata, tutafundisha classifier ya bigram kwa kutumia embeddings.\n",
|
||
"\n",
|
||
"> **Note:** Unaweza kuacha tu zile ngrams ambazo zinatokea kwenye maandishi zaidi ya idadi maalum ya mara. Hii itahakikisha kwamba bigrams zisizotokea mara nyingi zitaachwa, na itapunguza ukubwa wa vipimo kwa kiasi kikubwa. Ili kufanya hivyo, weka parameter `min_freq` kwa thamani ya juu zaidi, na angalia mabadiliko ya urefu wa msamiati.\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## Mzunguko wa Neno na Upekee wa Hati TF-IDF\n",
|
||
"\n",
|
||
"Katika uwakilishi wa BoW, matukio ya maneno yanapewa uzito sawa, bila kujali neno lenyewe. Hata hivyo, ni wazi kwamba maneno yanayojirudia mara kwa mara, kama *a*, *in*, nk. yana umuhimu mdogo sana kwa uainishaji, ikilinganishwa na maneno maalum. Kwa kweli, katika kazi nyingi za NLP, baadhi ya maneno ni muhimu zaidi kuliko mengine.\n",
|
||
"\n",
|
||
"**TF-IDF** inasimama kwa **mzunguko wa neno–upekee wa hati**. Ni toleo la begi la maneno, ambapo badala ya thamani ya 0/1 ya binary inayoonyesha uwepo wa neno katika hati, thamani ya namba inayozunguka hutumika, ambayo inahusiana na mzunguko wa matukio ya neno katika mkusanyiko.\n",
|
||
"\n",
|
||
"Kwa ufafanuzi rasmi zaidi, uzito $w_{ij}$ wa neno $i$ katika hati $j$ hufafanuliwa kama:\n",
|
||
"$$\n",
|
||
"w_{ij} = tf_{ij}\\times\\log({N\\over df_i})\n",
|
||
"$$\n",
|
||
"ambapo\n",
|
||
"* $tf_{ij}$ ni idadi ya matukio ya $i$ katika $j$, yaani thamani ya BoW tuliyoiona awali\n",
|
||
"* $N$ ni idadi ya hati katika mkusanyiko\n",
|
||
"* $df_i$ ni idadi ya hati zinazojumuisha neno $i$ katika mkusanyiko mzima\n",
|
||
"\n",
|
||
"Thamani ya TF-IDF $w_{ij}$ huongezeka kulingana na idadi ya mara neno linavyoonekana katika hati na hupunguzwa na idadi ya hati katika mkusanyiko zinazojumuisha neno hilo, jambo ambalo husaidia kurekebisha ukweli kwamba baadhi ya maneno huonekana mara nyingi zaidi kuliko mengine. Kwa mfano, ikiwa neno linaonekana katika *kila* hati katika mkusanyiko, $df_i=N$, na $w_{ij}=0$, na maneno hayo yatapuuzwa kabisa.\n",
|
||
"\n",
|
||
"Unaweza kuunda kwa urahisi vectorization ya TF-IDF ya maandishi ukitumia Scikit Learn:\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 28,
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"data": {
|
||
"text/plain": [
|
||
"array([[0.43381609, 0. , 0.43381609, 0. , 0.65985664,\n",
|
||
" 0.43381609, 0. , 0. , 0. , 0. ,\n",
|
||
" 0. , 0. , 0. , 0. , 0. ,\n",
|
||
" 0. ]])"
|
||
]
|
||
},
|
||
"execution_count": 28,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"from sklearn.feature_extraction.text import TfidfVectorizer\n",
|
||
"vectorizer = TfidfVectorizer(ngram_range=(1,2))\n",
|
||
"vectorizer.fit_transform(corpus)\n",
|
||
"vectorizer.transform(['My dog likes hot dogs on a hot day.']).toarray()"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## Hitimisho\n",
|
||
"\n",
|
||
"Hata hivyo, ingawa uwakilishi wa TF-IDF unatoa uzito wa marudio kwa maneno tofauti, hauwezi kuwakilisha maana au mpangilio. Kama mwanaisimu maarufu J. R. Firth alivyosema mwaka 1935, “Maana kamili ya neno daima ni ya muktadha, na hakuna utafiti wa maana bila muktadha unaoweza kuchukuliwa kwa uzito.”. Tutajifunza baadaye katika kozi jinsi ya kunasa taarifa za muktadha kutoka kwa maandishi kwa kutumia uundaji wa lugha.\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"\n---\n\n**Kanusho**: \nHati hii imetafsiriwa kwa kutumia huduma ya tafsiri ya AI [Co-op Translator](https://github.com/Azure/co-op-translator). Ingawa tunajitahidi kuhakikisha usahihi, tafadhali fahamu kuwa tafsiri za kiotomatiki zinaweza kuwa na makosa au kutokuwa sahihi. Hati asilia katika lugha yake ya awali inapaswa kuchukuliwa kama chanzo cha mamlaka. Kwa taarifa muhimu, tafsiri ya kitaalamu ya binadamu inapendekezwa. Hatutawajibika kwa kutoelewana au tafsiri zisizo sahihi zinazotokana na matumizi ya tafsiri hii.\n"
|
||
]
|
||
}
|
||
],
|
||
"metadata": {
|
||
"interpreter": {
|
||
"hash": "16af2a8bbb083ea23e5e41c7f5787656b2ce26968575d8763f2c4b17f9cd711f"
|
||
},
|
||
"kernelspec": {
|
||
"display_name": "Python 3.8.12 ('py38')",
|
||
"language": "python",
|
||
"name": "python3"
|
||
},
|
||
"language_info": {
|
||
"codemirror_mode": {
|
||
"name": "ipython",
|
||
"version": 3
|
||
},
|
||
"file_extension": ".py",
|
||
"mimetype": "text/x-python",
|
||
"name": "python",
|
||
"nbconvert_exporter": "python",
|
||
"pygments_lexer": "ipython3",
|
||
"version": "3.8.12"
|
||
},
|
||
"coopTranslator": {
|
||
"original_hash": "7b9040985e748e4e2d4c689892456ad7",
|
||
"translation_date": "2025-08-29T16:39:08+00:00",
|
||
"source_file": "lessons/5-NLP/13-TextRep/TextRepresentationPyTorch.ipynb",
|
||
"language_code": "sw"
|
||
}
|
||
},
|
||
"nbformat": 4,
|
||
"nbformat_minor": 2
|
||
} |