RomanNum3ral
/
AI-For-Beginners
mirror of https://github.com/microsoft/AI-For-Beginners.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
AI-For-Beginners/translations/sw/lessons/5-NLP/13-TextRep/TextRepresentationTF.ipynb

647 lines
26 KiB
Plaintext
Raw Blame History

{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Kazi ya Uainishaji wa Maandishi\n",
"\n",
"Katika moduli hii, tutaanza na kazi rahisi ya uainishaji wa maandishi kwa kutumia seti ya data ya **[AG_NEWS](http://www.di.unipi.it/~gulli/AG_corpus_of_news_articles.html)**: tutatambua vichwa vya habari vya habari katika mojawapo ya makundi 4: Dunia, Michezo, Biashara, na Sayansi/Tecknolojia.\n",
"\n",
"## Seti ya Data\n",
"\n",
"Ili kupakia seti ya data, tutatumia API ya **[TensorFlow Datasets](https://www.tensorflow.org/datasets)**.\n"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"import tensorflow as tf\n",
"from tensorflow import keras\n",
"import tensorflow_datasets as tfds\n",
"\n",
"# In this tutorial, we will be training a lot of models. In order to use GPU memory cautiously,\n",
"# we will set tensorflow option to grow GPU memory allocation when required.\n",
"physical_devices = tf.config.list_physical_devices('GPU') \n",
"if len(physical_devices)>0:\n",
" tf.config.experimental.set_memory_growth(physical_devices[0], True)\n",
"\n",
"dataset = tfds.load('ag_news_subset')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Sasa tunaweza kufikia sehemu za mafunzo na majaribio za seti ya data kwa kutumia `dataset['train']` na `dataset['test']` mtawaliwa:\n"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Length of train dataset = 120000\n",
"Length of test dataset = 7600\n"
]
}
],
"source": [
"ds_train = dataset['train']\n",
"ds_test = dataset['test']\n",
"\n",
"print(f\"Length of train dataset = {len(ds_train)}\")\n",
"print(f\"Length of test dataset = {len(ds_test)}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Hebu tuchapishe vichwa vya habari 10 vya kwanza kutoka kwenye seti yetu ya data:\n"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"3 (Sci/Tech) -> b'AMD Debuts Dual-Core Opteron Processor' b'AMD #39;s new dual-core Opteron chip is designed mainly for corporate computing applications, including databases, Web services, and financial transactions.'\n",
"1 (Sports) -> b\"Wood's Suspension Upheld (Reuters)\" b'Reuters - Major League Baseball\\\\Monday announced a decision on the appeal filed by Chicago Cubs\\\\pitcher Kerry Wood regarding a suspension stemming from an\\\\incident earlier this season.'\n",
"2 (Business) -> b'Bush reform may have blue states seeing red' b'President Bush #39;s quot;revenue-neutral quot; tax reform needs losers to balance its winners, and people claiming the federal deduction for state and local taxes may be in administration planners #39; sights, news reports say.'\n",
"3 (Sci/Tech) -> b\"'Halt science decline in schools'\" b'Britain will run out of leading scientists unless science education is improved, says Professor Colin Pillinger.'\n",
"1 (Sports) -> b'Gerrard leaves practice' b'London, England (Sports Network) - England midfielder Steven Gerrard injured his groin late in Thursday #39;s training session, but is hopeful he will be ready for Saturday #39;s World Cup qualifier against Austria.'\n"
]
}
],
"source": [
"classes = ['World', 'Sports', 'Business', 'Sci/Tech']\n",
"\n",
"for i,x in zip(range(5),ds_train):\n",
" print(f\"{x['label']} ({classes[x['label']]}) -> {x['title']} {x['description']}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Uwakilishi wa maandishi kwa namba\n",
"\n",
"Sasa tunahitaji kubadilisha maandishi kuwa **namba** ambazo zinaweza kuwakilishwa kama tensors. Ikiwa tunataka uwakilishi wa kiwango cha maneno, tunahitaji kufanya mambo mawili:\n",
"\n",
"* Tumia **tokenizer** kugawanya maandishi kuwa **tokeni**.\n",
"* Unda **msamiati** wa tokeni hizo.\n",
"\n",
"### Kuweka kikomo kwa ukubwa wa msamiati\n",
"\n",
"Katika mfano wa dataset ya AG News, ukubwa wa msamiati ni mkubwa sana, zaidi ya maneno 100k. Kwa ujumla, hatuhitaji maneno ambayo yanapatikana mara chache sana katika maandishi — sentensi chache tu zitakuwa nayo, na modeli haitajifunza kutoka kwao. Kwa hivyo, ina mantiki kuweka kikomo kwa ukubwa wa msamiati kwa idadi ndogo kwa kupitisha hoja kwa mjenzi wa vectorizer:\n",
"\n",
"Hatua zote mbili zinaweza kushughulikiwa kwa kutumia safu ya **TextVectorization**. Hebu tuunde kitu cha vectorizer, kisha tuitie njia ya `adapt` ili kupitia maandishi yote na kujenga msamiati:\n"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [],
"source": [
"vocab_size = 50000\n",
"vectorizer = keras.layers.experimental.preprocessing.TextVectorization(max_tokens=vocab_size)\n",
"vectorizer.adapt(ds_train.take(500).map(lambda x: x['title']+' '+x['description']))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"> **Kumbuka** tunatumia sehemu tu ya dataset nzima kujenga msamiati. Tunafanya hivi ili kuharakisha muda wa utekelezaji na kuepuka kukufanya usubiri. Hata hivyo, tunachukua hatari kwamba baadhi ya maneno kutoka dataset nzima hayatajumuishwa kwenye msamiati, na yatapuuzwa wakati wa mafunzo. Kwa hivyo, kutumia ukubwa wa msamiati mzima na kupitia dataset yote wakati wa `adapt` kunapaswa kuongeza usahihi wa mwisho, lakini si kwa kiwango kikubwa.\n",
"\n",
"Sasa tunaweza kufikia msamiati halisi:\n"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"['', '[UNK]', 'the', 'to', 'a', 'in', 'of', 'and', 'on', 'for']\n",
"Length of vocabulary: 5335\n"
]
}
],
"source": [
"vocab = vectorizer.get_vocabulary()\n",
"vocab_size = len(vocab)\n",
"print(vocab[:10])\n",
"print(f\"Length of vocabulary: {vocab_size}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Kutumia vectorizer, tunaweza kwa urahisi kuweka maandishi yoyote katika seti ya nambari:\n"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"<tf.Tensor: shape=(7,), dtype=int64, numpy=array([ 112, 3695, 3, 304, 11, 1041, 1], dtype=int64)>"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"vectorizer('I love to play with my words')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Uwakilishi wa maandishi kwa kutumia Bag-of-words\n",
"\n",
"Kwa sababu maneno yanawakilisha maana, wakati mwingine tunaweza kuelewa maana ya kipande cha maandishi kwa kuangalia tu maneno binafsi, bila kujali mpangilio wao katika sentensi. Kwa mfano, wakati wa kuainisha habari, maneno kama *hali ya hewa* na *theluji* yanaweza kuashiria *utabiri wa hali ya hewa*, wakati maneno kama *hisa* na *dola* yangeelekea kuelekea *habari za kifedha*.\n",
"\n",
"**Bag-of-words** (BoW) ni uwakilishi wa vekta wa jadi ambao ni rahisi zaidi kueleweka. Kila neno linaunganishwa na faharasa ya vekta, na kipengele cha vekta kinaonyesha idadi ya mara neno fulani linavyotokea katika hati fulani.\n",
"\n",
"![Picha inayoonyesha jinsi uwakilishi wa vekta wa bag-of-words unavyohifadhiwa kwenye kumbukumbu.](../../../../../translated_images/sw/bag-of-words-example.606fc1738f1d7ba9.webp) \n",
"\n",
"> **Note**: Unaweza pia kufikiria BoW kama jumla ya vekta zote za one-hot-encoded kwa maneno binafsi katika maandishi.\n",
"\n",
"Hapo chini kuna mfano wa jinsi ya kuunda uwakilishi wa bag-of-words kwa kutumia maktaba ya python ya Scikit Learn:\n"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([[1, 1, 0, 2, 0, 0, 0, 0, 0]], dtype=int64)"
]
},
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from sklearn.feature_extraction.text import CountVectorizer\n",
"sc_vectorizer = CountVectorizer()\n",
"corpus = [\n",
" 'I like hot dogs.',\n",
" 'The dog ran fast.',\n",
" 'Its hot outside.',\n",
" ]\n",
"sc_vectorizer.fit_transform(corpus)\n",
"sc_vectorizer.transform(['My dog likes hot dogs on a hot day.']).toarray()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Tunaweza pia kutumia Keras vectorizer tuliyofafanua hapo juu, kubadilisha kila nambari ya neno kuwa one-hot encoding na kuongeza vektori zote hizo:\n"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([0., 5., 0., ..., 0., 0., 0.], dtype=float32)"
]
},
"execution_count": 9,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"def to_bow(text):\n",
" return tf.reduce_sum(tf.one_hot(vectorizer(text),vocab_size),axis=0)\n",
"\n",
"to_bow('My dog likes hot dogs on a hot day.').numpy()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"> **Kumbuka**: Unaweza kushangaa kwamba matokeo yanatofautiana na mfano wa awali. Sababu ni kwamba katika mfano wa Keras urefu wa vector unalingana na ukubwa wa msamiati, ambao ulijengwa kutoka kwenye seti nzima ya data ya AG News, ilhali katika mfano wa Scikit Learn tulijenga msamiati kutoka kwenye maandishi ya sampuli papo hapo.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Kufundisha Kainishaji wa BoW\n",
"\n",
"Sasa kwa kuwa tumejifunza jinsi ya kujenga uwakilishi wa mfuko-wa-maneno kwa maandishi yetu, hebu tufundishe kainishaji inayoutumia. Kwanza, tunahitaji kubadilisha seti yetu ya data kuwa uwakilishi wa mfuko-wa-maneno. Hii inaweza kufanyika kwa kutumia kazi ya `map` kwa njia ifuatayo:\n"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [],
"source": [
"batch_size = 128\n",
"\n",
"ds_train_bow = ds_train.map(lambda x: (to_bow(x['title']+x['description']),x['label'])).batch(batch_size)\n",
"ds_test_bow = ds_test.map(lambda x: (to_bow(x['title']+x['description']),x['label'])).batch(batch_size)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Sasa hebu tueleze mtandao wa neva wa kuainisha rahisi ambao una safu moja ya mstari. Ukubwa wa ingizo ni `vocab_size`, na ukubwa wa matokeo unahusiana na idadi ya madarasa (4). Kwa sababu tunatatua kazi ya uainishaji, kazi ya mwisho ya uanzishaji ni **softmax**:\n"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"938/938 [==============================] - 66s 70ms/step - loss: 0.6144 - acc: 0.8427 - val_loss: 0.4416 - val_acc: 0.8697\n"
]
},
{
"data": {
"text/plain": [
"<keras.callbacks.History at 0x20c70a947f0>"
]
},
"execution_count": 12,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"model = keras.models.Sequential([\n",
" keras.layers.Dense(4,activation='softmax',input_shape=(vocab_size,))\n",
"])\n",
"model.compile(loss='sparse_categorical_crossentropy',optimizer='adam',metrics=['acc'])\n",
"model.fit(ds_train_bow,validation_data=ds_test_bow)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Kwa kuwa tuna madarasa 4, usahihi wa zaidi ya 80% ni matokeo mazuri.\n",
"\n",
"## Kufundisha classifier kama mtandao mmoja\n",
"\n",
"Kwa sababu vectorizer pia ni safu ya Keras, tunaweza kufafanua mtandao unaojumuisha vectorizer, na kuufundisha kutoka mwanzo hadi mwisho. Kwa njia hii hatuhitaji kugeuza dataset kwa kutumia `map`, tunaweza tu kupitisha dataset ya asili kwenye ingizo la mtandao.\n",
"\n",
"> **Note**: Bado tungehitaji kutumia `map` kwenye dataset yetu ili kubadilisha sehemu kutoka kwa kamusi (kama `title`, `description` na `label`) kuwa tuples. Hata hivyo, tunapopakia data kutoka diski, tunaweza kujenga dataset yenye muundo unaohitajika moja kwa moja.\n"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Model: \"model\"\n",
"_________________________________________________________________\n",
" Layer (type) Output Shape Param # \n",
"=================================================================\n",
" input_1 (InputLayer) [(None, 1)] 0 \n",
" \n",
" text_vectorization (TextVec (None, None) 0 \n",
" torization) \n",
" \n",
" tf.one_hot (TFOpLambda) (None, None, 5335) 0 \n",
" \n",
" tf.math.reduce_sum (TFOpLam (None, 5335) 0 \n",
" bda) \n",
" \n",
" dense_2 (Dense) (None, 4) 21344 \n",
" \n",
"=================================================================\n",
"Total params: 21,344\n",
"Trainable params: 21,344\n",
"Non-trainable params: 0\n",
"_________________________________________________________________\n",
"938/938 [==============================] - 73s 77ms/step - loss: 0.6057 - acc: 0.8414 - val_loss: 0.4202 - val_acc: 0.8736\n"
]
},
{
"data": {
"text/plain": [
"<keras.callbacks.History at 0x20c721521f0>"
]
},
"execution_count": 13,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"def extract_text(x):\n",
" return x['title']+' '+x['description']\n",
"\n",
"def tupelize(x):\n",
" return (extract_text(x),x['label'])\n",
"\n",
"inp = keras.Input(shape=(1,),dtype=tf.string)\n",
"x = vectorizer(inp)\n",
"x = tf.reduce_sum(tf.one_hot(x,vocab_size),axis=1)\n",
"out = keras.layers.Dense(4,activation='softmax')(x)\n",
"model = keras.models.Model(inp,out)\n",
"model.summary()\n",
"\n",
"model.compile(loss='sparse_categorical_crossentropy',optimizer='adam',metrics=['acc'])\n",
"model.fit(ds_train.map(tupelize).batch(batch_size),validation_data=ds_test.map(tupelize).batch(batch_size))\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Bigrams, trigrams na n-grams\n",
"\n",
"Kikwazo kimoja cha mbinu ya bag-of-words ni kwamba baadhi ya maneno ni sehemu ya misemo ya maneno mengi, kwa mfano, neno 'hot dog' lina maana tofauti kabisa na maneno 'hot' na 'dog' katika muktadha mwingine. Ikiwa tutawakilisha maneno 'hot' na 'dog' kila mara kwa kutumia vekta zile zile, inaweza kuichanganya modeli yetu.\n",
"\n",
"Ili kushughulikia hili, **uwakilishi wa n-gram** mara nyingi hutumika katika mbinu za uainishaji wa nyaraka, ambapo marudio ya kila neno, maneno mawili au maneno matatu ni kipengele muhimu kwa kufundisha viainishi. Katika uwakilishi wa bigram, kwa mfano, tutaongeza jozi zote za maneno kwenye msamiati, pamoja na maneno ya asili.\n",
"\n",
"Hapa chini kuna mfano wa jinsi ya kuunda uwakilishi wa bigram bag-of-words kwa kutumia Scikit Learn:\n"
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Vocabulary:\n",
" {'i': 7, 'like': 11, 'hot': 4, 'dogs': 2, 'i like': 8, 'like hot': 12, 'hot dogs': 5, 'the': 16, 'dog': 0, 'ran': 14, 'fast': 3, 'the dog': 17, 'dog ran': 1, 'ran fast': 15, 'its': 9, 'outside': 13, 'its hot': 10, 'hot outside': 6}\n"
]
},
{
"data": {
"text/plain": [
"array([[1, 0, 1, 0, 2, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]],\n",
" dtype=int64)"
]
},
"execution_count": 14,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"bigram_vectorizer = CountVectorizer(ngram_range=(1, 2), token_pattern=r'\\b\\w+\\b', min_df=1)\n",
"corpus = [\n",
" 'I like hot dogs.',\n",
" 'The dog ran fast.',\n",
" 'Its hot outside.',\n",
" ]\n",
"bigram_vectorizer.fit_transform(corpus)\n",
"print(\"Vocabulary:\\n\",bigram_vectorizer.vocabulary_)\n",
"bigram_vectorizer.transform(['My dog likes hot dogs on a hot day.']).toarray()\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Hasara kuu ya mbinu ya n-gram ni kwamba ukubwa wa msamiati huanza kukua kwa kasi sana. Kwa vitendo, tunahitaji kuchanganya uwakilishi wa n-gram na mbinu ya kupunguza vipimo, kama vile *embeddings*, ambayo tutajadili katika kipengele kinachofuata.\n",
"\n",
"Ili kutumia uwakilishi wa n-gram katika seti yetu ya data ya **AG News**, tunahitaji kupitisha kipengele cha `ngrams` kwa mjenzi wetu wa `TextVectorization`. Urefu wa msamiati wa bigram ni **kubwa sana**, katika hali yetu ni zaidi ya tokeni milioni 1.3! Kwa hivyo, inafaa kupunguza tokeni za bigram kwa idadi fulani ya busara.\n",
"\n",
"Tunaweza kutumia msimbo ule ule kama hapo juu kufundisha classifier, hata hivyo, hii haitakuwa na ufanisi wa kumbukumbu. Katika kipengele kinachofuata, tutafundisha classifier ya bigram kwa kutumia embeddings. Kwa sasa, unaweza kujaribu mafunzo ya classifier ya bigram katika daftari hili na uone kama unaweza kupata usahihi wa juu zaidi.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Kuhesabu Vekta za BoW Kiotomatiki\n",
"\n",
"Katika mfano hapo juu tulihesabu vekta za BoW kwa mkono kwa kujumlisha usimbaji wa moja kwa moja wa maneno ya kibinafsi. Hata hivyo, toleo la hivi karibuni la TensorFlow linaturuhusu kuhesabu vekta za BoW kiotomatiki kwa kupitisha kipengele `output_mode='count` kwenye mjenzi wa vectorizer. Hii inafanya kufafanua na kufundisha modeli yetu kuwa rahisi zaidi:\n"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Training vectorizer\n",
"938/938 [==============================] - 7s 7ms/step - loss: 0.5929 - acc: 0.8486 - val_loss: 0.4168 - val_acc: 0.8772\n"
]
},
{
"data": {
"text/plain": [
"<keras.callbacks.History at 0x20c725217c0>"
]
},
"execution_count": 15,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"model = keras.models.Sequential([\n",
" keras.layers.experimental.preprocessing.TextVectorization(max_tokens=vocab_size,output_mode='count'),\n",
" keras.layers.Dense(4,input_shape=(vocab_size,), activation='softmax')\n",
"])\n",
"print(\"Training vectorizer\")\n",
"model.layers[0].adapt(ds_train.take(500).map(extract_text))\n",
"model.compile(loss='sparse_categorical_crossentropy',optimizer='adam',metrics=['acc'])\n",
"model.fit(ds_train.map(tupelize).batch(batch_size),validation_data=ds_test.map(tupelize).batch(batch_size))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Mzunguko wa Neno - Mzunguko wa Nyaraka Kinyume (TF-IDF)\n",
"\n",
"Katika uwakilishi wa BoW, matukio ya maneno yanapimwa kwa kutumia mbinu ile ile bila kujali neno lenyewe. Hata hivyo, ni wazi kwamba maneno yanayojirudia mara kwa mara kama *a* na *in* yana umuhimu mdogo sana kwa uainishaji ikilinganishwa na maneno maalum. Katika kazi nyingi za NLP, baadhi ya maneno ni muhimu zaidi kuliko mengine.\n",
"\n",
"**TF-IDF** inasimama kwa **mzunguko wa neno - mzunguko wa nyaraka kinyume**. Ni toleo la bag-of-words, ambapo badala ya thamani ya 0/1 ya binary inayoonyesha uwepo wa neno katika nyaraka, thamani ya namba ya desimali hutumika, ambayo inahusiana na mzunguko wa neno hilo katika mkusanyiko.\n",
"\n",
"Kwa ufafanuzi rasmi zaidi, uzito $w_{ij}$ wa neno $i$ katika nyaraka $j$ hufafanuliwa kama:\n",
"$$\n",
"w_{ij} = tf_{ij}\\times\\log({N\\over df_i})\n",
"$$\n",
"ambapo\n",
"* $tf_{ij}$ ni idadi ya matukio ya $i$ katika $j$, yaani thamani ya BoW tuliyoiona awali\n",
"* $N$ ni idadi ya nyaraka katika mkusanyiko\n",
"* $df_i$ ni idadi ya nyaraka zinazojumuisha neno $i$ katika mkusanyiko mzima\n",
"\n",
"Thamani ya TF-IDF $w_{ij}$ huongezeka kulingana na idadi ya mara neno linavyoonekana katika nyaraka na hupunguzwa na idadi ya nyaraka katika mkusanyiko ambazo zinajumuisha neno hilo, jambo ambalo husaidia kurekebisha ukweli kwamba baadhi ya maneno huonekana mara nyingi zaidi kuliko mengine. Kwa mfano, ikiwa neno linaonekana katika *kila* nyaraka katika mkusanyiko, $df_i=N$, na $w_{ij}=0$, na maneno hayo yatapuuzwa kabisa.\n",
"\n",
"Unaweza kuunda kwa urahisi uakisi wa maandishi wa TF-IDF kwa kutumia Scikit Learn:\n"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([[0.43381609, 0. , 0.43381609, 0. , 0.65985664,\n",
" 0.43381609, 0. , 0. , 0. , 0. ,\n",
" 0. , 0. , 0. , 0. , 0. ,\n",
" 0. ]])"
]
},
"execution_count": 16,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from sklearn.feature_extraction.text import TfidfVectorizer\n",
"vectorizer = TfidfVectorizer(ngram_range=(1,2))\n",
"vectorizer.fit_transform(corpus)\n",
"vectorizer.transform(['My dog likes hot dogs on a hot day.']).toarray()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Katika Keras, safu ya `TextVectorization` inaweza kuhesabu mara kwa mara za TF-IDF kiotomatiki kwa kupitisha kipengele `output_mode='tf-idf'`. Hebu turudie msimbo tuliotumia hapo juu ili kuona kama kutumia TF-IDF kunaongeza usahihi:\n"
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Training vectorizer\n",
"938/938 [==============================] - 12s 12ms/step - loss: 0.4197 - acc: 0.8662 - val_loss: 0.3432 - val_acc: 0.8849\n"
]
},
{
"data": {
"text/plain": [
"<keras.callbacks.History at 0x20c729dfd30>"
]
},
"execution_count": 17,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"model = keras.models.Sequential([\n",
" keras.layers.experimental.preprocessing.TextVectorization(max_tokens=vocab_size,output_mode='tf-idf'),\n",
" keras.layers.Dense(4,input_shape=(vocab_size,), activation='softmax')\n",
"])\n",
"print(\"Training vectorizer\")\n",
"model.layers[0].adapt(ds_train.take(500).map(extract_text))\n",
"model.compile(loss='sparse_categorical_crossentropy',optimizer='adam',metrics=['acc'])\n",
"model.fit(ds_train.map(tupelize).batch(batch_size),validation_data=ds_test.map(tupelize).batch(batch_size))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Hitimisho\n",
"\n",
"Ingawa uwakilishi wa TF-IDF unatoa uzito wa marudio kwa maneno tofauti, hauwezi kuwakilisha maana au mpangilio. Kama mwanaisimu maarufu J. R. Firth alivyosema mwaka 1935, \"Maana kamili ya neno daima ni ya muktadha, na hakuna uchambuzi wa maana bila muktadha unaoweza kuchukuliwa kwa uzito.\" Tutajifunza jinsi ya kunasa taarifa za muktadha kutoka kwa maandishi kwa kutumia uundaji wa lugha baadaye katika kozi.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n---\n\n**Kanusho**: \nHati hii imetafsiriwa kwa kutumia huduma ya kutafsiri ya AI [Co-op Translator](https://github.com/Azure/co-op-translator). Ingawa tunajitahidi kuhakikisha usahihi, tafadhali fahamu kuwa tafsiri za kiotomatiki zinaweza kuwa na makosa au kutokuwa sahihi. Hati ya asili katika lugha yake ya awali inapaswa kuzingatiwa kama chanzo cha mamlaka. Kwa taarifa muhimu, tafsiri ya kitaalamu ya binadamu inapendekezwa. Hatutawajibika kwa kutoelewana au tafsiri zisizo sahihi zinazotokana na matumizi ya tafsiri hii.\n"
]
}
],
"metadata": {
"interpreter": {
"hash": "0cb620c6d4b9f7a635928804c26cf22403d89d98d79684e4529119355ee6d5a5"
},
"kernel_info": {
"name": "conda-env-py37_tensorflow-py"
},
"kernelspec": {
"display_name": "py37_tensorflow",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.12"
},
"nteract": {
"version": "nteract-front-end@1.0.0"
},
"coopTranslator": {
"original_hash": "19b43951d55b377a76209c24c1f017e4",
"translation_date": "2025-08-29T16:44:48+00:00",
"source_file": "lessons/5-NLP/13-TextRep/TextRepresentationTF.ipynb",
"language_code": "sw"
}
},
"nbformat": 4,
"nbformat_minor": 4
}
Reference in New Issue View Git Blame Copy Permalink