647 lines
25 KiB
Plaintext
647 lines
25 KiB
Plaintext
{
|
||
"cells": [
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"# 文本分類任務\n",
|
||
"\n",
|
||
"在本模組中，我們將從一個基於 **[AG_NEWS](http://www.di.unipi.it/~gulli/AG_corpus_of_news_articles.html)** 數據集的簡單文本分類任務開始：我們將把新聞標題分類為以下四個類別之一：世界、體育、商業和科學/技術。\n",
|
||
"\n",
|
||
"## 數據集\n",
|
||
"\n",
|
||
"為了載入數據集，我們將使用 **[TensorFlow Datasets](https://www.tensorflow.org/datasets)** API。\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 1,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"import tensorflow as tf\n",
|
||
"from tensorflow import keras\n",
|
||
"import tensorflow_datasets as tfds\n",
|
||
"\n",
|
||
"# In this tutorial, we will be training a lot of models. In order to use GPU memory cautiously,\n",
|
||
"# we will set tensorflow option to grow GPU memory allocation when required.\n",
|
||
"physical_devices = tf.config.list_physical_devices('GPU') \n",
|
||
"if len(physical_devices)>0:\n",
|
||
" tf.config.experimental.set_memory_growth(physical_devices[0], True)\n",
|
||
"\n",
|
||
"dataset = tfds.load('ag_news_subset')"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"我們現在可以分別使用 `dataset['train']` 和 `dataset['test']` 訪問數據集的訓練部分和測試部分：\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 3,
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Length of train dataset = 120000\n",
|
||
"Length of test dataset = 7600\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"ds_train = dataset['train']\n",
|
||
"ds_test = dataset['test']\n",
|
||
"\n",
|
||
"print(f\"Length of train dataset = {len(ds_train)}\")\n",
|
||
"print(f\"Length of test dataset = {len(ds_test)}\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"讓我們列印出資料集中前10個新的標題：\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 4,
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"3 (Sci/Tech) -> b'AMD Debuts Dual-Core Opteron Processor' b'AMD #39;s new dual-core Opteron chip is designed mainly for corporate computing applications, including databases, Web services, and financial transactions.'\n",
|
||
"1 (Sports) -> b\"Wood's Suspension Upheld (Reuters)\" b'Reuters - Major League Baseball\\\\Monday announced a decision on the appeal filed by Chicago Cubs\\\\pitcher Kerry Wood regarding a suspension stemming from an\\\\incident earlier this season.'\n",
|
||
"2 (Business) -> b'Bush reform may have blue states seeing red' b'President Bush #39;s quot;revenue-neutral quot; tax reform needs losers to balance its winners, and people claiming the federal deduction for state and local taxes may be in administration planners #39; sights, news reports say.'\n",
|
||
"3 (Sci/Tech) -> b\"'Halt science decline in schools'\" b'Britain will run out of leading scientists unless science education is improved, says Professor Colin Pillinger.'\n",
|
||
"1 (Sports) -> b'Gerrard leaves practice' b'London, England (Sports Network) - England midfielder Steven Gerrard injured his groin late in Thursday #39;s training session, but is hopeful he will be ready for Saturday #39;s World Cup qualifier against Austria.'\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"classes = ['World', 'Sports', 'Business', 'Sci/Tech']\n",
|
||
"\n",
|
||
"for i,x in zip(range(5),ds_train):\n",
|
||
" print(f\"{x['label']} ({classes[x['label']]}) -> {x['title']} {x['description']}\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## 文本向量化\n",
|
||
"\n",
|
||
"現在我們需要將文本轉換為可以表示為張量的**數字**。如果我們想要詞級別的表示，需要完成兩件事：\n",
|
||
"\n",
|
||
"* 使用**分詞器**將文本拆分為**詞元**。\n",
|
||
"* 建立這些詞元的**詞彙表**。\n",
|
||
"\n",
|
||
"### 限制詞彙表大小\n",
|
||
"\n",
|
||
"在 AG News 數據集的例子中，詞彙表的大小相當大，超過 10 萬個單詞。一般來說，我們不需要那些在文本中很少出現的單詞——只有少數句子會包含它們，模型也無法從中學習。因此，通過向向量化器構造函數傳遞參數，將詞彙表的大小限制為較小的數量是有意義的：\n",
|
||
"\n",
|
||
"這兩個步驟都可以使用 **TextVectorization** 層來處理。我們來實例化向量化器對象，然後調用 `adapt` 方法來遍歷所有文本並建立詞彙表：\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 5,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"vocab_size = 50000\n",
|
||
"vectorizer = keras.layers.experimental.preprocessing.TextVectorization(max_tokens=vocab_size)\n",
|
||
"vectorizer.adapt(ds_train.take(500).map(lambda x: x['title']+' '+x['description']))"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"> **注意** 我們僅使用整個數據集的一部分來建立詞彙表。這樣做是為了加快執行速度，避免讓您等待太久。然而，我們承擔了一些風險，即整個數據集中的某些詞可能不會被包含在詞彙表中，並在訓練過程中被忽略。因此，使用完整的詞彙表大小並在 `adapt` 過程中遍歷整個數據集應該能提高最終的準確性，但提升幅度不會太大。\n",
|
||
"\n",
|
||
"現在我們可以訪問實際的詞彙表：\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 6,
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"['', '[UNK]', 'the', 'to', 'a', 'in', 'of', 'and', 'on', 'for']\n",
|
||
"Length of vocabulary: 5335\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"vocab = vectorizer.get_vocabulary()\n",
|
||
"vocab_size = len(vocab)\n",
|
||
"print(vocab[:10])\n",
|
||
"print(f\"Length of vocabulary: {vocab_size}\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"使用向量化器，我們可以輕鬆地將任何文本編碼為一組數字：\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 7,
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"data": {
|
||
"text/plain": [
|
||
"<tf.Tensor: shape=(7,), dtype=int64, numpy=array([ 112, 3695, 3, 304, 11, 1041, 1], dtype=int64)>"
|
||
]
|
||
},
|
||
"execution_count": 7,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"vectorizer('I love to play with my words')"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## Bag-of-words 文本表示法\n",
|
||
"\n",
|
||
"由於文字具有意義，有時我們僅需查看句子中的個別文字，就能理解一段文字的含義，而不必在意它們的排列順序。例如，在分類新聞時，像 *weather* 和 *snow* 這樣的詞可能表明是 *天氣預報*，而像 *stocks* 和 *dollar* 則可能屬於 *財經新聞*。\n",
|
||
"\n",
|
||
"**Bag-of-words** (BoW) 向量表示法是最容易理解的傳統向量表示法。每個詞都與向量索引相關聯，向量中的元素則表示某個詞在特定文件中出現的次數。\n",
|
||
"\n",
|
||
" \n",
|
||
"\n",
|
||
"> **Note**: 你也可以將 BoW 理解為文本中每個詞的獨熱編碼向量的總和。\n",
|
||
"\n",
|
||
"以下是一個使用 Scikit Learn Python 庫生成 Bag-of-words 表示法的範例：\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 8,
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"data": {
|
||
"text/plain": [
|
||
"array([[1, 1, 0, 2, 0, 0, 0, 0, 0]], dtype=int64)"
|
||
]
|
||
},
|
||
"execution_count": 8,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"from sklearn.feature_extraction.text import CountVectorizer\n",
|
||
"sc_vectorizer = CountVectorizer()\n",
|
||
"corpus = [\n",
|
||
" 'I like hot dogs.',\n",
|
||
" 'The dog ran fast.',\n",
|
||
" 'Its hot outside.',\n",
|
||
" ]\n",
|
||
"sc_vectorizer.fit_transform(corpus)\n",
|
||
"sc_vectorizer.transform(['My dog likes hot dogs on a hot day.']).toarray()"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"我們也可以使用我們在上面定義的 Keras 向量化器，將每個單詞編號轉換為一個獨熱編碼，然後將所有這些向量相加：\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 9,
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"data": {
|
||
"text/plain": [
|
||
"array([0., 5., 0., ..., 0., 0., 0.], dtype=float32)"
|
||
]
|
||
},
|
||
"execution_count": 9,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"def to_bow(text):\n",
|
||
" return tf.reduce_sum(tf.one_hot(vectorizer(text),vocab_size),axis=0)\n",
|
||
"\n",
|
||
"to_bow('My dog likes hot dogs on a hot day.').numpy()"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"> **注意**：您可能會驚訝地發現結果與之前的範例不同。原因在於，在 Keras 的範例中，向量的長度對應於詞彙表的大小，而該詞彙表是基於整個 AG News 數據集構建的；而在 Scikit Learn 的範例中，我們是即時從樣本文本中構建詞彙表的。\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## 訓練 BoW 分類器\n",
|
||
"\n",
|
||
"現在我們已經學會如何建立文字的詞袋表示法，接下來讓我們訓練一個使用該表示法的分類器。首先，我們需要將數據集轉換為詞袋表示法。這可以通過以下方式使用 `map` 函數來實現：\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 11,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"batch_size = 128\n",
|
||
"\n",
|
||
"ds_train_bow = ds_train.map(lambda x: (to_bow(x['title']+x['description']),x['label'])).batch(batch_size)\n",
|
||
"ds_test_bow = ds_test.map(lambda x: (to_bow(x['title']+x['description']),x['label'])).batch(batch_size)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"現在讓我們定義一個簡單的分類器神經網絡，其中包含一個線性層。輸入大小為 `vocab_size`，輸出大小對應於類別數量（4）。由於我們正在解決分類任務，最終的激活函數是 **softmax**：\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 12,
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"938/938 [==============================] - 66s 70ms/step - loss: 0.6144 - acc: 0.8427 - val_loss: 0.4416 - val_acc: 0.8697\n"
|
||
]
|
||
},
|
||
{
|
||
"data": {
|
||
"text/plain": [
|
||
"<keras.callbacks.History at 0x20c70a947f0>"
|
||
]
|
||
},
|
||
"execution_count": 12,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"model = keras.models.Sequential([\n",
|
||
" keras.layers.Dense(4,activation='softmax',input_shape=(vocab_size,))\n",
|
||
"])\n",
|
||
"model.compile(loss='sparse_categorical_crossentropy',optimizer='adam',metrics=['acc'])\n",
|
||
"model.fit(ds_train_bow,validation_data=ds_test_bow)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"由於我們有四個類別，超過 80% 的準確率是一個不錯的結果。\n",
|
||
"\n",
|
||
"## 將分類器作為一個網絡進行訓練\n",
|
||
"\n",
|
||
"由於向量化器本身也是一個 Keras 層，我們可以定義一個包含它的網絡，並進行端到端的訓練。這樣，我們就不需要使用 `map` 來向量化數據集，只需將原始數據集傳遞到網絡的輸入即可。\n",
|
||
"\n",
|
||
"> **注意**: 我們仍然需要對數據集應用映射操作，將字典中的字段（例如 `title`、`description` 和 `label`）轉換為元組。然而，在從磁盤加載數據時，我們可以一開始就構建具有所需結構的數據集。\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 13,
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Model: \"model\"\n",
|
||
"_________________________________________________________________\n",
|
||
" Layer (type) Output Shape Param # \n",
|
||
"=================================================================\n",
|
||
" input_1 (InputLayer) [(None, 1)] 0 \n",
|
||
" \n",
|
||
" text_vectorization (TextVec (None, None) 0 \n",
|
||
" torization) \n",
|
||
" \n",
|
||
" tf.one_hot (TFOpLambda) (None, None, 5335) 0 \n",
|
||
" \n",
|
||
" tf.math.reduce_sum (TFOpLam (None, 5335) 0 \n",
|
||
" bda) \n",
|
||
" \n",
|
||
" dense_2 (Dense) (None, 4) 21344 \n",
|
||
" \n",
|
||
"=================================================================\n",
|
||
"Total params: 21,344\n",
|
||
"Trainable params: 21,344\n",
|
||
"Non-trainable params: 0\n",
|
||
"_________________________________________________________________\n",
|
||
"938/938 [==============================] - 73s 77ms/step - loss: 0.6057 - acc: 0.8414 - val_loss: 0.4202 - val_acc: 0.8736\n"
|
||
]
|
||
},
|
||
{
|
||
"data": {
|
||
"text/plain": [
|
||
"<keras.callbacks.History at 0x20c721521f0>"
|
||
]
|
||
},
|
||
"execution_count": 13,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"def extract_text(x):\n",
|
||
" return x['title']+' '+x['description']\n",
|
||
"\n",
|
||
"def tupelize(x):\n",
|
||
" return (extract_text(x),x['label'])\n",
|
||
"\n",
|
||
"inp = keras.Input(shape=(1,),dtype=tf.string)\n",
|
||
"x = vectorizer(inp)\n",
|
||
"x = tf.reduce_sum(tf.one_hot(x,vocab_size),axis=1)\n",
|
||
"out = keras.layers.Dense(4,activation='softmax')(x)\n",
|
||
"model = keras.models.Model(inp,out)\n",
|
||
"model.summary()\n",
|
||
"\n",
|
||
"model.compile(loss='sparse_categorical_crossentropy',optimizer='adam',metrics=['acc'])\n",
|
||
"model.fit(ds_train.map(tupelize).batch(batch_size),validation_data=ds_test.map(tupelize).batch(batch_size))\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## 二元組、三元組和 n 元組\n",
|
||
"\n",
|
||
"詞袋方法的一個限制是，有些詞屬於多詞表達，例如「熱狗」這個詞在不同語境下與「熱」和「狗」的意思完全不同。如果我們始終用相同的向量表示「熱」和「狗」，可能會讓模型感到困惑。\n",
|
||
"\n",
|
||
"為了解決這個問題，**n 元組表示法**通常用於文件分類方法中，其中每個詞、雙詞或三詞的頻率是訓練分類器的一個有用特徵。例如，在二元組表示法中，我們會將所有的詞對加入詞彙表中，除了原始詞之外。\n",
|
||
"\n",
|
||
"以下是一個使用 Scikit Learn 生成二元組詞袋表示法的示例：\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 14,
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Vocabulary:\n",
|
||
" {'i': 7, 'like': 11, 'hot': 4, 'dogs': 2, 'i like': 8, 'like hot': 12, 'hot dogs': 5, 'the': 16, 'dog': 0, 'ran': 14, 'fast': 3, 'the dog': 17, 'dog ran': 1, 'ran fast': 15, 'its': 9, 'outside': 13, 'its hot': 10, 'hot outside': 6}\n"
|
||
]
|
||
},
|
||
{
|
||
"data": {
|
||
"text/plain": [
|
||
"array([[1, 0, 1, 0, 2, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]],\n",
|
||
" dtype=int64)"
|
||
]
|
||
},
|
||
"execution_count": 14,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"bigram_vectorizer = CountVectorizer(ngram_range=(1, 2), token_pattern=r'\\b\\w+\\b', min_df=1)\n",
|
||
"corpus = [\n",
|
||
" 'I like hot dogs.',\n",
|
||
" 'The dog ran fast.',\n",
|
||
" 'Its hot outside.',\n",
|
||
" ]\n",
|
||
"bigram_vectorizer.fit_transform(corpus)\n",
|
||
"print(\"Vocabulary:\\n\",bigram_vectorizer.vocabulary_)\n",
|
||
"bigram_vectorizer.transform(['My dog likes hot dogs on a hot day.']).toarray()\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"n-gram 方法的主要缺點是詞彙量會迅速增長。在實際應用中，我們需要將 n-gram 表示法與降維技術（例如 *嵌入*）結合使用，我們會在下一單元中討論這一點。\n",
|
||
"\n",
|
||
"要在我們的 **AG News** 數據集中使用 n-gram 表示法，我們需要將 `ngrams` 參數傳遞給 `TextVectorization` 構造函數。二元語法詞彙的長度**顯著增加**，在我們的例子中，超過了 130 萬個詞元！因此，限制二元語法詞元的數量在合理範圍內是有意義的。\n",
|
||
"\n",
|
||
"我們可以使用與上面相同的代碼來訓練分類器，但這樣做會非常耗費記憶體。在下一單元中，我們將使用嵌入來訓練二元語法分類器。與此同時，您可以在這個筆記本中嘗試訓練二元語法分類器，看看是否能獲得更高的準確率。\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## 自動計算 BoW 向量\n",
|
||
"\n",
|
||
"在上述範例中，我們透過手動方式計算 BoW 向量，方法是將個別單字的一次性編碼相加。然而，最新版本的 TensorFlow 允許我們透過在向量化器建構函數中傳入 `output_mode='count` 參數，自動計算 BoW 向量。這使得定義和訓練模型變得更加簡單：\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 15,
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Training vectorizer\n",
|
||
"938/938 [==============================] - 7s 7ms/step - loss: 0.5929 - acc: 0.8486 - val_loss: 0.4168 - val_acc: 0.8772\n"
|
||
]
|
||
},
|
||
{
|
||
"data": {
|
||
"text/plain": [
|
||
"<keras.callbacks.History at 0x20c725217c0>"
|
||
]
|
||
},
|
||
"execution_count": 15,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"model = keras.models.Sequential([\n",
|
||
" keras.layers.experimental.preprocessing.TextVectorization(max_tokens=vocab_size,output_mode='count'),\n",
|
||
" keras.layers.Dense(4,input_shape=(vocab_size,), activation='softmax')\n",
|
||
"])\n",
|
||
"print(\"Training vectorizer\")\n",
|
||
"model.layers[0].adapt(ds_train.take(500).map(extract_text))\n",
|
||
"model.compile(loss='sparse_categorical_crossentropy',optimizer='adam',metrics=['acc'])\n",
|
||
"model.fit(ds_train.map(tupelize).batch(batch_size),validation_data=ds_test.map(tupelize).batch(batch_size))"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## 詞頻 - 逆文件頻率 (TF-IDF)\n",
|
||
"\n",
|
||
"在 BoW 表示法中，單詞的出現次數無論對於哪個單詞，都是使用相同的技術進行加權。然而，很明顯，像 *a* 和 *in* 這樣的高頻單詞對分類的影響遠不如專業術語。在大多數自然語言處理任務中，有些單詞比其他單詞更為重要。\n",
|
||
"\n",
|
||
"**TF-IDF** 是 **詞頻 - 逆文件頻率** 的縮寫。這是一種基於詞袋模型的變體，其中不再使用二元的 0/1 值來表示單詞是否出現在文件中，而是使用與單詞在語料庫中出現頻率相關的浮點值。\n",
|
||
"\n",
|
||
"更正式地說，單詞 $i$ 在文件 $j$ 中的權重 $w_{ij}$ 定義為：\n",
|
||
"$$\n",
|
||
"w_{ij} = tf_{ij}\\times\\log({N\\over df_i})\n",
|
||
"$$\n",
|
||
"其中：\n",
|
||
"* $tf_{ij}$ 是單詞 $i$ 在文件 $j$ 中出現的次數，也就是我們之前看到的 BoW 值\n",
|
||
"* $N$ 是集合中的文件總數\n",
|
||
"* $df_i$ 是整個集合中包含單詞 $i$ 的文件數量\n",
|
||
"\n",
|
||
"TF-IDF 值 $w_{ij}$ 與單詞在文件中出現的次數成正比，並且會根據語料庫中包含該單詞的文件數量進行調整，這有助於平衡某些單詞出現頻率較高的情況。例如，如果某個單詞出現在集合中的*每一個*文件中，則 $df_i=N$，而 $w_{ij}=0$，這些詞將被完全忽略。\n",
|
||
"\n",
|
||
"您可以使用 Scikit Learn 輕鬆地對文本進行 TF-IDF 向量化：\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 16,
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"data": {
|
||
"text/plain": [
|
||
"array([[0.43381609, 0. , 0.43381609, 0. , 0.65985664,\n",
|
||
" 0.43381609, 0. , 0. , 0. , 0. ,\n",
|
||
" 0. , 0. , 0. , 0. , 0. ,\n",
|
||
" 0. ]])"
|
||
]
|
||
},
|
||
"execution_count": 16,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"from sklearn.feature_extraction.text import TfidfVectorizer\n",
|
||
"vectorizer = TfidfVectorizer(ngram_range=(1,2))\n",
|
||
"vectorizer.fit_transform(corpus)\n",
|
||
"vectorizer.transform(['My dog likes hot dogs on a hot day.']).toarray()"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"在 Keras 中，可以通過傳遞參數 `output_mode='tf-idf'`，讓 `TextVectorization` 層自動計算 TF-IDF 頻率。我們重複上面使用的程式碼，來看看使用 TF-IDF 是否能提高準確性：\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 17,
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Training vectorizer\n",
|
||
"938/938 [==============================] - 12s 12ms/step - loss: 0.4197 - acc: 0.8662 - val_loss: 0.3432 - val_acc: 0.8849\n"
|
||
]
|
||
},
|
||
{
|
||
"data": {
|
||
"text/plain": [
|
||
"<keras.callbacks.History at 0x20c729dfd30>"
|
||
]
|
||
},
|
||
"execution_count": 17,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"model = keras.models.Sequential([\n",
|
||
" keras.layers.experimental.preprocessing.TextVectorization(max_tokens=vocab_size,output_mode='tf-idf'),\n",
|
||
" keras.layers.Dense(4,input_shape=(vocab_size,), activation='softmax')\n",
|
||
"])\n",
|
||
"print(\"Training vectorizer\")\n",
|
||
"model.layers[0].adapt(ds_train.take(500).map(extract_text))\n",
|
||
"model.compile(loss='sparse_categorical_crossentropy',optimizer='adam',metrics=['acc'])\n",
|
||
"model.fit(ds_train.map(tupelize).batch(batch_size),validation_data=ds_test.map(tupelize).batch(batch_size))"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## 結論\n",
|
||
"\n",
|
||
"儘管 TF-IDF 表示法為不同的詞提供了頻率權重，但它無法表達詞義或順序。正如著名語言學家 J. R. Firth 在 1935 年所說：「詞語的完整意義總是與上下文相關，任何脫離上下文的意義研究都不應被認真對待。」在課程的後續部分，我們將學習如何通過語言建模來捕捉文本中的上下文信息。\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"\n---\n\n**免責聲明**： \n本文件已使用 AI 翻譯服務 [Co-op Translator](https://github.com/Azure/co-op-translator) 進行翻譯。儘管我們致力於提供準確的翻譯，請注意自動翻譯可能包含錯誤或不準確之處。原始文件的母語版本應被視為權威來源。對於關鍵資訊，建議使用專業人工翻譯。我們對因使用此翻譯而引起的任何誤解或錯誤解釋不承擔責任。\n"
|
||
]
|
||
}
|
||
],
|
||
"metadata": {
|
||
"interpreter": {
|
||
"hash": "0cb620c6d4b9f7a635928804c26cf22403d89d98d79684e4529119355ee6d5a5"
|
||
},
|
||
"kernel_info": {
|
||
"name": "conda-env-py37_tensorflow-py"
|
||
},
|
||
"kernelspec": {
|
||
"display_name": "py37_tensorflow",
|
||
"language": "python",
|
||
"name": "python3"
|
||
},
|
||
"language_info": {
|
||
"codemirror_mode": {
|
||
"name": "ipython",
|
||
"version": 3
|
||
},
|
||
"file_extension": ".py",
|
||
"mimetype": "text/x-python",
|
||
"name": "python",
|
||
"nbconvert_exporter": "python",
|
||
"pygments_lexer": "ipython3",
|
||
"version": "3.8.12"
|
||
},
|
||
"nteract": {
|
||
"version": "nteract-front-end@1.0.0"
|
||
},
|
||
"coopTranslator": {
|
||
"original_hash": "19b43951d55b377a76209c24c1f017e4",
|
||
"translation_date": "2025-08-31T10:59:55+00:00",
|
||
"source_file": "lessons/5-NLP/13-TextRep/TextRepresentationTF.ipynb",
|
||
"language_code": "tw"
|
||
}
|
||
},
|
||
"nbformat": 4,
|
||
"nbformat_minor": 4
|
||
} |