{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# 文本分類任務\n", "\n", "在本模組中，我們將從一個簡單的文本分類任務開始，基於 **[AG_NEWS](http://www.di.unipi.it/~gulli/AG_corpus_of_news_articles.html)** 數據集：我們將把新聞標題分類為以下四個類別之一：世界、體育、商業和科學/技術。\n", "\n", "## 數據集\n", "\n", "為了載入數據集，我們將使用 **[TensorFlow Datasets](https://www.tensorflow.org/datasets)** API。\n" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "import tensorflow as tf\n", "from tensorflow import keras\n", "import tensorflow_datasets as tfds\n", "\n", "# In this tutorial, we will be training a lot of models. In order to use GPU memory cautiously,\n", "# we will set tensorflow option to grow GPU memory allocation when required.\n", "physical_devices = tf.config.list_physical_devices('GPU') \n", "if len(physical_devices)>0:\n", " tf.config.experimental.set_memory_growth(physical_devices[0], True)\n", "\n", "dataset = tfds.load('ag_news_subset')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "我們現在可以分別使用 `dataset['train']` 和 `dataset['test']` 訪問數據集的訓練和測試部分：\n" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Length of train dataset = 120000\n", "Length of test dataset = 7600\n" ] } ], "source": [ "ds_train = dataset['train']\n", "ds_test = dataset['test']\n", "\n", "print(f\"Length of train dataset = {len(ds_train)}\")\n", "print(f\"Length of test dataset = {len(ds_test)}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "讓我們列印出資料集中前10個新的標題：\n" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "3 (Sci/Tech) -> b'AMD Debuts Dual-Core Opteron Processor' b'AMD #39;s new dual-core Opteron chip is designed mainly for corporate computing applications, including databases, Web services, and financial transactions.'\n", "1 (Sports) -> b\"Wood's Suspension Upheld (Reuters)\" b'Reuters - Major League Baseball\\\\Monday announced a decision on the appeal filed by Chicago Cubs\\\\pitcher Kerry Wood regarding a suspension stemming from an\\\\incident earlier this season.'\n", "2 (Business) -> b'Bush reform may have blue states seeing red' b'President Bush #39;s quot;revenue-neutral quot; tax reform needs losers to balance its winners, and people claiming the federal deduction for state and local taxes may be in administration planners #39; sights, news reports say.'\n", "3 (Sci/Tech) -> b\"'Halt science decline in schools'\" b'Britain will run out of leading scientists unless science education is improved, says Professor Colin Pillinger.'\n", "1 (Sports) -> b'Gerrard leaves practice' b'London, England (Sports Network) - England midfielder Steven Gerrard injured his groin late in Thursday #39;s training session, but is hopeful he will be ready for Saturday #39;s World Cup qualifier against Austria.'\n" ] } ], "source": [ "classes = ['World', 'Sports', 'Business', 'Sci/Tech']\n", "\n", "for i,x in zip(range(5),ds_train):\n", " print(f\"{x['label']} ({classes[x['label']]}) -> {x['title']} {x['description']}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 文本向量化\n", "\n", "現在我們需要將文本轉換成可以表示為張量的**數字**。如果我們想要詞級表示，需要完成以下兩件事：\n", "\n", "* 使用**分詞器**將文本拆分成**詞元**。\n", "* 建立這些詞元的**詞彙表**。\n", "\n", "### 限制詞彙表大小\n", "\n", "在 AG News 數據集的例子中，詞彙表的大小相當大，超過 10 萬個詞。一般來說，我們不需要那些在文本中很少出現的詞——只有少數句子會包含它們，而模型無法從中學習。因此，通過向向量化器構造函數傳遞參數，限制詞彙表大小到一個較小的數量是合理的。\n", "\n", "以上兩個步驟都可以使用 **TextVectorization** 層來處理。接下來，我們來實例化向量化器對象，然後調用 `adapt` 方法，遍歷所有文本並建立詞彙表：\n" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [], "source": [ "vocab_size = 50000\n", "vectorizer = keras.layers.experimental.preprocessing.TextVectorization(max_tokens=vocab_size)\n", "vectorizer.adapt(ds_train.take(500).map(lambda x: x['title']+' '+x['description']))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "> **注意** 我們僅使用整個數據集的一部分來建立詞彙表。這樣做是為了加快執行速度，避免讓您等待太久。然而，這樣做也存在一定風險，即整個數據集中的某些詞可能不會被包含在詞彙表中，並在訓練過程中被忽略。因此，使用完整的詞彙表大小並在 `adapt` 過程中遍歷整個數據集，應該能提升最終的準確性，但提升幅度不會太大。\n", "\n", "現在我們可以訪問實際的詞彙表：\n" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "['', '[UNK]', 'the', 'to', 'a', 'in', 'of', 'and', 'on', 'for']\n", "Length of vocabulary: 5335\n" ] } ], "source": [ "vocab = vectorizer.get_vocabulary()\n", "vocab_size = len(vocab)\n", "print(vocab[:10])\n", "print(f\"Length of vocabulary: {vocab_size}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "使用向量化器，我們可以輕鬆地將任何文本編碼為一組數字：\n" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "vectorizer('I love to play with my words')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 詞袋（Bag-of-words）文本表示法\n", "\n", "由於文字具有意義，有時我們可以僅通過查看單個詞語來理解一段文本的含義，而不需要考慮它們在句子中的順序。例如，在分類新聞時，像 *weather* 和 *snow* 這樣的詞語可能表明這是與 *天氣預報* 有關的內容，而像 *stocks* 和 *dollar* 則可能屬於 *財經新聞*。\n", "\n", "**詞袋**（Bag-of-words, BoW）向量表示法是最簡單易懂的傳統向量表示法。每個詞語都對應到向量中的一個索引，而向量中的元素則表示該詞語在特定文檔中出現的次數。\n", "\n", "![顯示詞袋向量表示法在記憶體中如何表示的圖片。](../../../../../translated_images/bag-of-words-example.606fc1738f1d7ba98a9d693e3bcd706c6e83fa7bf8221e6e90d1a206d82f2ea4.mo.png) \n", "\n", "> **注意**：你也可以將 BoW 理解為文本中每個詞語的單熱編碼（one-hot-encoded）向量的總和。\n", "\n", "以下是一個使用 Scikit Learn Python 庫生成詞袋表示法的範例：\n" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "array([[1, 1, 0, 2, 0, 0, 0, 0, 0]], dtype=int64)" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from sklearn.feature_extraction.text import CountVectorizer\n", "sc_vectorizer = CountVectorizer()\n", "corpus = [\n", " 'I like hot dogs.',\n", " 'The dog ran fast.',\n", " 'Its hot outside.',\n", " ]\n", "sc_vectorizer.fit_transform(corpus)\n", "sc_vectorizer.transform(['My dog likes hot dogs on a hot day.']).toarray()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "我們也可以使用我們在上面定義的 Keras 向量化器，將每個單詞編號轉換為一個獨熱編碼，然後將所有這些向量相加：\n" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "array([0., 5., 0., ..., 0., 0., 0.], dtype=float32)" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "def to_bow(text):\n", " return tf.reduce_sum(tf.one_hot(vectorizer(text),vocab_size),axis=0)\n", "\n", "to_bow('My dog likes hot dogs on a hot day.').numpy()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "> **注意**：您可能會驚訝地發現結果與之前的例子不同。原因是在 Keras 的例子中，向量的長度對應於詞彙表的大小，而該詞彙表是基於整個 AG News 數據集構建的；而在 Scikit Learn 的例子中，我們是即時從樣本文本中構建詞彙表的。\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 訓練 BoW 分類器\n", "\n", "現在我們已經學會如何建立文字的詞袋表示法，接下來讓我們訓練一個使用該表示法的分類器。首先，我們需要將數據集轉換為詞袋表示法。這可以通過以下方式使用 `map` 函數來實現：\n" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [], "source": [ "batch_size = 128\n", "\n", "ds_train_bow = ds_train.map(lambda x: (to_bow(x['title']+x['description']),x['label'])).batch(batch_size)\n", "ds_test_bow = ds_test.map(lambda x: (to_bow(x['title']+x['description']),x['label'])).batch(batch_size)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "現在讓我們定義一個簡單的分類器神經網絡，其中包含一個線性層。輸入大小為 `vocab_size`，輸出大小對應於類別數量（4）。由於我們正在解決分類任務，最終的激活函數是 **softmax**：\n" ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "938/938 [==============================] - 66s 70ms/step - loss: 0.6144 - acc: 0.8427 - val_loss: 0.4416 - val_acc: 0.8697\n" ] }, { "data": { "text/plain": [ "" ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "model = keras.models.Sequential([\n", " keras.layers.Dense(4,activation='softmax',input_shape=(vocab_size,))\n", "])\n", "model.compile(loss='sparse_categorical_crossentropy',optimizer='adam',metrics=['acc'])\n", "model.fit(ds_train_bow,validation_data=ds_test_bow)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "由於我們有四個類別，準確率超過 80% 就是一個不錯的結果。\n", "\n", "## 將分類器作為一個網絡進行訓練\n", "\n", "因為向量化器也是一個 Keras 層，我們可以定義一個包含它的網絡，並進行端到端的訓練。這樣我們就不需要使用 `map` 來向量化數據集，只需將原始數據集傳遞到網絡的輸入即可。\n", "\n", "> **注意**：我們仍然需要對數據集應用映射操作，將字典中的字段（例如 `title`、`description` 和 `label`）轉換為元組。然而，當從磁盤加載數據時，我們可以一開始就構建具有所需結構的數據集。\n" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Model: \"model\"\n", "_________________________________________________________________\n", " Layer (type) Output Shape Param # \n", "=================================================================\n", " input_1 (InputLayer) [(None, 1)] 0 \n", " \n", " text_vectorization (TextVec (None, None) 0 \n", " torization) \n", " \n", " tf.one_hot (TFOpLambda) (None, None, 5335) 0 \n", " \n", " tf.math.reduce_sum (TFOpLam (None, 5335) 0 \n", " bda) \n", " \n", " dense_2 (Dense) (None, 4) 21344 \n", " \n", "=================================================================\n", "Total params: 21,344\n", "Trainable params: 21,344\n", "Non-trainable params: 0\n", "_________________________________________________________________\n", "938/938 [==============================] - 73s 77ms/step - loss: 0.6057 - acc: 0.8414 - val_loss: 0.4202 - val_acc: 0.8736\n" ] }, { "data": { "text/plain": [ "" ] }, "execution_count": 13, "metadata": {}, "output_type": "execute_result" } ], "source": [ "def extract_text(x):\n", " return x['title']+' '+x['description']\n", "\n", "def tupelize(x):\n", " return (extract_text(x),x['label'])\n", "\n", "inp = keras.Input(shape=(1,),dtype=tf.string)\n", "x = vectorizer(inp)\n", "x = tf.reduce_sum(tf.one_hot(x,vocab_size),axis=1)\n", "out = keras.layers.Dense(4,activation='softmax')(x)\n", "model = keras.models.Model(inp,out)\n", "model.summary()\n", "\n", "model.compile(loss='sparse_categorical_crossentropy',optimizer='adam',metrics=['acc'])\n", "model.fit(ds_train.map(tupelize).batch(batch_size),validation_data=ds_test.map(tupelize).batch(batch_size))\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 二元詞組、三元詞組與 n 元詞組\n", "\n", "袋裝詞語方法的一個限制是，有些詞語屬於多詞組表達，例如「熱狗」這個詞的意思與「熱」和「狗」在其他語境中的意思完全不同。如果我們始終用相同的向量來表示「熱」和「狗」，可能會讓模型感到困惑。\n", "\n", "為了解決這個問題，**n 元詞組表示法**通常用於文件分類方法中，其中每個詞語、二詞組或三詞組的頻率都是訓練分類器的有用特徵。例如，在二元詞組表示法中，我們會將所有的詞語對加入詞彙表中，除了原始詞語之外。\n", "\n", "以下是一個使用 Scikit Learn 生成二元詞組袋裝詞語表示法的範例：\n" ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Vocabulary:\n", " {'i': 7, 'like': 11, 'hot': 4, 'dogs': 2, 'i like': 8, 'like hot': 12, 'hot dogs': 5, 'the': 16, 'dog': 0, 'ran': 14, 'fast': 3, 'the dog': 17, 'dog ran': 1, 'ran fast': 15, 'its': 9, 'outside': 13, 'its hot': 10, 'hot outside': 6}\n" ] }, { "data": { "text/plain": [ "array([[1, 0, 1, 0, 2, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]],\n", " dtype=int64)" ] }, "execution_count": 14, "metadata": {}, "output_type": "execute_result" } ], "source": [ "bigram_vectorizer = CountVectorizer(ngram_range=(1, 2), token_pattern=r'\\b\\w+\\b', min_df=1)\n", "corpus = [\n", " 'I like hot dogs.',\n", " 'The dog ran fast.',\n", " 'Its hot outside.',\n", " ]\n", "bigram_vectorizer.fit_transform(corpus)\n", "print(\"Vocabulary:\\n\",bigram_vectorizer.vocabulary_)\n", "bigram_vectorizer.transform(['My dog likes hot dogs on a hot day.']).toarray()\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "n-gram 方法的主要缺點是詞彙表的大小會開始以極快的速度增長。實際操作中，我們需要將 n-gram 表示法與降維技術（例如 *嵌入*）結合使用，我們將在下一單元中討論這一點。\n", "\n", "要在我們的 **AG News** 數據集中使用 n-gram 表示法，我們需要將 `ngrams` 參數傳遞給 `TextVectorization` 構造函數。二元語法詞彙表的長度**顯著更大**，在我們的例子中，它超過了 130 萬個詞元！因此，限制二元語法詞元的數量在某個合理範圍內是有意義的。\n", "\n", "我們可以使用與上面相同的代碼來訓練分類器，但這樣做會非常浪費記憶體。在下一單元中，我們將使用嵌入來訓練二元語法分類器。與此同時，你可以在這個 notebook 中嘗試訓練二元語法分類器，看看是否能獲得更高的準確率。\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 自動計算 BoW 向量\n", "\n", "在上面的例子中，我們透過手動方式計算 BoW 向量，方法是將個別單詞的一次性編碼相加。然而，最新版本的 TensorFlow 允許我們透過在向量化器構造函數中傳入 `output_mode='count` 參數，自動計算 BoW 向量。這使得定義和訓練模型變得更加簡單：\n" ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Training vectorizer\n", "938/938 [==============================] - 7s 7ms/step - loss: 0.5929 - acc: 0.8486 - val_loss: 0.4168 - val_acc: 0.8772\n" ] }, { "data": { "text/plain": [ "" ] }, "execution_count": 15, "metadata": {}, "output_type": "execute_result" } ], "source": [ "model = keras.models.Sequential([\n", " keras.layers.experimental.preprocessing.TextVectorization(max_tokens=vocab_size,output_mode='count'),\n", " keras.layers.Dense(4,input_shape=(vocab_size,), activation='softmax')\n", "])\n", "print(\"Training vectorizer\")\n", "model.layers[0].adapt(ds_train.take(500).map(extract_text))\n", "model.compile(loss='sparse_categorical_crossentropy',optimizer='adam',metrics=['acc'])\n", "model.fit(ds_train.map(tupelize).batch(batch_size),validation_data=ds_test.map(tupelize).batch(batch_size))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 詞頻 - 逆文件頻率 (TF-IDF)\n", "\n", "在詞袋表示法中，詞的出現次數使用相同的技術進行加權，而不考慮詞本身。然而，很明顯像 *a* 和 *in* 這樣的常見詞對分類的作用遠不如專業術語。在大多數自然語言處理任務中，有些詞比其他詞更具相關性。\n", "\n", "**TF-IDF** 代表 **詞頻 - 逆文件頻率**。這是一種詞袋的變體，其中不是用二進制的 0/1 值來表示詞在文件中的出現，而是使用浮點值，該值與詞在語料庫中的出現頻率相關。\n", "\n", "更正式地說，詞 $i$ 在文件 $j$ 中的權重 $w_{ij}$ 定義為：\n", "$$\n", "w_{ij} = tf_{ij}\\times\\log({N\\over df_i})\n", "$$\n", "其中：\n", "* $tf_{ij}$ 是詞 $i$ 在文件 $j$ 中的出現次數，也就是我們之前看到的詞袋值\n", "* $N$ 是集合中的文件數量\n", "* $df_i$ 是包含詞 $i$ 的文件數量\n", "\n", "TF-IDF 值 $w_{ij}$ 與詞在文件中出現的次數成正比，並且會根據語料庫中包含該詞的文件數量進行調整，這有助於平衡某些詞出現頻率較高的情況。例如，如果某個詞出現在集合中的 *每一個* 文件中，則 $df_i=N$，而 $w_{ij}=0$，這些詞將被完全忽略。\n", "\n", "您可以使用 Scikit Learn 輕鬆地創建文本的 TF-IDF 向量化：\n" ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "array([[0.43381609, 0. , 0.43381609, 0. , 0.65985664,\n", " 0.43381609, 0. , 0. , 0. , 0. ,\n", " 0. , 0. , 0. , 0. , 0. ,\n", " 0. ]])" ] }, "execution_count": 16, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from sklearn.feature_extraction.text import TfidfVectorizer\n", "vectorizer = TfidfVectorizer(ngram_range=(1,2))\n", "vectorizer.fit_transform(corpus)\n", "vectorizer.transform(['My dog likes hot dogs on a hot day.']).toarray()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "在 Keras 中，`TextVectorization` 層可以通過傳遞 `output_mode='tf-idf'` 參數自動計算 TF-IDF 頻率。我們重複上面使用的代碼來看看使用 TF-IDF 是否能提高準確性：\n" ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Training vectorizer\n", "938/938 [==============================] - 12s 12ms/step - loss: 0.4197 - acc: 0.8662 - val_loss: 0.3432 - val_acc: 0.8849\n" ] }, { "data": { "text/plain": [ "" ] }, "execution_count": 17, "metadata": {}, "output_type": "execute_result" } ], "source": [ "model = keras.models.Sequential([\n", " keras.layers.experimental.preprocessing.TextVectorization(max_tokens=vocab_size,output_mode='tf-idf'),\n", " keras.layers.Dense(4,input_shape=(vocab_size,), activation='softmax')\n", "])\n", "print(\"Training vectorizer\")\n", "model.layers[0].adapt(ds_train.take(500).map(extract_text))\n", "model.compile(loss='sparse_categorical_crossentropy',optimizer='adam',metrics=['acc'])\n", "model.fit(ds_train.map(tupelize).batch(batch_size),validation_data=ds_test.map(tupelize).batch(batch_size))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 結論\n", "\n", "儘管 TF-IDF 表示法為不同的詞語提供了頻率權重，但它無法表達詞語的意義或順序。正如著名語言學家 J. R. Firth 在 1935 年所說：「一個詞語的完整意義總是與上下文相關，任何脫離上下文的意義研究都不應被認真對待。」在課程的後續部分，我們將學習如何通過語言模型從文本中捕捉上下文信息。\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n---\n\n**免責聲明**： \n本文件已使用 AI 翻譯服務 [Co-op Translator](https://github.com/Azure/co-op-translator) 進行翻譯。儘管我們努力確保翻譯的準確性，但請注意，自動翻譯可能包含錯誤或不準確之處。原始文件的母語版本應被視為權威來源。對於關鍵信息，建議尋求專業人工翻譯。我們對因使用此翻譯而引起的任何誤解或錯誤解釋不承擔責任。\n" ] } ], "metadata": { "interpreter": { "hash": "0cb620c6d4b9f7a635928804c26cf22403d89d98d79684e4529119355ee6d5a5" }, "kernel_info": { "name": "conda-env-py37_tensorflow-py" }, "kernelspec": { "display_name": "py37_tensorflow", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.12" }, "nteract": { "version": "nteract-front-end@1.0.0" }, "coopTranslator": { "original_hash": "19b43951d55b377a76209c24c1f017e4", "translation_date": "2025-08-28T12:35:41+00:00", "source_file": "lessons/5-NLP/13-TextRep/TextRepresentationTF.ipynb", "language_code": "mo" } }, "nbformat": 4, "nbformat_minor": 4 }