{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 文本分類任務\n",
    "\n",
    "如前所述，我們將專注於基於 **AG_NEWS** 數據集的簡單文本分類任務，該任務是將新聞標題分類為以下四個類別之一：世界、體育、商業和科學/技術。\n",
    "\n",
    "## 數據集\n",
    "\n",
    "此數據集已內建於 [`torchtext`](https://github.com/pytorch/text) 模組中，因此我們可以輕鬆訪問它。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torchtext\n",
    "import os\n",
    "import collections\n",
    "os.makedirs('./data',exist_ok=True)\n",
    "train_dataset, test_dataset = torchtext.datasets.AG_NEWS(root='./data')\n",
    "classes = ['World', 'Sports', 'Business', 'Sci/Tech']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在這裡，`train_dataset` 和 `test_dataset` 包含分別返回標籤（類別數字）和文本的集合，例如：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(3,\n",
       " \"Wall St. Bears Claw Back Into the Black (Reuters) Reuters - Short-sellers, Wall Street's dwindling\\\\band of ultra-cynics, are seeing green again.\")"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(train_dataset)[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "那麼，讓我們從數據集中列印出前 10 條新標題：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "**Sci/Tech** -> Wall St. Bears Claw Back Into the Black (Reuters) Reuters - Short-sellers, Wall Street's dwindling\\band of ultra-cynics, are seeing green again.\n",
      "**Sci/Tech** -> Carlyle Looks Toward Commercial Aerospace (Reuters) Reuters - Private investment firm Carlyle Group,\\which has a reputation for making well-timed and occasionally\\controversial plays in the defense industry, has quietly placed\\its bets on another part of the market.\n",
      "**Sci/Tech** -> Oil and Economy Cloud Stocks' Outlook (Reuters) Reuters - Soaring crude prices plus worries\\about the economy and the outlook for earnings are expected to\\hang over the stock market next week during the depth of the\\summer doldrums.\n",
      "**Sci/Tech** -> Iraq Halts Oil Exports from Main Southern Pipeline (Reuters) Reuters - Authorities have halted oil export\\flows from the main pipeline in southern Iraq after\\intelligence showed a rebel militia could strike\\infrastructure, an oil official said on Saturday.\n",
      "**Sci/Tech** -> Oil prices soar to all-time record, posing new menace to US economy (AFP) AFP - Tearaway world oil prices, toppling records and straining wallets, present a new economic menace barely three months before the US presidential elections.\n"
     ]
    }
   ],
   "source": [
    "for i,x in zip(range(5),train_dataset):\n",
    "    print(f\"**{classes[x[0]]}** -> {x[1]}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "因為數據集是迭代器，如果我們想多次使用數據，就需要將其轉換為列表：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_dataset, test_dataset = torchtext.datasets.AG_NEWS(root='./data')\n",
    "train_dataset = list(train_dataset)\n",
    "test_dataset = list(test_dataset)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 分詞\n",
    "\n",
    "現在我們需要將文本轉換為可以表示為張量的**數字**。如果我們想要基於單詞的表示，需要完成以下兩件事：\n",
    "* 使用**分詞器**將文本拆分為**詞元**\n",
    "* 建立這些詞元的**詞彙表**。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['he', 'said', 'hello']"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer = torchtext.data.utils.get_tokenizer('basic_english')\n",
    "tokenizer('He said: hello')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "counter = collections.Counter()\n",
    "for (label, line) in train_dataset:\n",
    "    counter.update(tokenizer(line))\n",
    "vocab = torchtext.vocab.vocab(counter, min_freq=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "使用詞彙，我們可以輕鬆地將標記化的字串編碼為一組數字：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Vocab size if 95810\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[599, 3279, 97, 1220, 329, 225, 7368]"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vocab_size = len(vocab)\n",
    "print(f\"Vocab size if {vocab_size}\")\n",
    "\n",
    "stoi = vocab.get_stoi() # dict to convert tokens to indices\n",
    "\n",
    "def encode(x):\n",
    "    return [stoi[s] for s in tokenizer(x)]\n",
    "\n",
    "encode('I love to play with my words')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 詞袋文字表示法\n",
    "\n",
    "由於文字代表意義，有時我們可以僅透過查看個別的文字來理解文本的含義，而不需要考慮它們在句子中的順序。例如，在分類新聞時，像 *weather*、*snow* 這樣的詞可能表明是 *天氣預報*，而像 *stocks*、*dollar* 則可能屬於 *財經新聞*。\n",
    "\n",
    "**詞袋** (BoW) 向量表示法是最常用的傳統向量表示法之一。每個詞都與向量索引相關聯，向量元素包含某個詞在特定文件中出現的次數。\n",
    "\n",
    "![顯示詞袋向量表示法如何在記憶體中表示的圖片。](../../../../../lessons/5-NLP/13-TextRep/images/bag-of-words-example.png) \n",
    "\n",
    "> **Note**: 你也可以將 BoW 想像成文本中每個詞的所有獨立一熱編碼向量的總和。\n",
    "\n",
    "以下是一個使用 Scikit Learn Python 庫生成詞袋表示法的範例：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1, 1, 0, 2, 0, 0, 0, 0, 0]], dtype=int64)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "vectorizer = CountVectorizer()\n",
    "corpus = [\n",
    "        'I like hot dogs.',\n",
    "        'The dog ran fast.',\n",
    "        'Its hot outside.',\n",
    "    ]\n",
    "vectorizer.fit_transform(corpus)\n",
    "vectorizer.transform(['My dog likes hot dogs on a hot day.']).toarray()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "要從我們的 AG_NEWS 數據集的向量表示計算詞袋向量，可以使用以下函數：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([2., 1., 2.,  ..., 0., 0., 0.])\n"
     ]
    }
   ],
   "source": [
    "vocab_size = len(vocab)\n",
    "\n",
    "def to_bow(text,bow_vocab_size=vocab_size):\n",
    "    res = torch.zeros(bow_vocab_size,dtype=torch.float32)\n",
    "    for i in encode(text):\n",
    "        if i<bow_vocab_size:\n",
    "            res[i] += 1\n",
    "    return res\n",
    "\n",
    "print(to_bow(train_dataset[0][1]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> **注意：** 這裡我們使用全域變數 `vocab_size` 來指定詞彙表的預設大小。由於詞彙表的大小通常相當大，我們可以將詞彙表的大小限制為最常出現的詞彙。嘗試降低 `vocab_size` 的值並執行以下程式碼，看看它如何影響準確性。你應該預期準確性會有所下降，但不會太劇烈，以換取更高的效能。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 訓練 BoW 分類器\n",
    "\n",
    "現在我們已經學會如何構建文本的詞袋（Bag-of-Words）表示，接下來讓我們基於它來訓練一個分類器。首先，我們需要將數據集轉換為適合訓練的格式，也就是將所有的位置向量表示轉換為詞袋表示。這可以通過將 `bowify` 函數作為 `collate_fn` 參數傳遞給標準的 torch `DataLoader` 來實現：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torch.utils.data import DataLoader\n",
    "import numpy as np \n",
    "\n",
    "# this collate function gets list of batch_size tuples, and needs to \n",
    "# return a pair of label-feature tensors for the whole minibatch\n",
    "def bowify(b):\n",
    "    return (\n",
    "            torch.LongTensor([t[0]-1 for t in b]),\n",
    "            torch.stack([to_bow(t[1]) for t in b])\n",
    "    )\n",
    "\n",
    "train_loader = DataLoader(train_dataset, batch_size=16, collate_fn=bowify, shuffle=True)\n",
    "test_loader = DataLoader(test_dataset, batch_size=16, collate_fn=bowify, shuffle=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "現在讓我們定義一個簡單的分類器神經網絡，其中包含一個線性層。輸入向量的大小等於 `vocab_size`，輸出大小對應於類別數量（4）。由於我們正在解決分類任務，最終的激活函數是 `LogSoftmax()`。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "net = torch.nn.Sequential(torch.nn.Linear(vocab_size,4),torch.nn.LogSoftmax(dim=1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "現在我們將定義標準的 PyTorch 訓練循環。由於我們的數據集相當大，出於教學目的，我們將僅訓練一個 epoch，有時甚至少於一個 epoch（通過指定 `epoch_size` 參數可以限制訓練）。我們還會在訓練過程中報告累積的訓練準確率；報告的頻率是通過 `report_freq` 參數指定的。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train_epoch(net,dataloader,lr=0.01,optimizer=None,loss_fn = torch.nn.NLLLoss(),epoch_size=None, report_freq=200):\n",
    "    optimizer = optimizer or torch.optim.Adam(net.parameters(),lr=lr)\n",
    "    net.train()\n",
    "    total_loss,acc,count,i = 0,0,0,0\n",
    "    for labels,features in dataloader:\n",
    "        optimizer.zero_grad()\n",
    "        out = net(features)\n",
    "        loss = loss_fn(out,labels) #cross_entropy(out,labels)\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "        total_loss+=loss\n",
    "        _,predicted = torch.max(out,1)\n",
    "        acc+=(predicted==labels).sum()\n",
    "        count+=len(labels)\n",
    "        i+=1\n",
    "        if i%report_freq==0:\n",
    "            print(f\"{count}: acc={acc.item()/count}\")\n",
    "        if epoch_size and count>epoch_size:\n",
    "            break\n",
    "    return total_loss.item()/count, acc.item()/count"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3200: acc=0.8028125\n",
      "6400: acc=0.8371875\n",
      "9600: acc=0.8534375\n",
      "12800: acc=0.85765625\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(0.026090790722161722, 0.8620069296375267)"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_epoch(net,train_loader,epoch_size=15000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## BiGrams、TriGrams 和 N-Grams\n",
    "\n",
    "袋裝詞方法的一個限制是，有些詞是多詞表達的一部分。例如，詞語「熱狗」的意思與「熱」和「狗」在其他語境中的意思完全不同。如果我們始終用相同的向量表示「熱」和「狗」，可能會讓模型感到困惑。\n",
    "\n",
    "為了解決這個問題，**N-Gram 表示法**通常用於文件分類的方法中，其中每個詞、雙詞或三詞的頻率是訓練分類器的有用特徵。例如，在雙詞表示法中，我們會將所有的詞對加入詞彙表，除了原始詞語之外。\n",
    "\n",
    "以下是一個使用 Scikit Learn 生成雙詞袋裝詞表示法的範例：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Vocabulary:\n",
      " {'i': 7, 'like': 11, 'hot': 4, 'dogs': 2, 'i like': 8, 'like hot': 12, 'hot dogs': 5, 'the': 16, 'dog': 0, 'ran': 14, 'fast': 3, 'the dog': 17, 'dog ran': 1, 'ran fast': 15, 'its': 9, 'outside': 13, 'its hot': 10, 'hot outside': 6}\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array([[1, 0, 1, 0, 2, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]],\n",
       "      dtype=int64)"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bigram_vectorizer = CountVectorizer(ngram_range=(1, 2), token_pattern=r'\\b\\w+\\b', min_df=1)\n",
    "corpus = [\n",
    "        'I like hot dogs.',\n",
    "        'The dog ran fast.',\n",
    "        'Its hot outside.',\n",
    "    ]\n",
    "bigram_vectorizer.fit_transform(corpus)\n",
    "print(\"Vocabulary:\\n\",bigram_vectorizer.vocabulary_)\n",
    "bigram_vectorizer.transform(['My dog likes hot dogs on a hot day.']).toarray()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "N-gram 方法的主要缺點是詞彙量會以極快的速度增長。實際應用中，我們需要將 N-gram 表示法與一些降維技術結合使用，例如 *嵌入*，我們會在下一單元中討論這些技術。\n",
    "\n",
    "為了在我們的 **AG News** 數據集中使用 N-gram 表示法，我們需要構建專門的 ngram 詞彙表：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Bigram vocabulary length =  1308842\n"
     ]
    }
   ],
   "source": [
    "counter = collections.Counter()\n",
    "for (label, line) in train_dataset:\n",
    "    l = tokenizer(line)\n",
    "    counter.update(torchtext.data.utils.ngrams_iterator(l,ngrams=2))\n",
    "    \n",
    "bi_vocab = torchtext.vocab.vocab(counter, min_freq=1)\n",
    "\n",
    "print(\"Bigram vocabulary length = \",len(bi_vocab))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我們可以使用上述相同的程式碼來訓練分類器，但這樣做會非常耗費記憶體。在下一個單元中，我們將使用嵌入來訓練二元分類器。\n",
    "\n",
    "> **注意:** 你只能保留那些在文本中出現次數超過指定數量的 ngram。這將確保不常見的二元組會被省略，並顯著降低維度。為此，將 `min_freq` 參數設置為更高的值，並觀察詞彙表的長度變化。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 詞頻-逆文檔頻率 TF-IDF\n",
    "\n",
    "在 BoW 表示法中，詞的出現次數被均等地加權，無論該詞本身如何。然而，很明顯，像 *a*、*in* 等這些常見詞對分類的作用遠不如專業術語重要。事實上，在大多數 NLP 任務中，有些詞比其他詞更具相關性。\n",
    "\n",
    "**TF-IDF** 代表 **詞頻–逆文檔頻率**。它是袋子模型（BoW）的變體，與使用二進制 0/1 值表示詞在文檔中的出現不同，TF-IDF 使用浮點值，該值與詞在語料庫中的出現頻率相關。\n",
    "\n",
    "更正式地說，詞 $i$ 在文檔 $j$ 中的權重 $w_{ij}$ 定義為：\n",
    "$$\n",
    "w_{ij} = tf_{ij}\\times\\log({N\\over df_i})\n",
    "$$\n",
    "其中：\n",
    "* $tf_{ij}$ 是詞 $i$ 在文檔 $j$ 中的出現次數，即我們之前看到的 BoW 值\n",
    "* $N$ 是集合中的文檔數量\n",
    "* $df_i$ 是整個集合中包含詞 $i$ 的文檔數量\n",
    "\n",
    "TF-IDF 值 $w_{ij}$ 隨著詞在文檔中出現次數的增加而增加，同時受到包含該詞的文檔數量的影響，這有助於調整某些詞比其他詞出現更頻繁的情況。例如，如果某個詞出現在集合中的*每一個*文檔中，則 $df_i=N$，而 $w_{ij}=0$，這些詞將被完全忽略。\n",
    "\n",
    "您可以使用 Scikit Learn 輕鬆地創建文本的 TF-IDF 向量化：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.43381609, 0.        , 0.43381609, 0.        , 0.65985664,\n",
       "        0.43381609, 0.        , 0.        , 0.        , 0.        ,\n",
       "        0.        , 0.        , 0.        , 0.        , 0.        ,\n",
       "        0.        ]])"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "vectorizer = TfidfVectorizer(ngram_range=(1,2))\n",
    "vectorizer.fit_transform(corpus)\n",
    "vectorizer.transform(['My dog likes hot dogs on a hot day.']).toarray()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 結論\n",
    "\n",
    "然而，儘管 TF-IDF 表示法為不同的詞提供了頻率權重，但它無法表達詞義或順序。正如著名語言學家 J. R. Firth 在1935年所說：「詞語的完整意義總是與上下文相關，任何脫離上下文的意義研究都不應被認真對待。」在課程的後續部分，我們將學習如何通過語言建模從文本中捕捉上下文信息。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n---\n\n**免責聲明**：  \n本文件已使用 AI 翻譯服務 [Co-op Translator](https://github.com/Azure/co-op-translator) 進行翻譯。儘管我們努力確保翻譯的準確性，但請注意，自動翻譯可能包含錯誤或不準確之處。原始文件的母語版本應被視為權威來源。對於關鍵資訊，建議使用專業人工翻譯。我們對因使用此翻譯而引起的任何誤解或錯誤解釋不承擔責任。\n"
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "16af2a8bbb083ea23e5e41c7f5787656b2ce26968575d8763f2c4b17f9cd711f"
  },
  "kernelspec": {
   "display_name": "Python 3.8.12 ('py38')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.12"
  },
  "coopTranslator": {
   "original_hash": "7b9040985e748e4e2d4c689892456ad7",
   "translation_date": "2025-08-31T10:57:29+00:00",
   "source_file": "lessons/5-NLP/13-TextRep/TextRepresentationPyTorch.ipynb",
   "language_code": "tw"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}