{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 文本分类任务\n",
    "\n",
    "如前所述，我们将专注于基于 **AG_NEWS** 数据集的简单文本分类任务，该任务是将新闻标题分类为以下四个类别之一：国际、体育、商业和科技。\n",
    "\n",
    "## 数据集\n",
    "\n",
    "此数据集已集成到 [`torchtext`](https://github.com/pytorch/text) 模块中，因此我们可以轻松访问它。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torchtext\n",
    "import os\n",
    "import collections\n",
    "os.makedirs('./data',exist_ok=True)\n",
    "train_dataset, test_dataset = torchtext.datasets.AG_NEWS(root='./data')\n",
    "classes = ['World', 'Sports', 'Business', 'Sci/Tech']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在这里，`train_dataset` 和 `test_dataset` 包含返回标签（类别编号）和文本对的集合，例如：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(3,\n",
       " \"Wall St. Bears Claw Back Into the Black (Reuters) Reuters - Short-sellers, Wall Street's dwindling\\\\band of ultra-cynics, are seeing green again.\")"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(train_dataset)[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "那么，让我们打印出数据集中前10条新标题：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "**Sci/Tech** -> Wall St. Bears Claw Back Into the Black (Reuters) Reuters - Short-sellers, Wall Street's dwindling\\band of ultra-cynics, are seeing green again.\n",
      "**Sci/Tech** -> Carlyle Looks Toward Commercial Aerospace (Reuters) Reuters - Private investment firm Carlyle Group,\\which has a reputation for making well-timed and occasionally\\controversial plays in the defense industry, has quietly placed\\its bets on another part of the market.\n",
      "**Sci/Tech** -> Oil and Economy Cloud Stocks' Outlook (Reuters) Reuters - Soaring crude prices plus worries\\about the economy and the outlook for earnings are expected to\\hang over the stock market next week during the depth of the\\summer doldrums.\n",
      "**Sci/Tech** -> Iraq Halts Oil Exports from Main Southern Pipeline (Reuters) Reuters - Authorities have halted oil export\\flows from the main pipeline in southern Iraq after\\intelligence showed a rebel militia could strike\\infrastructure, an oil official said on Saturday.\n",
      "**Sci/Tech** -> Oil prices soar to all-time record, posing new menace to US economy (AFP) AFP - Tearaway world oil prices, toppling records and straining wallets, present a new economic menace barely three months before the US presidential elections.\n"
     ]
    }
   ],
   "source": [
    "for i,x in zip(range(5),train_dataset):\n",
    "    print(f\"**{classes[x[0]]}** -> {x[1]}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "因为数据集是迭代器，如果我们想多次使用数据，则需要将其转换为列表：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_dataset, test_dataset = torchtext.datasets.AG_NEWS(root='./data')\n",
    "train_dataset = list(train_dataset)\n",
    "test_dataset = list(test_dataset)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 分词\n",
    "\n",
    "现在我们需要将文本转换为可以表示为张量的**数字**。如果我们想要基于单词的表示，需要完成两件事：\n",
    "* 使用**分词器**将文本拆分为**标记**\n",
    "* 构建这些标记的**词汇表**。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['he', 'said', 'hello']"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer = torchtext.data.utils.get_tokenizer('basic_english')\n",
    "tokenizer('He said: hello')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "counter = collections.Counter()\n",
    "for (label, line) in train_dataset:\n",
    "    counter.update(tokenizer(line))\n",
    "vocab = torchtext.vocab.vocab(counter, min_freq=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "使用词汇表，我们可以轻松地将标记化的字符串编码为一组数字：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Vocab size if 95810\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[599, 3279, 97, 1220, 329, 225, 7368]"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vocab_size = len(vocab)\n",
    "print(f\"Vocab size if {vocab_size}\")\n",
    "\n",
    "stoi = vocab.get_stoi() # dict to convert tokens to indices\n",
    "\n",
    "def encode(x):\n",
    "    return [stoi[s] for s in tokenizer(x)]\n",
    "\n",
    "encode('I love to play with my words')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 词袋文本表示法\n",
    "\n",
    "由于单词代表了意义，有时我们可以通过仅仅查看单个单词，而不考虑它们在句子中的顺序，就能理解一段文本的含义。例如，在分类新闻时，像 *weather*（天气）、*snow*（雪）这样的单词可能表明是 *天气预报*，而像 *stocks*（股票）、*dollar*（美元）这样的单词则可能属于 *财经新闻*。\n",
    "\n",
    "**词袋**（Bag of Words, BoW）向量表示法是最常用的传统向量表示法。每个单词都与一个向量索引相关联，向量的元素包含某个单词在给定文档中出现的次数。\n",
    "\n",
    "![展示词袋向量表示在内存中如何表示的图片。](../../../../../lessons/5-NLP/13-TextRep/images/bag-of-words-example.png) \n",
    "\n",
    "> **Note**: 你也可以将 BoW 理解为文本中每个单词的独热编码向量的总和。\n",
    "\n",
    "下面是一个使用 Scikit Learn Python 库生成词袋表示的示例：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1, 1, 0, 2, 0, 0, 0, 0, 0]], dtype=int64)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "vectorizer = CountVectorizer()\n",
    "corpus = [\n",
    "        'I like hot dogs.',\n",
    "        'The dog ran fast.',\n",
    "        'Its hot outside.',\n",
    "    ]\n",
    "vectorizer.fit_transform(corpus)\n",
    "vectorizer.transform(['My dog likes hot dogs on a hot day.']).toarray()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "要从我们的 AG_NEWS 数据集的向量表示计算词袋向量，可以使用以下函数：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([2., 1., 2.,  ..., 0., 0., 0.])\n"
     ]
    }
   ],
   "source": [
    "vocab_size = len(vocab)\n",
    "\n",
    "def to_bow(text,bow_vocab_size=vocab_size):\n",
    "    res = torch.zeros(bow_vocab_size,dtype=torch.float32)\n",
    "    for i in encode(text):\n",
    "        if i<bow_vocab_size:\n",
    "            res[i] += 1\n",
    "    return res\n",
    "\n",
    "print(to_bow(train_dataset[0][1]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> **注意：** 这里我们使用全局变量 `vocab_size` 来指定默认的词汇表大小。由于词汇表的大小通常非常大，我们可以将词汇表的大小限制为最常用的词汇。尝试降低 `vocab_size` 的值并运行下面的代码，看看它如何影响准确性。你应该预期会有一些准确性的下降，但不会太剧烈，以换取更高的性能。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 训练 BoW 分类器\n",
    "\n",
    "现在我们已经学习了如何构建文本的词袋表示，让我们在其基础上训练一个分类器。首先，我们需要将数据集转换为适合训练的形式，即将所有位置向量表示转换为词袋表示。这可以通过将 `bowify` 函数作为标准 torch `DataLoader` 的 `collate_fn` 参数传递来实现：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torch.utils.data import DataLoader\n",
    "import numpy as np \n",
    "\n",
    "# this collate function gets list of batch_size tuples, and needs to \n",
    "# return a pair of label-feature tensors for the whole minibatch\n",
    "def bowify(b):\n",
    "    return (\n",
    "            torch.LongTensor([t[0]-1 for t in b]),\n",
    "            torch.stack([to_bow(t[1]) for t in b])\n",
    "    )\n",
    "\n",
    "train_loader = DataLoader(train_dataset, batch_size=16, collate_fn=bowify, shuffle=True)\n",
    "test_loader = DataLoader(test_dataset, batch_size=16, collate_fn=bowify, shuffle=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "现在让我们定义一个简单的分类器神经网络，它包含一个线性层。输入向量的大小等于 `vocab_size`，输出大小对应于类别的数量（4）。由于我们正在解决分类任务，最终的激活函数是 `LogSoftmax()`。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "net = torch.nn.Sequential(torch.nn.Linear(vocab_size,4),torch.nn.LogSoftmax(dim=1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "现在我们将定义标准的PyTorch训练循环。由于我们的数据集相当大，为了教学目的，我们将只训练一个周期，有时甚至少于一个周期（通过指定`epoch_size`参数可以限制训练）。我们还将在训练过程中报告累计的训练准确率；报告的频率通过`report_freq`参数指定。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train_epoch(net,dataloader,lr=0.01,optimizer=None,loss_fn = torch.nn.NLLLoss(),epoch_size=None, report_freq=200):\n",
    "    optimizer = optimizer or torch.optim.Adam(net.parameters(),lr=lr)\n",
    "    net.train()\n",
    "    total_loss,acc,count,i = 0,0,0,0\n",
    "    for labels,features in dataloader:\n",
    "        optimizer.zero_grad()\n",
    "        out = net(features)\n",
    "        loss = loss_fn(out,labels) #cross_entropy(out,labels)\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "        total_loss+=loss\n",
    "        _,predicted = torch.max(out,1)\n",
    "        acc+=(predicted==labels).sum()\n",
    "        count+=len(labels)\n",
    "        i+=1\n",
    "        if i%report_freq==0:\n",
    "            print(f\"{count}: acc={acc.item()/count}\")\n",
    "        if epoch_size and count>epoch_size:\n",
    "            break\n",
    "    return total_loss.item()/count, acc.item()/count"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3200: acc=0.8028125\n",
      "6400: acc=0.8371875\n",
      "9600: acc=0.8534375\n",
      "12800: acc=0.85765625\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(0.026090790722161722, 0.8620069296375267)"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_epoch(net,train_loader,epoch_size=15000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 二元组、三元组和N元组\n",
    "\n",
    "词袋方法的一个局限性是，有些词是多词表达的一部分。例如，“热狗”这个词的含义与“热”和“狗”在其他语境中的含义完全不同。如果我们总是用相同的向量表示“热”和“狗”，可能会让模型感到困惑。\n",
    "\n",
    "为了解决这个问题，**N元组表示**通常用于文档分类方法中，其中每个单词、双词或三词的频率是训练分类器的有用特征。例如，在二元组表示中，我们会将所有的词对添加到词汇表中，除了原始单词之外。\n",
    "\n",
    "下面是一个使用Scikit Learn生成二元组词袋表示的示例：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Vocabulary:\n",
      " {'i': 7, 'like': 11, 'hot': 4, 'dogs': 2, 'i like': 8, 'like hot': 12, 'hot dogs': 5, 'the': 16, 'dog': 0, 'ran': 14, 'fast': 3, 'the dog': 17, 'dog ran': 1, 'ran fast': 15, 'its': 9, 'outside': 13, 'its hot': 10, 'hot outside': 6}\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array([[1, 0, 1, 0, 2, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]],\n",
       "      dtype=int64)"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bigram_vectorizer = CountVectorizer(ngram_range=(1, 2), token_pattern=r'\\b\\w+\\b', min_df=1)\n",
    "corpus = [\n",
    "        'I like hot dogs.',\n",
    "        'The dog ran fast.',\n",
    "        'Its hot outside.',\n",
    "    ]\n",
    "bigram_vectorizer.fit_transform(corpus)\n",
    "print(\"Vocabulary:\\n\",bigram_vectorizer.vocabulary_)\n",
    "bigram_vectorizer.transform(['My dog likes hot dogs on a hot day.']).toarray()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "N-gram方法的主要缺点是词汇量会迅速增长。在实际应用中，我们需要将N-gram表示与一些降维技术结合使用，例如*嵌入*，我们将在下一单元中讨论。\n",
    "\n",
    "为了在我们的**AG News**数据集中使用N-gram表示，我们需要构建一个特殊的N-gram词汇：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Bigram vocabulary length =  1308842\n"
     ]
    }
   ],
   "source": [
    "counter = collections.Counter()\n",
    "for (label, line) in train_dataset:\n",
    "    l = tokenizer(line)\n",
    "    counter.update(torchtext.data.utils.ngrams_iterator(l,ngrams=2))\n",
    "    \n",
    "bi_vocab = torchtext.vocab.vocab(counter, min_freq=1)\n",
    "\n",
    "print(\"Bigram vocabulary length = \",len(bi_vocab))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们可以使用上述相同的代码来训练分类器，但这样会非常占用内存。在下一节中，我们将使用嵌入来训练二元分类器。\n",
    "\n",
    "> **注意:** 你只能保留那些在文本中出现次数超过指定数量的n元组。这将确保不常见的二元组被省略，并显著减少维度。为此，可以将`min_freq`参数设置为更高的值，并观察词汇表长度的变化。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 词频-逆文档频率 TF-IDF\n",
    "\n",
    "在 BoW 表示法中，单词的出现被均等对待，无论单词本身如何。然而，很明显，像 *a*、*in* 这样的常见词对于分类的作用远不如一些专业术语重要。实际上，在大多数 NLP 任务中，有些词比其他词更相关。\n",
    "\n",
    "**TF-IDF** 是 **词频-逆文档频率** 的缩写。它是袋子模型的一种变体，其中不是用二进制的 0/1 值来表示单词是否出现在文档中，而是使用一个浮点值，该值与单词在语料库中的出现频率相关。\n",
    "\n",
    "更正式地说，单词 $i$ 在文档 $j$ 中的权重 $w_{ij}$ 定义为：\n",
    "$$\n",
    "w_{ij} = tf_{ij}\\times\\log({N\\over df_i})\n",
    "$$\n",
    "其中：\n",
    "* $tf_{ij}$ 是单词 $i$ 在文档 $j$ 中出现的次数，即我们之前看到的 BoW 值\n",
    "* $N$ 是语料库中的文档总数\n",
    "* $df_i$ 是整个语料库中包含单词 $i$ 的文档数量\n",
    "\n",
    "TF-IDF 值 $w_{ij}$ 随着单词在文档中出现的次数成比例增加，同时会受到语料库中包含该单词的文档数量的影响，从而调整某些单词出现频率较高的情况。例如，如果某个单词出现在语料库中的*每一篇*文档中，那么 $df_i=N$，且 $w_{ij}=0$，这些词将被完全忽略。\n",
    "\n",
    "你可以使用 Scikit Learn 轻松实现文本的 TF-IDF 向量化：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.43381609, 0.        , 0.43381609, 0.        , 0.65985664,\n",
       "        0.43381609, 0.        , 0.        , 0.        , 0.        ,\n",
       "        0.        , 0.        , 0.        , 0.        , 0.        ,\n",
       "        0.        ]])"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "vectorizer = TfidfVectorizer(ngram_range=(1,2))\n",
    "vectorizer.fit_transform(corpus)\n",
    "vectorizer.transform(['My dog likes hot dogs on a hot day.']).toarray()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 结论\n",
    "\n",
    "尽管TF-IDF表示法为不同的单词提供了频率权重，但它无法表达意义或顺序。正如著名语言学家J.R. Firth在1935年所说：“一个词的完整意义总是与上下文相关，任何脱离上下文的意义研究都不应被认真对待。” 在课程的后续部分，我们将学习如何通过语言建模从文本中捕捉上下文信息。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n---\n\n**免责声明**：  \n本文档使用AI翻译服务[Co-op Translator](https://github.com/Azure/co-op-translator)进行翻译。尽管我们努力确保翻译的准确性，但请注意，自动翻译可能包含错误或不准确之处。应以原始语言的文档作为权威来源。对于关键信息，建议使用专业人工翻译。我们对因使用本翻译而引起的任何误解或误读不承担责任。\n"
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "16af2a8bbb083ea23e5e41c7f5787656b2ce26968575d8763f2c4b17f9cd711f"
  },
  "kernelspec": {
   "display_name": "Python 3.8.12 ('py38')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.12"
  },
  "coopTranslator": {
   "original_hash": "7b9040985e748e4e2d4c689892456ad7",
   "translation_date": "2025-08-31T10:58:15+00:00",
   "source_file": "lessons/5-NLP/13-TextRep/TextRepresentationPyTorch.ipynb",
   "language_code": "zh"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}