647 lines
25 KiB
Plaintext
647 lines
25 KiB
Plaintext
{
|
||
"cells": [
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"# 文本分类任务\n",
|
||
"\n",
|
||
"在本模块中，我们将从一个简单的文本分类任务开始，基于 **[AG_NEWS](http://www.di.unipi.it/~gulli/AG_corpus_of_news_articles.html)** 数据集：我们将把新闻标题分类为以下四个类别之一：国际、体育、商业和科技。\n",
|
||
"\n",
|
||
"## 数据集\n",
|
||
"\n",
|
||
"为了加载数据集，我们将使用 **[TensorFlow Datasets](https://www.tensorflow.org/datasets)** API。\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 1,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"import tensorflow as tf\n",
|
||
"from tensorflow import keras\n",
|
||
"import tensorflow_datasets as tfds\n",
|
||
"\n",
|
||
"# In this tutorial, we will be training a lot of models. In order to use GPU memory cautiously,\n",
|
||
"# we will set tensorflow option to grow GPU memory allocation when required.\n",
|
||
"physical_devices = tf.config.list_physical_devices('GPU') \n",
|
||
"if len(physical_devices)>0:\n",
|
||
" tf.config.experimental.set_memory_growth(physical_devices[0], True)\n",
|
||
"\n",
|
||
"dataset = tfds.load('ag_news_subset')"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"我们现在可以通过使用 `dataset['train']` 和 `dataset['test']` 分别访问数据集的训练和测试部分：\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 3,
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Length of train dataset = 120000\n",
|
||
"Length of test dataset = 7600\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"ds_train = dataset['train']\n",
|
||
"ds_test = dataset['test']\n",
|
||
"\n",
|
||
"print(f\"Length of train dataset = {len(ds_train)}\")\n",
|
||
"print(f\"Length of test dataset = {len(ds_test)}\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"让我们打印出数据集中前10条新标题：\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 4,
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"3 (Sci/Tech) -> b'AMD Debuts Dual-Core Opteron Processor' b'AMD #39;s new dual-core Opteron chip is designed mainly for corporate computing applications, including databases, Web services, and financial transactions.'\n",
|
||
"1 (Sports) -> b\"Wood's Suspension Upheld (Reuters)\" b'Reuters - Major League Baseball\\\\Monday announced a decision on the appeal filed by Chicago Cubs\\\\pitcher Kerry Wood regarding a suspension stemming from an\\\\incident earlier this season.'\n",
|
||
"2 (Business) -> b'Bush reform may have blue states seeing red' b'President Bush #39;s quot;revenue-neutral quot; tax reform needs losers to balance its winners, and people claiming the federal deduction for state and local taxes may be in administration planners #39; sights, news reports say.'\n",
|
||
"3 (Sci/Tech) -> b\"'Halt science decline in schools'\" b'Britain will run out of leading scientists unless science education is improved, says Professor Colin Pillinger.'\n",
|
||
"1 (Sports) -> b'Gerrard leaves practice' b'London, England (Sports Network) - England midfielder Steven Gerrard injured his groin late in Thursday #39;s training session, but is hopeful he will be ready for Saturday #39;s World Cup qualifier against Austria.'\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"classes = ['World', 'Sports', 'Business', 'Sci/Tech']\n",
|
||
"\n",
|
||
"for i,x in zip(range(5),ds_train):\n",
|
||
" print(f\"{x['label']} ({classes[x['label']]}) -> {x['title']} {x['description']}\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## 文本向量化\n",
|
||
"\n",
|
||
"现在我们需要将文本转换为可以表示为张量的**数字**。如果我们想要词级别的表示，需要完成以下两件事：\n",
|
||
"\n",
|
||
"* 使用一个**分词器**将文本拆分为**标记**。\n",
|
||
"* 构建这些标记的**词汇表**。\n",
|
||
"\n",
|
||
"### 限制词汇表大小\n",
|
||
"\n",
|
||
"在 AG News 数据集的例子中，词汇表的大小相当大，超过了 10 万个单词。一般来说，我们不需要那些在文本中很少出现的单词——只有少数句子会包含它们，而模型也无法从中学习。因此，通过向向量化器构造函数传递一个参数，将词汇表大小限制为一个较小的数字是合理的：\n",
|
||
"\n",
|
||
"以上两个步骤都可以通过 **TextVectorization** 层来处理。我们可以实例化一个向量化器对象，然后调用 `adapt` 方法来遍历所有文本并构建词汇表：\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 5,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"vocab_size = 50000\n",
|
||
"vectorizer = keras.layers.experimental.preprocessing.TextVectorization(max_tokens=vocab_size)\n",
|
||
"vectorizer.adapt(ds_train.take(500).map(lambda x: x['title']+' '+x['description']))"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"> **注意** 我们仅使用整个数据集的一部分来构建词汇表。这样做是为了加快执行速度，避免让您等待太久。然而，这样做存在一定风险，即整个数据集中的某些词可能不会被包含在词汇表中，并在训练过程中被忽略。因此，使用完整的词汇表大小并在`adapt`过程中遍历整个数据集应该可以提高最终的准确性，但提升幅度不会太大。\n",
|
||
"\n",
|
||
"现在我们可以访问实际的词汇表：\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 6,
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"['', '[UNK]', 'the', 'to', 'a', 'in', 'of', 'and', 'on', 'for']\n",
|
||
"Length of vocabulary: 5335\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"vocab = vectorizer.get_vocabulary()\n",
|
||
"vocab_size = len(vocab)\n",
|
||
"print(vocab[:10])\n",
|
||
"print(f\"Length of vocabulary: {vocab_size}\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"使用向量化器，我们可以轻松地将任何文本编码为一组数字：\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 7,
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"data": {
|
||
"text/plain": [
|
||
"<tf.Tensor: shape=(7,), dtype=int64, numpy=array([ 112, 3695, 3, 304, 11, 1041, 1], dtype=int64)>"
|
||
]
|
||
},
|
||
"execution_count": 7,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"vectorizer('I love to play with my words')"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## 词袋文本表示法\n",
|
||
"\n",
|
||
"由于单词承载着意义，有时我们可以通过单独查看单词，而不考虑它们在句子中的顺序，来理解一段文本的含义。例如，在新闻分类中，像 *天气* 和 *雪* 这样的词可能表明 *天气预报*，而像 *股票* 和 *美元* 这样的词则可能属于 *财经新闻*。\n",
|
||
"\n",
|
||
"**词袋** (BoW) 向量表示法是最容易理解的传统向量表示法。每个单词都与一个向量索引相关联，向量中的元素表示某个文档中每个单词出现的次数。\n",
|
||
"\n",
|
||
" \n",
|
||
"\n",
|
||
"> **注意**: 你也可以将 BoW 理解为文本中每个单词的独热编码向量的总和。\n",
|
||
"\n",
|
||
"下面是一个使用 Scikit Learn Python 库生成词袋表示法的示例：\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 8,
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"data": {
|
||
"text/plain": [
|
||
"array([[1, 1, 0, 2, 0, 0, 0, 0, 0]], dtype=int64)"
|
||
]
|
||
},
|
||
"execution_count": 8,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"from sklearn.feature_extraction.text import CountVectorizer\n",
|
||
"sc_vectorizer = CountVectorizer()\n",
|
||
"corpus = [\n",
|
||
" 'I like hot dogs.',\n",
|
||
" 'The dog ran fast.',\n",
|
||
" 'Its hot outside.',\n",
|
||
" ]\n",
|
||
"sc_vectorizer.fit_transform(corpus)\n",
|
||
"sc_vectorizer.transform(['My dog likes hot dogs on a hot day.']).toarray()"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"我们还可以使用上面定义的Keras向量化器，将每个单词编号转换为独热编码，并将所有这些向量相加：\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 9,
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"data": {
|
||
"text/plain": [
|
||
"array([0., 5., 0., ..., 0., 0., 0.], dtype=float32)"
|
||
]
|
||
},
|
||
"execution_count": 9,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"def to_bow(text):\n",
|
||
" return tf.reduce_sum(tf.one_hot(vectorizer(text),vocab_size),axis=0)\n",
|
||
"\n",
|
||
"to_bow('My dog likes hot dogs on a hot day.').numpy()"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"> **注意**：您可能会惊讶地发现结果与之前的示例不同。原因在于，在 Keras 示例中，向量的长度对应于词汇表的大小，而该词汇表是基于整个 AG News 数据集构建的；而在 Scikit Learn 示例中，我们是根据示例文本动态构建词汇表的。\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## 训练 BoW 分类器\n",
|
||
"\n",
|
||
"现在我们已经了解了如何构建文本的词袋表示，让我们训练一个使用它的分类器。首先，我们需要将数据集转换为词袋表示。这可以通过以下方式使用 `map` 函数来实现：\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 11,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"batch_size = 128\n",
|
||
"\n",
|
||
"ds_train_bow = ds_train.map(lambda x: (to_bow(x['title']+x['description']),x['label'])).batch(batch_size)\n",
|
||
"ds_test_bow = ds_test.map(lambda x: (to_bow(x['title']+x['description']),x['label'])).batch(batch_size)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"现在让我们定义一个简单的分类器神经网络，它包含一个线性层。输入大小是`vocab_size`，输出大小对应于类别数量（4）。由于我们正在解决一个分类任务，最终的激活函数是**softmax**：\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 12,
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"938/938 [==============================] - 66s 70ms/step - loss: 0.6144 - acc: 0.8427 - val_loss: 0.4416 - val_acc: 0.8697\n"
|
||
]
|
||
},
|
||
{
|
||
"data": {
|
||
"text/plain": [
|
||
"<keras.callbacks.History at 0x20c70a947f0>"
|
||
]
|
||
},
|
||
"execution_count": 12,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"model = keras.models.Sequential([\n",
|
||
" keras.layers.Dense(4,activation='softmax',input_shape=(vocab_size,))\n",
|
||
"])\n",
|
||
"model.compile(loss='sparse_categorical_crossentropy',optimizer='adam',metrics=['acc'])\n",
|
||
"model.fit(ds_train_bow,validation_data=ds_test_bow)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"由于我们有4个类别，超过80%的准确率是一个不错的结果。\n",
|
||
"\n",
|
||
"## 将分类器训练为一个网络\n",
|
||
"\n",
|
||
"因为向量化器也是一个 Keras 层，我们可以定义一个包含它的网络，并进行端到端的训练。这样我们就不需要使用 `map` 来向量化数据集，只需将原始数据集传递到网络的输入即可。\n",
|
||
"\n",
|
||
"> **注意**: 我们仍然需要对数据集应用映射操作，以将字段从字典（例如 `title`、`description` 和 `label`）转换为元组。然而，当从磁盘加载数据时，我们可以一开始就构建一个具有所需结构的数据集。\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 13,
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Model: \"model\"\n",
|
||
"_________________________________________________________________\n",
|
||
" Layer (type) Output Shape Param # \n",
|
||
"=================================================================\n",
|
||
" input_1 (InputLayer) [(None, 1)] 0 \n",
|
||
" \n",
|
||
" text_vectorization (TextVec (None, None) 0 \n",
|
||
" torization) \n",
|
||
" \n",
|
||
" tf.one_hot (TFOpLambda) (None, None, 5335) 0 \n",
|
||
" \n",
|
||
" tf.math.reduce_sum (TFOpLam (None, 5335) 0 \n",
|
||
" bda) \n",
|
||
" \n",
|
||
" dense_2 (Dense) (None, 4) 21344 \n",
|
||
" \n",
|
||
"=================================================================\n",
|
||
"Total params: 21,344\n",
|
||
"Trainable params: 21,344\n",
|
||
"Non-trainable params: 0\n",
|
||
"_________________________________________________________________\n",
|
||
"938/938 [==============================] - 73s 77ms/step - loss: 0.6057 - acc: 0.8414 - val_loss: 0.4202 - val_acc: 0.8736\n"
|
||
]
|
||
},
|
||
{
|
||
"data": {
|
||
"text/plain": [
|
||
"<keras.callbacks.History at 0x20c721521f0>"
|
||
]
|
||
},
|
||
"execution_count": 13,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"def extract_text(x):\n",
|
||
" return x['title']+' '+x['description']\n",
|
||
"\n",
|
||
"def tupelize(x):\n",
|
||
" return (extract_text(x),x['label'])\n",
|
||
"\n",
|
||
"inp = keras.Input(shape=(1,),dtype=tf.string)\n",
|
||
"x = vectorizer(inp)\n",
|
||
"x = tf.reduce_sum(tf.one_hot(x,vocab_size),axis=1)\n",
|
||
"out = keras.layers.Dense(4,activation='softmax')(x)\n",
|
||
"model = keras.models.Model(inp,out)\n",
|
||
"model.summary()\n",
|
||
"\n",
|
||
"model.compile(loss='sparse_categorical_crossentropy',optimizer='adam',metrics=['acc'])\n",
|
||
"model.fit(ds_train.map(tupelize).batch(batch_size),validation_data=ds_test.map(tupelize).batch(batch_size))\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## 二元组、三元组和 n 元组\n",
|
||
"\n",
|
||
"词袋模型的一个局限性在于，有些词是多词表达的一部分。例如，“hot dog”这个词的含义与单独的“hot”和“dog”在其他语境中的含义完全不同。如果我们始终用相同的向量表示“hot”和“dog”，可能会让模型感到困惑。\n",
|
||
"\n",
|
||
"为了解决这个问题，**n 元组表示法**常被用于文档分类方法中，其中每个单词、双词组或三词组的频率是训练分类器的一个有用特征。例如，在二元组表示中，我们会将所有的单词对添加到词汇表中，除了原始单词之外。\n",
|
||
"\n",
|
||
"以下是一个使用 Scikit Learn 生成二元组词袋表示的示例：\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 14,
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Vocabulary:\n",
|
||
" {'i': 7, 'like': 11, 'hot': 4, 'dogs': 2, 'i like': 8, 'like hot': 12, 'hot dogs': 5, 'the': 16, 'dog': 0, 'ran': 14, 'fast': 3, 'the dog': 17, 'dog ran': 1, 'ran fast': 15, 'its': 9, 'outside': 13, 'its hot': 10, 'hot outside': 6}\n"
|
||
]
|
||
},
|
||
{
|
||
"data": {
|
||
"text/plain": [
|
||
"array([[1, 0, 1, 0, 2, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]],\n",
|
||
" dtype=int64)"
|
||
]
|
||
},
|
||
"execution_count": 14,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"bigram_vectorizer = CountVectorizer(ngram_range=(1, 2), token_pattern=r'\\b\\w+\\b', min_df=1)\n",
|
||
"corpus = [\n",
|
||
" 'I like hot dogs.',\n",
|
||
" 'The dog ran fast.',\n",
|
||
" 'Its hot outside.',\n",
|
||
" ]\n",
|
||
"bigram_vectorizer.fit_transform(corpus)\n",
|
||
"print(\"Vocabulary:\\n\",bigram_vectorizer.vocabulary_)\n",
|
||
"bigram_vectorizer.transform(['My dog likes hot dogs on a hot day.']).toarray()\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"n-gram 方法的主要缺点是词汇表的大小会迅速增长。在实际应用中，我们需要将 n-gram 表示与降维技术（例如 *embeddings*）结合使用，这将在下一单元中讨论。\n",
|
||
"\n",
|
||
"为了在我们的 **AG News** 数据集中使用 n-gram 表示，我们需要将 `ngrams` 参数传递给 `TextVectorization` 构造函数。二元组词汇表的长度会**显著增加**，在我们的例子中，它超过了 130 万个标记！因此，将二元组标记限制在一个合理的数量范围内是有意义的。\n",
|
||
"\n",
|
||
"我们可以使用与上面相同的代码来训练分类器，但这样做会非常占用内存。在下一单元中，我们将使用 embeddings 来训练二元组分类器。同时，你可以在这个笔记本中尝试训练二元组分类器，看看是否能获得更高的准确率。\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## 自动计算 BoW 向量\n",
|
||
"\n",
|
||
"在上面的例子中，我们通过手动将单个词的一次性编码相加来计算 BoW 向量。然而，最新版本的 TensorFlow 允许我们通过向向量化器构造函数传递 `output_mode='count` 参数来自动计算 BoW 向量。这使得定义和训练我们的模型变得更加简单：\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 15,
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Training vectorizer\n",
|
||
"938/938 [==============================] - 7s 7ms/step - loss: 0.5929 - acc: 0.8486 - val_loss: 0.4168 - val_acc: 0.8772\n"
|
||
]
|
||
},
|
||
{
|
||
"data": {
|
||
"text/plain": [
|
||
"<keras.callbacks.History at 0x20c725217c0>"
|
||
]
|
||
},
|
||
"execution_count": 15,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"model = keras.models.Sequential([\n",
|
||
" keras.layers.experimental.preprocessing.TextVectorization(max_tokens=vocab_size,output_mode='count'),\n",
|
||
" keras.layers.Dense(4,input_shape=(vocab_size,), activation='softmax')\n",
|
||
"])\n",
|
||
"print(\"Training vectorizer\")\n",
|
||
"model.layers[0].adapt(ds_train.take(500).map(extract_text))\n",
|
||
"model.compile(loss='sparse_categorical_crossentropy',optimizer='adam',metrics=['acc'])\n",
|
||
"model.fit(ds_train.map(tupelize).batch(batch_size),validation_data=ds_test.map(tupelize).batch(batch_size))"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## 词频-逆文档频率 (TF-IDF)\n",
|
||
"\n",
|
||
"在 BoW 表示法中，无论单词本身如何，单词出现的权重都使用相同的技术进行计算。然而，很明显，像 *a* 和 *in* 这样的高频词对于分类的意义远不如一些专业术语。在大多数 NLP 任务中，有些词比其他词更重要。\n",
|
||
"\n",
|
||
"**TF-IDF** 是 **词频-逆文档频率** 的缩写。它是袋子模型（bag-of-words）的一个变体，其中不是用二进制的 0/1 值来表示单词是否出现在文档中，而是使用一个浮点值，该值与单词在语料库中出现的频率相关。\n",
|
||
"\n",
|
||
"更正式地说，单词 $i$ 在文档 $j$ 中的权重 $w_{ij}$ 定义为：\n",
|
||
"$$\n",
|
||
"w_{ij} = tf_{ij}\\times\\log({N\\over df_i})\n",
|
||
"$$\n",
|
||
"其中：\n",
|
||
"* $tf_{ij}$ 是单词 $i$ 在文档 $j$ 中出现的次数，即我们之前看到的 BoW 值\n",
|
||
"* $N$ 是语料库中的文档总数\n",
|
||
"* $df_i$ 是整个语料库中包含单词 $i$ 的文档数量\n",
|
||
"\n",
|
||
"TF-IDF 值 $w_{ij}$ 随着单词在文档中出现的次数成比例增加，同时会根据语料库中包含该单词的文档数量进行调整。这有助于解决某些单词比其他单词更频繁出现的问题。例如，如果某个单词出现在语料库的*每一篇*文档中，那么 $df_i=N$，且 $w_{ij}=0$，这些词将被完全忽略。\n",
|
||
"\n",
|
||
"你可以使用 Scikit Learn 轻松创建文本的 TF-IDF 向量化：\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 16,
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"data": {
|
||
"text/plain": [
|
||
"array([[0.43381609, 0. , 0.43381609, 0. , 0.65985664,\n",
|
||
" 0.43381609, 0. , 0. , 0. , 0. ,\n",
|
||
" 0. , 0. , 0. , 0. , 0. ,\n",
|
||
" 0. ]])"
|
||
]
|
||
},
|
||
"execution_count": 16,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"from sklearn.feature_extraction.text import TfidfVectorizer\n",
|
||
"vectorizer = TfidfVectorizer(ngram_range=(1,2))\n",
|
||
"vectorizer.fit_transform(corpus)\n",
|
||
"vectorizer.transform(['My dog likes hot dogs on a hot day.']).toarray()"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"在 Keras 中，通过传递参数 `output_mode='tf-idf'`，`TextVectorization` 层可以自动计算 TF-IDF 频率。让我们重复上面的代码，看看使用 TF-IDF 是否能提高准确性：\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 17,
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Training vectorizer\n",
|
||
"938/938 [==============================] - 12s 12ms/step - loss: 0.4197 - acc: 0.8662 - val_loss: 0.3432 - val_acc: 0.8849\n"
|
||
]
|
||
},
|
||
{
|
||
"data": {
|
||
"text/plain": [
|
||
"<keras.callbacks.History at 0x20c729dfd30>"
|
||
]
|
||
},
|
||
"execution_count": 17,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"model = keras.models.Sequential([\n",
|
||
" keras.layers.experimental.preprocessing.TextVectorization(max_tokens=vocab_size,output_mode='tf-idf'),\n",
|
||
" keras.layers.Dense(4,input_shape=(vocab_size,), activation='softmax')\n",
|
||
"])\n",
|
||
"print(\"Training vectorizer\")\n",
|
||
"model.layers[0].adapt(ds_train.take(500).map(extract_text))\n",
|
||
"model.compile(loss='sparse_categorical_crossentropy',optimizer='adam',metrics=['acc'])\n",
|
||
"model.fit(ds_train.map(tupelize).batch(batch_size),validation_data=ds_test.map(tupelize).batch(batch_size))"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## 结论\n",
|
||
"\n",
|
||
"尽管TF-IDF表示法为不同的词赋予了频率权重，但它无法表达词语的含义或顺序。正如著名语言学家J.R. Firth在1935年所说：“一个词的完整含义总是与上下文相关，任何脱离上下文的意义研究都不值得认真对待。”在课程的后续部分，我们将学习如何通过语言模型从文本中捕捉上下文信息。\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"\n---\n\n**免责声明**： \n本文档使用AI翻译服务[Co-op Translator](https://github.com/Azure/co-op-translator)进行翻译。尽管我们努力确保准确性，但请注意，自动翻译可能包含错误或不准确之处。应以原始语言的文档作为权威来源。对于关键信息，建议使用专业人工翻译。因使用本翻译而导致的任何误解或误读，我们概不负责。\n"
|
||
]
|
||
}
|
||
],
|
||
"metadata": {
|
||
"interpreter": {
|
||
"hash": "0cb620c6d4b9f7a635928804c26cf22403d89d98d79684e4529119355ee6d5a5"
|
||
},
|
||
"kernel_info": {
|
||
"name": "conda-env-py37_tensorflow-py"
|
||
},
|
||
"kernelspec": {
|
||
"display_name": "py37_tensorflow",
|
||
"language": "python",
|
||
"name": "python3"
|
||
},
|
||
"language_info": {
|
||
"codemirror_mode": {
|
||
"name": "ipython",
|
||
"version": 3
|
||
},
|
||
"file_extension": ".py",
|
||
"mimetype": "text/x-python",
|
||
"name": "python",
|
||
"nbconvert_exporter": "python",
|
||
"pygments_lexer": "ipython3",
|
||
"version": "3.8.12"
|
||
},
|
||
"nteract": {
|
||
"version": "nteract-front-end@1.0.0"
|
||
},
|
||
"coopTranslator": {
|
||
"original_hash": "19b43951d55b377a76209c24c1f017e4",
|
||
"translation_date": "2025-08-31T11:00:44+00:00",
|
||
"source_file": "lessons/5-NLP/13-TextRep/TextRepresentationTF.ipynb",
|
||
"language_code": "zh"
|
||
}
|
||
},
|
||
"nbformat": 4,
|
||
"nbformat_minor": 4
|
||
} |