414 lines
20 KiB
Plaintext
414 lines
20 KiB
Plaintext
{
|
||
"cells": [
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"# 生成式網絡\n",
|
||
"\n",
|
||
"循環神經網絡（RNNs）及其門控單元變體，例如長短期記憶單元（LSTMs）和門控循環單元（GRUs），提供了一種語言建模的機制，也就是說，它們可以學習詞語的排列順序，並對序列中的下一個詞進行預測。這使得我們可以使用 RNNs 進行**生成任務**，例如普通文本生成、機器翻譯，甚至是圖像描述。\n",
|
||
"\n",
|
||
"在上一單元中討論的 RNN 架構中，每個 RNN 單元都會生成下一個隱藏狀態作為輸出。然而，我們也可以為每個循環單元添加另一個輸出，這樣就可以輸出一個**序列**（其長度與原始序列相等）。此外，我們還可以使用不在每一步接受輸入的 RNN 單元，而僅僅接受一些初始狀態向量，然後生成一系列輸出。\n",
|
||
"\n",
|
||
"在這份筆記中，我們將專注於幫助生成文本的簡單生成模型。為了簡化，我們將構建**字元級網絡**，逐字生成文本。在訓練過程中，我們需要使用一些文本語料庫，並將其拆分為字元序列。\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 1,
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Loading dataset...\n",
|
||
"Building vocab...\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"import torch\n",
|
||
"import torchtext\n",
|
||
"import numpy as np\n",
|
||
"from torchnlp import *\n",
|
||
"train_dataset,test_dataset,classes,vocab = load_dataset()"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## 建立字元詞彙表\n",
|
||
"\n",
|
||
"要建立字元級生成網絡，我們需要將文本拆分為單個字元，而不是單詞。這可以通過定義不同的分詞器來完成：\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 2,
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Vocabulary size = 82\n",
|
||
"Encoding of 'a' is 1\n",
|
||
"Character with code 13 is c\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"def char_tokenizer(words):\n",
|
||
" return list(words) #[word for word in words]\n",
|
||
"\n",
|
||
"counter = collections.Counter()\n",
|
||
"for (label, line) in train_dataset:\n",
|
||
" counter.update(char_tokenizer(line))\n",
|
||
"vocab = torchtext.vocab.vocab(counter)\n",
|
||
"\n",
|
||
"vocab_size = len(vocab)\n",
|
||
"print(f\"Vocabulary size = {vocab_size}\")\n",
|
||
"print(f\"Encoding of 'a' is {vocab.get_stoi()['a']}\")\n",
|
||
"print(f\"Character with code 13 is {vocab.get_itos()[13]}\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"讓我們看看如何從我們的數據集中編碼文本的示例：\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 3,
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"data": {
|
||
"text/plain": [
|
||
"tensor([ 0, 1, 2, 2, 3, 4, 5, 6, 3, 7, 8, 1, 9, 10, 3, 11, 2, 1,\n",
|
||
" 12, 3, 7, 1, 13, 14, 3, 15, 16, 5, 17, 3, 5, 18, 8, 3, 7, 2,\n",
|
||
" 1, 13, 14, 3, 19, 20, 8, 21, 5, 8, 9, 10, 22, 3, 20, 8, 21, 5,\n",
|
||
" 8, 9, 10, 3, 23, 3, 4, 18, 17, 9, 5, 23, 10, 8, 2, 2, 8, 9,\n",
|
||
" 10, 24, 3, 0, 1, 2, 2, 3, 4, 5, 9, 8, 8, 5, 25, 10, 3, 26,\n",
|
||
" 12, 27, 16, 26, 2, 27, 16, 28, 29, 30, 1, 16, 26, 3, 17, 31, 3, 21,\n",
|
||
" 2, 5, 9, 1, 23, 13, 32, 16, 27, 13, 10, 24, 3, 1, 9, 8, 3, 10,\n",
|
||
" 8, 8, 27, 16, 28, 3, 28, 9, 8, 8, 16, 3, 1, 28, 1, 27, 16, 6])"
|
||
]
|
||
},
|
||
"execution_count": 3,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"def enc(x):\n",
|
||
" return torch.LongTensor(encode(x,voc=vocab,tokenizer=char_tokenizer))\n",
|
||
"\n",
|
||
"enc(train_dataset[0][1])"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## 訓練生成式 RNN\n",
|
||
"\n",
|
||
"我們將以以下方式訓練 RNN 來生成文本。在每一步中，我們會取一段長度為 `nchars` 的字元序列，並讓網路為每個輸入字元生成下一個輸出字元：\n",
|
||
"\n",
|
||
"\n",
|
||
"\n",
|
||
"根據實際情況，我們可能還需要加入一些特殊字元，例如 *序列結束符* `<eos>`。在我們的例子中，我們只希望訓練網路進行無限文本生成，因此我們會將每個序列的大小固定為 `nchars` 個標記。因此，每個訓練樣本將包含 `nchars` 個輸入和 `nchars` 個輸出（輸出是將輸入序列向左移動一個符號後的結果）。一個小批次（minibatch）將由多個這樣的序列組成。\n",
|
||
"\n",
|
||
"我們生成小批次的方式是取每段長度為 `l` 的新聞文本，並從中生成所有可能的輸入-輸出組合（這樣的組合會有 `l-nchars` 個）。這些組合將構成一個小批次，而每次訓練步驟的小批次大小會有所不同。\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 4,
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"data": {
|
||
"text/plain": [
|
||
"(tensor([[ 0, 1, 2, ..., 28, 29, 30],\n",
|
||
" [ 1, 2, 2, ..., 29, 30, 1],\n",
|
||
" [ 2, 2, 3, ..., 30, 1, 16],\n",
|
||
" ...,\n",
|
||
" [20, 8, 21, ..., 1, 28, 1],\n",
|
||
" [ 8, 21, 5, ..., 28, 1, 27],\n",
|
||
" [21, 5, 8, ..., 1, 27, 16]]),\n",
|
||
" tensor([[ 1, 2, 2, ..., 29, 30, 1],\n",
|
||
" [ 2, 2, 3, ..., 30, 1, 16],\n",
|
||
" [ 2, 3, 4, ..., 1, 16, 26],\n",
|
||
" ...,\n",
|
||
" [ 8, 21, 5, ..., 28, 1, 27],\n",
|
||
" [21, 5, 8, ..., 1, 27, 16],\n",
|
||
" [ 5, 8, 9, ..., 27, 16, 6]]))"
|
||
]
|
||
},
|
||
"execution_count": 4,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"nchars = 100\n",
|
||
"\n",
|
||
"def get_batch(s,nchars=nchars):\n",
|
||
" ins = torch.zeros(len(s)-nchars,nchars,dtype=torch.long,device=device)\n",
|
||
" outs = torch.zeros(len(s)-nchars,nchars,dtype=torch.long,device=device)\n",
|
||
" for i in range(len(s)-nchars):\n",
|
||
" ins[i] = enc(s[i:i+nchars])\n",
|
||
" outs[i] = enc(s[i+1:i+nchars+1])\n",
|
||
" return ins,outs\n",
|
||
"\n",
|
||
"get_batch(train_dataset[0][1])"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"現在我們來定義生成器網路。它可以基於我們在上一單元討論過的任何一種循環單元（簡單的、LSTM 或 GRU）。在我們的例子中，我們將使用 LSTM。\n",
|
||
"\n",
|
||
"由於網路的輸入是字符，且詞彙表的大小相對較小，因此我們不需要嵌入層，使用獨熱編碼的輸入可以直接傳遞到 LSTM 單元。然而，因為我們是以字符的數字作為輸入，所以在傳遞給 LSTM 之前需要對它們進行獨熱編碼。這可以在 `forward` 傳遞過程中調用 `one_hot` 函數來完成。輸出編碼器將是一個線性層，用於將隱藏狀態轉換為獨熱編碼的輸出。\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 5,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"class LSTMGenerator(torch.nn.Module):\n",
|
||
" def __init__(self, vocab_size, hidden_dim):\n",
|
||
" super().__init__()\n",
|
||
" self.rnn = torch.nn.LSTM(vocab_size,hidden_dim,batch_first=True)\n",
|
||
" self.fc = torch.nn.Linear(hidden_dim, vocab_size)\n",
|
||
"\n",
|
||
" def forward(self, x, s=None):\n",
|
||
" x = torch.nn.functional.one_hot(x,vocab_size).to(torch.float32)\n",
|
||
" x,s = self.rnn(x,s)\n",
|
||
" return self.fc(x),s"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"在訓練過程中，我們希望能夠抽樣生成的文本。為了達到這個目的，我們將定義一個 `generate` 函數，該函數會生成長度為 `size` 的輸出字串，並以初始字串 `start` 作為起點。\n",
|
||
"\n",
|
||
"其運作方式如下：首先，我們會將整個初始字串通過網絡，並獲得輸出狀態 `s` 和下一個預測字符 `out`。由於 `out` 是獨熱編碼（one-hot encoded），我們使用 `argmax` 來獲取該字符在詞彙表中的索引 `nc`，然後利用 `itos` 找出實際字符，並將其附加到結果字符列表 `chars` 中。這個生成單個字符的過程會重複執行 `size` 次，以生成所需數量的字符。\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 8,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"def generate(net,size=100,start='today '):\n",
|
||
" chars = list(start)\n",
|
||
" out, s = net(enc(chars).view(1,-1).to(device))\n",
|
||
" for i in range(size):\n",
|
||
" nc = torch.argmax(out[0][-1])\n",
|
||
" chars.append(vocab.get_itos()[nc])\n",
|
||
" out, s = net(nc.view(1,-1),s)\n",
|
||
" return ''.join(chars)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"現在開始訓練吧！訓練迴圈幾乎和之前的例子一樣，但這次我們不是打印準確率，而是每隔 1000 個 epoch 打印生成的樣本文字。\n",
|
||
"\n",
|
||
"需要特別注意的是計算損失的方式。我們需要根據 one-hot 編碼的輸出 `out` 和期望的文字 `text_out`（即字符索引的列表）來計算損失。幸運的是，`cross_entropy` 函數的第一個參數是未正規化的網絡輸出，第二個參數是類別編號，這正好符合我們的需求。它還會自動對小批量的大小進行平均。\n",
|
||
"\n",
|
||
"此外，我們通過 `samples_to_train` 限制訓練樣本的數量，以避免等待過久。我們鼓勵你進行實驗，嘗試更長時間的訓練，可能是多個 epoch（在這種情況下，你需要在這段代碼外再建立一個迴圈）。\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 9,
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Current loss = 4.398899078369141\n",
|
||
"today sr sr sr sr sr sr sr sr sr sr sr sr sr sr sr sr sr sr sr sr sr sr sr sr sr sr sr sr sr sr sr sr sr s\n",
|
||
"Current loss = 2.161320447921753\n",
|
||
"today and to the tor to to the tor to to the tor to to the tor to to the tor to to the tor to to the tor t\n",
|
||
"Current loss = 1.6722588539123535\n",
|
||
"today and the court to the could to the could to the could to the could to the could to the could to the c\n",
|
||
"Current loss = 2.423795223236084\n",
|
||
"today and a second to the conternation of the conternation of the conternation of the conternation of the \n",
|
||
"Current loss = 1.702607274055481\n",
|
||
"today and the company to the company to the company to the company to the company to the company to the co\n",
|
||
"Current loss = 1.692358136177063\n",
|
||
"today and the company to the company to the company to the company to the company to the company to the co\n",
|
||
"Current loss = 1.9722288846969604\n",
|
||
"today and the control the control the control the control the control the control the control the control \n",
|
||
"Current loss = 1.8705692291259766\n",
|
||
"today and the second to the second to the second to the second to the second to the second to the second t\n",
|
||
"Current loss = 1.7626899480819702\n",
|
||
"today and a security and a security and a security and a security and a security and a security and a secu\n",
|
||
"Current loss = 1.5574463605880737\n",
|
||
"today and the company and the company and the company and the company and the company and the company and \n",
|
||
"Current loss = 1.5620026588439941\n",
|
||
"today and the be that the be the be that the be the be that the be the be that the be the be that the be t\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"net = LSTMGenerator(vocab_size,64).to(device)\n",
|
||
"\n",
|
||
"samples_to_train = 10000\n",
|
||
"optimizer = torch.optim.Adam(net.parameters(),0.01)\n",
|
||
"loss_fn = torch.nn.CrossEntropyLoss()\n",
|
||
"net.train()\n",
|
||
"for i,x in enumerate(train_dataset):\n",
|
||
" # x[0] is class label, x[1] is text\n",
|
||
" if len(x[1])-nchars<10:\n",
|
||
" continue\n",
|
||
" samples_to_train-=1\n",
|
||
" if not samples_to_train: break\n",
|
||
" text_in, text_out = get_batch(x[1])\n",
|
||
" optimizer.zero_grad()\n",
|
||
" out,s = net(text_in)\n",
|
||
" loss = torch.nn.functional.cross_entropy(out.view(-1,vocab_size),text_out.flatten()) #cross_entropy(out,labels)\n",
|
||
" loss.backward()\n",
|
||
" optimizer.step()\n",
|
||
" if i%1000==0:\n",
|
||
" print(f\"Current loss = {loss.item()}\")\n",
|
||
" print(generate(net))"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"這個範例已經能生成相當不錯的文本，但仍有幾個方面可以進一步改進：\n",
|
||
"\n",
|
||
"* **更好的小批次生成**。我們在準備訓練數據時，是從一個樣本中生成一個小批次。這種方式並不理想，因為小批次的大小各不相同，有些甚至無法生成，因為文本小於 `nchars`。此外，過小的小批次無法充分利用 GPU 的性能。更明智的做法是從所有樣本中提取一大段文本，然後生成所有的輸入-輸出對，將它們打亂，並生成大小相等的小批次。\n",
|
||
"\n",
|
||
"* **多層 LSTM**。嘗試使用 2 或 3 層的 LSTM 單元是有意義的。如我們在前一單元提到的，每一層 LSTM 都會從文本中提取某些模式。在字符級生成器的情況下，我們可以預期較低層的 LSTM 負責提取音節，而較高層則負責提取單詞及單詞組合。這可以通過向 LSTM 構造函數傳遞層數參數來簡單實現。\n",
|
||
"\n",
|
||
"* 你也可以嘗試使用 **GRU 單元**，看看哪種表現更好，還可以嘗試 **不同的隱藏層大小**。隱藏層過大可能導致過擬合（例如，網絡會學習到精確的文本），而過小則可能無法產生良好的結果。\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## 軟性文本生成與溫度\n",
|
||
"\n",
|
||
"在之前 `generate` 的定義中，我們總是選擇機率最高的字元作為生成文本中的下一個字元。這導致生成的文本經常在相同的字元序列之間不斷循環，例如以下例子：\n",
|
||
"```\n",
|
||
"today of the second the company and a second the company ...\n",
|
||
"```\n",
|
||
"\n",
|
||
"然而，如果我們觀察下一個字元的機率分佈，可能會發現幾個最高機率之間的差異並不大，例如一個字元的機率是 0.2，另一個是 0.19，等等。例如，在尋找序列 *play* 的下一個字元時，下一個字元可能同樣有可能是空格，或者是 **e**（如單詞 *player* 中的情況）。\n",
|
||
"\n",
|
||
"這讓我們得出一個結論：選擇機率最高的字元並不總是「公平」的，因為選擇第二高的字元也可能生成有意義的文本。更明智的做法是從網絡輸出的機率分佈中**抽樣**字元。\n",
|
||
"\n",
|
||
"這種抽樣可以通過 `multinomial` 函數來實現，該函數實現了所謂的**多項分佈**。下面定義了一個實現這種**軟性**文本生成的函數：\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 10,
|
||
"metadata": {
|
||
"scrolled": true
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"--- Temperature = 0.3\n",
|
||
"Today and a company and complete an all the land the restrational the as a security and has provers the pay to and a report and the computer in the stand has filities and working the law the stations for a company and with the company and the final the first company and refight of the state and and workin\n",
|
||
"\n",
|
||
"--- Temperature = 0.8\n",
|
||
"Today he oniis its first to Aus bomblaties the marmation a to manan boogot that pirate assaid a relaid their that goverfin the the Cappets Ecrotional Assonia Cition targets it annight the w scyments Blamity #39;s TVeer Diercheg Reserals fran envyuil that of ster said access what succers of Dour-provelith\n",
|
||
"\n",
|
||
"--- Temperature = 1.0\n",
|
||
"Today holy they a 11 will meda a toket subsuaties, engins for Chanos, they's has stainger past to opening orital his thempting new Nattona was al innerforder advan-than #36;s night year his religuled talitatian what the but with Wednesday to Justment will wemen of Mark CCC Camp as Timed Nae wome a leaders\n",
|
||
"\n",
|
||
"--- Temperature = 1.3\n",
|
||
"Today gpone 2.5 fech atcusion poor cocles toparsdorM.cht Line Pamage put 43 his calt lowed to the book, that has authh-the silia rruch ailing to'ory andhes beutirsimi- Aefffive heading offil an auf eacklets is charged evis, Gunymy oy) Mony has it after-sloythyor loveId out filme, the Natabl -Najuntaxiggs \n",
|
||
"\n",
|
||
"--- Temperature = 1.8\n",
|
||
"Today plary, P.slan chly\\401 mardregationly #39;t 8.1Mide) closes ,filtcon alfly playin roven!\\grea.-QFBEP: Iss onfarchQ/itilia CCf Zivesigntwasta orce.-Peul-aw.uicrin of fuglinfsut aftaningwo, MIEX awayew Aice Woiduar Corvagiugge oppo esig ThusBratourid canthly-RyI.co lagitems\\eexciaishes.conBabntusmor I\n",
|
||
"\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"def generate_soft(net,size=100,start='today ',temperature=1.0):\n",
|
||
" chars = list(start)\n",
|
||
" out, s = net(enc(chars).view(1,-1).to(device))\n",
|
||
" for i in range(size):\n",
|
||
" #nc = torch.argmax(out[0][-1])\n",
|
||
" out_dist = out[0][-1].div(temperature).exp()\n",
|
||
" nc = torch.multinomial(out_dist,1)[0]\n",
|
||
" chars.append(vocab.get_itos()[nc])\n",
|
||
" out, s = net(nc.view(1,-1),s)\n",
|
||
" return ''.join(chars)\n",
|
||
" \n",
|
||
"for i in [0.3,0.8,1.0,1.3,1.8]:\n",
|
||
" print(f\"--- Temperature = {i}\\n{generate_soft(net,size=300,start='Today ',temperature=i)}\\n\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"我們引入了一個名為 **temperature** 的參數，用於指示我們應該多大程度地堅持最高概率。如果 temperature 為 1.0，我們進行公平的多項式抽樣；當 temperature 趨於無窮大時，所有概率變得相等，我們隨機選擇下一個字符。在下面的例子中，我們可以觀察到，當我們將 temperature 增加得過高時，文本變得毫無意義；而當 temperature 趨近於 0 時，文本則類似於「循環」的硬生成文本。\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"\n---\n\n**免責聲明**： \n本文件已使用 AI 翻譯服務 [Co-op Translator](https://github.com/Azure/co-op-translator) 進行翻譯。雖然我們致力於提供準確的翻譯，但請注意，自動翻譯可能包含錯誤或不準確之處。原始文件的母語版本應被視為權威來源。對於關鍵信息，建議尋求專業人工翻譯。我們對因使用此翻譯而引起的任何誤解或錯誤解釋不承擔責任。\n"
|
||
]
|
||
}
|
||
],
|
||
"metadata": {
|
||
"interpreter": {
|
||
"hash": "16af2a8bbb083ea23e5e41c7f5787656b2ce26968575d8763f2c4b17f9cd711f"
|
||
},
|
||
"kernelspec": {
|
||
"display_name": "Python 3.8.12 ('py38')",
|
||
"language": "python",
|
||
"name": "python3"
|
||
},
|
||
"language_info": {
|
||
"codemirror_mode": {
|
||
"name": "ipython",
|
||
"version": 3
|
||
},
|
||
"file_extension": ".py",
|
||
"mimetype": "text/x-python",
|
||
"name": "python",
|
||
"nbconvert_exporter": "python",
|
||
"pygments_lexer": "ipython3",
|
||
"version": "3.8.12"
|
||
},
|
||
"coopTranslator": {
|
||
"original_hash": "7673cd150d96c74c6d6011460094efb4",
|
||
"translation_date": "2025-08-28T12:03:32+00:00",
|
||
"source_file": "lessons/5-NLP/17-GenerativeNetworks/GenerativePyTorch.ipynb",
|
||
"language_code": "mo"
|
||
}
|
||
},
|
||
"nbformat": 4,
|
||
"nbformat_minor": 4
|
||
} |