479 lines
22 KiB
Plaintext
479 lines
22 KiB
Plaintext
{
|
||
"cells": [
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"# 循環神經網絡\n",
|
||
"\n",
|
||
"在上一個模組中，我們使用了豐富的語義表示來處理文本，並在嵌入層之上使用了一個簡單的線性分類器。這種架構能夠捕捉句子中詞語的聚合意義，但它並未考慮詞語的**順序**，因為嵌入層上的聚合操作已經將原始文本中的這種信息移除了。由於這些模型無法建模詞語的順序，因此它們無法解決更複雜或更具歧義的任務，例如文本生成或問答系統。\n",
|
||
"\n",
|
||
"為了捕捉文本序列的意義，我們需要使用另一種神經網絡架構，稱為**循環神經網絡**（Recurrent Neural Network，簡稱 RNN）。在 RNN 中，我們將句子逐個符號地傳遞給網絡，網絡會生成某種**狀態**，然後我們將該狀態與下一個符號一起再次傳遞給網絡。\n",
|
||
"\n",
|
||
"給定輸入的符號序列 $X_0,\\dots,X_n$，RNN 會創建一個神經網絡塊的序列，並通過反向傳播對這個序列進行端到端的訓練。每個網絡塊接收一對 $(X_i,S_i)$ 作為輸入，並生成 $S_{i+1}$ 作為結果。最終的狀態 $S_n$ 或輸出 $X_n$ 會進入線性分類器以生成結果。所有的網絡塊共享相同的權重，並通過一次反向傳播進行端到端的訓練。\n",
|
||
"\n",
|
||
"由於狀態向量 $S_0,\\dots,S_n$ 會在網絡中傳遞，因此它能夠學習詞語之間的順序依賴關係。例如，當單詞 *not* 出現在序列中的某處時，網絡可以學會在狀態向量中否定某些元素，從而實現否定的效果。\n",
|
||
"\n",
|
||
"> 由於圖片中所有 RNN 塊的權重是共享的，因此同一張圖片可以表示為一個帶有循環反饋迴路的單一塊（右側），該迴路將網絡的輸出狀態傳回到輸入。\n",
|
||
"\n",
|
||
"現在讓我們看看循環神經網絡如何幫助我們對新聞數據集進行分類。\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 1,
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Loading dataset...\n",
|
||
"Building vocab...\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"import torch\n",
|
||
"import torchtext\n",
|
||
"from torchnlp import *\n",
|
||
"train_dataset, test_dataset, classes, vocab = load_dataset()\n",
|
||
"vocab_size = len(vocab)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## 簡單 RNN 分類器\n",
|
||
"\n",
|
||
"在簡單 RNN 的情況下，每個循環單元是一個簡單的線性網絡，它接收連接的輸入向量和狀態向量，並生成一個新的狀態向量。PyTorch 使用 `RNNCell` 類來表示這個單元，而由這些單元組成的網絡則表示為 `RNN` 層。\n",
|
||
"\n",
|
||
"為了定義一個 RNN 分類器，我們首先會應用一個嵌入層來降低輸入詞彙的維度，然後在其上添加一個 RNN 層：\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 2,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"class RNNClassifier(torch.nn.Module):\n",
|
||
" def __init__(self, vocab_size, embed_dim, hidden_dim, num_class):\n",
|
||
" super().__init__()\n",
|
||
" self.hidden_dim = hidden_dim\n",
|
||
" self.embedding = torch.nn.Embedding(vocab_size, embed_dim)\n",
|
||
" self.rnn = torch.nn.RNN(embed_dim,hidden_dim,batch_first=True)\n",
|
||
" self.fc = torch.nn.Linear(hidden_dim, num_class)\n",
|
||
"\n",
|
||
" def forward(self, x):\n",
|
||
" batch_size = x.size(0)\n",
|
||
" x = self.embedding(x)\n",
|
||
" x,h = self.rnn(x)\n",
|
||
" return self.fc(x.mean(dim=1))"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"> **注意：** 為了簡化，我們在這裡使用未經訓練的嵌入層，但如果想要更好的效果，可以使用預訓練的嵌入層，例如 Word2Vec 或 GloVe 嵌入，這在之前的單元中已經描述過。為了更好地理解，您可能需要調整此程式碼以使用預訓練的嵌入。\n",
|
||
"\n",
|
||
"在我們的情況下，我們將使用填充的數據加載器，因此每個批次都會包含一些相同長度的填充序列。RNN 層將接收嵌入張量的序列，並產生兩個輸出：\n",
|
||
"* $x$ 是每一步中 RNN 單元輸出的序列\n",
|
||
"* $h$ 是序列最後一個元素的最終隱藏狀態\n",
|
||
"\n",
|
||
"接著，我們應用一個全連接的線性分類器來獲得類別數。\n",
|
||
"\n",
|
||
"> **注意：** RNN 的訓練相當困難，因為一旦 RNN 單元沿著序列長度展開，反向傳播所涉及的層數會非常多。因此，我們需要選擇較小的學習率，並在更大的數據集上訓練網絡以獲得良好的結果。這可能需要相當長的時間，因此建議使用 GPU。\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 3,
|
||
"metadata": {
|
||
"scrolled": true
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"3200: acc=0.3090625\n",
|
||
"6400: acc=0.38921875\n",
|
||
"9600: acc=0.4590625\n",
|
||
"12800: acc=0.511953125\n",
|
||
"16000: acc=0.5506875\n",
|
||
"19200: acc=0.57921875\n",
|
||
"22400: acc=0.6070089285714285\n",
|
||
"25600: acc=0.6304296875\n",
|
||
"28800: acc=0.6484027777777778\n",
|
||
"32000: acc=0.66509375\n",
|
||
"35200: acc=0.6790056818181818\n",
|
||
"38400: acc=0.6929166666666666\n",
|
||
"41600: acc=0.7035817307692308\n",
|
||
"44800: acc=0.7137276785714286\n",
|
||
"48000: acc=0.72225\n",
|
||
"51200: acc=0.73001953125\n",
|
||
"54400: acc=0.7372794117647059\n",
|
||
"57600: acc=0.7436631944444444\n",
|
||
"60800: acc=0.7503947368421052\n",
|
||
"64000: acc=0.75634375\n",
|
||
"67200: acc=0.7615773809523809\n",
|
||
"70400: acc=0.7662642045454545\n",
|
||
"73600: acc=0.7708423913043478\n",
|
||
"76800: acc=0.7751822916666666\n",
|
||
"80000: acc=0.7790625\n",
|
||
"83200: acc=0.7825\n",
|
||
"86400: acc=0.7858564814814815\n",
|
||
"89600: acc=0.7890513392857142\n",
|
||
"92800: acc=0.7920474137931034\n",
|
||
"96000: acc=0.7952708333333334\n",
|
||
"99200: acc=0.7982258064516129\n",
|
||
"102400: acc=0.80099609375\n",
|
||
"105600: acc=0.8037594696969697\n",
|
||
"108800: acc=0.8060569852941176\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=16, collate_fn=padify, shuffle=True)\n",
|
||
"net = RNNClassifier(vocab_size,64,32,len(classes)).to(device)\n",
|
||
"train_epoch(net,train_loader, lr=0.001)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## 長短期記憶網絡 (LSTM)\n",
|
||
"\n",
|
||
"傳統 RNN 的主要問題之一是所謂的 **梯度消失** 問題。由於 RNN 是通過一次反向傳播從頭到尾進行訓練的，因此在將誤差傳遞到網絡的第一層時會遇到困難，導致網絡無法學習遠距離的詞元之間的關係。為了解決這個問題，可以通過使用所謂的 **門控機制** 引入 **顯式狀態管理**。其中最知名的兩種架構是 **長短期記憶網絡** (LSTM) 和 **門控循環單元** (GRU)。\n",
|
||
"\n",
|
||
"\n",
|
||
"\n",
|
||
"LSTM 網絡的結構與 RNN 類似，但有兩個狀態會從一層傳遞到下一層：實際狀態 $c$ 和隱藏向量 $h$。在每個單元中，隱藏向量 $h_i$ 與輸入 $x_i$ 拼接在一起，並通過 **門控機制** 控制狀態 $c$ 的變化。每個門控機制都是一個帶有 sigmoid 激活函數（輸出範圍為 $[0,1]$）的神經網絡，可以將其視為在與狀態向量相乘時的位掩碼。以下是這些門控機制（如上圖從左到右所示）：\n",
|
||
"* **遺忘門** 接收隱藏向量並決定向量 $c$ 的哪些部分需要遺忘，哪些需要保留。\n",
|
||
"* **輸入門** 從輸入和隱藏向量中提取一些信息，並將其插入到狀態中。\n",
|
||
"* **輸出門** 通過帶有 $\\tanh$ 激活函數的線性層轉換狀態，然後使用隱藏向量 $h_i$ 選擇其部分組件以生成新的狀態 $c_{i+1}$。\n",
|
||
"\n",
|
||
"狀態 $c$ 的組件可以被視為一些可以開啟或關閉的標誌。例如，當我們在序列中遇到名字 *Alice* 時，我們可能會假設它指的是女性角色，並在狀態中設置一個標誌，表示句子中有一個女性名詞。當我們進一步遇到短語 *and Tom* 時，我們會設置一個標誌，表示句子中有複數名詞。因此，通過操控狀態，我們可以追蹤句子部分的語法屬性。\n",
|
||
"\n",
|
||
"> **Note**: 理解 LSTM 內部結構的一個很棒的資源是 Christopher Olah 的這篇文章 [Understanding LSTM Networks](https://colah.github.io/posts/2015-08-Understanding-LSTMs/)。\n",
|
||
"\n",
|
||
"雖然 LSTM 單元的內部結構看起來很複雜，但 PyTorch 將這些實現隱藏在 `LSTMCell` 類中，並提供了 `LSTM` 對象來表示整個 LSTM 層。因此，LSTM 分類器的實現將與我們之前看到的簡單 RNN 非常相似：\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 4,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"class LSTMClassifier(torch.nn.Module):\n",
|
||
" def __init__(self, vocab_size, embed_dim, hidden_dim, num_class):\n",
|
||
" super().__init__()\n",
|
||
" self.hidden_dim = hidden_dim\n",
|
||
" self.embedding = torch.nn.Embedding(vocab_size, embed_dim)\n",
|
||
" self.embedding.weight.data = torch.randn_like(self.embedding.weight.data)-0.5\n",
|
||
" self.rnn = torch.nn.LSTM(embed_dim,hidden_dim,batch_first=True)\n",
|
||
" self.fc = torch.nn.Linear(hidden_dim, num_class)\n",
|
||
"\n",
|
||
" def forward(self, x):\n",
|
||
" batch_size = x.size(0)\n",
|
||
" x = self.embedding(x)\n",
|
||
" x,(h,c) = self.rnn(x)\n",
|
||
" return self.fc(h[-1])"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": []
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 5,
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"3200: acc=0.259375\n",
|
||
"6400: acc=0.25859375\n",
|
||
"9600: acc=0.26177083333333334\n",
|
||
"12800: acc=0.2784375\n",
|
||
"16000: acc=0.313\n",
|
||
"19200: acc=0.3528645833333333\n",
|
||
"22400: acc=0.3965625\n",
|
||
"25600: acc=0.4385546875\n",
|
||
"28800: acc=0.4752777777777778\n",
|
||
"32000: acc=0.505375\n",
|
||
"35200: acc=0.5326704545454546\n",
|
||
"38400: acc=0.5557552083333334\n",
|
||
"41600: acc=0.5760817307692307\n",
|
||
"44800: acc=0.5954910714285714\n",
|
||
"48000: acc=0.6118333333333333\n",
|
||
"51200: acc=0.62681640625\n",
|
||
"54400: acc=0.6404779411764706\n",
|
||
"57600: acc=0.6520138888888889\n",
|
||
"60800: acc=0.662828947368421\n",
|
||
"64000: acc=0.673546875\n",
|
||
"67200: acc=0.6831547619047619\n",
|
||
"70400: acc=0.6917897727272727\n",
|
||
"73600: acc=0.6997146739130434\n",
|
||
"76800: acc=0.707109375\n",
|
||
"80000: acc=0.714075\n",
|
||
"83200: acc=0.7209134615384616\n",
|
||
"86400: acc=0.727037037037037\n",
|
||
"89600: acc=0.7326674107142858\n",
|
||
"92800: acc=0.7379633620689655\n",
|
||
"96000: acc=0.7433645833333333\n",
|
||
"99200: acc=0.7479032258064516\n",
|
||
"102400: acc=0.752119140625\n",
|
||
"105600: acc=0.7562405303030303\n",
|
||
"108800: acc=0.76015625\n",
|
||
"112000: acc=0.7641339285714286\n",
|
||
"115200: acc=0.7677777777777778\n",
|
||
"118400: acc=0.7711233108108108\n"
|
||
]
|
||
},
|
||
{
|
||
"data": {
|
||
"text/plain": [
|
||
"(0.03487814127604167, 0.7728)"
|
||
]
|
||
},
|
||
"execution_count": 5,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"net = LSTMClassifier(vocab_size,64,32,len(classes)).to(device)\n",
|
||
"train_epoch(net,train_loader, lr=0.001)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## 打包序列\n",
|
||
"\n",
|
||
"在我們的例子中，我們需要用零向量填充小批量中的所有序列。雖然這會導致一些記憶體浪費，但對於 RNN 而言，更關鍵的是為填充的輸入項目創建了額外的 RNN 單元，這些單元參與了訓練，但並未攜帶任何重要的輸入資訊。如果能僅針對實際的序列長度來訓練 RNN，效果會更好。\n",
|
||
"\n",
|
||
"為此，PyTorch 引入了一種特殊格式來存儲填充的序列。假設我們有一個填充過的小批量輸入，看起來像這樣：\n",
|
||
"```\n",
|
||
"[[1,2,3,4,5],\n",
|
||
" [6,7,8,0,0],\n",
|
||
" [9,0,0,0,0]]\n",
|
||
"```\n",
|
||
"這裡的 0 代表填充的值，而輸入序列的實際長度向量是 `[5,3,1]`。\n",
|
||
"\n",
|
||
"為了有效地用填充序列訓練 RNN，我們希望先用較大的小批量（`[1,6,9]`）開始訓練第一組 RNN 單元，然後結束第三個序列的處理，並繼續用較小的小批量（`[2,7]`，`[3,8]`）進行訓練，依此類推。因此，打包序列被表示為一個向量——在我們的例子中是 `[1,6,9,2,7,3,8,4,5]`，以及長度向量（`[5,3,1]`），我們可以根據這些輕鬆重建原始的填充小批量。\n",
|
||
"\n",
|
||
"要生成打包序列，我們可以使用 `torch.nn.utils.rnn.pack_padded_sequence` 函數。所有的循環層，包括 RNN、LSTM 和 GRU，都支持將打包序列作為輸入，並生成打包輸出，這些輸出可以通過 `torch.nn.utils.rnn.pad_packed_sequence` 解碼。\n",
|
||
"\n",
|
||
"為了能夠生成打包序列，我們需要將長度向量傳遞給網絡，因此我們需要一個不同的函數來準備小批量：\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 6,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"def pad_length(b):\n",
|
||
" # build vectorized sequence\n",
|
||
" v = [encode(x[1]) for x in b]\n",
|
||
" # compute max length of a sequence in this minibatch and length sequence itself\n",
|
||
" len_seq = list(map(len,v))\n",
|
||
" l = max(len_seq)\n",
|
||
" return ( # tuple of three tensors - labels, padded features, length sequence\n",
|
||
" torch.LongTensor([t[0]-1 for t in b]),\n",
|
||
" torch.stack([torch.nn.functional.pad(torch.tensor(t),(0,l-len(t)),mode='constant',value=0) for t in v]),\n",
|
||
" torch.tensor(len_seq)\n",
|
||
" )\n",
|
||
"\n",
|
||
"train_loader_len = torch.utils.data.DataLoader(train_dataset, batch_size=16, collate_fn=pad_length, shuffle=True)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"實際的網絡會與上面的 `LSTMClassifier` 非常相似，但在 `forward` 傳遞時，會同時接收填充過的小批量數據以及序列長度的向量。在計算嵌入後，我們會計算打包序列，將其傳遞給 LSTM 層，然後再將結果解包回來。\n",
|
||
"\n",
|
||
"> **注意**：我們實際上並沒有使用解包後的結果 `x`，因為在後續的計算中，我們使用的是隱藏層的輸出。因此，我們可以完全移除這段代碼中的解包部分。我們之所以將其放在這裡，是為了方便你在需要使用網絡輸出進行進一步計算時，能夠輕鬆修改這段代碼。\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 7,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"class LSTMPackClassifier(torch.nn.Module):\n",
|
||
" def __init__(self, vocab_size, embed_dim, hidden_dim, num_class):\n",
|
||
" super().__init__()\n",
|
||
" self.hidden_dim = hidden_dim\n",
|
||
" self.embedding = torch.nn.Embedding(vocab_size, embed_dim)\n",
|
||
" self.embedding.weight.data = torch.randn_like(self.embedding.weight.data)-0.5\n",
|
||
" self.rnn = torch.nn.LSTM(embed_dim,hidden_dim,batch_first=True)\n",
|
||
" self.fc = torch.nn.Linear(hidden_dim, num_class)\n",
|
||
"\n",
|
||
" def forward(self, x, lengths):\n",
|
||
" batch_size = x.size(0)\n",
|
||
" x = self.embedding(x)\n",
|
||
" pad_x = torch.nn.utils.rnn.pack_padded_sequence(x,lengths,batch_first=True,enforce_sorted=False)\n",
|
||
" pad_x,(h,c) = self.rnn(pad_x)\n",
|
||
" x, _ = torch.nn.utils.rnn.pad_packed_sequence(pad_x,batch_first=True)\n",
|
||
" return self.fc(h[-1])"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": []
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 8,
|
||
"metadata": {
|
||
"scrolled": true
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"3200: acc=0.285625\n",
|
||
"6400: acc=0.33359375\n",
|
||
"9600: acc=0.3876041666666667\n",
|
||
"12800: acc=0.44078125\n",
|
||
"16000: acc=0.4825\n",
|
||
"19200: acc=0.5235416666666667\n",
|
||
"22400: acc=0.5559821428571429\n",
|
||
"25600: acc=0.58609375\n",
|
||
"28800: acc=0.6116666666666667\n",
|
||
"32000: acc=0.63340625\n",
|
||
"35200: acc=0.6525284090909091\n",
|
||
"38400: acc=0.668515625\n",
|
||
"41600: acc=0.6822596153846154\n",
|
||
"44800: acc=0.6948214285714286\n",
|
||
"48000: acc=0.7052708333333333\n",
|
||
"51200: acc=0.71521484375\n",
|
||
"54400: acc=0.7239889705882353\n",
|
||
"57600: acc=0.7315277777777778\n",
|
||
"60800: acc=0.7388486842105263\n",
|
||
"64000: acc=0.74571875\n",
|
||
"67200: acc=0.7518303571428572\n",
|
||
"70400: acc=0.7576988636363636\n",
|
||
"73600: acc=0.7628940217391305\n",
|
||
"76800: acc=0.7681510416666667\n",
|
||
"80000: acc=0.7728125\n",
|
||
"83200: acc=0.7772235576923077\n",
|
||
"86400: acc=0.7815393518518519\n",
|
||
"89600: acc=0.7857700892857142\n",
|
||
"92800: acc=0.7895043103448276\n",
|
||
"96000: acc=0.7930520833333333\n",
|
||
"99200: acc=0.7959072580645161\n",
|
||
"102400: acc=0.798994140625\n",
|
||
"105600: acc=0.802064393939394\n",
|
||
"108800: acc=0.8051378676470589\n",
|
||
"112000: acc=0.8077857142857143\n",
|
||
"115200: acc=0.8104600694444445\n",
|
||
"118400: acc=0.8128293918918919\n"
|
||
]
|
||
},
|
||
{
|
||
"data": {
|
||
"text/plain": [
|
||
"(0.029785829671223958, 0.8138166666666666)"
|
||
]
|
||
},
|
||
"execution_count": 8,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"net = LSTMPackClassifier(vocab_size,64,32,len(classes)).to(device)\n",
|
||
"train_epoch_emb(net,train_loader_len, lr=0.001,use_pack_sequence=True)\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"> **注意：** 您可能已經注意到我們傳遞給訓練函數的參數 `use_pack_sequence`。目前，`pack_padded_sequence` 函數要求長度序列張量位於 CPU 設備上，因此訓練函數需要避免在訓練時將長度序列數據移動到 GPU。您可以查看 [`torchnlp.py`](../../../../../lessons/5-NLP/16-RNN/torchnlp.py) 文件中 `train_emb` 函數的實現。\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## 雙向和多層 RNN\n",
|
||
"\n",
|
||
"在我們的例子中，所有的循環神經網絡都是單向運行的，從序列的開頭到結尾。這看起來很自然，因為它類似於我們閱讀和聆聽語音的方式。然而，在許多實際情況下，我們可以隨機訪問輸入序列，因此在兩個方向上運行循環計算可能更有意義。這樣的網絡被稱為 **雙向** RNN，可以通過在 RNN/LSTM/GRU 構造函數中傳遞參數 `bidirectional=True` 來創建。\n",
|
||
"\n",
|
||
"處理雙向網絡時，我們需要兩個隱藏狀態向量，每個方向一個。PyTorch 將這些向量編碼為一個大小是原來兩倍的向量，這非常方便，因為通常我們會將生成的隱藏狀態傳遞給全連接線性層，只需在創建該層時考慮到這個大小的增加即可。\n",
|
||
"\n",
|
||
"無論是單向還是雙向的循環網絡，都能在序列中捕捉某些模式，並將其存儲到狀態向量中或傳遞到輸出中。與卷積網絡類似，我們可以在第一層之上構建另一個循環層，以捕捉更高層次的模式，這些模式是由第一層提取的低層次模式構建而成的。這引出了 **多層 RNN** 的概念，它由兩層或更多的循環網絡組成，其中前一層的輸出作為下一層的輸入。\n",
|
||
"\n",
|
||
"\n",
|
||
"\n",
|
||
"*圖片來源於 [這篇精彩的文章](https://towardsdatascience.com/from-a-lstm-cell-to-a-multilayer-lstm-network-with-pytorch-2899eb5696f3) 作者 Fernando López*\n",
|
||
"\n",
|
||
"PyTorch 讓構建這樣的網絡變得非常簡單，因為只需在 RNN/LSTM/GRU 構造函數中傳遞參數 `num_layers`，即可自動構建多層循環網絡。這也意味著隱藏/狀態向量的大小會按比例增加，處理循環層的輸出時需要考慮到這一點。\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## RNNs 用於其他任務\n",
|
||
"\n",
|
||
"在本單元中，我們已經看到 RNNs 可以用於序列分類，但事實上，它們還能處理更多任務，例如文本生成、機器翻譯等等。我們會在下一單元探討這些任務。\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"\n---\n\n**免責聲明**： \n本文件已使用人工智能翻譯服務 [Co-op Translator](https://github.com/Azure/co-op-translator) 進行翻譯。儘管我們致力於提供準確的翻譯，但請注意，自動翻譯可能包含錯誤或不準確之處。原始語言的文件應被視為權威來源。對於重要資訊，建議使用專業人工翻譯。我們對因使用此翻譯而引起的任何誤解或錯誤解釋概不負責。\n"
|
||
]
|
||
}
|
||
],
|
||
"metadata": {
|
||
"interpreter": {
|
||
"hash": "16af2a8bbb083ea23e5e41c7f5787656b2ce26968575d8763f2c4b17f9cd711f"
|
||
},
|
||
"kernelspec": {
|
||
"display_name": "Python 3.8.12 ('py38')",
|
||
"language": "python",
|
||
"name": "python3"
|
||
},
|
||
"language_info": {
|
||
"codemirror_mode": {
|
||
"name": "ipython",
|
||
"version": 3
|
||
},
|
||
"file_extension": ".py",
|
||
"mimetype": "text/x-python",
|
||
"name": "python",
|
||
"nbconvert_exporter": "python",
|
||
"pygments_lexer": "ipython3",
|
||
"version": "3.8.12"
|
||
},
|
||
"coopTranslator": {
|
||
"original_hash": "522ee52ae3d5ae933e283286254e9a55",
|
||
"translation_date": "2025-08-31T10:50:36+00:00",
|
||
"source_file": "lessons/5-NLP/16-RNN/RNNPyTorch.ipynb",
|
||
"language_code": "hk"
|
||
}
|
||
},
|
||
"nbformat": 4,
|
||
"nbformat_minor": 2
|
||
} |