Merge pull request #126 from shwars/main
Add contributions from students of MAI/HSE
This commit is contained in:
carlotta94c
GitHub
commit
32a0d200e2
|
|
@ -81,7 +81,7 @@ For a gentle introduction to *AI in the Cloud* topics you may consider taking th
|
|||
<td></td></tr>
|
||||
<tr><td>13</td><td>Text Representation. Bow/TF-IDF</td><td><a href="lessons/5-NLP/13-TextRep/README.md">Text</a></td><td><a href="lessons/5-NLP/13-TextRep/TextRepresentationPyTorch.ipynb">PyTorch</a></td><td><a href="lessons/5-NLP/13-TextRep/TextRepresentationTF.ipynb">TensorFlow</td><td></td></tr>
|
||||
<tr><td>14</td><td>Semantic word embeddings. Word2Vec and GloVe</td><td><a href="lessons/5-NLP/14-Embeddings/README.md">Text</td><td><a href="lessons/5-NLP/14-Embeddings/EmbeddingsPyTorch.ipynb">PyTorch</a></td><td><a href="lessons/5-NLP/14-Embeddings/EmbeddingsTF.ipynb">TensorFlow</a></td><td></td></tr>
|
||||
<tr><td>15</td><td>Language Modeling. Training your own embeddings</td><td><a href="lessons/5-NLP/15-LanguageModeling/README.md">Text</a></td><td></td><td><a href="lessons/5-NLP/15-LanguageModeling/CBoW-TF.ipynb">TensorFlow</a></td><td><a href="lessons/5-NLP/15-LanguageModeling/lab/README.md">Lab</a></td></tr>
|
||||
<tr><td>15</td><td>Language Modeling. Training your own embeddings</td><td><a href="lessons/5-NLP/15-LanguageModeling/README.md">Text</a></td><td><a href="lessons/5-NLP/15-LanguageModeling/CBoW-PyTorch.ipynb">PyTorch</a></td><td><a href="lessons/5-NLP/15-LanguageModeling/CBoW-TF.ipynb">TensorFlow</a></td><td><a href="lessons/5-NLP/15-LanguageModeling/lab/README.md">Lab</a></td></tr>
|
||||
<tr><td>16</td><td>Recurrent Neural Networks</td><td><a href="lessons/5-NLP/16-RNN/README.md">Text</a></td><td><a href="lessons/5-NLP/16-RNN/RNNPyTorch.ipynb">PyTorch</a></td><td><a href="lessons/5-NLP/16-RNN/RNNTF.ipynb">TensorFlow</a></td><td></td></tr>
|
||||
<tr><td>17</td><td>Generative Recurrent Networks</td><td><a href="lessons/5-NLP/17-GenerativeNetworks/README.md">Text</a></td><td><a href="lessons/5-NLP/17-GenerativeNetworks/GenerativePyTorch.md">PyTorch</a></td><td><a href="lessons/5-NLP/17-GenerativeNetworks/GenerativeTF.md">TensorFlow</a></td><td><a href="lessons/5-NLP/17-GenerativeNetworks/lab/README.md">Lab</a></td></tr>
|
||||
<tr><td>18</td><td>Transformers. BERT.</td><td><a href="lessons/5-NLP/18-Transformers/README.md">Text</a></td><td><a href="lessons/5-NLP/18-Transformers/TransformersPyTorch.ipynb">PyTorch</a></td><td><a href="lessons/5-NLP/18-Transformers/TransformersTF.ipynb">TensorFlow</a></td><td></td></tr>
|
||||
|
|
@ -89,7 +89,7 @@ For a gentle introduction to *AI in the Cloud* topics you may consider taking th
|
|||
<tr><td>20</td><td>Large Language Models, Prompt Programming and Few-Shot Tasks</td><td><a href="lessons/5-NLP/20-LangModels/README.md">Text</a></td><td><a href="lessons/5-NLP/20-LangModels/GPT-PyTorch.ipynb">PyTorch</td><td></td><td></td></tr>
|
||||
<tr><td>VI</td><td colspan="4"><b>Other AI Techniques</b></td><td></td></tr>
|
||||
<tr><td>21</td><td>Genetic Algorithms</td><td><a href="lessons/6-Other/21-GeneticAlgorithms/README.md">Text</a><td colspan="2"><a href="lessons/6-Other/21-GeneticAlgorithms/Genetic.ipynb">Notebook</a></td><td></td></tr>
|
||||
<tr><td>22</td><td>Deep Reinforcement Learning</td><td><a href="lessons/6-Other/22-DeepRL/README.md">Text</a></td><td></td><td><a href="lessons/6-Other/22-DeepRL/CartPole-RL-TF.ipynb">TensorFlow</td><td><a href="lessons/6-Other/22-DeepRL/lab/README.md">Lab</a></td></tr>
|
||||
<tr><td>22</td><td>Deep Reinforcement Learning</td><td><a href="lessons/6-Other/22-DeepRL/README.md">Text</a></td><td><a href="lessons/6-Other/22-DeepRL/CartPole-RL-PyTorch.ipynb">PyTorch</a></td><td><a href="lessons/6-Other/22-DeepRL/CartPole-RL-TF.ipynb">TensorFlow</a></td><td><a href="lessons/6-Other/22-DeepRL/lab/README.md">Lab</a></td></tr>
|
||||
<tr><td>23</td><td>Multi-Agent Systems</td><td><a href="lessons/6-Other/23-MultiagentSystems/README.md">Text</a></td><td></td><td></td><td></td></tr>
|
||||
<tr><td>VII</td><td colspan="4"><b>AI Ethics</b></td><td></td></tr>
|
||||
<tr><td>24</td><td>AI Ethics and Responsible AI</td><td><a href="lessons/7-Ethics/README.md">Text</a></td><td colspan="2"><a href="https://docs.microsoft.com/learn/paths/responsible-ai-business-principles/?WT.mc_id=academic-77998-cacaste"><i>MS Learn: Responsible AI Principles</i></a></td><td></td></tr>
|
||||
|
|
|
|||
File diff suppressed because it is too large
Load Diff
|
|
@ -0,0 +1,563 @@
|
|||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "NXTSugt6ieXh"
|
||||
},
|
||||
"source": [
|
||||
"## Training CBoW Model\n",
|
||||
"\n",
|
||||
"This notebooks is a part of [AI for Beginners Curriculum](http://aka.ms/ai-beginners)\n",
|
||||
"\n",
|
||||
"In this example, we will look at training CBoW language model to get our own Word2Vec embedding space. We will use AG News dataset as the source of text."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"import torch\n",
|
||||
"import torchtext\n",
|
||||
"import os\n",
|
||||
"import collections\n",
|
||||
"import builtins\n",
|
||||
"import random\n",
|
||||
"import numpy as np"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "q-UiiJUKaxHj"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "TFbR8CZaTZ1q"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"First let's load our dataset and define tokenizer and vocabulary. We will set `vocab_size` to 5000 to limit computations a bit."
|
||||
],
|
||||
"metadata": {
|
||||
"id": "HIwC7lI5T-ov"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"def load_dataset(ngrams = 1, min_freq = 1, vocab_size = 5000 , lines_cnt = 500):\n",
|
||||
" tokenizer = torchtext.data.utils.get_tokenizer('basic_english')\n",
|
||||
" print(\"Loading dataset...\")\n",
|
||||
" test_dataset, train_dataset = torchtext.datasets.AG_NEWS(root='./data')\n",
|
||||
" train_dataset = list(train_dataset)\n",
|
||||
" test_dataset = list(test_dataset)\n",
|
||||
" classes = ['World', 'Sports', 'Business', 'Sci/Tech']\n",
|
||||
" print('Building vocab...')\n",
|
||||
" counter = collections.Counter()\n",
|
||||
" for i, (_, line) in enumerate(train_dataset):\n",
|
||||
" counter.update(torchtext.data.utils.ngrams_iterator(tokenizer(line),ngrams=ngrams))\n",
|
||||
" if i == lines_cnt:\n",
|
||||
" break\n",
|
||||
" vocab = torchtext.vocab.Vocab(collections.Counter(dict(counter.most_common(vocab_size))), min_freq=min_freq)\n",
|
||||
" return train_dataset, test_dataset, classes, vocab, tokenizer"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "wdZuygtgiuLG"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"train_dataset, test_dataset, _, vocab, tokenizer = load_dataset()"
|
||||
],
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"base_uri": "https://localhost:8080/"
|
||||
},
|
||||
"id": "4d1nU1gsivGu",
|
||||
"outputId": "949fe272-ae0e-49f5-c373-6703458b3a74"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": [
|
||||
{
|
||||
"output_type": "stream",
|
||||
"name": "stdout",
|
||||
"text": [
|
||||
"Loading dataset...\n",
|
||||
"Building vocab...\n"
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"def encode(x, vocabulary, tokenizer = tokenizer):\n",
|
||||
" return [vocabulary[s] for s in tokenizer(x)]"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "1XDYNhG8ToFV"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "LIlQk6_PaHVY"
|
||||
},
|
||||
"source": [
|
||||
"## CBoW Model\n",
|
||||
"\n",
|
||||
"CBoW learns to predict a word based on the $2N$ neighboring words. For example, when $N=1$, we will get the following pairs from the sentence *I like to train networks*: (like,I), (I, like), (to, like), (like,to), (train,to), (to, train), (networks, train), (train,networks). Here, first word is the neighboring word used as an input, and second word is the one we are predicting.\n",
|
||||
"\n",
|
||||
"To build a network to predict next word, we will need to supply neighboring word as input, and get word number as output. The architecture of CBoW network is the following:\n",
|
||||
"\n",
|
||||
"* Input word is passed through the embedding layer. This very embedding layer would be our Word2Vec embedding, thus we will define it separately as `embedder` variable. We will use embedding size = 30 in this example, even though you might want to experiment with higher dimensions (real word2vec has 300)\n",
|
||||
"* Embedding vector would then be passed to a linear layer that will predict output word. Thus it has the `vocab_size` neurons.\n",
|
||||
"\n",
|
||||
"For the output, if we use `CrossEntropyLoss` as loss function, we would also have to provide just word numbers as expected results, without one-hot encoding."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"vocab_size = len(vocab)\n",
|
||||
"\n",
|
||||
"embedder = torch.nn.Embedding(num_embeddings = vocab_size, embedding_dim = 30)\n",
|
||||
"model = torch.nn.Sequential(\n",
|
||||
" embedder,\n",
|
||||
" torch.nn.Linear(in_features = 30, out_features = vocab_size),\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"print(model)"
|
||||
],
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"base_uri": "https://localhost:8080/"
|
||||
},
|
||||
"id": "akKTcKQKkfl2",
|
||||
"outputId": "da687e3e-a8ec-4c1a-e456-ab8cd6ac7dad"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": [
|
||||
{
|
||||
"output_type": "stream",
|
||||
"name": "stdout",
|
||||
"text": [
|
||||
"Sequential(\n",
|
||||
" (0): Embedding(5002, 30)\n",
|
||||
" (1): Linear(in_features=30, out_features=5002, bias=True)\n",
|
||||
")\n"
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "Nud6jgGPaHVa"
|
||||
},
|
||||
"source": [
|
||||
"## Preparing Training Data\n",
|
||||
"\n",
|
||||
"Now let's program the main function that will compute CBoW word pairs from text. This function will allow us to specify window size, and will return a set of pairs - input and output word. Note that this function can be used on words, as well as on vectors/tensors - which will allow us to encode the text, before passing it to `to_cbow` function."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"base_uri": "https://localhost:8080/"
|
||||
},
|
||||
"id": "x-dsXygOieXn",
|
||||
"outputId": "c2218280-e540-40ba-9546-efe48d0d714f"
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"output_type": "stream",
|
||||
"name": "stdout",
|
||||
"text": [
|
||||
"[['like', 'I'], ['to', 'I'], ['I', 'like'], ['to', 'like'], ['train', 'like'], ['I', 'to'], ['like', 'to'], ['train', 'to'], ['networks', 'to'], ['like', 'train'], ['to', 'train'], ['networks', 'train'], ['to', 'networks'], ['train', 'networks']]\n",
|
||||
"[[232, 172], [5, 172], [172, 232], [5, 232], [0, 232], [172, 5], [232, 5], [0, 5], [1202, 5], [232, 0], [5, 0], [1202, 0], [5, 1202], [0, 1202]]\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"def to_cbow(sent,window_size=2):\n",
|
||||
" res = []\n",
|
||||
" for i,x in enumerate(sent):\n",
|
||||
" for j in range(max(0,i-window_size),min(i+window_size+1,len(sent))):\n",
|
||||
" if i!=j:\n",
|
||||
" res.append([sent[j],x])\n",
|
||||
" return res\n",
|
||||
"\n",
|
||||
"print(to_cbow(['I','like','to','train','networks']))\n",
|
||||
"print(to_cbow(encode('I like to train networks', vocab)))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "XVaaDLjaaHVb"
|
||||
},
|
||||
"source": [
|
||||
"Let's prepare the training dataset. We will go through all news, call `to_cbow` to get the list of word pairs, and add those pairs to `X` and `Y`. For the sake of time, we will only consider first 10k news items - you can easily remove the limitation in case you have more time to wait, and want to get better embeddings :)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "54b-Gd9TieXo"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"X = []\n",
|
||||
"Y = []\n",
|
||||
"for i, x in zip(range(10000), train_dataset):\n",
|
||||
" for w1, w2 in to_cbow(encode(x[1], vocab), window_size = 5):\n",
|
||||
" X.append(w1)\n",
|
||||
" Y.append(w2)\n",
|
||||
"\n",
|
||||
"X = torch.tensor(X)\n",
|
||||
"Y = torch.tensor(Y)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"We will also convert that data to one dataset, and create dataloader:"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "cwWy0PzXWhN5"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"class SimpleIterableDataset(torch.utils.data.IterableDataset):\n",
|
||||
" def __init__(self, X, Y):\n",
|
||||
" super(SimpleIterableDataset).__init__()\n",
|
||||
" self.data = []\n",
|
||||
" for i in range(len(X)):\n",
|
||||
" self.data.append( (Y[i], X[i]) )\n",
|
||||
" random.shuffle(self.data)\n",
|
||||
"\n",
|
||||
" def __iter__(self):\n",
|
||||
" return iter(self.data)"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "mfoAcGPFZU8p"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "e4NQ_-5waHVc"
|
||||
},
|
||||
"source": [
|
||||
"We will also convert that data to one dataset, and create dataloader:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "AbLUcojlieXo"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"ds = SimpleIterableDataset(X, Y)\n",
|
||||
"dl = torch.utils.data.DataLoader(ds, batch_size = 256)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "pKQr7sXeaHVc"
|
||||
},
|
||||
"source": [
|
||||
"Now let's do the actual training. We will use `SGD` optimizer with pretty high learning rate. You can also try playing around with other optimizers, such as `Adam`. We will train for 10 epochs to begin with - and you can re-run this cell if you want even lower loss."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"def train_epoch(net, dataloader, lr = 0.01, optimizer = None, loss_fn = torch.nn.CrossEntropyLoss(), epochs = None, report_freq = 1):\n",
|
||||
" optimizer = optimizer or torch.optim.Adam(net.parameters(), lr = lr)\n",
|
||||
" loss_fn = loss_fn.to(device)\n",
|
||||
" net.train()\n",
|
||||
"\n",
|
||||
" for i in range(epochs):\n",
|
||||
" total_loss, j = 0, 0, \n",
|
||||
" for labels, features in dataloader:\n",
|
||||
" optimizer.zero_grad()\n",
|
||||
" features, labels = features.to(device), labels.to(device)\n",
|
||||
" out = net(features)\n",
|
||||
" loss = loss_fn(out, labels)\n",
|
||||
" loss.backward()\n",
|
||||
" optimizer.step()\n",
|
||||
" total_loss += loss\n",
|
||||
" j += 1\n",
|
||||
" if i % report_freq == 0:\n",
|
||||
" print(f\"Epoch: {i+1}: loss={total_loss.item()/j}\")\n",
|
||||
"\n",
|
||||
" return total_loss.item()/j"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "HeeCYKr_KF1w"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"train_epoch(net = model, dataloader = dl, optimizer = torch.optim.SGD(model.parameters(), lr = 0.1), loss_fn = torch.nn.CrossEntropyLoss(), epochs = 10)"
|
||||
],
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"base_uri": "https://localhost:8080/"
|
||||
},
|
||||
"id": "KVgwGtDHgDlT",
|
||||
"outputId": "2447833f-f0e3-4566-c33d-addbfe2f451d"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": [
|
||||
{
|
||||
"output_type": "stream",
|
||||
"name": "stdout",
|
||||
"text": [
|
||||
"Epoch: 1: loss=5.664632366860172\n",
|
||||
"Epoch: 2: loss=5.632101973960962\n",
|
||||
"Epoch: 3: loss=5.610399051405015\n",
|
||||
"Epoch: 4: loss=5.594621561080262\n",
|
||||
"Epoch: 5: loss=5.582538017415446\n",
|
||||
"Epoch: 6: loss=5.572900234519603\n",
|
||||
"Epoch: 7: loss=5.564951676341915\n",
|
||||
"Epoch: 8: loss=5.558288112064614\n",
|
||||
"Epoch: 9: loss=5.552576955031129\n",
|
||||
"Epoch: 10: loss=5.547634165194347\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"output_type": "execute_result",
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"5.547634165194347"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"execution_count": 16
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "W8u2qXZmaHVd"
|
||||
},
|
||||
"source": [
|
||||
"## Trying out Word2Vec\n",
|
||||
"\n",
|
||||
"To use Word2Vec, let's extract vectors corresponding to all words in our vocabulary:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "r8TatcXjkU_t"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"vectors = torch.stack([embedder(torch.tensor(vocab[s])) for s in vocab.itos], 0)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "3OcX21UOaHVd"
|
||||
},
|
||||
"source": [
|
||||
"Let's see, for example, how the word **Paris** is encoded into a vector:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"base_uri": "https://localhost:8080/"
|
||||
},
|
||||
"id": "bz6tAeLzieXp",
|
||||
"outputId": "5b20850e-4342-45e9-f840-cfac2b4d61d8"
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"output_type": "stream",
|
||||
"name": "stdout",
|
||||
"text": [
|
||||
"tensor([-0.0915, 2.1224, -0.0281, -0.6819, 1.1219, 0.6458, -1.3704, -1.3314,\n",
|
||||
" -1.1437, 0.4496, 0.2301, -0.3515, -0.8485, 1.0481, 0.4386, -0.8949,\n",
|
||||
" 0.5644, 1.0939, -2.5096, 3.2949, -0.2601, -0.8640, 0.1421, -0.0804,\n",
|
||||
" -0.5083, -1.0560, 0.9753, -0.5949, -1.6046, 0.5774],\n",
|
||||
" grad_fn=<EmbeddingBackward>)\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"paris_vec = embedder(torch.tensor(vocab['paris']))\n",
|
||||
"print(paris_vec)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "pHTJlaeYaHVd"
|
||||
},
|
||||
"source": [
|
||||
"It is interesting to use Word2Vec to look for synonyms. The following function will return `n` closest words to a given input. To find them, we compute the norm of $|w_i - v|$, where $v$ is the vector corresponding to our input word, and $w_i$ is the encoding of $i$-th word in the vocabulary. We then sort the array and return corresponding indices using `argsort`, and take first `n` elements of the list, which encode positions of closest words in the vocabulary. "
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"base_uri": "https://localhost:8080/"
|
||||
},
|
||||
"id": "NlZyi-_olFar",
|
||||
"outputId": "b5dbb163-88c4-4d5a-eaf2-6751f700e98c"
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"output_type": "execute_result",
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"['microsoft', 'quoted', 'lp', 'rate', 'top']"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"execution_count": 56
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"def close_words(x, n = 5):\n",
|
||||
" vec = embedder(torch.tensor(vocab[x]))\n",
|
||||
" top5 = np.linalg.norm(vectors.detach().numpy() - vec.detach().numpy(), axis = 1).argsort()[:n]\n",
|
||||
" return [ vocab.itos[x] for x in top5 ]\n",
|
||||
"\n",
|
||||
"close_words('microsoft')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"base_uri": "https://localhost:8080/"
|
||||
},
|
||||
"id": "-dQq7xeAln0U",
|
||||
"outputId": "66f768c3-c248-4bfd-ce4f-c8ffc6d0dd0d"
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"output_type": "execute_result",
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"['basketball', 'lot', 'sinai', 'states', 'healthdaynews']"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"execution_count": 51
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"close_words('basketball')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"base_uri": "https://localhost:8080/"
|
||||
},
|
||||
"id": "fJXqK26b29sa",
|
||||
"outputId": "78f0baba-ffd0-485a-dd87-0a12bedfd7fa"
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"output_type": "execute_result",
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"['funds', 'travel', 'sydney', 'japan', 'business']"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"execution_count": 77
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"close_words('funds')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "My0VeTDd3Ji8"
|
||||
},
|
||||
"source": [
|
||||
"## Takeaway\n",
|
||||
"\n",
|
||||
"Using clever techniques such as CBoW, we can train Word2Vec model. You may also try to train skip-gram model that is trained to predict the neighboring word given the central one, and see how well it performs. "
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"collapsed_sections": [],
|
||||
"name": "CBoW-PyTorch.ipynb",
|
||||
"provenance": []
|
||||
},
|
||||
"interpreter": {
|
||||
"hash": "16af2a8bbb083ea23e5e41c7f5787656b2ce26968575d8763f2c4b17f9cd711f"
|
||||
},
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3.8.12 ('py38')",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.8.12"
|
||||
},
|
||||
"orig_nbformat": 4,
|
||||
"gpuClass": "standard"
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 0
|
||||
}
|
||||
|
|
@ -23,6 +23,8 @@ In our previous examples, we used pre-trained semantic embeddings, but it is int
|
|||
Continue your learning in the following notebooks:
|
||||
|
||||
* [Training CBoW Word2Vec with TensorFlow](CBoW-TF.ipynb)
|
||||
* [Training CBoW Word2Vec with PyTorch](CBoW-PyTorch.ipynb)
|
||||
|
||||
|
||||
## Conclusion
|
||||
|
||||
|
|
|
|||
|
|
@ -0,0 +1,490 @@
|
|||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Training RL to do Cartpole Balancing\n",
|
||||
"\n",
|
||||
"This notebooks is part of [AI for Beginners Curriculum](http://aka.ms/ai-beginners). It has been inspired by [official PyTorch tutorial](https://pytorch.org/tutorials/intermediate/reinforcement_q_learning.html) and [this Cartpole Pytorch implementation](https://github.com/yc930401/Actor-Critic-pytorch).\n",
|
||||
"\n",
|
||||
"In this example, we will use RL to train a model to balance a pole on a cart that can move left and right on horizontal scale. We will use [OpenAI Gym](https://www.gymlibrary.ml/) environment to simulate the pole.\n",
|
||||
"\n",
|
||||
"> **Note**: You can run this lesson's code locally (eg. from Visual Studio Code), in which case the simulation will open in a new window. When running the code online, you may need to make some tweaks to the code, as described [here](https://towardsdatascience.com/rendering-openai-gym-envs-on-binder-and-google-colab-536f99391cc7).\n",
|
||||
"\n",
|
||||
"We will start by making sure Gym is installed:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import sys\n",
|
||||
"!{sys.executable} -m pip install gym"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Now let's create the CartPole environment and see how to operate on it. An environment has the following properties:\n",
|
||||
"\n",
|
||||
"* **Action space** is the set of possible actions that we can perform at each step of the simulation\n",
|
||||
"* **Observation space** is the space of observations that we can make"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import gym\n",
|
||||
"\n",
|
||||
"env = gym.make(\"CartPole-v1\")\n",
|
||||
"\n",
|
||||
"print(f\"Action space: {env.action_space}\")\n",
|
||||
"print(f\"Observation space: {env.observation_space}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Let's see how the simulation works. The following loop runs the simulation, until `env.step` does not return the termination flag `done`. We will randomly chose actions using `env.action_space.sample()`, which means the experiment will probably fail very fast (CartPole environment terminates when the speed of CartPole, its position or angle are outside certain limits).\n",
|
||||
"\n",
|
||||
"> Simulation will open in the new window. You can run the code several times and see how it behaves."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"env.reset()\n",
|
||||
"\n",
|
||||
"done = False\n",
|
||||
"total_reward = 0\n",
|
||||
"while not done:\n",
|
||||
" env.render()\n",
|
||||
" obs, rew, done, info = env.step(env.action_space.sample())\n",
|
||||
" total_reward += rew\n",
|
||||
" print(f\"{obs} -> {rew}\")\n",
|
||||
"print(f\"Total reward: {total_reward}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Youn can notice that observations contain 4 numbers. They are:\n",
|
||||
"- Position of cart\n",
|
||||
"- Velocity of cart\n",
|
||||
"- Angle of pole\n",
|
||||
"- Rotation rate of pole\n",
|
||||
"\n",
|
||||
"`rew` is the reward we receive at each step. You can see that in CartPole environment you are rewarded 1 point for each simulation step, and the goal is to maximize total reward, i.e. the time CartPole is able to balance without falling.\n",
|
||||
"\n",
|
||||
"During reinforcement learning, our goal is to train a **policy** $\\pi$, that for each state $s$ will tell us which action $a$ to take, so essentially $a = \\pi(s)$.\n",
|
||||
"\n",
|
||||
"If you want probabilistic solution, you can think of policy as returning a set of probabilities for each action, i.e. $\\pi(a|s)$ would mean a probability that we should take action $a$ at state $s$.\n",
|
||||
"\n",
|
||||
"## Policy Gradient Method\n",
|
||||
"\n",
|
||||
"In simplest RL algorithm, called **Policy Gradient**, we will train a neural network to predict the next action."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import numpy as np\n",
|
||||
"import matplotlib.pyplot as plt\n",
|
||||
"import torch\n",
|
||||
"\n",
|
||||
"num_inputs = 4\n",
|
||||
"num_actions = 2\n",
|
||||
"\n",
|
||||
"model = torch.nn.Sequential(\n",
|
||||
" torch.nn.Linear(num_inputs, 128, bias=False, dtype=torch.float32),\n",
|
||||
" torch.nn.ReLU(),\n",
|
||||
" torch.nn.Linear(128, num_actions, bias = False, dtype=torch.float32),\n",
|
||||
" torch.nn.Softmax(dim=1)\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"We will train the network by running many experiments, and updating our network after each run. Let's define a function that will run the experiment and return the results (so-called **trace**) - all states, actions (and their recommended probabilities), and rewards:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def run_episode(max_steps_per_episode = 10000,render=False): \n",
|
||||
" states, actions, probs, rewards = [],[],[],[]\n",
|
||||
" state = env.reset()\n",
|
||||
" for _ in range(max_steps_per_episode):\n",
|
||||
" if render:\n",
|
||||
" env.render()\n",
|
||||
" action_probs = model(torch.from_numpy(np.expand_dims(state,0)))[0]\n",
|
||||
" action = np.random.choice(num_actions, p=np.squeeze(action_probs.detach().numpy()))\n",
|
||||
" nstate, reward, done, info = env.step(action)\n",
|
||||
" if done:\n",
|
||||
" break\n",
|
||||
" states.append(state)\n",
|
||||
" actions.append(action)\n",
|
||||
" probs.append(action_probs.detach().numpy())\n",
|
||||
" rewards.append(reward)\n",
|
||||
" state = nstate\n",
|
||||
" return np.vstack(states), np.vstack(actions), np.vstack(probs), np.vstack(rewards)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"You can run one episode with untrained network and observe that total reward (AKA length of episode) is very low:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"s, a, p, r = run_episode()\n",
|
||||
"print(f\"Total reward: {np.sum(r)}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"One of the tricky aspects of policy gradient algorithm is to use **discounted rewards**. The idea is that we compute the vector of total rewards at each step of the game, and during this process we discount the early rewards using some coefficient $gamma$. We also normalize the resulting vector, because we will use it as weight to affect our training: "
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"eps = 0.0001\n",
|
||||
"\n",
|
||||
"def discounted_rewards(rewards,gamma=0.99,normalize=True):\n",
|
||||
" ret = []\n",
|
||||
" s = 0\n",
|
||||
" for r in rewards[::-1]:\n",
|
||||
" s = r + gamma * s\n",
|
||||
" ret.insert(0, s)\n",
|
||||
" if normalize:\n",
|
||||
" ret = (ret-np.mean(ret))/(np.std(ret)+eps)\n",
|
||||
" return ret"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Now let's do the actual training! We will run 300 episodes, and at each episode we will do the following:\n",
|
||||
"\n",
|
||||
"1. Run the experiment and collect the trace\n",
|
||||
"1. Calculate the difference (`gradients`) between the actions taken, and by predicted probabilities. The less the difference is, the more we are sure that we have taken the right action.\n",
|
||||
"1. Calculate discounted rewards and multiply gradients by discounted rewards - that will make sure that steps with higher rewards will make more effect on the final result than lower-rewarded ones\n",
|
||||
"1. Expected target actions for our neural network would be partly taken from the predicted probabilities during the run, and partly from calculated gradients. We will use `alpha` parameter to determine to which extent gradients and rewards are taken into account - this is called *learning rate* of reinforcement algorithm.\n",
|
||||
"1. Finally, we train our network on states and expected actions, and repeat the process "
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"optimizer = torch.optim.Adam(model.parameters(), lr=0.01)\n",
|
||||
"\n",
|
||||
"def train_on_batch(x, y):\n",
|
||||
" x = torch.from_numpy(x)\n",
|
||||
" y = torch.from_numpy(y)\n",
|
||||
" optimizer.zero_grad()\n",
|
||||
" predictions = model(x)\n",
|
||||
" loss = -torch.mean(torch.log(predictions) * y)\n",
|
||||
" loss.backward()\n",
|
||||
" optimizer.step()\n",
|
||||
" return loss"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"alpha = 1e-4\n",
|
||||
"\n",
|
||||
"history = []\n",
|
||||
"for epoch in range(300):\n",
|
||||
" states, actions, probs, rewards = run_episode()\n",
|
||||
" one_hot_actions = np.eye(2)[actions.T][0]\n",
|
||||
" gradients = one_hot_actions-probs\n",
|
||||
" dr = discounted_rewards(rewards)\n",
|
||||
" gradients *= dr\n",
|
||||
" target = alpha*np.vstack([gradients])+probs\n",
|
||||
" train_on_batch(states,target)\n",
|
||||
" history.append(np.sum(rewards))\n",
|
||||
" if epoch%100==0:\n",
|
||||
" print(f\"{epoch} -> {np.sum(rewards)}\")\n",
|
||||
"\n",
|
||||
"plt.plot(history)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Now let's run the episode with rendering to see the result:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"_ = run_episode(render=True)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Hopefully, you can see that pole can now balance pretty well!\n",
|
||||
"\n",
|
||||
"## Actor-Critic Model\n",
|
||||
"\n",
|
||||
"Actor-Critic model is the further development of policy gradients, in which we build a neural network to learn both the policy and estimated rewards. The network will have two outputs (or you can view it as two separate networks):\n",
|
||||
"* **Actor** will recommend the action to take by giving us the state probability distribution, as in policy gradient model\n",
|
||||
"* **Critic** would estimate what the reward would be from those actions. It returns total estimated rewards in the future at the given state.\n",
|
||||
"\n",
|
||||
"Let's define such a model: "
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from itertools import count\n",
|
||||
"import torch.nn.functional as F"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
|
||||
"env = gym.make(\"CartPole-v1\")\n",
|
||||
"\n",
|
||||
"state_size = env.observation_space.shape[0]\n",
|
||||
"action_size = env.action_space.n\n",
|
||||
"lr = 0.0001\n",
|
||||
"\n",
|
||||
"class Actor(torch.nn.Module):\n",
|
||||
" def __init__(self, state_size, action_size):\n",
|
||||
" super(Actor, self).__init__()\n",
|
||||
" self.state_size = state_size\n",
|
||||
" self.action_size = action_size\n",
|
||||
" self.linear1 = torch.nn.Linear(self.state_size, 128)\n",
|
||||
" self.linear2 = torch.nn.Linear(128, 256)\n",
|
||||
" self.linear3 = torch.nn.Linear(256, self.action_size)\n",
|
||||
"\n",
|
||||
" def forward(self, state):\n",
|
||||
" output = F.relu(self.linear1(state))\n",
|
||||
" output = F.relu(self.linear2(output))\n",
|
||||
" output = self.linear3(output)\n",
|
||||
" distribution = torch.distributions.Categorical(F.softmax(output, dim=-1))\n",
|
||||
" return distribution\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"class Critic(torch.nn.Module):\n",
|
||||
" def __init__(self, state_size, action_size):\n",
|
||||
" super(Critic, self).__init__()\n",
|
||||
" self.state_size = state_size\n",
|
||||
" self.action_size = action_size\n",
|
||||
" self.linear1 = torch.nn.Linear(self.state_size, 128)\n",
|
||||
" self.linear2 = torch.nn.Linear(128, 256)\n",
|
||||
" self.linear3 = torch.nn.Linear(256, 1)\n",
|
||||
"\n",
|
||||
" def forward(self, state):\n",
|
||||
" output = F.relu(self.linear1(state))\n",
|
||||
" output = F.relu(self.linear2(output))\n",
|
||||
" value = self.linear3(output)\n",
|
||||
" return value"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"We would need to slightly modify our `discounted_rewards` and `run_episode` functions:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def discounted_rewards(next_value, rewards, masks, gamma=0.99):\n",
|
||||
" R = next_value\n",
|
||||
" returns = []\n",
|
||||
" for step in reversed(range(len(rewards))):\n",
|
||||
" R = rewards[step] + gamma * R * masks[step]\n",
|
||||
" returns.insert(0, R)\n",
|
||||
" return returns\n",
|
||||
"\n",
|
||||
"def run_episode(actor, critic, n_iters):\n",
|
||||
" optimizerA = torch.optim.Adam(actor.parameters())\n",
|
||||
" optimizerC = torch.optim.Adam(critic.parameters())\n",
|
||||
" for iter in range(n_iters):\n",
|
||||
" state = env.reset()\n",
|
||||
" log_probs = []\n",
|
||||
" values = []\n",
|
||||
" rewards = []\n",
|
||||
" masks = []\n",
|
||||
" entropy = 0\n",
|
||||
" env.reset()\n",
|
||||
"\n",
|
||||
" for i in count():\n",
|
||||
" env.render()\n",
|
||||
" state = torch.FloatTensor(state).to(device)\n",
|
||||
" dist, value = actor(state), critic(state)\n",
|
||||
"\n",
|
||||
" action = dist.sample()\n",
|
||||
" next_state, reward, done, _ = env.step(action.cpu().numpy())\n",
|
||||
"\n",
|
||||
" log_prob = dist.log_prob(action).unsqueeze(0)\n",
|
||||
" entropy += dist.entropy().mean()\n",
|
||||
"\n",
|
||||
" log_probs.append(log_prob)\n",
|
||||
" values.append(value)\n",
|
||||
" rewards.append(torch.tensor([reward], dtype=torch.float, device=device))\n",
|
||||
" masks.append(torch.tensor([1-done], dtype=torch.float, device=device))\n",
|
||||
"\n",
|
||||
" state = next_state\n",
|
||||
"\n",
|
||||
" if done:\n",
|
||||
" print('Iteration: {}, Score: {}'.format(iter, i))\n",
|
||||
" break\n",
|
||||
"\n",
|
||||
"\n",
|
||||
" next_state = torch.FloatTensor(next_state).to(device)\n",
|
||||
" next_value = critic(next_state)\n",
|
||||
" returns = discounted_rewards(next_value, rewards, masks)\n",
|
||||
"\n",
|
||||
" log_probs = torch.cat(log_probs)\n",
|
||||
" returns = torch.cat(returns).detach()\n",
|
||||
" values = torch.cat(values)\n",
|
||||
"\n",
|
||||
" advantage = returns - values\n",
|
||||
"\n",
|
||||
" actor_loss = -(log_probs * advantage.detach()).mean()\n",
|
||||
" critic_loss = advantage.pow(2).mean()\n",
|
||||
"\n",
|
||||
" optimizerA.zero_grad()\n",
|
||||
" optimizerC.zero_grad()\n",
|
||||
" actor_loss.backward()\n",
|
||||
" critic_loss.backward()\n",
|
||||
" optimizerA.step()\n",
|
||||
" optimizerC.step()\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Now we will run the main training loop. We will use manual network training process by computing proper loss functions and updating network parameters:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"\n",
|
||||
"actor = Actor(state_size, action_size).to(device)\n",
|
||||
"critic = Critic(state_size, action_size).to(device)\n",
|
||||
"run_episode(actor, critic, n_iters=100)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Finally, let's close the environment."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"env.close()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Takeaway\n",
|
||||
"\n",
|
||||
"We have seen two RL algorithms in this demo: simple policy gradient, and more sophisticated actor-critic. You can see that those algorithms operate with abstract notions of state, action and reward - thus they can be applied to very different environments.\n",
|
||||
"\n",
|
||||
"Reinforcement learning allows us to learn the best strategy to solve the problem just by looking at the final reward. The fact that we do not need labelled datasets allows us to repeat simulations many times to optimize our models. However, there are still many challenges in RL, which you may learn if you decide to focus more on this interesting area of AI. "
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3.10.4 64-bit",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.10.4"
|
||||
},
|
||||
"orig_nbformat": 4,
|
||||
"vscode": {
|
||||
"interpreter": {
|
||||
"hash": "916dbcbb3f70747c44a77c7bcd40155683ae19c65e1c03b4aa3499c5328201f1"
|
||||
}
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2
|
||||
}
|
||||
File diff suppressed because one or more lines are too long
|
|
@ -83,6 +83,7 @@ After running one of those algorithms, we can expect our CartPole to behave like
|
|||
Continue your learning in the following notebooks:
|
||||
|
||||
* [RL in TensorFlow](CartPole-RL-TF.ipynb)
|
||||
* [RL in PyTorch](CartPole-RL-PyTorch.ipynb)
|
||||
|
||||
## Other RL Tasks
|
||||
|
||||
|
|
|
|||
File diff suppressed because one or more lines are too long
Loading…
Reference in New Issue