diff --git a/README.md b/README.md index 1e6aaf04..aab56ef2 100644 --- a/README.md +++ b/README.md @@ -82,7 +82,7 @@ For a gentle introduction to *AI in the Cloud* topics you may consider taking th 13Text Representation. Bow/TF-IDFTextPyTorchTensorFlow 14Semantic word embeddings. Word2Vec and GloVeTextPyTorchTensorFlow -15Language Modeling. Training your own embeddingsTextPyTorchTensorFlow +15Language Modeling. Training your own embeddingsTextTensorFlowLab 16Recurrent Neural NetworksTextPyTorchTensorFlow 17Generative Recurrent NetworksTextPyTorchTensorFlowLab 18Transformers. BERT.TextPyTorchTensorFlow diff --git a/lessons/5-NLP/15-LanguageModeling/CBoW-TF.ipynb b/lessons/5-NLP/15-LanguageModeling/CBoW-TF.ipynb new file mode 100644 index 00000000..ee49610a --- /dev/null +++ b/lessons/5-NLP/15-LanguageModeling/CBoW-TF.ipynb @@ -0,0 +1,3344 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": { + "id": "NXTSugt6ieXh" + }, + "source": [ + "## Training CBoW Model\n", + "\n", + "This notebooks is a part of [AI for Beginners Curriculum](http://aka.ms/ai-beginners)\n", + "\n", + "In this example, we will look at training CBoW language model to get our own Word2Vec embedding space. We will use AG News dataset as the source of text." + ] + }, + { + "cell_type": "code", + "execution_count": 30, + "metadata": { + "id": "hvf7izZpieXk" + }, + "outputs": [], + "source": [ + "from tensorflow import keras\n", + "import tensorflow as tf\n", + "import tensorflow_datasets as tfds\n", + "import numpy as np" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We will start by loading the dateset:" + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 299, + "referenced_widgets": [ + "a6a32befb28542228cde3d444d6411f6", + "9f5a040885564d41934f6c458761bf33", + "a7651b06cb974b52a35e34e8f96c226c", + "ee267b7dcf05457b8e3f545df150f09f", + "55cc8f6b3b0c49ddbc4bfe526906ccbf", + "9d4d315121e9440c8578a62fbe88e415", + "a1f8b53e8a1d4ebd8ed0116219490877", + "074db709c0a14cd4acfe13ed54e92cbc", + "713f10e7274e4f0484d34759b5505842", + "5ac5967b468d4af59cea0693ce9a8217", + "7916d209cbe04da2912830b16e5f747c", + "1a467f2a5e4b421db16e4fd4329f9bc1", + "40ac80ed694940c191002b92e30a9f39", + "4d62b43b387b45c89471f9820c041718", + "ee572078162448bd89bd2c52fbe39aa7", + "2bddf650279242d7a259db4209de3253", + "9bba163f10a4461ca232b54d79c9b74d", + "f91bb89f9f144f8e97f8f0c97f7d9f55", + "bec242f072394c4cabc6f39e43350603", + "74a70de4caa94d358606308df816725b", + "be1b974e61b44ecd807a77a94f6f7991", + "72693d5e2c034fb9ad03a32a8eb2999f", + "3832f856d7644da2aedc8dc843732269", + "3e31ac504d4242b8be208b68219bf064", + "bb16a77431264209935f8e747918e430", + "b1c32e56d326473db36bcda7abca7010", + "7b38647b718d4cb58c20480e6387d749", + "490cee33a6c14ead8209f36ca7dc1351", + "5eee10ee14ff47f9bea181870bb973e3", + "15d88a4607524d07be1f3b91345243ba", + "a585d2e5ac5240679587990dbf53dfd2", + "bbce51f4b75a4f999f4b3c170083e724", + "25c18271a4594c1fa8c694d50dd356a7", + "3edc08cbd6774e7485d42bdd13164ed6", + "1cf5cb0c39cc4c5ca66be45895aa1860", + "b70cac0930eb4d2da24a9f5b042f4a9e", + "b38d4f6271234adfb41e8309a115e95b", + "d7ef0d4ec19749c3bee8a0be8ad2d468", + "bfb1e75c5bc744f28544515c660a0b9b", + "62aa64e5318d445f844b8083ae6c40f4", + "7ab6c716b4a04052bf048d0ede312365", + "94db6867a26a4c988f549d20b3cb51f3", + "30ffd5f13a524b0eabd5d2f20885ce50", + "5022afdb2026474081a3f54cb4c81351", + "52d3dd9cf2994e6da25a10ea42be4beb", + "82ee379245d64fc39d1ed9a2586e20a2", + "24283dc34e944591877888871a5e584b", + "027dbec8c35d4ec3ab43ed2878a32eb9", + "b459e5715d3b44eeb379108510261336", + "b58f7dc6368b42d0a387e47bce4ce88e", + "2252a0aeca7c4b7784370704181f1628", + "dc49356d1ba943ad87b88ee6e451e7fb", + "7095c6398b0c433db8c4284620c9e335", + "19ece8654d8149ac87538fd162bd1aeb", + "e630a16615414ceeba5868d162f55a20", + "13ee77a308634d928662b651ad2bb9e7", + "260fcdf1d0404d149732d566b6ccbbab", + "f3ad889117ba43b783e34a82113b325c", + "4dace863d2be4961ae72c729405da6cc", + "2bbd772ad6284273b3cf97c6afeda6e0", + "ca245734f2f54c4e805e761d23652eca", + "bcedd81ebcef4d9ca31eea1ae4ab795d", + "532f40fc7a1e4826b4495be24ee0f8ed", + "b26304339073463b9f0ba2cce4835d13", + "abaa80c91f5642649996c844ceb0fcd1", + "8655ea4b7b6c4399adaf6e04613869ea", + "fc94257ae5094ce0b04695ad29bdf72b", + "30224d6b4c274faf85dbd4d2c1892aa7", + "295d430b24444986a46a9382c5d5f80d", + "9a4eedfb4c6a466ba6f6f21ce76a64bb", + "9e28f7897bf142aebd4d374559320812", + "0798ebda763a40bc86235a40dfc1adec", + "bcd9ea70684742b6991d4e2c7556efa6", + "44e94cb4f240446da537579caa8e6d2f", + "8591f95a707d4214a17e9f187df6e1c4", + "75dd999664ac40f18168f6e1870a878e", + "91d15913f17040da828ece1c3b5fa6c6" + ] + }, + "id": "pWPCrm2jieXl", + "outputId": "7ffa325f-d5d2-4044-d318-0a521f4f5c98" + }, + "outputs": [], + "source": [ + "ds_train, ds_test = tfds.load('ag_news_subset').values()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## CBoW Model\n", + "\n", + "CBoW learns to predict a word based on the $2N$ neighboring words. For example, when $N=1$, we will get the following pairs from the sentence *I like to train networks*: (like,I), (I, like), (to, like), (like,to), (train,to), (to, train), (networks, train), (train,networks). Here, first word is the neighboring word used as an input, and second word is the one we are predicting.\n", + "\n", + "To build a network to predict next word, we will need to supply neighboring word as input, and get word number as output. The architecture of CBoW network is the following:\n", + "\n", + "* Input word is passed through the embedding layer. This very embedding layer would be our Word2Vec embedding, thus we will define it separately as `embedder` variable. We will use embedding size = 30 in this example, even though you might want to experiment with higher dimensions (real word2vec has 300)\n", + "* Embedding vector would then be passed to a dense layer that will predict output word. Thus it has the `vocab_size` neurons.\n", + "\n", + "Embedding layer in Keras automatically knows how to convert numeric input into one-hot encoding, so that we do not have to one-hot-encode input word separately. We specify `input_length=1` to indicate that we want just one word in the input sequence - normally embedding layer is designed to work with longer sequences.\n", + "\n", + "For the output, if we use `sparse_categorical_crossentropy` as loss function, we would also have to provide just word numbers as expected results, without one-hot encoding.\n", + "\n", + "We will set `vocab_size` to 5000 to limit computations a bit. We will also define a vectorizer which we will use later. " + ] + }, + { + "cell_type": "code", + "execution_count": 68, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "6PHiH8oRieXl", + "outputId": "0259a0d5-b5f1-4bc9-d632-73c31893fa3f" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Model: \"sequential_1\"\n", + "_________________________________________________________________\n", + " Layer (type) Output Shape Param # \n", + "=================================================================\n", + " embedding_1 (Embedding) (None, 1, 30) 150000 \n", + " \n", + " dense_1 (Dense) (None, 1, 5000) 155000 \n", + " \n", + "=================================================================\n", + "Total params: 305,000\n", + "Trainable params: 305,000\n", + "Non-trainable params: 0\n", + "_________________________________________________________________\n" + ] + } + ], + "source": [ + "vocab_size = 5000\n", + "\n", + "vectorizer = keras.layers.experimental.preprocessing.TextVectorization(max_tokens=vocab_size,input_shape=(1,))\n", + "embedder = keras.layers.Embedding(vocab_size,30,input_length=1)\n", + "\n", + "model = keras.Sequential([\n", + " embedder,\n", + " keras.layers.Dense(vocab_size,activation='softmax')\n", + "])\n", + "\n", + "model.summary()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Let's initialize the vectorizer and get out the vocabulary:" + ] + }, + { + "cell_type": "code", + "execution_count": 69, + "metadata": { + "id": "rWnylDAIieXn" + }, + "outputs": [], + "source": [ + "def extract_text(x):\n", + " return x['title']+' '+x['description']\n", + "\n", + "vectorizer.adapt(ds_train.take(500).map(extract_text))\n", + "vocab = vectorizer.get_vocabulary()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Preparing Training Data\n", + "\n", + "Now let's program the main function that will compute CBoW word pairs from text. This function will allow us to specify window size, and will return a set of pairs - input and output word. Note that this function can be used on words, as well as on vectors/tensors - which will allow us to encode the text, before passing it to `to_cbow` function." + ] + }, + { + "cell_type": "code", + "execution_count": 70, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "x-dsXygOieXn", + "outputId": "11828ef5-5961-4909-f777-ff7b9b93adbd" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "[['like', 'I'], ['to', 'I'], ['I', 'like'], ['to', 'like'], ['train', 'like'], ['I', 'to'], ['like', 'to'], ['train', 'to'], ['networks', 'to'], ['like', 'train'], ['to', 'train'], ['networks', 'train'], ['to', 'networks'], ['train', 'networks']]\n", + "[[, ], [, ], [, ], [, ], [, ], [, ], [, ], [, ], [, ], [, ], [, ], [, ], [, ], [, ]]\n" + ] + } + ], + "source": [ + "def to_cbow(sent,window_size=2):\n", + " res = []\n", + " for i,x in enumerate(sent):\n", + " for j in range(max(0,i-window_size),min(i+window_size+1,len(sent))):\n", + " if i!=j:\n", + " res.append([sent[j],x])\n", + " return res\n", + "\n", + "print(to_cbow(['I','like','to','train','networks']))\n", + "print(to_cbow(vectorizer('I like to train networks')))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Let's prepare the training dataset. We will go through all news, call `to_cbow` to get the list of word pairs, and add those pairs to `X` and `Y`. For the sake of time, we will only consider first 10k news items - you can easily remove the limitation in case you have more time to wait, and want to get better embeddings :)" + ] + }, + { + "cell_type": "code", + "execution_count": 100, + "metadata": { + "id": "54b-Gd9TieXo" + }, + "outputs": [], + "source": [ + "X = []\n", + "Y = []\n", + "for i,x in zip(range(10000),ds_train.map(extract_text).as_numpy_iterator()):\n", + " for w1, w2 in to_cbow(vectorizer(x),window_size=1):\n", + " X.append(tf.expand_dims(w1,0))\n", + " Y.append(tf.expand_dims(w2,0))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We will also convert that data to one dataset, and batch it for training:" + ] + }, + { + "cell_type": "code", + "execution_count": 101, + "metadata": { + "id": "AbLUcojlieXo" + }, + "outputs": [], + "source": [ + "ds = tf.data.Dataset.from_tensor_slices((X,Y)).batch(256)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Now let's do the actual training. We will use `SGD` optimizer with pretty high learning rate. You can also try playing around with other optimizers, such as `Adam`. We will train for 200 epochs to begin with - and you can re-run this cell if you want even lower loss." + ] + }, + { + "cell_type": "code", + "execution_count": 102, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "xAcGAQtVieXp", + "outputId": "bbab8c44-de25-49b9-ec3f-07db878a0818" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Epoch 1/200\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "/usr/local/lib/python3.7/dist-packages/keras/optimizer_v2/gradient_descent.py:102: UserWarning: The `lr` argument is deprecated, use `learning_rate` instead.\n", + " super(SGD, self).__init__(name, **kwargs)\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.6134\n", + "Epoch 2/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.5431\n", + "Epoch 3/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.5029\n", + "Epoch 4/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.4754\n", + "Epoch 5/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.4548\n", + "Epoch 6/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.4382\n", + "Epoch 7/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.4243\n", + "Epoch 8/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.4123\n", + "Epoch 9/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.4019\n", + "Epoch 10/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.3926\n", + "Epoch 11/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.3843\n", + "Epoch 12/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.3767\n", + "Epoch 13/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.3697\n", + "Epoch 14/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.3632\n", + "Epoch 15/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.3571\n", + "Epoch 16/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.3513\n", + "Epoch 17/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.3459\n", + "Epoch 18/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.3408\n", + "Epoch 19/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.3359\n", + "Epoch 20/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.3312\n", + "Epoch 21/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.3266\n", + "Epoch 22/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.3223\n", + "Epoch 23/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.3181\n", + "Epoch 24/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.3140\n", + "Epoch 25/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.3101\n", + "Epoch 26/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.3062\n", + "Epoch 27/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.3025\n", + "Epoch 28/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.2989\n", + "Epoch 29/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.2953\n", + "Epoch 30/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.2919\n", + "Epoch 31/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.2885\n", + "Epoch 32/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.2852\n", + "Epoch 33/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.2819\n", + "Epoch 34/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.2787\n", + "Epoch 35/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.2756\n", + "Epoch 36/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.2725\n", + "Epoch 37/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.2695\n", + "Epoch 38/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.2665\n", + "Epoch 39/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.2636\n", + "Epoch 40/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.2607\n", + "Epoch 41/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.2578\n", + "Epoch 42/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.2550\n", + "Epoch 43/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.2523\n", + "Epoch 44/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.2495\n", + "Epoch 45/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.2468\n", + "Epoch 46/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.2442\n", + "Epoch 47/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.2416\n", + "Epoch 48/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.2390\n", + "Epoch 49/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.2364\n", + "Epoch 50/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.2339\n", + "Epoch 51/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.2314\n", + "Epoch 52/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.2290\n", + "Epoch 53/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.2266\n", + "Epoch 54/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.2242\n", + "Epoch 55/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.2218\n", + "Epoch 56/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.2195\n", + "Epoch 57/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.2172\n", + "Epoch 58/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.2149\n", + "Epoch 59/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.2126\n", + "Epoch 60/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.2104\n", + "Epoch 61/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.2082\n", + "Epoch 62/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.2060\n", + "Epoch 63/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.2038\n", + "Epoch 64/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.2017\n", + "Epoch 65/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1996\n", + "Epoch 66/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1975\n", + "Epoch 67/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1954\n", + "Epoch 68/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1933\n", + "Epoch 69/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1913\n", + "Epoch 70/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1893\n", + "Epoch 71/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1873\n", + "Epoch 72/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1853\n", + "Epoch 73/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1833\n", + "Epoch 74/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1814\n", + "Epoch 75/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1795\n", + "Epoch 76/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1775\n", + "Epoch 77/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1756\n", + "Epoch 78/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1737\n", + "Epoch 79/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1719\n", + "Epoch 80/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1700\n", + "Epoch 81/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1682\n", + "Epoch 82/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1663\n", + "Epoch 83/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1645\n", + "Epoch 84/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1627\n", + "Epoch 85/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1609\n", + "Epoch 86/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1592\n", + "Epoch 87/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1574\n", + "Epoch 88/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1557\n", + "Epoch 89/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1539\n", + "Epoch 90/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1522\n", + "Epoch 91/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1505\n", + "Epoch 92/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1488\n", + "Epoch 93/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1471\n", + "Epoch 94/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1454\n", + "Epoch 95/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1438\n", + "Epoch 96/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1421\n", + "Epoch 97/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1405\n", + "Epoch 98/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1389\n", + "Epoch 99/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1372\n", + "Epoch 100/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1356\n", + "Epoch 101/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1341\n", + "Epoch 102/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1325\n", + "Epoch 103/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1309\n", + "Epoch 104/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1293\n", + "Epoch 105/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1278\n", + "Epoch 106/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1263\n", + "Epoch 107/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1247\n", + "Epoch 108/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1232\n", + "Epoch 109/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1217\n", + "Epoch 110/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1202\n", + "Epoch 111/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1187\n", + "Epoch 112/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1173\n", + "Epoch 113/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1158\n", + "Epoch 114/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1144\n", + "Epoch 115/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1129\n", + "Epoch 116/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1115\n", + "Epoch 117/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1101\n", + "Epoch 118/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1086\n", + "Epoch 119/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1072\n", + "Epoch 120/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1058\n", + "Epoch 121/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1045\n", + "Epoch 122/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1031\n", + "Epoch 123/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1017\n", + "Epoch 124/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1004\n", + "Epoch 125/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0990\n", + "Epoch 126/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0977\n", + "Epoch 127/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0963\n", + "Epoch 128/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0950\n", + "Epoch 129/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0937\n", + "Epoch 130/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0924\n", + "Epoch 131/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0911\n", + "Epoch 132/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0898\n", + "Epoch 133/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0885\n", + "Epoch 134/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0873\n", + "Epoch 135/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0860\n", + "Epoch 136/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0848\n", + "Epoch 137/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0835\n", + "Epoch 138/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0823\n", + "Epoch 139/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0810\n", + "Epoch 140/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0798\n", + "Epoch 141/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0786\n", + "Epoch 142/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0774\n", + "Epoch 143/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0762\n", + "Epoch 144/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0750\n", + "Epoch 145/200\n", + "2156/2156 [==============================] - 8s 4ms/step - loss: 5.0739\n", + "Epoch 146/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0727\n", + "Epoch 147/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0715\n", + "Epoch 148/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0704\n", + "Epoch 149/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0692\n", + "Epoch 150/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0681\n", + "Epoch 151/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0670\n", + "Epoch 152/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0658\n", + "Epoch 153/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0647\n", + "Epoch 154/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0636\n", + "Epoch 155/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0625\n", + "Epoch 156/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0614\n", + "Epoch 157/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0603\n", + "Epoch 158/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0593\n", + "Epoch 159/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0582\n", + "Epoch 160/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0571\n", + "Epoch 161/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0561\n", + "Epoch 162/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0550\n", + "Epoch 163/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0539\n", + "Epoch 164/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0529\n", + "Epoch 165/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0519\n", + "Epoch 166/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0508\n", + "Epoch 167/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0498\n", + "Epoch 168/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0488\n", + "Epoch 169/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0478\n", + "Epoch 170/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0468\n", + "Epoch 171/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0458\n", + "Epoch 172/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0448\n", + "Epoch 173/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0438\n", + "Epoch 174/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0428\n", + "Epoch 175/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0418\n", + "Epoch 176/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0409\n", + "Epoch 177/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0399\n", + "Epoch 178/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0389\n", + "Epoch 179/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0380\n", + "Epoch 180/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0370\n", + "Epoch 181/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0361\n", + "Epoch 182/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0351\n", + "Epoch 183/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0342\n", + "Epoch 184/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0333\n", + "Epoch 185/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0323\n", + "Epoch 186/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0314\n", + "Epoch 187/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0305\n", + "Epoch 188/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0296\n", + "Epoch 189/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0287\n", + "Epoch 190/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0278\n", + "Epoch 191/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0269\n", + "Epoch 192/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0260\n", + "Epoch 193/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0251\n", + "Epoch 194/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0242\n", + "Epoch 195/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0233\n", + "Epoch 196/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0225\n", + "Epoch 197/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0216\n", + "Epoch 198/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0207\n", + "Epoch 199/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0199\n", + "Epoch 200/200\n", + "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0190\n" + ] + }, + { + "data": { + "text/plain": [ + "" + ] + }, + "execution_count": 102, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "model.compile(optimizer=keras.optimizers.SGD(lr=0.1),loss='sparse_categorical_crossentropy')\n", + "model.fit(ds,epochs=200)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Trying out Word2Vec\n", + "\n", + "To use Word2Vec, let's extract vectors corresponding to all words in our vocabulary:" + ] + }, + { + "cell_type": "code", + "execution_count": 103, + "metadata": { + "id": "r8TatcXjkU_t" + }, + "outputs": [], + "source": [ + "vectors = embedder(vectorizer(vocab))\n", + "vectors = tf.reshape(vectors,(-1,30)) # we need reshape to get rid of extra dimension" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Let's see, for example, how the word **Paris** is encoded into a vector:" + ] + }, + { + "cell_type": "code", + "execution_count": 104, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "bz6tAeLzieXp", + "outputId": "c0422bc7-ca08-4f99-bced-e46d8b9b93e3" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "tf.Tensor(\n", + "[-0.13308628 0.50972325 0.00344684 0.185389 -0.03176536 0.22262476\n", + " -0.3856765 -0.6854793 0.5185803 -0.7215402 -0.16101503 0.15622072\n", + " 0.00653811 -0.14954254 0.03379822 -0.01243829 0.27907634 -0.32538188\n", + " 0.21718933 0.31112966 -0.24142407 0.15589055 0.2915561 0.19029242\n", + " 0.08425518 -0.0941902 -0.54313695 -0.24854654 0.26196313 0.18027727], shape=(30,), dtype=float32)\n" + ] + } + ], + "source": [ + "paris_vec = embedder(vectorizer('paris'))[0]\n", + "print(paris_vec)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "It is interesting to use Word2Vec to look for synonyms. The following function will return `n` closest words to a given input. To find them, we compute the norm of $|w_i - v|$, where $v$ is the vector corresponding to our input word, and $w_i$ is the encoding of $i$-th word in the vocabulary. We then sort the array and return corresponding indices using `argsort`, and take first `n` elements of the list, which encode positions of closest words in the vocabulary. " + ] + }, + { + "cell_type": "code", + "execution_count": 105, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "NlZyi-_olFar", + "outputId": "4e4543db-4472-4b46-affd-71f39df4d342" + }, + "outputs": [ + { + "data": { + "text/plain": [ + "['paris', 'philippines', 'seoul', 'jakarta', 'zoo']" + ] + }, + "execution_count": 105, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "def close_words(x,n=5):\n", + " vec = embedder(vectorizer(x))[0]\n", + " top5 = np.linalg.norm(vectors-vec,axis=1).argsort()[:n]\n", + " return [ vocab[x] for x in top5 ]\n", + "\n", + "close_words('paris')" + ] + }, + { + "cell_type": "code", + "execution_count": 112, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "-dQq7xeAln0U", + "outputId": "3fdf5f9b-554c-4546-d84e-b88a96dc0e01" + }, + "outputs": [ + { + "data": { + "text/plain": [ + "['china', 'russia', 'pakistan', 'israel', 'turkey']" + ] + }, + "execution_count": 112, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "close_words('china')" + ] + }, + { + "cell_type": "code", + "execution_count": 113, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "fJXqK26b29sa", + "outputId": "7a51e71f-1a1d-409e-c050-cffebb145095" + }, + "outputs": [ + { + "data": { + "text/plain": [ + "['official', 'military', 'office', 'police', 'sources']" + ] + }, + "execution_count": 113, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "close_words('official')" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "My0VeTDd3Ji8" + }, + "source": [ + "## Takeaway\n", + "\n", + "Using clever techniques such as CBoW, we can train Word2Vec model. You may also try to train skip-gram model that is trained to predict the neighboring word given the central one, and see how well it performs. " + ] + } + ], + "metadata": { + "accelerator": "GPU", + "colab": { + "collapsed_sections": [], + "name": "CBoW-TF.ipynb", + "provenance": [] + }, + "interpreter": { + "hash": "16af2a8bbb083ea23e5e41c7f5787656b2ce26968575d8763f2c4b17f9cd711f" + }, + "kernelspec": { + "display_name": "Python 3.8.12 ('py38')", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.8.12" + }, + "orig_nbformat": 4, + "widgets": { + "application/vnd.jupyter.widget-state+json": { + "027dbec8c35d4ec3ab43ed2878a32eb9": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "HTMLModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HTMLModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HTMLView", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_19ece8654d8149ac87538fd162bd1aeb", + "placeholder": "​", + "style": "IPY_MODEL_e630a16615414ceeba5868d162f55a20", + "value": " 119999/120000 [00:00<00:00, 291181.43 examples/s]" + } + }, + "074db709c0a14cd4acfe13ed54e92cbc": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "1.2.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": "20px" + } + }, + "0798ebda763a40bc86235a40dfc1adec": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "1.2.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "13ee77a308634d928662b651ad2bb9e7": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "HBoxModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HBoxModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HBoxView", + "box_style": "", + "children": [ + "IPY_MODEL_260fcdf1d0404d149732d566b6ccbbab", + "IPY_MODEL_f3ad889117ba43b783e34a82113b325c", + "IPY_MODEL_4dace863d2be4961ae72c729405da6cc" + ], + "layout": "IPY_MODEL_2bbd772ad6284273b3cf97c6afeda6e0" + } + }, + "15d88a4607524d07be1f3b91345243ba": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "1.2.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": "20px" + } + }, + "19ece8654d8149ac87538fd162bd1aeb": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "1.2.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "1a467f2a5e4b421db16e4fd4329f9bc1": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "HBoxModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HBoxModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HBoxView", + "box_style": "", + "children": [ + "IPY_MODEL_40ac80ed694940c191002b92e30a9f39", + "IPY_MODEL_4d62b43b387b45c89471f9820c041718", + "IPY_MODEL_ee572078162448bd89bd2c52fbe39aa7" + ], + "layout": "IPY_MODEL_2bddf650279242d7a259db4209de3253" + } + }, + "1cf5cb0c39cc4c5ca66be45895aa1860": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "HTMLModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HTMLModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HTMLView", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_bfb1e75c5bc744f28544515c660a0b9b", + "placeholder": "​", + "style": "IPY_MODEL_62aa64e5318d445f844b8083ae6c40f4", + "value": "" + } + }, + "2252a0aeca7c4b7784370704181f1628": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "DescriptionStyleModel", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "DescriptionStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "description_width": "" + } + }, + "24283dc34e944591877888871a5e584b": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "FloatProgressModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "FloatProgressModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "ProgressView", + "bar_style": "danger", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_dc49356d1ba943ad87b88ee6e451e7fb", + "max": 120000, + "min": 0, + "orientation": "horizontal", + "style": "IPY_MODEL_7095c6398b0c433db8c4284620c9e335", + "value": 119999 + } + }, + "25c18271a4594c1fa8c694d50dd356a7": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "DescriptionStyleModel", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "DescriptionStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "description_width": "" + } + }, + "260fcdf1d0404d149732d566b6ccbbab": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "HTMLModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HTMLModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HTMLView", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_ca245734f2f54c4e805e761d23652eca", + "placeholder": "​", + "style": "IPY_MODEL_bcedd81ebcef4d9ca31eea1ae4ab795d", + "value": "" + } + }, + "295d430b24444986a46a9382c5d5f80d": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "FloatProgressModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "FloatProgressModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "ProgressView", + "bar_style": "danger", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_44e94cb4f240446da537579caa8e6d2f", + "max": 7600, + "min": 0, + "orientation": "horizontal", + "style": "IPY_MODEL_8591f95a707d4214a17e9f187df6e1c4", + "value": 7599 + } + }, + "2bbd772ad6284273b3cf97c6afeda6e0": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "1.2.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "2bddf650279242d7a259db4209de3253": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "1.2.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "30224d6b4c274faf85dbd4d2c1892aa7": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "HTMLModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HTMLModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HTMLView", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_0798ebda763a40bc86235a40dfc1adec", + "placeholder": "​", + "style": "IPY_MODEL_bcd9ea70684742b6991d4e2c7556efa6", + "value": "100%" + } + }, + "30ffd5f13a524b0eabd5d2f20885ce50": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "1.2.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "3832f856d7644da2aedc8dc843732269": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "HBoxModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HBoxModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HBoxView", + "box_style": "", + "children": [ + "IPY_MODEL_3e31ac504d4242b8be208b68219bf064", + "IPY_MODEL_bb16a77431264209935f8e747918e430", + "IPY_MODEL_b1c32e56d326473db36bcda7abca7010" + ], + "layout": "IPY_MODEL_7b38647b718d4cb58c20480e6387d749" + } + }, + "3e31ac504d4242b8be208b68219bf064": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "HTMLModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HTMLModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HTMLView", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_490cee33a6c14ead8209f36ca7dc1351", + "placeholder": "​", + "style": "IPY_MODEL_5eee10ee14ff47f9bea181870bb973e3", + "value": "Extraction completed...: 100%" + } + }, + "3edc08cbd6774e7485d42bdd13164ed6": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "HBoxModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HBoxModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HBoxView", + "box_style": "", + "children": [ + "IPY_MODEL_1cf5cb0c39cc4c5ca66be45895aa1860", + "IPY_MODEL_b70cac0930eb4d2da24a9f5b042f4a9e", + "IPY_MODEL_b38d4f6271234adfb41e8309a115e95b" + ], + "layout": "IPY_MODEL_d7ef0d4ec19749c3bee8a0be8ad2d468" + } + }, + "40ac80ed694940c191002b92e30a9f39": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "HTMLModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HTMLModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HTMLView", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_9bba163f10a4461ca232b54d79c9b74d", + "placeholder": "​", + "style": "IPY_MODEL_f91bb89f9f144f8e97f8f0c97f7d9f55", + "value": "Dl Size...: " + } + }, + "44e94cb4f240446da537579caa8e6d2f": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "1.2.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "490cee33a6c14ead8209f36ca7dc1351": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "1.2.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "4d62b43b387b45c89471f9820c041718": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "FloatProgressModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "FloatProgressModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "ProgressView", + "bar_style": "success", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_bec242f072394c4cabc6f39e43350603", + "max": 1, + "min": 0, + "orientation": "horizontal", + "style": "IPY_MODEL_74a70de4caa94d358606308df816725b", + "value": 0 + } + }, + "4dace863d2be4961ae72c729405da6cc": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "HTMLModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HTMLModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HTMLView", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_abaa80c91f5642649996c844ceb0fcd1", + "placeholder": "​", + "style": "IPY_MODEL_8655ea4b7b6c4399adaf6e04613869ea", + "value": " 7221/0 [00:01<00:00, 4326.56 examples/s]" + } + }, + "5022afdb2026474081a3f54cb4c81351": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "DescriptionStyleModel", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "DescriptionStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "description_width": "" + } + }, + "52d3dd9cf2994e6da25a10ea42be4beb": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "HBoxModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HBoxModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HBoxView", + "box_style": "", + "children": [ + "IPY_MODEL_82ee379245d64fc39d1ed9a2586e20a2", + "IPY_MODEL_24283dc34e944591877888871a5e584b", + "IPY_MODEL_027dbec8c35d4ec3ab43ed2878a32eb9" + ], + "layout": "IPY_MODEL_b459e5715d3b44eeb379108510261336" + } + }, + "532f40fc7a1e4826b4495be24ee0f8ed": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "1.2.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": "20px" + } + }, + "55cc8f6b3b0c49ddbc4bfe526906ccbf": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "1.2.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "5ac5967b468d4af59cea0693ce9a8217": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "1.2.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "5eee10ee14ff47f9bea181870bb973e3": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "DescriptionStyleModel", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "DescriptionStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "description_width": "" + } + }, + "62aa64e5318d445f844b8083ae6c40f4": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "DescriptionStyleModel", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "DescriptionStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "description_width": "" + } + }, + "7095c6398b0c433db8c4284620c9e335": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "ProgressStyleModel", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "ProgressStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "bar_color": null, + "description_width": "" + } + }, + "713f10e7274e4f0484d34759b5505842": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "ProgressStyleModel", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "ProgressStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "bar_color": null, + "description_width": "" + } + }, + "72693d5e2c034fb9ad03a32a8eb2999f": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "DescriptionStyleModel", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "DescriptionStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "description_width": "" + } + }, + "74a70de4caa94d358606308df816725b": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "ProgressStyleModel", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "ProgressStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "bar_color": null, + "description_width": "" + } + }, + "75dd999664ac40f18168f6e1870a878e": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "1.2.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "7916d209cbe04da2912830b16e5f747c": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "DescriptionStyleModel", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "DescriptionStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "description_width": "" + } + }, + "7ab6c716b4a04052bf048d0ede312365": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "1.2.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": "20px" + } + }, + "7b38647b718d4cb58c20480e6387d749": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "1.2.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "82ee379245d64fc39d1ed9a2586e20a2": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "HTMLModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HTMLModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HTMLView", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_b58f7dc6368b42d0a387e47bce4ce88e", + "placeholder": "​", + "style": "IPY_MODEL_2252a0aeca7c4b7784370704181f1628", + "value": "100%" + } + }, + "8591f95a707d4214a17e9f187df6e1c4": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "ProgressStyleModel", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "ProgressStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "bar_color": null, + "description_width": "" + } + }, + "8655ea4b7b6c4399adaf6e04613869ea": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "DescriptionStyleModel", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "DescriptionStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "description_width": "" + } + }, + "91d15913f17040da828ece1c3b5fa6c6": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "DescriptionStyleModel", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "DescriptionStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "description_width": "" + } + }, + "94db6867a26a4c988f549d20b3cb51f3": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "ProgressStyleModel", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "ProgressStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "bar_color": null, + "description_width": "" + } + }, + "9a4eedfb4c6a466ba6f6f21ce76a64bb": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "HTMLModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HTMLModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HTMLView", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_75dd999664ac40f18168f6e1870a878e", + "placeholder": "​", + "style": "IPY_MODEL_91d15913f17040da828ece1c3b5fa6c6", + "value": " 7599/7600 [00:00<00:00, 106078.71 examples/s]" + } + }, + "9bba163f10a4461ca232b54d79c9b74d": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "1.2.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "9d4d315121e9440c8578a62fbe88e415": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "1.2.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "9e28f7897bf142aebd4d374559320812": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "1.2.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "9f5a040885564d41934f6c458761bf33": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "HTMLModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HTMLModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HTMLView", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_9d4d315121e9440c8578a62fbe88e415", + "placeholder": "​", + "style": "IPY_MODEL_a1f8b53e8a1d4ebd8ed0116219490877", + "value": "Dl Completed...: " + } + }, + "a1f8b53e8a1d4ebd8ed0116219490877": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "DescriptionStyleModel", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "DescriptionStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "description_width": "" + } + }, + "a585d2e5ac5240679587990dbf53dfd2": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "ProgressStyleModel", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "ProgressStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "bar_color": null, + "description_width": "" + } + }, + "a6a32befb28542228cde3d444d6411f6": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "HBoxModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HBoxModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HBoxView", + "box_style": "", + "children": [ + "IPY_MODEL_9f5a040885564d41934f6c458761bf33", + "IPY_MODEL_a7651b06cb974b52a35e34e8f96c226c", + "IPY_MODEL_ee267b7dcf05457b8e3f545df150f09f" + ], + "layout": "IPY_MODEL_55cc8f6b3b0c49ddbc4bfe526906ccbf" + } + }, + "a7651b06cb974b52a35e34e8f96c226c": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "FloatProgressModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "FloatProgressModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "ProgressView", + "bar_style": "success", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_074db709c0a14cd4acfe13ed54e92cbc", + "max": 1, + "min": 0, + "orientation": "horizontal", + "style": "IPY_MODEL_713f10e7274e4f0484d34759b5505842", + "value": 0 + } + }, + "abaa80c91f5642649996c844ceb0fcd1": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "1.2.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "b1c32e56d326473db36bcda7abca7010": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "HTMLModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HTMLModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HTMLView", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_bbce51f4b75a4f999f4b3c170083e724", + "placeholder": "​", + "style": "IPY_MODEL_25c18271a4594c1fa8c694d50dd356a7", + "value": " 1/1 [00:00<00:00, 2.47 file/s]" + } + }, + "b26304339073463b9f0ba2cce4835d13": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "ProgressStyleModel", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "ProgressStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "bar_color": null, + "description_width": "" + } + }, + "b38d4f6271234adfb41e8309a115e95b": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "HTMLModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HTMLModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HTMLView", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_30ffd5f13a524b0eabd5d2f20885ce50", + "placeholder": "​", + "style": "IPY_MODEL_5022afdb2026474081a3f54cb4c81351", + "value": " 119817/0 [00:28<00:00, 3712.60 examples/s]" + } + }, + "b459e5715d3b44eeb379108510261336": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "1.2.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "b58f7dc6368b42d0a387e47bce4ce88e": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "1.2.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "b70cac0930eb4d2da24a9f5b042f4a9e": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "FloatProgressModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "FloatProgressModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "ProgressView", + "bar_style": "info", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_7ab6c716b4a04052bf048d0ede312365", + "max": 1, + "min": 0, + "orientation": "horizontal", + "style": "IPY_MODEL_94db6867a26a4c988f549d20b3cb51f3", + "value": 1 + } + }, + "bb16a77431264209935f8e747918e430": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "FloatProgressModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "FloatProgressModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "ProgressView", + "bar_style": "success", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_15d88a4607524d07be1f3b91345243ba", + "max": 1, + "min": 0, + "orientation": "horizontal", + "style": "IPY_MODEL_a585d2e5ac5240679587990dbf53dfd2", + "value": 1 + } + }, + "bbce51f4b75a4f999f4b3c170083e724": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "1.2.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "bcd9ea70684742b6991d4e2c7556efa6": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "DescriptionStyleModel", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "DescriptionStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "description_width": "" + } + }, + "bcedd81ebcef4d9ca31eea1ae4ab795d": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "DescriptionStyleModel", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "DescriptionStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "description_width": "" + } + }, + "be1b974e61b44ecd807a77a94f6f7991": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "1.2.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "bec242f072394c4cabc6f39e43350603": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "1.2.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": "20px" + } + }, + "bfb1e75c5bc744f28544515c660a0b9b": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "1.2.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "ca245734f2f54c4e805e761d23652eca": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "1.2.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "d7ef0d4ec19749c3bee8a0be8ad2d468": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "1.2.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "dc49356d1ba943ad87b88ee6e451e7fb": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "1.2.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "e630a16615414ceeba5868d162f55a20": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "DescriptionStyleModel", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "DescriptionStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "description_width": "" + } + }, + "ee267b7dcf05457b8e3f545df150f09f": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "HTMLModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HTMLModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HTMLView", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_5ac5967b468d4af59cea0693ce9a8217", + "placeholder": "​", + "style": "IPY_MODEL_7916d209cbe04da2912830b16e5f747c", + "value": " 0/0 [00:00<?, ? url/s]" + } + }, + "ee572078162448bd89bd2c52fbe39aa7": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "HTMLModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HTMLModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HTMLView", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_be1b974e61b44ecd807a77a94f6f7991", + "placeholder": "​", + "style": "IPY_MODEL_72693d5e2c034fb9ad03a32a8eb2999f", + "value": " 0/0 [00:00<?, ? MiB/s]" + } + }, + "f3ad889117ba43b783e34a82113b325c": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "FloatProgressModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "FloatProgressModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "ProgressView", + "bar_style": "info", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_532f40fc7a1e4826b4495be24ee0f8ed", + "max": 1, + "min": 0, + "orientation": "horizontal", + "style": "IPY_MODEL_b26304339073463b9f0ba2cce4835d13", + "value": 1 + } + }, + "f91bb89f9f144f8e97f8f0c97f7d9f55": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "DescriptionStyleModel", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "DescriptionStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "description_width": "" + } + }, + "fc94257ae5094ce0b04695ad29bdf72b": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "HBoxModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HBoxModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HBoxView", + "box_style": "", + "children": [ + "IPY_MODEL_30224d6b4c274faf85dbd4d2c1892aa7", + "IPY_MODEL_295d430b24444986a46a9382c5d5f80d", + "IPY_MODEL_9a4eedfb4c6a466ba6f6f21ce76a64bb" + ], + "layout": "IPY_MODEL_9e28f7897bf142aebd4d374559320812" + } + } + } + } + }, + "nbformat": 4, + "nbformat_minor": 0 +} diff --git a/lessons/5-NLP/15-LanguageModeling/README.md b/lessons/5-NLP/15-LanguageModeling/README.md index 29023275..f269e738 100644 --- a/lessons/5-NLP/15-LanguageModeling/README.md +++ b/lessons/5-NLP/15-LanguageModeling/README.md @@ -9,21 +9,25 @@ The main idea behind language modeling is training them on unlabeled datasets in ## Training Embeddings -In our previous examples, we used pre-trained semantic embeddings, but it is interesting to see how those embeddings can be trained using either CBoW, or Skip-gram architectures. +In our previous examples, we used pre-trained semantic embeddings, but it is interesting to see how those embeddings can be trained. There are several possible ideas the can be used: + +* **N-Gram** language modeling, when we predict a token by looking at N previous tokens (N-gram) +* **Continuous Bag-of-Words** (CBoW), when we predict the middle token $W_0$ in a token sequence $W_{-N}$, ..., $W_N$. +* **Skip-gram**, where we predict a set of neighboring tokens $\{W_{-N},\dots, W_{-1}, W_1,\dots, W_N\}$ from the middle token $W_0$. ![image from paper on converting words to vectors](../14-Embeddings/images/example-algorithms-for-converting-words-to-vectors.png) > Image from [this paper](https://arxiv.org/pdf/1301.3781.pdf) -The idea underpinning CBoW involves how to predict a missing word, but to do this we take a small sliding window of text tokens. We can denote them from W-2 to W2, and train a model to predict the central word W0 from a few surrounding words. +## ✍️ Example Notebooks: Training CBoW model + +Continue your learning in the following notebooks: + +* [Training CBoW Word2Vec with TensorFlow](CBoW-TF.ipynb) ## Conclusion -TBD - -## 🚀 Challenge - -TBD +In the previous lesson we have seen that words embeddings work like magic! Now we know that training word embeddings is not a very complex task, and we should be able to train our own word embeddings for domain specific text if needed. ## [Post-lecture quiz](https://black-ground-0cc93280f.1.azurestaticapps.net/quiz/215) @@ -33,4 +37,6 @@ TBD * [Official TensorFlow tutorial on training Word2Vec model](https://www.TensorFlow.org/tutorials/text/word2vec). * Using the **gensim** framework to train most commonly used embeddings in a few lines of code is described [in this documentation](https://pytorch.org/tutorials/beginner/nlp/word_embeddings_tutorial.html). -## [Assignment: Notebooks](assignment.md) - TBD +## 🚀 [Assignment: Train Skip-Gram Model](lab/README.md) + +In the lab, we challenge you to modify the code from this lesson to train skip-gram model instead of CBoW. [Read the details](lab/README.md) diff --git a/lessons/5-NLP/15-LanguageModeling/lab/README.md b/lessons/5-NLP/15-LanguageModeling/lab/README.md new file mode 100644 index 00000000..e7d04400 --- /dev/null +++ b/lessons/5-NLP/15-LanguageModeling/lab/README.md @@ -0,0 +1,27 @@ +# Training Skip-Gram Model + +Lab Assignment from [AI for Beginners Curriculum](https://github.com/microsoft/ai-for-beginners). + +## Task + +In this lab, you we challenge you to train Word2Vec model using Skip-Gram technique. Train a network with embedding to predict neighboring words in $N$-tokens-wide Skip-Gram window. You can use the [code from this lesson](../CBoW-TF.ipynb), and slightly modify it. + +## The Dataset + +You are welcome to use any book. You can find a lot of free texts at [Project Gutenberg](https://www.gutenberg.org/), for example, here is a direct link to [Alice's Adventures in Wonderland](https://www.gutenberg.org/files/11/11-0.txt)) by Lewis Carroll. Or, you can use Shakespeare's plays, which you can get using the following code: + +```python +path_to_file = tf.keras.utils.get_file( + 'shakespeare.txt', + 'https://storage.googleapis.com/download.tensorflow.org/data/shakespeare.txt') +text = open(path_to_file, 'rb').read().decode(encoding='utf-8') +``` + +## Explore! + +If you have time and want to get deeper into the subject, try to explore several things: + +* How does embedding size affects the results? +* How does different text styles affect the result? +* Take several very different types of words and their synonyms, obtain their vector representations, apply PCA to reduce dimensions to 2, and plot them in 2D space. Do you see any patterns? + \ No newline at end of file