"
- },
- "metadata": {}
- }
- ],
- "execution_count": 20,
- "metadata": {}
+ ]
},
{
"cell_type": "markdown",
+ "metadata": {},
"source": [
- "One of the reasons we're not seeing higher accuracy is because some words from our dataset are missing in the pretrained GloVe vocabulary, and thus they are essentially ignored. To overcome this, we can train our own embeddings based on our dataset. \n",
- "\n",
- "\n",
- "## Training your own embeddings\n",
- "\n",
- "In our examples, we have been using pretrained semantic embeddings, but it is interesting to see how those embeddings can be trained using either CBoW, or skip-gram architectures. This exercise goes beyond this module, but those interested might want to check out this [official TensorFlow tutorial on training Word2Vec model](https://www.tensorflow.org/tutorials/text/word2vec). Also, the **gensim** framework can be used to train the most commonly used embeddings in a few lines of code, as described [in the official documentation](https://radimrehurek.com/gensim/auto_examples/tutorials/run_word2vec.html#training-your-own-model)."
- ],
- "metadata": {}
+ "One of the reasons we're not seeing higher accuracy is because some words from our dataset are missing in the pretrained GloVe vocabulary, and thus they are essentially ignored. To overcome this, we can train our own embeddings based on our dataset. "
+ ]
},
{
"cell_type": "markdown",
+ "metadata": {},
"source": [
"## Contextual embeddings\n",
"\n",
@@ -1066,30 +650,32 @@
"- John wants to **play** with his friends.\n",
"\n",
"The pretrained embeddings we talked about represent both meanings of the word 'play' in the same embedding. To overcome this limitation, we need to build embeddings based on the **language model**, which is trained on a large corpus of text, and *knows* how words can be put together in different contexts. Discussing contextual embeddings is out of scope for this tutorial, but we will come back to them when talking about language models in the next unit.\n"
- ],
- "metadata": {}
+ ]
}
],
"metadata": {
+ "interpreter": {
+ "hash": "0cb620c6d4b9f7a635928804c26cf22403d89d98d79684e4529119355ee6d5a5"
+ },
+ "kernel_info": {
+ "name": "conda-env-py37_tensorflow-py"
+ },
"kernelspec": {
- "name": "conda-env-py37_tensorflow-py",
+ "display_name": "py37_tensorflow",
"language": "python",
- "display_name": "py37_tensorflow"
+ "name": "python3"
},
"language_info": {
- "name": "python",
- "version": "3.7.9",
- "mimetype": "text/x-python",
"codemirror_mode": {
"name": "ipython",
"version": 3
},
- "pygments_lexer": "ipython3",
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
"nbconvert_exporter": "python",
- "file_extension": ".py"
- },
- "kernel_info": {
- "name": "conda-env-py37_tensorflow-py"
+ "pygments_lexer": "ipython3",
+ "version": "3.8.12"
},
"nteract": {
"version": "nteract-front-end@1.0.0"
@@ -1097,4 +683,4 @@
},
"nbformat": 4,
"nbformat_minor": 4
-}
\ No newline at end of file
+}
diff --git a/5-NLP/14-Embeddings/README.md b/5-NLP/14-Embeddings/README.md
new file mode 100644
index 00000000..e69de29b
diff --git a/5-NLP/14-Embeddings/images/embedding-classifier-example.png b/5-NLP/14-Embeddings/images/embedding-classifier-example.png
new file mode 100644
index 00000000..3958e3fa
Binary files /dev/null and b/5-NLP/14-Embeddings/images/embedding-classifier-example.png differ
diff --git a/5-NLP/14-Embeddings/images/example-algorithms-for-converting-words-to-vectors.png b/5-NLP/14-Embeddings/images/example-algorithms-for-converting-words-to-vectors.png
new file mode 100644
index 00000000..500bad2f
Binary files /dev/null and b/5-NLP/14-Embeddings/images/example-algorithms-for-converting-words-to-vectors.png differ
diff --git a/5-NLP/14-Embeddings/images/offset-sequence-representation.png b/5-NLP/14-Embeddings/images/offset-sequence-representation.png
new file mode 100644
index 00000000..2eb982e8
Binary files /dev/null and b/5-NLP/14-Embeddings/images/offset-sequence-representation.png differ
diff --git a/5-NLP/15-LanguageModeling/README.md b/5-NLP/15-LanguageModeling/README.md
new file mode 100644
index 00000000..7541e1f5
--- /dev/null
+++ b/5-NLP/15-LanguageModeling/README.md
@@ -0,0 +1,20 @@
+
+# Language Modeling
+
+Semantic embeddings, such as Word2Vec and GloVe, are in fact a first step towards **language modeling** - creating models that somehow *understand* (or *represent*) the nature of the language.
+
+The main idea behind language modeling is training them on unlabeled datesets in unsupervised manner. It is important, because we have huge amounts of unlabeled text available, while the amount of labeled text would always be limited by the amount of effort we can spend on labeling. Most often, we build language models that can **predict missing words** in the text, because it is easy to mask out a random word in text and use it as a training sample.
+
+## Training embeddings
+
+In our previous examples, we have been using pre-trained semantic embeddings, but it is interesting to see how those embeddings can be trained using either CBoW, or Skip-gram architectures.
+
+
+
+The idea of CBoW is exactly predicting a missing word, however, to do this we take a small sliding window of text tokens (we can denote them from W-2 to W2), and train a model to predict the central word W0 from few surrounding words.
+
+## More Info
+
+* [Official PyTorch tutorial on Language Modeling](https://pytorch.org/tutorials/beginner/nlp/word_embeddings_tutorial.html).
+* [Official TensorFlow tutorial on training Word2Vec model](https://www.tensorflow.org/tutorials/text/word2vec).
+* Using **gensim** framework to train most commonly used embeddings in a few lines of code is as described [in this documentation](https://pytorch.org/tutorials/beginner/nlp/word_embeddings_tutorial.html).
diff --git a/5-NLP/README.md b/5-NLP/README.md
index 5ea06aa8..fa9f7b3f 100644
--- a/5-NLP/README.md
+++ b/5-NLP/README.md
@@ -33,7 +33,23 @@ pip install -r requirements-torch.txt
pip install -r requirements-tf.txt
```
+## GPU Warning
+
+In this section, in some of the examples we will be training quite large models. It is advisable to run notebooks on GPU-enabled compute to minimize waiting time.
+
+When running on GPU, you may experience situations when you run out of GPU memory. During training, the amount of GPU memory consumed depends on many factors, including minibatch size. If you experience any memory problems - you may try to minimize the minibatch size in the code.
+
+Also, some older versions of Tensorflow do not release GPU memory correctly if we are training multiple models in one Python kernel. In order to use GPU memory cautiously, you may set tensorflow option to grow GPU memory allocation only when required. You would need to include the following code in your notebooks:
+
+```python
+physical_devices = tf.config.list_physical_devices('GPU')
+if len(physical_devices)>0:
+ tf.config.experimental.set_memory_growth(physical_devices[0], True)
+```
+
## Contents
* [Representing text as tensors](13-TextRep/README.md)
* [Word Embeddings](14-Emdeddings/README.md)
+* [Language Modeling](15-LanguageModeling/README.md)
+
diff --git a/README.md b/README.md
index 0080b20a..42d2a503 100644
--- a/README.md
+++ b/README.md
@@ -62,8 +62,8 @@ For a gentle introduction to *AI in the Cloud* topic you may consider taking [Ge
| MS Learn |
PAT |
| 13 | Text Representation. Bow/TF-IDF | Text | PyTorch | Tensorflow | |
-| 14 | Semantic word embeddings | Text | PyTorch | Tensorflow | |
-| 15 | Training your own embeddings | Text | PyTorch | Tensorflow | |
+| 14 | Semantic word embeddings. Word2Vec and GloVe | Text | PyTorch | Tensorflow | |
+| 15 | Language Modeling. Training your own embeddings | Text | PyTorch | Tensorflow | |
| 16 | Recurrent Neural Networks | Text | PyTorch | Tensorflow | |
| 17 | Generative Recurrent Networks | Text | PyTorch | Tensorflow | |
| 18 | Language Modelling. Transformers. BERT. | Text | PyTorch | Tensorflow | |
We understand what each letter **represents**, and how all characters come together to form the words of a sentence. However, computers by themselves do not have such an understanding, and neural network has to learn the meaning during training.
@@ -34,7 +34,7 @@ In some cases, we may consider using tri-grams -- combinations of three words --
When solving tasks like text classification, we need to be able to represent text by one fixed-size vector, which we will use as an input to final dense classifier. One of the simplest ways to do that is to combine all individual word representations, eg. by adding them. If we add one-hot encodings of each word, we will end up with a vector of frequencies, showing how many times each word appears inside the text. Such representation of text is called **bag of words** (BOW).
-
+