Add CBoW training

README.md

@@ -82,7 +82,7 @@ For a gentle introduction to *AI in the Cloud* topics you may consider taking th
    <td></td></tr>
 <tr><td>13</td><td>Text Representation. Bow/TF-IDF</td><td><a href="lessons/5-NLP/13-TextRep/README.md">Text</a></td><td><a href="lessons/5-NLP/13-TextRep/TextRepresentationPyTorch.ipynb">PyTorch</a></td><td><a href="lessons/5-NLP/13-TextRep/TextRepresentationTF.ipynb">TensorFlow</td><td></td></tr>
 <tr><td>14</td><td>Semantic word embeddings. Word2Vec and GloVe</td><td><a href="lessons/5-NLP/14-Embeddings/README.md">Text</td><td><a href="lessons/5-NLP/14-Embeddings/EmbeddingsPyTorch.ipynb">PyTorch</a></td><td><a href="lessons/5-NLP/14-Embeddings/EmbeddingsTF.ipynb">TensorFlow</a></td><td></td></tr>
-<tr><td>15</td><td>Language Modeling. Training your own embeddings</td><td><a href="lessons/5-NLP/15-LanguageModeling">Text</a></td><td>PyTorch</td><td>TensorFlow</td><td></td></tr>
+<tr><td>15</td><td>Language Modeling. Training your own embeddings</td><td><a href="lessons/5-NLP/15-LanguageModeling/README.md">Text</a></td><td></td><td><a href="lessons/5-NLP/15-LanguageModeling/CBoW-TF.ipynb">TensorFlow</a></td><td><a href="lessons/5-NLP/15-LanguageModeling/lab/README.md">Lab</a></td></tr>
 <tr><td>16</td><td>Recurrent Neural Networks</td><td><a href="lessons/5-NLP/16-RNN/README.md">Text</a></td><td><a href="lessons/5-NLP/16-RNN/RNNPyTorch.ipynb">PyTorch</a></td><td><a href="lessons/5-NLP/16-RNN/RNNTF.ipynb">TensorFlow</a></td><td></td></tr>
 <tr><td>17</td><td>Generative Recurrent Networks</td><td><a href="lessons/5-NLP/17-GenerativeNetworks/README.md">Text</a></td><td><a href="lessons/5-NLP/17-GenerativeNetworks/GenerativePyTorch.md">PyTorch</a></td><td><a href="lessons/5-NLP/17-GenerativeNetworks/GenerativeTF.md">TensorFlow</a></td><td><a href="lessons/5-NLP/17-GenerativeNetworks/lab/README.md">Lab</a></td></tr>
 <tr><td>18</td><td>Transformers. BERT.</td><td><a href="lessons/5-NLP/18-Transformers/README.md">Text</a></td><td><a href="lessons/5-NLP/18-Transformers/TransformersPyTorch.md">PyTorch</a></td><td><a href="lessons/5-NLP/18-Transformers/TransformersTF.md">TensorFlow</a></td><td></td></tr>

lessons/5-NLP/15-LanguageModeling/README.md

@@ -9,21 +9,25 @@ The main idea behind language modeling is training them on unlabeled datasets in
 
 ## Training Embeddings
 
-In our previous examples, we used pre-trained semantic embeddings, but it is interesting to see how those embeddings can be trained using either CBoW, or Skip-gram architectures.
+In our previous examples, we used pre-trained semantic embeddings, but it is interesting to see how those embeddings can be trained. There are several possible ideas the can be used:
+
+* **N-Gram** language modeling, when we predict a token by looking at N previous tokens (N-gram)
+* **Continuous Bag-of-Words** (CBoW), when we predict the middle token $W_0$ in a token sequence $W_{-N}$, ..., $W_N$.
+* **Skip-gram**, where we predict a set of neighboring tokens $\{W_{-N},\dots, W_{-1}, W_1,\dots, W_N\}$ from the middle token $W_0$.
 
 ![image from paper on converting words to vectors](../14-Embeddings/images/example-algorithms-for-converting-words-to-vectors.png)
 
 > Image from [this paper](https://arxiv.org/pdf/1301.3781.pdf)
 
-The idea underpinning CBoW involves how to predict a missing word, but to do this we take a small sliding window of text tokens. We can denote them from W<sub>-2</sub> to W<sub>2</sub>, and train a model to predict the central word W<sub>0</sub> from a few surrounding words.
+## ✍️ Example Notebooks: Training CBoW model
+
+Continue your learning in the following notebooks:
+
+* [Training CBoW Word2Vec with TensorFlow](CBoW-TF.ipynb)
 
 ## Conclusion
 
-TBD
-
-## 🚀 Challenge
-
-TBD
+In the previous lesson we have seen that words embeddings work like magic! Now we know that training word embeddings is not a very complex task, and we should be able to train our own word embeddings for domain specific text if needed. 
 
 ## [Post-lecture quiz](https://black-ground-0cc93280f.1.azurestaticapps.net/quiz/215)
 
@@ -33,4 +37,6 @@ TBD
 * [Official TensorFlow tutorial on training Word2Vec model](https://www.TensorFlow.org/tutorials/text/word2vec).
 * Using the **gensim** framework to train most commonly used embeddings in a few lines of code is described [in this documentation](https://pytorch.org/tutorials/beginner/nlp/word_embeddings_tutorial.html).
 
-## [Assignment: Notebooks](assignment.md) - TBD
+## 🚀 [Assignment: Train Skip-Gram Model](lab/README.md)
+
+In the lab, we challenge you to modify the code from this lesson to train skip-gram model instead of CBoW. [Read the details](lab/README.md) 

lessons/5-NLP/15-LanguageModeling/lab/README.md

@@ -0,0 +1,27 @@
+# Training Skip-Gram Model
+
+Lab Assignment from [AI for Beginners Curriculum](https://github.com/microsoft/ai-for-beginners).
+
+## Task
+
+In this lab, you we challenge you to train Word2Vec model using Skip-Gram technique. Train a network with embedding to predict neighboring words in $N$-tokens-wide Skip-Gram window. You can use the [code from this lesson](../CBoW-TF.ipynb), and slightly modify it.
+
+## The Dataset
+
+You are welcome to use any book. You can find a lot of free texts at [Project Gutenberg](https://www.gutenberg.org/), for example, here is a direct link to [Alice's Adventures in Wonderland](https://www.gutenberg.org/files/11/11-0.txt)) by Lewis Carroll. Or, you can use Shakespeare's plays, which you can get using the following code:
+
+```python
+path_to_file = tf.keras.utils.get_file(
+   'shakespeare.txt', 
+   'https://storage.googleapis.com/download.tensorflow.org/data/shakespeare.txt')
+text = open(path_to_file, 'rb').read().decode(encoding='utf-8')
+```
+
+## Explore!
+
+If you have time and want to get deeper into the subject, try to explore several things:
+
+* How does embedding size affects the results?
+* How does different text styles affect the result?
+* Take several very different types of words and their synonyms, obtain their vector representations, apply PCA to reduce dimensions to 2, and plot them in 2D space. Do you see any patterns?
+