719bf52bfa
|
||
|---|---|---|
| .. | ||
| lab | ||
| CBoW-PyTorch.ipynb | ||
| CBoW-TF.ipynb | ||
| README.md | ||
README.md
Language Modeling
Semantic embeddings, such as Word2Vec and GloVe, are essentially the first step toward language modeling—creating models that can somehow understand (or represent) the nature of language.
Pre-lecture quiz
The main idea behind language modeling is training models on unlabeled datasets in an unsupervised manner. This is crucial because there is an abundance of unlabeled text available, whereas the amount of labeled text is always limited by the effort required for labeling. Typically, language models are designed to predict missing words in text, as it is straightforward to mask a random word in a sentence and use it as a training example.
Training Embeddings
In previous examples, we used pre-trained semantic embeddings, but it’s interesting to explore how these embeddings can be trained. Several approaches can be used:
- N-Gram language modeling, where a token is predicted based on the N preceding tokens (N-gram).
- Continuous Bag-of-Words (CBoW), where the middle token
W_0in a sequenceW_{-N}, ...,W_Nis predicted. - Skip-gram, where a set of neighboring tokens {
W_{-N},\dots, W_{-1}, W_1,\dots, W_N} is predicted from the middle tokenW_0.
Image from this paper
✍️ Example Notebooks: Training CBoW model
Continue your learning with the following notebooks:
Conclusion
In the previous lesson, we saw that word embeddings work like magic! Now we understand that training word embeddings is not overly complex, and we should be able to train our own embeddings for domain-specific text when needed.
Post-lecture quiz
Review & Self Study
- Official PyTorch tutorial on Language Modeling.
- Official TensorFlow tutorial on training Word2Vec model.
- Learn how to use the gensim framework to train commonly used embeddings in just a few lines of code in this documentation.
🚀 Assignment: Train Skip-Gram Model
In the lab, your challenge is to modify the code from this lesson to train a skip-gram model instead of CBoW. Read the details