From f4896c504933c4583b0f476641b18a473fb76316 Mon Sep 17 00:00:00 2001 From: Jen Looper Date: Thu, 12 May 2022 11:27:35 -0400 Subject: [PATCH] BOW changed to BoW, and ch 14 edits --- etc/docsify-to-pdf/static/main.md | 6 ++-- lessons/5-NLP/13-TextRep/README.md | 6 ++-- .../14-Embeddings/EmbeddingsPyTorch.ipynb | 2 +- lessons/5-NLP/14-Embeddings/README.md | 35 ++++++++++++------- lessons/5-NLP/14-Embeddings/assignment.md | 3 ++ 5 files changed, 32 insertions(+), 20 deletions(-) create mode 100644 lessons/5-NLP/14-Embeddings/assignment.md diff --git a/etc/docsify-to-pdf/static/main.md b/etc/docsify-to-pdf/static/main.md index e2f8eeb5..02b2b5e0 100644 --- a/etc/docsify-to-pdf/static/main.md +++ b/etc/docsify-to-pdf/static/main.md @@ -948,15 +948,15 @@ In some cases, we may consider using tri-grams -- combinations of three words -- ## Bag-of-Words and TF/IDF -When solving tasks like text classification, we need to be able to represent text by one fixed-size vector, which we will use as an input to final dense classifier. One of the simplest ways to do that is to combine all individual word representations, eg. by adding them. If we add one-hot encodings of each word, we will end up with a vector of frequencies, showing how many times each word appears inside the text. Such representation of text is called **bag of words** (BOW). +When solving tasks like text classification, we need to be able to represent text by one fixed-size vector, which we will use as an input to final dense classifier. One of the simplest ways to do that is to combine all individual word representations, eg. by adding them. If we add one-hot encodings of each word, we will end up with a vector of frequencies, showing how many times each word appears inside the text. Such representation of text is called **bag of words** (BoW). > Image by author -BOW essentially represents which words appear in text and in which quantities, which can indeed be a good indication of what the text is about. For example, news article on politics is likely to contains words such as *president* and *country*, while scientific publication would have something like *collider*, *discovered*, etc. Thus, word frequencies can in many cases be a good indicator of text content. +BoW essentially represents which words appear in text and in which quantities, which can indeed be a good indication of what the text is about. For example, news article on politics is likely to contains words such as *president* and *country*, while scientific publication would have something like *collider*, *discovered*, etc. Thus, word frequencies can in many cases be a good indicator of text content. -The problem with BOW is that certain common words, such as *and*, *is*, etc. appear in most of the texts, and they have highest frequencies, masking out the words that are really important. We may lower the importance of those words by taking into account the frequency at which words occur in the whole document collection. This is the main idea behind TF/IDF approach, which is covered in more detail in the notebooks below. +The problem with BoW is that certain common words, such as *and*, *is*, etc. appear in most of the texts, and they have highest frequencies, masking out the words that are really important. We may lower the importance of those words by taking into account the frequency at which words occur in the whole document collection. This is the main idea behind TF/IDF approach, which is covered in more detail in the notebooks below. However, none of those approaches can fully take into account the semantics of text. We need more powerful neural networks models, which we will discuss later in this course. diff --git a/lessons/5-NLP/13-TextRep/README.md b/lessons/5-NLP/13-TextRep/README.md index d1983acc..2a3a08f8 100644 --- a/lessons/5-NLP/13-TextRep/README.md +++ b/lessons/5-NLP/13-TextRep/README.md @@ -37,15 +37,15 @@ In some cases, we may consider using tri-grams -- combinations of three words -- ## Bag-of-Words and TF/IDF -When solving tasks like text classification, we need to be able to represent text by one fixed-size vector, which we will use as an input to final dense classifier. One of the simplest ways to do that is to combine all individual word representations, eg. by adding them. If we add one-hot encodings of each word, we will end up with a vector of frequencies, showing how many times each word appears inside the text. Such representation of text is called **bag of words** (BOW). +When solving tasks like text classification, we need to be able to represent text by one fixed-size vector, which we will use as an input to final dense classifier. One of the simplest ways to do that is to combine all individual word representations, eg. by adding them. If we add one-hot encodings of each word, we will end up with a vector of frequencies, showing how many times each word appears inside the text. Such representation of text is called **bag of words** (BoW). > Image by author -A BOW essentially represents which words appear in text and in which quantities, which can indeed be a good indication of what the text is about. For example, news article on politics is likely to contains words such as *president* and *country*, while scientific publication would have something like *collider*, *discovered*, etc. Thus, word frequencies can in many cases be a good indicator of text content. +A BoW essentially represents which words appear in text and in which quantities, which can indeed be a good indication of what the text is about. For example, news article on politics is likely to contains words such as *president* and *country*, while scientific publication would have something like *collider*, *discovered*, etc. Thus, word frequencies can in many cases be a good indicator of text content. -The problem with BOW is that certain common words, such as *and*, *is*, etc. appear in most of the texts, and they have highest frequencies, masking out the words that are really important. We may lower the importance of those words by taking into account the frequency at which words occur in the whole document collection. This is the main idea behind TF/IDF approach, which is covered in more detail in the notebooks attached to this lesson. +The problem with BoW is that certain common words, such as *and*, *is*, etc. appear in most of the texts, and they have highest frequencies, masking out the words that are really important. We may lower the importance of those words by taking into account the frequency at which words occur in the whole document collection. This is the main idea behind TF/IDF approach, which is covered in more detail in the notebooks attached to this lesson. However, none of those approaches can fully take into account the **semantics** of text. We need more powerful neural networks models to do this, which we will discuss later in this section. diff --git a/lessons/5-NLP/14-Embeddings/EmbeddingsPyTorch.ipynb b/lessons/5-NLP/14-Embeddings/EmbeddingsPyTorch.ipynb index 179c5fb9..17214e29 100644 --- a/lessons/5-NLP/14-Embeddings/EmbeddingsPyTorch.ipynb +++ b/lessons/5-NLP/14-Embeddings/EmbeddingsPyTorch.ipynb @@ -419,7 +419,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "Both CBOW and Skip-Grams are “predictive” embeddings, in that they only take local contexts into account. Word2Vec does not take advantage of global context. \n", + "Both CBoW and Skip-Grams are “predictive” embeddings, in that they only take local contexts into account. Word2Vec does not take advantage of global context. \n", "\n", "**FastText**, builds on Word2Vec by learning vector representations for each word and the charachter n-grams found within each word. The values of the representations are then averaged into one vector at each training step. While this adds a lot of additional computation to pre-training it enables word embeddings to encode sub-word information. \n", "\n", diff --git a/lessons/5-NLP/14-Embeddings/README.md b/lessons/5-NLP/14-Embeddings/README.md index 92b2c448..40536fed 100644 --- a/lessons/5-NLP/14-Embeddings/README.md +++ b/lessons/5-NLP/14-Embeddings/README.md @@ -2,28 +2,29 @@ ## [Pre-lecture quiz](https://black-ground-0cc93280f.1.azurestaticapps.net/quiz/114) -When training classifiers based on BoW or TF/IDF, we operated on high-dimensional bag-of-words vectors with length `vocab_size`, and we were explicitly converting from low-dimensional positional representation vectors into sparse one-hot representation. This one-hot representation is not memory-efficient, in addition, each word is treated independently from each other, i.e. one-hot encoded vectors do not express any semantic similarity between words. +When training classifiers based on BoW or TF/IDF, we operated on high-dimensional bag-of-words vectors with length `vocab_size`, and we were explicitly converting from low-dimensional positional representation vectors into sparse one-hot representation. This one-hot representation, however, is not memory-efficient. In addition, each word is treated independently from each other, i.e. one-hot encoded vectors do not express any semantic similarity between words. -The idea of **embedding** is to represent words by lower-dimensional dense vectors, which somehow reflect semantic meaning of a word. We will later discuss how to build meaningful word embeddings, but for now let's just think of embeddings as a way to lower dimensionality of a word vector. +The idea of **embedding** is to represent words by lower-dimensional dense vectors, which somehow reflect the semantic meaning of a word. We will later discuss how to build meaningful word embeddings, but for now let's just think of embeddings as a way to lower dimensionality of a word vector. -So, embedding layer would take a word as an input, and produce an output vector of specified `embedding_size`. In a sense, it is very similar to `Linear` layer, but instead of taking one-hot encoded vector, it will be able to take a word number as an input, allowing us to avoid creating large one-hot-encoded vectors. +So, the embedding layer would take a word as an input, and produce an output vector of specified `embedding_size`. In a sense, it is very similar to a `Linear` layer, but instead of taking a one-hot encoded vector, it will be able to take a word number as an input, allowing us to avoid creating large one-hot-encoded vectors. -By using embedding layer as a first layer in our classifier network, we can switch from bag-of-words to **embedding bag** model, where we first convert each word in our text into corresponding embedding, and then compute some aggregate function over all those embeddings, such as `sum`, `average` or `max`. +By using an embedding layer as a first layer in our classifier network, we can switch from a bag-of-words to **embedding bag** model, where we first convert each word in our text into corresponding embedding, and then compute some aggregate function over all those embeddings, such as `sum`, `average` or `max`. ![Image showing an embedding classifier for five sequence words.](images/embedding-classifier-example.png) > Image by author -## Continue in Notebooks +## ✍️ Exercises: Embeddings +Continue your learning in the following notebooks: * [Embeddings with PyTorch](EmbeddingsPyTorch.ipynb) * [Embeddings TensorFlow](EmbeddingsTF.ipynb) ## Semantic Embeddings: Word2Vec -While embedding layer learned to map words to vector representation, however, this representation did not necessarily have much semantical meaning. It would be nice to learn such vector representation that similar words or synonyms correspond to vectors that are close to each other in terms of some vector distance (eg. Euclidean distance). +While the embedding layer learned to map words to vector representation, however, this representation did not necessarily have much semantical meaning. It would be nice to learn a vector representation such that similar words or synonyms correspond to vectors that are close to each other in terms of some vector distance (eg. Euclidean distance). -To do that, we need to pre-train our embedding model on a large collection of text in a specific way. One of the first ways to train semantic embeddings is called [Word2Vec](https://en.wikipedia.org/wiki/Word2vec). It is based on two main architectures that are used to produce a distributed representation of words: +To do that, we need to pre-train our embedding model on a large collection of text in a specific way. One way to train semantic embeddings is called [Word2Vec](https://en.wikipedia.org/wiki/Word2vec). It is based on two main architectures that are used to produce a distributed representation of words: - **Continuous bag-of-words** (CBoW) — in this architecture, we train the model to predict a word from surrounding context. Given the ngram $(W_{-2},W_{-1},W_0,W_1,W_2)$, the goal of the model is to predict $W_0$ from $(W_{-2},W_{-1},W_1,W_2)$. - **Continuous skip-gram** is opposite to CBoW. The model uses surrounding window of context words to predict the current word. @@ -34,23 +35,31 @@ CBoW is faster, while skip-gram is slower, but does a better job of representing > Image from [this paper](https://arxiv.org/pdf/1301.3781.pdf) -Word2Vec pre-trained embeddings (as well as other similar models, such as GloVe) can also be used in place of embedding layer in neural networks. However, we need to deal with vocabularies, because the vocabulary used to pre-train Word2Vec/GloVe is likely to differ from the vocabulary in our text corpus. Have a look into Notebooks to see how this problem can be resolved. +Word2Vec pre-trained embeddings (as well as other similar models, such as GloVe) can also be used in place of embedding layer in neural networks. However, we need to deal with vocabularies, because the vocabulary used to pre-train Word2Vec/GloVe is likely to differ from the vocabulary in our text corpus. Have a look into the above Notebooks to see how this problem can be resolved. ## Contextual Embeddings -One key limitation of tradition pretrained embedding representations such as Word2Vec is the problem of word sense disambiguation. While pretrained embeddings can capture some of the meaning of words in context, every possible meaning of a word is encoded into the same embedding. This can cause problems in downstream models, since many words such as the word 'play' have different meanings depending on the context they are used in. +One key limitation of traditional pretrained embedding representations such as Word2Vec is the problem of word sense disambiguation. While pretrained embeddings can capture some of the meaning of words in context, every possible meaning of a word is encoded into the same embedding. This can cause problems in downstream models, since many words such as the word 'play' have different meanings depending on the context they are used in. For example word 'play' in those two different sentences have quite different meaning: -- I went to a **play** at the theature. +- I went to a **play** at the theatre. - John wants to **play** with his friends. The pretrained embeddings above represent both of these meanings of the word 'play' in the same embedding. To overcome this limitation, we need to build embeddings based on the **language model**, which is trained on a large corpus of text, and *knows* how words can be put together in different contexts. Discussing contextual embeddings is out of scope for this tutorial, but we will come back to them when talking about language models later in the course. +## Conclusion + +In this lesson, you discovered how to build and use embedding layers in TensorFlow and Pytorch to better reflect the semantic meanings of words. + +## 🚀 Challenge + +Word2Vec has been used for some interesting applications, including generating song lyrics and poetry. Take a look at [this article](https://www.politetype.com/blog/word2vec-color-poems) which walks through how the author used Word2Vec to generate poetry. Watch [this video by Dan Shiffmann](https://www.youtube.com/watch?v=LSS_bos_TPI&ab_channel=TheCodingTrain) as well to discover a different explanation of this technique. Then try to apply these techniques to your own text corpus, perhaps sourced from Kaggle. + ## [Post-lecture quiz](https://black-ground-0cc93280f.1.azurestaticapps.net/quiz/214) -> ✅ Todo: Assignment, conclusions, challenge. +## Review & Self Study -## References +Read through this paper on Word2Vec: [Efficient Estimation of Word Representations in Vector Space](https://arxiv.org/pdf/1301.3781.pdf) -* Paper on Word2Vec: [Efficient Estimation of Word Representations in Vector Space](https://arxiv.org/pdf/1301.3781.pdf) +## [Assignment: Notebooks](assignment.md) diff --git a/lessons/5-NLP/14-Embeddings/assignment.md b/lessons/5-NLP/14-Embeddings/assignment.md new file mode 100644 index 00000000..95b66f80 --- /dev/null +++ b/lessons/5-NLP/14-Embeddings/assignment.md @@ -0,0 +1,3 @@ +# Assignment: Notebooks + +Using the notebooks associated to this lesson (either the PyTorch or the TensorFlow version), rerun them using your own dataset, perhaps one from Kaggle, used with attribution. Rewrite the notebook to underline your own findings. Try a different kind of dataset and document your findings, using text such as [these Beatles lyrics](https://www.kaggle.com/datasets/jenlooper/beatles-lyrics). \ No newline at end of file