RomanNum3ral
/
AI-For-Beginners
mirror of https://github.com/microsoft/AI-For-Beginners.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #73 from CinnamonXI/main 

Lesson 13
This commit is contained in:
Jen Looper 2022-05-09 14:40:38 -04:00 committed by GitHub
parent 123ef6cd91 25d6deb267
commit 4354f45904
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
11 changed files with 309 additions and 36 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
etc/quiz-app/src/assets/translations/en/index.js
View File

@ -8,6 +8,8 @@ import x8 from "./lesson-8.json";
import x9 from "./lesson-9.json";
import x10 from "./lesson-10.json";
import x12 from "./lesson-12.json";
import x13 from "./lesson-13.json";
import x14 from "./lesson-14.json";
import x23 from "./lesson-23.json";
const quiz = { 0 : x1[0], 1 : x2[0], 2 : x3[0], 3 : x4[0], 4 : x5[0], 5 : x7[0], 6 : x8[0], 7 : x9[0], 8 : x10[0], 9 : x12[0], 10 : x23[0] };
const quiz = { 0 : x1[0], 1 : x2[0], 2 : x3[0], 3 : x4[0], 4 : x5[0], 5 : x7[0], 6 : x8[0], 7 : x9[0], 8 : x10[0], 9 : x12[0], 10 : x13[0], 11 : x14[0], 12 : x23[0] };
export default quiz;

115
etc/quiz-app/src/assets/translations/en/lesson-13.json Normal file
View File

@ -0,0 +1,115 @@
[
{
"title": "AI for Beginners: Quizzes",
"complete": "Congratulations, you completed the quiz!",
"error": "Sorry, try again",
"quizzes": [
{
"id": 113,
"title": "Text Representation: Pre Quiz",
"quiz": [
{
"questionText": "Each word in a Bag of Words is linked to a vector index",
"answerOptions": [
{
"answerText": "true",
"isCorrect": true
},
{
"answerText": "false",
"isCorrect": false
}
]
},
{
"questionText": "Text can be represented using _____ approaches",
"answerOptions": [
{
"answerText": "1",
"isCorrect": false
},
{
"answerText": "2",
"isCorrect": true
},
{
"answerText": "3",
"isCorrect": false
}
]
},
{
"questionText": "Character level representation represents each _____ as a number",
"answerOptions": [
{
"answerText": "letter",
"isCorrect": true
},
{
"answerText": "word",
"isCorrect": false
},
{
"answerText": "symbol",
"isCorrect": false
}
]
}
]
},
{
"id": 213,
"title": "Text Representation: Post Quiz",
"quiz": [
{
"questionText": "Word level representation represents _____ as a number",
"answerOptions": [
{
"answerText": "letter",
"isCorrect": false
},
{
"answerText": "word",
"isCorrect": true
},
{
"answerText": "symbol",
"isCorrect": false
}
]
},
{
"questionText": "N-Grams refers to _____",
"answerOptions": [
{
"answerText": "combination of n number of words and symbols",
"isCorrect": false
},
{
"answerText": "combination of n number of letters",
"isCorrect": false
},
{
"answerText": "combination of n number of Words",
"isCorrect": true
}
]
},
{
"questionText": "The main drawback of N-gram is that the vocabulary size grows fast",
"answerOptions": [
{
"answerText": "true",
"isCorrect": true
},
{
"answerText": "false",
"isCorrect": false
}
]
}
]
}
]
}
]

111
etc/quiz-app/src/assets/translations/en/lesson-14.json Normal file
View File

@ -0,0 +1,111 @@
[
{
"title": "AI for Beginners: Quizzes",
"complete": "Congratulations, you completed the quiz!",
"error": "Sorry, try again",
"quizzes": [
{
"id": 114,
"title": "Embeddings: Pre Quiz",
"quiz": [
{
"questionText": "Embedding is to represent words with _____ dimensional dense vectors",
"answerOptions": [
{
"answerText": "lower",
"isCorrect": true
},
{
"answerText": "higher",
"isCorrect": false
},
{
"answerText": "average",
"isCorrect": false
}
]
},
{
"questionText": "Word2Vec pre-trained embeddings can also be used in place of embedding layer in neural networks",
"answerOptions": [
{
"answerText": "True",
"isCorrect": true
},
{
"answerText": "False",
"isCorrect": false
}
]
},
{
"questionText": "Using embedding layer we cannot switch from bag-of-words to embedding bag",
"answerOptions": [
{
"answerText": "True",
"isCorrect": false
},
{
"answerText": "false",
"isCorrect": true
}
]
}
]
},
{
"id": 214,
"title": "Embeddings: Post Quiz",
"quiz": [
{
"questionText": "Word2Vec has _____ main architectures",
"answerOptions": [
{
"answerText": "1",
"isCorrect": false
},
{
"answerText": "2",
"isCorrect": true
},
{
"answerText": "3",
"isCorrect": false
}
]
},
{
"questionText": "Word sense disambiguation is a limitation of traditional pretrained embedding representations",
"answerOptions": [
{
"answerText": "True",
"isCorrect": true
},
{
"answerText": "False",
"isCorrect": false
}
]
},
{
"questionText": "An embedding layer takes _____ as input",
"answerOptions": [
{
"answerText": "word",
"isCorrect": true
},
{
"answerText": "symbol",
"isCorrect": false
},
{
"answerText": "number",
"isCorrect": false
}
]
}
]
}
]
}
]

51
etc/quiz-src/questions-en.txt
View File

@ -266,6 +266,57 @@ Lesson 12E Segmentation: Post Quiz
- height normalization
- weight normalization
Lesson 13B Text Representation: Pre Quiz
* Each word in a Bag of Words is linked to a vector index
+ true
- false
* Text can be represented using _____ approaches
- 1
+ 2
- 3
* Character level representation represents each _____ as a number
+ letter
- word
- symbol
Lesson 13E Text Representation: Post Quiz
* Word level representation represents _____ as a number
- letter
+ word
- symbol
* N-Grams refers to _____
- combination of n number of words and symbols
- combination of n number of letters
+ combination of n number of Words
* The main drawback of N-gram is that the vocabulary size grows fast
+ true
- false
Lesson 14B Embeddings: Pre Quiz
* Embedding is to represent words with _____ dimensional dense vectors
+ lower
- higher
- average
* Word2Vec pre-trained embeddings can also be used in place of embedding layer in neural networks
+ True
- False
* Using embedding layer we cannot switch from bag-of-words to embedding bag
- True
+ false
Lesson 14E Embeddings: Post Quiz
* Word2Vec has _____ main architectures
- 1
+ 2
- 3
* Word sense disambiguation is a limitation of traditional pretrained embedding representations
+ True
- False
* An embedding layer takes _____ as input
+ word
- symbol
- number
Lesson 23B Multi-Agent Modeling: Pre Quiz
* By modeling the behavior of simple agents, we can understand more complex behaviors of a system.
+ true

1
lessons/4-ComputerVision/12-Segmentation/README.md
View File

@ -64,4 +64,3 @@ This [wikipedia article](https://wikipedia.org/wiki/Image_segmentation) offers a
## [Assignment](lab/README.md)
In this lab, try **human body segmentation** using [Segmentation Full Body MADS Dataset](https://www.kaggle.com/datasets/tapakah68/segmentation-full-body-mads-dataset) from Kaggle.

7
lessons/5-NLP/13-TextRep/README.md
View File

@ -1,5 +1,7 @@
# Representing Text as Tensors
## [Pre-lecture quiz](https://black-ground-0cc93280f.1.azurestaticapps.net/quiz/113)
## Text Classification
Throughout the first part of this course, we will focus on **text classification** task. We will use [AG News](https://www.kaggle.com/amananandrai/ag-news-classification-dataset) Dataset, which contains news articles like the following:
@ -21,6 +23,7 @@ If we want to solve Natural Language Processing (NLP) tasks with neural networks
We understand what each letter **represents**, and how all characters come together to form the words of a sentence. However, computers by themselves do not have such an understanding, and neural network has to learn the meaning during training.
Therefore, we can use different approaches when representing text:
* **Character-level representation**, when we represent text by treating each character as a number. Given that we have *C* different characters in our text corpus, the word *Hello* would be represented by 5x*C* tensor. Each letter would correspond to a tensor column in one-hot encoding.
* **Word-level representation**, in which we create a **vocabulary** of all words in our text, and then represent words using one-hot encoding. This approach is somehow better, because each letter by itself does not have much meaning, and thus by using higher-level semantic concepts - words - we simplify the task for the neural network. However, given large dictionary size, we need to deal with high-dimensional sparse tensors.
@ -50,3 +53,7 @@ However, none of those approaches can fully take into account the semantics of t
* [Text Representation with PyTorch](TextRepresentationPyTorch.ipynb)
* [Text Representation with TensorFlow](TextRepresentationTF.ipynb)
## [Post-lecture quiz](https://black-ground-0cc93280f.1.azurestaticapps.net/quiz/213)
> ✅ Todo: conclusion, assignment, challenge, review.

5
lessons/5-NLP/13-TextRep/TextRepresentationPyTorch.ipynb
View File

@ -542,11 +542,6 @@
"\n",
"However even though TF-IDF representations provide frequency weight to different words they are unable to represent meaning or order. As the famous linguist J. R. Firth said in 1935, “The complete meaning of a word is always contextual, and no study of meaning apart from context can be taken seriously.”. We will learn later in the course how to capture contextual information from text using language modeling.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": []
}
],
"metadata": {

15
lessons/5-NLP/14-Embeddings/EmbeddingsPyTorch.ipynb
View File

@ -61,7 +61,7 @@
"\n",
"So, embedding layer would take a word as an input, and produce an output vector of specified `embedding_size`. In a sense, it is very similar to `Linear` layer, but instead of taking one-hot encoded vector, it will be able to take a word number as an input.\n",
"\n",
"By using embedding layer as a first layer in our network, we can switch from bag-or-words to **embedding bag** model, where we first convert each word in our text into corresponding embedding, and then compute some aggregate function over all those embeddings, such as `sum`, `average` or `max`. \n",
"By using embedding layer as a first layer in our network, we can switch from bag-of-words to **embedding bag** model, where we first convert each word in our text into corresponding embedding, and then compute some aggregate function over all those embeddings, such as `sum`, `average` or `max`. \n",
"\n",
"![Image showing an embedding classifier for five sequence words.](images/embedding-classifier-example.png)\n",
"\n",
@ -235,7 +235,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"Note, that unlike in all previous examples, our network now accepts two parameters: data vector and offset vector, which are of different sizes. Sililarly, our data loader also provides us with 3 values instead of 2: both text and offset vectors are provided as features. Therefore, we need to slightly adjust our training function to take care of that:"
"Note, that unlike in all previous examples, our network now accepts two parameters: data vector and offset vector, which are of different sizes. Similarly, our data loader also provides us with 3 values instead of 2: both text and offset vectors are provided as features. Therefore, we need to slightly adjust our training function to take care of that:"
]
},
{
@ -304,7 +304,7 @@
"\n",
"## Semantic Embeddings: Word2Vec\n",
"\n",
"In our previous example, the model embedding layer learnt to map words to vector representation, however, this representation did not have much semantical meaning. It would be nice to learn such vector representation, that similar words or symonims would correspond to vectors that are close to each other in terms of some vector distance (eg. euclidian distance).\n",
"In our previous example, the model embedding layer learnt to map words to vector representation, however, this representation did not have much semantical meaning. It would be nice to learn such vector representation, that similar words or synonyms would correspond to vectors that are close to each other in terms of some vector distance (eg. euclidian distance).\n",
"\n",
"To do that, we need to pre-train our embedding model on a large collection of text in a specific way. One of the first ways to train semantic embeddings is called [Word2Vec](https://en.wikipedia.org/wiki/Word2vec). It is based on two main architectures that are used to produce a distributed representation of words:\n",
"\n",
@ -517,7 +517,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"In our case we do not see huge increase in accuracy, which is likely to quite different vocalularies. \n",
"In our case we do not see huge increase in accuracy, which is likely to quite different vocabularies. \n",
"To overcome the problem of different vocabularies, we can use one of the following solutions:\n",
"* Re-train word2vec model on our vocabulary\n",
"* Load our dataset with the vocabulary from the pre-trained word2vec model. Vocabulary used to load the dataset can be specified during loading.\n",
@ -678,7 +678,7 @@
"source": [
"## Contextual Embeddings\n",
"\n",
"One key limitation of tradition pretrained embedding representaitons such as Word2Vec is the problem of word sense disambigioution. While pretrained embeddings can capture some of the meaning of words in context, every possible meaning of a word is encoded into the same embedding. This can cause problems in downstream models, since many words such as the word 'play' have different meanings depending on the context they are used in.\n",
"One key limitation of traditional pretrained embedding representations such as Word2Vec is the problem of word sense disambiguation. While pretrained embeddings can capture some of the meaning of words in context, every possible meaning of a word is encoded into the same embedding. This can cause problems in downstream models, since many words such as the word 'play' have different meanings depending on the context they are used in.\n",
"\n",
"For example word 'play' in those two different sentences have quite different meaning:\n",
"- I went to a **play** at the theature.\n",
@ -686,11 +686,6 @@
"\n",
"The pretrained embeddings above represent both of these meanings of the word 'play' in the same embedding. To overcome this limitation, we need to build embeddings based on the **language model**, which is trained on a large corpus of text, and *knows* how words can be put together in different contexts. Discussing contextual embeddings is out of scope for this tutorial, but we will come back to them when talking about language models in the next unit.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": []
}
],
"metadata": {

12
lessons/5-NLP/14-Embeddings/EmbeddingsTF.ipynb
View File

@ -36,7 +36,7 @@
"\n",
"So, an embedding layer takes a word as input, and produces an output vector of specified `embedding_size`. In a sense, it is very similar to a `Dense` layer, but instead of taking a one-hot encoded vector as input, it's able to take a word number.\n",
"\n",
"By using an embedding layer as the first layer in our network, we can switch from bag-or-words to an **embedding bag** model, where we first convert each word in our text into the corresponding embedding, and then compute some aggregate function over all those embeddings, such as `sum`, `average` or `max`. \n",
"By using an embedding layer as the first layer in our network, we can switch from bag-of-words to an **embedding bag** model, where we first convert each word in our text into the corresponding embedding, and then compute some aggregate function over all those embeddings, such as `sum`, `average` or `max`. \n",
"\n",
"![Image showing an embedding classifier for five sequence words.](images/embedding-classifier-example.png)\n",
"\n",
@ -292,16 +292,6 @@
"> **Note:** When you first create word vectors, downloading them can take some time!"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [],
"source": [
"import gensim.downloader as api\n",
"w2v = api.load('word2vec-google-news-300')"
]
},
{
"cell_type": "code",
"execution_count": 12,

15
lessons/5-NLP/14-Embeddings/README.md
View File

@ -1,12 +1,14 @@
# Embeddings
## [Pre-lecture quiz](https://black-ground-0cc93280f.1.azurestaticapps.net/quiz/114)
When training classifiers based on BoW or TF/IDF, we operated on high-dimensional bag-of-words vectors with length `vocab_size`, and we were explicitly converting from low-dimensional positional representation vectors into sparse one-hot representation. This one-hot representation is not memory-efficient, in addition, each word is treated independently from each other, i.e. one-hot encoded vectors do not express any semantic similarity between words.
The idea of **embedding** is to represent words by lower-dimensional dense vectors, which somehow reflect semantic meaning of a word. We will later discuss how to build meaningful word embeddings, but for now let's just think of embeddings as a way to lower dimensionality of a word vector.
The idea of **embedding** is to represent words by lower-dimensional dense vectors, which somehow reflect semantic meaning of a word. We will later discuss how to build meaningful word embeddings, but for now let's just think of embeddings as a way to lower dimensionality of a word vector.
So, embedding layer would take a word as an input, and produce an output vector of specified `embedding_size`. In a sense, it is very similar to `Linear` layer, but instead of taking one-hot encoded vector, it will be able to take a word number as an input, allowing us to avoid creating large one-hot-encoded vectors.
By using embedding layer as a first layer in our classifier network, we can switch from bag-or-words to **embedding bag** model, where we first convert each word in our text into corresponding embedding, and then compute some aggregate function over all those embeddings, such as `sum`, `average` or `max`.
By using embedding layer as a first layer in our classifier network, we can switch from bag-of-words to **embedding bag** model, where we first convert each word in our text into corresponding embedding, and then compute some aggregate function over all those embeddings, such as `sum`, `average` or `max`.
![Image showing an embedding classifier for five sequence words.](images/embedding-classifier-example.png)
@ -19,7 +21,7 @@ By using embedding layer as a first layer in our classifier network, we can swit
## Semantic Embeddings: Word2Vec
While embedding layer learnt to map words to vector representation, however, this representation did not necessarily have much semantical meaning. It would be nice to learn such vector representation that similar words or symonims correspond to vectors that are close to each other in terms of some vector distance (eg. Euclidean distance).
While embedding layer learnt to map words to vector representation, however, this representation did not necessarily have much semantical meaning. It would be nice to learn such vector representation that similar words or synonyms correspond to vectors that are close to each other in terms of some vector distance (eg. Euclidean distance).
To do that, we need to pre-train our embedding model on a large collection of text in a specific way. One of the first ways to train semantic embeddings is called [Word2Vec](https://en.wikipedia.org/wiki/Word2vec). It is based on two main architectures that are used to produce a distributed representation of words:
@ -39,11 +41,16 @@ Word2Vec pre-trained embeddings (as well as other similar models, such as GloVe)
One key limitation of tradition pretrained embedding representations such as Word2Vec is the problem of word sense disambiguation. While pretrained embeddings can capture some of the meaning of words in context, every possible meaning of a word is encoded into the same embedding. This can cause problems in downstream models, since many words such as the word 'play' have different meanings depending on the context they are used in.
For example word 'play' in those two different sentences have quite different meaning:
- I went to a **play** at the theature.
- John wants to **play** with his friends.
The pretrained embeddings above represent both of these meanings of the word 'play' in the same embedding. To overcome this limitation, we need to build embeddings based on the **language model**, which is trained on a large corpus of text, and *knows* how words can be put together in different contexts. Discussing contextual embeddings is out of scope for this tutorial, but we will come back to them when talking about language models later in the course.
## [Post-lecture quiz](https://black-ground-0cc93280f.1.azurestaticapps.net/quiz/214)
> ✅ Todo: Assignment, conclusions, challenge.
## References
* Paper on Word2Vec: [Efficient Estimation of Word Representations in Vector Space](https://arxiv.org/pdf/1301.3781.pdf)
* Paper on Word2Vec: [Efficient Estimation of Word Representations in Vector Space](https://arxiv.org/pdf/1301.3781.pdf)

9
lessons/5-NLP/README.md
View File

@ -1,10 +1,11 @@
# Natural Language Processing
![Summary of NLP tasks in a doodle](../sketchnotes/ai-nlp.png)
In this section, we will focus on using Neural Networks to handle tasks related to natural language processing (NLP). There are many NLP problems that we want computers to be able to solve:
* **Text classification** is typical classification problem on text sequences. Examples include classifying e-mail messages on spam vs. no-spam, or attributing news article into one of the pre-defined categories (sport, business, politics, etc.). Also, when developing chat bots, we often need to understand what a used wanted to say -- in this case we are dealing with **intent classificaton**. Often, in intent classification we need to deal with many categories.
* **Sentiment analysis** is typical regression problem, where we need to attribute a number -- sentiment -- corresponding to how positive/negative the meaning of a sentence is. More advanced version of sentiment analysis is **aspect-based sentiment analysis** (ABSA), where we attribute sentiment not the the whole sentence, but to different parts of it (aspects), eg. *In this restaurant, I liked the cuisine, but the atmosphere was awful*.
* **Text classification** is typical classification problem on text sequences. Examples include classifying e-mail messages on spam vs. no-spam, or attributing news article into one of the pre-defined categories (sport, business, politics, etc.). Also, when developing chat bots, we often need to understand what a user wanted to say -- in this case we are dealing with **intent classificaton**. Often, in intent classification we need to deal with many categories.
* **Sentiment analysis** is typical regression problem, where we need to attribute a number -- sentiment -- corresponding to how positive/negative the meaning of a sentence is. More advanced version of sentiment analysis is **aspect-based sentiment analysis** (ABSA), where we attribute sentiment not to the whole sentence, but to different parts of it (aspects), eg. *In this restaurant, I liked the cuisine, but the atmosphere was awful*.
* **Named Entity Recognition** (NER) refers to the problem of extracting certain entities from text. For example, we might need to understand that in the phrase *I need to fly to Paris tomorrow* the word *tomorrow* refers to DATE, and *Paris* is a LOCATION.
* **Keyword extraction** is similar to NER, but we need to extract words important to the meaning of the sentence automatically, without pre-training for specific entity types.
* **Text clustering** can be useful when we want to group together similar sentences, for example, similar requests in technical support conversations.
@ -19,7 +20,7 @@ Many classical NLP methods are implemented in [Natural Language Processing Toolk
In our course, we will mostly focus on using Neural Networks for NLP, and we will use NLTK where needed.
We have already learnt about using neural networks for dealing with tabular data and with images. The main difference between those types of data and text is that text is a sequence of variable length, while the input size in case of images is known in advance. While convolutional networks can extract patterns from input data, patterns in text are more complex. Eg., we can have negation being separated from the subject be arbitrary many words (eg. *I do not like organges*, vs. *I do not like those big colorful tasty oranges*), and that should still be interpreted as one pattern. Thus, to handle language we need to introduce new neural network types, such as *recurrent networks* and *transformers*.
We have already learnt about using neural networks for dealing with tabular data and with images. The main difference between those types of data and text is that text is a sequence of variable length, while the input size in case of images is known in advance. While convolutional networks can extract patterns from input data, patterns in text are more complex. Eg., we can have negation being separated from the subject be arbitrary many words (eg. *I do not like organges*, vs. *I do not like those big colorful tasty oranges*), and that should still be interpreted as one pattern. Thus, to handle language we need to introduce new neural network types, such as *recurrent networks* and *transformers*.
## Install Libraries
@ -36,7 +37,7 @@ pip install -r requirements-tf.txt
## GPU Warning
In this section, in some of the examples we will be training quite large models. It is advisable to run notebooks on GPU-enabled compute to minimize waiting time.
In this section, in some of the examples we will be training quite large models. It is advisable to run notebooks on GPU-enabled computer to minimize waiting time.
When running on GPU, you may experience situations when you run out of GPU memory. During training, the amount of GPU memory consumed depends on many factors, including minibatch size. If you experience any memory problems - you may try to minimize the minibatch size in the code.