Add embeddings
This commit is contained in:
Dmitri Soshnikov
parent
005d7503b4
commit
8296947c89
|
|
@ -15,6 +15,7 @@
|
|||
|
||||
.ipynb_checkpoints/
|
||||
data/
|
||||
.vector_cache/
|
||||
|
||||
# Mono auto generated files
|
||||
mono_crash.*
|
||||
|
|
|
|||
|
|
@ -14,7 +14,7 @@ Our goal would be to classify the news item into one of the categories based on
|
|||
|
||||
If we want to solve Natural Language Processing (NLP) tasks with neural networks, we need some way to represent text as tensors. Computers already represent textual characters as numbers that map to fonts on your screen using encodings such as ASCII or UTF-8.
|
||||
|
||||
|
||||
<img alt="Image showing diagram mapping a character to an ASCII and binary representation" src="images/ascii-character-map.png" width="50%"/>
|
||||
|
||||
We understand what each letter **represents**, and how all characters come together to form the words of a sentence. However, computers by themselves do not have such an understanding, and neural network has to learn the meaning during training.
|
||||
|
||||
|
|
@ -34,7 +34,7 @@ In some cases, we may consider using tri-grams -- combinations of three words --
|
|||
|
||||
When solving tasks like text classification, we need to be able to represent text by one fixed-size vector, which we will use as an input to final dense classifier. One of the simplest ways to do that is to combine all individual word representations, eg. by adding them. If we add one-hot encodings of each word, we will end up with a vector of frequencies, showing how many times each word appears inside the text. Such representation of text is called **bag of words** (BOW).
|
||||
|
||||
<img src="images/bow.png" width="30%"/>
|
||||
<img src="images/bow.png" width="90%"/>
|
||||
|
||||
BOW essentially represents which words appear in text and in which quantities, which can indeed be a good indication of what the text is about. For example, news article on politics is likely to contains words such as *president* and *country*, while scientific publication would have something like *collider*, *discovered*, etc. Thus, word frequencies can in many cases be a good indicator of text content.
|
||||
|
||||
|
|
|
|||
|
|
@ -8,7 +8,7 @@
|
|||
"\n",
|
||||
"As we have mentioned, we will focus on simple text classification task based on **AG_NEWS** dataset, which is to classify news headlines into one of 4 categories: World, Sports, Business and Sci/Tech.\n",
|
||||
"\n",
|
||||
"### The Dataset\n",
|
||||
"## The Dataset\n",
|
||||
"\n",
|
||||
"This dataset is built into [`torchtext`](https://github.com/pytorch/text) module, so we can easily access it."
|
||||
]
|
||||
|
|
@ -116,7 +116,7 @@
|
|||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Tokenization\n",
|
||||
"## Tokenization\n",
|
||||
"\n",
|
||||
"Now we need to convert text into **numbers** that can be represented as tensors. If we want word-level representation, we need to do two things:\n",
|
||||
"* use **tokenizer** to split text into **tokens**\n",
|
||||
|
|
@ -200,7 +200,7 @@
|
|||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Bag of Words text representation\n",
|
||||
"## Bag of Words text representation\n",
|
||||
"\n",
|
||||
"Because words represent meaning, sometimes we can figure out the meaning of a text by just looking at the individual words, regardless of their order in the sentence. For example, when classifying news, words like *weather*, *snow* are likely to indicate *weather forecast*, while words like *stocks*, *dollar* would count towards *financial news*.\n",
|
||||
"\n",
|
||||
|
|
@ -285,7 +285,7 @@
|
|||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Training BoW classifier\n",
|
||||
"## Training BoW classifier\n",
|
||||
"\n",
|
||||
"Now that we have learned how to build Bag-of-Words representation of our text, let's train a classifier on top of it. First, we need to convert our dataset for training in such a way, that all positional vector representations are converted to bag-of-words representation. This can be achieved by passing `bowify` function as `collate_fn` parameter to standard torch `DataLoader`:"
|
||||
]
|
||||
|
|
@ -396,7 +396,7 @@
|
|||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### BiGrams, TriGrams and N-Grams\n",
|
||||
"## BiGrams, TriGrams and N-Grams\n",
|
||||
"\n",
|
||||
"One limitation of a bag of words approach is that some words are part of multi word expressions, for example, the word 'hot dog' has a completely different meaning than the words 'hot' and 'dog' in other contexts. If we represent words 'hot` and 'dog' always by the same vectors, it can confuse our model.\n",
|
||||
"\n",
|
||||
|
|
@ -488,7 +488,7 @@
|
|||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Term Frequency Inverse Document Frequency TF-IDF\n",
|
||||
"## Term Frequency Inverse Document Frequency TF-IDF\n",
|
||||
"\n",
|
||||
"In BoW representation, word occurrences are evenly weighted, regardless of the word itself. However, it is clear that frequent words, such as *a*, *in*, etc. are much less important for the classification, than specialized terms. In fact, in most NLP tasks some words are more relevant than others.\n",
|
||||
"\n",
|
||||
|
|
@ -538,10 +538,15 @@
|
|||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Conclusion \n",
|
||||
"## Conclusion \n",
|
||||
"\n",
|
||||
"However even though TF-IDF representations provide frequency weight to different words they are unable to represent meaning or order. As the famous linguist J. R. Firth said in 1935, “The complete meaning of a word is always contextual, and no study of meaning apart from context can be taken seriously.”. We will learn later in the course how to capture contextual information from text using language modeling.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": []
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
|
|
|
|||
|
|
@ -8,59 +8,16 @@
|
|||
"\n",
|
||||
"In this module, we will start with a simple text classification task based on the **[AG_NEWS](http://www.di.unipi.it/~gulli/AG_corpus_of_news_articles.html)** dataset: we'll classify news headlines into one of 4 categories: World, Sports, Business and Sci/Tech. \n",
|
||||
"\n",
|
||||
"### The Dataset\n",
|
||||
"## The Dataset\n",
|
||||
"\n",
|
||||
"To load the dataset, we will use the **[TensorFlow Datasets](https://www.tensorflow.org/datasets)** API."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"execution_count": 1,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"\u001b[1mDownloading and preparing dataset Unknown size (download: Unknown size, generated: Unknown size, total: Unknown size) to C:\\Users\\dmitryso\\tensorflow_datasets\\ag_news_subset\\1.0.0...\u001b[0m\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "stderr",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Dl Completed...: 0 url [00:00, ? url/s]\n",
|
||||
"Dl Completed...: 0%| | 0/1 [00:00<?, ? url/s]\n",
|
||||
"Dl Completed...: 0%| | 0/1 [00:32<?, ? url/s]\n",
|
||||
"Dl Completed...: 0%| | 0/1 [00:32<?, ? url/s]\n",
|
||||
"Dl Completed...: 0%| | 0/1 [00:32<?, ? url/s]\n",
|
||||
"Dl Completed...: 0%| | 0/1 [00:32<?, ? url/s]\n",
|
||||
"Dl Completed...: 0%| | 0/1 [00:32<?, ? url/s]\n",
|
||||
"Dl Completed...: 0%| | 0/1 [00:32<?, ? url/s]\n",
|
||||
"Dl Completed...: 0%| | 0/1 [00:32<?, ? url/s]\n",
|
||||
"Dl Completed...: 0%| | 0/1 [00:32<?, ? url/s]\n",
|
||||
"Dl Completed...: 0%| | 0/1 [00:33<?, ? url/s]\n",
|
||||
"Dl Completed...: 0%| | 0/1 [00:33<?, ? url/s]\n",
|
||||
"Dl Completed...: 0%| | 0/1 [00:33<?, ? url/s]\n",
|
||||
"Dl Completed...: 0%| | 0/1 [00:33<?, ? url/s]\n",
|
||||
"Dl Completed...: 100%|██████████| 1/1 [00:33<00:00, 33.33s/ url]\n",
|
||||
"Dl Completed...: 100%|██████████| 1/1 [00:33<00:00, 33.33s/ url]\n",
|
||||
"\u001b[A\n",
|
||||
"Dl Completed...: 100%|██████████| 1/1 [00:34<00:00, 33.33s/ url]\n",
|
||||
"Extraction completed...: 100%|██████████| 1/1 [00:34<00:00, 34.43s/ file]\n",
|
||||
"Dl Size...: 100%|██████████| 11/11 [00:34<00:00, 3.13s/ MiB]\n",
|
||||
"Dl Completed...: 100%|██████████| 1/1 [00:34<00:00, 34.45s/ url]\n",
|
||||
" \r"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"\u001b[1mDataset ag_news_subset downloaded and prepared to C:\\Users\\dmitryso\\tensorflow_datasets\\ag_news_subset\\1.0.0. Subsequent calls will reuse this data.\u001b[0m\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import tensorflow as tf\n",
|
||||
"from tensorflow import keras\n",
|
||||
|
|
@ -84,15 +41,15 @@
|
|||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"execution_count": 3,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"120000\n",
|
||||
"7600\n"
|
||||
"Length of train dataset = 120000\n",
|
||||
"Length of test dataset = 7600\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
|
|
@ -113,7 +70,7 @@
|
|||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
|
|
@ -156,7 +113,7 @@
|
|||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 16,
|
||||
"execution_count": 5,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
|
|
@ -176,15 +133,15 @@
|
|||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 17,
|
||||
"execution_count": 6,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"['', '[UNK]', 'the', 'to', 'a', 'of', 'in', 'and', 'on', 'for']\n",
|
||||
"Length of vocabulary: 50000\n"
|
||||
"['', '[UNK]', 'the', 'to', 'a', 'in', 'of', 'and', 'on', 'for']\n",
|
||||
"Length of vocabulary: 5335\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
|
|
@ -199,21 +156,21 @@
|
|||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Using the tokenizer, we can easily encode any text into a set of numbers:"
|
||||
"Using the vectorizer, we can easily encode any text into a set of numbers:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 18,
|
||||
"execution_count": 7,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"<tf.Tensor: shape=(7,), dtype=int64, numpy=array([ 372, 2297, 3, 312, 12, 1293, 2314])>"
|
||||
"<tf.Tensor: shape=(7,), dtype=int64, numpy=array([ 112, 3695, 3, 304, 11, 1041, 1], dtype=int64)>"
|
||||
]
|
||||
},
|
||||
"execution_count": 18,
|
||||
"execution_count": 7,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
|
|
@ -241,16 +198,16 @@
|
|||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 19,
|
||||
"execution_count": 8,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"array([[1, 1, 0, 2, 0, 0, 0, 0, 0]])"
|
||||
"array([[1, 1, 0, 2, 0, 0, 0, 0, 0]], dtype=int64)"
|
||||
]
|
||||
},
|
||||
"execution_count": 19,
|
||||
"execution_count": 8,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
|
|
@ -264,8 +221,7 @@
|
|||
" 'Its hot outside.',\n",
|
||||
" ]\n",
|
||||
"sc_vectorizer.fit_transform(corpus)\n",
|
||||
"sc_vectorizer.transform(['My dog likes hot dogs on a hot day.']).toarray()\n",
|
||||
"\n"
|
||||
"sc_vectorizer.transform(['My dog likes hot dogs on a hot day.']).toarray()"
|
||||
]
|
||||
},
|
||||
{
|
||||
|
|
@ -277,16 +233,16 @@
|
|||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 20,
|
||||
"execution_count": 9,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"array([0., 0., 0., ..., 0., 0., 0.], dtype=float32)"
|
||||
"array([0., 5., 0., ..., 0., 0., 0.], dtype=float32)"
|
||||
]
|
||||
},
|
||||
"execution_count": 20,
|
||||
"execution_count": 9,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
|
|
@ -316,7 +272,7 @@
|
|||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 21,
|
||||
"execution_count": 11,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
|
|
@ -335,23 +291,23 @@
|
|||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 22,
|
||||
"execution_count": 12,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"938/938 [==============================] - 88s 94ms/step - loss: 0.5466 - acc: 0.8759 - val_loss: 0.3682 - val_acc: 0.8950\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n"
|
||||
"938/938 [==============================] - 66s 70ms/step - loss: 0.6144 - acc: 0.8427 - val_loss: 0.4416 - val_acc: 0.8697\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"<tensorflow.python.keras.callbacks.History at 0x7fb00217a810>"
|
||||
"<keras.callbacks.History at 0x20c70a947f0>"
|
||||
]
|
||||
},
|
||||
"execution_count": 22,
|
||||
"execution_count": 12,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
|
|
@ -379,41 +335,44 @@
|
|||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 23,
|
||||
"execution_count": 13,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Model: \"functional_1\"\n",
|
||||
"Model: \"model\"\n",
|
||||
"_________________________________________________________________\n",
|
||||
"Layer (type) Output Shape Param # \n",
|
||||
" Layer (type) Output Shape Param # \n",
|
||||
"=================================================================\n",
|
||||
"input_1 (InputLayer) [(None, 1)] 0 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"text_vectorization_6 (TextVe (None, None) 0 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"tf_op_layer_OneHot (TensorFl [(None, None, 50000)] 0 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"tf_op_layer_Sum (TensorFlowO [(None, 50000)] 0 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dense_1 (Dense) (None, 4) 200004 \n",
|
||||
" input_1 (InputLayer) [(None, 1)] 0 \n",
|
||||
" \n",
|
||||
" text_vectorization (TextVec (None, None) 0 \n",
|
||||
" torization) \n",
|
||||
" \n",
|
||||
" tf.one_hot (TFOpLambda) (None, None, 5335) 0 \n",
|
||||
" \n",
|
||||
" tf.math.reduce_sum (TFOpLam (None, 5335) 0 \n",
|
||||
" bda) \n",
|
||||
" \n",
|
||||
" dense_2 (Dense) (None, 4) 21344 \n",
|
||||
" \n",
|
||||
"=================================================================\n",
|
||||
"Total params: 200,004\n",
|
||||
"Trainable params: 200,004\n",
|
||||
"Total params: 21,344\n",
|
||||
"Trainable params: 21,344\n",
|
||||
"Non-trainable params: 0\n",
|
||||
"_________________________________________________________________\n",
|
||||
"938/938 [==============================] - 79s 84ms/step - loss: 0.5221 - acc: 0.8804 - val_loss: 0.3447 - val_acc: 0.9024\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n"
|
||||
"938/938 [==============================] - 73s 77ms/step - loss: 0.6057 - acc: 0.8414 - val_loss: 0.4202 - val_acc: 0.8736\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"<tensorflow.python.keras.callbacks.History at 0x7fb003184d10>"
|
||||
"<keras.callbacks.History at 0x20c721521f0>"
|
||||
]
|
||||
},
|
||||
"execution_count": 23,
|
||||
"execution_count": 13,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
|
|
@ -451,7 +410,7 @@
|
|||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 24,
|
||||
"execution_count": 14,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
|
|
@ -465,10 +424,11 @@
|
|||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"array([[1, 0, 1, 0, 2, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]])"
|
||||
"array([[1, 0, 1, 0, 2, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]],\n",
|
||||
" dtype=int64)"
|
||||
]
|
||||
},
|
||||
"execution_count": 24,
|
||||
"execution_count": 14,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
|
|
@ -507,7 +467,7 @@
|
|||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 25,
|
||||
"execution_count": 15,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
|
|
@ -515,16 +475,16 @@
|
|||
"output_type": "stream",
|
||||
"text": [
|
||||
"Training vectorizer\n",
|
||||
"938/938 [==============================] - 10s 11ms/step - loss: 0.5207 - acc: 0.8826 - val_loss: 0.3430 - val_acc: 0.9051\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n"
|
||||
"938/938 [==============================] - 7s 7ms/step - loss: 0.5929 - acc: 0.8486 - val_loss: 0.4168 - val_acc: 0.8772\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"<tensorflow.python.keras.callbacks.History at 0x7fb002c0b290>"
|
||||
"<keras.callbacks.History at 0x20c725217c0>"
|
||||
]
|
||||
},
|
||||
"execution_count": 25,
|
||||
"execution_count": 15,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
|
|
@ -566,7 +526,7 @@
|
|||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 20,
|
||||
"execution_count": 16,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
|
|
@ -578,7 +538,7 @@
|
|||
" 0. ]])"
|
||||
]
|
||||
},
|
||||
"execution_count": 20,
|
||||
"execution_count": 16,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
|
|
@ -599,7 +559,7 @@
|
|||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 21,
|
||||
"execution_count": 17,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
|
|
@ -607,16 +567,16 @@
|
|||
"output_type": "stream",
|
||||
"text": [
|
||||
"Training vectorizer\n",
|
||||
"938/938 [==============================] - 94s 101ms/step - loss: 0.3203 - acc: 0.9039 - val_loss: 0.2542 - val_acc: 0.9186\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n"
|
||||
"938/938 [==============================] - 12s 12ms/step - loss: 0.4197 - acc: 0.8662 - val_loss: 0.3432 - val_acc: 0.8849\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"<tensorflow.python.keras.callbacks.History at 0x7f78f402e5d0>"
|
||||
"<keras.callbacks.History at 0x20c729dfd30>"
|
||||
]
|
||||
},
|
||||
"execution_count": 21,
|
||||
"execution_count": 17,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
|
|
@ -638,7 +598,7 @@
|
|||
"source": [
|
||||
"## Conclusion \n",
|
||||
"\n",
|
||||
"Even though TF-IDF representations provide frequency weights to different words, they are unable to represent meaning or order. As the famous linguist J. R. Firth said in 1935, \"The complete meaning of a word is always contextual, and no study of meaning apart from context can be taken seriously.\" We will learn how to capture contextual information from text using language modeling in a later unit."
|
||||
"Even though TF-IDF representations provide frequency weights to different words, they are unable to represent meaning or order. As the famous linguist J. R. Firth said in 1935, \"The complete meaning of a word is always contextual, and no study of meaning apart from context can be taken seriously.\" We will learn how to capture contextual information from text using language modeling later in the course."
|
||||
]
|
||||
}
|
||||
],
|
||||
|
|
|
|||
|
|
@ -8,22 +8,9 @@
|
|||
"\n",
|
||||
"In our previous example, we operated on high-dimensional bag-of-words vectors with length `vocab_size`, and we were explicitly converting from low-dimensional positional representation vectors into sparse one-hot representation. This one-hot representation is not memory-efficient, in addition, each word is treated independently from each other, i.e. one-hot encoded vectors do not express any semantic similarity between words.\n",
|
||||
"\n",
|
||||
"In this unit, we will continue exploring **News AG** dataset. To begin, let's load the data and get some definitions from the previous unit.\n"
|
||||
"In this unit, we will continue exploring **News AG** dataset. To begin, let's load the data and get some definitions from the previous notebook.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"metadata": {
|
||||
"jupyter": {
|
||||
"outputs_hidden": true
|
||||
}
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"!wget -q https://raw.githubusercontent.com/MicrosoftDocs/pytorchfundamentals/main/nlp-pytorch/torchnlp.py"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
|
|
@ -33,7 +20,21 @@
|
|||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Loading dataset...\n",
|
||||
"Loading dataset...\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "stderr",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"d:\\WORK\\ai-for-beginners\\5-NLP\\14-Embeddings\\data\\train.csv: 29.5MB [00:01, 18.8MB/s] \n",
|
||||
"d:\\WORK\\ai-for-beginners\\5-NLP\\14-Embeddings\\data\\test.csv: 1.86MB [00:00, 11.2MB/s] \n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Building vocab...\n",
|
||||
"Vocab size = 95812\n"
|
||||
]
|
||||
|
|
@ -54,7 +55,7 @@
|
|||
"metadata": {},
|
||||
"source": [
|
||||
"\n",
|
||||
"### What is embedding?\n",
|
||||
"## What is embedding?\n",
|
||||
"\n",
|
||||
"The idea of **embedding** is to represent words by lower-dimensional dense vectors, which somehow reflect semantic meaning of a word. We will later discuss how to build meaningful word embeddings, but for now let's just think of embeddings as a way to lower dimensionality of a word vector. \n",
|
||||
"\n",
|
||||
|
|
@ -62,7 +63,7 @@
|
|||
"\n",
|
||||
"By using embedding layer as a first layer in our network, we can switch from bag-or-words to **embedding bag** model, where we first convert each word in our text into corresponding embedding, and then compute some aggregate function over all those embeddings, such as `sum`, `average` or `max`. \n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"Our classifier neural network will start with embedding layer, then aggregation layer, and linear classifier on top of it:"
|
||||
]
|
||||
|
|
@ -136,19 +137,19 @@
|
|||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"3200: acc=0.6428125\n",
|
||||
"6400: acc=0.68453125\n",
|
||||
"9600: acc=0.7123958333333333\n",
|
||||
"12800: acc=0.725703125\n",
|
||||
"16000: acc=0.7365625\n",
|
||||
"19200: acc=0.7464583333333333\n",
|
||||
"22400: acc=0.7548214285714285\n"
|
||||
"3200: acc=0.6415625\n",
|
||||
"6400: acc=0.6865625\n",
|
||||
"9600: acc=0.7103125\n",
|
||||
"12800: acc=0.726953125\n",
|
||||
"16000: acc=0.739375\n",
|
||||
"19200: acc=0.75046875\n",
|
||||
"22400: acc=0.7572321428571429\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"(0.9526769402541186, 0.7595969289827256)"
|
||||
"(0.889799795315499, 0.7623160588611644)"
|
||||
]
|
||||
},
|
||||
"execution_count": 4,
|
||||
|
|
@ -176,7 +177,7 @@
|
|||
"\n",
|
||||
"In the previous architecture, we needed to pad all sequences to the same length in order to fit them into a minibatch. This is not the most efficient way to represent variable length sequences - another apporach would be to use **offset** vector, which would hold offsets of all sequences stored in one large vector.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"> **Note**: On the picture above, we show a sequence of characters, but in our example we are working with sequences of words. However, the general principle of representing sequences with offset vector remains the same.\n",
|
||||
"\n",
|
||||
|
|
@ -246,19 +247,19 @@
|
|||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"3200: acc=0.6334375\n",
|
||||
"6400: acc=0.68234375\n",
|
||||
"9600: acc=0.7072916666666667\n",
|
||||
"12800: acc=0.72375\n",
|
||||
"16000: acc=0.73575\n",
|
||||
"19200: acc=0.743125\n",
|
||||
"22400: acc=0.7497767857142857\n"
|
||||
"3200: acc=0.6153125\n",
|
||||
"6400: acc=0.6615625\n",
|
||||
"9600: acc=0.6932291666666667\n",
|
||||
"12800: acc=0.715078125\n",
|
||||
"16000: acc=0.7270625\n",
|
||||
"19200: acc=0.7382291666666667\n",
|
||||
"22400: acc=0.7486160714285715\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"(23.37446267194498, 0.754118682021753)"
|
||||
"(22.771553103007037, 0.7551983365323096)"
|
||||
]
|
||||
},
|
||||
"execution_count": 7,
|
||||
|
|
@ -338,13 +339,13 @@
|
|||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"neuronal -> 0.780479907989502\n",
|
||||
"neuronal -> 0.7804799675941467\n",
|
||||
"neurons -> 0.7326500415802002\n",
|
||||
"neural_circuits -> 0.7252851128578186\n",
|
||||
"neural_circuits -> 0.7252851724624634\n",
|
||||
"neuron -> 0.7174385190010071\n",
|
||||
"cortical -> 0.6941086053848267\n",
|
||||
"brain_circuitry -> 0.6923245787620544\n",
|
||||
"synaptic -> 0.6699119210243225\n",
|
||||
"cortical -> 0.6941086649894714\n",
|
||||
"brain_circuitry -> 0.6923246383666992\n",
|
||||
"synaptic -> 0.6699118614196777\n",
|
||||
"neural_circuitry -> 0.6638563275337219\n",
|
||||
"neurochemical -> 0.6555314064025879\n",
|
||||
"neuronal_activity -> 0.6531826257705688\n"
|
||||
|
|
@ -360,7 +361,7 @@
|
|||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"We can also extract vector embeddings from the word, to be used in training classification model (we only show first 20 components of the vector for clarity):"
|
||||
"We can also compute vector embeddings from the word, to be used in training classification model (we only show first 20 components of the vector for clarity):"
|
||||
]
|
||||
},
|
||||
{
|
||||
|
|
@ -396,7 +397,7 @@
|
|||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 11,
|
||||
"execution_count": 10,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
|
|
@ -405,7 +406,7 @@
|
|||
"('queen', 0.7118192911148071)"
|
||||
]
|
||||
},
|
||||
"execution_count": 11,
|
||||
"execution_count": 10,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
|
|
@ -424,7 +425,7 @@
|
|||
"\n",
|
||||
"Another method, **GloVe**, leverages the idea of co-occurence matrix, uses neural methods to decompose co-occurrence matrix into more expressive and non linear word vectors.\n",
|
||||
"\n",
|
||||
"You can play with the example by changing embeddings to FastText and GloVe, since gensim supports "
|
||||
"You can play with the example by changing embeddings to FastText and GloVe, since gensim supports several different word embedding models."
|
||||
]
|
||||
},
|
||||
{
|
||||
|
|
@ -438,7 +439,7 @@
|
|||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 12,
|
||||
"execution_count": 11,
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
|
|
@ -481,29 +482,29 @@
|
|||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 13,
|
||||
"execution_count": 12,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"3200: acc=0.63875\n",
|
||||
"6400: acc=0.693125\n",
|
||||
"9600: acc=0.7176041666666667\n",
|
||||
"12800: acc=0.7321875\n",
|
||||
"16000: acc=0.7454375\n",
|
||||
"19200: acc=0.7559375\n",
|
||||
"22400: acc=0.7631696428571428\n"
|
||||
"3200: acc=0.6359375\n",
|
||||
"6400: acc=0.68109375\n",
|
||||
"9600: acc=0.7067708333333333\n",
|
||||
"12800: acc=0.723671875\n",
|
||||
"16000: acc=0.73625\n",
|
||||
"19200: acc=0.7463541666666667\n",
|
||||
"22400: acc=0.7560714285714286\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"(218.64081493921944, 0.7667146513115803)"
|
||||
"(214.1013875559821, 0.7626759436980166)"
|
||||
]
|
||||
},
|
||||
"execution_count": 13,
|
||||
"execution_count": 12,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
|
|
@ -528,7 +529,15 @@
|
|||
"cell_type": "code",
|
||||
"execution_count": 14,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stderr",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"100%|█████████▉| 399999/400000 [00:15<00:00, 25411.14it/s]\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"vocab = torchtext.vocab.GloVe(name='6B', dim=50)"
|
||||
]
|
||||
|
|
@ -660,12 +669,7 @@
|
|||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"One of the reasons we are not seeing significant increase in accuracy is due to the fact that some words from our dataset are missing in the pre-trained GloVe vocabulary, and thus they are essentially ignored. To overcome this fact, we can train our own embeddings on our dataset. \n",
|
||||
"\n",
|
||||
"\n",
|
||||
"## Training your own embeddings\n",
|
||||
"\n",
|
||||
"In our examples, we have been using pre-trained semantic embeddings, but it is interesting to see how those embeddings can be trained using either CBoW, or Skip-gram architectures. This exercise goes beyond this module, but those interested might want to check out this [official PyTorch tutorial on Language Modeling](https://pytorch.org/tutorials/beginner/nlp/word_embeddings_tutorial.html). Also, **gensim** framework can be used to train most commonly used embeddings in a few lines of code, as described [in this documentation](https://pytorch.org/tutorials/beginner/nlp/word_embeddings_tutorial.html)."
|
||||
"One of the reasons we are not seeing significant increase in accuracy is due to the fact that some words from our dataset are missing in the pre-trained GloVe vocabulary, and thus they are essentially ignored. To overcome this fact, we can train our own embeddings on our dataset. "
|
||||
]
|
||||
},
|
||||
{
|
||||
|
|
@ -685,10 +689,13 @@
|
|||
}
|
||||
],
|
||||
"metadata": {
|
||||
"interpreter": {
|
||||
"hash": "0cb620c6d4b9f7a635928804c26cf22403d89d98d79684e4529119355ee6d5a5"
|
||||
},
|
||||
"kernelspec": {
|
||||
"display_name": "py37_pytorch",
|
||||
"language": "python",
|
||||
"name": "conda-env-py37_pytorch-py"
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
|
|
@ -700,7 +707,7 @@
|
|||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.7.7"
|
||||
"version": "3.8.12"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
|
|
|
|||
File diff suppressed because it is too large
Load Diff
Binary file not shown.
|
After Width: | Height: | Size: 39 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 15 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 38 KiB |
|
|
@ -0,0 +1,20 @@
|
|||
|
||||
# Language Modeling
|
||||
|
||||
Semantic embeddings, such as Word2Vec and GloVe, are in fact a first step towards **language modeling** - creating models that somehow *understand* (or *represent*) the nature of the language.
|
||||
|
||||
The main idea behind language modeling is training them on unlabeled datesets in unsupervised manner. It is important, because we have huge amounts of unlabeled text available, while the amount of labeled text would always be limited by the amount of effort we can spend on labeling. Most often, we build language models that can **predict missing words** in the text, because it is easy to mask out a random word in text and use it as a training sample.
|
||||
|
||||
## Training embeddings
|
||||
|
||||
In our previous examples, we have been using pre-trained semantic embeddings, but it is interesting to see how those embeddings can be trained using either CBoW, or Skip-gram architectures.
|
||||
|
||||
|
||||
|
||||
The idea of CBoW is exactly predicting a missing word, however, to do this we take a small sliding window of text tokens (we can denote them from W<sub>-2</sub> to W<sub>2</sub>), and train a model to predict the central word W<sub>0</sub> from few surrounding words.
|
||||
|
||||
## More Info
|
||||
|
||||
* [Official PyTorch tutorial on Language Modeling](https://pytorch.org/tutorials/beginner/nlp/word_embeddings_tutorial.html).
|
||||
* [Official TensorFlow tutorial on training Word2Vec model](https://www.tensorflow.org/tutorials/text/word2vec).
|
||||
* Using **gensim** framework to train most commonly used embeddings in a few lines of code is as described [in this documentation](https://pytorch.org/tutorials/beginner/nlp/word_embeddings_tutorial.html).
|
||||
|
|
@ -33,7 +33,23 @@ pip install -r requirements-torch.txt
|
|||
pip install -r requirements-tf.txt
|
||||
```
|
||||
|
||||
## GPU Warning
|
||||
|
||||
In this section, in some of the examples we will be training quite large models. It is advisable to run notebooks on GPU-enabled compute to minimize waiting time.
|
||||
|
||||
When running on GPU, you may experience situations when you run out of GPU memory. During training, the amount of GPU memory consumed depends on many factors, including minibatch size. If you experience any memory problems - you may try to minimize the minibatch size in the code.
|
||||
|
||||
Also, some older versions of Tensorflow do not release GPU memory correctly if we are training multiple models in one Python kernel. In order to use GPU memory cautiously, you may set tensorflow option to grow GPU memory allocation only when required. You would need to include the following code in your notebooks:
|
||||
|
||||
```python
|
||||
physical_devices = tf.config.list_physical_devices('GPU')
|
||||
if len(physical_devices)>0:
|
||||
tf.config.experimental.set_memory_growth(physical_devices[0], True)
|
||||
```
|
||||
|
||||
## Contents
|
||||
|
||||
* [Representing text as tensors](13-TextRep/README.md)
|
||||
* [Word Embeddings](14-Emdeddings/README.md)
|
||||
* [Language Modeling](15-LanguageModeling/README.md)
|
||||
|
||||
|
|
|
|||
|
|
@ -62,8 +62,8 @@ For a gentle introduction to *AI in the Cloud* topic you may consider taking [Ge
|
|||
<td><a href="https://docs.microsoft.com/learn/modules/intro-natural-language-processing-tensorflow/?WT.mc_id=academic-33554-dmitryso">MS Learn</a></td>
|
||||
<td>PAT</td></tr>
|
||||
<tr><td>13</td><td>Text Representation. Bow/TF-IDF</td><td><a href="5-NLP/13-TextRep/README.md">Text</a></td><td><a href="5-NLP/13-TextRep/TextRepresentationPyTorch.ipynb">PyTorch</a></td><td><a href="5-NLP/13-TextRep/TextRepresentationTF.ipynb">Tensorflow</td><td></td></tr>
|
||||
<tr><td>14</td><td>Semantic word embeddings</td><td><a href="5-NLP/14-Embeddings/README.md">Text</td><td><a href="5-NLP/14-Embeddings/EmbeddingsPyTorch.ipynb">PyTorch</a></td><td><a href="5-NLP/14-Embeddings/EmbeddingsTF.ipynb">Tensorflow</a></td><td></td></tr>
|
||||
<tr><td>15</td><td>Training your own embeddings</td><td>Text</td><td>PyTorch</td><td>Tensorflow</td><td></td></tr>
|
||||
<tr><td>14</td><td>Semantic word embeddings. Word2Vec and GloVe</td><td><a href="5-NLP/14-Embeddings/README.md">Text</td><td><a href="5-NLP/14-Embeddings/EmbeddingsPyTorch.ipynb">PyTorch</a></td><td><a href="5-NLP/14-Embeddings/EmbeddingsTF.ipynb">Tensorflow</a></td><td></td></tr>
|
||||
<tr><td>15</td><td>Language Modeling. Training your own embeddings</td><td><a href="5-NLP/15-LanguageModeling">Text</a></td><td>PyTorch</td><td>Tensorflow</td><td></td></tr>
|
||||
<tr><td>16</td><td>Recurrent Neural Networks</td><td>Text</td><td>PyTorch</td><td>Tensorflow</td><td></td></tr>
|
||||
<tr><td>17</td><td>Generative Recurrent Networks</td><td>Text</td><td>PyTorch</td><td>Tensorflow</td><td></td></tr>
|
||||
<tr><td>18</td><td>Language Modelling. Transformers. BERT.</td><td>Text</td><td>PyTorch</td><td>Tensorflow</td><td></td></tr>
|
||||
|
|
|
|||
Loading…
Reference in New Issue