"
+ },
+ "metadata": {}
+ }
+ ],
+ "execution_count": 20,
+ "metadata": {}
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "One of the reasons we're not seeing higher accuracy is because some words from our dataset are missing in the pretrained GloVe vocabulary, and thus they are essentially ignored. To overcome this, we can train our own embeddings based on our dataset. \n",
+ "\n",
+ "\n",
+ "## Training your own embeddings\n",
+ "\n",
+ "In our examples, we have been using pretrained semantic embeddings, but it is interesting to see how those embeddings can be trained using either CBoW, or skip-gram architectures. This exercise goes beyond this module, but those interested might want to check out this [official TensorFlow tutorial on training Word2Vec model](https://www.tensorflow.org/tutorials/text/word2vec). Also, the **gensim** framework can be used to train the most commonly used embeddings in a few lines of code, as described [in the official documentation](https://radimrehurek.com/gensim/auto_examples/tutorials/run_word2vec.html#training-your-own-model)."
+ ],
+ "metadata": {}
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Contextual embeddings\n",
+ "\n",
+ "One key limitation of traditional pretrained embedding representations such as Word2Vec is the fact that, even though they can capture some meaning of a word, they can't differentiate between different meanings. This can cause problems in downstream models.\n",
+ "\n",
+ "For example the word 'play' has different meaning in these two different sentences:\n",
+ "- I went to a **play** at the theater.\n",
+ "- John wants to **play** with his friends.\n",
+ "\n",
+ "The pretrained embeddings we talked about represent both meanings of the word 'play' in the same embedding. To overcome this limitation, we need to build embeddings based on the **language model**, which is trained on a large corpus of text, and *knows* how words can be put together in different contexts. Discussing contextual embeddings is out of scope for this tutorial, but we will come back to them when talking about language models in the next unit.\n"
+ ],
+ "metadata": {}
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "name": "conda-env-py37_tensorflow-py",
+ "language": "python",
+ "display_name": "py37_tensorflow"
+ },
+ "language_info": {
+ "name": "python",
+ "version": "3.7.9",
+ "mimetype": "text/x-python",
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "pygments_lexer": "ipython3",
+ "nbconvert_exporter": "python",
+ "file_extension": ".py"
+ },
+ "kernel_info": {
+ "name": "conda-env-py37_tensorflow-py"
+ },
+ "nteract": {
+ "version": "nteract-front-end@1.0.0"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}
\ No newline at end of file
diff --git a/5-NLP/14-Embeddings/torchnlp.py b/5-NLP/14-Embeddings/torchnlp.py
new file mode 100644
index 00000000..d6ca5e0c
--- /dev/null
+++ b/5-NLP/14-Embeddings/torchnlp.py
@@ -0,0 +1,104 @@
+import builtins
+import torch
+import torchtext
+import collections
+import os
+
+device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
+
+vocab = None
+tokenizer = torchtext.data.utils.get_tokenizer('basic_english')
+
+def load_dataset(ngrams=1,min_freq=1):
+ global vocab, tokenizer
+ print("Loading dataset...")
+ train_dataset, test_dataset = torchtext.datasets.AG_NEWS(root='./data')
+ train_dataset = list(train_dataset)
+ test_dataset = list(test_dataset)
+ classes = ['World', 'Sports', 'Business', 'Sci/Tech']
+ print('Building vocab...')
+ counter = collections.Counter()
+ for (label, line) in train_dataset:
+ counter.update(torchtext.data.utils.ngrams_iterator(tokenizer(line),ngrams=ngrams))
+ vocab = torchtext.vocab.Vocab(counter, min_freq=min_freq)
+ return train_dataset,test_dataset,classes,vocab
+
+def encode(x,voc=None,unk=0,tokenizer=tokenizer):
+ v = vocab if voc is None else voc
+ return [v.stoi.get(s,unk) for s in tokenizer(x)]
+
+def train_epoch(net,dataloader,lr=0.01,optimizer=None,loss_fn = torch.nn.CrossEntropyLoss(),epoch_size=None, report_freq=200):
+ optimizer = optimizer or torch.optim.Adam(net.parameters(),lr=lr)
+ loss_fn = loss_fn.to(device)
+ net.train()
+ total_loss,acc,count,i = 0,0,0,0
+ for labels,features in dataloader:
+ optimizer.zero_grad()
+ features, labels = features.to(device), labels.to(device)
+ out = net(features)
+ loss = loss_fn(out,labels) #cross_entropy(out,labels)
+ loss.backward()
+ optimizer.step()
+ total_loss+=loss
+ _,predicted = torch.max(out,1)
+ acc+=(predicted==labels).sum()
+ count+=len(labels)
+ i+=1
+ if i%report_freq==0:
+ print(f"{count}: acc={acc.item()/count}")
+ if epoch_size and count>epoch_size:
+ break
+ return total_loss.item()/count, acc.item()/count
+
+def padify(b,voc=None,tokenizer=tokenizer):
+ # b is the list of tuples of length batch_size
+ # - first element of a tuple = label,
+ # - second = feature (text sequence)
+ # build vectorized sequence
+ v = [encode(x[1],voc=voc,tokenizer=tokenizer) for x in b]
+ # compute max length of a sequence in this minibatch
+ l = max(map(len,v))
+ return ( # tuple of two tensors - labels and features
+ torch.LongTensor([t[0]-1 for t in b]),
+ torch.stack([torch.nn.functional.pad(torch.tensor(t),(0,l-len(t)),mode='constant',value=0) for t in v])
+ )
+
+def offsetify(b,voc=None):
+ # first, compute data tensor from all sequences
+ x = [torch.tensor(encode(t[1],voc=voc)) for t in b]
+ # now, compute the offsets by accumulating the tensor of sequence lengths
+ o = [0] + [len(t) for t in x]
+ o = torch.tensor(o[:-1]).cumsum(dim=0)
+ return (
+ torch.LongTensor([t[0]-1 for t in b]), # labels
+ torch.cat(x), # text
+ o
+ )
+
+def train_epoch_emb(net,dataloader,lr=0.01,optimizer=None,loss_fn = torch.nn.CrossEntropyLoss(),epoch_size=None, report_freq=200,use_pack_sequence=False):
+ optimizer = optimizer or torch.optim.Adam(net.parameters(),lr=lr)
+ loss_fn = loss_fn.to(device)
+ net.train()
+ total_loss,acc,count,i = 0,0,0,0
+ for labels,text,off in dataloader:
+ optimizer.zero_grad()
+ labels,text = labels.to(device), text.to(device)
+ if use_pack_sequence:
+ off = off.to('cpu')
+ else:
+ off = off.to(device)
+ out = net(text, off)
+ loss = loss_fn(out,labels) #cross_entropy(out,labels)
+ loss.backward()
+ optimizer.step()
+ total_loss+=loss
+ _,predicted = torch.max(out,1)
+ acc+=(predicted==labels).sum()
+ count+=len(labels)
+ i+=1
+ if i%report_freq==0:
+ print(f"{count}: acc={acc.item()/count}")
+ if epoch_size and count>epoch_size:
+ break
+ return total_loss.item()/count, acc.item()/count
+
diff --git a/5-NLP/README.md b/5-NLP/README.md
new file mode 100644
index 00000000..5ea06aa8
--- /dev/null
+++ b/5-NLP/README.md
@@ -0,0 +1,39 @@
+# Natural Language Processing
+
+In this section, we will focus on using Neural Networks to handle tasks related to natural language processing (NLP). There are many NLP problems that we want computers to be able to solve:
+
+* **Text classification** is typical classification problem on text sequences. Examples include classifying e-mail messages on spam vs. no-spam, or attributing news article into one of the pre-defined categories (sport, business, politics, etc.). Also, when developing chat bots, we often need to understand what a used wanted to say -- in this case we are dealing with **intent classificaton**. Often, in intent classification we need to deal with many categories.
+* **Sentiment analysis** is typical regression problem, where we need to attribute a number -- sentiment -- corresponding to how positive/negative the meaning of a sentence is. More advanced version of sentiment analysis is **aspect-based sentiment analysis** (ABSA), where we attribute sentiment not the the whole sentence, but to different parts of it (aspects), eg. *In this restaurant, I liked the cuisine, but the atmosphere was awful*.
+* **Named Entity Recognition** (NER) refers to the problem of extracting certain entities from text. For example, we might need to understand that in the phrase *I need to fly to Paris tomorrow* the word *tomorrow* refers to DATE, and *Paris* is a LOCATION.
+* **Keyword extraction** is similar to NER, but we need to extract words important to the meaning of the sentence automatically, without pre-training for specific entity types.
+* **Text clustering** can be useful when we want to group together similar sentences, for example, similar requests in technical support conversations.
+* **Question answering** refers to the ability of a model to answer a specific question. The model receives a text passage and a question as inputs, and it needs to provide a place in the text where the answer to the question is contained (or, sometimes, to generate the answer text).
+* **Text Generation** is the ability of a model to generate new text. It can be considered as classification task that predicts next letter/word based on some *text prompt*. Advanced text generation models, such as GPT-3, are able to solve other NLP tasks such as classification using a technique called [prompt programming](https://towardsdatascience.com/software-3-0-how-prompting-will-change-the-rules-of-the-game-a982fbfe1e0) or [prompt engineering](https://medium.com/swlh/openai-gpt-3-and-prompt-engineering-dcdc2c5fcd29)
+* **Text summarization** is a technique when we want a computer to "read" long text, and summarize it in a few sentences.
+* **Machine translation** can be viewed as a combination of text understanding in one language, and text generation in another one.
+
+Initially, most of NLP tasks were solved using traditional methods such as grammars. For example, in machine translation parsers were used to transform initial sentence into a syntax tree, then higher level semantic structures were extracted to represent the meaning of the sentence, and based on this meaning and grammar of the target language the result was generated. Nowadays, many NLP tasks are more effectively solved using neural networks.
+
+Many classical NLP methods are implemented in [Natural Language Processing Toolkit (NLTK)](https://www.nltk.org) Python library. There is a great [NLTK Book](https://www.nltk.org/book/) available online that covers how different NLP tasks can be solved using NLTK.
+
+In our course, we will mostly focus on using Neural Networks for NLP, and we will use NLTK where needed.
+
+We have already learnt about using neural networks for dealing with tabular data and with images. The main difference between those types of data and text is that text is a sequence of variable length, while the input size in case of images is known in advance. While convolutional networks can extract patterns from input data, patterns in text are more complex. Eg., we can have negation being separated from the subject be arbitrary many words (eg. *I do not like organges*, vs. *I do not like those big colorful tasty oranges*), and that should still be interpreted as one pattern. Thus, to handle language we need to introduce new neural network types, such as *recurrent networks* and *transformers*.
+
+## Install Libraries
+
+If you are using local Python installation to run this course, you may need to install all required libraries for NLP using the following commands:
+
+**For PyTorch**
+```bash
+pip install -r requirements-torch.txt
+```
+**For Tensorflow**
+```bash
+pip install -r requirements-tf.txt
+```
+
+## Contents
+
+* [Representing text as tensors](13-TextRep/README.md)
+* [Word Embeddings](14-Emdeddings/README.md)
diff --git a/5-NLP/requirements-pytorch.txt b/5-NLP/requirements-pytorch.txt
new file mode 100644
index 00000000..3545f559
--- /dev/null
+++ b/5-NLP/requirements-pytorch.txt
@@ -0,0 +1,15 @@
+gensim==3.8.3
+huggingface==0.0.1
+matplotlib
+nltk==3.5
+numpy==1.18.5
+opencv-python==4.5.1.48
+Pillow==7.1.2
+scikit-learn
+scipy
+torch==1.8.1
+torchaudio==0.8.1
+torchinfo==0.0.8
+torchtext==0.9.1
+torchvision==0.9.1
+transformers==4.3.3
\ No newline at end of file
diff --git a/5-NLP/requirements-tf.txt b/5-NLP/requirements-tf.txt
new file mode 100644
index 00000000..8b7689c0
--- /dev/null
+++ b/5-NLP/requirements-tf.txt
@@ -0,0 +1,12 @@
+gensim==3.8.3
+huggingface==0.0.1
+matplotlib
+nltk==3.5
+numpy==1.18.5
+opencv-python==4.5.1.48
+Pillow==7.1.2
+scikit-learn
+scipy
+tensorflow
+tensorflow_datasets
+transformers==4.3.3
\ No newline at end of file
diff --git a/README.md b/README.md
index 4706c59f..0080b20a 100644
--- a/README.md
+++ b/README.md
@@ -46,7 +46,7 @@ For a gentle introduction to *AI in the Cloud* topic you may consider taking [Ge
Text
Text |
PyTorch |
Keras/Tensorflow | |
-| IV | Computer Vision |
+
| IV | Computer Vision |
MS Learn |
MS Learn |
PAT |
@@ -57,17 +57,17 @@ For a gentle introduction to *AI in the Cloud* topic you may consider taking [Ge
| 10 | Generative Adversarial Networks | Text | PyTorch | Tensorflow | |
| 11 | Object Detection | Text | PyTorch | Tensorflow | |
| 12 | Instance Segmentation. U-Net | Text | PyTorch | Tensorflow | |
-| V | Natural Language Processing |
+
| V | Natural Language Processing |
MS Learn |
MS Learn |
PAT |
-| 13 | Text Representation. Bow/TF-IDF | Text | PyTorch | Tensorflow | |
-| 14 | Semantic Word Embeddings | Text | PyTorch | Tensorflow | |
+| 13 | Text Representation. Bow/TF-IDF | Text | PyTorch | Tensorflow | |
+| 14 | Semantic word embeddings | Text | PyTorch | Tensorflow | |
| 15 | Training your own embeddings | Text | PyTorch | Tensorflow | |
| 16 | Recurrent Neural Networks | Text | PyTorch | Tensorflow | |
| 17 | Generative Recurrent Networks | Text | PyTorch | Tensorflow | |
-| 18 | Language Modelling. BERT. Transformers. | Text | PyTorch | Tensorflow | |
-| 19 | Named Entity Recognition. | Text | PyTorch | Tensorflow | |
+| 18 | Language Modelling. Transformers. BERT. | Text | PyTorch | Tensorflow | |
+| 19 | Named Entity Recognition | Text | PyTorch | Tensorflow | |
| 20 | Text Generation using GPT | Text | PyTorch | Tensorflow | |
| VI | Other AI Techniques | PAT |
| 21 | Genetic Algorithms | Text | Notebook | |
+
## How to prevent overfitting
diff --git a/3-NeuralNetworks/05-Frameworks/README.md b/3-NeuralNetworks/05-Frameworks/README.md
index 0fb50286..df3e6a94 100644
--- a/3-NeuralNetworks/05-Frameworks/README.md
+++ b/3-NeuralNetworks/05-Frameworks/README.md
@@ -35,3 +35,4 @@ Low-Level API | [TensorFlow+Keras Notebook](IntroKerasTF.ipynb) | [PyTorch](Intr
--------------|-------------------------------------|--------------------------------
High-level API| [Keras](IntroKeras.ipynb) | *PyTorch Lightning*
+After mastering the frameworks, let's recap the notion of [overfitting](Overfiting.md).
\ No newline at end of file
diff --git a/3-NeuralNetworks/README.md b/3-NeuralNetworks/README.md
index 75fb6552..db82190b 100644
--- a/3-NeuralNetworks/README.md
+++ b/3-NeuralNetworks/README.md
@@ -39,6 +39,6 @@ where f is some non-linear **activation function**.
In this section we will learn about:
* [Perceptron](03-Perceptron/README.md), one of the earliest neural network models for two-class classification
-* [Modern multi-layered networks](04-OwnFramework/README.md) and [how to build our own framework](04-OwnFramework/OwnFramework.ipynb)
+* [Multi-layered networks](04-OwnFramework/README.md) and [how to build our own framework](04-OwnFramework/OwnFramework.ipynb)
* [Neural Network Frameworks](05-Frameworks/README.md), such as [PyTorch](05-Frameworks/IntroPyTorch.ipynb) and [Keras/Tensorflow](05-Frameworks/IntroKerasTF.ipynb)
-
+* [Overfitting](05-Frameworks/Overfitting.md)
diff --git a/5-NLP/13-TextRep/README.md b/5-NLP/13-TextRep/README.md
new file mode 100644
index 00000000..13ab39bc
--- /dev/null
+++ b/5-NLP/13-TextRep/README.md
@@ -0,0 +1,48 @@
+# Representing Text as Tensors
+
+## Text Classification
+
+Throughout the first part of this course, we will focus on **text classification** task. We will use [AG News](https://www.kaggle.com/amananandrai/ag-news-classification-dataset) Dataset, which contains news articles like the following:
+
+* Category: Sci/Tech
+* Title: Ky. Company Wins Grant to Study Peptides (AP)
+* Body: AP - A company founded by a chemistry researcher at the University of Louisville won a grant to develop...
+
+Our goal would be to classify the news item into one of the categories based on text.
+
+## Representing text
+
+If we want to solve Natural Language Processing (NLP) tasks with neural networks, we need some way to represent text as tensors. Computers already represent textual characters as numbers that map to fonts on your screen using encodings such as ASCII or UTF-8.
+
+
+
+We understand what each letter **represents**, and how all characters come together to form the words of a sentence. However, computers by themselves do not have such an understanding, and neural network has to learn the meaning during training.
+
+Therefore, we can use different approaches when representing text:
+* **Character-level representation**, when we represent text by treating each character as a number. Given that we have *C* different characters in our text corpus, the word *Hello* would be represented by 5x*C* tensor. Each letter would correspond to a tensor column in one-hot encoding.
+* **Word-level representation**, in which we create a **vocabulary** of all words in our text, and then represent words using one-hot encoding. This approach is somehow better, because each letter by itself does not have much meaning, and thus by using higher-level semantic concepts - words - we simplify the task for the neural network. However, given large dictionary size, we need to deal with high-dimensional sparse tensors.
+
+Regardless of the representation, we first need to convert text into a sequence of **tokens**, one token being either a character, a word, or sometimes even part of a word. Then, we convert token into a number, typically using **vocabulary**, and this number can be fed into a neural network using one-hot encoding.
+
+## N-Grams
+
+In natural language, precise meaning of words can only be determined in context. For example, meanings of *neural network* and *fishing network* are completely different. One of the ways to take this into account is to build our model on pairs of words, and considering word pairs as separate vocabulary tokens. In this way, the sentence *I like to go fishing* will be represented by the following sequence of tokens: *I like*, *like to*, *to go*, *go fishing*. The problem with this approach is that the dictionary size grows significantly, and combinations like *go fishing* and *go shopping* are presented by different tokens, which do not share any semantic similarity despite the same verb.
+
+In some cases, we may consider using tri-grams -- combinations of three words -- as well. Thus the approach is such is often called **n-grams**. Also, it makes sense to use n-grams with character-level representation, in which case n-grams will roughly correspond to different syllabi.
+
+## Bag-of-Words and TF/IDF
+
+When solving tasks like text classification, we need to be able to represent text by one fixed-size vector, which we will use as an input to final dense classifier. One of the simplest ways to do that is to combine all individual word representations, eg. by adding them. If we add one-hot encodings of each word, we will end up with a vector of frequencies, showing how many times each word appears inside the text. Such representation of text is called **bag of words** (BOW).
+
+
+
+BOW essentially represents which words appear in text and in which quantities, which can indeed be a good indication of what the text is about. For example, news article on politics is likely to contains words such as *president* and *country*, while scientific publication would have something like *collider*, *discovered*, etc. Thus, word frequencies can in many cases be a good indicator of text content.
+
+The problem with BOW is that certain common words, such as *and*, *is*, etc. appear in most of the texts, and they have highest frequencies, masking out the words that are really important. We may lower the importance of those words by taking into account the frequency at which words occur in the whole document collection. This is the main idea behind TF/IDF approach, which is covered in more detail in the notebooks below.
+
+However, none of those approaches can fully take into account the semantics of text. We need more powerful neural networks models, which we will discuss later in this course.
+
+## Continue to Notebooks
+
+* [Text Representation with PyTorch](TextRepresentationPyTorch.ipynb)
+* [Text Representation with Tensorflow](TextRepresentationTF.ipynb)
diff --git a/5-NLP/13-TextRep/TextRepresentationPyTorch.ipynb b/5-NLP/13-TextRep/TextRepresentationPyTorch.ipynb
new file mode 100644
index 00000000..463039a8
--- /dev/null
+++ b/5-NLP/13-TextRep/TextRepresentationPyTorch.ipynb
@@ -0,0 +1,571 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# Text classification task\n",
+ "\n",
+ "As we have mentioned, we will focus on simple text classification task based on **AG_NEWS** dataset, which is to classify news headlines into one of 4 categories: World, Sports, Business and Sci/Tech.\n",
+ "\n",
+ "### The Dataset\n",
+ "\n",
+ "This dataset is built into [`torchtext`](https://github.com/pytorch/text) module, so we can easily access it."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "d:\\WORK\\ai-for-beginners\\5-NLP\\13-TextRep\\data\\train.csv: 29.5MB [00:00, 48.1MB/s] \n",
+ "d:\\WORK\\ai-for-beginners\\5-NLP\\13-TextRep\\data\\test.csv: 1.86MB [00:00, 9.53MB/s] \n"
+ ]
+ }
+ ],
+ "source": [
+ "import torch\n",
+ "import torchtext\n",
+ "import os\n",
+ "import collections\n",
+ "os.makedirs('./data',exist_ok=True)\n",
+ "train_dataset, test_dataset = torchtext.datasets.AG_NEWS(root='./data')\n",
+ "classes = ['World', 'Sports', 'Business', 'Sci/Tech']"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Here, `train_dataset` and `test_dataset` contain iterators that return pairs of label (number of class) and text respectively, for example:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "(3,\n",
+ " \"Wall St. Bears Claw Back Into the Black (Reuters) Reuters - Short-sellers, Wall Street's dwindling\\\\band of ultra-cynics, are seeing green again.\")"
+ ]
+ },
+ "execution_count": 3,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "next(train_dataset)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "So, let's print out the first 10 new headlines from our dataset: "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 4,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "**Sci/Tech** -> Carlyle Looks Toward Commercial Aerospace (Reuters) Reuters - Private investment firm Carlyle Group,\\which has a reputation for making well-timed and occasionally\\controversial plays in the defense industry, has quietly placed\\its bets on another part of the market.\n",
+ "**Sci/Tech** -> Oil and Economy Cloud Stocks' Outlook (Reuters) Reuters - Soaring crude prices plus worries\\about the economy and the outlook for earnings are expected to\\hang over the stock market next week during the depth of the\\summer doldrums.\n",
+ "**Sci/Tech** -> Iraq Halts Oil Exports from Main Southern Pipeline (Reuters) Reuters - Authorities have halted oil export\\flows from the main pipeline in southern Iraq after\\intelligence showed a rebel militia could strike\\infrastructure, an oil official said on Saturday.\n",
+ "**Sci/Tech** -> Oil prices soar to all-time record, posing new menace to US economy (AFP) AFP - Tearaway world oil prices, toppling records and straining wallets, present a new economic menace barely three months before the US presidential elections.\n",
+ "**Sci/Tech** -> Stocks End Up, But Near Year Lows (Reuters) Reuters - Stocks ended slightly higher on Friday\\but stayed near lows for the year as oil prices surged past #36;46\\a barrel, offsetting a positive outlook from computer maker\\Dell Inc. (DELL.O)\n"
+ ]
+ }
+ ],
+ "source": [
+ "for i,x in zip(range(5),train_dataset):\n",
+ " print(f\"**{classes[x[0]]}** -> {x[1]}\")\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Because datasets are iterators, if we want to use the data multiple times we need to convert it to list:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 5,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "train_dataset, test_dataset = torchtext.datasets.AG_NEWS(root='./data')\n",
+ "train_dataset = list(train_dataset)\n",
+ "test_dataset = list(test_dataset)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Tokenization\n",
+ "\n",
+ "Now we need to convert text into **numbers** that can be represented as tensors. If we want word-level representation, we need to do two things:\n",
+ "* use **tokenizer** to split text into **tokens**\n",
+ "* build a **vocabulary** of those tokens."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 9,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "['he', 'said', 'hello']"
+ ]
+ },
+ "execution_count": 9,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "tokenizer = torchtext.data.utils.get_tokenizer('basic_english')\n",
+ "tokenizer('He said: hello')"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 10,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "counter = collections.Counter()\n",
+ "for (label, line) in train_dataset:\n",
+ " counter.update(tokenizer(line))\n",
+ "vocab = torchtext.vocab.Vocab(counter, min_freq=1)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Using vocabulary, we can easily encode out tokenized string into a set of numbers:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 11,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Vocab size if 95812\n"
+ ]
+ },
+ {
+ "data": {
+ "text/plain": [
+ "[283, 2321, 5, 337, 19, 1301, 2357]"
+ ]
+ },
+ "execution_count": 11,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "vocab_size = len(vocab)\n",
+ "print(f\"Vocab size if {vocab_size}\")\n",
+ "\n",
+ "def encode(x):\n",
+ " return [vocab.stoi[s] for s in tokenizer(x)]\n",
+ "\n",
+ "encode('I love to play with my words')"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Bag of Words text representation\n",
+ "\n",
+ "Because words represent meaning, sometimes we can figure out the meaning of a text by just looking at the individual words, regardless of their order in the sentence. For example, when classifying news, words like *weather*, *snow* are likely to indicate *weather forecast*, while words like *stocks*, *dollar* would count towards *financial news*.\n",
+ "\n",
+ "**Bag of Words** (BoW) vector representation is the most commonly used traditional vector representation. Each word is linked to a vector index, vector element contains the number of occurrences of a word in a given document.\n",
+ "\n",
+ " \n",
+ "\n",
+ "> **Note**: You can also think of BoW as a sum of all one-hot-encoded vectors for individual words in the text.\n",
+ "\n",
+ "Below is an example of how to generate a bag of word representation using the Scikit Learn python library:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 12,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "array([[1, 1, 0, 2, 0, 0, 0, 0, 0]], dtype=int64)"
+ ]
+ },
+ "execution_count": 12,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "from sklearn.feature_extraction.text import CountVectorizer\n",
+ "vectorizer = CountVectorizer()\n",
+ "corpus = [\n",
+ " 'I like hot dogs.',\n",
+ " 'The dog ran fast.',\n",
+ " 'Its hot outside.',\n",
+ " ]\n",
+ "vectorizer.fit_transform(corpus)\n",
+ "vectorizer.transform(['My dog likes hot dogs on a hot day.']).toarray()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "To compute bag-of-words vector from the vector representation of our AG_NEWS dataset, we can use the following function:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 13,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "tensor([0., 0., 2., ..., 0., 0., 0.])\n"
+ ]
+ }
+ ],
+ "source": [
+ "vocab_size = len(vocab)\n",
+ "\n",
+ "def to_bow(text,bow_vocab_size=vocab_size):\n",
+ " res = torch.zeros(bow_vocab_size,dtype=torch.float32)\n",
+ " for i in encode(text):\n",
+ " if i