RomanNum3ral
/
AI-For-Beginners
mirror of https://github.com/microsoft/AI-For-Beginners.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Add lesson on language models

This commit is contained in:
Dmitri Soshnikov 2022-05-24 14:05:07 +03:00
parent 838ae2916e
commit 4e63ff3b56
4 changed files with 406 additions and 4 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
README.md
View File

@ -87,7 +87,7 @@ For a gentle introduction to *AI in the Cloud* topics you may consider taking th
<tr><td>17</td><td>Generative Recurrent Networks</td><td><a href="lessons/5-NLP/17-GenerativeNetworks/README.md">Text</a></td><td><a href="lessons/5-NLP/17-GenerativeNetworks/GenerativePyTorch.md">PyTorch</a></td><td><a href="lessons/5-NLP/17-GenerativeNetworks/GenerativeTF.md">TensorFlow</a></td><td><a href="lessons/5-NLP/17-GenerativeNetworks/lab/README.md">Lab</a></td></tr>
<tr><td>18</td><td>Transformers. BERT.</td><td><a href="lessons/5-NLP/18-Transformers/README.md">Text</a></td><td><a href="lessons/5-NLP/18-Transformers/TransformersPyTorch.md">PyTorch</a></td><td><a href="lessons/5-NLP/18-Transformers/TransformersTF.md">TensorFlow</a></td><td></td></tr>
<tr><td>19</td><td>Named Entity Recognition</td><td><a href="lessons/5-NLP/19-NER/README.md">Text</a></td><td></td><td><a href="lessons/5-NLP/19-NER/NER-TF.ipynb">TensorFlow</a></td><td><a href="lessons/5-NLP/19-NER/lab/README.md">Lab</a></td></tr>
<tr><td>20</td><td>Large Language Models, Prompt Programming and Few-Shot Tasks</td><td>Text</td><td>PyTorch</td><td>TensorFlow</td><td></td></tr>
<tr><td>20</td><td>Large Language Models, Prompt Programming and Few-Shot Tasks</td><td><a href="lessons/5-NLP/20-LanguageModels/README.md">Text</a></td><td><a href="lessons/5-NLP/20-LanguageModels/GPT-PyTorch.ipynb">PyTorch</td><td></td><td></td></tr>
<tr><td>VI</td><td colspan="4"><b>Other AI Techniques</b></td><td></td></tr>
<tr><td>21</td><td>Genetic Algorithms</td><td><a href="lessons/6-Other/21-GeneticAlgorithms/README.md">Text</a><td colspan="2"><a href="lessons/6-Other/21-GeneticAlgorithms/Genetic.ipynb">Notebook</a></td><td></td></tr>
<tr><td>22</td><td>Deep Reinforcement Learning</td><td><a href="lessons/6-Other/22-DeepRL/README.md">Text</a></td><td></td><td><a href="lessons/6-Other/22-DeepRL/CartPole-RL-TF.ipynb">TensorFlow</td><td><a href="lessons/6-Other/22-DeepRL/lab/README.md">Lab</a></td></tr>

3
lessons/5-NLP/15-LanguageModeling/README.md
View File

@ -1,4 +1,3 @@
# Language Modeling
Semantic embeddings, such as Word2Vec and GloVe, are in fact a first step towards **language modeling** - creating models that somehow *understand* (or *represent*) the nature of the language.
@ -13,7 +12,7 @@ In our previous examples, we used pre-trained semantic embeddings, but it is int
* **N-Gram** language modeling, when we predict a token by looking at N previous tokens (N-gram)
* **Continuous Bag-of-Words** (CBoW), when we predict the middle token $W_0$ in a token sequence $W_{-N}$, ..., $W_N$.
* **Skip-gram**, where we predict a set of neighboring tokens $\{W_{-N},\dots, W_{-1}, W_1,\dots, W_N\}$ from the middle token $W_0$.
* **Skip-gram**, where we predict a set of neighboring tokens {$W_{-N},\dots, W_{-1}, W_1,\dots, W_N$} from the middle token $W_0$.
![image from paper on converting words to vectors](../14-Embeddings/images/example-algorithms-for-converting-words-to-vectors.png)

347
lessons/5-NLP/20-LangModels/GPT-PyTorch.ipynb Normal file
View File

@ -0,0 +1,347 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Experimenting with GPT-2\n",
"\n",
"This notebook is part of [AI for Beginners Curriculum](http://aka.ms/ai-beginners).\n",
"\n",
"In this notebook, we will explore how we can play with OpenAI GPT-2 model using Hugging Face `transformers` library.\n",
"\n",
"Without further ado, let's instantiate text generating pipeline and start generating! You can select smaller GPT-2 model in order to increase download time and speed of inference, but that would affect the quality."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Some weights of GPT2Model were not initialized from the model checkpoint at gpt2-large and are newly initialized: ['h.0.attn.masked_bias', 'h.1.attn.masked_bias', 'h.2.attn.masked_bias', 'h.3.attn.masked_bias', 'h.4.attn.masked_bias', 'h.5.attn.masked_bias', 'h.6.attn.masked_bias', 'h.7.attn.masked_bias', 'h.8.attn.masked_bias', 'h.9.attn.masked_bias', 'h.10.attn.masked_bias', 'h.11.attn.masked_bias', 'h.12.attn.masked_bias', 'h.13.attn.masked_bias', 'h.14.attn.masked_bias', 'h.15.attn.masked_bias', 'h.16.attn.masked_bias', 'h.17.attn.masked_bias', 'h.18.attn.masked_bias', 'h.19.attn.masked_bias', 'h.20.attn.masked_bias', 'h.21.attn.masked_bias', 'h.22.attn.masked_bias', 'h.23.attn.masked_bias', 'h.24.attn.masked_bias', 'h.25.attn.masked_bias', 'h.26.attn.masked_bias', 'h.27.attn.masked_bias', 'h.28.attn.masked_bias', 'h.29.attn.masked_bias', 'h.30.attn.masked_bias', 'h.31.attn.masked_bias', 'h.32.attn.masked_bias', 'h.33.attn.masked_bias', 'h.34.attn.masked_bias', 'h.35.attn.masked_bias']\n",
"You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n",
"Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n"
]
},
{
"data": {
"text/plain": [
"[{'generated_text': 'Hello! I am a neural network, and I want to say that I am an expert in the area of learning and understanding neural networks. I also know a bit about math. You might have seen \"How to make deep neural networks\" or \"What is a deep learning neural network?\". I would like to discuss the first one.\\n\\nHow to make deep neural networks is a very complicated topic, and I don\\'t know how to go into it in detail, but I want to do some'},\n",
" {'generated_text': 'Hello! I am a neural network, and I want to say that you can find a network algorithm, called Naive Bayes, which produces good results, but which is computationally too expensive to be useful, so I will not cover all of its details here.\\n\\nFirst we are going to define how an NN looks like.\\n\\nHere is a naive Bayes classification problem with a few variables:\\n\\nFeature Input Value (x) 1 2 3 4 5 1 3 4'},\n",
" {'generated_text': 'Hello! I am a neural network, and I want to say that… \"I\\'m done with this crap!\"\\n\\nNow, if you were to be able to tell me all about this, then you did something wrong with the previous part. If you weren\\'t able to tell I did this before, sorry! If your brain was too wired, then you don\\'t know anything about neural networks. I mean, it turns out neural networks have a big problem. It\\'s called lossy neural'},\n",
" {'generated_text': 'Hello! I am a neural network, and I want to say that I learned this from Wikipedia but also from lots of other books. They also gave me some tips on how to do the analysis, which is really useful for beginners.\\n\\nFor the code, I just used Python and scikit-learn. But I will warn you with two caveats –\\n\\nfirst, this code will have to be imported before you can run them. You have to import scikit-learn after'},\n",
" {'generated_text': 'Hello! I am a neural network, and I want to say that you\\'re awesome.\"'}]"
]
},
"execution_count": 1,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from transformers import pipeline\n",
"\n",
"model_name = 'gpt2-large' # try 'gpt2' for small model, 'gpt2-medium' for medium one\n",
"\n",
"generator = pipeline('text-generation', model=model_name)\n",
"\n",
"generator(\"Hello! I am a neural network, and I want to say that\", max_length=100, num_return_sequences=5)\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Prompt Engineering\n",
"\n",
"In some of the problems, you can use GPT-2 generation right away by designing correct prompts. Have a look at the examples below:"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n"
]
},
{
"data": {
"text/plain": [
"[{'generated_text': 'Synonyms of a word cat:\\n\\n(a) cat;\\n\\n(b) black'},\n",
" {'generated_text': 'Synonyms of a word cat: cat-bitch; cat-queen; cat-c'},\n",
" {'generated_text': 'Synonyms of a word cat: feline, feline form, feline spirit, fas'},\n",
" {'generated_text': 'Synonyms of a word cat:\\n\\n(a) cat-like, like a cat;'},\n",
" {'generated_text': 'Synonyms of a word cat:\\n\\nThe more common English words you need to understand the meaning'}]"
]
},
"execution_count": 10,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"generator(\"Synonyms of a word cat:\", max_length=20, num_return_sequences=5)"
]
},
{
"cell_type": "code",
"execution_count": 25,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n"
]
},
{
"data": {
"text/plain": [
"[{'generated_text': 'I love when you say this -> Positive\\nI have myself -> Negative\\nThis is awful for you to say this -> Negative\\nHow do you think this hurts us all? -> Positive\\nYou seem'},\n",
" {'generated_text': \"I love when you say this -> Positive\\nI have myself -> Negative\\nThis is awful for you to say this -> Negative\\nThis is such a horrible way to treat yourself -> Negative (You're\"},\n",
" {'generated_text': 'I love when you say this -> Positive\\nI have myself -> Negative\\nThis is awful for you to say this -> I am disappointed\\nI have to admit -> You are a good friend but you'},\n",
" {'generated_text': 'I love when you say this -> Positive\\nI have myself -> Negative\\nThis is awful for you to say this -> Positive\\nThis is awful for me to do this with you -> Negative\\nWhy'},\n",
" {'generated_text': \"I love when you say this -> Positive\\nI have myself -> Negative\\nThis is awful for you to say this -> I'm still sad for you!\\nMe? \\xa0I find this a\"}]"
]
},
"execution_count": 25,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"generator(\"I love when you say this -> Positive\\nI have myself -> Negative\\nThis is awful for you to say this ->\", max_length=40, num_return_sequences=5)"
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n"
]
},
{
"data": {
"text/plain": [
"[{'generated_text': 'Translate English to French: cat => chat, dog => chien, student => étudiant;\\n\\n\\nOther:\\n\\nTo learn'},\n",
" {'generated_text': 'Translate English to French: cat => chat, dog => chien, student => étude, french = le français = \"'},\n",
" {'generated_text': \"Translate English to French: cat => chat, dog => chien, student => été, mama => m'aime. We\"}]"
]
},
"execution_count": 20,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"generator(\"Translate English to French: cat => chat, dog => chien, student => \", top_k=50, max_length=30, num_return_sequences=3)"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n"
]
},
{
"data": {
"text/plain": [
"[{'generated_text': 'People who liked the movie The Matrix also liked \\xa0\"Dawn of the Dead\" (or in the case of John Goodman \"Vanish\"),\\xa0but their votes didn\\'t matter! The first'},\n",
" {'generated_text': 'People who liked the movie The Matrix also liked _____. It\\'s a true statement, as \"tasteful\" movies are often more popular for their plot, characters and themes than for their entertainment'},\n",
" {'generated_text': 'People who liked the movie The Matrix also liked \\xa0the book... \\xa0And so on, and so forth. Now at the other end of the spectrum...\\n...the real \"truths'},\n",
" {'generated_text': \"People who liked the movie The Matrix also liked 『The Grand Budapest Hotel』. 『The Grand Budapest Hotel』 is the movie people who like to watch their dream come true. I'll explain\"},\n",
" {'generated_text': 'People who liked the movie The Matrix also liked \\xa0a lot of the lines that were used, and I got to listen to the dialogue in the movie. \\xa0The characters were really enjoyable characters'}]"
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"generator(\"People who liked the movie The Matrix also liked \", max_length=40, num_return_sequences=5)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Text Sampling Strategies\n",
"\n",
"So far we have been using simple **greedy** sampling strategy, when we selected next word based on the highest probability. Here is how it works:"
]
},
{
"cell_type": "code",
"execution_count": 28,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n"
]
},
{
"data": {
"text/plain": [
"[{'generated_text': \"It was early evening when I can back from work. I usually work late, but this time it was an exception. When I entered a room, I saw my girlfriend with a very cute looking guy. I noticed that his eyes that looked into mine were a bit scary and kind of like a predator's eyes. And that's when I knew that I had to know something about him.\\n\\nAs I sat, studying him, my mind was racing in a crazy way. We didn't exactly\"},\n",
" {'generated_text': 'It was early evening when I can back from work. I usually work late, but this time it was an exception. When I entered a room, I saw my reflection in a small screen attached to a thin, white piece of glass. Suddenly, I thought about the possibility of making money making a computer, and decided to try it for myself! I started working on making an e-ink display. I was able to make it work by following some simple rules!\\n\\nThe first step is'},\n",
" {'generated_text': 'It was early evening when I can back from work. I usually work late, but this time it was an exception. When I entered a room, I saw a pile of clothing and furniture. They were completely ruined. The place had been abandoned. It had been a home, not a work place.\\n\\nWhen the police arrived, they found the crime scene and I walked with two officers to the police station. Before that I had not gone to the police station and I have never been to'},\n",
" {'generated_text': 'It was early evening when I can back from work. I usually work late, but this time it was an exception. When I entered a room, I saw a woman who could not be a mother of any kind. I saw the same woman with her eyes closed and her mouth wide open. She was sitting naked on a couch and was sitting topless. I saw at least three different women with similar posture. The only thing I kept telling myself was \"if this is how a woman eats,'},\n",
" {'generated_text': \"It was early evening when I can back from work. I usually work late, but this time it was an exception. When I entered a room, I saw a pale person with long white hair sitting quietly. He was a bit bigger and his eyes looked different. He was watching me.\\n\\nI didn't have to say anything. The man kept studying me, and after a while he turned his eyes away from me to the floor. In a moment he was gone, and he was still\"}]"
]
},
"execution_count": 28,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"prompt = \"It was early evening when I can back from work. I usually work late, but this time it was an exception. When I entered a room, I saw\"\n",
"generator(prompt,max_length=100,num_return_sequences=5)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Beam Search** allows the generator to explore several directions (*beams*) of text generation, and select the ones with highers overall score. You can do beam search by providing `num_beams` parameter. You can also specify `no_repeat_ngram_size` to penalize the model for repeating n-grams of a given size: "
]
},
{
"cell_type": "code",
"execution_count": 29,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n"
]
},
{
"data": {
"text/plain": [
"[{'generated_text': 'It was early evening when I can back from work. I usually work late, but this time it was an exception. When I entered a room, I saw a group of people sitting around a table. One of them was a middle-aged man. He was wearing a black suit and a white shirt. His eyes were closed and he was staring at the floor with his hands on his knees.\\n\\n\"What are you doing here?\" I asked him. \"What do you want?\"\\n'},\n",
" {'generated_text': 'It was early evening when I can back from work. I usually work late, but this time it was an exception. When I entered a room, I saw a man sitting at a table in front of a computer. He was wearing a black suit, a white shirt, and a red tie.\\n\\n\"Hello,\" he said to me. \"How are you?\" he asked me in a voice that sounded as if he was speaking to a child. There was a smile on his face.'},\n",
" {'generated_text': 'It was early evening when I can back from work. I usually work late, but this time it was an exception. When I entered a room, I saw a man sitting on the edge of the bed. He was staring at me.\\n\\n\"What are you doing here?\" he asked in a low voice. \"I don\\'t know why you\\'re here, and I\\'m not interested in what you have to say,\" he said, as if he didn\\'t understand what I was saying.'},\n",
" {'generated_text': 'It was early evening when I can back from work. I usually work late, but this time it was an exception. When I entered a room, I saw a man sitting in a chair in front of a desk. He was in his mid-thirties, and he was wearing a dark suit and a white shirt.\\n\\n\"What are you doing here?\" I asked him. The man didn\\'t answer. Instead, he walked over to the desk and sat down on it. \"'},\n",
" {'generated_text': 'It was early evening when I can back from work. I usually work late, but this time it was an exception. When I entered a room, I saw that the door was open, and a man was sitting on a chair. He was wearing a white shirt and black pants.\\n\\n\"What are you doing here?\" I asked. The man looked at me for a moment, then said, \"I have something to tell you.\" He handed me a piece of paper. It was a'}]"
]
},
"execution_count": 29,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"prompt = \"It was early evening when I can back from work. I usually work late, but this time it was an exception. When I entered a room, I saw\"\n",
"generator(prompt,max_length=100,num_return_sequences=5,num_beams=10,no_repeat_ngram_size=2)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Sampling** selects the next word non-deterministically, using the probability distribution returned by the model. You turn on sampling using `do_sample=True` parameter. You can also specify `temperature`, to make the model more or less deterministic."
]
},
{
"cell_type": "code",
"execution_count": 30,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n"
]
},
{
"data": {
"text/plain": [
"[{'generated_text': 'It was early evening when I can back from work. I usually work late, but this time it was an exception. When I entered a room, I saw one of my colleagues lying on his stomach. He was unconscious. I could smell a strong odor of alcohol on his breath. I was on my way out. I ran to his room and found him unconscious on the bed. I saw a bottle of wine on the bed. He was in a state of intoxication. He was unconscious. I'}]"
]
},
"execution_count": 30,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"prompt = \"It was early evening when I can back from work. I usually work late, but this time it was an exception. When I entered a room, I saw\"\n",
"generator(prompt,max_length=100,do_sample=True,temperature=0.8)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can also provide to additional parameters to sampling:\n",
"* `top_k` specifies the number of word options to consider when using sampling. This minimizes the chance of getting weird (low-probability) words in our text.\n",
"* `top_p` is similar, but we chose the smallest subset of most probable words, whose total probability is larger than p.\n",
"\n",
"Feel free to experiment with adding those parameters in."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Fine-Tuning GPT-2\n",
"\n",
"You can also fine-tune GPT-2 text generation on your own dataset. This will allow you to adjust the style of text, while keeping the major part of language model. The example of fine-tuning GPT-2 to generate song lyrics can be found [in this blog post](https://towardsdatascience.com/how-to-fine-tune-gpt-2-for-text-generation-ae2ea53bc272)."
]
}
],
"metadata": {
"interpreter": {
"hash": "16af2a8bbb083ea23e5e41c7f5787656b2ce26968575d8763f2c4b17f9cd711f"
},
"kernelspec": {
"display_name": "Python 3.8.12 ('py38')",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.12"
},
"orig_nbformat": 4
},
"nbformat": 4,
"nbformat_minor": 2
}

58
lessons/5-NLP/20-LangModels/README.md
View File

@ -1 +1,57 @@
Placeholder
# Pre-Trained Large Language Models
In all of our previous tasks, we were training a neural network to perform a certain task using labeled dataset. With large transformer models, such as BERT, we use language modelling in self-supervised fashion to build a language model, which is then specialized for specific downstream task with further domain-specific training. However, it has been demonstrated that large language models can also solve many tasks without ANY domain-specific training. A family of models capable of doing that is called **GPT**: Generative Pre-Trained Transformer.
## [Pre-lecture quiz](https://black-ground-0cc93280f.1.azurestaticapps.net/quiz/120)
## Text Generation and Perplexity
The idea of a neural network being able to do general tasks without downstream training is presented in [Language Models are Unsupervised Multitask Learners](https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf) paper. The main idea is the many other tasks can be modeled using **text generation**, because understanding text essentially means being able to produce it. Because the model is trained on a huge amount of text that encompasses human knowledge, it also becomes knowledgeable about wide variety of subjects.
> Understanding and being able to produce text also entails knowing something about the world around us. People also learn by reading to the large extent, and GPT network is similar in this respect.
Text generation networks work by predicting probability of the next word $P(w_N)$. However, unconditional probability of the next word equals to the frequency of the this word in the text corpus. GPT is able to give us **conditional probability** of the next word, given the previous ones $P(w_N | w_{n-1}, ..., w_0)$.
> You can read more about probabilities in our [Data Science for Beginers Curriculum](https://github.com/microsoft/Data-Science-For-Beginners/tree/main/1-Introduction/04-stats-and-probability)
Quality of language generating model can be defined using **perplexity**. It is intrinsic metric that allows us to measure the model quality without any task-specific dataset. It is based on the notion of *probability of a sentence* - the model assigns high probability to a sentence that is likely to be real (i.e. the model is not **perplexed** by it), and low probability to sentences that make less sense (eg. *Can it does what?*). When we give our model sentences from real text corpus, we would expect them to have high probability, and low **perplexity**. Mathematically, it is defined as normalized inverse probability of the test set:
$$
\mathrm{Perplexity}(W) = \sqrt[N]{1\over P(W_1,...,W_N)}
$$
**You can experiment with text generation using [GPT-powered text editor from Hugging Face](https://transformer.huggingface.co/doc/gpt2-large)**. In this editor, you start writing your text, and pressing **[TAB]** will offer you several completion options. If they are too short, or you are not satisfied with them - press [TAB] again, and you will have more options, including longer pieces of text.
## GPT is a Family
GPT is not a single model, but rather a collection of models developed and trained by [OpenAI](http://openai.org). The latest model openly available is [GPT-2](https://huggingface.co/docs/transformers/model_doc/gpt2#openai-gpt2), which has up to 1.5 billion parameters (there are several variations of the model, so you can select one for your tasks that is a good compromise between size/performance). Latest GPT-3 model has up to 175 billion parameters, and is available [as a cognitive service from Microsoft Azure](https://azure.microsoft.com/en-us/services/cognitive-services/openai-service/#overview?WT.mc_id=academic-57639-dmitryso), and as [OpenAI API](https://openai.com/api/).
## Prompt-based Inference
Because GPT has been trained on a vast volumes of data, it has some commonsense knowledge embedded directly inside the model. This allows us to force GPT to solve certain typical problems by just providing the right prompt. This presents a whole new approach for using pre-trained models, called [Prompt Engineering](https://en.wikipedia.org/wiki/Prompt_engineering). It is particularly useful with GPT-3, which has significantly more parameters, and consequently more embedded knowledge.
Here are a few example of using Prompt Engineering (answers from the model are *in italics*):
**Recommendation Systems**:<br/>
People, who liked the movie "The Matrix" also liked *Star Wars, Jupyter Ascending, Ex Machina*
**Translation**:<br/>
Translate from English to French:<br/>
cat => chat, dog => chien, student => *étudiant*
**Looking for words:**<br/>
Synonyms of a word cat: *feline, feline form, feline spirit*
[This article](https://www.gwern.net/GPT-3#prompts-as-programming) talks more about prompt engineering.
## ✍️ Example Notebook: [Playing with GPT-2](GPT-PyTorch.ipynb)
Continue your learning in the following notebooks:
* [Generating text with GPT-2 and Hugging Face Transformers](GPT-PyTorch.ipynb)
## Conclusion
New general pre-trained language models do not only model language structure, but also contain vast amount of commonsense knowledge. Thus, they can be effectively used to solve some NLP tasks in zero-shop or few-shot settings.
## [Post-lecture quiz](https://black-ground-0cc93280f.1.azurestaticapps.net/quiz/220)