RomanNum3ral
/
AI-For-Beginners
mirror of https://github.com/microsoft/AI-For-Beginners.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

updated langModels lesson

This commit is contained in:
Pikachú 2023-06-15 13:48:48 +03:00
parent a7e3a4906d
commit f900907d8c
3 changed files with 84 additions and 121 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
.gitignore vendored
View File

@ -3,6 +3,10 @@
##
## Get latest from https://github.com/github/gitignore/blob/master/VisualStudio.gitignore
# virtual environment
venv/
ENV/
# ignore all pdf creation files
package.json
package-lock.json

175
lessons/5-NLP/20-LangModels/GPT-PyTorch.ipynb
View File

@ -1,16 +1,17 @@
{
"cells": [
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Experimenting with GPT-2\n",
"## Experimenting with OpenAI GPT\n",
"\n",
"This notebook is part of [AI for Beginners Curriculum](http://aka.ms/ai-beginners).\n",
"\n",
"In this notebook, we will explore how we can play with OpenAI GPT-2 model using Hugging Face `transformers` library.\n",
"In this notebook, we will explore how we can play with OpenAI-GPT model using Hugging Face `transformers` library.\n",
"\n",
"Without further ado, let's instantiate text generating pipeline and start generating! You can select smaller GPT-2 model in order to increase download time and speed of inference, but that would affect the quality."
"Without further ado, let's instantiate text generating pipeline and start generating! "
]
},
{
@ -22,19 +23,30 @@
"name": "stderr",
"output_type": "stream",
"text": [
"Some weights of GPT2Model were not initialized from the model checkpoint at gpt2-large and are newly initialized: ['h.0.attn.masked_bias', 'h.1.attn.masked_bias', 'h.2.attn.masked_bias', 'h.3.attn.masked_bias', 'h.4.attn.masked_bias', 'h.5.attn.masked_bias', 'h.6.attn.masked_bias', 'h.7.attn.masked_bias', 'h.8.attn.masked_bias', 'h.9.attn.masked_bias', 'h.10.attn.masked_bias', 'h.11.attn.masked_bias', 'h.12.attn.masked_bias', 'h.13.attn.masked_bias', 'h.14.attn.masked_bias', 'h.15.attn.masked_bias', 'h.16.attn.masked_bias', 'h.17.attn.masked_bias', 'h.18.attn.masked_bias', 'h.19.attn.masked_bias', 'h.20.attn.masked_bias', 'h.21.attn.masked_bias', 'h.22.attn.masked_bias', 'h.23.attn.masked_bias', 'h.24.attn.masked_bias', 'h.25.attn.masked_bias', 'h.26.attn.masked_bias', 'h.27.attn.masked_bias', 'h.28.attn.masked_bias', 'h.29.attn.masked_bias', 'h.30.attn.masked_bias', 'h.31.attn.masked_bias', 'h.32.attn.masked_bias', 'h.33.attn.masked_bias', 'h.34.attn.masked_bias', 'h.35.attn.masked_bias']\n",
"c:\\Users\\bethanycheum\\Desktop\\AI-For-Beginners\\.venv\\lib\\site-packages\\tqdm\\auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
" from .autonotebook import tqdm as notebook_tqdm\n",
"Downloading model.safetensors: 100%|██████████| 479M/479M [04:28<00:00, 1.78MB/s] \n",
"c:\\Users\\bethanycheum\\Desktop\\AI-For-Beginners\\.venv\\lib\\site-packages\\huggingface_hub\\file_download.py:133: UserWarning: `huggingface_hub` cache-system uses symlinks by default to efficiently store duplicated files but your machine does not support them in C:\\Users\\bethanycheum\\.cache\\huggingface\\hub. Caching files will still work but in a degraded version that might require more space on your disk. This warning can be disabled by setting the `HF_HUB_DISABLE_SYMLINKS_WARNING` environment variable. For more details, see https://huggingface.co/docs/huggingface_hub/how-to-cache#limitations.\n",
"To support symlinks on Windows, you either need to activate Developer Mode or to run Python as an administrator. In order to see activate developer mode, see this article: https://docs.microsoft.com/en-us/windows/apps/get-started/enable-your-device-for-development\n",
" warnings.warn(message)\n",
"Some weights of OpenAIGPTLMHeadModel were not initialized from the model checkpoint at openai-gpt and are newly initialized: ['position_ids']\n",
"You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n",
"Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n"
"Downloading (…)neration_config.json: 100%|██████████| 74.0/74.0 [00:00<00:00, 48.8kB/s]\n",
"Downloading (…)olve/main/vocab.json: 100%|██████████| 816k/816k [00:00<00:00, 1.76MB/s]\n",
"Downloading (…)olve/main/merges.txt: 100%|██████████| 458k/458k [00:00<00:00, 1.11MB/s]\n",
"Downloading (…)/main/tokenizer.json: 100%|██████████| 1.27M/1.27M [00:00<00:00, 2.12MB/s]\n",
"Xformers is not installed correctly. If you want to use memory_efficient_attention to accelerate training use the following command to install Xformers\n",
"pip install xformers.\n"
]
},
{
"data": {
"text/plain": [
"[{'generated_text': 'Hello! I am a neural network, and I want to say that I am an expert in the area of learning and understanding neural networks. I also know a bit about math. You might have seen \"How to make deep neural networks\" or \"What is a deep learning neural network?\". I would like to discuss the first one.\\n\\nHow to make deep neural networks is a very complicated topic, and I don\\'t know how to go into it in detail, but I want to do some'},\n",
" {'generated_text': 'Hello! I am a neural network, and I want to say that you can find a network algorithm, called Naive Bayes, which produces good results, but which is computationally too expensive to be useful, so I will not cover all of its details here.\\n\\nFirst we are going to define how an NN looks like.\\n\\nHere is a naive Bayes classification problem with a few variables:\\n\\nFeature Input Value (x) 1 2 3 4 5 1 3 4'},\n",
" {'generated_text': 'Hello! I am a neural network, and I want to say that… \"I\\'m done with this crap!\"\\n\\nNow, if you were to be able to tell me all about this, then you did something wrong with the previous part. If you weren\\'t able to tell I did this before, sorry! If your brain was too wired, then you don\\'t know anything about neural networks. I mean, it turns out neural networks have a big problem. It\\'s called lossy neural'},\n",
" {'generated_text': 'Hello! I am a neural network, and I want to say that I learned this from Wikipedia but also from lots of other books. They also gave me some tips on how to do the analysis, which is really useful for beginners.\\n\\nFor the code, I just used Python and scikit-learn. But I will warn you with two caveats –\\n\\nfirst, this code will have to be imported before you can run them. You have to import scikit-learn after'},\n",
" {'generated_text': 'Hello! I am a neural network, and I want to say that you\\'re awesome.\"'}]"
"[{'generated_text': \"Hello! I am a neural network, and I want to say that i apologize for not coming to you yourself, for not helping you, and that i was too busy getting dressed and studying for a midterm. you know, the kind where the teachers are like that and they come in pairs with their boyfriends, but not with theirs. it's true, that i have had a girlfriend, and i'm only going on wednesdays and thursdays because i was too busy with college, but maybe\"},\n",
" {'generated_text': 'Hello! I am a neural network, and I want to say that we have been blessed with a wonderful gift ; no one of us has died at all. and our spirits are strong, very strong. in one very lucky moment of luck for you, all has been given direction and destiny, and for us there are no more mysteries. the earth has been chosen for you, and that earth is now ours, and you must be forever in our hearts. \" \\n the words, as one,'},\n",
" {'generated_text': 'Hello! I am a neural network, and I want to say that if you would just turn and face the general, you would have a nice day. \" \\n \" sure thing, \" said one of the soldiers, and started to run. the rest of the soldiers followed, shouting. the general turned to general zulu, raising his arm. the general said something in his native language, and the general immediately started to run. zulu started to move toward the wall, with the'},\n",
" {'generated_text': 'Hello! I am a neural network, and I want to say that i am not a doctor but an anthropologist to you, a specialist, a specialist in the field of astrobiological biology, and that i am very much involved in this investigation. i am not sure, i am not certain, but i can confirm your conclusions and therefore i will go to the top. i have a colleague who has just returned from this expedition and his findings confirm that you are a specialist. that is, he'},\n",
" {'generated_text': \"Hello! I am a neural network, and I want to say that everyone here is in agreement that no matter how many times i say to myself,'he was never a man of action on the battlefield,'or'he 'll never take a chance at killing any civilians,'or'he 'll never let his men go undefended against enemy forces of this caliber,'or'that's just what i need in a day like today. \\n you see, there are only three groups that\"}]"
]
},
"execution_count": 1,
@ -45,7 +57,7 @@
"source": [
"from transformers import pipeline\n",
"\n",
"model_name = 'gpt2-large' # try 'gpt2' for small model, 'gpt2-medium' for medium one\n",
"model_name = 'openai-gpt' \n",
"\n",
"generator = pipeline('text-generation', model=model_name)\n",
"\n",
@ -53,37 +65,31 @@
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Prompt Engineering\n",
"\n",
"In some of the problems, you can use GPT-2 generation right away by designing correct prompts. Have a look at the examples below:"
"In some of the problems, you can use openai-gpt generation right away by designing correct prompts. Have a look at the examples below:"
]
},
{
"cell_type": "code",
"execution_count": 10,
"execution_count": 2,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n"
]
},
{
"data": {
"text/plain": [
"[{'generated_text': 'Synonyms of a word cat:\\n\\n(a) cat;\\n\\n(b) black'},\n",
" {'generated_text': 'Synonyms of a word cat: cat-bitch; cat-queen; cat-c'},\n",
" {'generated_text': 'Synonyms of a word cat: feline, feline form, feline spirit, fas'},\n",
" {'generated_text': 'Synonyms of a word cat:\\n\\n(a) cat-like, like a cat;'},\n",
" {'generated_text': 'Synonyms of a word cat:\\n\\nThe more common English words you need to understand the meaning'}]"
"[{'generated_text': 'Synonyms of a word cat: the same cat i used to stare at, and you in'},\n",
" {'generated_text': 'Synonyms of a word cat: cat of the woods, cat of the hills, cat of'},\n",
" {'generated_text': 'Synonyms of a word cat: you! \\n \" it\\'s a girl. \" i said'},\n",
" {'generated_text': \"Synonyms of a word cat: big cat. but how come, we didn't hear it\"},\n",
" {'generated_text': 'Synonyms of a word cat: \" mea - o - c \" which makes them sound'}]"
]
},
"execution_count": 10,
"execution_count": 2,
"metadata": {},
"output_type": "execute_result"
}
@ -94,27 +100,20 @@
},
{
"cell_type": "code",
"execution_count": 25,
"execution_count": 3,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n"
]
},
{
"data": {
"text/plain": [
"[{'generated_text': 'I love when you say this -> Positive\\nI have myself -> Negative\\nThis is awful for you to say this -> Negative\\nHow do you think this hurts us all? -> Positive\\nYou seem'},\n",
" {'generated_text': \"I love when you say this -> Positive\\nI have myself -> Negative\\nThis is awful for you to say this -> Negative\\nThis is such a horrible way to treat yourself -> Negative (You're\"},\n",
" {'generated_text': 'I love when you say this -> Positive\\nI have myself -> Negative\\nThis is awful for you to say this -> I am disappointed\\nI have to admit -> You are a good friend but you'},\n",
" {'generated_text': 'I love when you say this -> Positive\\nI have myself -> Negative\\nThis is awful for you to say this -> Positive\\nThis is awful for me to do this with you -> Negative\\nWhy'},\n",
" {'generated_text': \"I love when you say this -> Positive\\nI have myself -> Negative\\nThis is awful for you to say this -> I'm still sad for you!\\nMe? \\xa0I find this a\"}]"
"[{'generated_text': 'I love when you say this -> Positive\\nI have myself -> Negative\\nThis is awful for you to say this -> positive this is so horrible - > positive that your brother is gay - >'},\n",
" {'generated_text': 'I love when you say this -> Positive\\nI have myself -> Negative\\nThis is awful for you to say this -> negative i will bring this on you -, < positive am i, i'},\n",
" {'generated_text': 'I love when you say this -> Positive\\nI have myself -> Negative\\nThis is awful for you to say this -> negative i have self - esteem i must take it - : \\n - -'},\n",
" {'generated_text': 'I love when you say this -> Positive\\nI have myself -> Negative\\nThis is awful for you to say this -> negative this is - : \\n if it were true that the devil would have'},\n",
" {'generated_text': \"I love when you say this -> Positive\\nI have myself -> Negative\\nThis is awful for you to say this -> positive i have you - > positive it's a bad thing, > positive\"}]"
]
},
"execution_count": 25,
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
@ -125,25 +124,18 @@
},
{
"cell_type": "code",
"execution_count": 20,
"execution_count": 4,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n"
]
},
{
"data": {
"text/plain": [
"[{'generated_text': 'Translate English to French: cat => chat, dog => chien, student => étudiant;\\n\\n\\nOther:\\n\\nTo learn'},\n",
" {'generated_text': 'Translate English to French: cat => chat, dog => chien, student => étude, french = le français = \"'},\n",
" {'generated_text': \"Translate English to French: cat => chat, dog => chien, student => été, mama => m'aime. We\"}]"
"[{'generated_text': 'Translate English to French: cat => chat, dog => chien, student => new and unusual. there were no more words to be'},\n",
" {'generated_text': 'Translate English to French: cat => chat, dog => chien, student => student \\n his eyes were huge in his lean face as'},\n",
" {'generated_text': \"Translate English to French: cat => chat, dog => chien, student => the teacher's words, their words, their words.\"}]"
]
},
"execution_count": 20,
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
@ -157,21 +149,14 @@
"execution_count": 5,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n"
]
},
{
"data": {
"text/plain": [
"[{'generated_text': 'People who liked the movie The Matrix also liked \\xa0\"Dawn of the Dead\" (or in the case of John Goodman \"Vanish\"),\\xa0but their votes didn\\'t matter! The first'},\n",
" {'generated_text': 'People who liked the movie The Matrix also liked _____. It\\'s a true statement, as \"tasteful\" movies are often more popular for their plot, characters and themes than for their entertainment'},\n",
" {'generated_text': 'People who liked the movie The Matrix also liked \\xa0the book... \\xa0And so on, and so forth. Now at the other end of the spectrum...\\n...the real \"truths'},\n",
" {'generated_text': \"People who liked the movie The Matrix also liked 『The Grand Budapest Hotel』. 『The Grand Budapest Hotel』 is the movie people who like to watch their dream come true. I'll explain\"},\n",
" {'generated_text': 'People who liked the movie The Matrix also liked \\xa0a lot of the lines that were used, and I got to listen to the dialogue in the movie. \\xa0The characters were really enjoyable characters'}]"
"[{'generated_text': 'People who liked the movie The Matrix also liked it, and there was the movie of the first man after us. \\n i wanted to laugh at how stupid these stupid actors were. no, they were'},\n",
" {'generated_text': \"People who liked the movie The Matrix also liked the movie, and the film was the result. and that's when the man in the story was brought into reality, after a few decades. \\n a\"},\n",
" {'generated_text': 'People who liked the movie The Matrix also liked the movie the matrix, because there was a very old movie movie called the matrix, where there was a great super hero, and the super hero came out'},\n",
" {'generated_text': \"People who liked the movie The Matrix also liked the movie that didn't have a chance to pay cash, if they could afford it. most often they got a good deal and a lot of money,\"},\n",
" {'generated_text': \"People who liked the movie The Matrix also liked the movie, and i didn't seem to have the same problem. \\n i 'd met the other half of my family. i spent most of my time\"}]"
]
},
"execution_count": 5,
@ -194,27 +179,20 @@
},
{
"cell_type": "code",
"execution_count": 28,
"execution_count": 6,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n"
]
},
{
"data": {
"text/plain": [
"[{'generated_text': \"It was early evening when I can back from work. I usually work late, but this time it was an exception. When I entered a room, I saw my girlfriend with a very cute looking guy. I noticed that his eyes that looked into mine were a bit scary and kind of like a predator's eyes. And that's when I knew that I had to know something about him.\\n\\nAs I sat, studying him, my mind was racing in a crazy way. We didn't exactly\"},\n",
" {'generated_text': 'It was early evening when I can back from work. I usually work late, but this time it was an exception. When I entered a room, I saw my reflection in a small screen attached to a thin, white piece of glass. Suddenly, I thought about the possibility of making money making a computer, and decided to try it for myself! I started working on making an e-ink display. I was able to make it work by following some simple rules!\\n\\nThe first step is'},\n",
" {'generated_text': 'It was early evening when I can back from work. I usually work late, but this time it was an exception. When I entered a room, I saw a pile of clothing and furniture. They were completely ruined. The place had been abandoned. It had been a home, not a work place.\\n\\nWhen the police arrived, they found the crime scene and I walked with two officers to the police station. Before that I had not gone to the police station and I have never been to'},\n",
" {'generated_text': 'It was early evening when I can back from work. I usually work late, but this time it was an exception. When I entered a room, I saw a woman who could not be a mother of any kind. I saw the same woman with her eyes closed and her mouth wide open. She was sitting naked on a couch and was sitting topless. I saw at least three different women with similar posture. The only thing I kept telling myself was \"if this is how a woman eats,'},\n",
" {'generated_text': \"It was early evening when I can back from work. I usually work late, but this time it was an exception. When I entered a room, I saw a pale person with long white hair sitting quietly. He was a bit bigger and his eyes looked different. He was watching me.\\n\\nI didn't have to say anything. The man kept studying me, and after a while he turned his eyes away from me to the floor. In a moment he was gone, and he was still\"}]"
"[{'generated_text': 'It was early evening when I can back from work. I usually work late, but this time it was an exception. When I entered a room, I saw my friend, a young man, sprawled across the bed in his bed. \\n \" hi, i\\'m mike eptirard. \" \\n there was silence on the other side of the door. i listened for any trace of life but there was nothing. my heart began to pound, i was starting to sweat, i took out my wallet'},\n",
" {'generated_text': 'It was early evening when I can back from work. I usually work late, but this time it was an exception. When I entered a room, I saw my mother on the bed, hugging her legs to her chest and sobbing. i saw my dad and mother from the corner of my eye. \\n elfin face was covered in tears as i entered the room. my dad and mother also wept ; just as they did every other time i came to work. but this time, they had different faces'},\n",
" {'generated_text': 'It was early evening when I can back from work. I usually work late, but this time it was an exception. When I entered a room, I saw the room had changed because it was dark. it still smelled like a hospital. a new light shined through from a vent in the ceiling. i found myself in a bathroom and a small room with a sink and a wall of glass. the bathroom billion years ago. not so different from all of the rest of the apartment. \\n now...'},\n",
" {'generated_text': 'It was early evening when I can back from work. I usually work late, but this time it was an exception. When I entered a room, I saw a large woman with dark hair and pale skin. she was asleep, but i noticed a faint movement of her face. i could sense she was awake. i got up and walked over to her. \\n \" hello miss. i am inspector michael o\\'dell ; we are investigating the case against you. i wanted to ask if you were the'},\n",
" {'generated_text': 'It was early evening when I can back from work. I usually work late, but this time it was an exception. When I entered a room, I saw i had an empty table and three empty chairs. that was all i needed. i had left a note on a table in the center of the room and had a pen in hand. \" \\n \" i think what you were doing was something he was doing to her. \" \\n \" yeah, \" i nodded with a grin. \" i'}]"
]
},
"execution_count": 28,
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
@ -233,27 +211,20 @@
},
{
"cell_type": "code",
"execution_count": 29,
"execution_count": 7,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n"
]
},
{
"data": {
"text/plain": [
"[{'generated_text': 'It was early evening when I can back from work. I usually work late, but this time it was an exception. When I entered a room, I saw a group of people sitting around a table. One of them was a middle-aged man. He was wearing a black suit and a white shirt. His eyes were closed and he was staring at the floor with his hands on his knees.\\n\\n\"What are you doing here?\" I asked him. \"What do you want?\"\\n'},\n",
" {'generated_text': 'It was early evening when I can back from work. I usually work late, but this time it was an exception. When I entered a room, I saw a man sitting at a table in front of a computer. He was wearing a black suit, a white shirt, and a red tie.\\n\\n\"Hello,\" he said to me. \"How are you?\" he asked me in a voice that sounded as if he was speaking to a child. There was a smile on his face.'},\n",
" {'generated_text': 'It was early evening when I can back from work. I usually work late, but this time it was an exception. When I entered a room, I saw a man sitting on the edge of the bed. He was staring at me.\\n\\n\"What are you doing here?\" he asked in a low voice. \"I don\\'t know why you\\'re here, and I\\'m not interested in what you have to say,\" he said, as if he didn\\'t understand what I was saying.'},\n",
" {'generated_text': 'It was early evening when I can back from work. I usually work late, but this time it was an exception. When I entered a room, I saw a man sitting in a chair in front of a desk. He was in his mid-thirties, and he was wearing a dark suit and a white shirt.\\n\\n\"What are you doing here?\" I asked him. The man didn\\'t answer. Instead, he walked over to the desk and sat down on it. \"'},\n",
" {'generated_text': 'It was early evening when I can back from work. I usually work late, but this time it was an exception. When I entered a room, I saw that the door was open, and a man was sitting on a chair. He was wearing a white shirt and black pants.\\n\\n\"What are you doing here?\" I asked. The man looked at me for a moment, then said, \"I have something to tell you.\" He handed me a piece of paper. It was a'}]"
"[{'generated_text': 'It was early evening when I can back from work. I usually work late, but this time it was an exception. When I entered a room, I saw a man sitting in a chair with his head in his hands. he didn\\'t look up as i approached. \\n \" excuse me, sir, \" i said. \" can i help you? \" \\n the man looked up at me. his eyes were red - rimmed and his face was pale, as if he hadn\\'t slept in days'},\n",
" {'generated_text': 'It was early evening when I can back from work. I usually work late, but this time it was an exception. When I entered a room, I saw a man sitting at a desk in the middle of the room. he had his back to me, so i couldn\\'t see what he was doing. \" \\n \" what did he look like? \" i asked as i sat down on the bed next to her. \\n she took a deep breath and looked at me with tears in her eyes'},\n",
" {'generated_text': 'It was early evening when I can back from work. I usually work late, but this time it was an exception. When I entered a room, I saw a woman sitting on the bed, reading a book. she looked up at me and smiled. \\n \" hi, \" she said. \" can i help you? \" \\n i sat down next to her and looked around the room. the walls were white, and there was a large window in the middle of the wall that looked out on'},\n",
" {'generated_text': 'It was early evening when I can back from work. I usually work late, but this time it was an exception. When I entered a room, I saw a man sitting at a table in the middle of the room. he looked up as i walked in, and when he saw me, he got up and walked over to me. \\n \" can i help you? \" he asked as he put his hand on the small of my back and led me to a chair at the other end of'},\n",
" {'generated_text': 'It was early evening when I can back from work. I usually work late, but this time it was an exception. When I entered a room, I saw a woman sitting on the edge of her bed, reading a book. she looked up at me and smiled. \\n \" hello, \" she said. \" can i help you? \" \\n i didn\\'t know what to say, so i just sat down in the chair next to the bed and looked at her. her hair was dark brown'}]"
]
},
"execution_count": 29,
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
@ -272,23 +243,16 @@
},
{
"cell_type": "code",
"execution_count": 30,
"execution_count": 8,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n"
]
},
{
"data": {
"text/plain": [
"[{'generated_text': 'It was early evening when I can back from work. I usually work late, but this time it was an exception. When I entered a room, I saw one of my colleagues lying on his stomach. He was unconscious. I could smell a strong odor of alcohol on his breath. I was on my way out. I ran to his room and found him unconscious on the bed. I saw a bottle of wine on the bed. He was in a state of intoxication. He was unconscious. I'}]"
"[{'generated_text': 'It was early evening when I can back from work. I usually work late, but this time it was an exception. When I entered a room, I saw her. she was on the bed, but she looked very different. \\n \" honey, what\\'s the matter? \" i asked. \\n she sat up. \" i can\\'t believe it\\'s real. i\\'ve been dreaming about you for the last two days. \" \\n \" i can\\'t believe it either. i guess that\\'s how'}]"
]
},
"execution_count": 30,
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
@ -310,12 +274,13 @@
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Fine-Tuning GPT-2\n",
"## Fine-Tuning your models\n",
"\n",
"You can also fine-tune GPT-2 text generation on your own dataset. This will allow you to adjust the style of text, while keeping the major part of language model. The example of fine-tuning GPT-2 to generate song lyrics can be found [in this blog post](https://towardsdatascience.com/how-to-fine-tune-gpt-2-for-text-generation-ae2ea53bc272)."
"You can also [fine-tune your model](https://learn.microsoft.com/en-us/azure/cognitive-services/openai/how-to/fine-tuning?pivots=programming-language-studio?WT.mc_id=academic-77998-bethanycheum) on your own dataset. This will allow you to adjust the style of text, while keeping the major part of language model. "
]
}
],
@ -338,7 +303,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.12"
"version": "3.10.11"
},
"orig_nbformat": 4
},

26
lessons/5-NLP/20-LangModels/README.md
View File

@ -23,34 +23,28 @@ $$
## GPT is a Family
GPT is not a single model, but rather a collection of models developed and trained by [OpenAI](http://openai.org). The latest model openly available is [GPT-2](https://huggingface.co/docs/transformers/model_doc/gpt2#openai-gpt2), which has up to 1.5 billion parameters (there are several variations of the model, so you can select one for your tasks that is a good compromise between size/performance). Latest GPT-3 model has up to 175 billion parameters, and is available [as a cognitive service from Microsoft Azure](https://azure.microsoft.com/en-us/services/cognitive-services/openai-service/#overview?WT.mc_id=academic-77998-cacaste), and as [OpenAI API](https://openai.com/api/).
GPT is not a single model, but rather a collection of models developed and trained by [OpenAI](https://openai.com).
## Prompt-based Inference
Under the GPT models, we have GPT-2, GPT-3, GPT-3.5 and now, GPT-4. [GPT-2](https://huggingface.co/docs/transformers/model_doc/gpt2#openai-gpt2), has up to 1.5 billion parameters. Next we have the GPT-3 model has up to 175 billion parameters which is 116 times larger than GPT-2.
Because GPT has been trained on a vast volumes of data, it has some commonsense knowledge embedded directly inside the model. This allows us to force GPT to solve certain typical problems by just providing the right prompt. This presents a whole new approach for using pre-trained models, called [Prompt Engineering](https://en.wikipedia.org/wiki/Prompt_engineering). It is particularly useful with GPT-3, which has significantly more parameters, and consequently more embedded knowledge.
The latest model openly available is [GPT-4](https://openai.com/gpt-4), which is a large multimodal model. GPT-4 accepts both image and text inputs and outputs text. The difference between GPT-3.5 and GPT-4 is subtle, but GPT-4 offere more reliable and creative output as well as is able to handle more nuanced instructions compared to GPT-3.5. [Learn more about GPT-4](https://openai.com/research/gpt-4)
Here are a few example of using Prompt Engineering (answers from the model are *in italics*):
The GPT-3 and GPT-4 models are available [as a cognitive service from Microsoft Azure](https://azure.microsoft.com/en-us/services/cognitive-services/openai-service/#overview?WT.mc_id=academic-77998-cacaste), and as [OpenAI API](https://openai.com/api/).
**Recommendation Systems**:<br/>
People, who liked the movie "The Matrix" also liked *Star Wars, Jupyter Ascending, Ex Machina*
## Prompt Engineering
**Translation**:<br/>
Translate from English to French:<br/>
cat => chat, dog => chien, student => *étudiant*
Because GPT has been trained on a vast volumes of data to understand language and code, they provide outputs in response to inputs (prompts). Prompts are GPT inputs or queries whereby one provides instructions to models on tasks they next completed. To elicit a desired outcome, you need the most effective prompt which involves selecting the right words, formats, phrases or even symbols. This approach is [Prompt Engineering](https://learn.microsoft.com/en-us/shows/ai-show/the-basics-of-prompt-engineering-with-azure-openai-service?WT.mc_id=academic-77998-bethanycheum)
**Looking for words:**<br/>
Synonyms of a word cat: *feline, feline form, feline spirit*
[This documentation](https://learn.microsoft.com/en-us/semantic-kernel/prompt-engineering/?WT.mc_id=academic-77998-bethanycheum) provides you with more information on prompt engineering.
[This article](https://www.gwern.net/GPT-3#prompts-as-programming) talks more about prompt engineering.
## ✍️ Example Notebook: [Playing with GPT-2](GPT-PyTorch.ipynb)
## ✍️ Example Notebook: [Playing with OpenAI-GPT](GPT-PyTorch.ipynb)
Continue your learning in the following notebooks:
* [Generating text with GPT-2 and Hugging Face Transformers](GPT-PyTorch.ipynb)
* [Generating text with OpenAI-GPT and Hugging Face Transformers](GPT-PyTorch.ipynb)
## Conclusion
New general pre-trained language models do not only model language structure, but also contain vast amount of commonsense knowledge. Thus, they can be effectively used to solve some NLP tasks in zero-shop or few-shot settings.
New general pre-trained language models do not only model language structure, but also contain vast amount of natural language. Thus, they can be effectively used to solve some NLP tasks in zero-shop or few-shot settings.
## [Post-lecture quiz](https://red-field-0a6ddfd03.1.azurestaticapps.net/quiz/220)