RomanNum3ral
/
AI-For-Beginners
mirror of https://github.com/microsoft/AI-For-Beginners.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
AI-For-Beginners/translations/pcm/lessons/5-NLP/17-GenerativeNetworks/GenerativeTF.ipynb

497 lines
27 KiB
Plaintext
Raw Blame History

{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Generative networks\n",
"\n",
"Recurrent Neural Networks (RNNs) and di gated cell dem like Long Short Term Memory Cells (LSTMs) and Gated Recurrent Units (GRUs) don give us way to do language modeling. Dis one mean say dem fit learn how words dey arrange and fit predict di next word for one sequence. Dis one make am possible to use RNNs for **generative tasks**, like normal text generation, machine translation, and even image captioning.\n",
"\n",
"For di RNN architecture wey we talk about for di last unit, each RNN unit dey produce di next hidden state as output. But, we fit still add another output to each recurrent unit, wey go allow us output one **sequence** (wey go get di same length as di original sequence). Plus, we fit use RNN units wey no dey collect input for every step, but go just take one initial state vector, and then produce one sequence of outputs.\n",
"\n",
"For dis notebook, we go focus on simple generative models wey go help us generate text. To make am simple, make we build **character-level network**, wey go generate text letter by letter. For training, we go need take one text corpus, and split am into letter sequences. \n"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"import tensorflow as tf\n",
"from tensorflow import keras\n",
"import tensorflow_datasets as tfds\n",
"import numpy as np\n",
"\n",
"ds_train, ds_test = tfds.load('ag_news_subset').values()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## How to build character vocabulary\n",
"\n",
"To build generative network wey dey work for character level, we go need split text into each character instead of words. `TextVectorization` layer wey we don dey use before no fit do am, so we get two options:\n",
"\n",
"* Load text manually and do tokenization 'by hand', like for [this official Keras example](https://keras.io/examples/generative/lstm_character_level_text_generation/)\n",
"* Use `Tokenizer` class to do tokenization for character level.\n",
"\n",
"We go use the second option. `Tokenizer` fit also tokenize into words, so e go easy to switch from char-level to word-level tokenization.\n",
"\n",
"To do tokenization for character level, we go need pass `char_level=True` parameter:\n"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
"def extract_text(x):\n",
" return x['title']+' '+x['description']\n",
"\n",
"def tupelize(x):\n",
" return (extract_text(x),x['label'])\n",
"\n",
"tokenizer = keras.preprocessing.text.Tokenizer(char_level=True,lower=False)\n",
"tokenizer.fit_on_texts([x['title'].numpy().decode('utf-8') for x in ds_train])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We wan use one special token wey go mean **end of sequence**, we go call am `<eos>`. Make we add am manually to the vocabulary:\n"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
"eos_token = len(tokenizer.word_index)+1\n",
"tokenizer.word_index['<eos>'] = eos_token\n",
"\n",
"vocab_size = eos_token + 1"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now, to change text to number sequence, we fit use:\n"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[[48, 2, 10, 10, 5, 44, 1, 25, 5, 8, 10, 13, 78]]"
]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"tokenizer.texts_to_sequences(['Hello, world!'])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## How we go take train generative RNN to fit generate titles\n",
"\n",
"Di way we go take train RNN to dey generate news titles na like dis. For each step, we go carry one title, wey we go put inside RNN, and for each character wey we put as input, we go tell di network make e generate di next character:\n",
"\n",
"![Image wey dey show example of RNN wey dey generate di word 'HELLO'.](../../../../../translated_images/pcm/rnn-generate.56c54afb52f9781d.webp)\n",
"\n",
"For di last character for our sequence, we go tell di network make e generate `<eos>` token.\n",
"\n",
"Di main difference for di generative RNN wey we dey use here be say we go dey collect output from each step of di RNN, no be only from di final cell. We fit do dis one by setting `return_sequences` parameter for di RNN cell.\n",
"\n",
"So, for di training, di input wey we go give di network go be sequence of encoded characters wey get some length, and di output go be sequence of di same length, but e go shift by one element and e go end with `<eos>`. Minibatch go get plenty of dis kind sequences, and we go need use **padding** to make all di sequences align.\n",
"\n",
"Make we create functions wey go help us transform di dataset. Because we wan pad sequences for minibatch level, we go first batch di dataset by calling `.batch()`, and then we go use `map` to do di transformation. So, di transformation function go take di whole minibatch as parameter:\n"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [],
"source": [
"def title_batch(x):\n",
" x = [t.numpy().decode('utf-8') for t in x]\n",
" z = tokenizer.texts_to_sequences(x)\n",
" z = tf.keras.preprocessing.sequence.pad_sequences(z)\n",
" return tf.one_hot(z,vocab_size), tf.one_hot(tf.concat([z[:,1:],tf.constant(eos_token,shape=(len(z),1))],axis=1),vocab_size)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Some important things we dey do for here:\n",
"* First, we go comot the real text from the string tensor\n",
"* `text_to_sequences` go change the list of strings to list of integer tensors\n",
"* `pad_sequences` go add padding for those tensors make dem reach dia maximum length\n",
"* At last, we go one-hot encode all the characters, plus do the shifting and `<eos>` join. We go soon see why we need one-hot-encoded characters\n",
"\n",
"But, dis function na **Pythonic**, e mean say e no fit automatically change to Tensorflow computational graph. If we try use dis function directly inside `Dataset.map` function, e go show error. We need to wrap dis Pythonic call by using `py_function` wrapper:\n"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [],
"source": [
"def title_batch_fn(x):\n",
" x = x['title']\n",
" a,b = tf.py_function(title_batch,inp=[x],Tout=(tf.float32,tf.float32))\n",
" return a,b"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"> **Note**: To sabi di difference between Pythonic and Tensorflow transformation functions fit be like e dey too hard small, and you fit dey wonder why we no dey use standard Python functions take transform di dataset before we pass am to `fit`. Even though e fit work like dat, using `Dataset.map` get big advantage, because di data transformation pipeline go run with Tensorflow computational graph, wey go use GPU computations well well, and e go reduce di need to dey pass data between CPU/GPU.\n",
"\n",
"Now we fit build our generator network and start to train am. E fit use any recurrent cell wey we talk about for di last unit (simple, LSTM or GRU). For our example, we go use LSTM.\n",
"\n",
"Because di network dey take characters as input, and di vocabulary size no too big, we no need embedding layer, one-hot-encoded input fit enter di LSTM cell directly. Di output layer go be `Dense` classifier wey go change di LSTM output into one-hot-encoded token numbers.\n",
"\n",
"Plus, since we dey work with variable-length sequences, we fit use `Masking` layer to create mask wey go ignore di padded part of di string. Dis one no dey strictly necessary, because we no too dey focus on anything wey dey after `<eos>` token, but we go use am make we sabi how dis layer type dey work. `input_shape` go be `(None, vocab_size)`, where `None` mean di sequence fit get any length, and di output shape go still be `(None, vocab_size)`, as you fit see from di `summary`:\n"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Model: \"sequential\"\n",
"_________________________________________________________________\n",
"Layer (type) Output Shape Param # \n",
"=================================================================\n",
"masking (Masking) (None, None, 84) 0 \n",
"_________________________________________________________________\n",
"lstm (LSTM) (None, None, 128) 109056 \n",
"_________________________________________________________________\n",
"dense (Dense) (None, None, 84) 10836 \n",
"=================================================================\n",
"Total params: 119,892\n",
"Trainable params: 119,892\n",
"Non-trainable params: 0\n",
"_________________________________________________________________\n",
"15000/15000 [==============================] - 229s 15ms/step - loss: 1.5385\n"
]
},
{
"data": {
"text/plain": [
"<tensorflow.python.keras.callbacks.History at 0x7fa40c1245e0>"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"model = keras.models.Sequential([\n",
" keras.layers.Masking(input_shape=(None,vocab_size)),\n",
" keras.layers.LSTM(128,return_sequences=True),\n",
" keras.layers.Dense(vocab_size,activation='softmax')\n",
"])\n",
"\n",
"model.summary()\n",
"model.compile(loss='categorical_crossentropy')\n",
"\n",
"model.fit(ds_train.batch(8).map(title_batch_fn))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## How to take output comot\n",
"\n",
"Now we don train di model, we wan use am take generate some output. First, we go need way wey go fit decode text wey dem represent as sequence of token numbers. To do dis one, we fit use `tokenizer.sequences_to_texts` function; but e no dey work well wit character-level tokenization. So, we go carry di dictionary of tokens from di tokenizer (dem dey call am `word_index`), build reverse map, and write our own decoding function:\n"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [],
"source": [
"reverse_map = {val:key for key, val in tokenizer.word_index.items()}\n",
"\n",
"def decode(x):\n",
" return ''.join([reverse_map[t] for t in x])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Okay, make we do generation. We go start wit one string `start`, encode am inside one sequence `inp`, and then for each step we go call our network to find di next character.\n",
"\n",
"Di output wey di network `out` go give na one vector wey get `vocab_size` elements wey dey represent di probabilities of each token, and we fit find di token wey get di highest probability by using `argmax`. After dat, we go add dis character to di list of tokens wey we don generate, and continue di generation process. Dis process wey dey generate one character go repeat `size` times to generate di number of characters wey we need, and we go stop early if we see `eos_token`.\n"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"'Today #39;s lead to strike for the strike for the strike for the strike (AFP)'"
]
},
"execution_count": 12,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"def generate(model,size=100,start='Today '):\n",
" inp = tokenizer.texts_to_sequences([start])[0]\n",
" chars = inp\n",
" for i in range(size):\n",
" out = model(tf.expand_dims(tf.one_hot(inp,vocab_size),0))[0][-1]\n",
" nc = tf.argmax(out)\n",
" if nc==eos_token:\n",
" break\n",
" chars.append(nc.numpy())\n",
" inp = inp+[nc]\n",
" return decode(chars)\n",
" \n",
"generate(model)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Sampling output during training \n",
"\n",
"Since we no get any useful metrics like *accuracy*, di only way we fit take see say our model dey improve na by **sampling** di string wey e generate during training. To do am, we go use **callbacks**, wey be functions wey we fit pass give di `fit` function, and dem go dey call from time to time during training.\n"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Epoch 1/3\n",
"15000/15000 [==============================] - 226s 15ms/step - loss: 1.2703\n",
"Today #39;s a lead in the company for the strike\n",
"Epoch 2/3\n",
"15000/15000 [==============================] - 227s 15ms/step - loss: 1.2057\n",
"Today #39;s the Market Service on Security Start (AP)\n",
"Epoch 3/3\n",
"15000/15000 [==============================] - 226s 15ms/step - loss: 1.1752\n",
"Today #39;s a line on the strike to start for the start\n"
]
},
{
"data": {
"text/plain": [
"<tensorflow.python.keras.callbacks.History at 0x7fa40c74e3d0>"
]
},
"execution_count": 13,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"sampling_callback = keras.callbacks.LambdaCallback(\n",
" on_epoch_end = lambda batch, logs: print(generate(model))\n",
")\n",
"\n",
"model.fit(ds_train.batch(8).map(title_batch_fn),callbacks=[sampling_callback],epochs=3)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Dis example dey already generate beta text, but e fit still beta well-well in plenty ways:\n",
"\n",
"* **Add more text**. We don only use title for di task, but you fit wan try use full text. Remember say RNNs no dey too sabi handle long sequence, so e go make sense if you split dem into short sentence, or you dey always train am for fixed sequence length wey get some predefined value `num_chars` (like 256). You fit try change di example wey dey up to dis kain architecture, use [official Keras tutorial](https://keras.io/examples/generative/lstm_character_level_text_generation/) as inspiration.\n",
"\n",
"* **Multilayer LSTM**. E go make sense if you try 2 or 3 layers of LSTM cells. As we don talk for di previous unit, each layer of LSTM dey extract some kain pattern from text, and for character-level generator, we fit expect say di lower LSTM level go dey responsible for extracting syllables, while di higher levels go dey handle words and word combinations. You fit simply implement am by passing number-of-layers parameter to di LSTM constructor.\n",
"\n",
"* You fit wan try experiment with **GRU units** to see which one go perform better, and also try **different hidden layer sizes**. If di hidden layer too big, e fit cause overfitting (like say di network go dey learn di exact text), and if e too small, e fit no produce beta result.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Soft text generation and temperature\n",
"\n",
"For di definition wey we give for `generate` before, we dey always pick di character wey get di highest probability as di next character for di text wey we dey generate. Dis one dey make di text dey \"repeat\" di same character sequence again and again, like dis example:\n",
"```\n",
"today of the second the company and a second the company ...\n",
"```\n",
"\n",
"But if we check di probability distribution for di next character, e fit be say di difference between di few characters wey get di highest probabilities no too big, e.g. one character fit get probability 0.2, another one fit get 0.19, etc. For example, if we dey look for di next character for di sequence '*play*', di next character fit be space or **e** (like for di word *player*).\n",
"\n",
"Dis one dey show say e no dey always \"fair\" to pick di character wey get di higher probability, because if we pick di second highest, e fit still give us meaningful text. E go make sense if we **sample** characters from di probability distribution wey di network output give us.\n",
"\n",
"We fit use `np.multinomial` function to do dis sampling, and dis function dey implement wetin dem dey call **multinomial distribution**. Below na di function wey dey implement dis **soft** text generation:\n"
]
},
{
"cell_type": "code",
"execution_count": 33,
"metadata": {
"scrolled": true
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"--- Temperature = 0.3\n",
"Today #39;s strike #39; to start at the store return\n",
"On Sunday PO to Be Data Profit Up (Reuters)\n",
"Moscow, SP wins straight to the Microsoft #39;s control of the space start\n",
"President olding of the blast start for the strike to pay &lt;b&gt;...&lt;/b&gt;\n",
"Little red riding hood ficed to the spam countered in European &lt;b&gt;...&lt;/b&gt;\n",
"\n",
"--- Temperature = 0.8\n",
"Today countie strikes ryder missile faces food market blut\n",
"On Sunday collores lose-toppy of sale of Bullment in &lt;b&gt;...&lt;/b&gt;\n",
"Moscow, IBM Diffeiting in Afghan Software Hotels (Reuters)\n",
"President Ol Luster for Profit Peaced Raised (AP)\n",
"Little red riding hood dace on depart talks #39; bank up\n",
"\n",
"--- Temperature = 1.0\n",
"Today wits House buiting debate fixes #39; supervice stake again\n",
"On Sunday arling digital poaching In for level\n",
"Moscow, DS Up 7, Top Proble Protest Caprey Mamarian Strike\n",
"President teps help of roubler stepted lessabul-Dhalitics (AFP)\n",
"Little red riding hood signs on cash in Carter-youb\n",
"\n",
"--- Temperature = 1.3\n",
"Today wits flawer ro, pSIA figat's co DroftwavesIs Talo up\n",
"On Sunday hround elitwing wint EU Powerburlinetien\n",
"Moscow, Bazz #39;s sentries olymen winnelds' next for Olympite Huc?\n",
"President lost securitys from power Elections in Smiltrials\n",
"Little red riding hood vides profit, exponituity, profitmainalist-at said listers\n",
"\n",
"--- Temperature = 1.8\n",
"Today #39;It: He deat: N.KA Asside\n",
"On Sunday i arry Par aldeup patient Wo stele1\n"
]
},
{
"ename": "KeyError",
"evalue": "0",
"output_type": "error",
"traceback": [
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[0;31mKeyError\u001b[0m Traceback (most recent call last)",
"\u001b[0;32m<ipython-input-33-db32367a0feb>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m 18\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34mf\"\\n--- Temperature = {i}\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 19\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mj\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mrange\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m5\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 20\u001b[0;31m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mgenerate_soft\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmodel\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0msize\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m300\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mstart\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mwords\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mj\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mtemperature\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mi\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
"\u001b[0;32m<ipython-input-33-db32367a0feb>\u001b[0m in \u001b[0;36mgenerate_soft\u001b[0;34m(model, size, start, temperature)\u001b[0m\n\u001b[1;32m 11\u001b[0m \u001b[0mchars\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mnc\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 12\u001b[0m \u001b[0minp\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0minp\u001b[0m\u001b[0;34m+\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mnc\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 13\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mdecode\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mchars\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 14\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 15\u001b[0m \u001b[0mwords\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m'Today '\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m'On Sunday '\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m'Moscow, '\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m'President '\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m'Little red riding hood '\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;32m<ipython-input-10-3f5fa6130b1d>\u001b[0m in \u001b[0;36mdecode\u001b[0;34m(x)\u001b[0m\n\u001b[1;32m 2\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 3\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mdecode\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 4\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0;34m''\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mjoin\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mreverse_map\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mt\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mt\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mx\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
"\u001b[0;32m<ipython-input-10-3f5fa6130b1d>\u001b[0m in \u001b[0;36m<listcomp>\u001b[0;34m(.0)\u001b[0m\n\u001b[1;32m 2\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 3\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mdecode\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 4\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0;34m''\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mjoin\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mreverse_map\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mt\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mt\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mx\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
"\u001b[0;31mKeyError\u001b[0m: 0"
]
}
],
"source": [
"def generate_soft(model,size=100,start='Today ',temperature=1.0):\n",
" inp = tokenizer.texts_to_sequences([start])[0]\n",
" chars = inp\n",
" for i in range(size):\n",
" out = model(tf.expand_dims(tf.one_hot(inp,vocab_size),0))[0][-1]\n",
" probs = tf.exp(tf.math.log(out)/temperature).numpy().astype(np.float64)\n",
" probs = probs/np.sum(probs)\n",
" nc = np.argmax(np.random.multinomial(1,probs,1))\n",
" if nc==eos_token:\n",
" break\n",
" chars.append(nc)\n",
" inp = inp+[nc]\n",
" return decode(chars)\n",
"\n",
"words = ['Today ','On Sunday ','Moscow, ','President ','Little red riding hood ']\n",
" \n",
"for i in [0.3,0.8,1.0,1.3,1.8]:\n",
" print(f\"\\n--- Temperature = {i}\")\n",
" for j in range(5):\n",
" print(generate_soft(model,size=300,start=words[j],temperature=i))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We don add one new parameter wey dem call **temperature**, e dey show how strong we go hold the highest probability. If temperature na 1.0, we go do fair multinomial sampling, but if temperature go infinity - all the probabilities go be the same, and we go randomly choose the next character. For the example wey dey below, we fit see say the text go dey meaningless if we increase the temperature too much, and e go resemble \"cycled\" hard-generated text if e near 0.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"---\n\n<!-- CO-OP TRANSLATOR DISCLAIMER START -->\n**Disclaimer**: \nDis dokyument don use AI transle-shon service [Co-op Translator](https://github.com/Azure/co-op-translator) do di transle-shon. Even as we dey try make am accurate, abeg make you sabi say transle-shon wey machine do fit get mistake or no dey correct well. Di original dokyument for im native language na di one wey you go take as di correct source. For important mata, e good make you use professional human transle-shon. We no go fit take blame for any misunderstanding or wrong interpretation wey fit happen because you use dis transle-shon.\n<!-- CO-OP TRANSLATOR DISCLAIMER END -->\n"
]
}
],
"metadata": {
"interpreter": {
"hash": "16af2a8bbb083ea23e5e41c7f5787656b2ce26968575d8763f2c4b17f9cd711f"
},
"kernelspec": {
"display_name": "Python 3.8.12 ('py38')",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.12"
},
"coopTranslator": {
"original_hash": "9fbb7d5fda708537649f71f5f646fcde",
"translation_date": "2025-11-18T19:32:47+00:00",
"source_file": "lessons/5-NLP/17-GenerativeNetworks/GenerativeTF.ipynb",
"language_code": "pcm"
}
},
"nbformat": 4,
"nbformat_minor": 4
}
Reference in New Issue View Git Blame Copy Permalink