From 09c0211e3accc4d4101beba0c448574f1a6e98c0 Mon Sep 17 00:00:00 2001 From: Dmitri Soshnikov Date: Tue, 17 May 2022 15:02:30 +0300 Subject: [PATCH] Add text generation lab and takeaways --- README.md | 2 +- lessons/5-NLP/17-GenerativeNetworks/README.md | 14 +++++++++++++- lessons/5-NLP/17-GenerativeNetworks/lab/README.md | 12 ++++++++++++ 3 files changed, 26 insertions(+), 2 deletions(-) create mode 100644 lessons/5-NLP/17-GenerativeNetworks/lab/README.md diff --git a/README.md b/README.md index e123bbce..9166eb95 100644 --- a/README.md +++ b/README.md @@ -78,7 +78,7 @@ For a gentle introduction to *AI in the Cloud* topics you may consider taking th 14Semantic word embeddings. Word2Vec and GloVeTextPyTorchTensorFlow 15Language Modeling. Training your own embeddingsTextPyTorchTensorFlow 16Recurrent Neural NetworksTextPyTorchTensorFlow -17Generative Recurrent NetworksTextPyTorchTensorFlow +17Generative Recurrent NetworksTextPyTorchTensorFlowLab 18Transformers. BERT.TextPyTorchTensorFlow 19Named Entity RecognitionTextPyTorchTensorFlow 20Large Language Models, Prompt Programming and Few-Shot TasksTextPyTorchTensorFlow diff --git a/lessons/5-NLP/17-GenerativeNetworks/README.md b/lessons/5-NLP/17-GenerativeNetworks/README.md index 5b3002f5..8a914332 100644 --- a/lessons/5-NLP/17-GenerativeNetworks/README.md +++ b/lessons/5-NLP/17-GenerativeNetworks/README.md @@ -50,4 +50,16 @@ Have a look at how this soft text generation is implemented in the notebooks. ## [Post-lecture quiz](https://black-ground-0cc93280f.1.azurestaticapps.net/quiz/217) -> ✅ Todo: conclusion, Assignment, challenge, reference. +## [Assignment](lab/README.md) + +We have seen how to generate text character-by-character. In the lab, you will explore word-level text generation. + +## References + +* Different approaches to text generation with Markov Chain, LSTM and GPT-2: [blog post](https://towardsdatascience.com/text-generation-gpt-2-lstm-markov-chain-9ea371820e1e) +* Learn Unit on Text Generation with [PyTorch](https://docs.microsoft.com/learn/modules/intro-natural-language-processing-pytorch/6-generative-networks/?WT.mc_id=academic-15963-dmitryso)/[TensorFlow](https://docs.microsoft.com/learn/modules/intro-natural-language-processing-tensorflow/5-generative-networks/?WT.mc_id=academic-15963-dmitryso) +* Text generation sample in [Keras documentation](https://keras.io/examples/generative/lstm_character_level_text_generation/) + +## Takeaway + +While text generation may be useful in its own right, the major benefits come from the ability to generate text using RNNs from some initial feature vector. For example, text generation is used as part of machine translation (sequence-to-sequence, in this case state vector from *encoder* is used to generate or *decode* translated message), generating textual description of an image (in which case feature vector would come from CNN extractor), etc. diff --git a/lessons/5-NLP/17-GenerativeNetworks/lab/README.md b/lessons/5-NLP/17-GenerativeNetworks/lab/README.md new file mode 100644 index 00000000..6db7214a --- /dev/null +++ b/lessons/5-NLP/17-GenerativeNetworks/lab/README.md @@ -0,0 +1,12 @@ +# Word-level Text Generation using RNNs + +Lab Assignment from [AI for Beginners Curriculum](https://github.com/microsoft/ai-for-beginners). + +## Task + +In this lab, you need to take any book, and use it as a dataset to train word-level text generator. + +## The Dataset + +You are welcome to use any book. You can find a lot of free texts at [Project Gutenberg](https://www.gutenberg.org/), for example, here is a direct link to [Alice's Adventures in Wonderland](https://www.gutenberg.org/files/11/11-0.txt)) by Lewis Carroll. +