Add section or DeepRL

README.md

@@ -84,7 +84,7 @@ For a gentle introduction to *AI in the Cloud* topics you may consider taking th
 <tr><td>20</td><td>Large Language Models, Prompt Programming and Few-Shot Tasks</td><td>Text</td><td>PyTorch</td><td>TensorFlow</td><td></td></tr>
 <tr><td>VI</td><td colspan="4"><b>Other AI Techniques</b></td><td></td></tr>
 <tr><td>21</td><td>Genetic Algorithms</td><td><a href="lessons/6-Other/21-GeneticAlgorithms/README.md">Text</a><td colspan="2"><a href="lessons/6-Other/21-GeneticAlgorithms/Genetic.ipynb">Notebook</a></td><td></td></tr>
-<tr><td>22</td><td>Deep Reinforcement Learning</td><td><a href="lessons/6-Other/22-DeepRL/README.md">Text</a></td><td>PyTorch</td><td>TensorFlow</td><td></td></tr>
+<tr><td>22</td><td>Deep Reinforcement Learning</td><td><a href="lessons/6-Other/22-DeepRL/README.md">Text</a></td><td></td><td><a href="lessons/6-Other/22-DeepRL/CartPole-RL-TF.ipynb">TensorFlow</td><td><a href="lessons/6-Other/22-DeepRL/lab/README.md">Lab</a></td></tr>
 <tr><td>23</td><td>Multi-Agent Systems</td><td><a href="lessons/6-Other/23-MultiagentSystems/README.md">Text</a></td><td></td><td></td><td></td></tr>
 <tr><td>VII</td><td colspan="4"><b>AI Ethics</b></td><td></td></tr>
 <tr><td>24</td><td>AI Ethics and Responsible AI</td><td><a href="lessons/7-Ethics/README.md">Text</a></td><td></td><td></td><td></td></tr>

lessons/6-Other/21-GeneticAlgorithms/README.md

@@ -4,6 +4,8 @@
 
 **Genetic Algorithms** (GA) are based on **evolutionary approach** to AI, in which methods of evolution of population is used to obtain an optimal solution for a given problem. They were proposed in 1975 by [John Henry Holland](https://en.wikipedia.org/wiki/John_Henry_Holland).
 
+> **Watch** [this great video](https://www.youtube.com/watch?v=qv6UVOQ0F44) talking about how computer can learn to play Super Mario using neural networks trained by genetic algorithms. We will learn more about computer learning to play games like that [at the next section](../22-DeepRL/README.md).
+
 Genetic Algorithms are based on the following ideas:
 
 * Valid solutions to the problem can be represented as **genes**

lessons/6-Other/22-DeepRL/README.md

@@ -1 +1,103 @@
-Placeholder
+# Deep Reinforcement Learning
+
+Reinforcement learning (RL) is seen as one of the basic machine learning paradigms, next to supervised learning and unsupervised learning. While in supervised learning we rely on the dataset with known outcomes, RL is based on **learning by doing**. For example, when we first see a computer game, we start playing, even not knowing the rules, and soon we are able to improve our skills just by the process of playing and adjusting our behavior.
+
+> **Learn more** about classical reinforcement learning in our [Machine Learning for Beginners Curriculum](https://github.com/microsoft/ML-For-Beginners/blob/main/8-Reinforcement/README.md).
+
+> **Watch** [this great video](https://www.youtube.com/watch?v=qv6UVOQ0F44) talking about how computer can learn to play Super Mario.
+
+To perform RL, we need:
+
+* An **environment** or **simulator** that sets the rules of the game. We should be able to run the experiments in the simulator and observe the results.
+* Some **Reward function**, which indicates how successful our experiment was. In case of learning to play computer game, the reward would be our final score. 
+
+Based on reward function, we should be able to adjust our behavior and improve our skills, so that next time we play better. The main difference between other types of machine learning and RL is that in RL we typically do not know whether we win or lose until we finish the game. Thus, we cannot say whether a certain move alone is good or not - we only receive a reward at the end of the game.
+
+During RL, we typically perform many experiments. During each experiment, we need to balance between following the optimal strategy that we have learnt so far (**exploitation**) and exploring new possible states (**exploration**).
+
+## OpenAI Gym
+
+A great tool for RL is [OpenAI Gym](https://gym.openai.com/) - a **simulation environment**, which can simulate ,many different environments - starting from Atari games, to the physics behind pole balancing. It is one of the most popular simulation environments for training reinforcement learning algorithms, and is maintained by [OpenAI](https://openai.com/).
+
+> **Note**: You can see all the environments available from OpenAI Gym [here](https://gym.openai.com/envs/#classic_control). 
+
+## CartPole Balancing
+
+You have probably all seen modern devices such as *Segway* or *Gyroscooters*. They are able to automatically balance by adjusting their wheels in response to a signal from accelerometer or gyroscope. In this section, we will learn how to solve a similar problem - balancing a pole. It is similar to a situation when a circus actor need to balance a pole on his hand - only in 1D.
+
+A simplified version of balancing is known as a **CartPole** problem. In the cartpole world, we have a horizontal slider that can move left or right, and the goal is to balance a vertical pole on top of the slider.
+
+<img alt="a cartpole" src="images/cartpole.png" width="200"/>
+
+To create and use this environment, we need a couple of lines of Python code:
+
+```python
+import gym
+env = gym.make("CartPole-v1")
+
+env.reset()
+done = False
+total_reward = 0
+while not done:
+   env.render()
+   action = env.action_space.sample()
+   observaton, reward, done, info = env.step(action)
+   total_reward += reward
+
+print(f"Total reward: {total_reward}")
+```
+
+Each environment can be accessed exactly in the same way:
+* `env.reset` starts a new experiment
+* `env.step` performs simulation step. It receives **action** from the **action space**, and returns **observation** (from the observation space), reward, and a termination flag.
+
+In the example above we perform random action at each step, that's why the experiment life is very short:
+
+![non-balancing cartpole](images/cartpole-nobalance.gif)
+
+The goal of RL algorithm is to train a model - so called **policy** &pi; - which will return the action in response to a given state. We can also consider policy to be probabilistic, eg. for any state *s* and action *a* it will return the probability &pi;(*a*|*s*) that we should take *a* in state *s*. 
+
+## Policy Gradients Algorithm
+
+The most obvious way to model a policy is by creating a neural network that will take states as input, and return corresponding actions (or rather probabilities of all actions). In a sense, it would be similar to a normal classification task, with a major difference - we do not know in advance which actions should we take at each of the steps.
+
+The idea here is to estimate those probabilities. We build a vector of **cumulative rewards**, which shows our total reward at each step of the experiment. We also apply **reward discounting** by multiplying earlier rewards by some coefficient &gamma;=0.99, in order to diminish the role of earlier rewards. Then, we reinforce those steps along the experiment path that yield larger rewards.
+
+> Learn more about Policy Gradient algorithm and see it in action in the [example notebook](CartPole-RL-TF.ipynb).
+
+## Actor-Critic Algorithm
+
+An improved version of Policy Gradients approach is called **Actor-Critic**. The main idea behind it is that the neural network would be trained to return two things:
+
+* Policy, which determines which action to take. This part is called **actor**
+* Estimation of the total reward we can expect to get at this state - this part is called **critic**.
+
+In a sense, this architecture resembles [GAN](../../4-ComputerVision/10-GANs/README.md), where we have two networks the are trained against each other. In actor-critic model, actor proposes the action we need to take, and critic tries to be critical and estimate the result. However, our goal is to train those networks in unison.
+
+Because we know both real cumulative rewards and results returned by the critic during the experiment, it is relatively easy to build a loss function that will minimize the difference between them. That would give us **critic loss**. We can computer **actor loss** by using the same approach as in the policy gradient algorithm.
+
+After running one of those algorithms, we can expect our CartPole to behave like this:
+
+![a balancing cartpole](images/cartpole-balance.gif)
+
+## ✍️ Exercises: Policy Gradients and Actor-Critic RL
+
+Continue your learning in the following notebooks:
+
+* [RL in Tensorflow](CartPole-RL-TF.ipynb)
+
+## Other RL Tasks
+
+Reinforcement Learning nowadays is a fast growing field of research. Some of the interesting examples of reinforcement learning are:
+
+* Teaching computer to play **Atari Games**. The challenging part in this problem is that we do not have simple state represented as a vector, but rather a screenshot - and we need to use CNN to convert this screen image to a feature vector, or to extract reward information. Atari games are available in Gym.
+* Teaching computer to play board games, such as Chess and Go. Recent state-of-the-art programs like **Alpha Zero** was trained from scratch by two agents playing against each other, and improving at each step.
+* In industry, RL is used to create control systems from simulation. A service called [Bonsai](https://azure.microsoft.com/services/project-bonsai/) is specifically designed for that.
+
+## Assignment: [Train a Mountain Car](labs/README.md)
+
+Your goal during this assignment would be to train a different Gym environment - [Mountain Car](https://www.gymlibrary.ml/environments/classic_control/mountain_car/).
+
+## Conclusion
+
+We have now learned how to train agents to achieve good results just by providing them a reward function that defines the desired state of the game, and by giving them an opportunity to intelligently explore the search space. We have successfully tried two algorithms, and achieved good result in relatively short time. However, this is just the beginning of your journey into RL, and you should definitely consider taking a separate course is you want to dig deeper.

lessons/6-Other/22-DeepRL/lab/MointainCar.ipynb

@@ -0,0 +1,98 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# # Training Mountain Car to Escape\n",
+    "\n",
+    "Lab Assignment from [AI for Beginners Curriculum](https://github.com/microsoft/ai-for-beginners).\n",
+    "\n",
+    "Your goal is to train the RL agent to control [Mountain Car](https://www.gymlibrary.ml/environments/classic_control/mountain_car/) in OpenAI Environment.\n",
+    "\n",
+    "Let's start by creating the environment:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import gym\n",
+    "env = gym.make('MountainCar-v0')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Let's see how the random experiment looks like:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "state = env.reset()\n",
+    "while True:\n",
+    "    env.render()\n",
+    "    action = env.action_space.sample()\n",
+    "    state, reward, done, info = env.step(action)\n",
+    "    if done:\n",
+    "        break"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Now the notebook is all yours - fell free to adopt Policy Gradients and Actor-Critic algorithms from the lesson to this problem! "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "## Lost of code here"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "env.close()"
+   ]
+  }
+ ],
+ "metadata": {
+  "interpreter": {
+   "hash": "16af2a8bbb083ea23e5e41c7f5787656b2ce26968575d8763f2c4b17f9cd711f"
+  },
+  "kernelspec": {
+   "display_name": "Python 3.8.12 ('py38')",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.8.12"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}

lessons/6-Other/22-DeepRL/lab/README.md

@@ -0,0 +1,22 @@
+# Training Mountain Car to Escape
+
+Lab Assignment from [AI for Beginners Curriculum](https://github.com/microsoft/ai-for-beginners).
+
+## Task
+
+Your goal is to train the RL agent to control [Mountain Car](https://www.gymlibrary.ml/environments/classic_control/mountain_car/) in OpenAI Environment.
+
+![Mountain Car](images/mountaincar.png)
+
+## The Environment
+
+Mountain Car environment consists of the car trapped inside a valley. Your goal is to jump out of the valley and reach the flag. The actions you can perform are to accelerate to the left, to the right, or do nothing. You can observe position of the car along x-axis, and velocity.
+
+## Stating Notebook
+
+Start the lab by opening [MountainCar.ipynb](MountainCar.ipynb)
+
+## Takeaway
+
+You should learn throughout this lab that adopting RL algorithms to a new environment is often quite straightforward, because the OpenAI Gym has the same interface for all environments, and algorithms as such do not largely depend on the nature of the environment. You can even restructure the Python code in such a way as to pass any environment to RL algorithm as a parameter.
+

lessons/X-Extras/X1-MultiModal/README.md

@@ -50,9 +50,9 @@ To generate an image corresponding to a text prompt, we start with some random e
 
 A great library that implements VQGAN+CLIP is [Pixray](http://github.com/pixray/pixray)
 
-![Picture produced by Pixray](images/a_closeup_watercolor_portrait_of_young_male_teacher_of_literature_with_a_book.png) |  ![Picture produced by pixray](images/a_closeup_oil_portrait_of_young_female_teacher_of_computer_science_with_a_computer.png)
-----|----
-Picture generated from prompt *a closeup watercolor portrait of young male teacher of literature with a book* | Picture generated from prompt *a closeup oil portrait of young female teacher of computer science with a computer*
+![Picture produced by Pixray](images/a_closeup_watercolor_portrait_of_young_male_teacher_of_literature_with_a_book.png) |  ![Picture produced by pixray](images/a_closeup_oil_portrait_of_young_female_teacher_of_computer_science_with_a_computer.png) | ![Picture produced by Pixray](a_closeup_oil_portrait_of_old_male_teacher_of_mathematics_in_front_of_blackboard.png)
+----|----|----
+Picture generated from prompt *a closeup watercolor portrait of young male teacher of literature with a book* | Picture generated from prompt *a closeup oil portrait of young female teacher of computer science with a computer* | Picture generated from prompt *a closeup oil portrait of old male teacher of mathematics in front of blackboard*
 
 > Pictures from **Artificial Teachers** collection by [Dmitry Soshnikov](http://soshnikov.com)