diff --git a/README.md b/README.md index e8b21431..f10e688b 100644 --- a/README.md +++ b/README.md @@ -84,7 +84,7 @@ For a gentle introduction to *AI in the Cloud* topics you may consider taking th 20Large Language Models, Prompt Programming and Few-Shot TasksTextPyTorchTensorFlow VIOther AI Techniques 21Genetic AlgorithmsTextNotebook -22Deep Reinforcement LearningTextPyTorchTensorFlow +22Deep Reinforcement LearningTextTensorFlowLab 23Multi-Agent SystemsText VIIAI Ethics 24AI Ethics and Responsible AIText diff --git a/lessons/6-Other/21-GeneticAlgorithms/README.md b/lessons/6-Other/21-GeneticAlgorithms/README.md index dbbbcea7..f4ded546 100644 --- a/lessons/6-Other/21-GeneticAlgorithms/README.md +++ b/lessons/6-Other/21-GeneticAlgorithms/README.md @@ -4,6 +4,8 @@ **Genetic Algorithms** (GA) are based on **evolutionary approach** to AI, in which methods of evolution of population is used to obtain an optimal solution for a given problem. They were proposed in 1975 by [John Henry Holland](https://en.wikipedia.org/wiki/John_Henry_Holland). +> **Watch** [this great video](https://www.youtube.com/watch?v=qv6UVOQ0F44) talking about how computer can learn to play Super Mario using neural networks trained by genetic algorithms. We will learn more about computer learning to play games like that [at the next section](../22-DeepRL/README.md). + Genetic Algorithms are based on the following ideas: * Valid solutions to the problem can be represented as **genes** diff --git a/lessons/6-Other/22-DeepRL/CartPole-RL-TF.ipynb b/lessons/6-Other/22-DeepRL/CartPole-RL-TF.ipynb new file mode 100644 index 00000000..5afa5a35 --- /dev/null +++ b/lessons/6-Other/22-DeepRL/CartPole-RL-TF.ipynb @@ -0,0 +1,567 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Training RL to do Cartpole Balancing\n", + "\n", + "This notebooks is part of [AI for Beginners Curriculum](http://aka.ms/ai-beginners). It has been inspired by [this blog post](https://medium.com/swlh/policy-gradient-reinforcement-learning-with-keras-57ca6ed32555), [official TensorFlow documentation](https://www.tensorflow.org/tutorials/reinforcement_learning/actor_critic) and [this Keras RL example](https://keras.io/examples/rl/actor_critic_cartpole/).\n", + "\n", + "In this example, we will use RL to train a model to balance a pole on a cart that can move left and right on horizontal scale. We will use [OpenAI Gym](https://www.gymlibrary.ml/) environment to simulate the pole.\n", + "\n", + "> **Note**: You can run this lesson's code locally (eg. from Visual Studio Code), in which case the simulation will open in a new window. When running the code online, you may need to make some tweaks to the code, as described [here](https://towardsdatascience.com/rendering-openai-gym-envs-on-binder-and-google-colab-536f99391cc7).\n", + "\n", + "We will start by making sure Gym is installed:" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Requirement already satisfied: gym in c:\\winapp\\miniconda3\\envs\\py38\\lib\\site-packages (0.23.1)\n", + "Collecting pygame\n", + " Downloading pygame-2.1.2-cp38-cp38-win_amd64.whl (8.4 MB)\n", + "Requirement already satisfied: importlib-metadata>=4.10.0 in c:\\winapp\\miniconda3\\envs\\py38\\lib\\site-packages (from gym) (4.11.3)\n", + "Requirement already satisfied: cloudpickle>=1.2.0 in c:\\winapp\\miniconda3\\envs\\py38\\lib\\site-packages (from gym) (2.0.0)\n", + "Requirement already satisfied: gym-notices>=0.0.4 in c:\\winapp\\miniconda3\\envs\\py38\\lib\\site-packages (from gym) (0.0.6)\n", + "Requirement already satisfied: numpy>=1.18.0 in c:\\winapp\\miniconda3\\envs\\py38\\lib\\site-packages (from gym) (1.22.3)\n", + "Requirement already satisfied: zipp>=0.5 in c:\\winapp\\miniconda3\\envs\\py38\\lib\\site-packages (from importlib-metadata>=4.10.0->gym) (3.6.0)\n", + "Installing collected packages: pygame\n", + "Successfully installed pygame-2.1.2\n" + ] + } + ], + "source": [ + "import sys\n", + "!{sys.executable} -m pip install gym pygame" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Now let's create the CartPole environment and see how to operate on it. An environment has the following properties:\n", + "\n", + "* **Action space** is the set of possible actions that we can perform at each step of the simulation\n", + "* **Observation space** is the space of observations that we can make" + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Action space: Discrete(2)\n", + "Observation space: Box([-4.8000002e+00 -3.4028235e+38 -4.1887903e-01 -3.4028235e+38], [4.8000002e+00 3.4028235e+38 4.1887903e-01 3.4028235e+38], (4,), float32)\n" + ] + } + ], + "source": [ + "import gym\n", + "\n", + "env = gym.make(\"CartPole-v1\")\n", + "\n", + "print(f\"Action space: {env.action_space}\")\n", + "print(f\"Observation space: {env.observation_space}\")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Let's see how the simulation works. The following loop runs the simulation, until `env.step` does not return the termination flag `done`. We will randomly chose actions using `env.action_space.sample()`, which means the experiment will probably fail very fast (CartPole environment terminates when the speed of CartPole, its position or angle are outside certain limits).\n", + "\n", + "> Simulation will open in the new window. You can run the code several times and see how it behaves." + ] + }, + { + "cell_type": "code", + "execution_count": 25, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "[-0.04825775 -0.21918646 -0.00796685 0.27188525] -> 1.0\n", + "[-0.05264147 -0.4141938 -0.00252914 0.5620448 ] -> 1.0\n", + "[-0.06092535 -0.21903647 0.00871175 0.26856613] -> 1.0\n", + "[-0.06530608 -0.41428167 0.01408308 0.56398404] -> 1.0\n", + "[-0.07359171 -0.60959834 0.02536276 0.8610703 ] -> 1.0\n", + "[-0.08578368 -0.4148308 0.04258416 0.57646877] -> 1.0\n", + "[-0.09408029 -0.610523 0.05411354 0.88225687] -> 1.0\n", + "[-0.10629076 -0.4161761 0.07175867 0.6070649 ] -> 1.0\n", + "[-0.11461428 -0.22212696 0.08389997 0.33781925] -> 1.0\n", + "[-0.11905681 -0.0282929 0.09065636 0.07272854] -> 1.0\n", + "[-0.11962268 0.16542032 0.09211093 -0.19003159] -> 1.0\n", + "[-0.11631427 -0.03089041 0.08831029 0.13022853] -> 1.0\n", + "[-0.11693208 -0.22715913 0.09091487 0.44941387] -> 1.0\n", + "[-0.12147526 -0.42344147 0.09990314 0.76931363] -> 1.0\n", + "[-0.12994409 -0.61978614 0.11528942 1.0916848 ] -> 1.0\n", + "[-0.14233981 -0.81622297 0.13712311 1.4182041 ] -> 1.0\n", + "[-0.15866427 -1.01275 0.1654872 1.7504154 ] -> 1.0\n", + "[-0.17891927 -1.2093195 0.2004955 2.0896728 ] -> 1.0\n", + "[-0.20310566 -1.4058217 0.24228896 2.4370732 ] -> 1.0\n", + "Total reward: 19.0\n" + ] + } + ], + "source": [ + "env.reset()\n", + "\n", + "done = False\n", + "total_reward = 0\n", + "while not done:\n", + " env.render()\n", + " obs, rew, done, info = env.step(env.action_space.sample())\n", + " total_reward += rew\n", + " print(f\"{obs} -> {rew}\")\n", + "print(f\"Total reward: {total_reward}\")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Youn can notice that observations contain 4 numbers. They are:\n", + "- Position of cart\n", + "- Velocity of cart\n", + "- Angle of pole\n", + "- Rotation rate of pole\n", + "\n", + "`rew` is the reward we receive at each step. You can see that in CartPole environment you are rewarded 1 point for each simulation step, and the goal is to maximize total reward, i.e. the time CartPole is able to balance without falling.\n", + "\n", + "During reinforcement learning, our goal is to train a **policy** $\\pi$, that for each state $s$ will tell us which action $a$ to take, so essentially $a = \\pi(s)$.\n", + "\n", + "If you want probabilistic solution, you can think of policy as returning a set of probabilities for each action, i.e. $\\pi(a|s)$ would mean a probability that we should take action $a$ at state $s$.\n", + "\n", + "## Policy Gradient Method\n", + "\n", + "In simplest RL algorithm, called **Policy Gradient**, we will train a neural network to predict the next action." + ] + }, + { + "cell_type": "code", + "execution_count": 100, + "metadata": {}, + "outputs": [], + "source": [ + "import numpy as np\n", + "import tensorflow as tf\n", + "from tensorflow import keras\n", + "import matplotlib.pyplot as plt\n", + "\n", + "num_inputs = 4\n", + "num_actions = 2\n", + "\n", + "model = keras.Sequential([\n", + " keras.layers.Dense(128, activation=\"relu\",input_shape=(num_inputs,)),\n", + " keras.layers.Dense(num_actions, activation=\"softmax\")\n", + "])\n", + "\n", + "model.compile(loss='categorical_crossentropy', optimizer=keras.optimizers.Adam(learning_rate=0.01))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We will train the network by running many experiments, and updating our network after each run. Let's define a function that will run the experiment and return the results (so-called **trace**) - all states, actions (and their recommended probabilities), and rewards:" + ] + }, + { + "cell_type": "code", + "execution_count": 101, + "metadata": {}, + "outputs": [], + "source": [ + "def run_episode(max_steps_per_episode = 10000,render=False): \n", + " states, actions, probs, rewards = [],[],[],[]\n", + " state = env.reset()\n", + " for _ in range(max_steps_per_episode):\n", + " if render:\n", + " env.render()\n", + " action_probs = model(np.expand_dims(state,0))[0]\n", + " action = np.random.choice(num_actions, p=np.squeeze(action_probs))\n", + " nstate, reward, done, info = env.step(action)\n", + " if done:\n", + " break\n", + " states.append(state)\n", + " actions.append(action)\n", + " probs.append(action_probs)\n", + " rewards.append(reward)\n", + " state = nstate\n", + " return np.vstack(states), np.vstack(actions), np.vstack(probs), np.vstack(rewards)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "You can run one episode with untrained network and observe that total reward (AKA length of episode) is very low:" + ] + }, + { + "cell_type": "code", + "execution_count": 102, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Total reward: 13.0\n" + ] + } + ], + "source": [ + "s,a,p,r = run_episode()\n", + "print(f\"Total reward: {np.sum(r)}\")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "One of the tricky aspects of policy gradient algorithm is to use **discounted rewards**. The idea is that we compute the vector of total rewards at each step of the game, and during this process we discount the early rewards using some coefficient $gamma$. We also normalize the resulting vector, because we will use it as weight to affect our training: " + ] + }, + { + "cell_type": "code", + "execution_count": 79, + "metadata": {}, + "outputs": [], + "source": [ + "eps = 0.0001\n", + "\n", + "def discounted_rewards(rewards,gamma=0.99,normalize=True):\n", + " ret = []\n", + " s = 0\n", + " for r in rewards[::-1]:\n", + " s = r + gamma * s\n", + " ret.insert(0, s)\n", + " if normalize:\n", + " ret = (ret-np.mean(ret))/(np.std(ret)+eps)\n", + " return ret" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Now let's do the actual training! We will run 300 episodes, and at each episode we will do the following:\n", + "\n", + "1. Run the experiment and collect the trace\n", + "1. Calculate the difference (`gradients`) between the actions taken, and by predicted probabilities. The less the difference is, the more we are sure that we have taken the right action.\n", + "1. Calculate discounted rewards and multiply gradients by discounted rewards - that will make sure that steps with higher rewards will make more effect on the final result than lower-rewarded ones\n", + "1. Expected target actions for our neural network would be partly taken from the predicted probabilities during the run, and partly from calculated gradients. We will use `alpha` parameter to determine to which extent gradients and rewards are taken into account - this is called *learning rate* of reinforcement algorithm.\n", + "1. Finally, we train our network on states and expected actions, and repeat the process " + ] + }, + { + "cell_type": "code", + "execution_count": 73, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "0 -> 44.0\n", + "100 -> 105.0\n", + "200 -> 145.0\n", + "300 -> 70.0\n", + "400 -> 190.0\n", + "500 -> 298.0\n", + "600 -> 289.0\n", + "700 -> 499.0\n", + "800 -> 499.0\n", + "900 -> 499.0\n" + ] + }, + { + "data": { + "text/plain": [ + "[]" + ] + }, + "execution_count": 73, + "metadata": {}, + "output_type": "execute_result" + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD4CAYAAAAXUaZHAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAABJLklEQVR4nO2deZgVxbn/v+/szDDDsAz7sMmOyuKICoiiRFBMMIkYk5iYq8bcBJOYxVzIdnNz471m3/WnWYm5RknUaGI0UaJGjQZxQUFlERAGEIZ9n2Fm6vfH6e7T3aequ6q7us+ZM/V5nnnmdHdtvb319ltvvUWMMRgMBoOhuCjJdwMMBoPBoB8j3A0Gg6EIMcLdYDAYihAj3A0Gg6EIMcLdYDAYipCyfDcAAPr168dGjBiR72YYDAZDl+KFF17Ywxhr4B0rCOE+YsQIrFq1Kt/NMBgMhi4FEb0lOmbMMgaDwVCEGOFuMBgMRYgR7gaDwVCEGOFuMBgMRYgR7gaDwVCESAl3ItpCRK8S0ctEtMra14eIHiWiDdb/3q70S4loIxGtI6J5STXeYDAYDHxUNPc5jLEpjLEma3sJgBWMsTEAVljbIKKJAK4EMAnAfAC3ElGpxjYbDAaDIYQ4fu4LAZxv/V4G4AkA/2Htv5sx1gpgMxFtBDAdwLMx6jKkwJrtB9HeyTClsR4A8LuVWzF2QC3OGJ75KGOM4Q8vNGPhlCGoKMvVC052dOL+l7bj8mlDUVJC2NRyBLc98SbePW0Idhw4gXdOHoTKslKs3LwP9dXlGDug1pP/dyu34tk39+L62aNw74vNuGjiQAzqVYVHX9uFD88YjsqyrI7w51d2YNehVpwxvDdefGs/ykoJdVXleG3nIZw42YHB9T3QNLw3nljXgiOt7Zg4qA5VFaV49s096F1dgR0HjqNXj3K8fegE+tRUYN/RNvSursDOgycwbmAtJg6qw8ot+3Dq4F44dOIkXt1+ECP71qCEABCho7MTNZVl2LLnKAbWVaGmsgz7jrVh75E2jOxXAwBobe9ER2cnyktLsGrLfvSqLkddVeaVO3GyE429ewAAdhw8gd7V5dh58ARG9qvB5j1HMbp/T8wZ1x87DhzH4dZ255r6+dPqHXhk7dsgAKP61WDTnqMAgOqKUtRWlaOmohQgAnyhvVs7OvHajkOY2ljvOd7WwbDr0Amc7OjEqIaezv7mA8dBIAyprwp+iDh1AcDxkx3Yf+wkBvfK5H/70An07VmJwydO4siJdgzrU+20q+VQK4Za12bTnqMYZV3PXYda0bOqLHNOAN5sOYpTGmo86TbvPYbhfapRQgARYUS/amxuOeo99/ZOtBxpxdD6Hp79e462obyE0KtHeU77Oxnw1r5jGNm3GgzAZle7AGDnwRMY1qca63cfcer3t7+yvBRnj+qDl7YeQL+eldjUcgQAMHZgLS49fXDwdY2ArHBnAP5GRAzA7YyxOwAMYIztBADG2E4i6m+lHQLgOVfeZmufByK6HsD1ADBs2LCIzTfo5NIfPw0A2HLLAhw8fhJL73sV1RWleO3r8wEAD695Gzf94RVs2XsUN80bn5P/jn9swrf/ug4EYFFTIy747pMAgN+/0AwA2LDrMJZeMgFX3P6sU4/NgWNtWHrfqwCAB1fvAAD86pktmDm6L57ZuBdThtXjzBF9AADN+4/hhrteSuAKZHhyfUtiZavwg8c2eLavaGrMSfPJ38lfB3L1Dbb8fWrDHs9xmeUdKLeP8ZTJSyNbrq50/jS8c3fv95cn235RW1TO5dLTB+dVuM9kjO2wBPijRPRGQFrerc85TauDuAMAmpqazIohBUZHZ+aWHGvrcPYdPH4SALDncBs3z76jbZ50flqOtArrO9nBfwRebT7oaQ+Q0byKgbs+ehbqqsqdTjWIQ4Jr6qaitARtHd5rc97YBjy5vgUXnzoQt111hrN/xJKHPOk+cNYw/M+7T8vZf9HEAZg3aSA+9/vVAIAfvX8q3jWZL4hu+v1q/P6FZkwe2gsP3DDLc8wu96FPzcKDq3fg9ic3eY7fee10nDumwUm3/GPn4OkNLfjR3zfig2cNw7/NHIm538soC5v/dwHufHYLvvLAWlx5ZiNG9++Jbzz0ulNWj/JSvP7f8zH2yw+jrb0Ts8c24DfXTM9py70fn+F8le450oqmbzwGAHjzfy5Bqe8raeJXH8Gxtg7c9sFp+N+H38DWfcfwnUWTcfkZQ3HbE2/im49kRWJtVRle/do8XHH7s1i5eR++9s6JeGX7Qdz34nZPmeMH1uKRG2dzr6UOpGzujLEd1v/dAO5Hxsyyi4gGAYD1f7eVvBmAW8UYCmCHrgYb0oG3Qpf9uLPcvtqXN4n26C9TlYF1ISaJCIi04Nx0Egk5Sco4phwVSnz1xivNLiO3lNKQ8/MfHtQrY1I5paGnc238aexNmWtAgt829vMne79kkLqnMQgV7kRUQ0S19m8AFwFYA+BBAFdbya4G8ID1+0EAVxJRJRGNBDAGwErdDTckC0+Whj2LyT6q+eXej8/A5WcM1VomgXKEpwgZGc1LY2ugUe9dSYk3b1A5cv0PP5F/PIEoWyBRbvsunNAfv/q3M3HNrJHOeed0RNamSLh7z4u4+3ln4P8l6lSyx4l73km/LzJmmQEA7rdOvgzAXYyxR4joeQDLiehaAFsBLAIAxthaIloO4DUA7QAWM8Y6+EUbCpUgTTlMiw7T7KO0w11mPjqRpJQsac1dIg1P+y0rlatA1A6/UBIJZ6nCAg75zSAEt5CkHC2XiDBnXGaYzxbqpUTo8Dwnmf3lpXwd1l1iiUDQB7U/7N7ZLSHipy1JeJZRqHBnjG0CMJmzfy+ACwV5bgZwc+zWGfIGT0DbL4tIdIsGp3STDwtNCZF2AU+Uq20K65cxLXDKKrUkSJhQFh33C6ZAzV1C8Ivy87RuT70BZTqaewkAlxpp5/d3HNnjbi1cshN0lev896XhfvVy9+XZLGPopsg+oe7DFCL8IzzMLOdHfkjiNXRrp6FpJToBXopySZt7kNB13zeZ0qJcq1wBnK1XpPk6KV2aO68doq8XEm4E1+XY3yN0CG5iDoeEUhDx3A2Fh1uWdnYyvPvWZzDAGlAUuoWFlqkuoYMGdtOEZ/fVVa5UuohlOUIzos09x56ckM09RzDnaO7iwu1zzLXbW2YZgf3DXb6soOVq4EKTVjYBt/0JD6ga4W7g0ukSqkfb2rG6+SCAgwDS9ZZhvv/5IolPaKJcW7IIGfMNz/xga61RW59rjopuegk65pe/7q8axoLLDBtQLRVq7q4vEsn7IGOictrt2s6H5m7MMgYukQS0bXMXiOKkbYxJQqI3NGaZ8mYZmTQ8m7us0BLb3GXbIeutac+AdiPbTn69xC3D3hKZpmRt+qI82bzROryk3wYj3A1cAmW70CwT5wUVVMW8//NFUl/QOl0heUnK7AHViCeQ6y0TrR3OMQLmnzowZ3+uWYaktGQgew1zNXdb6IeLOdn74G6XyL/e/6hmjueWL1tnVIxwN3Dh2bqdY6F5NbaD57WT8EvBQ2A1jVmmis1dRjvkmGViD6j6vEoCGyx/hYb44rrk2MshX6+d1e/xaGcpF5llJDsPf7uyA6qyefieVkk/xka4G7hEEdBJPqw6feejkJifu6zHhUQy7iQmSZu7cHIReY9Iae4SHYDfg4U3oJqbi49tjlH1lomCzGQn/26R+S3vM1QNBj8irZ5CjkerS1tRsUjKz122TN4nvP8689LE1dy129xtLduXONdeLm+WyZpH1Mwy7m5L2iwjlYqTz9jcDYWCW2745WvYJCat7eC0Jx8ko2TJdxg8+XT/S95AVHxXyBLhMRn8tm8ZIShjuPGbYXjhBzx+7gGlZs0y3jR2sDmpAVVZs4xnEJbfqWSPZ//z2m9s7oa8EMcMkm9BnASiFzQu8hpjbrrPLl8dWpYt2NxRNfnli9rnqztIgw6swUojmHDEC51g7wp3hSSnrW7scy4ThR+QOy1vHo/7JD+N331X9IVmbO6GvODR3Jn4mJtEXB2Z559Wbv/QGeGJLFRMKEmUqWLycGPb3EOFu6D8KFEhpcYHcjT34DKCigzT3IWBw+L6uUu0zZ/WjdHcDXkhSBSEessI9kd5lvkxbvRQz1lxR4z+F1Hla0BG+ATZ3MOFu9h04TVfiNsh6+cO5Hq2cDV34Ya/Xtu27hPuzNbcw80yUWaoSg+ogj9ZzWjuhrzAgozuAhL1lknA1iMTjMtJS8kMgMWZ9p6TJsDm3h7ZLCOvuTs26IBUkQdUg8r0tdVWCELNMp52Rb8RoVlFnYDR3A35gHl+ewVDuLeMxnYkaL/3C1YZjwydkMhHjoPMJzwvjXT/JRRActqqLLaQVhlQBSQncfkaF2qWiXAuMo6hOZOYBHUlrLgb4W7gE2hzF2XyaU5a2hFWZyx8AibI5IAEbO4hdbqRE27x2sMtU0WDljBCCzV3jp+7rDnIKaOE30axO6j6BfPGfZcvhe8to1y9Eka4GwQwzi/JnFo19+RU9xzNPSBtcpOYJNNF1NzlFXd+SpWvGxX8beWZZdx1ylQrWqpPuFhHFM3d5RpKrn2eNAF5/PuTxAh3Axev5i6nulPwYen6+AkiFBqCX8AEadGZSUx6X8aMdipXppQHSgwBIvSWKSG4xVWwzT08jQj+Yh3Wl2CIK6RThkAVFi7WodZEp13Z9VQlr62gLqO5G/ICE/wOIslJTEkgChGbJloHVGNYcYMMF57rEtfmLsjvN53kaL8SFYs1d303lncpwp4bscur0dwNeSDY5h62iGqERTkEZfLWUNVFji+1hK1Ya/3QG44saKJMmHlL6NaXY3IIsrlTYFm88mz4A6qeikMRfXmVCRfrUL/2Mln4i8sbm7uhQGAem7vfW4afJ3ydzsLC/74FDqgm4i0DkOQbKNO1JWVz92ircTX3nB8BaTmDl0GIIvuKF+uIgncsIKgct2bPS2P83A15gUWwyzhaYqQK5duT1KBesM09Ke1dH0FhyyPHc/eNC2gzD7nu503zxoWWIlOv6P7JLLMni0yH4x97IvClu5mhasgLgYHDwgZUu0hsmSgTdJJugwiZaxq0TmdUs0xOBxjTlsCrZ/Gc0dx08nHkvfi/NMULZEcwy/DyS7SNV5fR3A15wWOWUbS5y9rHO0NmTXrKTMRbxrsdZnPXLeDFA208wi8A11tGti2i/SSvQZOErJMOt+ArTyZXRVlGnF19zgjPfq2TmLxO/5J5ROMhyUp3s0C2gUsai3W0dXSiqqQ0U596dUq4Xdiy+/yDeJnt04b0wqvbD3qPIblBVRmkNPc4DRTk9e+NbXN3DNFh6bIJZF0hy0pKsPHmi3NcH0XhB6LA6+j8TeMOqEru04nR3A2hyA6oyh63aevoVGiDevlueG5yfoXO3uaZHhIZUAV/cgsPuQFVTh2y2qWwTH8bAwadrWPBmnu09sh2gmWlJTn3Sqfmzh20luioeGmMzd2QF6KEH1AVgG3tWeGetJ2eJ7BFA6oiz4YkXkVp4S5xffjeMpJmEKHNXd1rRQcqbqphiKNCRrC5e66F3NdOZp+xuRsKBK8rpGpeOdzCPbRMFr09gJxWa2/HsV2rQKQwoCpx1kF+7lHJuUYSaQM7lJQ6BzdCP/eY5WbNMsEliUx6RnM35IWg8ANCbxmSO25z0mWWSXoBbH7ERL7Nnf/pndAaqpJpI3vLyLZFkDczoEqebXEZ0esJSxdLc9c6oKqeX+jnrl69Eka4G7gE27hFIX/JOionqJU0d0971DsCGVupY3OPMRlIFZ22fK7mbv0Pu2KBrpCKDiJhXkcy5eR+MUS/TqLYMlHwdHQSYwxOOoHCkCRGuBty2H3oRLxojJJZWyPa3KO0jPca5fq5iyVPiU+D1QGBpKegy5wzT4jFH1CVL0+mLpUr6Jg9+LLRYUBdFQBg4qBa7nGRb77K/ZRx8wzMH1BmUhjhbshh+v+sCNTcZc0uYZxU8ZaJabWRsXkG2dyTUN39E3WCSDL0sd0W0X73oaQmc/HqdWL5s+DLP7mxHvd/YgY+PXcs97ioA1V5XrORIHPzy3yF8J8/+fqjIC3ciaiUiF4ioj9b232I6FEi2mD97+1Ku5SINhLROiKal0TDDcninaHqs7mH5ZWsw+MtI5kHiOqDz7O5C9IKPBvS8hSJCu+6xF2j1R/qOFhzD69LPgQxfzxExNRhvYXmF9HAZZTb6XZflc1PEDxTCXeUKpr7pwG87tpeAmAFY2wMgBXWNohoIoArAUwCMB/ArURUqqe5hvQQz1AVofqoqvi5x53mpKKN85wrkngNVcqUm8QUsxJumeoFyAy6htupg7fV2iNZiWRZzr0QmGr4USFz9wXFAtKBVPFENBTAAgA/d+1eCGCZ9XsZgMtc++9mjLUyxjYD2AhgupbWGlKj06O5ewkzEciaEDx1KKnjegZURRqdSto4KJkFIrpCxoXgM8sEae4y5Sl4mETJx0OsuUfouDj5Q10hSXRtCkNz/wGALwBwq1oDGGM7AcD639/aPwTANle6ZmufByK6nohWEdGqlpYW1XYbEibQFVKQJ8wVMreO9CKMxfUzzrdJJuqlkjYdiL5ifOYomWsWlCKqKSKOZ4lQuEfS3MNNVFwDTCHa3InoUgC7GWMvSJbJa3KuMx1jdzDGmhhjTQ0NDZJFG9IiyqShrCukZB2e+sLaI5+Wj7zNXTTTU7/rmt7yuDZ3WRu3oC28RTQCCtGGTnu0aIWmSDb3CKYv0cB50gqDTOCwmQDeRUSXAKgCUEdEvwWwi4gGMcZ2EtEgALut9M0AGl35hwLYobPRhuRJw1sm38guCg0EfVqnQ9IfOUFaKM+3O0pZToES5eh8lkQLokQaTxD8jkLeZ6gyxpYyxoYyxkYgM1D6d8bYVQAeBHC1lexqAA9Yvx8EcCURVRLRSABjAKzU3nJDoniFiaK3TBTVXSFpJD93nsBWeOkTsWcr2dyjlWfvCrsnwjFHgbsov4yYgl+iPVHQ6i3jMcvYNveQPAi+N0kRJ+TvLQCWE9G1ALYCWAQAjLG1RLQcwGsA2gEsZox1xG6pIVUC47l3ldU4XKh8TfNjy7hc4EiPJq3mLSMTz11/p+SPCik1aBrxWLSE4ejwc3fySOT/9TVe/xF/CAf3/iRREu6MsScAPGH93gvgQkG6mwHcHLNthnwS4C0TnlUuR1AHkpM2ps2dqzkp+Mj5BxU7Uu7gwmqb0liPgdZMTR6hrocBAlDaW6ZAbe46zR+8js79HN00bxymDevtzQPR86etWVzMDFUDlyjx0+2HXHYB7TTlI38SCR9RVEhy/dbSJo12mavOHi6ow8oeapaRHX+QMb2I0zimjIidTRT0ukIGe8vIe9AUgM3d0D1JQvAyMHzv0fXCY2F5ZdPyUHGFDPNz1/VSKpllIk7iih/P3au6x/Zzl2qNbpu7vkrCbOd8j6X82NyNcDdwCRKmIsFv24VF9uHdh1vxoxUbQstJAq7N3bXzz5+c5fzmzlB1u7MVoFeQ6NM/Lv6AaYH2dInLkw+PKp1+7r4SpMrJ9I8chSFhR3cj3A1c4vuVB5cJpOvnHmYCOXVIr2xwqJDBL13vpJJVRuGcG2orlesQXR9/vJa4g4DOXIgwM1FCdnJvW9RxC2R5zx/+JCajuRvyQqCfu8BEYKfrKr40qjFHsppp+uqnzDW1r/9Hzx2prV6/SSHYE0ZOk5WqN34R2bKEmnsUm3vu76jna+K5G/KCd4aqpFkm1G4eVEdIeyTqD0LG5m5vhtnUtWnuCiIsXNONVm42D5/SEm9pceWR251UJl2SRKnCc50VnPbzMUPVCHcDlyjCNGvWUK9DBV0DqireMu70ujQuNWeZaIHD5EPs8vcLFzQJLCP+9Unj6yianztHSIfNtoXIW0a9fhWMcDfwCZAlYs09RnWaIk2KUJlEEqa552NQUElzlzSjePIL9qvEc5eqJx/uMsIqInzhRLi2/nw2opg3ujDC3cAlcIaqovlFmC6ivNY2iUmUNqSQfDjLqNjcoyDu6MQdR04ZUmnifUnoRNcM1XATE3HvjfGWMeSFoJWYZPIo1xc9qxS810g8K1MwAGf91/VSpjGj05nEFLEtueMS6Zx7PjpQKSKMbQjHM4zmbsgHQa6HYQOqCsOEqs2KDH9AK1ggCsvS0SAXf1w8E7+5JmQ9G4meM6tJuswoMQVQSQnfQySoAYEeNQUrteUIm6HKzUP8J91o7oa8EBSFUSRmVDV3Fd/12AtkK6QNCzSlbYaqVc6UxnoMru8RmDaq5u0cV2iXt1y+R1FUpE3uKfQCkcwyEQeteWNGojVfdWGEu4GLx01R0ywmfeVEyKTwHqUVa1zFxVDlnHnlhmYPMEXJullK2dxlvXekUsVDp8toWD28+2fMMoa8EBg/PST8QJQ6wsRP1NgqNkqauyjOuzNJR/9LGVaizLXlxjWJWX/mK0Wjt4zzP/8eSVHqcH+1yUajIAI3imjSZpk48dwNRUygzT1khmqUOpTyRfJzl3+RwtImEX4grE6VM5b1bhHlCdqflj09FT933/ZfPnUuKsqC9d0ok8UIQCdPuCd8ikZzNwgI1N218P3H1qO9I7PmeleyuScheMI1d4kyYjRLHPLXFxclruYes7PRib9DnTi4DqP79wzOw/mKkWlrZ2fuDUz6FI1wN3CJEqhLVf5u3H0ED726UzGXPj93YdqQ1y6J8APxJwe5PWS8tQDhZp2g4Fpe2R5kcw8/iTQGSmWJ1JIIX0UgwNJhUsUIdwOXKHq7LT92H26Vr0cy2Fh8zV3FLCMqwz6uy1vGXXZ8s4wT/kHrjNJkhHEhyHht3jKh4QeIa5ZJGiPcDVzi+Lk/vOZtbGo5IlVPWan6GxbJWcZXzT3Xny1MGxo4LIG3Jtxbxh+8zbsdPqgXJoDE7YobnyYK+TDLSOXx/JbPb4S7oWDwhh9Qn6G6bf9xqTxlkpJS96sRvBRc8H5dNneZUuqry+XL49iA7Z9RzTKZxTok65dMJ1dWAaj2HKJ8FRHxhXvS4t4IdwOXIFkgeqjdWWQ1lfJS2yYs2TDo85cXkVbIXxn5lRXO3v3+bdVQCrn18NPly3xSCGYbHlHHlvNhczeukAYugTNUJaao8rwDePl2HWrFiCUPhbdHIfY7D1Hsdn5a/v4k/dzFbbE6P9dZX7fseTQHfBnFHSj0786Hh0uByvZIbqaM5ccsY4S7gYt3hqp6/g6OcOfx0tb96oVHQMlbJmREVd8MVRnvksx/xoAn17fgoVd24LHXdyvUES8dEeXFRFJIXjViFGzuku+DToxwN3CJFBXS9fuBl3fkHOdpL7IDqp4vCQ2ukIGTccLKUq8+BpnafvbUJuw50haQiq9SxveWUUmr78roKGnqsHq8tPWAhpLiIZqhmjRGuBu4eAS6rJ+7K13z/mOSNaUjKtVcIQX255DjqsgUY5uIggS7nrZITmJKCR113v+JmfELCUCljXmQ7WZA1ZAh19XO9Vu2DFdK3lco7wGXHhxlwg0pcjT3CDZ3p3ZNb6qMbIhi7ybB/iht6QrGkTQY1qcagC+2DCed6NngmSmTFvhGczcAyH3QvvnIG87vtw+eUC6DZ8rh7UtroElFSIV5nqSphamEF85OYsrui7vykb/+qvJwfVDH9Sk0m/tvrzsLL7y1H1Xlpc4+lTbmwyxjNHcDgFxdeNeh7CzTz/1+dWBa3v5OjuvXc5v25eyTHWcKWvYvGuIXM0yg6uqQZIRDWj7mQb797mO1VWK/+wKTx1ppqK3E/FMHevbxTld0T5N23+VhhLsBgJ6Hz12ErACU9SL4yeMbozQpi4aokPZeXY4PcmYZlXbb5QpsNIFtCfdzH9o7eEGRpDl3TD/cMGd0XtvghndreO8RY/LeYzoxZhkDAP2z5WT7CtlOYNu+rF93pPADGtPGjS2vgq746bJUlZfgxMnsZ5fSQHTCFvo7rz0r0fKTJA+y3Wjuhgw6vhq9A6qSmntKD71/kDR4taDg/T0r5UMCBCEjuFUGVHk2d9W2PPqZ8zx29ZKSZMwthWZTTxKi/Pi5hwp3IqoiopVEtJqI1hLRf1n7+xDRo0S0wfrf25VnKRFtJKJ1RDQvyRMw6EGLNhrBw0bVfn3Bd57Atx9Zp5QHUDRvhGigA+oq8fCnz1Vug2o9smkC8ysK0cY+1bhwwgBnu7KsNCC1vy6lqroVhTqg2grgAsbYZABTAMwnorMBLAGwgjE2BsAKaxtENBHAlQAmAZgP4FYikn9CDHkhSmyXlZv34dt/zXrVeAdU1Sc+ybBpz1Gs3JI7MKtKFDmUtWkDEwbVxW6DDPJxbMjTPlV4ncCccQ0Y2a8mUaFdW1X8luFM+IHsdm1l5pxt98qkCBXuLIMdv7Xc+mMAFgJYZu1fBuAy6/dCAHczxloZY5sBbAQwXWejDYXBFbc/i58+/qaz7R5MktXI0/Ii0OEKGUa/npWYNbqfQkUybUlrklcu75k21Dom6U4Zod4fvX9qhFyFgcqjO3dCf+f3VecMx/2fmIG5EwcE5IiPlM2diEqJ6GUAuwE8yhj7F4ABjLGdAGD9t1s/BMA2V/Zma5+/zOuJaBURrWppaYlxCgYdaLG5e7xl5PLwXCaTIHcSk1gUiSf0UGhe3YOtb+09KpVOFNBK2pVSQx+ittpV94EIWDhlCK4+ZzgAoKyEMHVY75Bc8ZES7oyxDsbYFABDAUwnolMDknMnbnHKvIMx1sQYa2poaJBqrCE5dAglbyRJ2QHVtDR3dZdCm1995Ezu/rB8KvUM7d0DH5kxAu+e6tWD8uFl4cfY0vmoXpdKawJUadIrY1soecswxg4AeAIZW/ouIhoEANZ/O1RdM4BGV7ahAHKjSBkKCt0yVlYjT014+TV3yWzjB9Zizvj+nn26Xk1vmADC1941KXSBZpmyouVX/5JRKcOQ9XUvKxThTkQNRFRv/e4BYC6ANwA8COBqK9nVAB6wfj8I4EoiqiSikQDGAFipud0GDbS2dzi/VWXs/S81O7/twVPv0nxd2eYe7eXryqLNaOfJYwv30iTWaeQgU8sgAI8T0SsAnkfG5v5nALcAeAcRbQDwDmsbjLG1AJYDeA3AIwAWM8Y6uCUb8sbf1r6NcV9+BGt3HASgJmTfPngCn7knG5JgyX2voLW9wxsiQLKs1MwyCn7usmX1KI/nBFZIvt5BLUl0sY4CMDulRdqae6gfEmPsFQA5Q9qMsb0ALhTkuRnAzbFbZ0iMR1/bBQBYu/0QJg3upfSOnWj32lyWr2rGrDENkcIPdKT0csexuWf3ew/86ZOzMPd7T3ryqfRVOl9xIoplWku7nymgfi01bF/3kkIxyxiKk5PWoo7lZZkHTUUw8AS3/3mVDgiWJ809MK3kch29Qxav1jHRSRfhlzlQd9fYkvzz2Gdn47YPTku93s5C09wNxcnJDvtBs/p3FeHOkdyrtx3w+rnLTmJKS3PPGVBVf8H8OfyavL/MnpXBr5fuNUd55aXt4ggUvqVldP9ajO5fm0jZQc9z1uZuNHdDgjiae2nmEVBxheQ9wD97arOnhL1H5VYOUrG5x9HydZhlZI93RfR0AuqFpBmELd/YZhkj3A2JYgv3Ck1mmaioFBUnbKrKgKpwEpNven+uJi9fRya/vpc8yeiRSYgiUZnnjS2+OS/2M27/L01JOzBmmW6KbZaxNXcVREGQosh8lY7iO39br15BBOK8e+7TCdNkC+kLIHDGrmRDdZzOz69uQqs1YD+wrgoAMLxvsjFYoqL6vNtfnmnddyPcuyltluZu29xVnlORAh3lE1vlBfn1Pzcrl2+jIypkNvwAPP+zx72k9PWdqTtBzT1NyktLHIXjHRMH4LfXnoUZp/TNc6viYd8b5mwbs4whQdot4W4LIBV7tiht0pp7nDg0Mq/TpMGZSI8y64SGseD0QaEDqoVEFDOVShkigm4/EWHWmH6puQ6qojzQbJ1rWqfTdZ4+g1Zss4z9bqnIZZHtO4pFXEW4x4mJLWMP/+GVU/H6zkOe9WN5eRwNPkDsfXD6sIIyy8QZuDSLdejBvgMqi57HwWju3RR7QNU/2COD0CwTSXOXTxtrQFUiTU1lGZpG9IkszIjII0TTFl9R7qWNFm+Z+EUUNWnb3I1wT5kVr+/CPkk3wSSxbe62MNLjkhbF5p7WJKZgn3TvsbCy5BKm6S0TRlhdwddDckBVQWqldd+ToK5HxuChqoGnbXM3ZpkUOdLajmuXrcLkob3wwA2z8toW5rfH5Olda08pLKSO10lGHun0p1eFN9Brtye08w6yuRuV3MPPP3wmHnp1JxoVV1KyO7S0bO5Gc0+RDsvOvXmP3AIMaWDLVi16e4RC4phaVFAKPyCpkfPSec0y+ZeKsl9kOlqqdo3zf22iMrBXFa6dNVI5X3ZA1djci45CnI3nmGU0r8QkS1rC3S++gt8v/sGw++cpk2TMMoVD4OpShffYFixBl8q+jmndd2OWyQOFoLXYn4jOIJyWlZjUy0hrVXitmrvjLVP4pGvXF/Pty0/Hi1sPpNWUgiRt5c4I9xQpRA3IMbkXueYeFirATYVg1m7Y+fnLDPv8LoRO3iYw/ICGZi5qasSipsbwhF2coEvlaO7GLFO8FNA77fiZ58dXpjBt7lVhi3A4NvfcQr3hB6SK0UbT8D4AEGmpPi3PZJTAYQWo8CRF1lsmnfqM5t7dcXyjNZhlClhz9xNkruhRwRfuYS11l0mg1AbObBY1DcXMMf0wpL6Hct60B3/fd2Yj/v7GbkwaUpdqvfnE2NwNqZD1gNQ3oBqF9Mwy8q9UWPgBGTd3BibvL68JIook2HW1RaWIeZMGYsstC+JXWmAED6jak5iMWcaQAjqFeqQB1TyZZYLeL9HaqP6XM6zMcJfKwrHPFU5LihfHLJNSfUa4p0ghmhfjTFnPLUw9S1reMiomkrgLXwOZL4VCEt6hl1mH5l44p1uQmPADRYxzc/PcDjfZAVUdrpDqpGZzDwnP66ZSqLl78/pNPYTC7MANhUHaA6pGuKdI2rElgsj6t3u345WpXojsWqtpoiPkb6GRRpybQpiRW0hUlmWeI/ur0bhCFjGF6PblF/KxyoqQJ7VJTAppK8tKcdmUweKyBOEHiApXvKn66Bvi891Fk3HDnNFoGt4bQPYr2djci5BCCj/AXP4yQP5cIeMswKFCTlTIkDfs+tmn5OyTuX+Fc4fVyJObe1HTv64Kn583LmexEaO5FyMF+ObrDBwWrf7C09x1lKXz/e3Xs0JfYQJ0CBwj24NJ28/dCPcUSdsVSgad3jIFbZbJuej6Z1MmdV/vvv6cyHlFMvtj542SSmfQh/3lZwZUFXlyfQv+uXFPvpsRSFo29yvveBaX3/ZPqbb4zTNxiGLaSeuaaNXcBX7uSZFEPZ84b7R0Wtl7ZDqIYLKau1msQ4mrf7kSAAp61ltaNvfnNu2TTptvzT0tVG3uPKR8xRO4CGmIAp11FKLjQCGQ9ZZJp76i0dy7Ap0p31wZtC7ElMBLPaCuUks5Sdjc4yzVp1RfCg9MUBWy1Reur1B6BHVsxixTxBTiupH+uO79ekYXpkl8mWgTGAqTmESoTPLUeSV0LMsW/ugZwZw0nSmbZYxwT5FCku1+c4wOrSKJ89Ol5Wh9oQR+7kmRhjDQE/JXQxndgILR3ImokYgeJ6LXiWgtEX3a2t+HiB4log3W/96uPEuJaCMRrSOieUmeQNekcN4Cf1TItBbvTRvehCNVwr683GXqNctoLExUR/JVGArQFbIdwOcYYxMAnA1gMRFNBLAEwArG2BgAK6xtWMeuBDAJwHwAtxJR/EhMXZyDx08WlOZu49fg48QgT0RzL6Bysq6sJCyzkCaqAekKbdNBBHfEzgzVQpnExBjbyRh70fp9GMDrAIYAWAhgmZVsGYDLrN8LAdzNGGtljG0GsBHAdM3t7lKs33UYk//rb7hn1dZ8NyUHv1kmjnDf2HJER5M8qLwIt7zntIByNDQmZEA8qVfWP8NRBVFXU1NZikG9qpztQoh3VAwED6hmKBizjBsiGgFgKoB/ARjAGNsJZDoAAP2tZEMAbHNla7b2+cu6nohWEdGqlpaWCE3vOmzYlRF6j79RGOe5de8xbD9wHEBGmzje1oGTHfE1zo279Qt3Fa6cPkx4jBfBMS5pCcQkaikrLcGzSy9E35oKbXWYDiKYtKPCSgt3IuoJ4F4ANzLGDgUl5ezLkRyMsTsYY02MsaaGhgbZZnRp0u65t+49xt1/6Y+fcn4zABO++ggu++kzAICSIhliv2neOM+2zmtOvv+8OqIKulH9anLri9F22axGLidP2lFhpV5lIipHRrD/H2PsPmv3LiIaZB0fBGC3tb8ZgHuZ86EAduhpbtcm6Z67rb0Tv3h6s7N90x9Wc9MdOtHu/F7+/DbPMb9Z5swRvZFPor4HUxrrA8uJNIkpBXv6xacNwgOLZ3r2pbEWa5BHTn11OQBg0RmNwjSZMgxBpD3PJXSGKmW6mV8AeJ0x9j3XoQcBXA3gFuv/A679dxHR9wAMBjAGwEqdje6qJD2g+rOnNuHbf13nbJ842RGaZ9Vb+z3bfkGiw1yTD3LfH31vlDDkr6Y6Jvs6piSR+ZqsrSrH+m9cjPJSI751kNZVlAk/MBPAhwC8SkQvW/u+iIxQX05E1wLYCmARADDG1hLRcgCvIeNps5gxFi5lugFJa34Hj5/0bB+XEO5+/A9ee1oxeQVE1nJCNPUogjhf3k56QkPEK6SiLPwj35h2QkjZWyZUuDPGnoa4s7lQkOdmADfHaFdRkrRw8K9qtH6X+gCn/7lr16C5f/O9p+E/7n01Ut6o2nASA6j+snnxamTv8bpvzNfYougU4qzpYiU7QzUdimT4rGuQ9GvEW7FO9eX1m2XaNSyD974zxV4sSRFmY49mc49ev5/KsnSnfoR1kjqUSaO5B2NiyxQxSax+/mbLERw8ljHH8D69VYVzrs09v2aZqLQcbvVs65zCL3P/dCrEehYv55fhn5gVBxM4LBgT8reICXtF2zs6wQCUl8r3uRd+90kMqe+BZ5ZcwBUobe2dSuUlYZaJQ9SO8LxxDThvbAOeXN8SWM43LjsVtVVyr0ExWzCM1p08JuRvEeJoTSHCYfa3Hsekr/5VuXz3hCQ/be1qmrffjpz3AdWI+eqqyvGD900JLeeqs4dj4ZScOXbBbZJolM4X2H1bT2nI9YPXgTHLJE9Bz1A1xCO7+jn/7u44eAJtMcwgXOGuWJ5/pnuHBpt7vtAxqchNmHmEiLSNq4jWTVX2eQ9JXsxfI/kg6BlhIe+/boxwLyJ4L2rrSVXh7n3wbNk+c3TfqM2KRVJuY/GKTf7lfGbJBc7vHuXZwddS1TgzskvkGXt5ahjNvQjJh7dMW4ear7tfdtjaxtwJA6I2KxZx3gO3wNLxQqW5QLbbm6Z3TQXGD6zN1KFZMvAG+c8bmwkHMnZArda6ujudCThUBGEGVFPAFjJJ31ye22Oros3d3zi7w1DWGAuMkb6YLZHiuTt5w9OKktx13VmRruWUxnq88fZh5Xj7fSzzzviBdYHp3MVe0dSIeZMGolePcqW6TOCwYNL2ljGae4okbd8UecuoINLc8/bixlPdHa46a7i2hUhExRAB9ZZAFHkozRjdD2eNUjdx2Zdf1eY+fmAdfv/v5+CLl0zgHhd1WKqC3RDOjz8wFZdNGZzYoLgfI9xTJPEZqhq8ZfzCwy6xrItr7iUlhBmn9HO2I52NxA38zqLJ+OqlE3H60F5RahBiD2z7/fdlOHNEH4nwATr83A1BjB9Yhx9cORVlCq7JcTDCvYjgiZ7W9k68vO0A1u44KFVGruae+V8aUXOfOCjYHJAkSX1sCBfroIx9/JpZI7V/6Ty3aR8A4O1DJwBkYr3cdd1ZmD22e4TLNqhjhHuKJB3yl6e5dzCGy376DBb86GmpMvz2QLvNUVcD+s21cotwyQSm0knS4Qd0w/tymjG6H2IHatQ4scaY3AsLI9xTwFmE2tpOyn7NtRooSiT/YB9z9kdqkvTsWFt4fevy06NVxCEpWSMaEEtyoCznOmruaYxcLj6McE+R5AdUcytQjUtS5lMF7a+BqAtGyGazzT5+r5ak6Eohf4Hkvmx0Khzd1VdeNOEs3xjhniJJx3Pn+bmrRg/wa4i2QEt6NSDb7OO3PsRylkl5AlSSl0h1oYyPnjsS3100WTp99xTLejhrVF9cM3MkgHQXWgnD+LmniJZFFxjDmy1HMbp/z5xjPJu7apV+227WLBM1rroc2fILV8zkM/Z5meLitl9aMFEpvV6be/eLafDVd07Ev58/Cv1rq/LdFAejuadAdhJT/LIeeHkH5n7vSTy+bnfOMZ7s4Qn8IHI190z+yMJdUmrYXwY6PS6Dikpj0WmdJPXVp7PDKtxuOR0KSbADRrinjPqL9JU/rsGIJQ8522u2Z1wa/+1Xz+P2J9/0lc7R3BWr9H/+y7hCBi2iLa+5W+kL2OUi7FIm2XL/fdQt7LurvbyYMcI9Jis373NC7oYhK2jd2tSdz73lOeaWfbf/Y5PnGM++/sMVG+QqtfBPsIhrlpGlVEJz/9VHzlQqM6ifKERbfj7QGYa2iC5LUWCEe0yuuP1ZzLzl71JpbRPJ9gPHMWLJQ3h6wx5BOrm6j7d5g4I9svbtnDSv7zwkV5iF3yzTGdPPXfaFzw6oijPMGd8fX7lUzZask9DOWbN0G9WvxonG2f2s2Ia4mAHVFPG/oH9ZsxOzxvTLSdfe2YnSkvA1Nk+0q0V8lEFklgmS7UFCT/Zz3/4yCJOPKjbiwLojyOHLpg7B/S9tx3XnjlTPHIG/f/5853dSg7k6VweyB311rLtriE9RaO6FvoK7cP1KQbNlF8hw5//jS9tVm8VF5JURZHMPaq2qn3tYZ6AyQKzbTNCnpgJ/+uQsDO1d7dk/bVh9pj691Xmwz/ojM0YkUr4OU1O55Yuf76UZDRmKRLjnuwVyyLYzyupHN97zsnIeHuVl/Jc8aZu7Y5bxPZF+oWMvLPWx2aNi1adrAHHLLQuU3Q7jcM4p+Vk0RQb7qy/OamIGfRSFcFd194vD/S8144W39uXsH7HkIew/2haYN/cLg9/ufC5tVy7S3FMaUA27ldmY+Mm2p6YyYxZTmTGbZJP8k8l0PfL2V6WOpleU2pq7Ee6FQFEI9zRF4WfuWY333vYs99hb+45x99uyWradJxSXxtOJKBaMf0D1TzfMkipPdUA1rKPO+t3LlSsirF3D+9bgVx85E9+WmuWZ/BMYN8ZPGDo6JvvZOWnMMgVBUQj3NDX3IEQxz22BlOOrLGj2joO5rpW2f/v+YyejN1ACf2wZG7/N/TRXvPKgMQ9Z84d96cLCJdhKoUw4hLgCa874/uhZKe9zkOi3REJfLDpXB8oKd6O5FwJFIdwLRLaHIjvwu43zBXDpjzMhe//wQnPk+j97z8t4/I3cma1uRDFMgoSp/6yuaBqak2bRGbn73Nhmn46Qa2QfTzrWjQppPH92FUmdtx7NPVOIEe6FQbcS7i9u3Y8HV+9IrB0iFzD7y8J/2N/uqvLM7TjWpt/FEQDue2k7PnX3S4FpxGYZ+Xq+dXnWlGELjTDhYQutsPEGUTiEIfU9hHl4X1RJiMg0JjeVW+fy8fNP0VqujpbbE+CMK2RhUBR+7rJmmffc+k8AwLsmD06kHe4l7fYdbcOmliNoGtFH2Pn4XSTDBJwOl89RDT2xetsB4XHREmCqgatsVAOH+e/lzFP6YuPuI852NgRxNs39n5iBqcMyIRBWfO48HDnRDiDbedYomFaiEHRXPn7+KTje1oFf/3NLvDrsAdUSwpZbFsQqy1Ou/UOLn3umEOMKWRh0K+GeNG7h/v47nsO6XYex5ZYFrgHV4Hba75ctxP1eB+O/8kjsNtaHLHzs1nKJXLFlIn7jyWqzpYKO7cuXTsSyZ7MhGOxL4i7Xbao4pSEbLbOqvBQ3zRuHeZMGRG6XCrwS/2P+eACIL9w1erUkhR1z3phlCoOiEO75Eu2dPkHU1pE1p6zbdRhARljJDqiST8B9/c+veY63Ki52zaMuRLi7O0oC39br1xxl+tawATv7w8DfUctEqQyS04vnjA5vXExSsbm7NHe9BWf+6RhQtRUDI9wLg1B9jIh+SUS7iWiNa18fInqUiDZY/3u7ji0loo1EtI6I5iXVcDcsD8/S0dZ2HLY+/23aOML3ZEen82KGebnbr5fdZzyyJjdWTFwqQlRwd3/l1m6DlssLnKHqlBXcLscsE3IveWaZKIOMydjcEyjUX0dS5eoYUDUzVAsKmY/tXwOY79u3BMAKxtgYACusbRDRRABXAphk5bmViMKDpMQkLbOMW1Of+t+PYvLX/+Y5ztOs2ztZ1hwT5gppuwMyhv/4wyvYfbg1bpNzuPfFYG8bkV0/6jJvygOqYd4yHFfIAnKcSYxsDBjNrpAJTGLS/nVhiEToG8sY+wcA/5TMhQCWWb+XAbjMtf9uxlgrY2wzgI0AputpqpgwgaALtxcAT0t378sOLnUKbe7+bfuVaOvoxD2rtmlocTDVFaU5S7FNGlzn/HYL+jCNX4S0zd3R3Pn30nal7OSYZSJp7hrlTxqxjewakurIdHQaVeWl+PxFY7H8Y+doaJEhLlFdIQcwxnYCgPW/v7V/CAC3VGq29iWKSCCEsXrbARw4FhwywE17iM3AjqmxZvtBpyNoc5llwpppv2CHjrcHJxQwdkDu0ntBXDC+P97r8j9/5WsXYXRDLTdt/AWag4XHGZa3S/+6Smff99+X6Xi23LLAmSnKW7C7UHzek1zwgnHOWye6Sr3hgjEYN5D/DBnSRbefO+8Z4Yo0IrqeiFYR0aqWlpZYlUb1q13402dwxe38UAJ+9hxpxc/+sTkwja25f8YVxKu9gzkCya/h3ffidmzZc9TZtt/bQyeizUJVffH92lopkUeIu1sbZHOXGVEMa9riOaPxt8/MxqTB2Zmv756aO/GpT01mpfn66uzAcBQrgE5BnKaFWbdoLxBHM0MCRBXuu4hoEABY/+1pj80AGl3phgLgzhpijN3BGGtijDU1NDREbEaGOIG21u86wt3/tQfX4tk39zrbX/jDK/j+Y+sDyxIOqFq/ec1810+edn7bL+6h49GEu2pwL3/qEp9wdyOauRq1rq/6Ft0oKSGMHRCu8X3i/NH41uWn452nZ+cqRFFmE1mTNIUPiEKeoWooLKIK9wcBXG39vhrAA679VxJRJRGNBDAGwMp4TQwniuZ+MCBGS0cnw6//uQXv/9lz2LznKO5/qRlHWsNNJbZwd3c2JztYoHp0yPK4WbP9oBM35u2DJ6TOwY+ycKfcbVEZQTbZKGJy9tjcRUpkqCgrwRVNjZ5Bu3wve5emK6TuU3Vs+QXtQW+IQqifOxH9DsD5APoRUTOA/wRwC4DlRHQtgK0AFgEAY2wtES0H8BqAdgCLGWPJzKV3oaq5v9p8EO90acx+bEFeQsCF330CnQyYPTb868K2uZ902ebbOzulls2zY8cAwKq39odn4KBDc3ejU2jlCiV9wiSKNpuEQE5SPDpeLYn5QiZUriFvhAp3xtj7BYcuFKS/GcDNcRqliqpwX7vjYOBx2yxSU1nm+LKLIj66cTR3l5/vNx9+A+dJdAw6iPt+JunB5tcM3XXd9sFpscqO0m6dsj0RE4+/DnsSk/aokAl3Goa8URSBw8K8WFSxBbo73OvfQ6IpAhk/9+b9x7DDZVZ5fF2L9ILXaeM3Z9iCw21fnzioDmFIzVDNMQFld1x82qDwAgKIprnruyn9azMePlMa67WVKSIxV8hkijXkkaIIP6B75SLbW0UlljeQMct8dvnqnP3+MAJJoXoV/C+0LTju/8RMx0x0z8fORksCk6kIwP+7alpkt8+46HxiRvevxUOfmoVxEgPCUUk65K+h+Og2wv2wy70wLLWtuatGE2xr78T2/bkLbYTRvJ+/gpMqysqoQJt2C5DaqnLUVonj0ZSQnFmC15HMPzWexu60IYJdRrfN3e3CmQSO+UR3udb/fA9KG/RTFGYZGeG+zBWVL+zFtm3uypp7eyeOtalrolsFy/Opoi7b+S+0bHTfp74wByu/NFeuLp/wCPPOOH+c/DhFd5jtnvgM1WSKNeSRotDc3a6QjDGPINm27xh++vhGDKirki7P1vLtRZJlOdnRGemz+d4Xtivn4aKojoqa6l9ST0Rjn2ql+mTqBoD137hYyfMnks09b7FE46J7QNUq1Uj3oqMohLtbcz/a1uFo3D98bIMz8WjW6KxfddiLbfueV5XLCffvLJqMX/9zM462dWDvUflwBjZhwbxkiWtzd/Yrvum8PqVHeSmOnxR7wQZVoRrqIIpgku3ACoaEhbDxcy8+is4s882H33B+u2eUujXBrz6wNrA8W3MPWrHITQkB1RVl+Mf6eGEU4qJqRxZq7hrsHH+9cbbHxTHIWyYuqoLpPVOHoG/PyvCEBUR2slEydLW+zhBOEWrufJu3O02QjX75qm342VOZGDJb9srZwktLCA21+RcWdtwVWYQ2d8GL/o+b5qCyXE4fGNa3GsP6Zs02/rp0yhLVvujd0xKPZaedNCJPGoqLotDc3Tb3mgp+f/X0xj3C/O7p/kvve1W5fiJCQ4gm+BVfLBUd+IXa9983RSm/ra3dNG8cBrrGJEQ27GF9q7ljF9H83KWbGUp3cA80Xi0GVYpCuLs18eqKjJ1cZamvs/93BQDgwLG2SD7zJQT06xmsNb/vzMbA434umpi77qcf/6LVypq7JScWzxmN576YnXCcxmILOgVytxDuznJ4ydANLmG3owiFe0Zz/8nfNyqXs+Reda0dyAiXoMFXIqCac/zTF45BrcDdskwiCmNSK1CpynaZVuT4uatVEVx2F3iKb//QGfjrjbMj5086towZUC0+usBrEY47/MDL2/Zj695jeKX5gFIZj722CweOq3u6ABlhWBkg3Bnja8PXnTsSc8b35+TI1cp5xBfu/Bc6CU+SnCK7mVlm3qSBWhax8Avhry88FZeePggzR0eLsumUW/iX0KBIUQh3t+b++LoWzP7245lQuwpc95tVypOWbIgIVRFWKiohEmroMoHKVD0+/EWKqkjCrjuiX423Do3SvTvIJVE/3tinGj/5wDRUlsVbqrg7XMPuRtEJdxsVm7uNjLbMg7FgzV1ECRF3gQ8g3CzTt6YCk4fWO9vffO9pAIBPXjBamMc/6Ugkw5XNMhJfEB+YPgx3ffQsxwxlBlTVMJONDKoUhXBv4wjyKMI96qzFliOtUpr7A4tneraJgF2H+AtzlIUsSP2vL16IirLcNz3IR112lq4OP3c/RIQZp/RzhJNOgWwEXnyMF07xURzCnbu8nbqgbhVo0WEca22X0twnN9Z7IgeWEKFN0M4ws0xZaQl3XdMge7l/0FdkGknyRU/CE6c7aO42JraMQZYuLdw3tRzBdcue58Zaj6K5P7Eu2gzTD58zwrN949wxwrQdLhNGCfE7JkDOROQW7ragLg0w5+w94g3dm8QM1Q+fMzzwuN356AzT3C0ChyU8iakb9Y/dhi4t3Ns7GR57fTee2pA7QYlnqkmCZ5ZcgB4VpWi14qjMnTAAN84dK0zf2ekW7iTshGybu1vQVleUYvrIPs42b9HqIM3dXnhjRN9qnDe2AYvn8O3zUYXlw58+F19feGpgGltz1+nG2R1MCklPYuoO17C70aWF+9gBtZgsWP1GpBGrMnN0X+GxAXWVGFLfAwBQX52ZQDQ+xN3NPZuWCGjnCPflHzvH+d3DZUphDLjz2ul45WsXAfBq7vZ3tUjrPm1IL2fafXVFGZZdM11og0/SzGF3PjqFe/fQ3DP/u8GpGjTRpYU7IA7utT9CdEYeQeYRt3yaPrIPll0znWuSGdwrK0TdsduJ4y2z+j8vwvSRfZz9fjt5ZVkp6qzFMyo4NneRrb6+uhyV1qBvmGBVFe52e2TcN2+7ahouPnUg+tfKh2AOoztpnd3oVA0x6fLCXcTRNnG4WRWC7M9+EXne2Aaul8sNF+QK/C8vmAAAngHV04b0Qq8eGUFpT8yqDPDCKeccE7W3vYM5mn6YvVtVE/7JB6biywsmYHT/nqFppw7rjduuOiMRj5wwTmnI+NrX91AL01AIjB+U+SKM68/u56Z547SWZygcurxw/8kHpiZavg4hxHOxvGbmSADA/FOzMWTcsWFOtmfy8OzqNt4B1Qw9BIHT2js7s8I9RHNXPef+dVW47txRBa9Bf3HBBNx57XScNjTZJfGS4Mfvn4q7rz9bOX5QGIvnjMaWWxZoLdNQGHT5kL/D+9SEJ4pBkHCNYza2Bxa/9s5J+NjsU3DP89twzayRznF7oNX9JeDvJCo4bZs7gR/OoL0zq7l3hmjuhS6ko1JZVopzx8gv31dI1FaV4+xR4vEfg8FPl9fcRfHFzx0TL9aGDc/mbptULpoUHrkxtPzSEjT2qcbn543zaGW2t49bO/d3Ju7B5DGW/3x9dYUzyOumo5M5HVW7pBviPA3nZzAY8kOX19xFNmkV22RtVRkOn8gu8vHoZ2bjHd//BwD+IOF1547Cu6YMRp/q5Gy3tubO085tzh3TgGeXXgAAGNQrV6C7ae9gjrklTHMHgFVfnusMlBoMhq5H19fcBUK8R4W8cH/HBK+GOsY1i1Rkf+5fWxUaIgAApg2rx4LTBkm3xcaeYeuuo5YjbAf16hEq2AHg3LH9nK+QMJs7APTrWam8lqnBYCgcuvzbK9LcVaI07j3aJvRnl4mrHsR9n5jp+MADGZfJMRJeJScds0ym/qryEvz+388JyhLIF+aNh21h0jk7NJ/Yg9IGgyGXri/cBTb3iYPrhHkevGEm3tfUiA+eNQwA0HK4FaUCf/Zpw3oDACYMEpenwvKPnYNHP3teaLovXjIBZwzvjTNHZGakfmTGSIzsF23wmCjzBWIvQXjh+OKwpX/1nRONp4fBIKDL29x5E3kWzzkFHzhrGHpWlqF3dQWu+80q59gF4/vj9KH1OP3yehxpbcf//WsrPnj2MKx4PTc+DQBcNnUIzh7VF419qvHImrelzT0NtZWhs1WDmDCoDvd+fAZufUJ9RalRDTXYfuC4sz22f6YdNZVleG7phegbsiSgwWDo+nR54V5WWoJ3TByARWcMxfV3vgAAuGneeADAoqZGZ6Zqrx7lWP2fF3ny9qwsczS/JwVBw8otbxYAmH/qQOl2Pf+luWonIiCKu+VPPzgNL761H+eP649/btzjWQFoYC99M0MNBkPh0uXNMgDwsw834aJJfMFbX12OG+eOwZ9umBVYhu1yaA9+zps0ALVV+e/7zhub8ct+x0S+/zqPuqpynD8uk37G6H7KKzYZDIauT/6ll0b+uHgmeld7PUqIKDBKo01NZcbcYmvnt3+oSX8DI3DqkF7GrmwwGJRJTLgT0XwAPwRQCuDnjLFbkqrLZoogQqQMX7pkInrXVCiZXgwGg6FQScQsQ0SlAH4K4GIAEwG8n4gmJlGXLnpVl2PpxRO4qxsZDAZDVyMpSTYdwEbG2CbGWBuAuwEsTKgug8FgMPhISrgPAbDNtd1s7XMgouuJaBURrWppiba8ncFgMBj4JCXcedM6PU59jLE7GGNNjLGmhoauGanPYDAYCpWkhHszgEbX9lAAOxKqy2AwGAw+khLuzwMYQ0QjiagCwJUAHkyoLoPBYDD4SMQVkjHWTkQ3APgrMq6Qv2SMrU2iLoPBYDDkkpifO2PsLwD+klT5BoPBYBBjnLoNBoOhCCEWZyFQXY0gagHwVowi+gHYo6k5XYHudr6AOefugjlnNYYzxrjuhgUh3ONCRKsYY4URDCYFutv5AuacuwvmnPVhzDIGg8FQhBjhbjAYDEVIsQj3O/LdgJTpbucLmHPuLphz1kRR2NwNBoPB4KVYNHeDwWAwuDDC3WAwGIqQLi3ciWg+Ea0joo1EtCTf7dEFETUS0eNE9DoRrSWiT1v7+xDRo0S0wfrf25VnqXUd1hHRvPy1PjpEVEpELxHRn63toj5fACCieiL6AxG9Yd3vc4r5vInoM9YzvYaIfkdEVcV4vkT0SyLaTURrXPuUz5OIziCiV61jPyIiXsRdPoyxLvmHTMyaNwGMAlABYDWAiflul6ZzGwRgmvW7FsB6ZFa0+haAJdb+JQC+af2eaJ1/JYCR1nUpzfd5RDjvzwK4C8Cfre2iPl/rXJYBuM76XQGgvljPG5k1HTYD6GFtLwfwkWI8XwCzAUwDsMa1T/k8AawEcA4yYdQfBnCxbBu6suZetKs9McZ2MsZetH4fBvA6Mi/GQmSEAaz/l1m/FwK4mzHWyhjbDGAjMteny0BEQwEsAPBz1+6iPV8AIKI6ZITALwCAMdbGGDuA4j7vMgA9iKgMQDUyocCL7nwZY/8AsM+3W+k8iWgQgDrG2LMsI+l/48oTSlcW7qGrPRUDRDQCwFQA/wIwgDG2E8h0AAD6W8mK4Vr8AMAXAHS69hXz+QKZr84WAL+yzFE/J6IaFOl5M8a2A/gOgK0AdgI4yBj7G4r0fDmonucQ67d/vxRdWbiHrvbU1SGingDuBXAjY+xQUFLOvi5zLYjoUgC7GWMvyGbh7Osy5+uiDJlP99sYY1MBHEXmc11Elz5vy8a8EBnTw2AANUR0VVAWzr4uc74KiM4z1vl3ZeFe1Ks9EVE5MoL9/xhj91m7d1mfarD+77b2d/VrMRPAu4hoCzLmtQuI6Lco3vO1aQbQzBj7l7X9B2SEfbGe91wAmxljLYyxkwDuAzADxXu+flTPs9n67d8vRVcW7kW72pM1Iv4LAK8zxr7nOvQggKut31cDeMC1/0oiqiSikQDGIDMQ0yVgjC1ljA1ljI1A5j7+nTF2FYr0fG0YY28D2EZE46xdFwJ4DcV73lsBnE1E1dYzfiEy40nFer5+lM7TMt0cJqKzrev1YVeecPI9qhxzRPoSZDxJ3gTwpXy3R+N5zULm8+sVAC9bf5cA6AtgBYAN1v8+rjxfsq7DOiiMqBfaH4DzkfWW6Q7nOwXAKute/xFA72I+bwD/BeANAGsA3ImMh0jRnS+A3yEzrnASGQ382ijnCaDJulZvAvgJrKgCMn8m/IDBYDAUIV3ZLGMwGAwGAUa4GwwGQxFihLvBYDAUIUa4GwwGQxFihLvBYDAUIUa4GwwGQxFihLvBYDAUIf8fYEwSgqJjBpAAAAAASUVORK5CYII=", + "text/plain": [ + "
" + ] + }, + "metadata": { + "needs_background": "light" + }, + "output_type": "display_data" + } + ], + "source": [ + "alpha = 1e-4\n", + "\n", + "history = []\n", + "for epoch in range(300):\n", + " states, actions, probs, rewards = run_episode()\n", + " one_hot_actions = np.eye(2)[actions.T][0]\n", + " gradients = one_hot_actions-probs\n", + " dr = discounted_rewards(rewards)\n", + " gradients *= dr\n", + " target = alpha*np.vstack([gradients])+probs\n", + " model.train_on_batch(states,target)\n", + " history.append(np.sum(rewards))\n", + " if epoch%100==0:\n", + " print(f\"{epoch} -> {np.sum(rewards)}\")\n", + "\n", + "plt.plot(history)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Now let's run the episode with rendering to see the result:" + ] + }, + { + "cell_type": "code", + "execution_count": 82, + "metadata": {}, + "outputs": [], + "source": [ + "_ = run_episode(render=True)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Hopefully, you can see that pole can now balance pretty well!\n", + "\n", + "## Actor-Critic Model\n", + "\n", + "Actor-Critic model is the further development of policy gradients, in which we build a neural network to learn both the policy and estimated rewards. The network will have two outputs (or you can view it as two separate networks):\n", + "* **Actor** will recommend the action to take by giving us the state probability distribution, as in policy gradient model\n", + "* **Critic** would estimate what the reward would be from those actions. It returns total estimated rewards in the future at the given state.\n", + "\n", + "Let's define such a model: " + ] + }, + { + "cell_type": "code", + "execution_count": 103, + "metadata": {}, + "outputs": [], + "source": [ + "num_inputs = 4\n", + "num_actions = 2\n", + "num_hidden = 128\n", + "\n", + "inputs = keras.layers.Input(shape=(num_inputs,))\n", + "common = keras.layers.Dense(num_hidden, activation=\"relu\")(inputs)\n", + "action = keras.layers.Dense(num_actions, activation=\"softmax\")(common)\n", + "critic = keras.layers.Dense(1)(common)\n", + "\n", + "model = keras.Model(inputs=inputs, outputs=[action, critic])" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We would need to slightly modify our `run_episode` function to return also critic results:" + ] + }, + { + "cell_type": "code", + "execution_count": 104, + "metadata": {}, + "outputs": [], + "source": [ + "def run_episode(max_steps_per_episode = 10000,render=False): \n", + " states, actions, probs, rewards, critic = [],[],[],[],[]\n", + " state = env.reset()\n", + " for _ in range(max_steps_per_episode):\n", + " if render:\n", + " env.render()\n", + " action_probs, est_rew = model(np.expand_dims(state,0))\n", + " action = np.random.choice(num_actions, p=np.squeeze(action_probs[0]))\n", + " nstate, reward, done, info = env.step(action)\n", + " if done:\n", + " break\n", + " states.append(state)\n", + " actions.append(action)\n", + " probs.append(tf.math.log(action_probs[0,action]))\n", + " rewards.append(reward)\n", + " critic.append(est_rew[0,0])\n", + " state = nstate\n", + " return states, actions, probs, rewards, critic" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Now we will run the main training loop. We will use manual network training process by computing proper loss functions and updating network parameters:" + ] + }, + { + "cell_type": "code", + "execution_count": 105, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "running reward: 5.82 at episode 10\n", + "running reward: 9.43 at episode 20\n", + "running reward: 10.30 at episode 30\n", + "running reward: 10.28 at episode 40\n", + "running reward: 11.00 at episode 50\n", + "running reward: 13.01 at episode 60\n", + "running reward: 21.78 at episode 70\n", + "running reward: 40.54 at episode 80\n", + "running reward: 73.70 at episode 90\n", + "running reward: 100.19 at episode 100\n", + "running reward: 159.20 at episode 110\n", + "Solved at episode 114!\n" + ] + } + ], + "source": [ + "optimizer = keras.optimizers.Adam(learning_rate=0.01)\n", + "huber_loss = keras.losses.Huber()\n", + "episode_count = 0\n", + "running_reward = 0\n", + "\n", + "while True: # Run until solved\n", + " state = env.reset()\n", + " episode_reward = 0\n", + " with tf.GradientTape() as tape:\n", + " _,_,action_probs, rewards, critic_values = run_episode()\n", + " episode_reward = np.sum(rewards)\n", + " \n", + " # Update running reward to check condition for solving\n", + " running_reward = 0.05 * episode_reward + (1 - 0.05) * running_reward\n", + "\n", + " # Calculate discounted rewards that will be labels for our critic\n", + " dr = discounted_rewards(rewards)\n", + "\n", + " # Calculating loss values to update our network\n", + " actor_losses = []\n", + " critic_losses = []\n", + " for log_prob, value, rew in zip(action_probs, critic_values, dr):\n", + " # When we took the action with probability `log_prob`, we received discounted reward of `rew`,\n", + " # while critic predicted it to be `value` \n", + " # First we calculate actor loss, to make actor predict actions that lead to higher rewards\n", + " diff = rew - value\n", + " actor_losses.append(-log_prob * diff)\n", + "\n", + " # The critic loss is to minimize the difference between predicted reward `value` and actual\n", + " # discounted reward `rew`\n", + " critic_losses.append(\n", + " huber_loss(tf.expand_dims(value, 0), tf.expand_dims(rew, 0))\n", + " )\n", + "\n", + " # Backpropagation\n", + " loss_value = sum(actor_losses) + sum(critic_losses)\n", + " grads = tape.gradient(loss_value, model.trainable_variables)\n", + " optimizer.apply_gradients(zip(grads, model.trainable_variables))\n", + "\n", + " # Log details\n", + " episode_count += 1\n", + " if episode_count % 10 == 0:\n", + " template = \"running reward: {:.2f} at episode {}\"\n", + " print(template.format(running_reward, episode_count))\n", + "\n", + " if running_reward > 195: # Condition to consider the task solved\n", + " print(\"Solved at episode {}!\".format(episode_count))\n", + " break\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Let's run the episode and see how good our model is:" + ] + }, + { + "cell_type": "code", + "execution_count": 99, + "metadata": {}, + "outputs": [], + "source": [ + "_ = run_episode(render=True)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Finally, let's close the environment." + ] + }, + { + "cell_type": "code", + "execution_count": 106, + "metadata": {}, + "outputs": [], + "source": [ + "env.close()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Takeaway\n", + "\n", + "We have seen two RL algorithms in this demo: simple policy gradient, and more sophisticated actor-critic. You can see that those algorithms operate with abstract notions of state, action and reward - thus they can be applied to very different environments.\n", + "\n", + "Reinforcement learning allows us to learn the best strategy to solve the problem just by looking at the final reward. The fact that we do not need labelled datasets allows us to repeat simulations many times to optimize our models. However, there are still many challenges in RL, which you may learn if you decide to focus more on this interesting area of AI. " + ] + } + ], + "metadata": { + "interpreter": { + "hash": "16af2a8bbb083ea23e5e41c7f5787656b2ce26968575d8763f2c4b17f9cd711f" + }, + "kernelspec": { + "display_name": "Python 3.8.12 ('py38')", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.8.12" + }, + "orig_nbformat": 4 + }, + "nbformat": 4, + "nbformat_minor": 2 +} diff --git a/lessons/6-Other/22-DeepRL/README.md b/lessons/6-Other/22-DeepRL/README.md index f6e0339a..15ecb6c8 100644 --- a/lessons/6-Other/22-DeepRL/README.md +++ b/lessons/6-Other/22-DeepRL/README.md @@ -1 +1,103 @@ -Placeholder \ No newline at end of file +# Deep Reinforcement Learning + +Reinforcement learning (RL) is seen as one of the basic machine learning paradigms, next to supervised learning and unsupervised learning. While in supervised learning we rely on the dataset with known outcomes, RL is based on **learning by doing**. For example, when we first see a computer game, we start playing, even not knowing the rules, and soon we are able to improve our skills just by the process of playing and adjusting our behavior. + +> **Learn more** about classical reinforcement learning in our [Machine Learning for Beginners Curriculum](https://github.com/microsoft/ML-For-Beginners/blob/main/8-Reinforcement/README.md). + +> **Watch** [this great video](https://www.youtube.com/watch?v=qv6UVOQ0F44) talking about how computer can learn to play Super Mario. + +To perform RL, we need: + +* An **environment** or **simulator** that sets the rules of the game. We should be able to run the experiments in the simulator and observe the results. +* Some **Reward function**, which indicates how successful our experiment was. In case of learning to play computer game, the reward would be our final score. + +Based on reward function, we should be able to adjust our behavior and improve our skills, so that next time we play better. The main difference between other types of machine learning and RL is that in RL we typically do not know whether we win or lose until we finish the game. Thus, we cannot say whether a certain move alone is good or not - we only receive a reward at the end of the game. + +During RL, we typically perform many experiments. During each experiment, we need to balance between following the optimal strategy that we have learnt so far (**exploitation**) and exploring new possible states (**exploration**). + +## OpenAI Gym + +A great tool for RL is [OpenAI Gym](https://gym.openai.com/) - a **simulation environment**, which can simulate ,many different environments - starting from Atari games, to the physics behind pole balancing. It is one of the most popular simulation environments for training reinforcement learning algorithms, and is maintained by [OpenAI](https://openai.com/). + +> **Note**: You can see all the environments available from OpenAI Gym [here](https://gym.openai.com/envs/#classic_control). + +## CartPole Balancing + +You have probably all seen modern devices such as *Segway* or *Gyroscooters*. They are able to automatically balance by adjusting their wheels in response to a signal from accelerometer or gyroscope. In this section, we will learn how to solve a similar problem - balancing a pole. It is similar to a situation when a circus actor need to balance a pole on his hand - only in 1D. + +A simplified version of balancing is known as a **CartPole** problem. In the cartpole world, we have a horizontal slider that can move left or right, and the goal is to balance a vertical pole on top of the slider. + +a cartpole + +To create and use this environment, we need a couple of lines of Python code: + +```python +import gym +env = gym.make("CartPole-v1") + +env.reset() +done = False +total_reward = 0 +while not done: + env.render() + action = env.action_space.sample() + observaton, reward, done, info = env.step(action) + total_reward += reward + +print(f"Total reward: {total_reward}") +``` + +Each environment can be accessed exactly in the same way: +* `env.reset` starts a new experiment +* `env.step` performs simulation step. It receives **action** from the **action space**, and returns **observation** (from the observation space), reward, and a termination flag. + +In the example above we perform random action at each step, that's why the experiment life is very short: + +![non-balancing cartpole](images/cartpole-nobalance.gif) + +The goal of RL algorithm is to train a model - so called **policy** π - which will return the action in response to a given state. We can also consider policy to be probabilistic, eg. for any state *s* and action *a* it will return the probability π(*a*|*s*) that we should take *a* in state *s*. + +## Policy Gradients Algorithm + +The most obvious way to model a policy is by creating a neural network that will take states as input, and return corresponding actions (or rather probabilities of all actions). In a sense, it would be similar to a normal classification task, with a major difference - we do not know in advance which actions should we take at each of the steps. + +The idea here is to estimate those probabilities. We build a vector of **cumulative rewards**, which shows our total reward at each step of the experiment. We also apply **reward discounting** by multiplying earlier rewards by some coefficient γ=0.99, in order to diminish the role of earlier rewards. Then, we reinforce those steps along the experiment path that yield larger rewards. + +> Learn more about Policy Gradient algorithm and see it in action in the [example notebook](CartPole-RL-TF.ipynb). + +## Actor-Critic Algorithm + +An improved version of Policy Gradients approach is called **Actor-Critic**. The main idea behind it is that the neural network would be trained to return two things: + +* Policy, which determines which action to take. This part is called **actor** +* Estimation of the total reward we can expect to get at this state - this part is called **critic**. + +In a sense, this architecture resembles [GAN](../../4-ComputerVision/10-GANs/README.md), where we have two networks the are trained against each other. In actor-critic model, actor proposes the action we need to take, and critic tries to be critical and estimate the result. However, our goal is to train those networks in unison. + +Because we know both real cumulative rewards and results returned by the critic during the experiment, it is relatively easy to build a loss function that will minimize the difference between them. That would give us **critic loss**. We can computer **actor loss** by using the same approach as in the policy gradient algorithm. + +After running one of those algorithms, we can expect our CartPole to behave like this: + +![a balancing cartpole](images/cartpole-balance.gif) + +## ✍️ Exercises: Policy Gradients and Actor-Critic RL + +Continue your learning in the following notebooks: + +* [RL in Tensorflow](CartPole-RL-TF.ipynb) + +## Other RL Tasks + +Reinforcement Learning nowadays is a fast growing field of research. Some of the interesting examples of reinforcement learning are: + +* Teaching computer to play **Atari Games**. The challenging part in this problem is that we do not have simple state represented as a vector, but rather a screenshot - and we need to use CNN to convert this screen image to a feature vector, or to extract reward information. Atari games are available in Gym. +* Teaching computer to play board games, such as Chess and Go. Recent state-of-the-art programs like **Alpha Zero** was trained from scratch by two agents playing against each other, and improving at each step. +* In industry, RL is used to create control systems from simulation. A service called [Bonsai](https://azure.microsoft.com/services/project-bonsai/) is specifically designed for that. + +## Assignment: [Train a Mountain Car](labs/README.md) + +Your goal during this assignment would be to train a different Gym environment - [Mountain Car](https://www.gymlibrary.ml/environments/classic_control/mountain_car/). + +## Conclusion + +We have now learned how to train agents to achieve good results just by providing them a reward function that defines the desired state of the game, and by giving them an opportunity to intelligently explore the search space. We have successfully tried two algorithms, and achieved good result in relatively short time. However, this is just the beginning of your journey into RL, and you should definitely consider taking a separate course is you want to dig deeper. \ No newline at end of file diff --git a/lessons/6-Other/22-DeepRL/images/cartpole-balance.gif b/lessons/6-Other/22-DeepRL/images/cartpole-balance.gif new file mode 100644 index 00000000..52dbd204 Binary files /dev/null and b/lessons/6-Other/22-DeepRL/images/cartpole-balance.gif differ diff --git a/lessons/6-Other/22-DeepRL/images/cartpole-nobalance.gif b/lessons/6-Other/22-DeepRL/images/cartpole-nobalance.gif new file mode 100644 index 00000000..65bd5177 Binary files /dev/null and b/lessons/6-Other/22-DeepRL/images/cartpole-nobalance.gif differ diff --git a/lessons/6-Other/22-DeepRL/images/cartpole.png b/lessons/6-Other/22-DeepRL/images/cartpole.png new file mode 100644 index 00000000..3f131cce Binary files /dev/null and b/lessons/6-Other/22-DeepRL/images/cartpole.png differ diff --git a/lessons/6-Other/22-DeepRL/lab/MointainCar.ipynb b/lessons/6-Other/22-DeepRL/lab/MointainCar.ipynb new file mode 100644 index 00000000..0495e452 --- /dev/null +++ b/lessons/6-Other/22-DeepRL/lab/MointainCar.ipynb @@ -0,0 +1,98 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# # Training Mountain Car to Escape\n", + "\n", + "Lab Assignment from [AI for Beginners Curriculum](https://github.com/microsoft/ai-for-beginners).\n", + "\n", + "Your goal is to train the RL agent to control [Mountain Car](https://www.gymlibrary.ml/environments/classic_control/mountain_car/) in OpenAI Environment.\n", + "\n", + "Let's start by creating the environment:" + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "metadata": {}, + "outputs": [], + "source": [ + "import gym\n", + "env = gym.make('MountainCar-v0')" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Let's see how the random experiment looks like:" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "metadata": {}, + "outputs": [], + "source": [ + "state = env.reset()\n", + "while True:\n", + " env.render()\n", + " action = env.action_space.sample()\n", + " state, reward, done, info = env.step(action)\n", + " if done:\n", + " break" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Now the notebook is all yours - fell free to adopt Policy Gradients and Actor-Critic algorithms from the lesson to this problem! " + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "## Lost of code here" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": {}, + "outputs": [], + "source": [ + "env.close()" + ] + } + ], + "metadata": { + "interpreter": { + "hash": "16af2a8bbb083ea23e5e41c7f5787656b2ce26968575d8763f2c4b17f9cd711f" + }, + "kernelspec": { + "display_name": "Python 3.8.12 ('py38')", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.8.12" + } + }, + "nbformat": 4, + "nbformat_minor": 4 +} diff --git a/lessons/6-Other/22-DeepRL/lab/README.md b/lessons/6-Other/22-DeepRL/lab/README.md new file mode 100644 index 00000000..42534c36 --- /dev/null +++ b/lessons/6-Other/22-DeepRL/lab/README.md @@ -0,0 +1,22 @@ +# Training Mountain Car to Escape + +Lab Assignment from [AI for Beginners Curriculum](https://github.com/microsoft/ai-for-beginners). + +## Task + +Your goal is to train the RL agent to control [Mountain Car](https://www.gymlibrary.ml/environments/classic_control/mountain_car/) in OpenAI Environment. + +![Mountain Car](images/mountaincar.png) + +## The Environment + +Mountain Car environment consists of the car trapped inside a valley. Your goal is to jump out of the valley and reach the flag. The actions you can perform are to accelerate to the left, to the right, or do nothing. You can observe position of the car along x-axis, and velocity. + +## Stating Notebook + +Start the lab by opening [MountainCar.ipynb](MountainCar.ipynb) + +## Takeaway + +You should learn throughout this lab that adopting RL algorithms to a new environment is often quite straightforward, because the OpenAI Gym has the same interface for all environments, and algorithms as such do not largely depend on the nature of the environment. You can even restructure the Python code in such a way as to pass any environment to RL algorithm as a parameter. + diff --git a/lessons/6-Other/22-DeepRL/lab/images/mountaincar.png b/lessons/6-Other/22-DeepRL/lab/images/mountaincar.png new file mode 100644 index 00000000..af05db59 Binary files /dev/null and b/lessons/6-Other/22-DeepRL/lab/images/mountaincar.png differ diff --git a/lessons/6-Other/22-DeepRL/notebook.ipynb b/lessons/6-Other/22-DeepRL/notebook.ipynb new file mode 100644 index 00000000..b6338c89 --- /dev/null +++ b/lessons/6-Other/22-DeepRL/notebook.ipynb @@ -0,0 +1,520 @@ +{ + "metadata": { + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.7.0" + }, + "orig_nbformat": 4, + "kernelspec": { + "name": "python3", + "display_name": "Python 3.7.0 64-bit ('3.7')" + }, + "interpreter": { + "hash": "70b38d7a306a849643e446cd70466270a13445e5987dfa1344ef2b127438fa4d" + } + }, + "nbformat": 4, + "nbformat_minor": 2, + "cells": [ + { + "source": [ + "## CartPole Skating\n", + "\n", + "> **Problem**: If Peter wants to escape from the wolf, he needs to be able to move faster than him. We will see how Peter can learn to skate, in particular, to keep balance, using Q-Learning.\n", + "\n", + "First, let's install the gym and import required libraries:" + ], + "cell_type": "markdown", + "metadata": {} + }, + { + "cell_type": "code", + "execution_count": 1, + "metadata": {}, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "Requirement already satisfied: gym in /Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages (0.18.3)\n", + "Requirement already satisfied: Pillow<=8.2.0 in /Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages (from gym) (7.0.0)\n", + "Requirement already satisfied: scipy in /Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages (from gym) (1.4.1)\n", + "Requirement already satisfied: numpy>=1.10.4 in /Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages (from gym) (1.19.2)\n", + "Requirement already satisfied: cloudpickle<1.7.0,>=1.2.0 in /Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages (from gym) (1.6.0)\n", + "Requirement already satisfied: pyglet<=1.5.15,>=1.4.0 in /Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages (from gym) (1.5.15)\n", + "\u001b[33mWARNING: You are using pip version 20.2.3; however, version 21.1.2 is available.\n", + "You should consider upgrading via the '/Library/Frameworks/Python.framework/Versions/3.7/bin/python3.7 -m pip install --upgrade pip' command.\u001b[0m\n" + ] + } + ], + "source": [ + "import sys\n", + "!pip install gym \n", + "\n", + "import gym\n", + "import matplotlib.pyplot as plt\n", + "import numpy as np\n", + "import random" + ] + }, + { + "source": [ + "## Create a cartpole environment" + ], + "cell_type": "markdown", + "metadata": {} + }, + { + "source": [ + "env = gym.make(\"CartPole-v1\")\n", + "print(env.action_space)\n", + "print(env.observation_space)\n", + "print(env.action_space.sample())" + ], + "cell_type": "code", + "metadata": {}, + "execution_count": 2, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "Discrete(2)\nBox(-3.4028234663852886e+38, 3.4028234663852886e+38, (4,), float32)\n0\n" + ] + } + ] + }, + { + "source": [ + "To see how the environment works, let's run a short simulation for 100 steps." + ], + "cell_type": "markdown", + "metadata": {} + }, + { + "source": [ + "env.reset()\n", + "\n", + "for i in range(100):\n", + " env.render()\n", + " env.step(env.action_space.sample())\n", + "env.close()" + ], + "cell_type": "code", + "metadata": {}, + "execution_count": 3, + "outputs": [ + { + "output_type": "stream", + "name": "stderr", + "text": [ + "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/gym/logger.py:30: UserWarning: \u001b[33mWARN: You are calling 'step()' even though this environment has already returned done = True. You should always call 'reset()' once you receive 'done = True' -- any further steps are undefined behavior.\u001b[0m\n warnings.warn(colorize('%s: %s'%('WARN', msg % args), 'yellow'))\n" + ] + } + ] + }, + { + "source": [ + "During simulation, we need to get observations in order to decide how to act. In fact, `step` function returns us back current observations, reward function, and the `done` flag that indicates whether it makes sense to continue the simulation or not:" + ], + "cell_type": "markdown", + "metadata": {} + }, + { + "source": [ + "env.reset()\n", + "\n", + "done = False\n", + "while not done:\n", + " env.render()\n", + " obs, rew, done, info = env.step(env.action_space.sample())\n", + " print(f\"{obs} -> {rew}\")\n", + "env.close()" + ], + "cell_type": "code", + "metadata": {}, + "execution_count": 4, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "[ 0.03044442 -0.19543914 -0.04496216 0.28125618] -> 1.0\n", + "[ 0.02653564 -0.38989186 -0.03933704 0.55942606] -> 1.0\n", + "[ 0.0187378 -0.19424049 -0.02814852 0.25461393] -> 1.0\n", + "[ 0.01485299 -0.38894946 -0.02305624 0.53828712] -> 1.0\n", + "[ 0.007074 -0.19351108 -0.0122905 0.23842953] -> 1.0\n", + "[ 0.00320378 0.00178427 -0.00752191 -0.05810469] -> 1.0\n", + "[ 0.00323946 0.19701326 -0.008684 -0.35315131] -> 1.0\n", + "[ 0.00717973 0.00201587 -0.01574703 -0.06321931] -> 1.0\n", + "[ 0.00722005 0.19736001 -0.01701141 -0.36082863] -> 1.0\n", + "[ 0.01116725 0.39271958 -0.02422798 -0.65882671] -> 1.0\n", + "[ 0.01902164 0.19794307 -0.03740452 -0.37387001] -> 1.0\n", + "[ 0.0229805 0.39357584 -0.04488192 -0.67810827] -> 1.0\n", + "[ 0.03085202 0.58929164 -0.05844408 -0.98457719] -> 1.0\n", + "[ 0.04263785 0.78514572 -0.07813563 -1.2950295 ] -> 1.0\n", + "[ 0.05834076 0.98116859 -0.10403622 -1.61111521] -> 1.0\n", + "[ 0.07796413 0.78741784 -0.13625852 -1.35259196] -> 1.0\n", + "[ 0.09371249 0.98396202 -0.16331036 -1.68461179] -> 1.0\n", + "[ 0.11339173 0.79106371 -0.1970026 -1.44691436] -> 1.0\n", + "[ 0.12921301 0.59883361 -0.22594088 -1.22169133] -> 1.0\n" + ] + } + ] + }, + { + "source": [ + "We can get min and max value of those numbers:" + ], + "cell_type": "markdown", + "metadata": {} + }, + { + "cell_type": "code", + "execution_count": 5, + "metadata": {}, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "[-4.8000002e+00 -3.4028235e+38 -4.1887903e-01 -3.4028235e+38]\n[4.8000002e+00 3.4028235e+38 4.1887903e-01 3.4028235e+38]\n" + ] + } + ], + "source": [ + "print(env.observation_space.low)\n", + "print(env.observation_space.high)" + ] + }, + { + "source": [ + "## State Discretization" + ], + "cell_type": "markdown", + "metadata": {} + }, + { + "cell_type": "code", + "execution_count": 6, + "metadata": {}, + "outputs": [], + "source": [ + "def discretize(x):\n", + " return tuple((x/np.array([0.25, 0.25, 0.01, 0.1])).astype(np.int))" + ] + }, + { + "source": [ + "Let's also explore other discretization method using bins:" + ], + "cell_type": "markdown", + "metadata": {} + }, + { + "cell_type": "code", + "execution_count": 7, + "metadata": {}, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "Sample bins for interval (-5,5) with 10 bins\n [-5. -4. -3. -2. -1. 0. 1. 2. 3. 4. 5.]\n" + ] + } + ], + "source": [ + "def create_bins(i,num):\n", + " return np.arange(num+1)*(i[1]-i[0])/num+i[0]\n", + "\n", + "print(\"Sample bins for interval (-5,5) with 10 bins\\n\",create_bins((-5,5),10))\n", + "\n", + "ints = [(-5,5),(-2,2),(-0.5,0.5),(-2,2)] # intervals of values for each parameter\n", + "nbins = [20,20,10,10] # number of bins for each parameter\n", + "bins = [create_bins(ints[i],nbins[i]) for i in range(4)]\n", + "\n", + "def discretize_bins(x):\n", + " return tuple(np.digitize(x[i],bins[i]) for i in range(4))" + ] + }, + { + "source": [ + "Let's now run a short simulation and observe those discrete environment values." + ], + "cell_type": "markdown", + "metadata": {} + }, + { + "cell_type": "code", + "execution_count": 8, + "metadata": {}, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "(0, 0, -1, -3)\n(0, 0, -2, 0)\n(0, 0, -2, -3)\n(0, 1, -3, -6)\n(0, 2, -4, -9)\n(0, 3, -6, -12)\n(0, 2, -8, -9)\n(0, 3, -10, -13)\n(0, 4, -13, -16)\n(0, 4, -16, -19)\n(0, 4, -20, -17)\n(0, 4, -24, -20)\n" + ] + } + ], + "source": [ + "env.reset()\n", + "\n", + "done = False\n", + "while not done:\n", + " #env.render()\n", + " obs, rew, done, info = env.step(env.action_space.sample())\n", + " #print(discretize_bins(obs))\n", + " print(discretize(obs))\n", + "env.close()" + ] + }, + { + "source": [ + "## Q-Table Structure" + ], + "cell_type": "markdown", + "metadata": {} + }, + { + "cell_type": "code", + "execution_count": 9, + "metadata": {}, + "outputs": [], + "source": [ + "Q = {}\n", + "actions = (0,1)\n", + "\n", + "def qvalues(state):\n", + " return [Q.get((state,a),0) for a in actions]" + ] + }, + { + "source": [ + "## Let's Start Q-Learning!" + ], + "cell_type": "markdown", + "metadata": {} + }, + { + "cell_type": "code", + "execution_count": 10, + "metadata": {}, + "outputs": [], + "source": [ + "# hyperparameters\n", + "alpha = 0.3\n", + "gamma = 0.9\n", + "epsilon = 0.90" + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "metadata": {}, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "0: 108.0, alpha=0.3, epsilon=0.9\n" + ] + } + ], + "source": [ + "def probs(v,eps=1e-4):\n", + " v = v-v.min()+eps\n", + " v = v/v.sum()\n", + " return v\n", + "\n", + "Qmax = 0\n", + "cum_rewards = []\n", + "rewards = []\n", + "for epoch in range(100000):\n", + " obs = env.reset()\n", + " done = False\n", + " cum_reward=0\n", + " # == do the simulation ==\n", + " while not done:\n", + " s = discretize(obs)\n", + " if random.random() Qmax:\n", + " Qmax = np.average(cum_rewards)\n", + " Qbest = Q\n", + " cum_rewards=[]" + ] + }, + { + "source": [ + "## Plotting Training Progress" + ], + "cell_type": "markdown", + "metadata": {} + }, + { + "cell_type": "code", + "execution_count": 20, + "metadata": {}, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "[]" + ] + }, + "metadata": {}, + "execution_count": 20 + }, + { + "output_type": "display_data", + "data": { + "text/plain": "
", + "image/svg+xml": "\r\n\r\n\r\n\r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n\r\n", + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD4CAYAAADiry33AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nO3deXxU9b3/8dcnCSTsa8CQgAEJIKIIBGSXTUWiYqu0Lq2o3MvV6nWhVlGrtbdasddq9dqfy9W2tr22WpdKXYu4W0VBRVBAQFACCEF2kCXk+/tjvkkm+yTMZCZn3s/HI4+c853vzPmenMl7vud7zpxjzjlERCS4UuLdABERiS0FvYhIwCnoRUQCTkEvIhJwCnoRkYBLi3cDADp37uxyc3Pj3QwRkSZl0aJFW5xzmXXVS4igz83NZeHChfFuhohIk2JmX0ZST0M3IiIBp6AXEQk4Bb2ISMAp6EVEAk5BLyIScBEFvZmtNbMlZvaxmS30ZR3NbJ6ZrfS/O/hyM7N7zWyVmX1iZoNjuQIiIlK7+vToxzvnjnfO5fv52cB851weMN/PA5wK5PmfmcD90WqsiIjU3+GcRz8VGOenHwVeB67z5X90oesfv2dm7c0syzm38XAa2pjWbd3Lj/+2mG7tMvjpaf3p3DqdbXsO8K/V31BwXBbOOf7+8XpO6NmJj77azsSjuzD5N2/yo3G9eXbxev484wTumb+S/lltOaJdBobxwBureePzIpbccjJmBsCLSzby+ooiphyXxYl9Kn7nYUnhDv7+8Xq6tk1n5tijyso/XredtBRjQHY7nHM8uaiQCf26cOvzy+jYqjnLNu7kX6u/4d5zB5HdvgWri3bz3UHZpKWm8PKnX3NMt7bc/sJybig4mpPueoNHpg+leZrxwdptzHlxOU9dOoIN2/dx+sBuPLbgK254Zglt0tO4cFQu//PqKm46rT+/eO4zlv9iMjf9fSl/W1RIn66tGdazI51apXPP/JX87PT+9OzcivfXbGXJ+h28tXJLg7fFJScexQNvrKZ3l9as2ry7rNwMUs0oLonfZbbbZKSxa19xxPVHHtWJpet3sHNfMcdmt2PJ+h306tyKL7bsAeDu7w/k6scXx6q53HJ6f7buPcgTH6zj6537YraccENzO/DB2m38YHgP1m/7luZpKbz86aZ6vcb5J/TgolG5TLrrzXov/4qJeaSacfcrn9da79YzB/DlN3v437fW8B9je/Hgm19UqfPdwdk8/eH6Gl9jcI/2PHXpSNZs2cOEX78RUftyOrTglVknktEsNaL6DWWRXI/ezNYA2wAHPOice8jMtjvn2ofV2eac62BmzwFznHNv+/L5wHXOuYWVXnMmoR4/PXr0GPLllxGd998ocmc/Xzad06EFb183ge8/+C4L1mzlvesnsm7bXqY98G5ZnYn9ujB/+eay+cqhFO73Fw1lfN8u7Nx3kONu+WdZ+do5BTW2Ifyx0vK1cwp4aelGLvnzh3Wuz7WT+3LxqJ70u+mlOuuWKv0HrUnz1BQOHCqJ+PVEksFvzxvMZY/V/T8Z7ofDj+QXZw5o0PLMbFHYKEuNIu3Rj3LObTCzLsA8M1te27KrKavyaeKcewh4CCA/Pz9h735SuO1bANZvD/0+eKiE3fsr9uLWbdtbYb6mkAfKeoDFhw5/lXd+G1lvcuvuA5TU8wYztYU8oJAXqcaufQfr/ZyiXftj0JKKIhqjd85t8L83A88Aw4BNZpYF4H+XdmkLge5hT88BNkSrwUGVO/t5LvnTong3Q0QCqM6gN7NWZtamdBo4GVgKzAWm+2rTgWf99FzgAn/2zXBgR1Man4+nlz79Ot5NEJEAimTopivwjD+AmAY85px7ycw+AJ4wsxnAV8A0X/8FYAqwCtgLXBT1VouIBETR7tgP3dQZ9M65L4CB1ZR/A0ysptwBl0WldQG0/+Ah5i7ewKijOjXaMuN4YoqI1OHLb/bWXekwJcRlipPJr15eQdGu/dxzzvGNtkyr7vC4iETdI2+viXcTqqVLIHhffrOHRV/WfqZJNJQeYd/5bf2PzotIYltZyxl38aQevXfif78OVD2fXUQklrY0whi9evRRYNV+dSBxOJf4bRSR2FHQi4gEnIZuoqAhBzsb80SY372zhjYZ2tQiyUo9+giMv/P1CvNNcRDknvkr490EEYkTBX0E1virC4qINEUK+nqq57XBouqv738Vv4WLSJOloI9QInzpaPbTS+LdBBFpghT0IiIBp6AH9h6o+7ru0R6yiecQkIgkFwU9sOCLrXXWKb0BSayHcL49cCi2CxCRpKOgr2TfwbqDNpad8UPq6otIlCnoK9l/sHFukVfbnkFjXPtCRJKHgr6eHv9gXUzv8bh19wHyb30lZq8vIslHQV9P9722it+/szZmr79174EqZbmzn+eQ7h4iIg2koAdcPUfdDxQf/gHTmobiP9uws9ryg4caZ0hJRIJHQd8Aq4sO/5IIroakX7l512G/tohIOAV9gvmwhrtcrdyUmHeuEZHEp6CvpL7DONG2uHBHteWn3/d2I7dERIJCQR8nlggXzxGRpKCgFxEJOAU98bnuTE0HY0VEok1BHwUahhGRRKagr8Sa5I0CRURqpqCvpCFn3SzbWP2XnEREEoGCnviM0Wu4R0Qai4JeRCTgFPTE9vryNS5TZ92ISCNR0MfJLf/4LN5NEJEkEXHQm1mqmX1kZs/5+Z5mtsDMVprZ42bW3Jen+/lV/vHc2DQ9etS7FpEgq0+P/kpgWdj8HcDdzrk8YBsww5fPALY553oDd/t6Ce3xD9bFuwn11uenL/L655vj3QwRaQIiCnozywEKgIf9vAETgCd9lUeBM/30VD+Pf3yiJfgpJis3l18Zsql07g8Ul/DCkq/j3QwRaQIi7dH/BrgWKL37RSdgu3Ou2M8XAtl+OhtYB+Af3+HrV2BmM81soZktLCoqamDzRUSkLnUGvZmdBmx2zi0KL66mqovgsfIC5x5yzuU75/IzMzMjamysxPvSxCIisZQWQZ1RwBlmNgXIANoS6uG3N7M032vPATb4+oVAd6DQzNKAdsDWqLc8SkpKHCVhd+lL7EEmEZH6q7NH75y73jmX45zLBc4BXnXOnQ+8Bpztq00HnvXTc/08/vFXXQKf1nLqPW+xfvu38W6GiEjMHM559NcBs8xsFaEx+Ed8+SNAJ18+C5h9eE2MrRWbdI9WEQm2SIZuyjjnXgde99NfAMOqqbMPmBaFtsVF4u57iIg0jL4ZKyIScAp6EZGAU9CLiARcvcbog+JAcQl9fvoiV0zoHe+miIjEXFL26PcVHwLg9++sjW9DREQaQVIGvYhIMlHQV6KzK0UkaBT0legKCCISNAp6EZGAU9CLiAScgr4SjdGLSNAo6CuZ95nu2iQiwZLUQb9rf3GVsi27D8ShJSIisZPUQS8ikgwU9JX8+p8r4t0EEZGoUtBXUqKjsSISMEkX9J9v2sW0+9+NdzNERBpN0gX9L19YptsHikhSSbqgFxFJNgp6EZGAU9CLiAScgl5EJOAU9CIiAaegFxEJuKQLet1YRESSTdIFvYhIslHQi4gEnIJeRCTgFPQiIgGnoBcRCbikC3oznXcjIsmlzqA3swwze9/MFpvZp2b2c1/e08wWmNlKM3vczJr78nQ/v8o/nhvbVRARkdpE0qPfD0xwzg0Ejgcmm9lw4A7gbudcHrANmOHrzwC2Oed6A3f7eglD/XkRSTZ1Br0L2e1nm/kfB0wAnvTljwJn+umpfh7/+ETTeImISNxENEZvZqlm9jGwGZgHrAa2O+eKfZVCINtPZwPrAPzjO4BO0Wy0iIhELqKgd84dcs4dD+QAw4Cjq6vmf1fXe69yJ1Yzm2lmC81sYVFRUaTtFRGReqrXWTfOue3A68BwoL2ZpfmHcoANfroQ6A7gH28HbK3mtR5yzuU75/IzMzMb1voG0CCSiCSbSM66yTSz9n66BTAJWAa8Bpztq00HnvXTc/08/vFXnXNVevQiItI40uquQhbwqJmlEvpgeMI595yZfQb81cxuBT4CHvH1HwH+ZGarCPXkz4lBu0VEJEJ1Br1z7hNgUDXlXxAar69cvg+YFpXWiYjIYUuKb8YeKnHcMvdTNmz/Nt5NERFpdJEM3TR576/Zyh/+tZbPN+2iZfPUeDdHRKRRJUWP3vmzO0t0TFhEklBSBH1FOr9SRJJLEga9iEhyUdCLiARcUgW9huhFJBklRdBb2Li8LoEgIskm0KdXOudYXbQn3s0QEYmrQPfoH3l7DZPueoPFhdvLyg6VaPxGRJJLoIP+o3WhgF+3dW9Z2avLN8erOSIicRHooBcRkSQLeg3aiEgySoqg15k2IpLMgh306sKLiAQ86D3T9W1EJIkF9jz63NnPl00/vnBdHFsiIhJfSdGjP1BcEu8miIjETVIEfRmN2YtIEgpU0O/eX8zU377D55t2xbspIiIJI1BB/69VW1i8bju/emlFvJsiIpIwAhX0dXl/7dZ4N0FEpNElVdCLiCSjQAW9jrWKiFQVqKAvpUseiIiUC2TQi4hIuUAFve4JKyJSVaCCvpRGbkREygUy6EVEpFzAgl5jNyIilQUs6EN01o2ISLlABr2IiJSrM+jNrLuZvWZmy8zsUzO70pd3NLN5ZrbS/+7gy83M7jWzVWb2iZkNjvVKlNJZNyIiVUXSoy8GfuycOxoYDlxmZv2B2cB851weMN/PA5wK5PmfmcD9UW91HXRHKRGRcnUGvXNuo3PuQz+9C1gGZANTgUd9tUeBM/30VOCPLuQ9oL2ZZUW95dXYrxuMiIhUUa8xejPLBQYBC4CuzrmNEPowALr4atlA+L37Cn1Z5deaaWYLzWxhUVFR/Vtejase/zgqryMiEiQRB72ZtQaeAq5yzu2srWo1ZVVGz51zDznn8p1z+ZmZmZE2IyI660ZEpFxEQW9mzQiF/P855572xZtKh2T8782+vBDoHvb0HGBDdJorIiL1FclZNwY8Aixzzt0V9tBcYLqfng48G1Z+gT/7Zjiwo3SIR0REGl9aBHVGAT8ElphZ6SD4DcAc4AkzmwF8BUzzj70ATAFWAXuBi6La4gho6EZEpFydQe+ce5uarxM2sZr6DrjsMNslIiJRom/GiogEXCCDXl+YEhEpF8igFxGRcgp6EZGAC2TQry7aHe8miIgkjEAG/fKvd8W7CSIiCSOQQS8iIuUU9CIiAaegFxEJOAW9iEjAKehFRAJOQS8iEnAKehGRgFPQi4gEnIJeRCTgFPQiIgGnoBcRCTgFvYhIwCnoRUQCTkEvIhJwCnoRkYBT0IuIBFxggr5w2954N0FEJCEFIujfXf0No+94Ld7NEBFJSIEI+uVf74x3E0REElYggt65eLdARCRxBSPo490AEZEEFoigFxGRmgUi6J3GbkREahSIoBcRkZo1+aDfd/AQtz6/LN7NEBFJWHUGvZn9zsw2m9nSsLKOZjbPzFb63x18uZnZvWa2ysw+MbPBsWw8wINvfBHrRYiINGmR9Oj/AEyuVDYbmO+cywPm+3mAU4E8/zMTuD86zazZ3oPFsV6EiEiTVmfQO+feBLZWKp4KPOqnHwXODCv/owt5D2hvZlnRamz1DYzpq4uINHkNHaPv6pzbCOB/d/Hl2cC6sHqFvqwKM5tpZgvNbGFRUVEDmwH/WLyhwc8VEUkG0T4Ya9WUVdvnds495JzLd87lZ2ZmNniBG3bsa/BzRUSSQUODflPpkIz/vdmXFwLdw+rlAOpyi4jEUUODfi4w3U9PB54NK7/An30zHNhROsQjIiLxkVZXBTP7CzAO6GxmhcDPgDnAE2Y2A/gKmOarvwBMAVYBe4GLYtBmERGphzqD3jl3bg0PTaymrgMuO9xGiYhI9DT5b8aKiEjtFPQiIgHXpIP+/TWVv8clIiKVNemg/8HDC+LdBBGRhNekg/7AoZJ4N0FEJOE16aAXEZG6KehFRAJOQS8iEnAKehGRgFPQi4gEnIJeRCTgFPQiInHUPC32MaygFxGJo7OH5MR8GQp6EZE4Om9Yj5gvQ0EvIhJHA7LbxXwZCnoRkTjp3aV1oyxHQS8iEidj8zIbZTkKehGROLnmlD6NshwFvYhInLRsXufdXKNCQS8iEnAKeklIfbo2zkEqaTzpjfDFIKme/vKSkJyLdwsk2sbkdY53E5KWgl6kEfzyO8dGXPfYRjivuql69OJhUX29Fs1So/p63Tu2iOrrRYuCXqSBxveNzalxd04bGPUAasq+Oyi7bPrEPuV/87F9av/7FxyXVedr5+d2qHd7Tu7ftcJ8r8xWjIvReyFaFPRS5vwTYv9V7Ibo27VNteV/nnFClbJOrZrHtC3nDO1eNn36wG4AXDDiSNbOKaj1ec1SrV7LuXh0br3bVpe1cwq45uTGOZ0vmobkdqBV86offJUDd3CP9hXmczrU3btOT0vh0nFHlQ0rRXKBsXF9u1SYP2twDtntQ8tqlpqYkZqYrZKo6pXZKqJ6t9VjeKHyP1ldLh/fO+K6o3tXHMt9+eqxVercd94gRsdhzLe095bZJp0zBnbjigm9+ckpfWt9zohenfhOWK80El3bZgBUG3CljvB1EtWpA4447Nd49rJRnDesBwtunMTim08GKgb43y4Zwc2n9WfBDRN57N+H07LS36tTq+ZMOrrm9+pPTunHdZNDPwBnDc7m/RsncvGonmV1hlbq9We2Secfl48um+/SJp3Zp/bjigm9Oe3YmvcirpjQm/wj678HEQ0K+iZkUKUeS6T+3/mDK7xxo2FafndunHI0AMNyO9ZZ/5pKYTiqd6da6885K/Sh819Tj6n28dOO6xZJMyOWF/ZV9PvOG8S8aj5cSv3homF8cOMk0lJTmHVyX9pkNAPgwpG5NK+mR/eXmcNJS03hhin9uHBkbll5Zpt0js5qW+0y+vi9mP+aOqDC3kL3ji147ZpxADzxHyPKyksD5KcFR5eVfS+/6lURv5ffvcK6Ho7OrdOrLX/1xyfy4pVjKqxruJqWX10IDuzeHjOjdXoa7VqG/s5XTepT9jpDczty8eiedG2bQUazVF798ThOCuuELLrpJB6enl82f+e0gfz7mPL/hTYZofPYB2S345Hp+fzs9GPo0iajQs/+4QuGcsvp/cvmT+rflWNz2vHFL6fwwA+GcPaQHNpkNGPWyX0Z2L3m/9ExfTI55ZjyD7+nLh1RY91oU9AniLp2Gbu0SeepS0Y26LWz2rXg5tP7c1xO+UG+W88cUOtzurWrvbc4rm8mk32P7ZJxvSJqx2XjjyqbPmdo7cNEQ47syNo5BVwwIheAD26cVGub75w2kGd+NLLsHzcSFjaaMm/WiRSE9cbyahguqs0tZxzD57edWuPjM8cexS1nlH9wDc3twItXjikbAgo3vFcn3r5uPGf5S9iWhvbLV42lZ+dWrJ1TQI9OLcvq//b8wVxzch9mjC4Psdu+cyzvXT+xwut2aZvBvFkncvNp/XllVs0fZvURvmdxZKeW9MpsXeMHWG2evDSy9/dZg7N569rxnNCramfhiHYZDO5R9QOjTXoak47uwtlDcrixoD9PXjKCn5zSl27ty/cOJh7dlQx/bCR8CLBdy2ZcOKon78yewJrbp5SVp6QYkwccgYW9kSaG7T3MOqkPr10zjldmjeXqSX3IP7JD2V7oHy4aypAj6+4gRYuCPgZKd+XrMx561/cG1vr4+zdOIiXFuOec4+vVlrVzCmjXItQTmnv5aOZ8N9RT7ndE7UFWenbDhSNzeSSsRwSw5vYpNEtNoXvHlqydU8CEflV3jR/4wZAqZT85pR9r5xSwdk5BhXBb/ovJQGg4qH9WW2af2q/KczPbpJeFeAffswt39pAcBvXoUHYlwNJeY/hxh/Drfg/s3p41txfwyqyxvHjlmBr+CiGDe7RniO9tRnMM9r/PDm3z/zl3EGvnFPC3S0YwMKcduZ1DAZ7ToTzIf3X2QNbOKajxm5Rd22Zw+YQ8zKys09AsNYUj2mUw7+qxvHRVxXW8eHRPenWuX8++8tlAE/t1oW/XNtz2ndAH8IhenXiyls5Ih5ah8ByTl1n2Plg7p4B7zjm+wvBKm/TaP6zNjO4dW9b4+FmDs8nr0pofDj+yrGzJz0/h4elDy+bzcztyWS3DiWP6hAK5Z+fyYc/s9i0qhHpNXrpqDPOuHssVE/Po2bkVvbu04cpJoW1zdFZb1s4pqDLOH2uN8/3bAMjr0pqnfjSS4275Z511Z4zuya59xfzbmF7c+c/PAUhLMYpLKp4cfnL/rvTLasuQIzsw6qjqhzLeu34i7cOC7YyB3fh80y4y0lJplpbCWYNzuP2FZTz90XpG9e7ElRP70KVNOu+s3sJx2VV3I78/tDsjj+pMj04tueOsY8nr2oY9+4tZun4nEOoZp1ioR/v2dePJateC1JSKb+7q3uxzLx/FkvU7uPGZpQBMHnAE/bPa8tnGndUeNAX4xZkDGNS9PRnNUnn3+gl0apVe655NwbFZLP96F5ecWL5n8OAPh1QYLik9KHbd5H4s/HIbV07MY8/+Yv7+8QbunDaQJxcVAnD/+YMB6N2l5g+8966fSIdWzUhPS2XvgWLunb+K8yI4YD0mrzOL123nzWvHY1T9Ww3IbsvufcW0qhRoQ3M78mzY2G8k/jRjGDu/La5Q9tx/juatlVvK5mvaOyndjOP6ZnJS/65l2y6rXQYbd+yjR8eWfLV1LwA/P+MYpo/MJXf287RJT2PX/mLyurbmjrOPA+Cta8eT3b4FKWHvlaP8h+2IXp1494tv6N+tLW9PGl/l2MLU47OZenzoGMaHN51Es1TjpLveJDuCg6nVKd1rORyl262+B9EB+h1R/72ZWDOXAN9Myc/PdwsXLqz380bcPp+NO/bV+3nXTu5L6/Q0bn72U1o0S+XpH40kNcW455WVPL9kIytvO5UHXl/Nr+eFQvrm0/pzsd8lds5x07NLuXBkLt978D227jnA9/JzSE1J4cqJeXz5zZ4Ku5TTf/c+Zw7qxstLN/HSp1+z+pdTqgRnuE8Kt3PGfe9w7rDuXDAiN6Jd4H0HD/H6is1MHlD36WSH465/rmDX/mJ+dnr14+YAS9fvoEXzVI7KbM3WPQf4dMMOxjTSFfoADhSXMH/Zpiq71KWeWlTIc59s4PcXVT0fe8XXu7jssQ956tKRZXtBySR39vMAvH/DRFYV7eaYrHZ8+NU29hwoZsqArLIQd87xwpKvmTzgiFrfy6VKShwvLo28fiLYX3yIs+7/Fz8t6M/waoaIEoWZLXLO5ddZLxZBb2aTgXuAVOBh59yc2uo3NOiLD5XwzEfr2VdcQtuMNIbmdqR5WgqpZlz+lw+57cxjSU0x9heXRHTd5/3Fh9iy+wDZ7VtQUuJYt20vR3aK7IyVSF77m90HKowJ1uTLb/bQo2PLiHYTRaJlz/5i9uwvpkuCn80j5eIW9GaWCnwOnAQUAh8A5zrnPqvpOQ0NehGRZBZp0MfiYOwwYJVz7gvn3AHgr8DUGCxHREQiEIugzwbWhc0X+rIKzGymmS00s4VFRUUxaIaIiEBsgr66geUq40POuYecc/nOufzMzMS+ToSISFMWi6AvBLqHzecAG2KwHBERiUAsgv4DIM/MeppZc+AcYG4MliMiIhGI+hemnHPFZnY58DKh0yt/55z7NNrLERGRyMTkm7HOuReAF2Lx2iIiUj+61o2ISMAlxCUQzKwI+LKBT+8MbKmzVrBonZOD1jk5HM46H+mcq/O0xYQI+sNhZgsj+WZYkGidk4PWOTk0xjpr6EZEJOAU9CIiAReEoH8o3g2IA61zctA6J4eYr3OTH6MXEZHaBaFHLyIitVDQi4gEXJMOejObbGYrzGyVmc2Od3vqw8y6m9lrZrbMzD41syt9eUczm2dmK/3vDr7czOxev66fmNngsNea7uuvNLPpYeVDzGyJf869liC3rDKzVDP7yMye8/M9zWyBb//j/hpJmFm6n1/lH88Ne43rffkKMzslrDzh3hNm1t7MnjSz5X57jwj6djazq/37eqmZ/cXMMoK2nc3sd2a22cyWhpXFfLvWtIxaOeea5A+h6+isBnoBzYHFQP94t6se7c8CBvvpNoTuytUf+BUw25fPBu7w01OAFwldBno4sMCXdwS+8L87+OkO/rH3gRH+OS8Cp8Z7vX27ZgGPAc/5+SeAc/z0A8ClfvpHwAN++hzgcT/d32/vdKCnfx+kJup7AngU+Dc/3RxoH+TtTOj+E2uAFmHb98KgbWdgLDAYWBpWFvPtWtMyam1rvP8JDuOPPAJ4OWz+euD6eLfrMNbnWUK3X1wBZPmyLGCFn36Q0C0ZS+uv8I+fCzwYVv6gL8sCloeVV6gXx/XMAeYDE4Dn/Jt4C5BWebsSujDeCD+d5utZ5W1dWi8R3xNAWx96Vqk8sNuZ8psPdfTb7TnglCBuZyCXikEf8+1a0zJq+2nKQzcR3cmqKfC7qoOABUBX59xGAP+7i69W0/rWVl5YTXm8/Qa4Fijx852A7c65Yj8f3s6ydfOP7/D16/u3iKdeQBHwez9c9bCZtSLA29k5tx64E/gK2Ehouy0i2Nu5VGNs15qWUaOmHPQR3ckq0ZlZa+Ap4Crn3M7aqlZT5hpQHjdmdhqw2Tm3KLy4mqqujseazDoT6qEOBu53zg0C9hDa3a5Jk19nP2Y8ldBwSzegFXBqNVWDtJ3rEtd1bMpB3+TvZGVmzQiF/P855572xZvMLMs/ngVs9uU1rW9t5TnVlMfTKOAMM1tL6KbxEwj18NubWekls8PbWbZu/vF2wFbq/7eIp0Kg0Dm3wM8/SSj4g7ydJwFrnHNFzrmDwNPASIK9nUs1xnataRk1aspB36TvZOWPoD8CLHPO3RX20Fyg9Mj7dEJj96XlF/ij98OBHX637WXgZDPr4HtSJxMav9wI7DKz4X5ZF4S9Vlw45653zuU453IJba9XnXPnA68BZ/tqlde59G9xtq/vfPk5/myNnkAeoQNXCfeecM59Dawzs76+aCLwGQHezoSGbIabWUvfptJ1Dux2DtMY27WmZdQsngdtonAgZAqhs1VWAzfGuz31bPtoQrtinwAf+58phMYm5wMr/e+Ovr4Bv9wf3t8AAACjSURBVPXrugTID3uti4FV/ueisPJ8YKl/zn1UOiAY5/UfR/lZN70I/QOvAv4GpPvyDD+/yj/eK+z5N/r1WkHYWSaJ+J4AjgcW+m39d0JnVwR6OwM/B5b7dv2J0JkzgdrOwF8IHYM4SKgHPqMxtmtNy6jtR5dAEBEJuKY8dCMiIhFQ0IuIBJyCXkQk4BT0IiIBp6AXEQk4Bb2ISMAp6EVEAu7/A6SijxMjKxrLAAAAAElFTkSuQmCC\n" + }, + "metadata": { + "needs_background": "light" + } + } + ], + "source": [ + "plt.plot(rewards)" + ] + }, + { + "source": [ + "From this graph, it is not possible to tell anything, because due to the nature of stochastic training process the length of training sessions varies greatly. To make more sense of this graph, we can calculate **running average** over series of experiments, let's say 100. This can be done conveniently using `np.convolve`:" + ], + "cell_type": "markdown", + "metadata": {} + }, + { + "cell_type": "code", + "execution_count": 22, + "metadata": {}, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "[]" + ] + }, + "metadata": {}, + "execution_count": 22 + }, + { + "output_type": "display_data", + "data": { + "text/plain": "
", + "image/svg+xml": "\r\n\r\n\r\n\r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n\r\n", + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXsAAAD4CAYAAAANbUbJAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nO2dd3gVZfbHvycdAiGUAKEZelGqkY4gICDo4rr6U3dVVKxrWdeKde2ylnXX1bWiYu8FpYmAKCol9AABAgQIBAglQALp7++PO3Mzd+70O7fk3vN5njyZeeedmXfu3HvmzHlPISEEGIZhmOgmLtwDYBiGYYIPC3uGYZgYgIU9wzBMDMDCnmEYJgZgYc8wDBMDJIR7AADQokULkZWVFe5hMAzD1CtWrVp1SAiRYaVvRAj7rKws5OTkhHsYDMMw9Qoi2mW1L5txGIZhYgAW9gzDMDEAC3uGYZgYgIU9wzBMDMDCnmEYJgZgYc8wDBMDsLBnGIaJAUyFPRGlENEKIlpHRBuJ6DGp/V0i2klEa6W/flI7EdFLRJRPROuJaECwL4JhwoUQAp/n7EFldW24h8IwhlgJqqoAMFoIUUpEiQCWEtFcads9QogvVP3PA9BV+hsE4FXpP8NEHXNz9+OeL9aj4HAZ7hnfI9zDYRhdTDV74aFUWk2U/owqnkwG8J603zIA6USUGfhQGSbyOHaqCgBw6ERlmEfCMMZYstkTUTwRrQVwEMACIcRyadNTkqnmRSJKltraAtij2L1QalMf8wYiyiGinOLi4gAugWHCB4V7AAxjEUvCXghRI4ToB6AdgIFEdAaA+wH0AHAWgGYA7pO6a33//d4EhBBvCCGyhRDZGRmW8vgwDMMwDrHljSOEKAHwE4AJQogiyVRTAeAdAAOlboUA2it2awdgnwtjZRiGYRxixRsng4jSpeUGAMYCyJPt8EREAC4EkCvtMgvAVZJXzmAAx4QQRUEZPcOEmVrpnVUYTmPFBt+v34dVu46GexiMDlY0+0wAi4loPYCV8NjsvwfwIRFtALABQAsAT0r95wDYASAfwJsA/ur6qBkmQnjn150AgFnr+OX11o/W4E+v/hbuYTA6mLpeCiHWA+iv0T5ap78AcEvgQ2OYyGfbQY+jWnmVr599eVUNVhYcwYiuPB/FRAYcQcswQeDRWRtx5YwVyNt/PNxDYRgALOyZKOPD5bvw2co95h2DjKzxnyivDtk5K6pr8I9vc1FVw9G8jD8s7Jmo4sGvc3Hvl+tt7fPBsl3YeajM1XHIE5Wh9MO/94v1mPn7Lpz97OIQnpWpL7CwZ2KSkpOVKD5RASEEHvomFxe+8mtAx5twemvNdgqhtP9l2yEAQNGxchwqrbC0z9wNRciaNtsbCcxELyzsmZhkwBMLcNZTP3pdJwMVdh2aN9Rs31tSHtBx7XCkrC5lw/8WbzfsW3yiAmUV1XhtiaffzkNluPWj1fh4xe6gjpEJHyzsmZjE6x8vAvOPv3poFgCgVVqK5vZtB04EdHynmD28znrqR5z/36UorfDMKdQKge/XF+H+rzb49T1cWoGsabOxcPMBS+cuq3A2T3Gqsiak8w3Xv5cTUw83FvZMTBNoKJRsptETrtW14Qm2+s6C3//OQ2XYXuyZqyg1mEjeVOTxKHrn1wJL5z79H/ORf9D+Q67nI/PQ9cG55h1dYsGmA5oPt2Bj1cTmNizsmZimNkDN/rf8wwCAlxZu09xeFaY895U2NeSftribjHDjPnsup1v2h+cNKNQs33EY2U/+iHm5oU8qwMKeiXpeXrQNd362VnNbgLIeW0zMNG5r9vtKTiFr2mx8u3avq8dtmZbsXe720FzNCetgpoQ4WRk6F1XA13zn5C3EKTmSl9a6wmMhO6cMC3sm6nn+h634arW5cHzw6w2ue6XY1bDNeG7+FgDA3z7Rfng5JbNJ3ZxDZXUt1u4p8a6T5ED6a/5h3dw3gc591ITY3KU83dh//YwCl11v9ZAfag0S40NyPiUs7JmoJGvabNz3hbm/vdKM8+Hy3brmGKe4bcZJSdT/yd43wXmlrMR4a6JAby5ALazJps9pqOc21Oa7/cdD4zU1e73HfPP1GnffzKzAwp6JWj7N8Y2k3V5c6tdHrZAGatZR47YMS06o0wgHPf0j8vYfR8GhMmRNm42PVuxyfNz4OGvCOU5HiKuFtcXDeQl11K/64ZSU4FwU7iguxUPfbECthZtdcPgkAGD/sdC55MqwsGdihjEvLPFrU/88F2856Oo50xsmunq8FMXr/4HjFXjz550Y9fxPAIA9R045Pq6eEAeA2RvqJhP1uqmFNdmMHS456dx8tufISaxTmJ2soH6ot0hN1u5ogZs+WIUPlu3GVhPbf/GJCnTKSAUAnKqqcXw+p7CwZ2Ia9eu8kfWhsroWk176Bb/mH7J8/GqXNdZklQYaqDeRTFqKdgLcE+VVPr7oeh9PpcpcdeSkvZq8pQ598wFgxLOLMdlmBLRbnxsAlFV4BHd1jf4xtx44gbOe+hE7ikMzN6AFC3sm5thz5KR3Wf2bH9lNPyXxvpJT2LjvuC3f7CqX7TjJicER9o00hP2O4lJMfTfHp03vYaiM3gX0Hx56lIdY01V/biWnnBeM31vieaP6YlWhbp/cvaH3vlHDwp6JOUYoEoXd+tFqn21G5gdZPOxWPCzMqDHQ9pyg9uLQe5b8aUA7W8fVemaMfmEJVhQc8WnTM/esUZlRGibZE/bqCeK1e0rwyLe5ul4+xSc8Ub2L8qxF9cpU1dTio+W7/Wz26jcTJ5yq1H9gac0XhRoW9kxMIycPk3n7151+Oej3HDmJDQ79oqtq3fbG8RX2esLQonONl2cll04zCo9qzwscV7ms2nXFVNv8L3zlV7z3+y6/ojAysqZ8rerNw4zXl2zHA19vwOc5vlp4WoPA51aM5me27GdhzzARx6SXlvqsj3h2MS54ealO7zpOf2SenznCyI7rBPVEpJ4Zx6p3jczPW61F0Cona5WotdpaIXD9eznIUb0Z6KGnWS/bcdjS/laRbeay6UXG7n06dqoKX632fWAcLtM3BVVUh35CVg0LeybieG5+HuZv3B+047dpop20TMZKgI+W5lpWWYN3fyvwaat2WbP/RFWYRe/wahu6kkADoLR4YcFWn/XiExVYsOkArn/PXPP+bt0+fLNW23//pI5ppOCw/0RnTa3Aw9/k+szJKDl4vBxfSf7tB1R+9XbnPvo+9gPu/GwdNu6re+NTH1NJhc7DbF/JqaDcDy1Y2DMRxyuLt+PG91f5tdfUCld+GOkNkxztpzy3nvfI9Ll5PutVLmv2TVTmBr0UBvM36tuyQyFb5DQSRy24VN728RpsLtLOpaPnovjYd5v82tbsPor3l+3C3z/Vji6+Q9HevplvSuqaWoH8g6XImjYby228TWzcWzfuHq0b6/ar0LiO938vwNDpi3Drx2ssny8QWNgz9YbOD8zBzR+sNu9owiYdwWIHPWHfv0O6z7rbrpcNk6xN0KrZUHgMD3/jmfB0w4PnyhnLMeCJBbrbW0spnxPsRlepsOOPLkft1kjX98rifOQfrLOV/7a9TojLkawyNULgmTmbAQCv/7xD9xzPzN2MborMnD9trYvLGNalhe5+WuUpP1qxR3MswcJU2BNRChGtIKJ1RLSRiB6T2jsS0XIi2kZEnxJRktSeLK3nS9uzgnsJTCwxL0DzTiBeF0rBeqqyRtNdsEtGI591ZWRpba0wfNW3NgZfQW1Vbl/y+m94f9kulFZUY6mNOAE9ftl2CEfKKnVt0bLg7dgiNaDznLKRIE1+rtTWCuwrOYXn5m/BJa/9ptlXbbOvqRVYmOcR3Ivy9APrXl+ywyff0dDOdQLeaJ5kh0buncY23VMDxYpmXwFgtBCiL4B+ACYQ0WAA/wTwohCiK4CjAKZK/acCOCqE6ALgRakfw0QE3wSQk0QpaEe/sASXvv67Rh/fdeXE3yuL8zHo6YW6NmUr2H1Y1dQKHC+v8nq1LNx8EFe/s9Lx+dVco3Msed7DKDLXCqcqrV+vLGxrhMDlby7z7C89kI+ZmJOcJmLLaFwXeWv3EFbSK7iJqbAXHuR3oUTpTwAYDeALqX0mgAul5cnSOqTtY8huViQmavlpy0EUn3CneMPGfcfw8iJ7ics+DKAykVogrCs85memUNvQlRO0SySPl0CSbvnbwI0FxqSXfkGfR3/wrt+hY892itI0okR+MDr55Zcoom+bNUqyLBTlB0tNLbBLykEjP59HPm9chN2p4FW+WdUKgRlLd+K5+Xn6OyjI0ckgGiws2eyJKJ6I1gI4CGABgO0ASoQQ8jtWIYC20nJbAHsAQNp+DEBzjWPeQEQ5RJRTXOxu4QQmMhFC4Op3VuKyN/w1Yidc8N+leP6HrYZ91JpwUYn1/DGnKmt8smC+9ctOvz5+9m/VqnKCVv5xV+j4jjvBzIyTF6aiIAelB7qT9ABK09fD3+Si0wNzLAljWdgfVXgiyZ+PWe6d71R28xcXbLVUhvGmDxSOBAJ44vtNeMWk/m+4sCTshRA1Qoh+ANoBGAigp1Y36b/Ws9zvTgkh3hBCZAshsjMy9EPUmehB/r1q2S9lRktJvewczwi1Nn7QxltFz0fm4V8Kl8IvV/uHw6vH0CzV19NHy/UykND8Sb0zHe/rFupJ5x83+QvFuZI/vlk+fy3ziVbZPjmD6c9bi5E1bbbmseT9lG9OAsJSKoZftvkqnP9ZuA1TZ3rcRn/NP4SsabNNJ9vdzLcTDGx54wghSgD8BGAwgHQikmcY2gGQHWULAbQHAGl7EwDWIiuYqMb7am/Qx+hBoIeRO6bd4KJAGdTJ9yVWK1gnkPS2Qzr7vSSHnNtUroLXafjSWw1SeusXf8+XCf/+xa/tsCTI//dTvu6xrnp7hV9bVY2wlD5Zzw8eAP7y1nIAwAfLjFNIq1NLRBpWvHEyiChdWm4AYCyAzQAWA7hY6jYFwLfS8ixpHdL2RSJUUQNMRFNnx3VXAMvfLq2vmVNty2nZP2WQDaBdlGOrSSlDNXn7j+OdXz0mJPXRwvHDmptr7hE1tIvnoWRUkWnBpgN4Zq41+7b8nXFiWtebV1BiZR5pywHjlAdHDQLZjLhueEdH+9nFiu9PJoCZRBQPz8PhMyHE90S0CcAnRPQkgDUAZkj9ZwB4n4jy4dHoLwvCuJl6iMvBpF4Kj57CjkOl2FfirzEfL3eWJ/3Vn5zZXZXeGYDH5FFRXYN4xQPOivBRImu61wzr6Gekj1Q9qpmUH753uya6faxE18pU1wg8Omujo7eitTZz3euR1sBYXOq9zew+bOx91ad9uuF2tzAV9kKI9QD6a7TvgMd+r24vB3CJK6NjogorZhw91HZXZaTq6Bd+QnWtwMCsZv47OpSFTic2m6uKYFTXCnR/aB7OPK2pt00vmZgVIkGzt4a7I3t/2S5NW74V1u62J+yzmjf0VpRSYpbBVK94ydnPGXsCZSu+G8GEI2iZkBHIBNYuxY/vVGUNXltSp3nLppKaCNRyZW1Pr1C3HY6drLKl2RYdc/5QCRR54nXFTnfs2E4FPQD8bjOZml49XrM6uQePOxtjqBzTWdgzISOQGBKl//oJHdOMlmdHuMW/m7VVNxYdw3u/+04SGj3fLvivvepNbmImGJduCzyKN1jojd0sqZ3Te223hKNTWNgzIcOJfVkIgQWbDvjYQ4+d0hb2WrbZUCv7ZRXVPuMwE3p6ZE2b7edi+MrifNw6uovlYwSiDQfKGoXppEDDw+qKGctDORxbqEs/ymjlt1GiDnjL23/c0kR/qBzGQpucgYlpnMi9GUt34snZmzFU4XJ47os/W95/9e7QRine9fk6n3Ut3+zz+xj7yiv91pVFU37NP+xXNtHtZ9nbV2fbLgiixU6FgB/1/E946fL+OHSiAteGyPMkGHyrk4ZZDy0XUk1Y2DPRhmyzlwXUF6sKMbxLC7Q2yC//5GxPJkK7Hiwyf/0w8CyZgaBVg9aozi3g67depkoE5nISTT9G92gVlOPeLvnmbztYig6q9MKRhtIcGIr8NWzGYaIOWdjLybnu/nxdRL/Ou4GWZm9nolrd1T/rpfvCKJjeIR+v2I1/zrPmW+8WRoF1fTVcQ5XC/sUfjdNx6GHnIREqMw4Le8YV9L7cQghvoJGydJ1snlDmG49GtD4WI+28TJUnXy3MQ6FpntFW3ze+PmKU0fLu8d392pR1g79Y5Z8iwwr/WWg9QV+o8kSysGcCZs6GInR6YI5m6t73ft+FSS8txW/5h/Dh8rqMk3IIuhJ19Gm0onYRnbOhCFnTZuO6mSv9hItSTt00srPfHMRelc9+kt1K4xo8MLEnLh/YPuDjuMnoHi2Dclx1MRjA15/eaaUxW8Le0Rnsw8KeCZj7vlwPAPhcQwvK2++pCrXzcJmpa5q60He0otbW5XmFHzcfRO5e3wee0uX0tSXbsXiLb8Ku3YoH7PIHxmDj4+NxaXZggjopIQ4jutpPTtiiUbJ5J4cESyAmJ/gLe6UHlds1hLXQ8y5zGxb2TMDILmk/aFSRypVqdC7bcSTkbpCRipFZQZ2Qy+wzi1MYfFulpSAxPk63Lm2wCaY1wqkLqxmZGs4BynkRvYLnbtKicfAekkpY2DOuoZViYIOkqS42KPWmh7KIRTRhJOzV+ffNkqZZqVyVZrH8nbKEoLqwuRXcKkqjRbA07OYabyNK000gZSz1yGru643EZhwmIjlUWoGsabPx3bp9OF5ehSe/3+Td1qJRku5+aSkJttMl9Htcv6B1faZWCNTUCgx5ZqFfmcRyVU1X2fU0EBIs2vHH9apzuxwaAamUlVRVh+5txWmJQquo8+4EWrrRKizsGVtskbT36XPz8MycPLy1tK56U28DL47z+7aJWjPOZzcOsdW/tLwaldW1KDpW7p3vkLFSaMMuWsLkvDNa+7UpTUiRVkk0GLni9dIvh8JOr4Rz4zARyQGpCtDeklPe/N1yHdbzemfins/XadaF/WHj/rDZkoON3QIpLy3K934Wai2y3IWShWkpviYYLcVeK9mX+s2rbXqDgMcSyXRt1UizPdiafbhgYc/oIoTw8xzZXHTcuyxPmsn/5+Xux+erCjXrwhYcPomhnVtonqf/4z/gjk/WaG6rDxxxULSi1yPzAfhPPBpVTDJiYu86TT1O9fDR0uzj4wjjT/eNllULuQYabolGhLgoWMDozTEEazJYD9bsmbDT+9EfcI6qJux36+oKM/+oKshcWmGcKEqtccocPVmFb2zmHXGbu8d1c7yvmxPJLR16ZijNLuoHtJawv/Ss9pg6vJNPmzrRl16qXz3s9jdjapDz6Dx/SV/N9mCbG9VzW2yzZ8JGaUU1sqbNRmlFtd9k0sET+vnU1bnLlfbnsT1bRrQZZ4jOW4cVurdu7No4lmwtNu+kgZHASIj33fbjnWdjcKfmGNixGWZMyfa2b1NFMyfpZH/Uw+lbiR7BNKe0SkvGsC6ee948Vd+xIBh8fP1g/H1snXLB3jhM2DjjH/N1tyl/fyO66gvI8qoa9Hh4nnd9cKfmPtkcI4U2kp91IIXJu7RshC9vHurWkByhHL16cjVBdW2dM+ps1crUCEoTHQAkxYfXLuM0VYEVFtw50rusNnsFkxFdW6Brq8b429iu3jZOl8BEPOkNPRqRlqDUCg5atsN9jwqnvHbFAKx8cKz3XSM+gB9cw6QEDOgQmjqiehjJq8Nllbhy8GnedaVwaZWmn3HULH97sLGS5O2/l/tVTLWE0qQYaHyAnYLh708d5F1u38wzAc6aPVNvsPK6LSBw48hOpv1CxYiuGchonOy1z8bZ+CXcf14Pv7Zwuyoanb/kZBUenNTT9jG3mAR0BZvINfoBtymKyDx0fi9Hx/j8xqF47YoBIXuzYGHP4EhZJY7rlPpzyk9b/CNmX9Dw0gkWPTPTDLfLNm55HsHOJNmNIzujkyLaNBIwG32lg0T46RpRtI0tRuO6QSA1i43o0lLb5RKw7hlz1zj/bJl2ad0kBRPOMC5k4yamwp6I2hPRYiLaTEQbiehvUvujRLSXiNZKfxMV+9xPRPlEtIWIxgfzApjAGfDEAvR59Afb+2nlapf52ydrfdZnLN2JvSXuFsD+6e5RutvMfrPqH7WVH/n401uhhzQZG2la51drjMvfNUryCOnbbJQ1/EPfNn5tTtIoOMXK/OzI7vYTtv2osNeridbAP8CaZl8N4C4hRE8AgwHcQkTye8uLQoh+0t8cAJC2XQbgdAATAPyPiOw57DJhxWjiVcncXP/EZ3ocOO5+3pQsA+3a6oSrnOjKimb/+pXZmHfH2dYGF2TsCt24OELB9Em2NNLxp3t895VvMWYxBdcN74ibRna2NTYtRvdoaclmn5aSiB1Pe/VM3DyqMz5Q2MXVPOTAnBUtmAp7IUSREGK1tHwCwGYAbQ12mQzgEyFEhRBiJ4B8AAPdGCzjLsdOVaGi2j88f3An37woe46c9EbORhp6KRrMZL0s2+VJSLtm02BUiLLD97cNd/2Y6jmVoV1a4Pf7R/t4rsjLKYnaouOh83uhUbI13U7P5XFsz1Z4++qz0L6ptfKFSpv39+v3GZp/AvG6knnnmrP82sxKTQLASw4nk93Cls2eiLIA9AcgV564lYjWE9HbRCTXMmsLYI9it0IYPxyYMNH3sR9w0f9+865nTZuNmb8V+PUb8exiDHp6YQhHZh09bwyt6M9hXeoeYuq6n+GeYLVLexfruA7q2AyAtsDKbNLAR0C2TW+AgumTsPyBsbqBaLLw7WpgGwf0I3RrpNw0vTVKBpqx58gpNG2o7zfvxl0epfE5vXlVXbyC3ptxqs2IZLexLOyJqBGALwHcIYQ4DuBVAJ0B9ANQBOAFuavG7n6PWiK6gYhyiCinuNhZIAnjnG2Sp8XGfb6+1f+YtTEcw3FMok7gT8Mk/4nEBIXLjSy/5PS/ekLg+hHBjeIMBKMso3aQg67s5P9q0iARt47u6tP2x/4enU42iZlNsOoVs1Gn4bBD4+QEpFp8s3CKlmKgDEB79A+na+53tgXtP5hYEvZElAiPoP9QCPEVAAghDgghaoQQtQDeRJ2pphCAslROOwB+sfBCiDeEENlCiOyMjPB+CLFIcWnwco87wUwL1EPrrXxsz5Z4WMMdTlkFSv7BJkuZD/Vs9r3bafvPy/t/fpO9jJdu4pYlSb72QLM9pjf0zCPEe4W9cX+9eRzZlVcvvYYR8fFkrNkH8Q1umuSSm9E4GZec2c5nW9v0Bq6nk7CLFW8cAjADwGYhxL8U7UqfoT8CyJWWZwG4jIiSiagjgK4AVrg3ZMYNjL54Rn7z/YMUPETkzJ6qtc9bU87yKxABAGWVdUFC8l7HpZJwejKgfVPjzI+h9E5R888/9XHlOHKErdF9//624XjtigGGx5Ft8LIZx+zhcUZbbfdYWaN/aFJPPGLRh13O0NmuaQOkN0zE+X0yNR/EWt/fyf38vY6ccOPZnbDj6YlIS0nEMxf19tlmN/VEMLAygmEArgQwWuVm+SwRbSCi9QDOAfB3ABBCbATwGYBNAOYBuEUIEfzaXowtWjXWj5w0ErqntzH2X7fCoxf0woWqH1gcEZo5yFGSoVP3VEuDG9WtpWK7578c6aun2evVVd15qMyvrbVBNGowOPO0puadLBAvmbeMhP0ZbZtY9gk/Q/qO9G9vPL42TbQfpLXSOFKTE3CtxehUeXK5b7t0EBFe/vMAnJXVzKdPzkNj0UfjTe2JC8+wdA4ziMj7oFMXjFGnrAgHVrxxlgohSAjRR+lmKYS4UgjRW2r/gxCiSLHPU0KIzkKI7kKIucG9BMYJRknJ9IT920t3ouRk4MFXU4ZmITXZ16ZeUV3ryMPFzmv5PEWNXHm/RtI4GupMniXreJ3IKGMH/jKog+WxuIHepdsVK3KQUdMAE4LJz4pBnZpj+QNjfNIua6FV/xUAaix+D/oqJnDJGySnzZJ7Ruk+uPXMRaMc+PDrYbVaWDAJ/wiYoCGEwKK8A5rBT0a/Jz0l5PHvN+H79UXaG21ARLhYZdOsrK7VtPEqPWiCid5Dw8yMfbKi7qX15lGB+5e7gd1H5l3juuH9qQP9NGG7KD25jHLuyOgJdavZLpVxFrKicEqnQLiTNMLPumQmA/yTzIUDFvZRRnVNLW7/eA027TuOV5dsx7Xv5uAJRZ1YGSNPCbVbYjDo38H3Fb9FoyTU1Aq/4tjn9vQtsBEIsouhFnoPuETJU2WgjiBUCqxQa29W71Oyib04MT4OI7oGrsUeVgVcDeroeVC/8mdtW79aB5FNHVZ81gHf5HVyLd+vdSKJzfLPTOrtb6IKdDJX77rDBQv7KGPHoTLMWrcPE1/6Bc/O2wIAmPn7LszLLUKZorjI+sJjeocIC/FxhFoh8CeVxq/8kVqN7NXjuhH6idi0BOel2e3RvFEyCqZPwmc6XjdWA4jcpGD6JM+CjixSm8OSwmRCaJqahILpkzCpT6amSadG9dok55dXpmA2QvnduELK6qlnSjMzmb94aT+seHCM3z5P/7G34wC2SX1Cl/fGCizsowy97/RNH6zGtK82eNeX79RPNzw3N3BTjV2ICEJ4XrevHprl0y5z7bCO+G3aaEfHv/+8HprauSwYSeOXcG4v/bcK2QunV6Z54I+ebdpNOiiCrNQvbZP7u+NtYkYPgyIu3Vt5Jm1lU9fAjs1QXeM70LvGdcODE3viAo2cPEr+fWk/AHXzLYDHx/+tq7LxmI6Pu555RyYpIQ4tVU4L8XGEPw/q4JPzvz7Dwj7KMHrz3H2kruqU0WTo6t0luttG92ipuy0QjpRVorSiGodKK5CSWKctKzWyuDhCG40i2B9ep58LRaZVWophGmOtj23fMf3EbfKErpXJxFJFXninWSO1JpCV9/rZi/Xty+eFKLOiUVKyW0d3wZc3D8V9E3rgzauy8f7UgX5BU8kJ8bj+7E5+DgK3j/EN3jq/TybuHtcN94yvy/NDRBjbq5WuKa2Rg8/9pMkDor7Bwj7qMJD2CsFkZQJN+xDByQkjuzJ+u3afz+u9cmJN71Vcfv0HgDeuPFP3HHYn6YwmCmWBVGthMvGEwnw2pJP2hLPaD1vtpy3nAFIH68go507BDLIAAB5KSURBVGDUIwpVJoi2Gg9imfg48rqKnturFZIT4i1PxGarXEwT4uNw6+iufh5dRlg1ZXXKqJv0VR5f+WCRuXxgB812La4ZlmWpXzBhYR9lGP2wjygKYzv10Q5iWVAvyrwvPpq9wcWdJgVR6b1yt0pLMfxstCbjurbUN0skeAOH7H0getegFkbqdXm/sxSTzMojpSrSQ6ifx6EoaD2sS3NcMeg0844K1MJezx1YmWbgkxsG2x8crJcenHVrnX1emcvmNI0gvWcu6o1bzrGWMtrouxQqWNhHGUZf6T1H6swSTos5Oy2IbQel0FAKLiOZNfOagbh9TFdN+/hzF/fBkM7NbQs9I/kgexPp+efroTeERJN6r17ThuLzUNqse2TWCZN+qijRUCj2fxrQznbFJav+9ACw4oEx+PLmoX4ZWa1iteyk8jNVmpMCVXL0ooVDSejKzjAhwaq72M/bIjf5nFJoLMqrq3hl5GqY1SIVd56rnYUxo7EnmMaugmv0WT5zUW9cMyzLtjlMT76ZuW2SV9YLRRvhg6mD8P6yAiQn1D10/tC3DVo1Tsalbyzz9gsWk/pkYvb6Ikdup2qFw+j+tkxLQcsAIpSdpOJQfm5OzZf5T52HgsMnDatjhQrW7KOI4+VVun7GMq8t2Q4AeOfXghCMKHDi4wj92ns0VacR57JA1hImsneIli+6kYxMSYzXDL03Q8scAAAX9PH1QFGfW09YDe/aAq9fme3XrpzIDmbuffnYTu6NOuulUVR3oITClKVFQnxcRAh6gDX7qMJKacHpc/NcqSQUKpQTl04LM8sapNbut47u6peq13u+IAiIjhrVtQZ0SEc7VcK1MT1bIS0lAcclTx5ZMFo1JyhTAARzmkWeS3fyWQWrxqwWgeamkYeqVaqxvsCaPeMKcnpbtyk8esorrALNJWXFnKFMtbxi5+HATqg5Bv+27q3T/K6tSYNEzL59hHf913zPWL5b55ctXJMmivsRTJlaG4Bm73TeyAlOFYVogoV9PWb5jsPImjYbBRoZGENF89QkbHlyQtBCw1ftOqpYc/aDlQWs2e99zcPn+nhjHCo1rrdqh/sm9ECvzDS0VmR6bCNNJv+hbxvNB5GWgDqh8Nm3SqVBYfhAkeW1k3kBdVCV1UnUYBMsxSXcsBkngjlwvBylFdW64ePfrPVoeb9uP2RYfDuYpCTGIzkh3kcMd2jW0CeAyy2cygLZVm8mkNRZH7cXlzo7oQY3j+rsnR8474zWmHBGa0zuV1etM2+/f6IsrdE6sWuv31NiOd+MXWQvIidmEqXb6gMTe0SMbXvtI+PCPYSgwMI+gpHrvnpzoahIkn5oVdXB09zMGCj5fSsF6ZmnNdUV9p0znD+UQq33bZVKN7rNq1f4B35pmVq0nk25e+1nT3QaQGeFJy48A+2aNsCo7vYjq5U2+xvOjux5pLQGHlHZKk07TXJ9gIV9PaZK0ozCWWJQjvRUCiYj7w+tdAdK1JWfOrZIDdjobJQmwYhQBsLIV9gzMw3T5c/Upcfb6J7BSXEBeIq7PDjJWjUpNS0b1x/BeU73lvjX//WNuORmdmCbfT1mzgZPwrJXFm8P2xjkPDZKsbS5SF8jNpuUO3bKtziK0uUwmP7iWgQrD5AW8gNySKfm6Bugq6kavaId4SZbSkx3ncVqVOGEiHDRgHY+8Qz1DRb29Ri5apS6xJ8dPphqnkTMCsrJxC0G5g8zd7uemZ5Iw2uHeQRAIAU1ZN95py8GoawbKrwTnYpGDWFvZ/Lwrauy8anD9AKhxEnAE2MfFvZRQEJ8nKWEXDJZ02Z7l4cHmCNexurP1UzwPj7Zk6L2ppGd0LRhok+6Y7s4qWmrxKzoh1WsJOHScmFs1rBu/PIEq53Sh2N7tcIgh+kFooV7xnc3TL0cS7Cwr8dcdlZ7AJ6MiP/7KT9o51ELXPm8SqxaWJSavVYmQNkdr2VaCtY8Mg7dFT9Uu5GgRcfKAQB7j+qnKjYiOdH5K/tfR3X2ZuDsoBM1q6QulqDug1SmIHj9yjPx0fWDcPc4a1kWGQ+3nNMF8+44O9zDiAh4grYekxAvuxQCz/+wNWjnaZPu682hLNwtY2RP3/T4eCzZUoybP1ztEwGqFU2qDqEH6kxEdmNwZBfQds2MJ4X1cKLZt01vgN5tm+DeCT0AeEwpfdqbF7/wPgT1iogTMLSzO29hkUboQqtiGxb2UUKz1CQcKXMvCEhJlSr4peRkFR6c2NNH6zZS7BsmJXiTWCknaLX20cqEKAfb2I24TEmMk87jzCbsRNj/qqqkNdag2pUdIiXgyE2i74oiG9NvMxG1J6LFRLSZiDYS0d+k9mZEtICItkn/m0rtREQvEVE+Ea0nosiquhtFyHKxvKomaIIe0M59f/3ZnXzyjJt5yshavFwrFAC+X+9f/vBsjcLX8gSeXWEve3uo3TmtEsoJWjPClcgrmLSV8gEp6xcwwcOKZl8N4C4hxGoiagxgFREtAHA1gIVCiOlENA3ANAD3ATgPQFfpbxCAV6X/jENOVlajYZL/rZJF39Nz8oJ6/gwL/tC5e40LmDeTik8r2bTPP0BIyzPj6Yt64+VF+bYLrjx6wem4emgWWjusAevWBK0bRGNul0m9M9HsuiQM6Rzbk8ihwvTbLIQoEkKslpZPANgMoC2AyQBmSt1mArhQWp4M4D3hYRmAdCKqv5EILiGEcJxqttcj83WOGciIrJNoISpJq/jGvRO64zmD2qhWldXOGY3w4qX9bGvaSQlx6NbKuSdGUnz99amuDxARhnZpEfL4iVjF1q+HiLIA9AewHEArIUQR4HkgAJAjUNoC2KPYrVBqUx/rBiLKIaKc4uLILaThFh3vn4O7PlsX8HFeX7IdP0gTpBVVdQWRz8pyVmbQCu2aNsDFOrVPZZR1YGW6t2qMS7L9PXdkjjtI6gUA/7msH2ZM8c/h7jaJCaETQt55BZ6tZIKEZWFPRI0AfAngDiGEUYIO7fxN6gYh3hBCZAshsjMygpOkKdL4yqSwiBHHpACqZ+bm4Yb3V/kdzzc7pDWmDDGvGdqjdWPExZGhhg5om1+ClcJ2cr+2GNPTnYlP7eN7gtQymzjz4nECK7dMsLEk7IkoER5B/6EQ4iup+YBsnpH+y/XjCgEo1bl2AKwl4Y5S7AQ86fHb9kM+6+8v2+V7DgensFLPU37FNnvV1hL2ai8eM+6T3BXDzX8u64+C6ZMcT+w6Qf74QlnQg4ktrHjjEIAZADYLIf6l2DQLwBRpeQqAbxXtV0leOYMBHJPNPbHKzsOB55tX1/h8+JvcgI95pgXTz+Yi/5c4rYhE2TWwqSKcv1xhZtIiSxVs9OeB1qNDo41mqZ5JcHWa5Wcu6o2+7cz99BnGDCveOMMAXAlgAxGtldoeADAdwGdENBXAbgCXSNvmAJgIIB/ASQDXuDriesjyHUcCPsb+4+V+2n2gNE4211wzNTxZtLR42TVQqZdWmKRevnNcd9z+8RrveiS5Ooaai/q3hRACF/b3nd66fGAHXB7DD0HGPUyFvRBiKfTjH8Zo9BcAbglwXFGFVlSomoMnypHRKNlrLlHb4P+7cBsOnnA3lXGDJHNvk/SG/vllxvVq7dcmW3mUJqvKamPNvqzCd4JWy6MnVoiLI8PJbIYJlNhVpUKIme03d+8xDHxqIT7PKfS2qQtnuCXo7SYH03IXvW10F782cqDZK/OZF0yf5GeqYhjGPfjXFQIqdDTcrGmzkTVtNj5cvhsAsKKgztwTjBzkfx/bDRdKpfCmnWdtMnScRri/VoBPqvSWoJxkNRP2IzSiZRmGCQ6cGycEmAm9j1d4hH2ipNl+tnIPdh1xPqk7Y0o2ps7M8WufOqIjGibG465x3ZCabH7rVz98LtIteqQkxMd5I2QfkiaP9R5yMpzHnGFCBwt7FymvqsHGfcf9wvorqqzViJWLNt/75fqAxqH0QR/ZLQNLtnqC1uLIo5VbEfRA4Pngza5blvWRlJaAYaIV/pW5yKOzNuJPr/6GXZKr5Wc5e5B/sNRUw5UJhqL77jVneZfdqmlqFbO5CiLCfRN6YNatw0M0IoaJXVizd5FNkk96yckqnNYcuPeL9UhOiMP1Izp5++wrOaVbdNvNHCEfXz8YLdOSfY7ptPC2Xc48rSlW7TqKszqalxS8eVTnEIyIYRgW9i6yvtCT+fFwWYU3VUBFda2PZp9g4F7oZhpbrUyCdo7fVueBZIUGUoUnKy6noeaFS/oikc1GTAzCwj4IrCw4iiGd6hKDKSdodxSXoWXjFM2UwMEOlQ+VEUf2l49EYf8nk4RuDBOtsIoTBI6dqsI9X9RluFSmDZCXH9JId/DubwX4dOVuW+d6aFJPy31DVQDjkQtOx9ndMnweeAzDhBfW7IPAR8t9BbZSs1+ytRiz1u7TzQh535cbbJ1rytAsPDl7s6W+ZgUwWqelYP9xT5FurefCm1dlW8rJ37FFKt67dqClMTEMExpY2IcApQviO78WAACSXIoWVdcm/UPfNo6PpXwWaAn7c12qp8owTOhhM04AlFVU47n5eag0CZoqq/Qv0lHpkj1bra2/eGk/x8dSeu6E2k2TYZjgwsI+AK6bmYNXFm/HN2uNi5L8ss3dbJVGaEWlWk0wdk4PZQFx14bEMEwEwMI+AH7fcRgAsGZ3ScjPTQTT6lEyP/x9JF68tK9pv39ccDoW3TUSDZPiMS1CCokwDOMObLN3gd0B5LExY94dIzDh37/4tV8x6DTLKXE7tkhFxxaputu/u3U45m/cj8T4OHTKaIRNj09wPF6GYSIT1uxdoE2TBkGrt9o8VTv75Ucr7LloGtG7XRPcPb67a8djGCbyYGHvAkO7NMftn6wx7+gAoarV3q99OoDgFfNmGCY6YWHvAmUVNZi9PjhldlMSfatJrd0T+vkBhmHqPyzsXeDRWRuDduyGiealA7+8eUjQzs8wTHTAwl6DaV+uxyWv/Wa5f3UQTSpWSvWVW8yXzzBM7MLeOBp8snKP4facgiO4+LXfvetxBITThB6MEoYMw0QXpmojEb1NRAeJKFfR9igR7SWitdLfRMW2+4kon4i2ENH4YA08nCgFPRCe8nqt0uoEfLdWjUJ+foZh6hdWNPt3AbwM4D1V+4tCiOeVDUTUC8BlAE4H0AbAj0TUTQhhrVRThPPIt7nomZnm1x7kzMSa3DWuzlXSzaInDMNEJ6bCXgjxMxFlWTzeZACfCCEqAOwkonwAAwH8brxbZLLrcBlOa14XjPTe77s0+wXTZq9HUUl5yM/JMEz9JZAJ2luJaL1k5pErbLcFoDR4F0ptfhDRDUSUQ0Q5xcXFAQwjeIx87iesL4xMV8fOLfUjYhmGYdQ4FfavAugMoB+AIgAvSO1a9gRNtVcI8YYQIlsIkZ2RkaHVJSLYfeSkX1v/Duk+6/dOCH30aeHRUz7rvds24XquDMPo4kjYCyEOCCFqhBC1AN6Ex1QDeDR5ZcKWdgD2BTbEyEOd+OyLnEJHx9kcQA6aU5W+0yDf3TYc93HyMoZhdHAk7IkoU7H6RwCyp84sAJcRUTIRdQTQFcCKwIYYXuS87ifKq3T77DjkLBFag6S6gKl5d4zwLmc09njaZDVvqLuv3IdhGMYKphO0RPQxgFEAWhBRIYB/ABhFRP3gMdEUALgRAIQQG4noMwCbAFQDuKW+e+I8Oz8Pk/pk4vI3l7l63LvHdfNZ79G6zstn5YNjTfcf2rm5q+NhGCa6seKNc7lG8wyD/k8BeCqQQUUSuw6fRL/Hf0DJSX3N3gm3ju7q1/bhdYOQf7DUrz37tKbI2XUUo3u0xKK8gwCAZAtpFBiGYWQ4XYIF3Bb0rdNSNNuHdWmBKUOz/Npfv/JMNE9Nwp3n1r0NWK0+xTAMA3C6BC+HSyuwcd9xnN0t+J5BN43s5F3u2z4dI7q0MOzfvFEyVj18rk9bYhw/pxmGsQ4Le4m/vLUceftPIPcxdzI8/HjnSGQ2ScGO4jJc8PJSn23tm9VNvH57yzBHx09MYGHPMIx1WGJIbDlwAgDw1OzNrhyvY4tUpCYnoHe7Jt422bumVxv/lAt2SWFhzzCMDVizB5C3/7g3v82+klPGnS2ilRxt0V2jcOxUFZqmJjk+7tpHzsX+4+WWUh8zDMPIsLAH8MmKugwPp6qC5ykaF0cBCXoASG+YhPSGgR2DYZjYI+aF/UPfbMAHy+qKd6/YeSSMo2EYhgkOMW8LUAp6hmGYaCXmhT3DMEwsENPCvromsNqt7107EL/ce45f+9y/jdDozTAMEz5i2mb/yKyNAe2vF4ClVc2KYRgmnMS0Zj9nQ5HjfXu3rfOfb8kZKBmGiXBiWrMPpHbsjKuzvctL7jkHOw6VYtJLS/GPC3r59f3qr0NRXROGQrUMwzASMS3sA6Fl47pkZg2S4nF6myYomD5Js++ADk012xmGYUJFTJtxGIZhYoWYFvbCoR1nbM+WLo+EYRgmuMS0sHfCiK4t8MaV2eYdGYZhIoiYFvZ6ev3401vp7pPVPBVxGknOGIZhIpmYFvYnyqs12+dvPKC7zxCu/cowTD0k5oR9Ta3A/mPleOTbXL9tZlWqLj6zHSb2zgzW0BiGYYJGzLle/nNeHt74eYfmtkbJxkW8OygqTDEMw9QnTDV7InqbiA4SUa6irRkRLSCibdL/plI7EdFLRJRPROuJaEAwB2+XlQVHdAU9ACSZFATp3yHd7SExDMOEBCtmnHcBTFC1TQOwUAjRFcBCaR0AzgPQVfq7AcCr7gzTHS557XfD7ftKyr3Lc24fgXeuPguXD+zgbRtuUhicYRgmUjEV9kKInwGoK3pMBjBTWp4J4EJF+3vCwzIA6URUb4zcKwrqLrNXmzSc06MlhnWpm5AlYi8chmHqJ04naFsJIYoAQPovRxm1BbBH0a9QavODiG4gohwiyikuLnY4DPdY8cAYzfZmXAKQYZgowG1vHC3VV9OdXQjxhhAiWwiRnZFh7AUTCL9tP4SPlu9GZbVx7vqWaSma7a2baLczDMPUJ5wK+wOyeUb6f1BqLwTQXtGvHYB9zocXOH9+czke+HoDKi0UKpHTFr91VV2EbAanL2YYJgpwKuxnAZgiLU8B8K2i/SrJK2cwgGOyuSfcHCmt1N3WOSMVABAvRcY2Ta0z3TROSQzuwBiGYUKAFdfLjwH8DqA7ERUS0VQA0wGcS0TbAJwrrQPAHAA7AOQDeBPAX4MyagekJOlfavfWjQEAXVs2AgA0aeAv4FulsYbPMEz9xTSoSghxuc4mvxlN4UkjeUuggwoGRsVD+rbz+M8/ceEZmNyvLbpIQl9GL089wzBMfSGq0yUcKq3wLg+dvki33/UjOgEAUhLjMbwr+9IzDBN9RLWwv+Kt5Zb6cRZLhmGinagW9nn7T4R7CAzDMBFBVAt7K2x8bHy4h8AwDBN0olbYV1vwq7+wXxukJsdc4k+GYWKQqBX2XR6ca9qndzvOYskwTGwQtcLeCuVVNeEeAsMwTEiIaWE/4YzW4R4CwzBMSIhpYS/046wYhmGiipgS9lufPM9nPS2FJ2cZhokNokra1dQKfLJyN/4vu71Pe3JCHH6+9xwkJcThgYk90CApAWN6tNRNa8wwDBNtRJWw/2LVHjz4dS5KTlZ52+4/rwduHNnZu37D2Z21dmUYholqosqMU1rh8a5Zs7sEAHDjyE4+gp5hGCZWiSphXyUFUv24+QAAYN2eknAOh2EYJmKIKmE/fW6ez3r+wbIwjYRhGCayiCphr0aZ4phhGCaWiWphzzAMw3iIamGf89DYcA+BYRgmIohqYd+iEdeNZRiGAaJY2PeQiogzDMMwUSzs371mYLiHwDAMEzEEFEFLRAUATgCoAVAthMgmomYAPgWQBaAAwP8JIY4GNkxz5HTF7Zs1wPSL+qB1E06FwDAMI+OGZn+OEKKfECJbWp8GYKEQoiuAhdJ60Bn+z0UAgPQGSRjWpUUoTskwDFNvCIYZZzKAmdLyTAAXBuEcfhwqrQQAbNh7LBSnYxiGqVcEKuwFgB+IaBUR3SC1tRJCFAGA9L9lgOewxcTeXJCEYRhGTaBZL4cJIfYRUUsAC4goz3QPCenhcAMAdOjQIcBh1PHipf1cOxbDMEy0EJBmL4TYJ/0/COBrAAMBHCCiTACQ/h/U2fcNIUS2ECI7IyMjkGEAAAZ08BQPT06ID/hYDMMw0YZjYU9EqUTUWF4GMA5ALoBZAKZI3aYA+DbQQVqhY4tGaM3FSBiGYTQJxIzTCsDXRCQf5yMhxDwiWgngMyKaCmA3gEsCH6YxczYU4cvVhcE+DcMwTL3FsbAXQuwA0Fej/TCAMYEMyi5//XB1KE/HMAxT74jaCFqGYRimDhb2DMMwMUC9F/alFdXhHgLDMEzEU++F/Wcr94R7CAzDMBFPvRf2S7YWe5ffuirboCfDMEzsEmgEbdhp3igJAPDtLcPQt316mEfDMAwTmdRrzf5kZTW+Wr0XANC+WcMwj4ZhGCZyqdfC/pFvN3qXGyXX+5cUhmGYoFGvhf1pCm0+KaFeXwrDMExQqdfq8G1juiI1OQF7S06FeygMwzARTb0W9gBw7fCO4R4CwzBMxMO2D4ZhmBiAhT3DMEwMwMKeYRgmBmBhzzAMEwOwsGcYhokBWNgzDMPEACzsGYZhYgAW9gzDMDEACSHCPQYQUTGAXQ53bwHgkIvDqQ/wNccGfM2xQSDXfJoQIsNKx4gQ9oFARDlCiJhKZM/XHBvwNccGobpmNuMwDMPEACzsGYZhYoBoEPZvhHsAYYCvOTbga44NQnLN9d5mzzAMw5gTDZo9wzAMYwILe4ZhmBigXgt7IppARFuIKJ+IpoV7PHYgovZEtJiINhPRRiL6m9TejIgWENE26X9TqZ2I6CXpWtcT0QDFsaZI/bcR0RRF+5lEtEHa5yUiotBfqT9EFE9Ea4joe2m9IxEtl8b/KRElSe3J0nq+tD1LcYz7pfYtRDRe0R5x3wkiSieiL4goT7rfQ6L9PhPR36XvdS4RfUxEKdF2n4nobSI6SES5irag31e9c5gihKiXfwDiAWwH0AlAEoB1AHqFe1w2xp8JYIC03BjAVgC9ADwLYJrUPg3AP6XliQDmAiAAgwEsl9qbAdgh/W8qLTeVtq0AMETaZy6A88J93dK47gTwEYDvpfXPAFwmLb8G4GZp+a8AXpOWLwPwqbTcS7rfyQA6St+D+Ej9TgCYCeA6aTkJQHo032cAbQHsBNBAcX+vjrb7DOBsAAMA5Cragn5f9c5hOt5w/xAC+KCHAJivWL8fwP3hHlcA1/MtgHMBbAGQKbVlAtgiLb8O4HJF/y3S9ssBvK5of11qywSQp2j36RfG62wHYCGA0QC+l77IhwAkqO8rgPkAhkjLCVI/Ut9ruV8kficApEmCj1TtUXuf4RH2eyQBliDd5/HReJ8BZMFX2Af9vuqdw+yvPptx5C+UTKHUVu+QXlv7A1gOoJUQoggApP8tpW5612vUXqjRHm7+DeBeALXSenMAJUKIamldOU7vtUnbj0n97X4W4aQTgGIA70imq7eIKBVRfJ+FEHsBPA9gN4AieO7bKkT3fZYJxX3VO4ch9VnYa9kl650fKRE1AvAlgDuEEMeNumq0CQftYYOIzgdwUAixStms0VWYbKs31wyPpjoAwKtCiP4AyuB59daj3l+zZEOeDI/ppQ2AVADnaXSNpvtsRtivsT4L+0IA7RXr7QDsC9NYHEFEifAI+g+FEF9JzQeIKFPangngoNSud71G7e002sPJMAB/IKICAJ/AY8r5N4B0IkqQ+ijH6b02aXsTAEdg/7MIJ4UACoUQy6X1L+AR/tF8n8cC2CmEKBZCVAH4CsBQRPd9lgnFfdU7hyH1WdivBNBVmuFPgmdiZ1aYx2QZaWZ9BoDNQoh/KTbNAiDPyE+Bx5Yvt18lzeoPBnBMeoWbD2AcETWVNKpx8NgziwCcIKLB0rmuUhwrLAgh7hdCtBNCZMFzvxYJIf4CYDGAi6Vu6muWP4uLpf5Car9M8uLoCKArPJNZEfedEELsB7CHiLpLTWMAbEIU32d4zDeDiaihNCb5mqP2PisIxX3VO4cx4ZzIcWFyZCI8XizbATwY7vHYHPtweF7L1gNYK/1NhMdWuRDANul/M6k/AXhFutYNALIVx7oWQL70d42iPRtArrTPy1BNEob5+kehzhunEzw/4nwAnwNIltpTpPV8aXsnxf4PSte1BQrvk0j8TgDoByBHutffwON1EdX3GcBjAPKkcb0Pj0dNVN1nAB/DMydRBY8mPjUU91XvHGZ/nC6BYRgmBqjPZhyGYRjGIizsGYZhYgAW9gzDMDEAC3uGYZgYgIU9wzBMDMDCnmEYJgZgYc8wDBMD/D9pwksMstgtRgAAAABJRU5ErkJggg==\n" + }, + "metadata": { + "needs_background": "light" + } + } + ], + "source": [ + "def running_average(x,window):\n", + " return np.convolve(x,np.ones(window)/window,mode='valid')\n", + "\n", + "plt.plot(running_average(rewards,100))" + ] + }, + { + "source": [ + "## Varying Hyperparameters and Seeing the Result in Action\n", + "\n", + "Now it would be interesting to actually see how the trained model behaves. Let's run the simulation, and we will be following the same action selection strategy as during training: sampling according to the probability distribution in Q-Table: " + ], + "cell_type": "markdown", + "metadata": {} + }, + { + "cell_type": "code", + "execution_count": 23, + "metadata": {}, + "outputs": [], + "source": [ + "obs = env.reset()\n", + "done = False\n", + "while not done:\n", + " s = discretize(obs)\n", + " env.render()\n", + " v = probs(np.array(qvalues(s)))\n", + " a = random.choices(actions,weights=v)[0]\n", + " obs,_,done,_ = env.step(a)\n", + "env.close()" + ] + }, + { + "source": [ + "\n", + "## Saving result to an animated GIF\n", + "\n", + "If you want to impress your friends, you may want to send them the animated GIF picture of the balancing pole. To do this, we can invoke `env.render` to produce an image frame, and then save those to animated GIF using PIL library:" + ], + "cell_type": "markdown", + "metadata": {} + }, + { + "cell_type": "code", + "execution_count": 26, + "metadata": {}, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "360\n" + ] + } + ], + "source": [ + "from PIL import Image\n", + "obs = env.reset()\n", + "done = False\n", + "i=0\n", + "ims = []\n", + "while not done:\n", + " s = discretize(obs)\n", + " img=env.render(mode='rgb_array')\n", + " ims.append(Image.fromarray(img))\n", + " v = probs(np.array([Qbest.get((s,a),0) for a in actions]))\n", + " a = random.choices(actions,weights=v)[0]\n", + " obs,_,done,_ = env.step(a)\n", + " i+=1\n", + "env.close()\n", + "ims[0].save('images/cartpole-balance.gif',save_all=True,append_images=ims[1::2],loop=0,duration=5)\n", + "print(i)" + ] + } + ] +} \ No newline at end of file diff --git a/lessons/6-Other/22-DeepRL/tmp.ipynb b/lessons/6-Other/22-DeepRL/tmp.ipynb new file mode 100644 index 00000000..ce101f32 --- /dev/null +++ b/lessons/6-Other/22-DeepRL/tmp.ipynb @@ -0,0 +1,300 @@ +{ + "cells": [ + { + "cell_type": "code", + "execution_count": 1, + "metadata": {}, + "outputs": [], + "source": [ + "import numpy as np\n", + "import tensorflow as tf\n", + "from tensorflow import keras\n", + "import matplotlib.pyplot as plt\n", + "import gym" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "metadata": {}, + "outputs": [], + "source": [ + "env = gym.make(\"CartPole-v1\")" + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "<>:74: DeprecationWarning: invalid escape sequence \\g\n", + "<>:93: DeprecationWarning: invalid escape sequence \\d\n" + ] + } + ], + "source": [ + "class REINFORCE:\n", + " def __init__(self, env, path=None):\n", + " self.env=env #import env\n", + " self.state_shape=env.observation_space.shape # the state space\n", + " self.action_shape=env.action_space.n # the action space\n", + " self.gamma=0.99 # decay rate of past observations\n", + " self.alpha=1e-4 # learning rate in the policy gradient\n", + " self.learning_rate=0.01 # learning rate in deep learning\n", + " \n", + " if not path:\n", + " self.model=self._create_model() #build model\n", + " else:\n", + " self.model=self.load_model(path) #import model\n", + "\n", + " # record observations\n", + " self.states=[]\n", + " self.gradients=[] \n", + " self.rewards=[]\n", + " self.probs=[]\n", + " self.discounted_rewards=[]\n", + " self.total_rewards=[]\n", + "\n", + " def hot_encode_action(self, action):\n", + " '''encoding the actions into a binary list'''\n", + "\n", + " action_encoded=np.zeros(self.action_shape, np.float32)\n", + " action_encoded[action]=1\n", + "\n", + " return action_encoded\n", + " \n", + " def remember(self, state, action, action_prob, reward):\n", + " '''stores observations'''\n", + " encoded_action=self.hot_encode_action(action)\n", + " self.gradients.append(encoded_action-action_prob)\n", + " self.states.append(state)\n", + " self.rewards.append(reward)\n", + " self.probs.append(action_prob)\n", + "\n", + " def _create_model(self):\n", + " ''' builds the model using keras'''\n", + " model=keras.Sequential()\n", + "\n", + " # input shape is of observations\n", + " model.add(keras.layers.Dense(24, input_shape=self.state_shape, activation=\"relu\"))\n", + " # add a relu layer \n", + " model.add(keras.layers.Dense(12, activation=\"relu\"))\n", + "\n", + " # output shape is according to the number of action\n", + " # The softmax function outputs a probability distribution over the actions\n", + " model.add(keras.layers.Dense(self.action_shape, activation=\"softmax\")) \n", + " model.compile(loss=\"categorical_crossentropy\",\n", + " optimizer=keras.optimizers.Adam(lr=self.learning_rate))\n", + " \n", + " return model\n", + "\n", + " def get_action(self, state):\n", + " '''samples the next action based on the policy probabilty distribution \n", + " of the actions'''\n", + "\n", + " # transform state\n", + " state=state.reshape([1, state.shape[0]])\n", + " # get action probably\n", + " action_probability_distribution=self.model.predict(state).flatten()\n", + " # norm action probability distribution\n", + " action_probability_distribution/=np.sum(action_probability_distribution)\n", + " \n", + " # sample action\n", + " action=np.random.choice(self.action_shape,1,\n", + " p=action_probability_distribution)[0]\n", + "\n", + " return action, action_probability_distribution\n", + "\n", + " def get_discounted_rewards(self, rewards): \n", + " '''Use gamma to calculate the total reward discounting for rewards\n", + " Following - \\gamma ^ t * Gt'''\n", + " \n", + " discounted_rewards=[]\n", + " cumulative_total_return=0\n", + " # iterate the rewards backwards and and calc the total return \n", + " for reward in rewards[::-1]: \n", + " cumulative_total_return=(cumulative_total_return*self.gamma)+reward\n", + " discounted_rewards.insert(0, cumulative_total_return)\n", + "\n", + " # normalize discounted rewards\n", + " mean_rewards=np.mean(discounted_rewards)\n", + " std_rewards=np.std(discounted_rewards)\n", + " norm_discounted_rewards=(discounted_rewards-\n", + " mean_rewards)/(std_rewards+1e-7) # avoiding zero div\n", + " \n", + " return norm_discounted_rewards\n", + "\n", + " def update_policy(self):\n", + " '''Updates the policy network using the NN model.\n", + " This function is used after the MC sampling is done - following\n", + " \\delta \\theta = \\alpha * gradient + log pi'''\n", + " \n", + " # get X\n", + " states=np.vstack(self.states)\n", + "\n", + " # get Y\n", + " gradients=np.vstack(self.gradients)\n", + " rewards=np.vstack(self.rewards)\n", + " discounted_rewards=self.get_discounted_rewards(rewards)\n", + " gradients*=discounted_rewards\n", + " gradients=self.alpha*np.vstack([gradients])+self.probs\n", + "\n", + " history=self.model.train_on_batch(states, gradients)\n", + " \n", + " self.states, self.probs, self.gradients, self.rewards=[], [], [], []\n", + "\n", + " return history\n", + "\n", + " def train(self, episodes, rollout_n=1, render_n=50):\n", + " '''train the model\n", + " episodes - number of training iterations \n", + " rollout_n- number of episodes between policy update\n", + " render_n - number of episodes between env rendering ''' \n", + " \n", + " env=self.env\n", + " total_rewards=np.zeros(episodes)\n", + "\n", + " for episode in range(episodes):\n", + " # each episode is a new game env\n", + " state=env.reset()\n", + " done=False \n", + " episode_reward=0 #record episode reward\n", + " \n", + " while not done:\n", + " # play an action and record the game state & reward per episode\n", + " action, prob=self.get_action(state)\n", + " next_state, reward, done, _=env.step(action)\n", + " self.remember(state, action, prob, reward)\n", + " state=next_state\n", + " episode_reward+=reward\n", + "\n", + " #if episode%render_n==0: ## render env to visualize.\n", + " #env.render()\n", + " if done:\n", + " # update policy \n", + " if episode%rollout_n==0:\n", + " history=self.update_policy()\n", + "\n", + " total_rewards[episode]=episode_reward\n", + " if episode%10==0:\n", + " print(f\"{episode} -> {episode_reward}\")\n", + " \n", + " self.total_rewards=total_rewards" + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "c:\\winapp\\Miniconda3\\envs\\py38\\lib\\site-packages\\keras\\optimizer_v2\\adam.py:105: UserWarning: The `lr` argument is deprecated, use `learning_rate` instead.\n", + " super(Adam, self).__init__(name, **kwargs)\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "0 -> 39.0\n", + "10 -> 50.0\n", + "20 -> 18.0\n", + "30 -> 15.0\n", + "40 -> 16.0\n", + "50 -> 17.0\n", + "60 -> 20.0\n", + "70 -> 11.0\n", + "80 -> 20.0\n", + "90 -> 40.0\n", + "100 -> 36.0\n", + "110 -> 18.0\n", + "120 -> 79.0\n", + "130 -> 43.0\n", + "140 -> 41.0\n", + "150 -> 37.0\n", + "160 -> 95.0\n", + "170 -> 72.0\n", + "180 -> 138.0\n", + "190 -> 102.0\n" + ] + } + ], + "source": [ + "r = REINFORCE(env)\n", + "\n", + "r.train(200)" + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "[]" + ] + }, + "execution_count": 10, + "metadata": {}, + "output_type": "execute_result" + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD4CAYAAAAXUaZHAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAABEU0lEQVR4nO29eZhlVXX3/1l3rLo1z13dVd3VM9000EAzi0FmiRHURDEGyfsaia+SmBijqDHRJMRMDkkcIkZ/EgcQFAQnEFtAEOimu2l6nqeq7prn8Y7798cZ6tyqW9U1T70+z1NP3bvvOffse+rW96zz3WuvLcYYFEVRlIWFb7Y7oCiKokw9Ku6KoigLEBV3RVGUBYiKu6IoygJExV1RFGUBEpjtDgCUlpaampqa2e6GoijKvGL79u0txpiyTK/NCXGvqalh27Zts90NRVGUeYWInBzptTHbMiLiF5HXROSn9vNiEXlGRA7bv4s8235CRI6IyEERuWVy3VcURVHGy3g89w8D+z3P7wM2G2NWA5vt54jIeuBO4HzgVuCrIuKfmu4qiqIoY2FM4i4iVcDvAv/jab4deNB+/CBwh6f9YWNM1BhzHDgCXD4lvVUURVHGxFgj9y8BHwNSnrYKY0w9gP273G5fAtR6tquz29IQkXtEZJuIbGtubh5vvxVFUZRROKu4i8hbgCZjzPYxvqdkaBtWwMYY84AxZpMxZlNZWcbBXkVRFGWCjCVb5hrgrSJyG5AF5IvId4FGEak0xtSLSCXQZG9fB1R79q8CzkxlpxVFUZTROWvkboz5hDGmyhhTgzVQ+mtjzB8BTwJ325vdDTxhP34SuFNEwiKyHFgNbJ3yniuKoigjMpkZqv8M3CQih4Gb7OcYY/YCjwD7gKeADxljkpPtqKIoykKhayDOj7bXMZ0l18c1ickY8xzwnP24FbhhhO3uB+6fZN8URVEWJL/c28hHH32dmtIcLl1WdPYdJoDWllEURZlh+mIJAH57pGXajqHiriiKMsMMxC2n+sXDKu6KoigLhoG4NWVox6l2eqOJaTmGiruiKMoM40TuiZRhy/HWaTmGiruiKMoMMxBPkRX0EQ74eGGarJk5UfJXURTlXGIgkSQ3HOTm8ytYlJ81LcdQcVcURZlhBuJJsoI+/ultF0zbMdSWURRFmWGi8RRZwemthK7iriiKMsM4kft0ouKuKIoywwwkkmQFNHJXFEWZ9zR1D/D9LacAJ1tGxV1RFGXe87Nd9Xzy8d209cbUllEURVkoxJPWrNSegQQD8SRhjdwVRVHmP/GkVd63N5awbBn13BVFUeY/yZQl7n2xBNGE2jKKoigLgoQt7j3RpA6oKoqiLBSSKctz74sm5saAqohkichWEXldRPaKyGft9s+IyGkR2Wn/3ObZ5xMickREDorILdP5ARRFUeYDTuTe2R8nkTLT7rmPpbZMFLjeGNMjIkHgRRH5hf3aF40x/+7dWETWYy2kfT6wGPiViKzRdVQVRTmXSdgDqq29MYDZt2WMRY/9NGj/jLaq6+3Aw8aYqDHmOHAEuHzSPVUURZnHOAOqba64z4EBVRHxi8hOoAl4xhizxX7pXhHZJSLfEhFnldclQK1n9zq7beh73iMi20RkW3Nz88Q/gaIoyjwgYXvurT1RgLmR526MSRpjNgJVwOUisgH4GrAS2AjUA5+3N5dMb5HhPR8wxmwyxmwqKyubQNcVRVHmD07k7tgy4cAciNwdjDEdwHPArcaYRlv0U8A3GLRe6oBqz25VwJnJd1VRFGX+4kxiapsrnruIlIlIof04G7gROCAilZ7N3gbssR8/CdwpImERWQ6sBrZOaa8VRVHmGW7k3jMz4j6WbJlK4EER8WNdDB4xxvxURL4jIhuxLJcTwJ8CGGP2isgjwD4gAXxIM2UURTnXSQwdUJ1mW+as4m6M2QVcnKH9rlH2uR+4f3JdUxRFWTg4k5hidgGxWbdlFEVRlMnjeO4OKu6KoigLAMdzd5gTee6KoijK5EgME3eN3BVFUeY9jufuoPXcFUVRFgBDPfew2jKKoijzH6/nLjLHZqgqiqIoE8PruYcDPkQyVWqZOlTcFUVRZgCv5z7dg6mg4q4oijIjJJIGnx2sT/dgKqi4K4qizAiJlCE/OwhMf447qLgriqLMCMmUocAVd43cFUVRFgSJVMoV9+leqANU3BVFUWaERNKQn2VH7tOcBgkq7oqiKDNCQm0ZRVGUhUcyZcjPtqqs64CqoijKAiGRTJEV9BPy+zRyVxRFWSgkUoaAT8jNCpA9F8RdRLJEZKuIvC4ie0Xks3Z7sYg8IyKH7d9Fnn0+ISJHROSgiNwynR9AURRlPpBIGfw+H//0tg388TU10368sUTuUeB6Y8xFwEbgVhG5ErgP2GyMWQ1stp8jIuuBO4HzgVuBr9rrryqKopyzJFOGoF+4dUMl5y3Kn/bjnVXcjUWP/TRo/xjgduBBu/1B4A778e3Aw8aYqDHmOHAEuHwqO60oijKfMMaQTBn8vuktFuZlTJ67iPhFZCfQBDxjjNkCVBhj6gHs3+X25kuAWs/udXbb0Pe8R0S2ici25ubmSXwERVGUuY1T7jcw18TdGJM0xmwEqoDLRWTDKJtn6r0Z1mDMA8aYTcaYTWVlZWPqrKIoynzEKffr981cDsu4jmSM6QCew/LSG0WkEsD+3WRvVgdUe3arAs5MtqOKoijzlcRcjNxFpExECu3H2cCNwAHgSeBue7O7gSfsx08Cd4pIWESWA6uBrVPcb0VRlHlD0l5iL+CfOXEPjGGbSuBBO+PFBzxijPmpiLwMPCIi7wNOAX8AYIzZKyKPAPuABPAhY0xyerqvKIoy90nYC3XMZOR+VnE3xuwCLs7Q3grcMMI+9wP3T7p3iqIoC4A577kriqIo42dOeu6KoijK5JgNz13FXVEUZZpxPPc5N4lJURRFmTiDtox67oqiKAuGRNIZUNXIXVEUZcHglB8IqueuKIqycFDPXVEUZQGinruiKMoCRD13RVGUBYhb8lc9d0VRlIXDbNSWUXFXFEWZZhxbRj13RVGUBcRg4TCN3BVFURYM6rkriqIsQNRzVxRFWYCo564oirIAcWwZ/1yyZUSkWkSeFZH9IrJXRD5st39GRE6LyE775zbPPp8QkSMiclBEbpnOD6AoijLXmY3FOsayhmoC+CtjzA4RyQO2i8gz9mtfNMb8u3djEVkP3AmcDywGfiUia3QdVUVRzlWSc9FzN8bUG2N22I+7gf3AklF2uR142BgTNcYcB44Al09FZxVFUeYj8bnuuYtIDdZi2VvspntFZJeIfEtEiuy2JUCtZ7c6MlwMROQeEdkmItuam5vH33NFUZR5wpz03B1EJBf4EfAXxpgu4GvASmAjUA983tk0w+5mWIMxDxhjNhljNpWVlY2334qiKPOGObtAtogEsYT9e8aYxwCMMY3GmKQxJgV8g0HrpQ6o9uxeBZyZui4riqLML5JzsZ67iAjwTWC/MeYLnvZKz2ZvA/bYj58E7hSRsIgsB1YDW6euy4qiKPOLuZotcw1wF7BbRHbabZ8E3i0iG7EslxPAnwIYY/aKyCPAPqxMmw9ppoyiKOcyiaTB7xOsWHlmOKu4G2NeJLOP/vNR9rkfuH8S/VIURVkwJFJmRi0Z0BmqiqIo004ylZpRSwZU3BVFUaadRMqouCuKoiw0EklDwD+zcqviriiKMs2o564oirIAUc9dURRlAZJImRldhQlU3BVFUaadRNLMaNEwUHFXFEWZdpLquSuKoiw8Euq5K4qiLDyS6rkriqIsPOJJg189d0VRlIVFUmeoKoqiLDwSqZQOqCqKoiw0NHJXFEVZgMS1toyiKMrCQyN3RVGUBcicLBwmItUi8qyI7BeRvSLyYbu9WESeEZHD9u8izz6fEJEjInJQRG6Zzg+gKIoy15mrhcMSwF8ZY9YBVwIfEpH1wH3AZmPMamCz/Rz7tTuB84Fbga+KiH86Oq8oijIfmJP13I0x9caYHfbjbmA/sAS4HXjQ3uxB4A778e3Aw8aYqDHmOHAEuHyK+60oijJvmPMrMYlIDXAxsAWoMMbUg3UBAMrtzZYAtZ7d6uy2oe91j4hsE5Ftzc3NE+i6oijK/GBOFw4TkVzgR8BfGGO6Rts0Q5sZ1mDMA8aYTcaYTWVlZWPthqIoyrxjzhYOE5EglrB/zxjzmN3cKCKV9uuVQJPdXgdUe3avAs5MTXcVRVHmH3OycJiICPBNYL8x5guel54E7rYf3w084Wm/U0TCIrIcWA1snbouK4qizC/is7BYR2AM21wD3AXsFpGddtsngX8GHhGR9wGngD8AMMbsFZFHgH1YmTYfMsYkp7rjiqIo84XZ8NzPKu7GmBfJ7KMD3DDCPvcD90+iX4qiKPOeeDLFf/36CD3RBIXZwRk9ts5QVRRFmSYefOkE/7n5MG+/ZAl/fE3NjB57LLaMoiiKMgF21nZQVZTNF965ccaPrZG7oijKNHGkqYc1FXmzcmwVd0VRlGkgkUxxrLmX1eW5s3J8FXdFUZRp4FRbH7FkilUq7oqiKAuHw009AKxWW0ZRFGXhcMQWd43cFUWZFFuPt3HnAy8TT6ZmuysKcLixm8UFWeSGZycpUcVdURYIO2vbeeVYG90DidnuioJly6yaJUsGVNwVZcEQTxr7t0bus00qZTja3MOqstmxZEDFXVEWDI6oq7jPPl0DcQbiKZYUZc9aH1TcFWWB4Ih6Ijls+YRzmmTK8NePvs7++tGWoZha2vviABRFZraejBcVd0VZICTUlslIa2+UR7fX8b0tJ2fsmO19MQAKVdwVRZksMdeW0cjdi3M+XjzcMmPH7HDFPTRjxxyKiruiLBBcWyalkbuXWMI6Hyda+6ht65uRY3a4toyKu6IokySecGwZjdy9eG2q3x6ZmehdPXdFUaaMeEqzZTLhRO4AL0yjuBtjePClEzR1D9DRF8MnkJ81h8VdRL4lIk0issfT9hkROS0iO+2f2zyvfUJEjojIQRG5Zbo6rihKOk7Ertky6ThjEaW5YV4+2oox03N+mruj/N2Te/nR9tO098UoyA7im+Gl9byMJXL/NnBrhvYvGmM22j8/BxCR9cCdwPn2Pl8VEf9UdVZRlJGJ2xFqXD33NJzzcllNEW29Meo7B6blOF32zOD6zn7a++Kz6rfDGMTdGPMboG2M73c78LAxJmqMOQ4cAS6fRP8URRkjzkBqPKHi7sWJ3C+sKgSYtnz3nqgl7mc6LFumYBb9dpic536viOyybZsiu20JUOvZps5uG4aI3CMi20RkW3Nz8yS6oSgKQMyxZVJqy3iJu+JeAEytuHcPxLntP15gz+lOugesQdT6zn465kPkPgJfA1YCG4F64PN2eyaDKeM3zRjzgDFmkzFmU1lZ2QS7oSiKg2vL6IBqGjE7i6goEmJpcYT99d1pr790pIXf/9pL9MXGX3DtZGsf++q72FnbQY9rywzQ0Ref1QlMMEFxN8Y0GmOSxpgU8A0GrZc6oNqzaRVwZnJdVBRlLGj5gcw4tkwoIKyrzBsWuT+zv5FtJ9t5em+D23akqYfGrrN7885M1M7+ON22LdPWG6O5Ozo/I3cRqfQ8fRvgZNI8CdwpImERWQ6sBrZOrouKooyFeErLD2TCuaMJ+n2sq8zneGtvWpR+qNGK5B/bcRqwrJZ3fO0l/uGn+8763k4+e9dA3I3cwbqgzGaOO8BZq8iLyEPAdUCpiNQBfwdcJyIbsSyXE8CfAhhj9orII8A+IAF8yBiTnJaeK4qSxmC2zPyP3Ova+yjJCZMdmnyyXdyN3H2sr8zHGDjY0M3FS62hwoMNPQR8wotHWqjv7OexHafp7I9zrLn3rO/tlBno6k+QE0q3dWaz9ACMQdyNMe/O0PzNUba/H7h/Mp1SFGX8DNoy8z9yf+uXf8v/ubqGP7th9aTfy7FlnMgdYF99FxcvLaK1J0pLT5S7rlzGd145yad/vJftJ63kwNq2PowxiIycq97ea0fu/XFyw+kXonlpyyiKMvdYKJ77QDxJW2+Mpu7olLxfzGPLVBVlU5wTYsfJDgAO2pbMzedX8Ke/s4IXjzTT3hfn9y5aTHc04douDj/cXsdXnj3iToTq6Lcj94E4PdEE+VmD8fKct2UURZkfODNUY/M8cneKbvXFpsbRdc5LOOBDRLi8ppgtx1sBy54BWLsoj2tXl/Hn16+mvrOfEy19/OT1M5xq66M4ZzAC/9+XT7CrrpOOvhifvG2d29fO/jgF2UFK88IE/T5ae+d3nruiKHOIhRK5d/Zbgtkfn5q1YL2RO8AVK4qpa+/ndEc/hxq7KYoEKcsNA5ATDrCqPI+lJREATg2pInm6vZ+8cIBvvHCcl4+1utkyXf1W5J4XDlBZmAWoLaMoyhSxUEr+OuI+dZF7Cp+A367zcsXyEgC2HGvlYEM3ayryhvnq1UWWuHtLBPfHkrT2xnj7Jda8zKNNPa5t09lvZcvkZgWoLLCW1lNxVxRlSkgsEFtmOsQ9FBiUuvMW5VGQHeQHr9ay90yXO8jqJTvkpzwvzMnWwYyZ0x39gFXGIOT3cdouMwBWXZnugQS54QBLiyPkhQNTkukzGVTcFWWBEFtotswUiXs0kXItGQCfT7isppgtx9soyQnxJ9cuz7jf0uJImi1zxhb3qqJsKguzON3RT3uvJe7JlKGha4C8rCD/77qVfOdPrpiSvk8GFXdFWSAslFTIwch9ajz3eDJFyJ8udb9/6RI2LMnn+++/kirbghnK0uIItW397nMncl9SlM3igmxOtfXRNZBgUX6W2+/ccIDS3DAbqwunpO+TQcVdURYAyZTBmbs03ycxddpWx1RF7kNtGYBbN1Ty0z+7lprSnBH3W1oS4UxnP9GE1Y/T7f34fcKi/CwWF2ZzyM60cQZfAfKy5k4Cooq7oiwAvCUH5nvJXydy750icY8NsWXGytLiCMZYog5W5L4oP4uA38eSwiz641b/lhUPintuWMVdUZQpxCvu873k71R77vGkGRa5jwXHbnEmU53u6GdJoZUJs9j+DbDME7nnauSuKMpU4l0Ue74XDnPEPZZMTcn4QSw5sci9xM59b+2xbKLT7f0stnPYlxR5xX3Q2smbxTVTh6LirigLAK8ILpRsGYC++OSj91giRcg//rVMS3KtPPXW3iiJZIqGrgFX1EeK3PPUllEUZSrx5rYvlMgdpsaayTSgOhaKIiF8Ai09MRq7oyRThiWFlpAvLhgU9+oitWUURZkm0myZBeC5ZwetCUBTMZEpPkFbxu8TinNCtPRE3UFVJ3LPDvkpzgnh9wmFkaA7kKoDqoqiTClpA6rzOHI3xtDZH3frs0wk1/1X+xr53paT7vOJZssAlOSEae2JuhOYltj9AlhcmEVhdhARoSDb8to1FVJRlCklvkBsmf54knjSUFlgiehEbJnvbTnJ158/5j6PTTBbBizfvbUn5k5g8nrtK8ty3eeOqOeF586A6ty5zCiKMmHSs2Xmry3j+O2L8i3RnIgt09Ybc2u+QOYZqmOlJDfMntOd1LX3U5wTIhIalMzP/N75DNgTnJzIPSc8u/VkvJz1E4vIt0SkSUT2eNqKReQZETls/y7yvPYJETkiIgdF5Jbp6riiKIN4l5Kbz1UhHXF3IvcJiXtfjK6BBEl77MGyZcafLQNQkhOipTualuPuUJQTcitA5mcHyQ76CUzwIjIdjKUn3wZuHdJ2H7DZGLMa2Gw/R0TWA3cC59v7fFVE5s6lTFEWKI64R0L+eZ0K2WmX0F1UMHHPva1nsMY6TDxbBqAsL0x3NMHxlp5h4u6lNDectqjHXOCsn9gY8xugbUjz7cCD9uMHgTs87Q8bY6LGmOPAEeDyqemqoigj4VgxkaB/Xpf87Zhk5D4QT7plC5z3mtyAqiXYtW39aX77UD58w2q+ftelEzrGdDHRe4gKY0w9gP273G5fAtR6tquz24YhIveIyDYR2dbc3DzBbswN6tr7+M2h+f0ZlPmNU08me75H7v3pkft4B1TbPV6747tPdIYqDM5ShfRZqUNZVJDFhiUFEzrGdDHVBlEmYyvjN80Y84AxZpMxZlNZWdkUd2Nm+daLJ3j//24jNc/zi5X5y6AtE5hwKuRTexoYmIIZoZPBsVKcSULjjdzbej3i7rFlwpPIlnEYzZaZi0xU3BtFpBLA/t1kt9cB1Z7tqoAzE+/e/KA3miCasKYnK8ps4Excyg76JzSJ6VRrHx/47nZ+tqt+qrs2Lhq7BggFfBRkBwkFfPSNcx3V9t7B2a2Ofz8ZW6Y0ZzByrxolcp+LTFTcnwTuth/fDTzhab9TRMIishxYDWydXBfnPk7pz6GL6SrKTOHYMpGwf0J57l0DlhDOdoCyv76bNRW5+HxCJOQ/qy1T39nP2776Wxrtfrf2Rt3XOvpibp37CYt73gKO3EXkIeBlYK2I1InI+4B/Bm4SkcPATfZzjDF7gUeAfcBTwIeMMbN7nzcDqLgrs81Ys2WMMWw/2Y4x6ds49kdzdzStPZUyfPB723n5aOsU9zhz3/bXd7HeXtM0EvSf1ZbZeryN1051sKuuE8Bd9g4sW8abIjoRIqEA2UE/2UE/hZG5M0FpLJx1EpMx5t0jvHTDCNvfD9w/mU7NNxyf8lSrirsyOzgilh0MjBq5v3KsjXd/4xV++IGr2FRT7LY7AcpQce8aiPPz3Q1UF0W4amXJNPR8kObuKK29MXfB6uwxRO51ds2Xlh6r3229MUSsC0NHX5yofUcz0Tx3sHz3rKAfkYm/x2wwdzLu5zFOdKGRuzJbOKmQ2SHfqOJ+orUXgOMtvWnt/XY+eVN3ui3TYfvWTUNEfzrYV98F4Ip7JBQYlufe1DXgWkgAtfb/XIvdv7a+GIXZQYpzQ3ROQeQOsKIsl/MW5U14/9lCyw9MAf0q7sos44hYTihAylh2is83PNJ0CmCd6UgX8ZFsGSe1cKjoTwf76601SdctGozch9oydz7wChurC/nCuzYCUNtui7sncnfKBHT0xQbFfRIzR7/2nkuYZ0E7oOI+JQyo567MMq4tE7ImhMdTKcK+4ZPDT7vi3p/WPpIt46QTNnbNTOS+pDCbAtvbjoT8aamNde19HGvpTRNa15axt3PEPSvop6M/Tsy1ZSYu7jlzqIzveFBbZgpw/jHaemN0e24ZlYVHb3T80+EnSjyZGnPeuWPLZNl10EcaVHXqkp/pHCLudoTcG0umfUZnIlDTDGTR7K/vci0ZsMTd25ctx6yJ8ida+xiIJ0mmjHuRcmyZ9t44xTkhCrKDdPZNjS0zXzn3PvE00B9PUmrPZKtt6z/L1sp8pbatj4s++0ter+0467ZHm3v4/357fFLH+8yTe7n7W2PLJLYWpBDXfhjJd3dE3RF5B+/ApTd6dzz3roEEA/Ek33n5BAcausb+IcZIPJniWHNPmrcdCQXS+rX1uCXuyZThaHMPDV0D7kXNsWVa7ci9MBK0I3fr9clE7vOVc+8TTwP9sSRrF+UCcKqt9yxbK/OV0x39JFKGk2Ow3x7dVsdnf7LPtQUmwuHGHnaf7hyWtpgJZ7UhJyskU9nfZMrQ0GlF4Kc7+tPe17tWaXPPcHEHONnax6ef2MsPXvVWGJka2ntjpMxg2QGwIndvv7Ycb2VFqbUY9aHGbncwdUVpDi09MYwxtPfFKIqEKMwO0dEXI2qX5A0F5qFpPklU3CdJKmWIJlKsLrciDo3cFy5OFNkzcHZrps2eTDMZG6elJ0pfLEl733CrL55McdXnNvP4a3X2c0PAJ27J2Uxlf1t6osSThpVlOUQTqTQ/2xshN3WlTwRyeOloCwDdY/j846XV7ou3sqJ3QLWxa4ATrX2887JqQn4fBxoGxX3j0kI6++O09lqTlpzIPWUGB4RD/nOvOK2K+yRxivVX5GchgnruC5heOy1vLILtCGfvBErWOjgRdG2GO4XT7f3Udw64GSZOWduAnSGTyXN3BlMvs/PbvRkz/bGk60s3ezJjOjyLVf/2iDWRaTq+4+0ZxD0SDBBLpEimjGvJXL2yhBVlORxq6KauvR8RuKiqELDudJz3yLcXz3Aspsnkuc9XVNwniRPxREJ+ckIBeqILfkLuOUuf/bftHoO4O5Fo7wS/DwPxpBshO+l+XhxryLmIOLaMI9CZyv6eGSLupz0ZM33xJJUFWQR8kmbLtPfF3ZoqW45Z4t7VP7EL1hefOcSfP/RaxtcyRe7OqkY90QSHG7vxCaxdlMfaRXkcbOimtr2Pyvws18rZcaodsKydQlvcnbuQoA6oKuPFyZTJDvotj3ASkZoyt5nqyP1wYzc3fuF5WnuGpxm2eiyTTFbfqWHibgj6fQR8ti2TKXK3B1EvX+5E7oPv2x9LkBMKUJobTrNlOvtiLC/NIeAT96LWHZ1Y5P7ysVZ+e6Ql42uOfVIUGRT3qqIIACdaejna3MvS4gjhgJ+1i/I40znAsweaqCqOuMkMT+9twGdH8oX2+zgXqsnkuc9Xzr1PPMU4qWpZIT854YC7UICy8Ogbj+fe40TuI2+7+3QnR5p6ONjQPey1Fk/GSqbI/ZQ909QbuQf8QsAdUM0cuedlBagqyiY76OfpvQ3c/MXnOdLUTX88SXbIT1leeFjkXpwToixvsDriRD33xq4BWnsHBzm9OJ+jyFO/Za2dOXOwsZujzT2sKLOSFq5cUULI72P94nz+4sbVlNnivquuk3WV+eSEA24dGOdCdS6mQs7P7Pw5RH/MLtjkRO4zmAetTD2plOG7W07y0131PHDXpW4ECINC3XOWu7NoIulGuaPZMk7t8sYMsz+d6oZBv7gTdbw4kbsT8TqLQAdHEffTHQMsKcxGRFhcmMUW28feV99NXyxJbjhAYXYwrTJkh519Up4Xpt7OtOnqH3/kbsxgpk5TV5Tq4kja6229MQqyg2lrkC4tjpAV9LG/vovjLb1cu7oUgEuWFnHwH291a71475YvXWYt5+xYNU65BU2FVMaNa8vYnvtkBtCU2ecTj+3mb5/Y61Yb9HK2yP3pvQ18/Ie70mqKjxa5OxFwptmfLd2WaK+vzKcuw4DqKduqce4QHFsm6GbLpNsyxhgONHSx1BbV8yrzWWwLYGdfjP5Ykqygn/L8sFs+N5kydA0kKMgOUpZnbbusJEL3QGJM6ZleOvsHi3g1ZpgQ1dYbc5e0c/D7hNXleTx7oIloIsVKO3IH0op4RUIBIvbMXEfc87OClOSEONZsifu5GLmfe594inHEPSvoJyd89hKlytzmuUNNXLXCqn54tLkn7bW+s3juT+9p4AfbajndMSjGo13snQJYmcTOsUY2VhdS196ftsqXMca1ZbqjCWKJ1KAt48s8iWnvmS7q2vu5cV0FAF9610ae+cjvAFYue388SSTkp6ooQkuPJfbOkneFkSDl+Zb1ccnSIhIp437vx4r3biBTzfi23hhFGRaYXlORxwm72urK8txhrzs4vvslS4vctmUlEXdgWbNllHHjZMtkB/1EwgF61JaZ13T2x9mwJJ+iSJCjzekT0pzxlJH+xk72iTfiH82WcSL3pkyRe0+U3HCAVRV5xJKptKqMbb0xemNJVpZZE3qcAlneSUxDB1R/vrsev0+4ab0l7kG/j5xwgJyQn87+OH0xR9ytzJi69j43x70oEmJVWS75WQEurCpI6/tYcSyZoY9PtPQST1o5997BVAfvjFVv5D6U0lzLOvKullRTkuM+1gFVZdwMpNkyfjddTpl/RBNJBuIpCrKDrCzLHR65O577COLuTO1PF/fBbVt6omm1YgZtmeGRbEtPjNLcENW2WHkHVR2/fWO1FaW29saIJ43tuQ+P3I0x/Hx3PVevLBkWHRdkW9P0B2xbxvHCa9v73MlTBZEgd121jOf++k3ugtHjzXX3fkbn8fOHmnnT55/jB6/WZrRlANbY4l4YCaalSQ7l7qtr+Kub16TZNcu84q62zPgQkRMisltEdorINrutWESeEZHD9u+is73PfCY9FXL8nnvXQJxP/3jPjBakUjLj2BAFkRArynI4NkTcR4vckylDvT0pyMm3Fhm0ZYwxvPW/XuRjP9zl7uPaMhkGVFu6o5TkDkai3lowpzwzM8GaAOTUlglkKD9woKGbE619vHlD5bDjFERCdPTF6bNtmWo7/bC2rZ/O/sHIPej3UZwTIi8rYJ+r8Ubu1p3H4oIsGrqiNHUN8JEf7MQYeL22wyobkEG811ZY4j5a1A5w+8YlvOuypWltNaWDg7Y6oDox3mSM2WiM2WQ/vw/YbIxZDWy2ny9YvLaM47mPZ7Bpy7E2vvPKSbadbJ+uLipjxFlQ2YncW3pibht4KidGhw8oNndH3UHM+s4B/D6hLDfsXrTrOwc40znAT3ad4XCjlfrY5RlQHfp+LT1RSnNDlNgLNHtLBTgrfl1kWyRO5B5IG1AdjNxfspfIu/688mGfuSA7QEtPlGTKEAkFKM0NkRX0UdvW5w4MOxOCwBqoBCty748lSYxxvdaGrgFKckJUFUdo7BzgC88coieaYGVZDq+eaCOeNBkj94r8MKW5YdZVjn+xDG/kHshQ236hMx2Xs9uBB+3HDwJ3TMMxpoRkyox71H8o7oBqyEckFCBp15oZK04qW1vv9NfLVkbHjdxtcQc42jIYvTtReDw5/G98ekh99KJIkNysgOu57z5trfFpDHz52SMAdNvHiyVS7rEdLHEPk58dxCfpNV72N3RRVZTNYnvB5nbbcw/5M5cf2H6yjaqi7LSiXA6F2SHXA3eWkqsqilDb3ueWHvB64fl25N41kOD3vvwin35iz/ATmYHGrgEq8rNYlJ9FQ9cAzx9q5oZ15dywrsIdMM0UuYsIP/zAVfz1zeeN6TheakqsyD0U8M27JfKmgsmKuwF+KSLbReQeu63CGFMPYP8eHi7MEd7yXy/yFfsfbSSeP9TM84eaR3y9P5bEJ9aATa5d1N/JmGnoHOA7L58Y9f2df9rWntio2ynTT5q425kZR5sGxd07njLUmnFmezqLOxfnhMgND9p0e0934hN471XL+MnrZ2jpidI1kHDF0psOGU+maO+LU5obxu8TCrKDtNnfE2eB60uXFbkRdZvHlnEidydLxLt9Jgqyg+4qS046YXVRtmXL9FnrkTpWDODWbGnpjnKkqYdHttVxsnV4JdS+WIL/3HzYPacNnQMsKsiiIj/MqbY+6jsHeMOqsrSIvDgn8wLUNaU57gIe46EwYtV1PxcHU2Hy4n6NMeYS4M3Ah0TkjWPdUUTuEZFtIrKtuXlk8Zwu4skUBxq6ONjYM+w1Y6yIfiCe5C9/sJN/f/rgiO/TH0+SbUc8zj+Hcyv+ox11fPqJvcNWt/HSZt/6em+7ldnBK+7VRdkE/ZKWMdMbGxTjoWMkjrhfbS8ibS31NrjYxJ4zXawqz+WNq8tIGWsFoa6BOKvsi4h3wNEpolVqzwotioTcwc3THf00dkW5dFkRAb/PEv7eGPFEKt2WsSN37/aZcKongkfciyPUtffxWm0HS4sjacv1OUK/317vNJkyfPXZo8Pe96k9DXzhmUPc96NdGGPcyL0if/Du4drVpWmLcxTnhIe9z2SpKYmck2mQMElxN8acsX83AY8DlwONIlIJYP9uGmHfB4wxm4wxm8rKyibTjQnR0hPFmPTbXYd7H3qNP/3Odn66q5623tio4uxM24bB5bicaM35h23PcAyHDteWUXGfbbziHvD7WFaS4/rjxhj6YknKbXEamgroTO3fsMTywUtywlbkbkf7e053smFxgStude19xBKpjOJ+xB7IdabVF+WE3O/JdntsxsnnLskJWeKesmvLOKmQtuc+dPuh5Hv8dGcVp+qiCF0DCV480sLtG5ekbZ8d9OP3ibuY9YVVBfxoR92wjB9n1aRf7Gngv58/RmtvjEWeIl9LiyNUF0dYWZbrRtbFGVIhJ8vy0hwioXNzIv6ExV1EckQkz3kM3AzsAZ4E7rY3uxt4YrKdnA6c2+BMwrvjZDu/3NfI39p+YmtvNG0SiRcnhQzwRO6DNahhdOF2jt86AXFPpgy3f+W3PLpt6hdPOBdxxN2Jzi9fXswrx1qJJpLEklbp2XI7mh4auTtT+5fZPm9xTsiuNZSgqWuApu4oG5YUuJOBjjZZdwSOuDt57KmU4V+fOkhpbphrVll3AUWRoHuHt+NkO5GQ383/LnLEPZki5BeC9iSm9t44D209xS/3NZLj2X4ohR67w/n+Ohk6xsDbL04XdxEhPyvAIfui98nb1pFIGR5/7XTadluOt3LjunJuXFfOvzx1AIBFBWEW2Re3N9ilBIJ+H6srrHNQnDv14v5XN6/li/Zi2ucak4ncK4AXReR1YCvwM2PMU8A/AzeJyGHgJvv5nMONqnvTB7IS9oSRcMBHXyzJirIc4kkzbMDLwZnZB4ORuzOT0fmHHVXcJ2HL7Kzt4PXaDl490TbufZXhdPbHyQsH3Pom168tpzeWZOvxNtdvdyLvTJ77ksJsd+JMSW7ISo2NJtlzxhpM3bCkgJKcECKe6DwvTEF2kMauAaKJJA+8cIydtR3c9+bzyLMzUwojnsj9VDsbqwvdPhY74p6wJzHZKw59d8tJPvHYbn62q56LPNsPpcATuWcHB20ZgE3LiqgpzRm2T15W0MqrD/i4vKaYS5YW8qPtdW5ygrOwxpUrSvjvP7qUj926ltLcEBdVF7KqPJfyvDBvuXAwLXN9ZT6hgI+c0NQvqFFdHHGrYJ5rTPh+xRhzDLgoQ3srcMNkOjVWmrujPL23gZvWV6R5eWPBWfB3qC3T0mOt5vKxW9aSlxUkFPDx0Udfp7knmnE03/HcYXjk7sw8HEvkPhFxf/aA5Xg1zMDK9POZx3bU8fTeBr5+16ZRt+vsi6fZFNesKiUc8PHrA00st0XOidyHivvpjn4uXVZEUU6IL77rIq5cUcK3XjxObzTBwQZLyNdV5hHw+yjJCXPEHqjNzwpSkR/m4a21PLKtloF4iiuWF6dFzI6A98eS7K/v5oPXrRx8LRJi8/5GUgYqC7Pd8gPN3VGWFGbz/muXc+mykcWtMDt95SOwrIzinBDvvbom4z752ZZsVBVm4/MJ77i0ik89voc9p7u4oKqAV+y671csLyHg9/HB61bxwetWuftv/dSNae/3wTet4vrzys/JjJbpZF6bUY1dA/zNj/dQnBPitguGT9AYfV97GbRYklgi5c5gc+perCrP5YZ1Fbxs5wi3dEdZUzH81rbfY8vkeiJ3Y4zr1bePKu4Tj9w32+Le2Dl8EowyyCPbannlWBv9scHxkUx09sfTI9mQn6tWlvDrA03caU+QKfOI+6sn2thYXUjUTmWsLLQCjLddXAXYCzzHk5xs7bUnAFnvXZ4Xdme/5mUF+ejNa3nhcAuhgI9rV5dyzarStEHMwkiQaMJKAEimjFsKFywrI2XgLRdWcvdVy/B79rt8eTF/fM3yUc9Nui1jfX9zwgG2/82NI4ptXtjap8qO8N9ywWI+++Q+HnutjguqCthyvI28cID1i/Mz7j+U5aU57sVTmTrmdY7QqvJcfELGethnwzsA1NE/KKwN9hRyZ+DH+WduzrCgAljlBxzBcP45eu2iS046WmtvjAMNXfzfb7+atlalMYaOvhg+sYRlpBXrM1Hf2c/++i5CAV/GQkyKRTSRdMsBDM1FH8pQcQdr4s/J1j722taKM6C642QHf/DfL/PEzjNuuuSK0vRZlM7F/kBDt1tGwHqPsJsnn5cV4ObzF/EPd2zg029Zz3Vry4fNpnTyzJ1ceWcRC7A88Y/evIYvvWsjAb+Vz+3kul8yQoaMl0y2DDBqFO1kzDifqSAS5JpVJTx/0Mp6e+lIC5ctL0670Cgzz7wW96ygn5qSnAmJu7cQU0dfnC88c4gfv3barVldWWB9cZ2MhZEyZry2jLMsWG80kfb+7X0xnjvYzK8PNLGztsNt744mSKSM63GOllUzlGcPWP9It21YRGd/PK1myULFGHPWv3VtWx/v+/arbpbL67WdrpBORNydFEKnpIDzfXjhsHX+d9d1cNA+1tohg5bOGMyhxm43yoVBawfSs1VGwhH3XXWWuHsvFKsr8rj3+tVpnrqTMXPpCBkyXrz546Pd1Xhx+uytyf6G1WUca+nlxcMtnGjt47q1M58Bp6Qzr8UdrJKgzsj9eGjsGnAHcNp7Yzz40gm+88pJGjoHCAV87oow+dkBQn7fiJG7V9yzAn5ErAJTjt/uE8tyqbMLPzkRIECHPZi6yp4NOR5rZvfpDooiQa5ZZWUdNJwD1sxTexq45Uu/Gfb3ru/s5/3/u40vPnOIOx94hc0Hmnh6bwMAW4+3utvVZVjRyEsmcXemsO87Y6X+5WVZlRSdi/f++m4ONnSTFfS5tdIdnIt9Xyzp1mwB0saHvBOERsL5Lu6u6yQS8o9aQAsg6LMGJ4debDKRGwrgBNiRMYr7YOTuEXf7e/iPP9sHwJvWztm5i+cM81/cF+VxorV33JFrU3fUrThX195PZ3+cfWe6qOvop7Igy70tFRFKc0Pu4glD6Y+lyLL/KXw+IRL00xtLurZPTWkObb0xt/DTntOD4u7MOnRmQ7aNY5bq4cYeVlfkufbRuWDNOAN13gtkIpniww/t5LmDTfzH5sN0D8QpyQmx57QlxluOt7G6PJeAT9KKb2Wisz8+bCZkbthaV/SAfceQHfKT6xHk/Q1dHGzoZnV53jAbIseTX+0tRetE7iKWuJ4NR8wPN3VTXRQ568BjwC9cvLRoTLaIz54BG/DJmItrOWMH1cWDn2lNhZUFc6ChmzUVucNWWlJmnnkv7uctyiNlcLMPxkI0kaStN+bm/jqpav3xJK8cbXVzcR2GrivpZcATuYN1K94XG7RlzluUR3tvzF0qbY8dAcKgDePU5R5rrrsxhsNNPawuz3X7mqls7EJju22NHPbMKv7ys0fYeqKNf/39C9nyyRv45V/+DleuLGHPmU7iyRTbT7Zz9coSKguzRrVlBuJJoonUsMgdrFmOTkmJnFDAtVsq8sN0DyTYeqIt42B7JDz4vfCKnbOqUW44kDZwOhLOUn8pky6oI3HXVTX8n2tqzrqdQ0F2cMyWDEBlQRYhv49lxYODoCLiRu/Xn1cx5vdSpo95L+7OP9UBjxebSKb496cPUptheTIY9M+dcqLeaLq1N0ZlQQZxz+C5G2PSbBmwxL03mqSp27J9qooitNri7vcJR5t73AkwThaNM5HFa8v0RhP808/3Z7RbmnuidPbHWV2eS0XByOLe1hvjc7/YPy/8+FTKjFr2uDeaYH+99Tc+bF/Ij7f08pVnj3D7xsW87eIqqzBVQRYXLCmgrr2fp/Y00BdLcs2qUpYUZqetRdoXS6RNTPPOTh2Kt7pgJOwnzxb3P7i0GrAKf2WaJOQMqALDBlRhsMLi2fBmtHgHU0fiIzet4YZ1YxfYgkgo7Tt8Nt5xSRVP/cW1w+5yfsf22W9ar5bMXGDei3tNSYRQwJfmw75wuIUvP3uER7fXZdzHSYNcVppDKOBz/VQniFpUkB4dleaGackQuceThmTKpEU9kZDfjdzL87MoioSIJlL0x5NcsbwYYwbrcjhpkDUlOYikR+6ffmIPD/zmGD/ddWbYcY/Ykevqijzywtb6kQ2dUeLJVFqVy5/trufrzx/jxcMtI5y9ucPXf3OMK/9p87Aa6g6v13aQTBkKI0E3O+UffrqPcMDPp353Xdq2GxZbJQC++MwhckJ+3rimjKqiiGvLRBNJ3vivz/EvTx9w9xlN3J3qgmAthO5E7u/cVI3jkKzJIO7OdiKwJIMtMxa/HaxZnM62XntnqijIDo7ZbweryuKKDPXVf+/CxTz+watHzatXZo55L+4Bv49VZblpkfsPd1iivtcTkXtxJjAtys+iKBKkN5YkHPBxQVUhQMbIvdWuee2N9py0xixv5G7PSmzuilKWF06rUX3rhkXA4J2CkwZZFAlRmB10y/4+/lodj+04nbatFydyXV2ei4iwKD+L0x193PzF3/CFZw652zkXrS2eQcWhJJIp/vv5o7Ne2+bx1+rojia49/uvZbzTcGqk3LFxCSdae3nhsJV99OEbVlOel/73Ot/Orz7W0sv16yrICvpZUphNY/cAsUSKFw+30NIT5du/PUFT9wDJlBk9crdzsEMBHwG/j6KcENXF2SwtibDcjuozRe6O516Rl0U4MPgdcdJrxxq5w2DGzHR42TetK+cW+7s5GXw+y+tX5gbzXtwBLqouZNuJNvpiCTr74zyzrxGw8oL7Ygmu+KdfpdVfcW7PK+zIGmBJYTYX2kWfhta9Ls0NkzJw1ze3cMdXf+uK+jdeOAZYIusQCfuteiLdA5TnhdNmtV6ytIjS3DBb7XIBbb0xCiMhfD5xZyEOxJN87ucHuHhpIdefV+7mNjtYfns3+VkBVyQq8rP49YEmjrf08tzBwQqbzh3C1uMjlyd4/lAz//yLA3x/y8mMr9d39vPYDmtqufdxa0+Urz13lF11HZOuiX+kqZtDjT3cuK6CffVdvO/BV90qiw7bT7WzpiKXS5YVkTLwr08dJC8c4K6rlg17v6KcEEvsWue32aJVVZSNMdbn+fnuBnJCfuLJFB/74S6u/Nxm/uz7rwGjR+5OdtV9t57H/7z3MgDWL7bWW/WmNzo42TJDffJwwE9hJOjO9BwLTsZM9RhsmfFy11U1fOLN686+oTKvmNczVB3u2LiYh7ae4um9DfTHUsQSKe7YuJgf7zzDj187Q2NXlKf3NvIHmyyP9MUjLe4Ua+efeUlRNhfYK9s4wuDgiKizos3fPbmHS5YW8ZXnjvDOTVW8cc1gTm9OKMDhxh5ae6PcsK4irUZ1dVGEt1+yhP954RjHW3rp6Iu7/7SluWH2nunimy8ep6k7ypfetZGtJ9p49mATvdEEOeEAn/vFfn65t5GAT1hlR+1gXYycZdX21XfRF0sQDvg52NBNwCfsOdNFTzRBbjiAMYa9Z7rYeryNd11WzS/2WCmDmw80ce/1q4ed2/t/tp+f7qqnqz/OD7bVsb++i+WlOTy1p4Gv/8a6uN191TI+e/uGtP121nYQjSe5YkXJWf9+v9ht9eEf79jATevL+exP9vHm/3iBJ++9hmUlObT3xnj5aCt3XlbtXkh3n+7kzsuq0+6avFxYVUBrb9T1gR1b5HhLL8/sa+DWDZWkjFXwak1Frju/IVOaoTNw6ExS80bP9735PBq7ohkzWJztM/nkN5xXwdpFoy8d58UJEqrGMKCqKLBAxP2ymmKqi7P59ksnOdPRz/rKfN59+VJ+vPMMX33OWozj1RNtpFKGgUSSl4+1cteVVsTnRO5VRdm89aLFhPw+97bewYnK3rxhETWlOXztuaM8sq2O8xfn85m3np+2bU7Y72Zl3LphkVujOjccID87wPuvXcGDL53gS786RG17n3v8D1y3knv+dxv/9vRBNi0r4qqVJfTHk65H39IT4+vPH8MnVtbEu+wLFQzmTV+9soSXjrays7aDRflZ9MeT3L5xMU/sPMO2E21ct7acTz6+h4e2ngLgWEsPv9zbQMjvY2dtB609UXcBZLB86F/uayTk9/GZn1j5y0G/8Oj2On61r5FrV5dSlhvmu1tO8b43rKCxe4BIyM+K0lz+5MFXae+L8593XszveopEgTUA+ZPXz/DGNWWU5ob42e56Ll1WxKKCLN512VIuqynmjq/8lj976DV++IGreWRbLdFEij+8YhnLSiLuOXj7JVUjfic+dut53HXlskFBtgX2y78+QtdAgtsuWMTFS4u4dnUpb7lwMa29UV6v7choexREghRFgm4k7qWqKDLiIKffJ1xWU+TWePfy+XcOK8s0KsU5ISvaH4eVo5zbLAhx9/mEt11cxX9uPkw44OM777vc/Yera+8nJ+Snsz/OwcZu6tr7iSVS7nqSRXZkvaQwm6ygnzuGlDgF2FhdyD/csYE7Ni4mK+inIi/MmkV5XLG8ZFgusSMmV60o4bKaYncNzqqibESEsrww77liGd/67XEA/vAKq2bJm9aW8+U/vIRPPb6bj96yFhFxa4M/vbeBH7xaywVLCvj728/nnu9s58qVg4NWG6sLWVyQxefefgG/82/PseNkO8vtqfB/dOUyfrarni3H27hgSQGPbqvljo2LCfh9fPcVS+Q/fMNq/mPzYZ472Mw7Lq2ivrOfA/Xd1HdaHvW3/ngTf/+TfbzlwsXUtvfx8NZTpAx89q3nc8myIn66u54/e2gHe850kRPy885N1bT0xFhVnsu9D+3g4z8KcNXKEr72nks43tLLnz+8k/31Xawsy+HqlaUcaOjmH+8YjPxXlOXyr79/IR/47g7u/f4O9tV3ceWKYndSzrKSHBKpFJtGmV4/tF7JooIsKguy2HayncqCLN6wupRwwO9eICoLst1ZyZlYZg96j5dHP3D1+HfKwAevW8UdG4d/NxVlJBaEuAP8waVVfH/LKT52y1rOW2RF3itKczjW0sv737iCL/3qMFuOtXKwsYfccIDLaixxLHQj95G9zIDf50b6wKjFmJz0tz+/wbI48rMD+H2SZvV8+IbVFOcEuXJFSdoiCrecv4ib11e4t/jledbiwN944Ti54QBf/sOLWVaSw9ZP3pBmA9y6YRG3nG/tt7o8l+0n2+mPJ/H7hAuWFLCppojHd5wmNxwgkTJ84LqVVORl8av9jcQTKf7fdSv5/tZTbD7QyG0XVPLeb27lcFMP4YCPVeW5vGltOW9aa1Xte/5QM0/sPENBdpDr15UTDvi587Jq/vflk1xUVcDR5l7+58XjXLWihG/cvYlvvnCcuvY+Ht1ex8d+tItnDzTh9wkfv/U8/nPzYb7zyknee9Uy3nNF+sr1t26o5G9+dx3/8tQB4knDJ28b9IT//vbzCQf8Y8oRdwj6fbz48evt5eh846578je/u85dAHs2WFWe66bMKspYWDDiXl0cYcsnb0j7pz1/SQHHWnp5zxXLeHRbHU++foaTrX1cu7rUrQLpeN5LpijF7B2XVlGeH+bKFdbFQ0RYU5HHRdWF7jYFkWBGf9vZ3vv4giX5PHuwmX96+wVuvnUmf9dpu3RZET/fXU9bX5yVZTlkBf389S1recfXXubzvzzI+sp89+L3lT+8hI6+OFlBP797QSXffukE++tf4ERrL3dftYyHX63lriuXpR3vDatKWVYS4eb1FW4GyF/euIZFBVncdeUynjvYzH0/2sVf3rSG3HCAD99ofU6/T3j41Voq8sP84J6rqCnN4YoVxeyq7eC9V9Vk/Ex/cu0Krl5ZyguHm7l5/WDe9rWrJ1a3xO8T/L6J1QzfVKPpfcr8Qiab6TAVbNq0yWzbtm3K33d3XSc7a9u566oaPvLITh7bcZqyvDDfvHsTF9ppj5v3N3Lv91/jxY+/Kc1vnkqSKYPAuCJNh63H2zjQ0MV7r6oZ0/Y/ef0Mf/aQlfnx7sur+dzbLwTgIz/YyWOvnebTb1nP+94w/M4jlkjxn5sP89XnjnDvm1bxkZvXEktYiy4PFd5oIknAN3L06y2h7NAfS/K154/ytouXaHlXRZkiRGS7MSbjQgULWty97K/v4snXz3DPtSvS0hOdWaYLZZ1FYwx17f3EkymqiyNuvZCWnihf/vUR/vKmNRnT/Ry6BqzViHThBEWZ+8yKuIvIrcB/AH7gf4wxIy63NxPiriiKstAYTdynZRKTiPiBrwBvBtYD7xaR9dNxLEVRFGU40zVD9XLgiDHmmDEmBjwM3D5Nx1IURVGGMF3ivgSo9Tyvs9sURVGUGWC6xD3TaFyauS8i94jINhHZ1tzcnGFzRVEUZaJMl7jXAdWe51VAWu1aY8wDxphNxphNZWW63qKiKMpUMl3i/iqwWkSWi0gIuBN4cpqOpSiKogxhWpK7jTEJEbkXeBorFfJbxpi903EsRVEUZTjTNnPHGPNz4OfT9f6KoijKyMyJGaoi0gxkXi1ibJQCc3EtOe3X+NB+jZ+52jft1/iYaL+WGWMyDlrOCXGfLCKybaRZWrOJ9mt8aL/Gz1ztm/ZrfExHvxbEMnuKoihKOiruiqIoC5CFIu4PzHYHRkD7NT60X+NnrvZN+zU+prxfC8JzVxRFUdJZKJG7oiiK4kHFXVEUZQEyr8VdRG4VkYMickRE7pvFflSLyLMisl9E9orIh+32z4jIaRHZaf/cNgt9OyEiu+3jb7PbikXkGRE5bP8uOtv7TEO/1nrOy04R6RKRv5iNcyYi3xKRJhHZ42kb8RyJyCfs79xBEbllhvv1byJyQER2icjjIlJot9eISL/nvP33dPVrlL6N+Leb5XP2A0+fTojITrt9xs7ZKBoxfd8zY8y8/MEqa3AUWAGEgNeB9bPUl0rgEvtxHnAIa5GSzwAfneXzdAIoHdL2r8B99uP7gH+ZA3/LBmDZbJwz4I3AJcCes50j++/6OhAGltvfQf8M9utmIGA//hdPv2q8283SOcv4t5vtczbk9c8DfzvT52wUjZi279l8jtznzIIgxph6Y8wO+3E3sJ+5Xb/+duBB+/GDwB2z1xUAbgCOGmMmM0t5whhjfgO0DWke6RzdDjxsjIkaY44DR7C+izPSL2PML40xCfvpK1gVV2ecEc7ZSMzqOXMQa2HgdwIPTcexR2MUjZi279l8Fvc5uSCIiNQAFwNb7KZ77Vvob82G/YFVR/+XIrJdRO6x2yqMMfVgfemA8lnol5c7Sf+Hm+1zBiOfo7n0vfu/wC88z5eLyGsi8ryIXDtLfcr0t5sr5+xaoNEYc9jTNuPnbIhGTNv3bD6L+1kXBJlpRCQX+BHwF8aYLuBrwEpgI1CPdUs401xjjLkEaz3bD4nIG2ehDyMiVknotwKP2k1z4ZyNxpz43onIp4AE8D27qR5Yaoy5GPgI8H0RyZ/hbo30t5sT5wx4N+lBxIyfswwaMeKmGdrGdc7ms7ifdUGQmUREglh/tO8ZYx4DMMY0GmOSxpgU8A2m6VZ0NIwxZ+zfTcDjdh8aRaTS7ncl0DTT/fLwZmCHMaYR5sY5sxnpHM36905E7gbeArzH2Aatffveaj/ejuXRrpnJfo3yt5sL5ywAvB34gdM20+csk0Ywjd+z+Szuc2ZBENvL+yaw3xjzBU97pWeztwF7hu47zf3KEZE85zHWYNwerPN0t73Z3cATM9mvIaRFU7N9zjyMdI6eBO4UkbCILAdWA1tnqlMicivwceCtxpg+T3uZiPjtxyvsfh2bqX7Zxx3pbzer58zmRuCAMabOaZjJczaSRjCd37OZGCmexhHo27BGnY8Cn5rFfrwB65ZpF7DT/rkN+A6w225/Eqic4X6twBpxfx3Y65wjoATYDBy2fxfP0nmLAK1Agadtxs8Z1sWlHohjRUzvG+0cAZ+yv3MHgTfPcL+OYHmxzvfsv+1t32H/jV8HdgC/NwvnbMS/3WyeM7v928AHhmw7Y+dsFI2Ytu+Zlh9QFEVZgMxnW0ZRFEUZARV3RVGUBYiKu6IoygJExV1RFGUBouKuKIqyAFFxVxRFWYCouCuKoixA/n8p29MXqYsyOwAAAABJRU5ErkJggg==", + "text/plain": [ + "
" + ] + }, + "metadata": { + "needs_background": "light" + }, + "output_type": "display_data" + } + ], + "source": [ + "plt.plot(r.total_rewards)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [] + } + ], + "metadata": { + "interpreter": { + "hash": "16af2a8bbb083ea23e5e41c7f5787656b2ce26968575d8763f2c4b17f9cd711f" + }, + "kernelspec": { + "display_name": "Python 3.8.12 ('py38')", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.8.12" + }, + "orig_nbformat": 4 + }, + "nbformat": 4, + "nbformat_minor": 2 +} diff --git a/lessons/X-Extras/X1-MultiModal/README.md b/lessons/X-Extras/X1-MultiModal/README.md index 3b81a155..dc7a056a 100644 --- a/lessons/X-Extras/X1-MultiModal/README.md +++ b/lessons/X-Extras/X1-MultiModal/README.md @@ -50,9 +50,9 @@ To generate an image corresponding to a text prompt, we start with some random e A great library that implements VQGAN+CLIP is [Pixray](http://github.com/pixray/pixray) -![Picture produced by Pixray](images/a_closeup_watercolor_portrait_of_young_male_teacher_of_literature_with_a_book.png) | ![Picture produced by pixray](images/a_closeup_oil_portrait_of_young_female_teacher_of_computer_science_with_a_computer.png) -----|---- -Picture generated from prompt *a closeup watercolor portrait of young male teacher of literature with a book* | Picture generated from prompt *a closeup oil portrait of young female teacher of computer science with a computer* +![Picture produced by Pixray](images/a_closeup_watercolor_portrait_of_young_male_teacher_of_literature_with_a_book.png) | ![Picture produced by pixray](images/a_closeup_oil_portrait_of_young_female_teacher_of_computer_science_with_a_computer.png) | ![Picture produced by Pixray](a_closeup_oil_portrait_of_old_male_teacher_of_mathematics_in_front_of_blackboard.png) +----|----|---- +Picture generated from prompt *a closeup watercolor portrait of young male teacher of literature with a book* | Picture generated from prompt *a closeup oil portrait of young female teacher of computer science with a computer* | Picture generated from prompt *a closeup oil portrait of old male teacher of mathematics in front of blackboard* > Pictures from **Artificial Teachers** collection by [Dmitry Soshnikov](http://soshnikov.com)