From 5e416b31aa9ee2a19d2532fffdadfe471280e32d Mon Sep 17 00:00:00 2001 From: Dmitri Soshnikov Date: Wed, 29 Sep 2021 23:44:44 +0300 Subject: [PATCH] Finish 1st iteration of ownframework --- .../04-OwnFramework/OwnFramework.ipynb | 833 +++++++----------- 3-NeuralNetworks/04-OwnFramework/README.md | 23 +- .../04-OwnFramework/images/ComputeGraph.png | Bin 0 -> 9655 bytes .../images/ComputeGraphGrad.png | Bin 0 -> 16389 bytes .../images/Cross-Entropy-Loss.png | Bin 0 -> 19022 bytes .../04-OwnFramework/images/NeuroArch.png | Bin 0 -> 11806 bytes .../04-OwnFramework/images/overfit.png | Bin 0 -> 18414 bytes 7 files changed, 349 insertions(+), 507 deletions(-) create mode 100644 3-NeuralNetworks/04-OwnFramework/images/ComputeGraph.png create mode 100644 3-NeuralNetworks/04-OwnFramework/images/ComputeGraphGrad.png create mode 100644 3-NeuralNetworks/04-OwnFramework/images/Cross-Entropy-Loss.png create mode 100644 3-NeuralNetworks/04-OwnFramework/images/NeuroArch.png create mode 100644 3-NeuralNetworks/04-OwnFramework/images/overfit.png diff --git a/3-NeuralNetworks/04-OwnFramework/OwnFramework.ipynb b/3-NeuralNetworks/04-OwnFramework/OwnFramework.ipynb index 968a6bcf..81883322 100644 --- a/3-NeuralNetworks/04-OwnFramework/OwnFramework.ipynb +++ b/3-NeuralNetworks/04-OwnFramework/OwnFramework.ipynb @@ -3,11 +3,14 @@ { "cell_type": "markdown", "source": [ - "# Введение в нейронные сети\n", - "\n", - "## Эпизод 2: Многослойный персептрон\n", - "\n", - "Дмитрий Сошников | dmitri@soshnikov.com" + "## Multi-Layered Perceptrons\r\n", + "## Building our own Neural Framework\r\n", + "\r\n", + "> This notebook is a part of [AI for Beginners Curricula](http://github.com/microsoft/ai-for-beginners). Visit the repository for complete set of learning materials.\r\n", + "\r\n", + "In this notebook, we will gradually build our own neural framework capable of solving multi-class classification tasks as well as regression with multi-layered preceptrons.\r\n", + "\r\n", + "First, let's import some required libraries." ], "metadata": { "slideshow": { @@ -15,79 +18,40 @@ } } }, - { - "cell_type": "markdown", - "source": [ - "Данная презентация представляет собой введение в современные нейронные сети на основе Microsoft Cognitive Toolkit (CNTK). Идея однодневного мастер-класса основана на Neural Network Workshop в Microsoft Research Cambridge. Материал и фрагменты кода частично взяты из презентаций [Katja Hoffmann](https://www.microsoft.com/en-us/research/people/kahofman/), [Matthew Johnson](https://www.microsoft.com/en-us/research/people/matjoh/) и [Ryoto Tomioka](https://www.microsoft.com/en-us/research/people/ryoto/) из Microsoft Research Cambridge. [NeuroWorkshop](http://github.com/shwars/NeuroWorkshop) подготовлен [Дмитрием Сошниковым](http://blog.soshnikov.com), Microsoft Russia." - ], - "metadata": { - "slideshow": { - "slide_type": "notes" - } - } - }, - { - "cell_type": "markdown", - "source": [ - "## Обучение с учителем\n", - "\n", - "**Дано:**\n", - " * Обучающая выборка $\\mathbf{X} \\in \\mathbb{R}^{n \\times k}$\n", - " * $n$ - размер выборки\n", - " * $x_i$ представлено вектором свойств размерности $k$\n", - " * Известные значения целевой функции $\\mathbf{Y}$ ($y_i$ соответствует вектору свойств $x_i$)\n", - " * $\\mathbf{Y} \\in \\mathbb{R}^{n \\times 1}$ (задачи регрессии)\n", - " * $\\mathbf{Y} \\in C^{n \\times 1}$, где $y_i \\in C$ (задачи классификации на $|C|$ классов)\n" - ], - "metadata": { - "slideshow": { - "slide_type": "notes" - } - } - }, - { - "cell_type": "markdown", - "source": [ - "## Задача\n", - "\n", - "**Дано:**\n", - " * Обучающая выборка $\\mathbf{X} \\in \\mathbb{R}^{n \\times k}$\n", - " * Входные значение целевой функции $\\mathbf{Y}$\n", - "\n", - "**Необходимо построить:**\n", - " * Функцию $f : \\mathbf{X} \\rightarrow \\mathbf{Y}$ который _точно предсказывает_ значение целевой функции на новом наборе входных данных $\\mathbf{X}_{new}$\n" - ], - "metadata": { - "slideshow": { - "slide_type": "notes" - } - } - }, { "cell_type": "code", - "execution_count": 1, + "execution_count": 4, "source": [ + "%matplotlib nbagg\r\n", "import matplotlib.pyplot as plt \r\n", "from matplotlib import gridspec\r\n", "from sklearn.datasets import make_classification\r\n", - "import numpy as np" - ], - "outputs": [], - "metadata": { - "slideshow": { - "slide_type": "skip" - } - } - }, - { - "cell_type": "code", - "execution_count": 2, - "source": [ + "import numpy as np\r\n", "# pick the seed for reproducability - change it to explore the effects of random variations\r\n", "np.random.seed(0)\r\n", "import random" ], - "outputs": [], + "outputs": [ + { + "output_type": "error", + "ename": "ImportError", + "evalue": "cannot import name 'animation' from partially initialized module 'matplotlib' (most likely due to a circular import) (C:\\winapp\\Miniconda3\\lib\\site-packages\\matplotlib\\__init__.py)", + "traceback": [ + "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m", + "\u001b[1;31mImportError\u001b[0m Traceback (most recent call last)", + "\u001b[1;32m\u001b[0m in \u001b[0;36m\u001b[1;34m\u001b[0m\n\u001b[1;32m----> 1\u001b[1;33m \u001b[0mget_ipython\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mrun_line_magic\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m'matplotlib'\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;34m'nbagg'\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m 2\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[0mmatplotlib\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mpyplot\u001b[0m \u001b[1;32mas\u001b[0m \u001b[0mplt\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m 3\u001b[0m \u001b[1;32mfrom\u001b[0m \u001b[0mmatplotlib\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[0mgridspec\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m 4\u001b[0m \u001b[1;32mfrom\u001b[0m \u001b[0msklearn\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mdatasets\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[0mmake_classification\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m 5\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[0mnumpy\u001b[0m \u001b[1;32mas\u001b[0m \u001b[0mnp\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n", + "\u001b[1;32mC:\\winapp\\Miniconda3\\lib\\site-packages\\IPython\\core\\interactiveshell.py\u001b[0m in \u001b[0;36mrun_line_magic\u001b[1;34m(self, magic_name, line, _stack_depth)\u001b[0m\n\u001b[0;32m 2346\u001b[0m \u001b[0mkwargs\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;34m'local_ns'\u001b[0m\u001b[1;33m]\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mget_local_scope\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mstack_depth\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m 2347\u001b[0m \u001b[1;32mwith\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mbuiltin_trap\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m-> 2348\u001b[1;33m \u001b[0mresult\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mfn\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m*\u001b[0m\u001b[0margs\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m 2349\u001b[0m \u001b[1;32mreturn\u001b[0m \u001b[0mresult\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m 2350\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n", + "\u001b[1;32mC:\\winapp\\Miniconda3\\lib\\site-packages\\decorator.py\u001b[0m in \u001b[0;36mfun\u001b[1;34m(*args, **kw)\u001b[0m\n\u001b[0;32m 230\u001b[0m \u001b[1;32mif\u001b[0m \u001b[1;32mnot\u001b[0m \u001b[0mkwsyntax\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m 231\u001b[0m \u001b[0margs\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mkw\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mfix\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0margs\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mkw\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0msig\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 232\u001b[1;33m \u001b[1;32mreturn\u001b[0m \u001b[0mcaller\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mfunc\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m*\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mextras\u001b[0m \u001b[1;33m+\u001b[0m \u001b[0margs\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m**\u001b[0m\u001b[0mkw\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m 233\u001b[0m \u001b[0mfun\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m__name__\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mfunc\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m__name__\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m 234\u001b[0m \u001b[0mfun\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m__doc__\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mfunc\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m__doc__\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n", + "\u001b[1;32mC:\\winapp\\Miniconda3\\lib\\site-packages\\IPython\\core\\magic.py\u001b[0m in \u001b[0;36m\u001b[1;34m(f, *a, **k)\u001b[0m\n\u001b[0;32m 185\u001b[0m \u001b[1;31m# but it's overkill for just that one bit of state.\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m 186\u001b[0m \u001b[1;32mdef\u001b[0m \u001b[0mmagic_deco\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0marg\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 187\u001b[1;33m \u001b[0mcall\u001b[0m \u001b[1;33m=\u001b[0m \u001b[1;32mlambda\u001b[0m \u001b[0mf\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m*\u001b[0m\u001b[0ma\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m**\u001b[0m\u001b[0mk\u001b[0m\u001b[1;33m:\u001b[0m \u001b[0mf\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m*\u001b[0m\u001b[0ma\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m**\u001b[0m\u001b[0mk\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m 188\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m 189\u001b[0m \u001b[1;32mif\u001b[0m \u001b[0mcallable\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0marg\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n", + "\u001b[1;32mC:\\winapp\\Miniconda3\\lib\\site-packages\\IPython\\core\\magics\\pylab.py\u001b[0m in \u001b[0;36mmatplotlib\u001b[1;34m(self, line)\u001b[0m\n\u001b[0;32m 97\u001b[0m \u001b[0mprint\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m\"Available matplotlib backends: %s\"\u001b[0m \u001b[1;33m%\u001b[0m \u001b[0mbackends_list\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m 98\u001b[0m \u001b[1;32melse\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 99\u001b[1;33m \u001b[0mgui\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mbackend\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mshell\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0menable_matplotlib\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0margs\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mgui\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mlower\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m \u001b[1;32mif\u001b[0m \u001b[0misinstance\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0margs\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mgui\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mstr\u001b[0m\u001b[1;33m)\u001b[0m \u001b[1;32melse\u001b[0m \u001b[0margs\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mgui\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m 100\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_show_matplotlib_backend\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0margs\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mgui\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mbackend\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m 101\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n", + "\u001b[1;32mC:\\winapp\\Miniconda3\\lib\\site-packages\\IPython\\core\\interactiveshell.py\u001b[0m in \u001b[0;36menable_matplotlib\u001b[1;34m(self, gui)\u001b[0m\n\u001b[0;32m 3515\u001b[0m \"\"\"\n\u001b[0;32m 3516\u001b[0m \u001b[1;32mfrom\u001b[0m \u001b[0mIPython\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mcore\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[0mpylabtools\u001b[0m \u001b[1;32mas\u001b[0m \u001b[0mpt\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m-> 3517\u001b[1;33m \u001b[1;32mfrom\u001b[0m \u001b[0mmatplotlib_inline\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mbackend_inline\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[0mconfigure_inline_support\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m 3518\u001b[0m \u001b[0mgui\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mbackend\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mpt\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mfind_gui_and_backend\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mgui\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mpylab_gui_select\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m 3519\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n", + "\u001b[1;32mC:\\winapp\\Miniconda3\\lib\\site-packages\\matplotlib_inline\\backend_inline.py\u001b[0m in \u001b[0;36m\u001b[1;34m\u001b[0m\n\u001b[0;32m 4\u001b[0m \u001b[1;31m# Distributed under the terms of the BSD 3-Clause License.\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m 5\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 6\u001b[1;33m \u001b[1;32mimport\u001b[0m \u001b[0mmatplotlib\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m 7\u001b[0m from matplotlib.backends.backend_agg import ( # noqa\n\u001b[0;32m 8\u001b[0m \u001b[0mnew_figure_manager\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n", + "\u001b[1;32mC:\\winapp\\Miniconda3\\lib\\site-packages\\matplotlib\\__init__.py\u001b[0m in \u001b[0;36m\u001b[1;34m\u001b[0m\n\u001b[0;32m 105\u001b[0m \u001b[1;31m# cbook must import matplotlib only within function\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m 106\u001b[0m \u001b[1;31m# definitions, so it is safe to import from it here.\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 107\u001b[1;33m \u001b[1;32mfrom\u001b[0m \u001b[1;33m.\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[0m_api\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mcbook\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mdocstring\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mrcsetup\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m 108\u001b[0m \u001b[1;32mfrom\u001b[0m \u001b[0mmatplotlib\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mcbook\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[0mMatplotlibDeprecationWarning\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0msanitize_sequence\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m 109\u001b[0m \u001b[1;32mfrom\u001b[0m \u001b[0mmatplotlib\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mcbook\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[0mmplDeprecation\u001b[0m \u001b[1;31m# deprecated\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n", + "\u001b[1;32mC:\\winapp\\Miniconda3\\lib\\site-packages\\matplotlib\\rcsetup.py\u001b[0m in \u001b[0;36m\u001b[1;34m\u001b[0m\n\u001b[0;32m 22\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[0mnumpy\u001b[0m \u001b[1;32mas\u001b[0m \u001b[0mnp\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m 23\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 24\u001b[1;33m \u001b[1;32mfrom\u001b[0m \u001b[0mmatplotlib\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[0m_api\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0manimation\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mcbook\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m 25\u001b[0m \u001b[1;32mfrom\u001b[0m \u001b[0mmatplotlib\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mcbook\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[0mls_mapper\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m 26\u001b[0m \u001b[1;32mfrom\u001b[0m \u001b[0mmatplotlib\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mcolors\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[0mColormap\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mis_color_like\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n", + "\u001b[1;31mImportError\u001b[0m: cannot import name 'animation' from partially initialized module 'matplotlib' (most likely due to a circular import) (C:\\winapp\\Miniconda3\\lib\\site-packages\\matplotlib\\__init__.py)" + ] + } + ], "metadata": { "slideshow": { "slide_type": "skip" @@ -97,8 +61,9 @@ { "cell_type": "markdown", "source": [ - "## Пример\n", - "Рассмотрим пример двухмерной задачи классификации на 2 класса. Примером такой задачи может быть классификация опухоли на 2 типа - доброкачественная и злокачественная, в зависимости от её размера и возраста.\n" + "## Sample Dataset\r\n", + "\r\n", + "As before, we will start with a simple sample dataset with two parameters.|\r\n" ], "metadata": { "slideshow": { @@ -108,7 +73,7 @@ }, { "cell_type": "code", - "execution_count": 3, + "execution_count": null, "source": [ "n = 100\r\n", "X, Y = make_classification(n_samples = n, n_features=2,\r\n", @@ -130,7 +95,7 @@ }, { "cell_type": "code", - "execution_count": 4, + "execution_count": null, "source": [ "def plot_dataset(suptitle, features, labels):\r\n", " # prepare the plot\r\n", @@ -154,32 +119,12 @@ }, { "cell_type": "code", - "execution_count": 5, + "execution_count": null, "source": [ - "plot_dataset('Scatterplot of the training data', train_x, train_labels)" - ], - "outputs": [ - { - "output_type": "stream", - "name": "stderr", - "text": [ - "C:\\winapp\\Miniconda3\\lib\\site-packages\\ipykernel_launcher.py:11: UserWarning: Matplotlib is currently using module://ipykernel.pylab.backend_inline, which is a non-GUI backend, so cannot show the figure.\n", - " # This is added back by InteractiveShellApp.init_path()\n" - ] - }, - { - "output_type": "display_data", - "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYQAAAEnCAYAAACpNTSTAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nOydd3hb1fnHv6/kve14JHbiOAtIAkmALAghgx2gYRYoFChQoKUtlA7oD2gptJRO2jJaoIPSQVkFQoAQCBlAAsQhg+yB7cTxTLynLOn8/vjqIlnWtLVsn8/z6EksXd177tW973vOO0UpBY1Go9FoTNEegEaj0WhiA60QNBqNRgNAKwSNRqPRONAKQaPRaDQAtELQaDQajQOtEDQajUYDQCuEsCEiF4nIOhGpE5FOEakQkVdF5NwwHe8OEbnEyzjuDMcxA0VErhcRJSIlQX6vRETuF5HxIR7PSBFZJiINjnHd4WW7GY7j53j4TInIz0I5Lg/H8Pibhmjf5SLyTD++V+I49+tDP6rgEZGFjvEs7Md37xeRxWEY1qBFK4QwICLfAfAKgH0AbgRwPgBDeITrBrwDgCfhcRGAqCqEAVAC4CcAQqoQAPwYwALwtzkFwH+9bDfDcfw+CiFCePtNQ8HFAB7sx/eqwWv2RmiHExV+gvA9j4OSuGgPYIjyfQCvKqVudHnvPQBPi8igV8IikqiU6o72OAbAZABblVKvRHsgoSLY30Qptbk/x3Ec46P+fFczCFBK6VeIXwDaAPwpwG3HAfgngBoA3QA+B/AHl89nAXgJQCWATgB7ADwEINllm3IAyu31jOPl/n65y/dyAfwJwGHHsXcDuNltfNc7vnc6gBcBNAHY4vjsGce4TgWwEUCXYyzf9rKPEpf34sFVUzkAi+PfnwGId3y+0MPYFYCFPq6lAPiu4xpZwNnsYwAyHJ+XeNlniYd9Xe9rW8f/fwbgOwDKALQCWAtgqod9XQIK0Q7H9XsRQLGf+8Ljb+r47H7H38cDeBu8315zfHY2gDcd594BYDuA7wEwe9j/Mx7Ody6AfwNoAVAF4I8Akly2M67h9S7vGffBiQDedxx3H4BbPZzXmQA2O+6V/QBucny/3Nf1cHw3D8B/HGNrAvAsuALudV8Ecg28/Lb3B/rMDdWXXiGEh08AXCcin4MP6l5PG4nIOMe2HeDydR+AMeANbVAMYAv40LQCmAqaPMYDuNKxzcXgA7AVFBYAUO/4Nw+8wb/k+LvbcewMAB8CSHZ8pwzAOQD+5JhtPuo23H8DeA7AZei9sswA8DyAX4IP+JUA/igirUqpZzxeHfIPAF8GH7QPQDPEvY7z+gqATwHcBuBxUOhudHxvp499/hzAjxzfeR3AFNAsMl1EFsBp7ngSgA3ANx3fq/awrzdAgX8vgMtB4eC+7TWgsLgdQAKAXwN4TUSOU0pZAUBEbgWV7t8BPAAgHbzea0VkmlKq1cu5+PpNDV4D8Ffw2tsd740HsArAo6DQnen4fh6Au70cy5V/gr/zJeC1uh9AI3h/+iIDFNa/B8/za+C9tEcptRoARGQKeF0/Ae+TBAD3Ach0Gb8v/gdgOoD/A5+VKxzn6U4g1+AUABvA5+pJx3vGbxzIMzc0ibZGGoovAMcA2AbnzOMI+JCd7bbds+DsrjDA/QoojK8BH6ARLp+VA/iXh+88A6DSw/v3gQ/LJLf3n3aMN87x9/WOc3jEy74VgCvd3n8HQAUAcdtHiePv4+EyI3P53r2O96c5/l7o+PvMAK5NjuN8nnF7/xrHPr7k8t4HANYEsE9j3BM9fKZAoRTv8t5ljvdPdfydBqAZwN/cvlsCrmDu8HN8b7/p/Y7j3B7g/XIPKNRNbvt+xsO5/tRtH8sB7HUbu6cVggKwyOW9RMd99JTLe/8BlVqKy3ujHL9buZ9zOcvLvfYWfKwc/VwDBeBn/X3mhuJr0NuzYxHFFcGJoOPy5+Bs42IAb4vIvS6bng1guVKqytu+RCRDRH4pIgfA2X0POIsTAJMGMMxzAXwMoExE4owXaIIYAc6uXfFmb7cBeNntvf+Cs6wiL9853fHvv9zeN/5e4GfsnpgLCiH3ff4XgLWf+/THO0qpHpe/P3P8W+z49xRw5vxvt2tcCZrnTsfA6PObiMgoEXlSRCpApdMDrnSyAOQHsE93Z/FncJ6PLzqUYyUAfOFr2Of23bkA3lRKdbhsVw1gfQD7PwXe77VeDPQahPGZi3m0yShMKKVsANY5XhCRQgArAPxERB5XSjWCgrfS+14A0NRwJrhk3QKgHcBs0CySNIAh5gOYCN7snhjh9rcnswoANLoJRQCodfxbBM/nZ0TtuO+zxu3zYPC4T6WUVUSO9nOf/mhw+9tw6hq/iyF83vXy/cYBHr/XuToCFpYBKARXEbtBG/hF4Aw5kPvF0zklBvA9T+fS7XbMUQDqPGxXC/+RZKPg+14DELJrEK5nLubRCiFCKKWqROQvAP4AzjI+AZfU3mbREJEkAEtB08ofXN4/IQRDOgo+nLd7+XyP29/Ky3bZIhLv9qAWOP497OU7htAZCeCAy/sjXcYWLK773GG86ZiRj+jnPgeKcczrXcfkgjf/QaC4/yYTQHv5V5VSX6yUROTCAR4nVFTD8wy9wMN7nr7r614zGNA1CPMzF/Nok1EYEJExXj46zvGvMRNeCeACERnlZftEAGb0ncVf72HbbtBBHOj7KxzjOaiUKvXwClRYmQFc6vbelQAOwrtCWOuynStXO/5d5zJ2eBm/Ox85tnff5xXgxGdtn2/4J5jje2I9KPQnernG7krX0/GDOXaK498v7hcRiYfzukabjwAsERFjnHDc+/MC+O4GeL/XXAnmGljQ9/oG88wNOfQKITxsF5HVoI23DLQjLwFwK4AXlFIHHdv9BExaWy8iD4FROkUAzlVKXaOUahaRjwB8T0SqwRXFDfC8qtgJYL6IXAAqnCNKqXLH+zki8g0ApQC6lFKfAXgEFJbvi8gj4IogFVQS85VSSwM811YAvxKRXNBmfBW43L5eObxy7iildojIcwDud8zg14M24vsAPKeU2ubYdC9o/79BRBpAAbnHk7JSSjWIyO8A/EhE2sEIncmg7fgD9C+Ryohouk1E/gEKiW1KKUsgX1ZKtYjIDwA8LiJ5oAO0Gfz9FoCO7f/4Ob6n39Qbu0Bn/s9FxOYY73cDGWuE+BnoeH9bRH4DCt/7QLOPzygjpdQ7IvIBgCdd7rUrwAAFV4K5BjsBnC8iK0CTV5VjJR/oMzf0iLZXeyi+QMG/DLwxu0Ab5GYAPwSQ4LbtBDAC6QiceQiPuHxeAgqSVtDE8xioRNxjr4+DMwbcNWY91bH/RvTNQ8gGFUMZOFuqc+zjDpdtrof3SJtn0DcPoQLAd9y2M/ZR4vKekYdQAT60FXDJQ3DZ7hbHNbG6n7OH8XjKQ3gcjjwEl+0CijJybPsTcKVjcz0HeIhQgYcIHMf7SwCsBuPnO0HF/zcAU/wc29tver/j7zgP35nhOL8Ox2/zABjr7379y+E5ymii2/7uB6B8nSO8R7Ktcb/OYLTQFjjv9VvAidPmAH6LPPBeboUzD2Gp+30RxDWYB2ATeN+65iGUIIBnbii+jLBAjSZoHLVwzlRKjY72WDSDExFJAxXkG6p3Zr8mCmiTkUajiRgi8ihoIqwCI4FuB1eqf/D1PU1k0ApBo9FEkiQws7oANOt9Aq4yt/n8liYiaJORRqPRaADosFONRqPRONAKQaPRaDQAtELQaDQajQOtEDQajUYDQCsEjUaj0TjQCkGj0Wg0ALRC0Gg0Go0DrRA0Go1GA0ArBI1Go9E40ApBo9FoNAC0QtBoNBqNA60QNBqNRgNAKwSNRqPROBjU5a9zc3NVSUlJtIeh0Wg0g4pNmzYdUUrlub8/qBVCSUkJSktLoz0MjUajGVSISIWn97XJSKPRaDQAtELQaDQajYNBbTLSaDQhoLsbaGsDEhKA9PRoj2bgKAXU1QEdHUBqKpCXB4hEe1SDAq0QNJrhSk0NsHIl8P77gM1GQTppEnD++cC0aYNPiCoFbNkCvPoqcPAgYDbzvMaNA5YuBaZPj/YIY55B3VN55syZSjuVYxuLBaio4L+ZmUBR0eCTM0OSAweAX/0KsNuB/HwgPp4CtaEBaG4GLroIuPjiwfVjvfkm8N//Ajk5vNlEeE5NTTyva68Fzjor2qOMCURkk1Jqpvv7eoWgCQtWK/D223xGOzv5bNrtQHExcNllwAknRHuEw5iODuD3vwdSUig4DUSAESP43iuvABMmDJ5Z9eefA88/D4wZQ+VmIAJkZ9N09O9/A8cey5tQ45GYcSqLyBgRWS0iu0Rkh4jcHu0xafqH1Qo8+SSfz/R0Pn9jxvDf5mbg178GPvww2qMcxnz6KX0GrsrAlbg4ICsLWL6cM+zBwKpVQFJSb2XgSkICP1uzJqLDGmzEjEIAYAXwPaXUZABzAdwmIlOiPCZNP/joI77GjeMzaiBCOVNYCPz971zFa6LA+vX+ncc5OcD+/UBLS2TGNBCUAj75hM5jX+TlcTuNV2LGZKSUqgZQ7fh/q4jsAlAEYGdUB6YJCqVoJvIV2JGURPPRRx8BS5ZEdnwa0IbnbSZtIAKYTHT+RIKWFuDjj4F167h6yc0FzjgDOPFEIDHR//etVjqRfWE2R+58BimxtEL4AhEpAXAigI89fHaziJSKSGl9fX2kh6bxQ3MzUF3tfwKamQls2kQ5sH8/fZydnZEZ47Bn5Eigvd33NlYrFUJaWvjHs3cvcNddwH/+Q/9GYiLDRp98EvjJTwB/z7kIMGoU0Nrqe7vWVi5PNV6JmRWCgYikAXgZwB1KqT7rVaXUUwCeAhhlFOHhafxgt/P59Bec0tXF1fsddziDQeLigEWLgAsvjIwcGrYsWABs2MCL7u2Hqq0FTj0VSE4O71hqa4Hf/IY/eEGB8/2kJDqDa2qA3/4W+OlPfa8UzjkH+NvfgIwM79s0NgJXXhm6sQ9BYkohiEg8qAz+rZT6X7THowme9HQ+y11dvf0HrrS20jIwYgQwY4Zzpd/Tw7D4zZtpLfjgA8qD5GTKptNP1xO8kHDMMcBxx3FmPmZMX6XQ0sL3zj03/GN57z3OIrwJ8pEjgbIyYOtWYPZsJtFVVHAFk53Nz0X42cqVQFVV35tEKeDwYaCkhCYojVdiJg9BRATAPwA0KKXuCOQ7Og8hNnnlFWDZMmDs2L6fKQWsXs2J4Xnn0cnsSkcHw1VTU4G5c/mv1Uqrgd0O3HgjMG9eZM5jSNPWBjz2GLBrF8NPU1OpkZub+fcdd1BxhBOrFfjmN+kvSEjwvl1DA51SU6cC77xDpWDEMU+YAHz5yxxrYyPw6KMMQU1I4Iqiq4vndcwxwG23eY+sGmYMhjyEeQC+CuAzEdnieO//lFJvRnFMmn6weLFzdl9Q0HsC2tDAydrUqX2fTcPRbDbTn5CQwP+bzcDo0Xy2n3qKeVSTJkX2nIYcaWnAD37AVcKaNc6l2CWXADNnUkGEG0NY+1IGAAX7O+/Q2VRY6DQdKcWZxUMPUYHNmAHcey+3++gjJqRlZwOnnELFMZiS7KJEzCgEpdQHAPQvNgTIzATuvht44gmu9uPi+LJYgEOHmK08fXrf57O+ntaKrCxOVJuaaBEwSEqinHrjDT7/mgFiNgOTJ/MVDRIT6bi22XxHCB04wBuipKT3TWMk0iUlAX/+M30NqalcDYR7dTNEiRmFoBla5OUBP/4xV+9btjCopaAAOHKEZmNPz//Bg72jIe32vtvk5tKc3No6NOqwDWvi42n7//RTRgl5wm4Hdu9maru3GX5qKm+sTZvoaNL0G60QNGFDhCv1CROc75WWcvXvic5OriSU4islpe82JhNf7e1aIQwJzjqLEU/d3Z6jiGpqeDOMH+97P6mpnHlohTAgYjIPQTN0Of54ZxSSO4mJ9DN2dzPoxJP/z27ny5Oy0AxCxo8HbriBySvV1TQfAU77YkcHbxp/yWlmM28ezYDQCkETUZKSGApeVdU3aXTsWD7/XV3eLQRHj9Ih7SvcXDPIOP10JqCdfDJvjIoKRh+ccw7w859zZuBP2Le10cegGRDaZKSJOAsWcCL43HP8NzmZVoGODgacTJrkuSxNdzd9BxdcEPkxD2tqamjnb2ykx//EE0OfEDJuHHDzzVwtWCxcERiOpvnz2bNh9GjP37XZ+NLxyANGKwRNxBFh4tns2fQpHDjA96ZOpZx59FGgvJwOZCMPoa6O/95wA3OqNBGgs5NVCD/5hI6b+HiGib70EkM8b7op9OGpRkiaK0uW8Eapr+87U7BauaJYsqR3prOmX8RMYlp/0IlpQ5O2NsqgFSsoAxISmKm8aJEuZR8xrFbgd79j4lpxcW/7nVJAZSXt/z/4gf88glBQVcVEuqoqp9IwEtSWLGEzH3/F7TRfMBgS0zQaAMyZWryYL1/ldjRhZNs2YPt2mnLcfwARmm/27KEpae7c8I+nsBD42c+YSLdtGx1NhYVMonNPd/dFdTWwcSNnGmlp9FvopLUv0ApBE3Gqq2kCEuHk09fzrJ/TKPH22842lJ4QYc+Et96KjEIAaLY67rj+2Qy7uoBnnmEGs8nE6AaLheMfP55lLUaMCPmQBxtaIWgiRnk5Kxzv3ctn0mDuXJajCWaipwkz5eX+G85kZjKbMNaXcXY7S2lv3sxQNnfz1+HD7C99333DvsyuDjvVRIQDBxhBePgwn8niYr6KitgX5ec/Z6kKTYxgNvtvn2m3Dw67vWHaclcGgLOXQm0tC3ANc7RC0IQdmw34058YkOLeSc1sZgXmhgbgxRejN8aA6exk+KWnzLqhxIknshyEL44cAaZNi+3VAcB+yykpvseZn88oBk/1UoYR2mSkCTu7d1N2+MobKixkBYMvfzlGKxTv30+B8emnzg5Ap5wCnH02NdpQwyhZa7X2DQMFqOU7Onj+sU5Fhf9MxpQU3qSdnZGp9Bqj6BWCJuzs3etZprhiWB4OHQr/eIJm3TrgwQeBHTsYXTNmDM0MH3/MDNutW6M9wtAzfjywdCmFqXu7zY4O+hjOPRc49tioDC8o4uL8z/yNAlqDwQQWRvQKQRNy2tpYZ6y2lgmnVVWBfzfmVuwVFWzN6FqHH6DgKCyksHzsMeAXv2Am3VBBhLH9ubnAq6/yOhgml4wMZgguXBj75iIAmDWLNdN9zfwbGpgi763N3zBBKwRNyLDbGcX3yiu0NMTH07JQW0tncna292fSZmPJ+4YG9lAYO7Z3JFLUWLXK2X3LE6mpLLC0fj3wpS9FdmzhRoR1hubN44qgvZ2mlXHjBtdM+rTTgOXLvVdUtdsZ0XDjjZEfW4yhFYImZLz2GvC//zF6yLWvwahRFPKrV7NkhWvfdqUoazZvptz95z/5fOblARddRDN91CahSlHQu3bp8URuLs1KQ00hGJjNvWuYDzby84HrrgP+8hfmGrjmV7S3s1bTmWeya9MwRysETUior3f2UXb3F8THM9dg7Vqa4Wc6EuaVYjLsjh2cuM2f73Qot7YyMunoUeDCCyN7Ll9gtXLp4s8BEh+vY2ZjnQULmOjyv/8xd8Js5swjMxP42tcGj/krzGiFoAkJGzbwefImO4uLaXnYuJHPZXIyhf3WrZzAzZ7dO7ooPZ3m3JdfZmTj2LGROY9exMUxUamzs/eyxp32dv9JXJroM306b6aqKs44kpIYIDCYzF9hRisETUg4cMB/kuf48Xz2LrmERTOXL6e/75hjPPsL4uP5Wr0auP76sAzbNyKsyW/YwbzR1ARcfnlQu1aK7UVra3nuxcWhryit8YAIsyE1HtEKQRMSTCb/ia3GdiefTLP86697VwYGubkM/Q9aIbS3s2TqypW0ZyUm0rm4YEFwkve001jX5+hRz7VuampYdvmkkwLe5b59LKtz+LDTSqEUe91fd51/l4VGEy5iIY5DMwSYNo3hpr7o7GSQSl6eM+zbn9nWZHJ2VQyYujrg/vuBf/yDBcwKCxkquWoVcO+9dBQHSlYW8MMf0nxUXs4wqPZ2KojychZ4+/73fZuUXNi9mxGqbW00g7m+yspYwqO2Nsjz1WhChFYImpAwaxbNO52dnj9XilVOzzmH2xlh/K2tvvfb3Byk/6Cnh3X8jZaK6enUKomJTCrLzweeeoo2rkAZMwZ46CHglls46IQE7vs736HiCdB/YLPx0FlZDMF1VYYiXBlYLOwkp9FEA20y0oSEtDTg619njlZ2du/Ivp4e+vGOPZbRfQZLlgBPP+29qoBSVBjnnBPEQLZvpxnHW52MpCTO5t94gwI9UJKTGQN7yilBDKY3u3ZxgeGrhEdBAR3tnpqDaTThRq8QNCFj1iw20EpLY2Sf8aqroyK4887eiaCzZtHRXFnZ1/+gFL97wglsrRkw69b5r0WTl8dUaveSDGGmrMx/QIvJREVaWRmZMWk0rugVgiakHH88y/4cPMiioHFxTGz1JKMTE6kknnoK+OwzCsOEBJpN7HZgzhw6k/2lAfSisdF/+QFD6nZ0RLSQWaDdanU4vCZaxJRCEJG/AbgAQJ1S6vhoj0fTP0ScjlJ/pKdTKRw+zGii5mb6aSdMYFjmI4/Q9j5pEhPXCgv9CMzsbFatTE/3vo3dTukcoCM4VJSU+HeQK8VtRo2KyJA0ml7ElEIA8AyAxwA8G+VxaCKI0aJ39Gj+/f77wG9+40wkFQHefZfVp886C7jySh+ml9NPZx0MX4Xm6uuBGTMi3h1ryhSeT1ub90PX1XGVpUNPNdEgpnwISql1ABqiPQ5N9Ni8mY7m3Fwma2Vm0ulsVJ1esYLFN71iSNO6Os+fd3XRVLRkSVjG74u4ONZPO3IEaGnp/ZlS1FMAcNVVER+aRgMgxhRCIIjIzSJSKiKl9cYTpBkSKAW88ALzvzy5AcxmmqHefLOvQP2C+Hjgu9+lOai8nNNxpeiYqKxkkP/Xvw5MnBjOU/HKtGlMWxBhRenycr4qKujrvucenUiriR6iAvV0RQgRKQGwPBAfwsyZM1VpaWnYx6SJDGVlwAMPcGXgy09QUcGM3oULfeysvZ0NbN5+m1PvpCQWU1q4MCYkrs3GxkE1Nc7SFSUl2qGsiQwiskkpNdP9/VjzIWiGMc3NzgAgX8TFOc0rXklNZRvIxYsDS4mOMGYzS1VMnhztkWg0TgadyUgzdElICCw002oNMkAoxpSBRhOrxNQKQUSeA7AQQK6IVAL4iVLqr9Edlaa/2GwsZWE202LjTy6PG0cXgMVC5WAgyo4RR/cio/kQFIDutjE4Yeox0PMZjSa0xJRCUErp+IohQEsLE4bffpumfKWYR3DeecCM47ogFeWsZ5GT0yuxIDmZYaWvv+60p484uhcnbXoaKR20EXV2ADNSgeLH84Cbv85yqRqNJiTElELQDH7q64GHH2bNnvx8RgwpBdRXduOD25chxfwuJpX0wGQSJhqUlABXXAEcdxwAdqE8eJDhp1MT9mHep79Ed2I6jqaVoK0NSMkFZpwGiKUJ+OUvgbvvprbRaDQDRq+5NSHDbmdxu/Z2hocadv44WzeW7P09Tmtejt31I3BQFTOpoLiYZaQffpi1hUBT0be/DdzwNYVZ2/+Oo91pqOvOQnc3i+MtWMAS2sjKYnbX3/8eeE0IjUbjE71C0ISMffsYEupezXNc2Srk1e9EU1YJknsEe/dSYYgIzUaJiWyg/LvfAampiI8HFhUfgP24anTlF0OBPog+2clZWTzggQNRyyvQaIYSeoWgCRkbN/Z2BgOAyW7FpH1voS1tJCCChAQ6mnsllqWm0pPsmlNSXQ2TKKSkClJTvZSqMLzU1dWhPhWNZliiVwiakNHezighV1Lb6xDf0wFrXBJS29gKrNOWAavVLW40NZWNABYs4N/BhIoOJKzUamWjgg8/dFbWmzeP9qlYbr7e1UUz23vvscJrRgaT7k4+2WFT02iCRysETcjIy6OcciWttQp59dsRZ+0GIFAA8i0KqbuKgJOmOoWXe6/MMWP4r6+kMsN3YGwbLPX1wB/+wJIWyck0XZWVUTkYHdFycvq373BSXQ389rcsipSZSXva0aPAX/8KvPwy8L3v9f+aaIY12mSkCZq2NvYG3rmTMslgzhzKdENOZ7RUYmbpn5HU1QxLQjq6kzLRbma1usSGasamdnRw4/Z2JiIYFBfzb9cDuHPkCLcpLg7+JNrbgV//mvsoKWGrsqwsFsYrKWGLt9/+tq+Gizbt7SwF29HBcWZnU5llZfFvux341a+42tFogkQrBE3AtLayb/0dd1Dm/Pa37JD2yCOcZBcVAbNn08+r7AonbXoaNnMijuQeh3hLO+x2Wmhy80yQjAzmImzbxjftdppqDESAG26ghqmr6x1JpBTfs9m4TX9MRh9/zH14qjNtt9OEtXs3sGZN8PsOJxs3cjWQn+/585wcKo0PP4zsuDRDAm0yGiJYrZzMJib2teOHgpYWRofW1DCXzOhiZrcDe/YAP/sZ8KMfUT53dwOH3y9DQk0FmkeMRXt6KuLrq4GedowcnepsUpaWxpn4nj3Al7/ct4nwmDHAvfcCzz7LSnCG4FeKCWnXXutsohAsK1b07Zlgt9NktG8fT6Knh3kOe/YAS5f6boYcLqqrqWFtNiqBt99mcocv8vKAlSujUuJbM7jRCmGQU1MDrFoFrF1LpSBC083ZZ4dWfr30EitHu1tnTCZOshsagCeeAH7xC+D224FDWeXoOAwc6RI0NaWgPuU0nNC1ES2HmmA9KkhKMaG70wZ7Wxc2ZMxGzZFLsXgfo0d7TfhHj6amOXyYygOgRioq6r8z2W7n6sC1pZvdziinykoqqsxMvmfYx7ZtY93qSFWjq6ujIty+vfd5bt4MzJ3r7AinFBWXCLW0CH0KtbUcv0kbATSBoxXCIGbXLobuK0UTeHw8J5KbNgEbNgA33dTbCtNfWlpogfDV1jEnh3X99+yhzBxb2IPqbIGtmhPalJR01GIREjsa0FVRh+4mKxKy05BZaMPhCadj62bB3nfKsGB6E86/KB7miS6NmN1bqg0UEefFMpY6Bw9SGWRnOwWw3c7tCgpoL3v0UdrJwt1688gR4KGHGJ/LhA3nZ1u3Ah98AJxyCm+f9KYAACAASURBVFcx+/ZxO4CRRpMmcSWRmKiL+vUHpbhK3LqVfpqCAkZuZWdHe2QRQSuEQUpTEwNkMjJ6tw82mym4u7qAv/yFMjSQ3sa+qKjgc+Kv2b3ZTMvO5MlAlRqFA/sV0ke6fk9Q1TUCdTIC8VmA1aJwslQgo+corvn8HqS2VKN1m+DQWoWSCXEsXX3xxZ675QwEEc6yP/6Yqw2lOPC0tN5CtKPD6ehOT6ft/tNPQ6NlffHCC/QDeOrbMGkSVytvvEGTV1oaHcpKUUGUljJy69ZbtUIIliNHmCB54ABv5rg45sf8+99ccl9+uf+HYJCj15ODlA0b+Px76yWflMTJ7apVAz+W3R6YbDGZaL0AgLcPTYElKRPJ9vZe+zl6lJPXuDgg3daEurYUHL/9OcT1dKIlqxjWUcXY1jQWliyHvfyRR3iioeaMM/iwWywUvp2dvbPqrFYKWVe7W3p67+S5cNDQwGN4W46NHcvVSkcHx2uM2TAVpaYynNZiCe84hxqGk+zwYV7jMWP4G4wdy1nVW28B//rXkC+TErRCEJFUEYnhjJ3hwbp1/n2L+fnA+vUUxAMhL4/78Pcs9PTw2VEK+PDjOOyZc+0XiWkAZZhh1k62tiJLmvBey0y0ZRShOykTEIHZzG3qm+L5MO7aBaxePbAT8ERJCXDNNTQTuYa2KkUF0doKzJjRd/kVbkFbVUXh7s32b0RAmUxUHsaPa7c7FcW8eVQqhilJ459VqzhbGTmy7+zHbOb9smYNTYtDGL8KQURMIvIVEXlDROoA7AZQLSI7ROTXIqJLTUaB9va+ZSLciYujmdyYtfeXUaPo7D161Ps2RoTT9On4Iry0dvTJ+GT2txBvaUNmUwXSWw8jv+cwCrrKYYINb2RdjYNSAmtcX5NQdzf4YI4cydmZ1Tqwk/DEmWcCd91Fs1B7OzN+m5tpL54/v6+tra2tfzkPweBP69bW0odRVMQZQVsbx9zaSvPXwoU8n54eYP/+8I51qNDTA7zzjucQZAOTiQ/U2rWRG1cUCMQgthrAuwB+BGC7UsoOACKSA2ARgIdF5BWl1L/CN0yNO7m5nCD6Ugrd3ZQd/hSHP0SAr3yFoaXNzQzAcT9OVRVDTg1/a3Y2J6hVRbNQO3I68mu2QZWVY2+3oDl3IspTpyKjfAtykuo8HvOL0NmUFGqio0fp4As1U6fyNWIEl1PFxZ59FoaWO+200I/BlcJCHstbhFBPj/P9E0/kNbFaKazcS22Ew9Q2FGlp4bVKTPS9XWYmHc5DmEBMRmcqpR5USm0zlAEAKKUalFIvK6UuBfB8+Iao8cQZZ9Cx7IvaWk6CQ+FbHDeOk2mlGE1UVeUMkT96FPja15xliAD64Iy+xzZzAqqLZuLg7MuwJudS7EuZDpvEoc2SiDmZe3sdx/BXuKckDNju5Y/LL+cD78nMYrfzpE8/nQI7nIwYQUFfW+v58+RkCi8j+slspiBzVwZKeXcwaXpjMgV2f9ntsV3fKgT4XSEopXpE5DgARQA+Vkq1GZ+JyLlKqRVKqQEaJTTBctJJ9BHU1nqeODc1UU64CumBcswxrJqwYwdD861WWlVOOqlvPbV585gbVV/vFO4JCcD48YyUjIsDRmfF44T4XegGHahKcbI2dqzLJN1i4caOsL/mZgYH7djhzE879dQQlBwqKGAS2h//SOGfnMyHv7OTdrfFi4Grr45M5M4VVwAPPshchLy83sdMTaVCOOUU78KpvZ2RR7okeGBkZfH3b2lh2J43mpvZ0m8II8qPzVJEvgPgNgC7AMwAcLtS6jXHZ58qpU4K+yi9MHPmTFUa7qiPGKaujmHxdXW8j43JY1MToxG/+10K4GhRXc08ifp6jicxkbLq008pZ8+aUYevfnYXmtOL0NUTh85OKrk5c1yi+yoqgHPPBa64AmvXMlfLbndOftsc05PLLmOLzgHL654eapvSUl7MwkIKX1/25XBQXc244QMHnAX+RCi8Ro1iwlpJSV+zUk8PHZ8330w/iCYw1q0Dnn6aD4ynm6iri8EHv/71kMhJEJFNSqmZfd4PQCF8BuAUpVSbiJQAeAnAP5VSfxCRzUqpE8Mx4EAY7goBoMzasoVBEg0NFLwLFwIzZ/L/0cZiYdj8++9zApadTTN8Vxf9eDmrX8ZJh15Fz8hiTJwcj1GjHBNfpbj8SU0F7rsPH+/JwmOPMYrJ3dRryMDrr6cpbcigFHDoEFcsNhtnsccey/effZYOzqQkmrqUolPcaqV2PP98nYcQDFYrU+1LS+mwN24ypbgyaGgAbrkl/DkoEWIgCmGnUmqKy99poFLYCWCxUmpGqAcbKFohDG6UAnq67ZA3lyPuzWUQwzlqlEydOBG45RbYcvLw/e9TUXhTcl1dzmQ9f77BIYGRUfvee7TBmUwM8YqEn2Oo0tPDiLYVK3hDiaPvd1ERa21NmxbtEYYMbwohkCijGhGZoZTaAgCOlcIFAP4G4IQQj1MzjBABEpJMwCVfAs5ezKVOXR1nvVOmfFG2Yd9uCntfGddJSXyGd+ygT2PII0LzRjRtgkON+HjgS1+iibKsjMvbzEwmqQ2T1VYgCuFaAL2CwJVSVgDXisiTYRmVZviRluY1pLOxMbBdiPjOlfCJUb7i3Xd7e6zPOovKSReJGz4kJNA0NwwJJMqo0sdnuui6JuwEWs5bqX6ai2w2liV47z165o0U8H37WF30lFNYKTAcdcW9jaeqiiaMrKzY7NqmGZIM7UpNmiGBURLbZvMeaWm4wib1J29+xQquDMaN670SyM9n2OdHH1EwX3VVP3YeBDYbldIbb9ADb9iwjz8euOSS3h3lNJowoBWCJqQYUZtr1zJKLz2dfs4ZM/pftDQri5P0DRu8V46orgZOOMF3iW6PdHcDy5czfMmTWUiENuRVq4ALLghfspfNBjz5JJVPQYHzRO12hp4++CBw551UDhpNmAhYIYiIALgawHil1AMiUgxgpFLqk1ANRkTOBfAHAGYAf1FKPRyqfWvCT0MDi5MeOsRo0eRkRuz9+c8U6nfe6b8UkBFpuXYtLTZmM4M7zjyTdejKy5kSYCgXi4XKID+fpTOCZvduZyEmb8TFMSxx+3ZqpnDwwQfUeO5x8CYTT66tDXj8cWYGftFyTqMJLcGsEJ4AYAewGMADAFoBvAxgVigG4qig+jiAswBUAtgoIsuUUjtDsX9NeOnuZpKc0bPelZwcKotf/YoTXW95PTYbS8+/9x7N9UZ4/fLlwOuv02py4onMXzDKYsTHs1PkOef4TjL1SkdHYNuJ0IwTDux2mony871Hs6SlOfsx6IQzTZgIRiHMUUqdJCKbAUAp1SgiAyyb1ovZAPYrpT4HABH5L4ClYL6DJsbZsoWl5L217czJYfLY++8zss8Tr75KYe9uyk9P50rghReA227jKuTIESqL3NwB5h0E2v0snLWBjh7lCflbPqWlARs3aoWgCRvBxNL1OGbxCgBEJA9cMYSKIgCHXP6udLzXCxG5WURKRaS03pgmaqLOu+/2rYLqjtEj3lMuZGsr8OabzDXwZMpPSOD3X3yRFhyjrfKAk9COPZbLDF99Dmw2DmrKFO/bDASjGbY/zOaB1zLXaHwQzArhjwBeAZAvIj8HcBmAe0M4Fk9PRB/RoZR6CsBTADOVQ3h8zQCoq3OatltaaOs/dIiyNC0NmDCBQryjg+Yldwfzli20nPjqUJiWxtJGn38ewrptyclMRHr1Vc+1gQynxqJFdISEg8xMHtfI1PZGWxsd3MFgs9FHEhcX2hTu7m76VKqqqFAnTuSPHMoErs5O3kzx8b17XWvCRkAKweFQXgdgE4AzQOF9kVJqVwjHUgnA9W4fDaAqhPvXhJGkJE5eq6oo3E0mZ2Mvi4Xh/Hv20BzkKZz/yJHAKguL0FEdUr70JTolPvyQjoicHB6osZGvk04CrrwyxAd1ISWFSXnr1nkX+EY/hkDNRU1N9MyvXOn0k0ybRmfL5Mn9F65K8Tr95z/cr9HiTimO/ZZbGLE1EGpqWELiww+5X7ud5rTzzwdmzdKKIYwEpBCUUkpEXlVKnQx2TAsHGwFMEpFxAA4DuBLAV8J0LE2ImT8f+Pvf2aQrPb33RDcxka/aWu85VikpnMwGwkAnuobJ6gu5EhcHfP3rwNy5tGnt3s2NJkwArr2WMbPhbq5+3nms693Q0PciufZjCETYVlUBv/wlZ9cFBc4eqEai3SWXAEuX9k+wrlvHKqyFhbThGShFX8gvfgHcd1//q8MeOMDoA5uN+4iLcxaYe/RRNtq45hqtFMJEMHf5RyIySym1MRwDUUpZReRbAN4Gw07/ppTaEY5jaULPqacygsjoNOiOzeZUDLt3s0mZK8cfTwtBbS33kZnZt9ObxcLVxYQJwY+vsxP45BPK++pqjnH2bFZHHT8ezuJw06d70BgRID+/dz+GpCQO0ujHsGhRYIKwpwf4/e/5HdfiT0b4ak4O8PLLnHEHW/Spo4NhYEVFfW1+IvTwV1cDL70EfOtbwe0boGnrD3+gGc/VPGeU/c7I4Ipn0iQqb03ICUYhLAJwi4hUAGgHzUZKKRWyEoBKqTcBvBmq/Wkih8nECd3Bg3QQG+YipShHLBbK2tRUhty7KoSKCuD557m6aG3lakGE8mzyZCoGpRjFdMEFgQcGGTQ2Mny/spJVKYqLKS9LSzmWK65w66UQbkVgt3PGqxSFnKFBi4uBhx8Gdu7k4Do7KXznzg18xr19Ox063sK94uKoFF5/nTG8wZzrp59S4fjKMCwo4HaeVjr+2LKFqxpvYzeZ+AMuX86mGXqVEHKCUQjnhW0UgxBDQLW3U4AVFQ3v+mcdHZwgjh9PwV7pUgGroIA+x9xcmrZdi9Xt30/rRnw8Z+vr11MmpKSw4OTRo1QkTU2UX0uXBjcuu52Whvr63pUf4uKY1dzTA/z3v/z/ieHu7NHdTQ30xhvO/qfp6bTrL1zIk46Lo62/v6WW16/v277OnawsauGGBmfdpkAoK/NvrzOZKKjr6oJXCB9/7L+JR2YmZx1Hj/KG0oSUgBWCUqoinAMZLCjFydsrr3B1bLRjzc+nsDrllOExcWlqYmLt1q3O3i2trVSMJ59MId7TQ5+jq+mnu9tpDTB6kqSmOt+bP5+yav9+Xsfqak6if/hDXttg68vt20c55q10thHAsmwZXQU+fzuLhbPfFSsYeRQXx05Eixd777Rl0NnJBIpdu3qXpujo4PJowwae5EBzHdra+tra3BHhjdvVFdy+jSVfIPTnIejs9P8DG2PX4bdhIZjSFT/29L5S6oHQDSf2efNNzihzc/lMG/d9Swvwpz8xQOLii4e2UvjwQzqQbTZnxGRZGf2BjY0U3HFxnn0JXV3OKtc7d3J7V2GdkEAT8YQJlL89PbSunHxy/4qNfvSR/+9lZdFsX1/f20/ai9ZWCvQDB6hBioo4Eygt5QW5+GJGK3n74V98kdrJXXGkpHDpUlnJi/qd7wR/kq7k5zMu11eIrN3OV7At9Y47jpmDvjByKor6pBD5Z9Qojt1XQosReZCQwG3tdjrN/SXBaAIiGJNRu8v/kwBcAPZZHjaUlzNbtri4r5DJyOCz/dprdJAec0xUhhh2tm1jDbbCwt6mZMMUvnIl5cz06X2/W1XFa7R/P5XB9u18nj1hMnH/SUlUGocOUR4FS1OT/6J6xqTTaxULpXjSFRW97U4mE4WY1UpH7ciRtG2709rK6JyiIu8Ko7CQEUB1dT60UgDMm8dwU6MPsyfq67kcClaInnACf9zWVu8rmepqLvP60791/nxg9WrfY6+s5MP3f//nVD52O/0sS5dy9aXpN8GYjH7r+reI/AbAspCPKIZZvZoTE28zzrg4OjzffXdoKgSluDoaMcKzkB05ktFGH39MBTF6NK9JWxv9LZWVfG/ZMr6/ezejijo7Key9+WCMZ74/5OSw+qq/87LbfZjeDx6k9vJmd4qL45LxlVcYJ+9+Ivv2cWbrK3TVsD3u3j0whTBxIi/m3r0U2hUVzsJPubn8kbq76Z0PloQE4BvfoIe+p6d3spjRwyEvj6ul/jBuHBXV1q29l98GR47ws4kT+TJMYzYbS3ps20ZFoVuI9puBuEFTAAyr/n2bNvF+90VeHs3M/kytra3AZ58xsOLQocBNs9Hk4EGnTd8bEyfSOTtqFP1+Bw/yebXbKaeOPZbPemEhzUKJiUxY8ya0DeuGv+vujblzKf98Xd/GRlpyvB6jtJTOEF92wIwMzu4PH/Z8gOpqYM0a+h/WruWFcbeDm0zUjgPBZKLQ7uhgNM7evZxJWyz0X7z3Hm1y3iJ5/DF1KvCjH1EZVFTwPA4e5HnPns3P+mu+EWFi2/Tp3Hd1NW2xDQ38e9cuKo0pU3r7Scxmrr6UolNqMDxMMUowPoTP4CwlYQaQB+DBcAwqVunp8R9JZDI5e8R7kh9tbQzT/uADCjqj8cu4cey/Essri6YmZxCJL7KyqBSuuILX4aWXmHjqnlM1apRzVXXgAGWUuyWivp4muP4qhAkTeE3LyjzndFksPK+bbvJxXo2NgWXDifS1O1VU0Ddw4ACXK93d3F9FBZXIokVOAapUaLqjbdhAIXnGGTxOayvHP2kSNfHu3fxBzj+/f/s/5hjg/vupCOrreaySEu9lbIMhORm4/Xb6B9ato6JJSmLU1b//7bt0R24ux3TgQAhrmwwvgvEhuK4xrQBqHb2Vhw1jxviPpmtp4TPnSXG0tzMJs7KS2xgWBNckzzvvpKk2FjHyAfxhtTrNLxYLe8t4CqNPSKCw37KFsvTgwd75CU1NVJZf/nL/x2wyMUfqd7+jUsjOptKx2SjLrFYmI/uM8szOpiD3h1K97U6NjcCvf83vJyRQOLvS0sIIo4suohY1LshA6O6mI6u4mErAk/kkIYF2u8WLg0/qMDASRbyZ0QaCCDW5awbiZ59R8fgyu4nwtWuXVgj9JBiT0TeVUhWO12FHZvEvwzayGOSccyikfAnFhgYmOXli+XKah4qLe9/XIlQyI0awmUwgsicaGHWIfBUGNezxhlxrauLKylskpGE2ttk4sauu5qSwooLP/913B1/PzZ3MTJqWv/EN/r+2lrJ4/nzggQfYfMcnM2c6l33eaGmh7d81uubDD7li6Oriy2bjhTC85WlpvDgrV3LWvnRp/wW0wa5d/hv+JCTwR9w5iCrLB1rXRFeEHRDBrBDOAnCX23vneXhvyDJjBm3gBw5QSLmaGJSiT62khPLDnc5Omm99tXhMS3P6zWbPDvnwB0xSEkvJLFvG8/RkYqmt5cTOMFH7C10Xof0+LY3bzZ9PZTllCrOU/RW86+zkNROhPPameBIT6U/oV8WD4mIu23bupN3J/cStVg7ittt6Lw1XrqSmX7uW0S8pKU4HrzEjMIroJSd7n0kEQ2trYNspFfi2scCIEc4ier5sllZrP/qoagz8KgQR+QaAbwIYLyLbXD5KB7A+XAOLReLjad586ikGNMTFUQB1d3MCM3kyZ6GeInCqqnzPlA2Sk+lgjUWFAAAXXsjZ++bNlHFGdGF3N3MwcnJ4DYxnNieH1pD2dt+dH7u6WFD07LMDG0dTE83gq1c7J+9JScBZZ3El5y9ZNyhEgJtvZo2g/ftpAsrI4IGPHOHJX3pp7x/NbneWbu7p4cmPGEF7VXMzLwjgrAQYH09tOlBhFkzlv1CWww43o0dzluGrJEZ3N6+jp5hnTUAEskL4D4C3APwCwN0u77cqpRrCMqoYJi0N+O53afopLaXtPyuL0YZjx3qfvPib2BgYTuZYJSEB+Pa32fnszTdp9xfhc3jhhfRjugaZmM30Xf7jHzQPeboGXV2cWHsK4ffEkSP0tzQ00ERuhAF3ddF8vmUL8IMf9C8U3ivp6cBdd1ETvvWW/0xlEWqo+vreq4aEBHrIDS95Tw+Vh9kcGoVgxO/66q1gNPyZPHlgx4okIsBXvsIfPiGh749rsdA5d+21IZ4NDC/8KgSlVDOAZgBXiUg2gElgYhpEBEqpdeEdYuwhQiuCv46HruTlUSnYbL7NIJ2dvXOfYpH4eMrAhQtp7bDbPVcnNTj9dMrR7dspwI3tjKrGDQ3ArbcGFq2oFPD004zWcvdnJiVxEnnoEH21N944kLP0QEICtVYgmksEWLAA+NvffNvM2tspmAOdMfigqQnYuDEDSeYFKHr/PaRPHYvcPOm9W6UoOOfPD1/Dn3BxzDGcjT35JGdiSUm8Zp2dVHDXXBOAQ0jji2DCTm8CcDvYuGYLgLkANgBYHJ6hDS0yM2lR+PRT73kzFgvv61mzIju2/mIUn/RHfDwrMixbxsoHhs9PKfpgb7gh8FpulZXMW/AV3FJYyBpvl10W5YoGCxeyZEV1tefPLRbODgoLqRX76T232ZgT98Yb/DvV/GUs6KxF4dufYXdOLqafls7ckdZWLq+mTGGM82Bk2jSGjG3ZwhmGzUan1axZvhNkNAERjFP5dgCzAHyklFokIscB+Gl4hjU0ufhiRs8dOdK3UGN3N4XdNdcMzfs6IYEC+oILGP5psXCC6ikh1Rd79/JfX98xm6lsDhwIvuR/SBk1imFSN9zAHz0nx5moYvgQ5syhr2HWrH7nILz4IpXB2LGGlSgJu0bdgZbDH6No65vYseIgZswAkscWMOFi7lz/zqxYJjEx8JWaJiiCUQhdSqkuEYGIJCqldovIsWEb2RCkoIDhj088wbpIJhOFl9XKWfS118b+itdqZbDNhg2UaXl5LJ/jzT/gTlLSwEzX3d2BlRlXimONOrNns+rhPffQzBEf70zkGjuWtq+MjH636KyrY9OfkpLepki7OR6Hik/DoTHzUFveif2nAl+9OXloV13UDJhgFEKliGQBeBXAOyLSCN3zOGiKithZ7MABhp5bLLQYTJ8e+76w6moW/KyrYzRUfDzD3t99lyv5W2/1HUkUCvLzA69rFIrE2ZAwfz696s8+y7pGcXHUak1NLON61VX9Xh2sX08Z79UvJYIRY1KwdiNw8dX0W5eW0nqUk0N95avmnmZ4IaofdT9EZAGATAArlFI+0pTCy8yZM1VpaWm0Dj+saGoCfvITzrrdy0goRUfusccC3/++/9yBgdDVxdDfnBzvUZOtrRzDww/HWNMio6tSXZ0zMiGYBjUeeOQRmuD86ZMDB2jBqqvrGy594olsKR3rExJN6BCRTUqpPhlTAT8uQq4RkR8rpdaCjuUZoRykJnZZs4aC1lNNIRH6Q3fu5KonlHR3c1K9cydXKImJwOWX09/iKSG1q4uRnldeGWPKAOCFGj2ajo0TTxywMgC4SvO3YurpYd7M4cO0Uo0ezZXWmDH8e+tWdpWLCRObJqoEYzJ6AoAdjCp6AEArgJdBR7NmCGOzMTrIV1VmEZqLVq3qXY+ov1gszHNYuZJC3iiBPWECaxtdeSVbEADOLOe2Ns5+b701ys7kCDJjBqvw+uomuX8/r+eECX1NQ4Yy37GDL53TNbwJRiHMUUqdJCKbAUAp1SgigzhUQRMoHR0M9fZXcTQ93XP152CxWIA//pERWYWFTkWkFG3gDz3EcPTf/IbObaPd5vHHM1hnoF0oPdLTwzDHTZu4bCkqYmu4KDdkOekk4F//8p4Jbrdz2JMn++43kZFB5asVwvAmGIXQIyJmOEpgi0geuGLQDHHi4iiMAykjEwo79Jo1NHG4Ry6JOJvzPPEE7ef9reAcFGVl1FCNjTx4XBwVw6uvMjX7qqt8V+EMIykpwDe/yWvR2cnrY1wzI5Q5Odn/qi0jg1nnmuFNMHfxHwG8AiBfRH4O4DIA94ZlVJqYIjmZFREOHfJtmmhsZAbzQLDZaCoaOdK78klNZVh/aSmzoMNKdTW900YatCt2O6fVSjFmOEpMm8ZwZqNts1FQMCHB2ebZXxSR14ZubW3AJ5/wZbHQAbFgQd9SHZohQSDF7f6plPoqgFwAPwRwBgABcJFSalj1VB7OnHceTTTZ2Z6jiDo6+P4ppwzsOHV1zNPyVxYkLY3lMMKuEF5/ndLVUwyrycRlzHvvsapeFKtsTppEpVBTQ8UcF+dsiVBZSUXhywd09CgXO7347DPg8ce51DCaZh8+zFpOqal82e0873POoUNjMBXM0/QhkDiMk0VkLIAbANQCeA4seFcrIiFo76QZDJxwArOMy8tZf8iIVrbbKcTr6tj9cKANv2y2wCaeJlMEyt63trJBtC8/gZFduD42Cv+OHEl/waRJTtl89tmc6Hsrmmix8LNeyrWsjHaotDSGIhmdhSwWpot/+CG1TE4OtckTT9C509wc9nPUhI9AFMKfAawAcByATQBKHS/j/wNGRC4XkR0iYhcRD90ENNFGhOGe3/oWJ4aHDnHmeeiQs07T3r2cLA9EJhjVHfyFQLa10WoRVhob+a+/xIrU1Jg2wE+ZAixZQmXu2gJBKeaXHDrEQqKuvX3w2mvUKK5VRevrWYwrPZ3e/ro6Lg2zs7lSqqpi/GqgmYOamCOQaqd/BPBHEfmTUuobYRrHdgCXAHgyTPvXhAARlo+ZPZuyoLKSVUWrq50JT93djHpZsoS1m4JNUktJYWLvunWeeyADnM3abCyZEVYMb7o/jE5oMYoI+1sXFbHAYEUFla7dzmt83XVMi/iCxkZ69d1/gD17qCQMZ4PJxJvAqCBYVMQMuH37mKWoGXQE4kMQRbwqA2Ob/g7C8EWIdlINCkQo//75TyoA9z4QVisnmBYLZ57BsmQJsHGj5yKANhtnukuWRCDis6CAws5fd5/2dpagiGFEqGjnzaMM7+riKRUWejDRtbTwTdc4VaM1nWv52Pj4vl3XEhJoPvOnELq6uNr48EMu9/LyWB02kDZ5mrARSJTRahF5GcBrSqkv1sWOHITTAFwHYDWAZ8IyQk1M8vbb3p2/cXEMyFm5Eli0KHhfa24u8KMfAY89RuFv1IPr7qac+tKXqBA2bGAdpSNHuLI4ctdV9wAAHlpJREFU/XQ6tUNW5t9s5oGefdZ7VE1LCyXrjMGRtG8yBdDHIz6+78rIqM3ueg1sNmd3IoOkJPoUfFFRQf9EUxOd1QkJNEeVltL0dPvtg69XwxAhEIVwLuhQfk5ExgFoAhvkmAGsBPCIUmqLv52IyLsARnr46B6l1GuBDlhEbgZwMwAUB9OhRhMyurrYunKkp1/TgdnM1wcf0PcQLIWFwM9+Rr/E1q08ZmEhG5RZrSwQWFNDuZGSQgfzCy9wZfKd79BuHhIWLWIFv40bqdmMRAvDm261sj2bp76pg5WRI6mVW1udWX6GknBNRrFa3RwPoNb21YTi6FHgV79yzhoMDF9FZSVbld5zT19lowk7gfgQusCyFU+ISDwYftqplGoK5kBKqZAUdlZKPQXgKYDF7UKxT01wNDZSFvgzm6enA59/3v/jmEzMfzjuOOd73d0sstfU1FueGP7Plhbg/vuBr36VFp9x4/xnWPskLo6ZX+++y3DLI0ecBvgZM4ClS3136xmMmEwMKXv6aa5+TCYmo2Rl0YmcnEwTUkpK34vb3e3bubNmDbW7t2ZARUVcFm7f7ubY0ESCoNIrlVI9ALy0f9IMFwI18RqtgkPJtm1cGbjniAGcsG/bxn9rali7B6BcueaaAYTExsUB557LZhVVVdSG2dkRr69dU0N9ZDbT7BPWUuOnnUZtvmoV058zM+kX+OADCnSjSY3rD1xTQ0e0N/+BzUbF6ishAuBM4t13tUKIAtHJt3dDRC4G8CiAPABviMgWpdQ54TiWxcLKmc3NvKePOWbgsfPDDUM++PO1traG/pl+7z3PHeVqauhTSE7myqC9nbJJhPlVDz0E3HdtBTI3r6H5p6eHs9Szz+ZMP5AoISPbK8J8/jnNYXv2OLOQzWb6YC+6KEyKwWRi9vXkycDy5QyrNSrhtbdTIxtVBdvbqany82n/9zYL6OzkCsJf8lpKCotWaSJOTCgEpdQrYFmMMB6Dk53//Y+rXsBpCp07lzPIcDd3GSqYzcxc/uc/vXdK6+yk/Jw9O7THPnqUQt8Vq5WlhVJTnXLdbuf7iYlAUaFC5kcrUPuNF5A5JY5mDpOJjszHH2cW1+23h6kq3sDYs4cm96Sk3u1Ge3o4id63D/jhD8PUy8BkcsYZNzXxgmZk0M6/ahXLWVit9Ddcey239fUQJSTwQbTbfdcmD1VRLE3Q9EshiMhIpVRNqAcTTl5/nbVeiop6r1htNt7XNTV8sNyFjcYzCxfS2WtUJDUmfd3ddAQfPsyon48+ojwJVbP79PS+SqG2lgLSkEVGXpQxUR1V/SnmVP0H+8zFGJsbj0RjMZCdTbt4WRnw5JPA974XU/V5LBbqq8zMvqui+Hi6LsrLeW9fcYWfnXV0UGtu3+6sIz5nTmBmL5He202YwNfXvx6cXTAhgUvGXbt8xww3NLAUhibi9LeFyJshHUWYqakBXnmFD5B7MIjZzFXwgQNMhtIERnw8o3mWLqWT+eBBypuXX+a1HDeOMui554A772Q/hf5nqjiZP5+TVVeOHu0tk9rbqaTi4gAoheN2v4LO1BGwSnyfsPkvmtZs385wyBhi+3Y6yT2ZyAwKC2lG6+rysaNNm1gv/G9/40537WJG4fe+x0qC/f1hfPbu9MI55/DG8FZ3pK3N6Z/QRJz+KoTYmUYFwIcfcoXqq0LxyJEMIvFW70XTl4QE4JJLGCV4wQV874wzqCSOPZaWhOJiTgaffTY0Ctfod+CqFNwjIa1WYOJE/p3eVo30lsPoTvSxRBHhzbFx48AHGEK2bfMfzZqQwPOtrPSywfbtLN2dmckZUV4ef5ixYxlG+9xztD1FimOPZXejQ4dosjOUkc1Gh31jI/Dtb4duSakJiv4qhKdDOoows2OH//srJYWzMV2bK3gSEujQnTSJwt/d6pKQQFPd88/TpOSNo0c5mf3kE5pCPE1cU1O54uju5qqku5vWDIuFTuy2NiYNGxaOeEs7YDLDrjgo19I8vUhMpKkihrDZAmsDanST64NSFPjZ2Z5t8vHxXB29+CKdPpHivPOAu+/mMvLgQSqHqiquCn7609C03NP0i/46lf8X0lFoBjX79zPIxFc4flISTXc7dvRtb9nQAPznP1QGIs78pzFjmE9wzDG9tx83jklra9dyctvTQ4VQWMicBdckV2t8MmC3o61VoahIvM+4/SVURYGSEkZ5+sJu58tjrkV5ubORsjcSE3nxtm5ldEWkmDKFr7Y2Zx0N7cCLOsPChzB1qv+Zf0cHbbUxJhMGBYFOrEVoJXClsZEhoVu3UgEUF1N+jR3LFdvDD1OJuDNiBM1Vjz3G/Kmnn2awgHtya3NaEeqkAGloxeTJXgamFO0uoQ6JGiCzZvkv811b23tF1IuGhr7lJjwRF0dtHQ3S0mjC0sogJhgWPoR585xhiN6oqeFKVtfVCp5Au0fa7X3D/V96if6AoqLe5hEjsCU7mwFA3oSiycTVx/z5wG23cbJZXu58VRwUHJ59EeZOqEdqopcboKqKy5Bx4wI7kQiRkQFceimtKhZL388bGnidLr7Yyw4C/WFsNt3YRgOg/yajQeVDGDmSD40RdupqNrDZuKqeODEC3beGKBMmUDDZbN4VquEPmDTJ+V5zM30P7uVwXElPp2DfscN//bg5c2iO2rWLM+e4OI5tzOjZkGWXMtQsKYlLCZOJToejR7ksue22mAo5NTjvPA715ZepFBMTeZ2tVp7G975HU5lHxo93NpfwphwM+5xrfRDNsKVfCkEp9USoBxJuLryQZsqXX6awMASUCCtkXn21XrX2l+xsCuNPPvFeoqa2lkmvrsL/8OHAIhfj45mgFUhB0fh49hjujTD0aepUJlSVllKqjhoF3HQTbTMxWpxOhFUzTjuNPpbDh7nKmjKFATs+r116Or+4bp33DOu6Oq6MPNUC0Qw7YiJTORKIMCRy/nzOIJub+WDp0hWh4aqraNooL+eM1TAN9fSwgU5WFmWv6yQ80PB3r1E0wSDC5cmkSc5ZcSAhPDFCWhp72wfN5Zcz8a68nArQMA1Zrfxh0tKAW2+NydWRJvIMG4VgkJAATJ8e7VEMPTIy2MPgrbcY+WP4a0wmYPFi4Pzz+zo+CwqcUTK+ZLPFEuJ2mSLDRwCmpgJ33QWsWMEfpqvLef6nn84EkhEjoj1KTYwQSMe0QObP9mDLYWsCQylnFnVlJRXanDm0cniNqY8SaWmckF54odMsl5fnvbxNbi7NQDt3em+i09lJa45W4gMgJYUhWeedx25mGzdSy7a3O1tgBuqA1gxpxF/nSxHpAlAF35FFZqVUxMtAzpw5U5WWlkb6sBGjq4vhlKWlFIppaZx5t7Rw5X/bbcAJJ0R7lAOjupo5BQAVhOvEvaOD0V/f+Ab9PJoB0NTEjOXPP+fNlJREpdDRwaiL7343Aj1Je1NbywAvo3qIe7tUTfgQkU1KqZl93g9AIWxWSvksYhzINuFgKCsEpYA//Qn4+GP6+9wtHG1tDDv88Y8Hvz+wspKhpYcO8W+j/0x6OqvQ6rI2A8RiAX7+c0pfTyFddXVUED/9aUQqvh4+zETEHTucpkKluFq86ir/7RI0A8ebQghknRjI3EzP30LMoUOM2vGkDACuFtrbgWXLWGRuMDN6NPDAA5y8HjjgDACaOlV3UQwJW7awcJ+3mUN+Pj9fvz7sVUYrK6mbACYfGve23U7T4YMPAvfeG/HFisaB3zALRwtNj4jI1/xto+kf69dTGPryfebl8Vl3r/45GBFhzsDZZ9PUPWOGVgYh4513/Kfg5+UxIiAUJWm9oBRNoCZT35pXJhMnAT09LISoiQ4Djbv7aUhGoelDdbX/vAijKoEuyKfxiRFe6ouUFM4sfNXJGCDl5VyI+PIVFBTQlFStG/VGhUCijLZ5+wiAXtiFiZQU36U2DDyVg9BoepGYyJvJ15LLSDMPY7RReTn/9bXqNSJiDx70Hnk2LFGKduT16+mNT0lh7a2pU0P6mwWypwIA5wBodHtfAKwP2Ug0vZg9m2UdfNHaSvOvtrdqfHLqqWyE4y2NHKBj2aimFyaCsUYNOBFxKNHZCfzlL0xVj4+n6aCnh41e8vKAO+7wXf8lCAL59ZcDSFNKVbi9ygGsCckoNH04/njmC3mrJGq38xm+4IJBlXCriQbz5/Nfbz0PenoY43zmmWEdxsiR/vMBjSRyPclxYLczBG/TJnrhi4pYWqGggEECnZ0sCXz0aEgOF4hT+UallMeq7Eqpr4RkFJo+xMdT8dtsXCkapl2laOotK2MpjnnzojtOzSAgP5/9j2trOYswpt9KUZAcPMguZkabuTBx7LHMVu/TxtSFpiZGncVY4dnosX8/sHlz75AsV3JzGW743nshOZyeW8YwY8YA999PwV9fT8Vw8CADRr71LeDaa/XqQBMgc+cC99zDWk6HDjlvplGjgB/+kKFdYcZsBr72NTZTamvr+3lLC5XFddcNn8oiflmzhjkivi7IyJEs2hiCgIBAEtM+VUqdNNBtwsFQTkxzp7ubD0x8PBWCfmA0/aa5mbPK5GQvnXXCy9atwF//yvvZqNZqs9EScvPNuhJ3L+67j2Yhf1FiBw8Cv/1twL/nQBLTJvuINALoXNZ9xsJMYqKXNomaoU17O7B9O1vLJSezhvhAU3kzM6PaGnD6dMqunTsZeWQy0UQ0ebJuUNWHhATPyylXDMdLCKKNAtlDIPraNtCBaDQaF+x24PXXgeXLaQow6nmIADNn0q4Sa9UNgyA+nopBFy30w5w5wH//61uBNzVRo4bgfvCrEJRSFa5/i8gDAMwAtgDYopTaN+BRaDQaJ0pRCLz1FhvbuOYP2O3Ap5/SqXTXXbqr01Bnzhz2mW1v91w22G5nKOJXvxoSO3LQLkml1I8B/BFAK4BLRWTA7TRF5NcisltEtonIKyKSNdB9ajSDlkOHgJUrGVbonkxmMjHaoKwM+MBj8J9mKJGZCXzzm/TEu0aIAfTAl5UBZ50FnHxySA4XsEIQkd+LUAUppWqVUiuUUg8rpb4egnG8A+B4pdQ0AHsB/CgE+9RoBidr1tAe7Mugnp/PFYTO4Br6zJjBin/uEWLx8cAtt7AkcIiiTILxQrQBWCYiVyql2kXkbAA/UUoNOBJeKbXS5c+PAFw20H1qNIOWPXv8R4ukplIotLWxXZ1maDNhAntWNDYyPCshgclpIY47D1ghKKXuFZGvAFgjIt0A2gHcHdLRkBsAPB+G/Wo0gwORwOo8KKXjj4cb2dlhDRUOxmR0BoCvg4ogD8B3lFLvB/H9d0Vku4fXUpdt7gFgBfBvH/u5WURKRaS0vr4+0MNrNIOHE07gTNAXLS2cIQ7iSCNN7BGMyegeAPcppT4QkRMAPC8idyqlAsqZVkr5LJQiItcBuADAGcpHtpxS6ikATwFMTAt49BrNYOH004EVKxhu6qlCqVJ0Mt5449BfIbS3A9u2scSGkYdRWBjtUQ1ZgjEZLXb5/2cich6AlwGcOtBBiMi5AO4CsEAp1THQ/Wk0g5pRo4BLLwWef57/T0lxftbTwx6U06axiulQxW5nDsbrr/OczWamMwNcQd14I5ClgxFDjd/SFT6/LJKslPJSQjGo/ewHkAjAKNn3kVLqVn/fG06lKzTDDKWAdeuA//2P5iHDr2A2A4sXU2EkJkZ7lOHjhReoDNzzMJRib+i8POD//k+bzPrJQEpXeCUUysCxn/CWWdRoBhsiwIIFLGe7dy+VQmIicMwxnhOUhhKHD7N/Q0lJ39BbEZaALitjeO4FF0RjhEOW8LVH0mg0AycuDpgyJdqjiCzvv+8/D2PkSPpZzjlHN98OIbp4skajiS327fOfW5GcDHR0cOWkCRl6haDR+KGzk2ZrpZggrPPAwkygeRjGtpqQoRWCRuOFtjb6NVevZo96Q07NnQtcdNHAq1BrvDBtGvDqq74rfLa1MUErimW8hyLaZKTReKC1FfjFL1hjLjeXwS5jxtCfuXEj8OCDQHV1tEc5RDHCaS0Wz58rxUJvS5boBgohRisEjcYDL7wA1NSwlW1CgvN9s5lKwWYD/vznwC0bmiDIzQWuvhqorOzbHKanhzWcpk4F5s+PzviGMNpkpNG40dwMrF/vOyE2Lw+oqGD04/jxkRvbsGHxYobXvvACL7ThKzCbgTPPZB6Gq6bWhAStEDQaN8rKnDlg3hDha8+eYaQQOjuBzz/nLD0zk3kC4XLqitBZM2sWcOAAtXR8PDBxok5GCyNaIWiihs3G6r2xFihiC7AhrMnk3cw9pLBYgGXLgHfeoTIAqDELCoArrgBOPDF8xzabmYyniQhaIWgiSkMDG3298w4dt0lJTMhduJBle2KBESNYSsdfdWmbLXbGHDZ6eoBHH2WBuaKi3maalhbgkUeAm25iQT7NoEcrBE3EKCsDfvMbWh7y8yl4LRZg1Srg3XeB224DTjop2qOkI3nMGPYu91Z6vrublSSmTYvs2CLO++8DW7bQLuauHTMyqCCefZYF58JYp18TGXSUkSYitLYCv/sdzcDFxVwZAJQno0dTOTz+OMvYRBsR4Ctfodm6vb3v5xYLA2CuvNJ5HkMSu501hQoKvC+VkpK43YYNkR2bJixohaCJCJ98wghCbxWLU1Jok1+1KrLj8sbkyexY2NYGlJcDtbUMfa+o4L/XXUcz15CmsZE2Pn9O3MxMriI0gx5tMtJEhNWrgZwc39sUFNBCcfXVsZFvNH06TeRbtgA7dtCnMH48A1/S06M9uggQaItOkcA98ZqYRisETURobe3d58UTcXEsEWGxsHZZLJCUxOjHuXOjPZIokJlJR4nhMPFGSwu1pGbQo01GmoiQlQV0dfnexmKhT2Eo930ZVMTHA2edxZRtb9hs1OI6ymhIoBWCJiIsXsyoHV/U1QGLFtGXoIkRzjiDpSSqq/vW6bBa6WA56yzd53iIoB89TUSYNYurhCNHPH/e2kpT9KJFkR2Xxg8ZGcDddzMWt6KCr8OH/7+9+4/1qq7jOP58CcgPUZl4mYok2hil5mCaM3UNJia5pmFlOio3Xc6ZkySbGk1NcqOxtVazZZss28jGJhnTsotkqS1KMFQU8UeTgRKCv/ACegXe/XE+V75cv/fe871wv+ec7309tju+53vP93te4uX7vp/POefzzv7cvDlb9vWyy8p3d6H1ywH1VC6aeypXy2uvwcKF2Uhh7NjsPEFnZ1Ykhg2DOXMGX3OwyojIFpV79tls7q+tLbtD2c0hKqmnnsouCNZUHR3Z8tHLl++7onHatGzF476uQjKzg6OnguCrjKypRo/OpoU8NWRWPj6HYGZmgAuCmZklLghmZga4IJiZWeKCYGZmQEkKgqT5kp6RtEZSuyTf9mhm1mSlKAjAwog4LSKmAA8CtxYdyMxssClFQYiI7TWbhwHVvVvOzKyiSnNjmqQ7gW8B7wK+bcnMrMmaNkKQ9IiktXW+LgaIiHkRMQFYDFzXy/tcLWmVpFVbt25tVnwzs5ZXurWMJJ0APBQRp/a1r9cyMjNrXE9rGZXiHIKkSTWbFwEvFJXFzGywKss5hAWSJgN7gQ3ANQXnMTMbdEpRECLiK0VnMDMb7EoxZWRmZsVzQTAzM8AFwczMEhcEMzMDXBDMzCxxQTAzM8AFwczMEhcEMzMDXBDMzCwpxZ3K1hwffghvvQUSHHUUDPX/fTOr4Y+EQaCjA1asgPZ2eP/97LnRo2HmTJg+HUaMKDafmZWDC0KLe/ddWLAANm+GY46Btrbs+Z074b77YPVqmDsXRo0qNqeZFc/nEFrcokWwbRtMnLj/SGDUKDjxRHjlFViypLB4ZlYiLggtbPNmePppOO64+t+XYPx4ePxx2L69/j5mNni4ILSw9euzP6We9xk6FPbuhZdeak4mMysvF4QW9sEHvReDLlJ2BZKZDW4uCC1s7FjI0zI7AsaMGfg8ZlZuLggt7JRTYPhw6OzseZ8dO7JiMGlSz/uY2eDggtDCRo6EWbNg40bYvfvj3+/shC1b4NJLYciQ5uczs3LxfQgt7oILYNcuWLYMDjkEjjgimyLquqroiivgrLOKzWhm5eCC0OKkbJRwzjnwxBPw4ovZczNmwNlnZ0tYmJmBC8KgMW4cXHJJ0SnMrMx8DsHMzAAXBDMzS1wQzMwMcEEwM7NEkedW1pKStBXYMMCHORrYNsDHGAjO3VxVzQ3Vze7c/XdCRLR1f7LSBaEZJK2KiDOKztEo526uquaG6mZ37oPPU0ZmZga4IJiZWeKC0LdfFx2gn5y7uaqaG6qb3bkPMp9DMDMzwCMEMzNLXBBykDRf0jOS1khql9RDl+JykbRQ0gsp+x8kVaINjqSvSXpO0l5Jpbwao5akmZLWS3pZ0s1F58lL0iJJb0haW3SWvCRNkPSopHXpZ2RO0ZnykjRC0r8lPZ2y/6joTN15yigHSUdExPb0+Hrg5Ii4puBYfZL0BeCvEbFb0k8AIuKmgmP1SdKngb3A3cCNEbGq4Eg9kjQEeBE4H9gEPAlcHhHPFxosB0mfBzqA30bEqUXnyUPSscCxEfGUpMOB1cCXK/L3LeCwiOiQNAx4ApgTESsLjvYRjxBy6CoGyWFAJapoRLRHRFdrnJXA8UXmySsi1kXE+qJz5HQm8HJE/DciOoHfAxcXnCmXiHgMeKvoHI2IiM0R8VR6/B6wDhhfbKp8ItORNoelr1J9lrgg5CTpTkkbgdnArUXn6YcrgT8XHaIFjQc21mxvoiIfUFUnaSIwFfhXsUnykzRE0hrgDWB5RJQquwtCIukRSWvrfF0MEBHzImICsBi4rti0+/SVO+0zD9hNlr0U8uSuCNV5rlS/9bUiSaOB+4HvdhvBl1pE7ImIKWSj9TMllWqqzg1ykoiYkXPX3wEPAbcNYJzc+sot6QrgS8B5UaITRg38fZfdJmBCzfbxwOsFZRkU0vz7/cDiiFhadJ7+iIh3JP0NmAmU5qS+Rwg5SJpUs3kR8EJRWRohaSZwE3BRROwsOk+LehKYJOlESYcClwHLCs7UstKJ2XuAdRHx06LzNEJSW9eVfpJGAjMo2WeJrzLKQdL9wGSyK182ANdExGvFpuqbpJeB4cCb6amVFbk6ahbwC6ANeAdYExEXFJuqZ5IuBH4GDAEWRcSdBUfKRdJ9wDSy1Te3ALdFxD2FhuqDpHOBx4Fnyf49AvwgIv5UXKp8JJ0G3Ev2c3IIsCQi7ig21f5cEMzMDPCUkZmZJS4IZmYGuCCYmVnigmBmZoALgpmZJS4IZmYGuCCYmVnigmAtQdJESbvSwmFdz32sT4GkkamvRaekow/wmCMl/T0tgY2k69M6/Q2tGSVpjKRrDyRLjmN8rPeBpEMlPSbJS9gY4IJgreWVtHBYV5+Cu4AvAicDl0s6OSJ2pX0OxnpDVwJLI2JP2r4WuDAiZjf4PmPSaxuiTN5/w78hWzfnI2m57hXA1xs9trUmFwSrjNQp6/z0+MeSft7L7s3oUzAb+GPK8yvgJGCZpBskfSN1x1oj6e6aUcQDklanjllXp/dZAHwy7bswjXZqf5O/UdLt6fHENAr5JfAUMKGnY9XqpffBA+m/w8wFwSrlNmCepNlk6+Df0Mu+A9qnIC1kd1JEvAqQ1oh6HZgOPEz2W/c5aTSyh30fuldGxOnAGcD1ksYCN5NGNxHx/RyHn0zW5WwqMKqXY+WxFvhsA/tbC/PcoVVGRDyWVrucC0zrmqqRNJ9sBcxaA92n4GiyhffqOQ84HXgyi8tIsoYokBWBWenxBGAS8L8Gj72hpu1ib8fqU0TsSedTDk8dyGwQc0GwypD0GeBYYFvXh5ekY6j/c9xwnwJJ3wG+nTYvBGbVbkdE7et3ASN6eivg3oi4pdv7TyNb8vhzEbEzrYdf7z12s//ovfs+O/o6VoOGA+8fwOutRXjKyCpBWXP1xWTnAXZI6loOeyqwps5LGu5TEBF3pWmbKRHxevftbvu+DQyRVO8DfQXwVUnjUvajJJ0AHAm8nYrBp4Cz0v7vAYfXvH4LME7SWEnDyRoc9aSnY+WSpqy2RsSHeV9jrcsFwUpP0ihgKfC9iFgHzAduT9+eQp2CEBG7yVqd/oWsEfuSiHjuIEdrB86tc+zngR8C7ZKeAZaTjWweBoam5+YDK9P+bwL/SC1EF6YP5zvIegU/SC9NVHo51n5S74N/ApMlbZJ0VfrWdKD0vQSsOdwPwSpN0j1k0zqfAB6MiFw9aiW9CpwREdsO4NhTgbkR8c3+vkfRJC0FbomI9UVnseJ5hGCVFhFXRcResqtrjqy9Ma2erhvTgGHs67jV32P/B3i03mWeVZCm0h5wMbAuHiGYmRngEYKZmSUuCGZmBrggmJlZ4oJgZmaAC4KZmSUuCGZmBrggmJlZ4oJgZmYA/B84qOy0PfvCMwAAAABJRU5ErkJggg==", - "text/plain": [ - "
" - ] - }, - "metadata": { - "needs_background": "light" - } - } + "plot_dataset('Scatterplot of the training data', train_x, train_labels)\r\n", + "plt.show()" ], + "outputs": [], "metadata": { "scrolled": false, "slideshow": { @@ -189,39 +134,26 @@ }, { "cell_type": "code", - "execution_count": 6, + "execution_count": null, "source": [ "print(train_x[:5])\r\n", "print(train_labels[:5])" ], - "outputs": [ - { - "output_type": "stream", - "name": "stdout", - "text": [ - "[[ 1.3382818 -0.98613256]\n", - " [ 0.5128146 0.43299454]\n", - " [-0.4473693 -0.2680512 ]\n", - " [-0.9865851 -0.28692 ]\n", - " [-1.0693829 0.41718036]]\n", - "[1 1 0 0 0]\n" - ] - } - ], + "outputs": [], "metadata": {} }, { "cell_type": "markdown", "source": [ - "## Подход\n", - "\n", - " * Задаём функцию потерь (loss function) $\\mathcal{L}$\n", - " * Определяем модель $f_{\\theta}$ с параметрами $\\theta$\n", - " * Подстраиваем $\\theta$ для минимизации $\\mathcal{L_{\\theta}}$ на обучающей выборке\n", - "$\\theta = \\mathrm{argmin}_\\theta \\mathcal{L_\\theta}(X,Y)$\n", - " * Проверяем качество модели на тестовой выборке\n", - "\n", - "Результат: $f_{\\theta}$, которая делает предсказания на новых данных: $\\hat{Y} = f_{\\theta}(X_{new})$" + "## Machine Learning Problem\r\n", + "\r\n", + "Suppose we have input dataset $\\langle X,Y\\rangle$, where $X$ is a set of features, and $Y$ - corresponding labels. For regression problem, $y_i\\in\\mathbb{R}$, and for classification is is represented by a class number $y_i\\in\\{0,\\dots,n\\}$. \r\n", + "\r\n", + "Any machine learning model can be represented by function $f_\\theta(x)$, where $\\theta$ is a set of **parameters**. Our goal is to find such parameters $\\theta$ that our model fits the dataset in the best way. The criteria is defined by **loss function** $\\mathcal{L}$:\r\n", + "\r\n", + "$$\r\n", + "\\theta = \\mathrm{argmin}_\\theta \\mathcal{L}(f_\\theta(X),Y)\r\n", + "$$" ], "metadata": { "slideshow": { @@ -229,19 +161,19 @@ } } }, + { + "cell_type": "markdown", + "source": [], + "metadata": {} + }, { "cell_type": "markdown", "source": [ - "## Функции потерь\n", - "\n", - "* Определяют (формализуют) цель обучения, т.е. фразу _\"точно предсказать\"_\n", - "* Выбор обусловлен требуемыми свойствами (непрерывность, дифференцируемость)\n", - "\n", - "**Часто используемые функции для регрессии**\n", - "\n", - "Абсолютная ошибка: $\\mathcal{L}_{abs}(\\theta) = \\sum_{i=1}^n |y_i - f_{\\theta}(x_i)|$\n", - "\n", - "Среднеквадратичная ошибка: $\\mathcal{L}_{sq}(\\theta) = \\sum_{i=1}^n (y_i - f_{\\theta}(x_i))^2$\n" + "Loss functions depend on the problem being solved.\r\n", + "\r\n", + "### Loss functions for regression\r\n", + "\r\n", + "For regression, we often use **abosolute error** $\\mathcal{L}_{abs}(\\theta) = \\sum_{i=1}^n |y_i - f_{\\theta}(x_i)|$, or **mean squared error**: $\\mathcal{L}_{sq}(\\theta) = \\sum_{i=1}^n (y_i - f_{\\theta}(x_i))^2$" ], "metadata": { "slideshow": { @@ -251,7 +183,7 @@ }, { "cell_type": "code", - "execution_count": 7, + "execution_count": null, "source": [ "# helper function for plotting various loss functions\r\n", "def plot_loss_functions(suptitle, functions, ylabels, xlabel):\r\n", @@ -274,7 +206,7 @@ }, { "cell_type": "code", - "execution_count": 8, + "execution_count": null, "source": [ "x = np.linspace(-2, 2, 101)\r\n", "plot_loss_functions(\r\n", @@ -284,20 +216,7 @@ " '$\\mathcal{L}_{sq}$ (squared loss)'],\r\n", " xlabel = '$y - f(x_i)$')" ], - "outputs": [ - { - "output_type": "display_data", - "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAAAi0AAADZCAYAAADsdQBUAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nOzdd3hUZdrH8e+dSknoPYGE3iEUKVbEhhUQQbD31dUVlLWvvXfBsrvWtQIiiNhFBUGlSAmhQyiB0EJPKOn3+8cM+2ZjIJlkZs6U+3Nd52Imc8pvJuHMc57zFFFVjDHGGGMCXYTTAYwxxhhjKsIKLcYYY4wJClZoMcYYY0xQsEKLMcYYY4KCFVqMMcYYExSs0GKMMcaYoGCFFmOCiIioiLTx07Eai8hsEckRkRf9ccwSxz4oIq38eDwRkfdEZJ+ILPDXcQONvz93YzxlhRYTMkTkMhFZ6D7xbheRb0XkZKdzBbGbgN1ALVUd66uDiMgsEbmh5M9UNU5VN/jqmGU4GTgLSFTVPn48bkBx4HM3xiNWaDEhQUTuBF4BngIaAy2AN4DBTuYKcknASg2PESiTgE2qesjTDUUkyhvrePuYxoQkVbXFlqBegNrAQWD4cdaJxVWo2eZeXgFi3a8NADKBu4EsYDswBDgPWAvsBe4vsa9HgMnAR0AOsAxoB9zn3n4LcHaJ9ZsB0937SQduLLWvT4EP3PtaAfQ+zvtQoE2J9/0BsAvIAP4BRLhfawP8AhzAVVsyyf1zAV525zwApAFdyjjOf4ACIN/92Z7p/tkTJdYZAGSWeL4J+Lt7nweASUC1Eq8PBlKBbGA9MAh4EigCct3Hec3D93kN8CvwArAP2AicW+KY1wAb3J/tRuDyMt7r9e7jF7kzPOr++Y3u39de9++vWanfw63AOmBjGftMdq9zPbAZmO3+eT/gd2A/sBQYUGKblsBsd9YfgdeBj6qwvzLf+7H+Nrz5udtii68WxwPYYktVF/eXXyEQdZx1HgPmAY2Ahu4T/ePu1wa4t38IiHZ/We0CPgHigc7uL7VW7vUfcT8/B4hyn9g3Ag+U2H5jiWP/gqvWpxqQ4t73GaX2dR4QCTwNzDvO+yj5pfIB8IU7YzKuAtb17tcmuPNEuI97svvn5wCLgDq4CjAdgabHONZ/+N9CSunnA/hzoWUBrkJaPWAVcLP7tT7uL8mz3JkSgA7u12YBN1TyfV6Dq3B1o/vzuwVXoVSAmrgKSO3d6zYFOh/jvV4D/Fri+UBcX+g9cRV4X8VdUCiRb4b7fVYvY3/J7nU+cOeo7n7Pe9y/6wj3Z7EHaOjeZi6uQkAMrttV2fy50FKh/R3vvR/rb8Nbn7vT5wNbQntxPIAttlR1AS4HdpSzznrgvBLPz8F1OwBcX75HgEj383j3ybtvifUXAUPcjx8BZpR47UJcV+ilt68DNMd1BR9fYv2ngf+U2NePJV7rBBw5zvtQXFfKkUAe0KnEa38BZrkffwC8iauNRsntB7q/fPrhvmo+zrH+g+eFlitKPH8O+Jf78b+Bl49xnFkco9BSgfd5DZBe4rUa7m2b4Pri3g8Mo4yCRanjXcP/FlreAZ4r8TzO/SWdXCLfwOPsL9m9TqsSP7sH+LDUet8DV+O6nVkI1Cjx2kf8udBS0f0d870f62/DW5+7r/6f22KLqlqbFhMS9gANyrnP3wxXFfdRGe6f/XcfqlrkfnzE/e/OEq8fwfXFxTFe213G9nHuY+xV1ZxSx04o8XxHiceHgWoVaLPQANcVeen3dHS/d+OqbVggIitE5DoAVf0ZeA3XrYedIvKmiNQq51ieKP1ejn5mzXEVHD1V3vv8n2Oq6mH3wzh1tU+5FLgZ2C4iX4tIhwoe93/+XlT1IK6/s5LH3VKB/ZRcJwkYLiL7jy64alSa8v9/J4ePsa1H+yvnvZf5t1FKpT/3MvZljNdYocWEgrm4brEMOc4623Cd5I9q4f6Zr20D6olIfKljb63ifnfjuvIv/Z62AqjqDlW9UVWb4bpCfuNoV2lVHa+qvXDd9moH3FXBYx7CdUV9VBMP8m4BWh/jNT3Odsd9n+VR1e9V9SxcBYPVwFsV2Y5Sfy8iUhOoX+q4x8td1jpbcNWM1Cmx1FTVZ3C1o6onIiU/3+ZV2N8x3/vx/jZKqNLnboyvWKHFBD1VPYCrPcrrIjJERGqISLSInCsiz7lXmwD8Q0QaikgD9/of+SHbFlztZ54WkWoi0g1XY8qPq7jfIlwNeJ8UkXgRSQLuxP2eRGS4iCS6V9+H68uuSEROEJG+IhKNqxBytAFqRaQC54lIPRFpAozxIPI7wLUicoaIRIhIQokr/51AmWODlPc+j8c9zsxF7gJHHq5beBV9r5+486aISCyuXmnzVXVTBbcvy0fAhSJyjohEuv8eBohIoqpmAAuBR0QkRkT647rtWKn9He+9H+tvo+SOq/K5G+NLVmgxIUFVX8J1Uv0HroauW4DbgGnuVZ7A9aWQhqu3z2L3z/xhFK42CduAz4GHVXWGF/b7N1wFjw24enJ8Arzrfu0EYL6IHMTV82W0qm4EauG64t6Hq7p/D67GnxXxIa4eKpuAH3D1DqoQVV0AXIur59IBXI2Tj17FjwMucQ/sNt7D93k8EcBYXJ/7XuA04K8VzPsT8CAwBVctSGtgZEW2Pc4+t+DqQXU///83ehf/fx6+HOiP63fyBK7PN6+S+zveez/W30Zplf3cjfEZUa1IDacxxhh/EpFJwGpVfdjpLMYECqtpMcaYAOC+ddfafftsEK5alGnlbWdMOLFRFY0xJjA0AabiavCbCdyiqkucjWRMYLHbQ8YYY4wJCnZ7yBhjjDFBwQotxhhjjAkKVmgxxhhjTFCwQosxxhhjgoIVWowxxhgTFKzQYowxxpigYIUWY4wxxgSFoB9crkGDBpqcnOx0DGOC1qJFi3arakOnc4QKOycZUzXHOycFfaElOTmZhQsXOh3DmKAlIhlOZwgldk4ypmqOd06y20PGGGOMCQp+K7SISHMRmSkiq0RkhYiMLmMdEZHxIpIuImki0tNf+Ywx5lhEJFJElojIV05nMSac+bOmpRAYq6odgX7ArSLSqdQ65wJt3ctNwD/9mM8YY45lNLDK6RDGhDu/FVpUdbuqLnY/zsF1Akgotdpg4AN1mQfUEZGmVTlucbGydMv+quzCmKC0ZPM+pyOEBBFJBM4H3vbG/gqKilm+9YA3dmVM0Ch0/91XdZJmR9q0iEgy0AOYX+qlBGBLieeZ/Llg45FXf07nkn/9bicJE1a+W76DoW/8zvSl25yOEgpeAe4Gir2xs+e/X8PF//yd7NwCb+zOmKDw2/o9XPDqr8xas6tK+/F7oUVE4oApwBhVzS79chmb/KlYJiI3ichCEVm4a9fxP4Cr+idRr2YMoycu4Uh+UaVzGxMsdhzI5d6paXRNqM2gzk2cjhPUROQCIEtVF5WzXoXPSed2aUJ+YTHfLdvhzajGBLRpS7ZSq1oU/VvXr9J+/FpoEZFoXAWWj1V1ahmrZALNSzxPBP50qaiqb6pqb1Xt3bDh8YeXqFszhheHp7B+1yGe+HplFdIbE/iKi5Wxk1PJKyhm3MgUYqKsg2AVnQRcJCKbgInAQBH5qPRKnpyTUprXIbl+DT5fstUngY0JNIfyCvlu+Q7O79aUatGRVdqXP3sPCfAOsEpVXzrGatOBq9y9iPoBB1R1e1WPfXLbBtx4Sks+nr+ZH1furOrujAlYb/+6gd/S9/DwhZ1o1TDO6ThBT1XvU9VEVU0GRgI/q+oVVdmniDCkRwLzNu5h2/4jXslpTCCbsXInRwqKGJJSpdYegH9rWk4CrsR1pZLqXs4TkZtF5Gb3Ot8AG4B04C3gr946+N/PaU+nprW4e0oaWdm53tqtMQFj+dYDPP/9Gs7p3JhLT2he/gbGMUNSElDF2hyZsPD5kq0k1KnOCcn1qrwvf/Ye+lVVRVW7qWqKe/lGVf+lqv9yr6OqequqtlbVrqrqtWElY6MiGT8qhcP5hYydvJTi4qq1YDYmkBzJL2L0xCXUqxnDMxd3w1WxabxJVWep6gXe2Fdyg5r0aFGHzxfbLSIT2nbl5PFr+m4GpzQjIqLq56WwuuHdplE8/zi/E3PW7ea93zc5HccYr3ni65Vs2H2Il0akULdmjNNxTAVc3COBNTtzWLHNejaa0PVF6laKipWhPap+awjCrNACcHnfFpzZsRHPfruaVdtLd14yJvjMWLmTj+dv5qZTWnFSmwZOxzEVdEG3ZkRHClOttsWEsKmLt9ItsTZtG8d7ZX9hV2gREZ4d1o3aNaK5fcIScgusG7QJXlnZudwzJY3OzWpx59ntnI5jPFC3ZgxndGjMF6lbKSzyyhAwxgSUVduzWbk9m2E9E722z7ArtADUj4vlheHdWZd1kKe/sZG5TXBydW9eyuH8QsaNTCE2qmpdCY3/Xdwzgd0H85mzbrfTUYzxus+XbCUqQriwezOv7TMsCy0Ap7VryHUnteT9uRn8vNq6QZvg897vm5izbjf/OL8TbRp5p+rV+NeA9o2oWyOazxZnOh3FGK8qKlamLdnK6R0aUc+L7ezCttACcPeg9nRoEs/dn6WxKyfP6TjGVNiq7dk8++1qzurUmMv7tnA6jqmkmKgILurejBkrd3LgiA3rb0LHnHW7yMrJY1hP7zTAPSqsCy3VoiMZP6oHObmF3P3Z0ipP5GSMP+QWFHH7hCXUrhHNMxd3te7NQe6SXs3JLyzmqzQbs8WEjs8WZVK3RjQDOzT26n7DutAC0K5xPPef15GZa3bxvnWDNkHg6W9WsS7rIC+N6E79uFin45gq6pJQi/aN4/lskd0iMqHhwOECfli5k8EpCV6fSiTsCy3gmlTx9PYNeerb1azdmeN0HGOO6efVO3l/bgbXn9ySU9oef44bExxEhEt6JbJk837Ssw46HceYKvsybRv5hcVc0st7vYaOskILrpPGc5d0p1a1KOsGbQLWrpw87pqc5mqHNai903GMFw3u0YzICGGKNcg1IeCzRZl0aBJP52a1vL5vK7S4NYyP5flLurN6Rw7Pfrfa6TjG/A9V5e+Tl3Iwr5Dxo3pY9+YQ0yi+GgPaNWTq4kwbs8UEtXU7c0jdsp9hPRN90t7OCi0lnN6hEdecmMx7v21i1posp+MY81/v/76JX9bu4v7zOtLOSyNLmsAyvHdzdmbn2ZgtJqhNXpRJVIQw1Mu9ho6yQksp957bgXaN4/j75DT2HLRu0MZ5a3bk8NS3qxnYoRFX9U9yOo7xkYEdGlG/ZgyfLtzidBRjKqWgqJipizM5o2MjGviok4AVWkqpFh3JuJE9yD5SwD1T0qwbtHFUboFr9uZa1aJ57hKbvTmUxURFMLRHAj+u2mkXTCYozVydxe6D+Yzo3dxnx7BCSxk6Nq3FPed24MdVWXw8f7PTcUwYe+67NazekcPzw7v57MrFBI4RJzSnoEj5fIlNomiCz6cLt9AoPpbT2vmuZ6PHhRYRqSkiId8K8NoTkzm1XUOe+Hol6VnWDdr43y9rd/Hubxu55sRkTm/fyOk4xg/aNY4npXkdPl24xWp5TVDZmZ3LzDW7uLhnIlGRvqsPKXfPIhIhIpeJyNcikgWsBraLyAoReV5E2vosnYMiIoQXhnejRkwUt09IJa/QukEb/9lzMI+xny6lfeN47j23g9NxjB+NPKE5a3ceZMmW/U5HMabCPluUSVGxcukJvrs1BBWraZkJtAbuA5qoanNVbQScAswDnhGRK3yY0TGN4qvx3LBurNyezQvfr3E6jgkTqsrdn6WRnVvAuFEpVIsO+YpNU8IF3ZtRIyaSSQusQa4JDsXFyqcLt9CvVT1aNqjp02NVpNBypqo+rqppqvrfAQRUda+qTlHVYcAk30V01pmdGnNFvxa8NWcjv1pXROMHH83fzE+rs7jv3A50aOL9wZlMYIuLjeLCbs34Mm0bB/MKnY5jTLnmbdxDxp7DjDzB95O3lltoUdUCABEZLiLx7scPishUEelZcp1Q9cB5nWjTKI47P01l36F8p+OYEJaelcMTX63ktHYNuebEZKfjhIRgbId3aZ/mHM4v4sulNomiCXwTF2yhVrUoBnVp4vNjedJa5kFVzRGRk4GzgfeBf/omVmCpHhPJuJEp7Ducb92gjc/kFRZx+4RU4mKjeH64dW+urFBoh9ejeR3aN45nwgLrvWgC295D+Xy3fAdDeyT45Va2J4WWoy1Rzwf+qapfADHejxSYOjerzd3ndOCHlTuZ+Ifdazbe98L3a1i5PZvnLulGo/hqTscJZkHfDk9EGNWnOWmZB1i+9YDTcYw5pqmLM8kvKuayvv4Z+NKTQstWEfk3MAL4RkRiPdw+6F1/cktObtOAx75cyfpdNhur8Z5f1+3mrTkbubJfEmd0bOx0nGAXEu3whvZMJDYqgk+stsUEKFXlkwWb6ZVUl/ZN/DO9iCeFjhHA98AgVd0P1APu8kmqABURIbw4ojux0RGMnriE/EKb2MxU3d5D+dz5aSptGsXxwPkdnY4T9EKlHV7t6tFc0K0ZXyzZag1yTUCat2EvG3YdYlQf3zfAParChRZVPayqU1V1nfv5dlX9wXfRAlPjWtV4dlg3lm/N5qUZa52OY4KcqnLPlDT2Hy5g/Mge1r3Zu4K+Hd5lfVtwKL+I6anWINcEngkLNlOrWhQXdGvqt2NWuNBS6qrlHyWvWiq4/bsikiUiy4/x+gAROSAiqe7loYru29/O6dyEUX1a8O/Z6/l9vXWDNpU3YcEWZqzcyd2D2tOpmXVv9rIqt8MTkWoiskBElrob8j7q9ZTH0bNFHTo0iefj+RnWAcAElN0H8/h2+XYu7pno14utyvYeOgfPr1r+AwwqZ505qpriXh7zYN9+9+AFHWlZvyZ3TlrK/sPWDdp4bv2ugzz+1UpOaduA605q6XScUOSNdnh5wEBV7Q6kAINEpJ+Xcx6TiHB5vyRWbMsm1UbINQHk04VbKChSrujnv1tD4MfeQ6o6G9jrwfECWo2YKMaP6sGeQ3ncN3WZXQUZj+QXFjN64hKqRUfwwvDuRERY92YfKN0Ory4etsNTl6Ot7qPdi1//sw/tkUDNmEg+mmcNck1gKCpWPpm/mX6t6tGmkX8a4B4VaL2H+rurYb8Vkc7HWklEbhKRhSKycNeuXV6OUHFdEmoz9uz2fLt8B5MXZjqWwwSfF2esYfnWbJ4Z1o3Gtax7s4+cD8xQ1XUi8g/gDcDj+7kiEikiqUCWe3/zy1jHZ+ekuNgohvRI4Ku0bVarawLC7HW7yNx3hCv6+aebc0mB1HtoMZDkroZ9FZh2rBVV9U1V7a2qvRs29N0U2BVx0ymt6N+qPo98uYKNuw85msUEh9/Td/Pm7A2M6tOcczr7fgTJMFbVW9oAqGqRqqYAiUAfEelSxjo+PSdd3jeJvMJiPltkF0fGeR/NzaBBXCxnd/L/+cuj3kPAeuAcEbkNaOTN3kOqmn20GlZVvwGiRaSBt/bvKxERwkuXdic6MoIxE5dQUGTdoM2x7T+cz52fLqVlg5o8eEEnp+OEOq8OiOm+WJtF+W3zvK5Ts1r0TqrLh/MyKC62W9HGOVv2HubnNVmM6tOcmCj/D9XmSe+h0cDHQCP38pGI/M1bQUSkibjHLReRPu5se7y1f19qWrs6z1zclaWZB3jlR+sGbcqmqtw3dRl7DuUxfmQPasREOR0p1FX5lraINBSROu7H1YEzcU0L4HdX9k8iY89hflnn3C1xYz6al0GECJf19W8D3KM8+Q98PdBXVR9S1YeAfsCNFd1YRCYAc4H2IpIpIteLyM0icrN7lUuA5SKyFBgPjNQgat16btemjOidyBuz1jN/Q1CUtYyfTV6YybfLdzD27PZ0SajtdJxw4I1b2k2BmSKSBvyBq03LV96NWTHndmlKg7hYPpyb4cThjSG3oIhJC7dwdqfGNK1d3ZEMnlzqCf9f3Yr7cYW7PKjqqHJefw14zYM8AefhCzuzYONe7piUyrejT6V2jWinI5kAsXH3IR75cgUntq7PTae0cjpOWFDVwyJy9Jb2ObiGVPDolraqpgE9fBLQQzFREVzWpzmvzkwnY88hkurXdDqSCTPTl25j/+ECruzv/wa4R3lS0/IeMF9EHhGRR3BNPPaOT1IFqZqxUYwb2YOsnDzun2bdoI1LQZGre3N0ZAQvjrDuzf7i61vaTrisbxKRIlbbYvxOVXn/9020bRRH/1b1HcvhSUPcl4DrcI21sg+4VlVf8VWwYNW9eR3uOKsdX6dtZ+rirU7HMQHg5RlrScs8wDMXd3WsSjVMVemWdiBqUrsag7o0YdLCLRyy+YiMHy3M2MeKbdlcc1Iy7uanjvCoUZqqLlLV8ao6TlWX+CpUsLv5tNb0aVmPh75YTsYe6wYdzuZt2MM/f1nPiN6JnNvVf/NzGKCKt7QD1bUnJZOTW8jnS+yiyPjPf37fRK1qUQztkeBojnILLSKSIyLZZSw5IpLtj5DBJjJCePnSFCIihDGTUq0bdJg6cLiAOyelkly/Jg9feMyxEo3vhOQt7Z4t6tI1oTb/+X2T3YI2frH9wBG+W76DkX1aON7rsdxCi6rGq2qtMpZ4VbUZ3o4hoU51nhralSWb9/PqT+ucjmP8TFW5f9oysnLyeOXSFGrGWvdmfwvVW9oiwjUnJpOedZA562zCVuN7H851Tdh5pQMj4Jbm/5FhwsiF3ZsxrGcir81M549NITPtkqmAKYu38nXadu44qx3dm9dxOk7YCtVb2hd0d3V/fve3jU5HMSHuSH4RnyzYzNmdmtC8Xg2n41ihxdceHdyZxLo1GDMxlezcAqfjGD/I2HOIh79YTp+W9bj5tNZOxwk74XBLOzYqkiv7JTFrzS7Ssw6Wv4ExlTR1SSb7Dxdw3cmBMRO9FVp8LC42ildGprAjO5eHpi13Oo7xMVf35tT/tmuKtO7Nfhcut7Qv79eCmKgI3rPaFuMjxcXKu79upGtCbU5Irut0HMCzYfxFRK4QkYfcz1u4h9s35ejZoi6jz2jLtNRtTLMW/yHt1Z/WkbplP09d3JWEOta92fhOg7hYhqQ0Y8riTPYdstmfjff9sm4X63cd4rqTne3mXJInNS1vAP2BoyPb5gCvez1RiPrrgNb0TqrLg9OWs2XvYafjGB/4Y9NeXpuZzsU9E7igWzOn45gwcP3JrcgtKObj+TbYnPG+t+dsoHGtWM7vGjjnM08KLX1V9VYgF0BV91GFGVPDTVRkBC9fmgLAmEmpFFo36JCSnVvAmImpJNatwWODuzgdx4SJ9k3iOa1dQ/7zewa5BUXlb2BMBa3YdoDf0vdw7UktHZnN+Vg8SVIgIpGAgmv2U8C+eT3QvF4NnhjahUUZ+3h95nqn4xgvenDacnZk5/LKyBTirHuz8aMbT2nF7oN5TE/d5nQUE0LenrORmjGRjOrjzGzOx+LJ2XU88DnQSESexDUr84M+SRXCBqckMHN1FuN/XsfJbRvQKykwGjeZypu2ZCtfpG7jzrPa0bOF/T6dJiJ3Hu919/gtIeOkNvXp2LQWb83ZwCW9Em1uK1Nl2w8c4cul27iqfzK1qwfWxL+ezD30MXA38DSwHRiiqp/6Klgoe2xIF5rWrsaYSUvIsW7QQW3L3sP8Y9pyeifV5a8DrHtzgIh3L72BW4AE93Iz0MnBXD4hItx0akvWZR1k5posp+OYEPDOnI0orikjAo0nvYeeVdXVqvq6qr6mqqtE5FlfhgtVtapF88qlKWzdd4SHp69wOo6ppMKiYsZMSkWAly9NISoycO77hjNVfVRVHwUaAD1VdayqjgV6AYnOpvONC7o1I6FOdf79ywano5ggd+BwARMWbObCbk0DYjC50jw5y55Vxs/O9VaQcNM7uR63DWzL1MVbmb7U7kUHo9dnrmdRxj6eGNolIP9zG1oAJfsC5wPJzkTxrejICK4/uSULNu1lUcY+p+OYIPbR/AwO5Rdx06mBWXNckQkTbxGRZUAHEUlzL8tEZCOwzPcRQ9ftA9vQo0UdHvh8GZn7rBt0MFmUsY/xP69jaI8EBqc4O+upOaYPgQXuCRMfBuYDHzicyWdG9mlOnRrR/OsXa+RvKie3oIj3ftvIae0a0qlZYI7DWJGalk+AC4EvgAvcjy8Aeqnq5T7MFvKiIiMYd2kPVOHOSUspKrYZW4NBTm4BYyYtoWntajw62GZvDlSq+iRwLa7JEvfjmjDxKWdT+U6NmCiu6p/MjJU7Wbczx+k4JghNXpTJ7oP5/OW0Vk5HOaaKzPJ8QFU3AauBa4Cr3cttR0fHNZXXon4NHr2oMws27eWfs9KdjmMq4OEvVrB13xHGjUyhVrXAallv/p+4hvDsBNRW1XHAnlAfxfvaE5OpHh3JP2dZbYvxTGFRMf/+ZT09WtShf6v6Tsc5Jk/atBwEDrmXIlztWZJ9kCnsuEZQbcorP7qGgDeB64vUrUxdspW/DWxLr6R6Tscxxxd2o3jXrRnDZX1b8MXSbTbytvHIl2nbyNx3hFsHtAmYIfvL4kmX5xdLLE8CA3B1IzRVJCI8ObQrjWtVY/TEJRzKK3Q6kilD5j5X9+aeLerwt4FtnI5jyheWo3jfeEorIgT+PdtqW0zFFBcrb8xcT4cm8Qzs0MjpOMdVlT6aNYDAvfEVZGpXj+alEd3Zsvcwj1g36IBTVKzcOWkpqvDKpT2se3NwCMtRvJvUrsYlvRL5dGEmO7NznY5jgsAPK3ewLusgtwxoHfCDE3oyTsuyEr2HVgBrgHG+ixZ++raqz18HtGHyoky+TtvudBxTwj9npbNg014eG9yZFvWte3OQKD2K969AyDbELemW09pQVKy8OdvGbTHHp6q8+nM6LRvUDIqJXj0Zxv+CEo8LgZ2qavcxvGz0mW2Zs24X901No0eLOjSrU93pSGEvdct+Xv5xHRd0a8rQHnZHNBi4G+HOBhYBZwCCaxTvVY4G85MW9WswOKUZH8/P4JYBrWkQF+t0JBOgfl6dxYpt2Tx/STciA7yWBTxr05JRYtnqaYFFRN4VkSwRWX6M10VExotIurs2p6cn+w8V0ZERjBvZg8Ji5c5PU60btMMO5RUyeuISmtSqxpNDu/oks9oAACAASURBVAZ0AzXz/1RVgWmlR/H2dD8i0lxEZorIKhFZISKjfRDXJ249vQ15hcW8NcdqW0zZVJXxP6eTWLc6Q4Lkgqwig8vliEh2iSWn5L8eHOs/wKDjvH4u0Na93AT804N9h5TkBjV55KLOzNuw16p3HfbI9BVs2XuYl0Z0D7iJw0y55onICVXcRyEwVlU7Av2AW0UkKOYvat0wjou6N+PDuRnsOZjndBwTgGat3cXSLfu59fQ2RAdJO72KjNMSr6q1SizxJf+t6IFUdTaw9zirDAY+UJd5QB0RaVrR/Yea4b0SOa9rE178YQ1pmdYN2glfp21n8qJM/jqgDX0DeNwCc0ynA3NFZH2JkbzTPNmBqm5X1cXuxznAKoKo1+TfBrblSEERb83Z6HQUE2BUlVd+XEdi3eoM6xk8U3J5VLQSke4icpt76eblLAnAlhLPMznGyUFEbhKRhSKycNeuXV6OERhEhKeGdqVhfCxjJqZyON+aD/nTtv1HuG9qGt0TazP6zLZOxzGVcy7QGhjI/4/kfWFldyYiyUAPXNMBlH4tIM9JbRq5als+mLvJalvM/5i1xlXLctvpbYiJCo5aFvCs99Bo4GOgkXv5WET+5sUsZTUWKLNBh6q+qaq9VbV3w4YNvRghsNSpEcOLI7qzcc8hHv9qpdNxwkZRsXLHpFQKi5VxI3sETbWp+V+qmgFkA42BpBKLx0QkDpgCjFHVP90WD+Rz0t8GtiW3oIh/261m46aqvPzjWlctS6/gqWUBz2parsc1WNNDqvoQrvu7N3oxSybQvMTzRCDspz8+sXUD/nJqayYs2MJ3y60btD+8OXsD8zfu5ZGLOpPcoKbTcUwlicgNuHoQfQ886v73kUrsJxpXgeVjVZ3qzYz+0KZRHENSEvhg7iaybNwWA8xYuZO0zAPcPrBt0F2UeZJWcA3ff1QRZdeOVNZ04Cp3L6J+wAFVtW9p4M6z2tE1oTb3Tl3GjgN20vGltMz9vPjDGs7r2oThQXYFYv5kNHACkKGqp+O6tePRvRt31+l3gFWq+pL3I/rH6DPbUlCkvGFzEoW94mLlpRlradmgJhf3DJrmWf/lSaHlPWC+e5r3R4F5uP4zV4iITADmAu1FJFNErheRm0XkZvcq3wAbgHTgLeCvHmQLaTFREbwyMoW8gmLGTk6l2LpB+8Th/EJGT0ylYXwsT1n35lCQq6q5ACISq6qrgfYe7uMk4EpgoIikupfzvB3U15Lq12RE70Q+mb+ZrfuPOB3HOOjrZdtZvSOHMWe2DcqRvSs8uJyqviQis4CT3T+6RlVTPdh+VDmvK3BrRfcXblo3jOOhCztx39RlvPPrRm481WZQ8LbHv1rJpj2H+PiGvtSpEfJT1ISDTBGpA0wDZojIPjy85ayqv+LdGmXH3DawLVMWbWXcj2t57pLuTscxDigsKublGWtp3zieC4Ng9NuyeNIQdziwTlXHA7WBh0Skh8+SmT8ZeUJzzuncmOe+X83yrQecjhNSvlu+gwkLtvCXU1tzYusGTscxXqCqQ1V1v6o+AjyIq2Z4sLOpnJNQpzpX9Evis0WZpGcddDqOccDkRZls2H2Iv5/TPuDnGDoWT+qGHlTVHBE5GTgLeB/4l29imbKICM9c3I16NWMYPXEJR/KLyt/IlGvHgVzunZpG14Ta3HlWO6fjGC8RkYeOLsBpQApwn8OxHHXr6a2pHh3JC9+vcTqK8bMj+UW88uNaeraow5kdA3sm5+PxpNBy9BvyfOBfqvoFYTDNe6CpWzOGF4ensH7XIZ78xrpBV1VxsTJ2cip5BcW8MjIlqMYrMOU6VGIpwjVuS7KTgZxWPy6WG09txXcrdrBk8z6n4xg/en/uJnZm53HPoA5B3V7PkzP0VhH5NzAC+EZEYj3c3njJyW0bcOMpLflo3mZmrNzpdJyg9vavG/gtfQ8PXdiJ1g3jnI5jvEhVXyyxPAkMIIhGs/WVG05pRYO4GJ75djWupoQm1O07lM/rM9M5vX3DoB/d25NCxwhc4xwMUtX9QD3gLp+kMuX6+znt6dS0Fnd/ttTGXqik5VsP8Pz3azinc2NGntC8/A1MsKsBhH0L9rjYKEaf0Zb5G/fy8+osp+MYP3h9ZjqH8gq599yOTkepMk9meT6sqlNVdZ37+XZV/cF30czxxEZFMn5UCkcKihg7eal1g/bQkfwiRk9cQr2aMTxzcbegri41ZTs615B7WQGsAcY5nSsQjOzTglYNavLMt6spLCp2Oo7xoS17D/PB3AyG92pO+ybxTsepMk96D1UTkTtFZKqITBGRO0Skmi/DmeNr0yief5zfiTnrdvPe75ucjhNUnvh6Jet3HeKlESnUrWlNs0LU0bmGLgTOBpqp6mvORgoM0ZER3D2oA+uyDjJp4ZbyNzBB69nvVhMRAXeESCcDT24PfQB0Bl4FXgM6Ah/6IpSpuMv7tuDMjo149tvVrNz2pylRTBlmrNzJx/M3c9OprTipjXVvDlWqmlFi2aqqNutoCed0bkyflvV46Ye1ZOcWOB3H+MCijL18lbadm05tTZPaoVHHUOHB5YD2qlpyRKKZIrLU24GMZ0SEZ4d1Y9C4OYyeuIQv/3Yy1aIjnY4VsLKyc7lnShqdmtZi7NmhceVhyiYidx7v9WAelt8bRIQHz+/Eha/9yhsz13PvuR2cjmS8qLhYeeyrVTSKj+Xm00KnKZcnNS1L3HMCASAifYHfvB/JeKp+XCwvDO/OuqyDPP3NKqfjBCxX9+alHM4vZPyoFGKjrHAX4noDt+DqMZQA3Ax0AuLdS9jrmlibi3sm8O6vG9m857DTcYwXTV+6jaVb9nPXOe2pEeNJ/URgK7fQcrQxG9AX+F1ENonIJlzzCJ3q43ymgk5r15DrTmrJ+3Mz+Hm1dYMuy3u/b2LOut08cH4n2jSy76ww0ADoqapjVXUs0AtIVNVHVfVRh7MFjHsGdSAqUnjiaxv3KVQcyivk6W9X0TWhNsN6htbErxWpaTnamG0Q0BLXyJKnuR+f77toxlN3D2pPhybx3DU5jV05eU7HCSgrt2Xz7LerObNjY67o28LpOMY/WgD5JZ7nE+aDy5Wlca1q3Hp6G35YuZM56zyaBNsEqDdmpbMzO49HLuoctMP1H0u5hZaSjdmAbKAxkFRiMQGiWnQk40f14GBeIXd9ttQGjnLLLXB1b65dI5pnh9nszWHkQ2CBe2b6R4AFuDoUmFKuP7klLerV4NEvV1JgXaCDWsaeQ7w1eyNDeyTQK6mu03G8zpMuzzcAs3ENMPeo+99HfBPLVFa7xvHcf15HZq3ZxQdzM5yOExCe/mYV67IO8sLw7tSPi3U6jvET9yi41wL7gL3A1ar6lLOpAlO16EgeuqAT6VkHee+3jU7HMZWkqjwyfQXRkRKyDas9aYg7GjgByFDV04EegNUlBqCr+idxevuGPPnNKtbsyHE6jqN+Xr2T9+dmcN1JLTmtXUOn4xg/KjEz/ThsZvpyndmpMWd0aMQrP65j+4EjTscxlTBj5U5mrtnFHWe1o3Gt0OjiXJonhZZcVc0FEJFYVV0NtPdNLFMVIsLzw7tTq1oUoycuIbcgPGeD3pWTx12T0+jQJJ67B9mfahiymek99PCFnSkqVp742nohBpsj+UU8+uVK2jWO4+oTk52O4zOeFFoyRaQOMA2YISJfANt8E8tUVYO4WJ4f3p3VO3J4PgynoVdV7vpsKQfzChk/qoeNXROebGZ6D7WoX4NbT2/D12nbmbXG5iUKJuN+WsfW/Ud4fHAXoiNDdy5jT+YeGqqq+1X1EeBB4B1giK+Cmao7vX0jrjkxmXd+3cgva8PrTt77v29i1ppdPHB+R9o1tu7NYcpmpq+Ev5zWilYNa/LgF8s5kh+etbTBZvWObN6es4ERvRODfhbn8lRknJY/dbVQ1V9Udbqq5h9rHRMY7j23A+0bx/P3yUvZczA8ukGv2ZHDU9+uZmCHRlzZzzq4hTGbmb4SYqMieWpoV7bsPcK4n9Y5HceUo7hYuX/qMmpVj+a+EJjFuTwVueqYKSJ/E5H/GdxCRGJEZKCIvA9c7Zt4pqqqRUcyblQKB44UcM+UtJDvBp1bUMTtE5ZQq1oUz11iszeHo6MXUcebmd4utI6vX6v6DO+VyFtzNrBi2wGn45jj+HBeBos37+cf53cMi8lfK1JoGYTr3vAEEdkmIitFZAOwDhgFvKyq//FhRlNFHZrU4t5BHfhxVRYfzd/sdByfeva71azZmcPzw7vTwLo3hyu70PKCB87vSN0aMdwzJY1CG7slIGXuO8yz363mlLYNGNojwek4flGRweVyVfUNVT0J12ByZ+AaGjtJVW9U1VSfpzRVds2JyZzariFPfr2S9KzQ7AY9a00W7/22iWtOTOb09o2cjmOcYxdaXlCnRgyPDe7M8q3ZvDXHxm4JNKrKA58vB+CpoeEzaKZHjdJUtcBdxbrfV4GMb0RECC8M70aNmChun5BKXmFoNbDbfTCPv09Oo33j+JAdVMlUjF1oec+5XZpwTufGvPzj2pC92AlWkxdl8svaXdx1Tnua16vhdBy/sZb0YaRRfDWeG9aNlduzeSGEukGrKvd8lkZ2bgHjRqVY92bzX3ahVTUiwhNDulIzJpKxk+02UaDYtv8Ij3+5kj4t63F1/2Sn4/iVFVrCzJmdGnNFvxa8NWcjv67b7XQcr/ho/mZ+Wp3FvYM60KFJLafjGBNSGsbH8tjgLizdsp9/z97gdJywp6rcMyWNIlVeuKR7yE2IWJ5KFVpEZKD736YiUuHLWhEZJCJrRCRdRO4t4/VrRGSXiKS6lxsqk88c3wPndaJNozjGTk5l36H88jcIYOlZOTzx1UpOa9eQa09KdjqOCUEi8q6IZInIcqezOOWCbk05v2tTXvlxLcu3Wm8iJ30wN4M563Zz33kdaVE/fG4LHVXZmpZBIpKIa0jslyuygbtw8zpwLtAJGCUincpYdZKqpriXtyuZzxxH9ZhIxo1MYd+hAu6dGrzdoPMKi/jbhFRqxkbx/HDr3myOTUSqMuLWf3A17g1brttEXahbI4Yxk1LDdmoQp6Vn5fDUN6sY0L4hV/RtUf4GIaiyhZY6wD3A3UBFRyzrA6Sr6gb3oHQTgcGVPL6pos7NanP3oPZ8v2InE//Y4nScSnn+uzWs2p7N85d0o1F8aE4OZrzmSxH5QkTeFJG/i8iAim6oqrNxzRId1urWjOGF4d1JzzrI09/Y3ET+lldYxJhJqdSIieS5YeF7kVbZQstjwCxVXcP/z+9RngSg5LdjpvtnpQ0TkTQR+UxEmpe1IxG5SUQWisjCXbvCa3h6b7rupJac3KYBj325kvW7DjodxyNz1u3i7V83cmW/JM7o2NjpOCbw/aKqg4G/A82BJt7cebick05t15DrTmrJ+3Mz+HHlTqfjhJXnv1vD8q3ZPDusG41CdAbniqhsoeV54Cz3IE3fVHCbsoqFpe9LfAkkq2o34Edcs7L+eSPVN1W1t6r2btiwYUUzm1IiIoQXR3SnWnQEYyamkl8YHD0D9h7KZ+ynS2nTKI77zwv9YauNV8SJSE/gCFBPVSd6c+fhdE6659z2dGpai7s+W8qOA7lOxwkLM9dk8favG7mqfxJnd/ZqeTvoVLbQskpVb1bVq4GRFdwmE9cVzlGJlJolWlX3qOrR201vAb0qmc9UUONa1XhmWDeWbT3ASzPWOh2nXEdbzu8/XMC4kSlUj7HuzaZCpgP9ga+A7xzOEtRioyJ59bIe5BYUc/uEJdYN2se2HzjC2E+X0r5xvF2kUYlCi4i8hauW5Xb3feGKjn/wB9BWRFqKSAyuws70UvtuWuLpRYDdOPWDczo3YVSfFvx79np+Xx/Y3aAnLNjCjJU7uXtQezo3q+10HBM8zgK+AHKBvg5nCXqtG8bx1MVdWLBpb1Bc7ASrgiJXwTCvoIjXL+9pY1BRiUKLqt6I6wTwB9AdqFCrfFUtBG7DNevqKuBTVV0hIo+JyEXu1W4XkRUishS4HbjG03ymch68oCMtG9TkzklL2X84MLtBp2cd5LGvVnBymwZcd1JLp+OY4FKy84BH9zREZAIwF2gvIpkicr0P8gWdoT0SGdWnBW/MWs9Pq6x9iy88//0a/ti0j6cu7kqbRnFOxwkIlb09dA/wENANSKvoRqr6jaq2U9XWqvqk+2cPqep09+P7VLWzqnZX1dNVdXUl8xkP1YiJYvzIHuw5lMf9ny8LuG7Q+YXFjJm0hOrRkbw4IvwGVDJV9hjwhbvzgEf3M1R1lKo2VdVoVU1U1Xd8EzH4PHxhJzo3q8WYSals3H3I6Tgh5au0bbw5ewNX9kticEp4TIZYEZUttNQF5gFPAO29F8c4qUtCbcae3Z5vlu1g8qJMp+P8jxdnuFrOPzOsG43DuOW8qRxVzQS2uh//aWBLUznVoiP51xW9iIoQ/vLhQg7lFTodKSSs2ZHD3Z+l0SupLg9eUNZwZuGrsoWWvUAkkIWNXxBSbjqlFSe2rs8j01ewKUCunH5P382bszdwWd8WnBPmLedNxYjIXSLyu4i0KfHjTBG52bFQIap5vRqMH9WD9KyD3DEpleLiwKqlDTZ7D+Vzwwd/UDM2ijcu70lMlM22U1KlPg1VfQzXaLjjARvTOYQc7QYdHRnB6IlLKHC4Z8D+w/nc+elSWjaoyT/Ot5bzpsLaAHdQoqOAquYAFzqWKISd0rYhD5zfiR9W7uTlH61hbmXlFxZzy0eL2Jmdx5tX9rJa5TJUpQhXC7hBVSs0jL8JHk1rV+eZi7uyNPMA435c51gOVeW+qcvYcyiPcZf2oEZMlGNZTND5CVeHgYKjPxCRBsBJjiUKcdedlMzIE5rz6s/pTF0cWLeXg4Gq8o9py5i/cS/PDetGjxZ1nY4UkCpUaDlGVetW4C++iWWcdm7Xpozoncjrs9KZv2GPIxkmL8zk2+U7GHt2e7omWvdmU3Gq+imuWuB0EflDRJ4ETgTWOJssdIkIjw3uwomt63PPlLSAHz4h0Lw+M51PF2Zy+8A2DOlhDW+PpaI1LVbVGoYevrAzSfVqcOenSzlwpKD8Dbxo4+5DPPLlCk5sXZ+bTmnl12Ob0KCqrwItgIdxtcH7O5DjaKgQFxMVwT+v6EVy/Zr85cNFrN6R7XSkoDBlUSYv/LCWi3skcMdZ7ZyOE9AqWmixqtYwVDM2inEje7AzO5cH/NgNuqComNETlxAdGWHdm02VqOoR91AL96rqqcDjTmcKdbWrR/PetSdQMyaKq95ZwJa9h52OFNB+Xr2Tu6ekcVKb+jw9rGvYToRYURUqtFhVa/jq3rwOd5zVjq/StjN18Va/HPOVH9eSlnmAZy7uStPa1f1yTBMeVPUXpzOEg8S6NXj/uj7kFhRx5TvzycqxOYrKsmDjXv768WI6Na3Fv6/sTWyUjXhbngo3xLWq1vB182mt6duyHg9PX8HmPb69apq3YQ9vzFrPiN6JnNu1afkbGGMCUvsm8bx37Qlk5eRxxdvz2XsoMEfadsqSzfu49r0FJNSpznvXnkBcrHU0qAiPeg9ZVWt4iowQXro0hQiB0ZN8N0HagcMF3DkpleT6NXn4ws4+OYYxxn96JdXj7at7k7HnMJdbweW/lm7Zz9XvLqB+XCwf39CPBnGxTkcKGlUatcaqWsNHQp3qPHVxV5Zs3s/4n9O9vn9V5f5py8jKyeOVS1OoaVcdxoSEE1s34M2rerN+10Eue2seew7mOR3JUYs37+OKt+dTq3o0n9zYlya1bSwWT9hQe6bCLujWjGE9E3nt53Us3OTdgZCnLN7K12nbueOsdnRvXser+zbGOOu0dg159+oT2Lj7EJe+OY/tB444HckRc9fv4ap3FlAvLoZJf+lPYt0aTkcKOlZoMR555KJOJNatwZhJqWTneqcbdMaeQzz8xXL6tKzHzae19so+jTGB5eS2DXj/uj7sOJDLJf+cy4ZdB52O5Fc/rNjB1e8toGntaky6qT8JdayTQWVYocV4JL5aNK+MTGH7gVwemra8yvtzdW9OJTJCePnSFCKte7MxIatfq/pMuLEfRwqKGPbP31mUsc/pSH7x4bwMbv5oER2b1uLTv/S3W0JVYIUW47GeLeoy+oy2TEvdxrQlVesG/epP60jdsp8nh3a1Kw9jwkDXxNpMueVEaleP5rK35vFV2janI/lMUbHy1DereHDacga0b8QnN/Slbs0Yp2MFNSu0mEr564DW9E6qy4PTlld68Kg/Nu3ltZnpDOuZyIXdm3k5oTEmULVsUJMpt5xIl4Ta3PbJEl78YU3IzQ594EgB17//B2/O3sCV/ZJ488pe1sHAC6zQYiolKjKCly9NAeCOSaked4M+cKSAMRNTSaxbg0cHW/dmY8JN/bhYPrmxLyN6J/Lqz+lc//4f7AuRLtErt2Uz5PXf+HXdbp4Y0oXHh3QhKtK+br3BPkVTac3r1eDxIV1YmLGPN2at92jbh75Yzo7sXMaNTLFBlYwJU7FRkTw7rBuPD+nCb+l7OH/8HK/3TPQnVeWT+ZsZ+sZvHM4v5JMb+3FFvySnY4UUK7SYKhnSI4EhKc0Y99O6Cjeq+3xJJl+kbmPMGW1t+nVjwpyIcGW/JKbcciJRkRGM+Pdcnv9+NfmFvhnE0ld2H8zjxg8Wcv/ny+jTsh5f334KfVrWczpWyLFCi6myx4Z0oUmtaoyZtISccrpBb9l7mAenraB3Ul3+enobPyU0xgS6rom1+Wb0KVzSK5HXZ67nwld/JXXLfqdjlUtVmbo4kzNf+oXZ63bz0AWdeP/aPjbKrY9YocVUWS13N+it+47wyPSVx1yvsKiYMZNSEbDuzcaYP4mLjeK5S7rzztW9OXCkgKFv/Mb9ny8L2LYua3fmcPnb87nz06W0alCTb24/metObmkz0/uQNSYwXnFCcj1uO70N439OZ0D7hmX2Bnp95noWZexj3MgUmtezkSCNMWU7o2Nj+rSsx0sz1vLB3Ay+TtvOXwe05uoTk6kW7fxMyDuzcxn/0zom/rGFuNgoHh/cmcv6JtmFmB9YocV4ze1ntGVO+m7u/3wZPZPq/s+4K4sy9jH+53UMSWnG4JQEB1MaY4JBfLVoHr6wMyNPaMHT367i6W9X8+5vG7nxlFZc1rcFNWL8//W1bf8R3pqzgQkLNlNYpFzetwVjzmxHPRt7xW9ENbj7xvfu3VsXLlzodAzjlrHnEOeNm0PnhNpMuLEfkRFCTm4B542fgyp8M/oUalWLdjqmKUFEFqlqb6dzhAo7J/nGvA17eOXHtczbsJfa1aMZ0TuRy/smkdygpk+Pq6oszNjHB3Mz+HbZdhQYnNKM0We0Jam+b48dro53TvJrUVVEBgHjgEjgbVV9ptTrscAHQC9gD3Cpqm7yZ0ZTNUn1a/LY4C6MnbyUf/2ynltPb8PD01ewdd8RJt/c3wosJuiUd94y/tGvVX0m3tSfRRl7effXTbz72ybemrORXkl1uah7M87q1JhmXhpVW1VZsS2bH1bu5IvUrWTsOUx8tSiu6p/MtScl2+1tB/mt0CIikcDrwFlAJvCHiExX1ZItN68H9qlqGxEZCTwLXOqvjMY7Lu6ZwMw1Wbw8Yy05uYVMXbyV0We0pVeSdf8zwaWC5y3jR72S6tErqR47s3OZungrny/J5OHpK3h4+graNY6jf6v69GhRly4JtUiqX5PoCgzqdiS/iLU7c1i29QCLMvYxd/0edmTnIgL9W9XnttPbcH63po7ckjL/y5+/gT5AuqpuABCRicBgoOR//sHAI+7HnwGviYhosN/DCjMiwpNDurI4Yx//+mU9PVvU4W8DrXuzCUoVOW8ZBzSuVY1bBrTmlgGtWb/rIDNW7uS39N18ujCT9+dmABAZISTUqU7D+Fjq1oihekwkURFCflExh/MK2Xsonx3ZuezMzvvvfhvExdC3VX1ObduAMzo2tq7LAcafhZYEYEuJ55lA32Oto6qFInIAqA/sLrmSiNwE3ATQokULX+U1VVC7RjSvXtaDF75fy7PDutkQ1iZYVeS8Zeckh7VuGEfr0+K4+bTWFBYVk77rIMu3ZrNp9yEy9h5md04emfsOk1dYTGFxMdGREVSPjqR+XCxtG8eTXL8GrRvG0SWhNol1qyNivYAClT8LLWX9FZSuQanIOqjqm8Cb4Gr0VvVoxhd6JdVjwk39nI5hTFXYOSnIREVG0KFJLTo0qeV0FOMD/rz8zQSal3ieCJSek/y/64hIFFAbCN6JKIwxwa4i5y1jjJ/4s9DyB9BWRFqKSAwwEpheap3pwNXux5cAP1t7FmOMgypy3jLG+Infbg+526jcBnyPq+vgu6q6QkQeAxaq6nTgHeBDEUnHVcMy0l/5jDGmtGOdtxyOZUzY8mv/LVX9Bvim1M8eKvE4Fxjuz0zGGHM8ZZ23jDHOsC4dxhhjjAkKQT+Mv4jsAjLKWa0BpbpNB5lgzh/M2SE88iepakN/hAkHdk4KeMGcHYI7f0WzH/OcFPSFlooQkYXBPLdKMOcP5uxg+Y1vBPvvJZjzB3N2CO783shut4eMMcYYExSs0GKMMcaYoBAuhZY3nQ5QRcGcP5izg+U3vhHsv5dgzh/M2SG481c5e1i0aTHGGGNM8AuXmhZjjDHGBDkrtBhjjDEmKIRNoUVEnheR1SKSJiKfi0gdpzN5QkSGi8gKESkWkaDo7iYig0RkjYiki8i9TufxhIi8KyJZIrLc6SyeEpHmIjJTRFa5/2ZGO53J/Jmdk/zPzknO8OY5KWwKLcAMoIuqdgPWAvc5nMdTy4GLgdlOB6kIEYkEXgfOBToBo0Skk7OpPPIfYJDTISqpEBirqh2BfsCtQfbZhws7J/mRnZMc5bVzUtgUWlT1B1UtdD+dh2uK+aChqqtUdY3TOTzQB0hX1Q2qmg9MBAY7nKnCVHU2rkk7g46qblfVxe7HOcAqIMHZVKY0Oyf5TpGZpQAABAtJREFUnZ2THOLNc1LYFFpKuQ741ukQIS4B2FLieSb2xel3IpIM9ADmO5vElMPOSb5n56QAUNVzkl9nefY1EfkRaFLGSw+o6hfudR7AVVX1sT+zVURF8gcRKeNn1r/ej0QkDpgCjFHVbKfzhCM7JwUUOyc5zBvnpJAqtKjqmcd7XUSuBi4AztAAHKCmvPxBJhNoXuJ5IrDNoSxhR0SicZ0cPlbVqU7nCVd2Tgoodk5ykLfOSWFze0hEBgH3ABep6mGn84SBP4C2ItJSRGKAkcB0hzOFBRER4B1glaq+5HQeUzY7J/mdnZMc4s1zUtgUWoDXgHhghoikisi/nA7kCREZKiKZQH/gaxH53ulMx+NuYHgb8D2uRlefquoKZ1NVnIhMAOYC7UUkU0SudzqTB04CrgQGuv/WU0XkPKdDmT+xc5If2TnJUV47J9kw/sYYY4wJCuFU02KMMcaYIGaFFmOMMcYEBSu0GGOMMSYoWKHFGGOMMUHBCi3GGGOMCQpWaDHGGGNMULBCizHGGGOCghVajF+IyIsislJE3hKRX9zTxFdkuxgRmS0iITXlhDHGWXZOCk5WaDE+JyKtgJNUtROQCkxV1aKKbOueQv4n4FIfRjTGhBE7JwUvK7SYMolIVxH5rcTzniLycyX20x74BUgSkSXADcAXJV6fKSJnuR8/ISLjy9jNNOByT49tjAkddk4yYMP4m2MQkQhcM6AmqGqRiMwExqrq4krs6wlgE/ABsFlVm5R47VTgMeAt4DJck8cVldo+Etihqg0r+36MMcHNzkkGrKbFHIOqFgMrgM4iMgzXf+z/OTmIyI8isryMZXCp3XUFlgINgP2ljjMbEOBOYOTRk4OIPF5inSIgX/6vvftniSOKwjD+HIT4B1LZaJ3SBGKfJiBol09gFWws06ULJE36tBYp0ppUFja2Qoo0guYDWClYiBKxeFPsCIuSweg2d/f5dbN39nAHlpczdy6zVU9HfJmSGmEmCcCNROqzz+DfOTeBtduDSVbuWWeJQdhMAzPDA1X1AlgETpOcd58tcPe3OQ38+Z/JSxo7ZtKEc6VFffaBT8D3JMcPKdDdiVwnuUxyBkxV1Uw3tgh8A94AF1W12n1tmcHmuJsa88BJkuuHX4qkMWAmTTibFvU5Aq6Az4+o8Rw4GDreBV5V1RywzeCZ9CHwEfjQnfOSoYAAXgM7j5iDpPFgJk04N+Lqn6rqC/AzydcR1lwG3iVZ7zlnC9jonmFTVdvA+yS/RzUPSe0xk+RKi+6oqmdVdQTMjjIcAJL8Avb6XuSU5O1QODwBfhgO0uQyk3TDlRZJktQEV1okSVITbFokSVITbFokSVITbFokSVITbFokSVITbFokSVITbFokSVIT/gLBqJo0q8gj7wAAAABJRU5ErkJggg==", - "text/plain": [ - "
" - ] - }, - "metadata": { - "needs_background": "light" - } - } - ], + "outputs": [], "metadata": { "slideshow": { "slide_type": "slide" @@ -307,21 +226,25 @@ { "cell_type": "markdown", "source": [ - "## Функции ошибки для классификации\n", - "\n", - "**0-1 loss**\n", - "\n", - "Предполагается решающая функция вида $\\hat{y} = \\mathrm{sign}(f_{\\theta}(x))$:\n", - "\n", - "$\\mathcal{L}_{0-1}(\\theta) = \\sum_{i=1}^n l_i \\quad l_i = \\begin{cases}\n", - " 0 & y_i f_{\\theta}(x) > 0 \\\\\n", - " 1 & в\\ противном\\ случае\n", - " \\end{cases} \\\\\n", - "$\n", - "\n", - "**логистическая функция ошибки**\n", - "\n", - "$\\mathcal{L}_{log}(\\theta) = \\sum_{i=1}^n \\frac{1}{\\log(2)} \\log(1 + e^{-y_i f_{\\theta}(x)})$" + "### Loss functions for classification\r\n", + "\r\n", + "Let's consider binary classification for a moment. In this case we have two classes, numbered 0 and 1. The output of the network $f_\\theta(x_i)\\in [0,1]$ essentially defines the probability of choosing the class 1.\r\n", + "\r\n", + "**0-1 loss**\r\n", + "\r\n", + "0-1 loss is the same as calculating accuracy of the model - we compute the number of correct classifications:\r\n", + "\r\n", + "$$\\mathcal{L}_{0-1} = \\sum_{i=1}^n l_i \\quad l_i = \\begin{cases}\r\n", + " 0 & (f(x_i)<0.5 \\land y_i=0) \\lor (f(x_i)<0.5 \\land y_i=1) \\\\\r\n", + " 1 & \\mathrm{ otherwise}\r\n", + " \\end{cases} \\\\\r\n", + "$$\r\n", + "\r\n", + "However, accuracy itself does not show how far are we from the right classification. It could be that we missed the correct class just by a little bit, and that is in a way \"better\" (in a sense that we need to correct weights much less) than missing significantly. Thus, more often logistic loss is used, which takes this into account.\r\n", + "\r\n", + "**Logistic Loss**\r\n", + "\r\n", + "$$\\mathcal{L}_{log} = \\sum_{i=1}^n -y\\log(f_{\\theta}(x_i)) - (1-y)\\log(1-f_\\theta(x_i))$$" ], "metadata": { "slideshow": { @@ -331,48 +254,33 @@ }, { "cell_type": "code", - "execution_count": 9, + "execution_count": null, "source": [ - "# define and vectorize zero-one loss\r\n", + "x = np.linspace(0,1,100)\r\n", "def zero_one(d):\r\n", - " if d < 0:\r\n", - " return 1\r\n", - " return 0\r\n", + " if d < 0.5:\r\n", + " return 0\r\n", + " return 1\r\n", + "zero_one_v = np.vectorize(zero_one)\r\n", "\r\n", "def logistic_loss(fx):\r\n", " # assumes y == 1\r\n", - " y = 1\r\n", - " return 1 / np.log(2) * np.log(1 + np.exp(-y * fx))\r\n", - "\r\n", - "zero_one_v = np.vectorize(zero_one)" + " return -np.log(fx)" ], "outputs": [], "metadata": {} }, { "cell_type": "code", - "execution_count": 10, + "execution_count": null, "source": [ - "plot_loss_functions(suptitle = 'Common loss functions for classification',\r\n", + "plot_loss_functions(suptitle = 'Common loss functions for classification (class=1)',\r\n", " functions = [zero_one_v(x), logistic_loss(x)],\r\n", " ylabels = ['$\\mathcal{L}_{0-1}}$ (0-1 loss)',\r\n", " '$\\mathcal{L}_{log}$ (logistic loss)'],\r\n", - " xlabel = '$y f(x_i)$')\r\n" - ], - "outputs": [ - { - "output_type": "display_data", - "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAAAi0AAADZCAYAAADsdQBUAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nO3dd5iU5dXH8e9vZpfeewcFRAGRXtQYG7GLxobYUJRYo69Go4lRYolJTOxYQBGxERULMRg1EcWGAlKUoiKgNBUFKQLSzvvH8ywZ12V3dndmnpnd87muuZiZp53ZGe45c1eZGc4555xz2S4WdQDOOeecc8nwpMU555xzOcGTFuecc87lBE9anHPOOZcTPGlxzjnnXE7wpMU555xzOcGTFueyiCST1CFD12oqaYqk9ZL+nolrJlx7g6TdM3g9SXpY0hpJ76f5WmMl3ZTG8+/820mqLumfktZKelrSaZJeScM1fybp41Sf17nSyos6AOeSJWkIcDmwJ7AemAXcbGZvRRpY7hoOfAPUsTRO2CTpdeAxM3uw4Dkzq5Wu6+3C/sBAoJWZfZ/ha6dUob/diUBToKGZbQufe7y815BkQEczWxhe802gU3nP61x5eU2LywmSLgfuAP5EUEi3Ae4FBkUZV45rC8xLZ8KSRdoCS8qSsEjK5h93bYFPEhIW5yo2M/Ob37L6BtQFNgAnFbNPVYKkZkV4uwOoGm47EFgGXAV8DawEjgOOBD4BVgO/SzjXCOBp4DGCGp0PgT2Aa8LjlwK/SNi/BTAxPM9C4LxC53oKGBeeay7Qu5jXYUCHhNc9DlgFfA5cC8TCbR2AN4C1BLUl/wifF3B7GOdaYA7QtYjrjAW2AlvCv+2h4XM3JexzILAs4fES4DfhOdcC/wCqJWwfRFD7tQ74DDgcuBnYDmwOr3NPKV/nUOAt4G/AGmAxcETCNYcCi8K/7WLgtCJe67Dw+tvDGP4YPn9e+H6tDt+/FoXeh4uAT4HFu3iv9gfeAb4LPxNDE/62N4X36wMvhq9tTXi/VUnx7+r9TfzbAX8M37+t4esaVvD3Sti3C/Bq+Bq/IvycA32Bd8PYVwL3AFXCbVPCa3wfnveUIj4LewGvh8fPBY4t9NkaCfwrfF3vAe2jLkf8VjFukQfgN7+VdAu//LYBecXscwMwFWgCNA6/TG4Mtx0YHn8dkB9+Wa0CngBqhwX7ZmD3cP8R4ePDCJpQx4VfKL9POH5xwrXfIKj1qQZ0D899SKFzHQnEgVuAqcW8jsQv83HAC2GM7QgSrGHhtifDeGLhdfcPnz8MmAHUI0hg9gKa7+JaY/lxklL4ceEvqiXA+wRJWgNgPnB+uK0vwRfswDCmlsCe4bbXgXPL+DqHEnwpnxf+/S4gSEoF1CRIkDqF+zYHuuzitQ7lx1/mBxMkAz0JEt67gSmF4ns1fJ3VizhfG4Iv5FPDz0RDoHvhv2P4/AlAjfD1PQ08H27bZfy7en+L+NuNIGh6+8nrDK+3ErgiPEdtoF+4rRfQn+Dz3S58Ly8r6hqFPwvh610I/A6oEv4t1ye8jrEESVLf8PyPA+OjLkf8VjFu3jzkckFD4Bsrvgr8NOAGM/vazFYR/Ao9I2H7VoL+L1uB8UAj4E4zW29mcwl+LXZL2P9NM3s5vObTBInQnxOObyepnqTWBL+4f2tmm81sFvBgoWu/ZWaTzGw78CiwT0kvWFKc4BfuNWGMS4C/J5x3K0HTQIvwum8lPF+boN+PzGy+ma0s6XqlcJeZrTCz1cA/CZI0CH7ljzGzV81sh5ktN7MFJZ0sidcJ8LmZjQ7/fo8QfLk3DbftALpKqm5mK8P3MhmnhfF+YGY/ENSiDZDULmGfW8xstZlt2sXx/zGzJ81sq5l9G773PxI+P8HMNprZeoKap58n7LKr+Hf1/pbG0cCXZvb38Bzrzey9MK4ZZjbVzLaFf/MHCsVVnP5ALYL/D1vM7DWCGqRTE/Z51szeD///PM7/PifOlYsnLS4XfAs0KqFvQQuCpoUCn4fP7TxH+KUHUPAl9FXC9k0EBTG72PZNEcfXCq+xOvxCSrx2y4THXybc3whUS6KfRCOCX7GFX1PBea8iqG14X9JcSecAhF8g9xBUz38laZSkOiVcqzQKv5aCv1lrgiah0irpdf7omma2Mbxby4L+KacA5wMrJf1L0p5JXvdHnxcz20DwOUu87tJijk/q9UqqIekBSZ9LWkfQ9FJPUryE+It8f0tplzFK2kPSi5K+DOP6E8F7kYwWwFIz25HwXEmf+Ux3vHYVlCctLhe8S9DEclwx+6wg+GVaoE34XLqtABpIql3o2svLed5v+N+v7Z+c18y+NLPzzKwF8Cvg3oKh0mZ2l5n1Imj22gO4Mslrfk/QjFGgWSniXQq038W24jr6Fvs6SxLWhg0kqH1ZAIxO5jgKfV4k1SSo0Uu8bnFxF/d6E11BMOqmn5nVAQ4ouGRx8Rf3/pZCcTHeF16vYxjX7wpiSsIKoLWkxO+PVHzmnSuRJy0u65nZWoL+KCMlHRf+es2XdISkv4a7PQlcK6mxpEbh/o9lILalBP1nbpFUTVI3gqaScg07DWt1ngJullRbUluC4d6PAUg6SVKrcPc1BF+w2yX1kdRPUj5BElLQATUZs4AjJTWQ1Ay4rBQhPwScLekQSTFJLRNqDb4CipyTpaTXWZxwnpljw4TjB4JOo8m+1ifCeLtLqkpQ0/Be2FSSjMeBQyWdLClPUkNJRTWB1CaomftOUgPg+mTi39X7m2RsBV4Emkm6TFLV8O/bLyGudcCG8H26oNCxu3zPCDrWfg9cFf4/PBA4hqDZ1Lm08qTF5QQzu43gy+xago6uS4GLgefDXW4CphOMbPkQ+CB8LhNOJejMuAJ4DrjezF5NwXkvIfhyWEQwguYJYEy4rQ/wnqQNBCNfLjWzxUAdgl/rawiq7L8lGHmTjEeB2QQdbl8hGB2UFDN7HzibYOTSWoLOyQU1GXcCJ4YTu91VytdZnBhBTcYKgo6fPwcuTDLe/wJ/ACYQdFZtDwxO5tjw+C8IOldfEV57FkX3VboDqE5QozQV+HeS8e/q/U1a2GQ5kCCh+JJgJNRB4ebfAEMIOtCO5qfv9QjgEUnfSTq50Hm3AMcCR4Sv617gzGT6MDlXXjKrDFM0OOeccy7XeU2Lc84553KCJy3OOeecywmetDjnnHMuJ3jS4pxzzrmc4EmLc84553KCJy3OOeecywmetDjnnHMuJ3jS4pxzzrmc4EmLc84553KCJy3OOeecywl5UQdQXo0aNbJ27dpFHYZzOWvGjBnfmFnjqOOoKLxMcq58iiuTcj5padeuHdOnT486DOdylqTPo46hIvEyybnyKa5M8uYh55xzzuWEjCUtksZI+lrSR7vYLkl3SVooaY6knpmKzTnnnHPZL5M1LWOBw4vZfgTQMbwNB+7LQEzOOeecyxEZ69NiZlMktStml0HAODMzYKqkepKam9nK8lz3mw0/sHbT1vKcwpVBnWr5NK5dNeownMs6G7ds44vVG9mzWZ2oQ3Eu52RTR9yWwNKEx8vC58qVtIycvJCH315SnlO4MojHxNRrDvHExblCfvP0bN5fvJqJF+9Pi3rVow7HuZySTUmLinjOitxRGk7QhESbNm2KPekve7Sie+t65Q7OJW/mF98x9p0lrN20xZMWl7UkVQOmAFUJysJnzOz6QvtUBcYBvYBvgVPMbEl5rnv5wE4cP/Jthj86nad/tS/Vq8TLczrnKpVsSlqWAa0THrcCVhS1o5mNAkYB9O7du8jEpsDereqyd6u6qYrRJaFKPMbYd5awbUexb41zUfsBONjMNkjKB96S9JKZTU3YZxiwxsw6SBoM/AU4pTwX7dCkFncM7s6546bz2wlzuHNwd6SifrM55wrLpiHPE4Ezw1FE/YG15e3P4qIRiwUF8HZPWlwWs8CG8GF+eCv8oR0EPBLefwY4RCnIMA7Zqym/+UUnJs5ewQNTFpX3dM5VGhmraZH0JHAg0EjSMuB6gkICM7sfmAQcCSwENgJnZyo2l1p5nrS4HCEpDswAOgAjzey9Qrvs7GtnZtskrQUaAt8UOk/STdYFLjywPfNWruMv/15Ap6a1OWjPJuV6Lc5VBpkcPXRqCdsNuChD4bg0Kqhp8eYhl+3MbDvQXVI94DlJXc0scS6ppPralabJeueJJW49sRuLV33Pr5+cyXMX7UeHJrXK8CqcqzyyqXnIVRAFNS07PGlxOcLMvgNe56dzSe3saycpD6gLrE7VdWtUyWPUmb2okhdj+LjpPj2DcyXwpMWlXNxrWlwOkNQ4rGFBUnXgUGBBod0mAmeF908EXgtrhVOmVf0a3Hd6L5au2cglT85k2/YdqTy9cxWKJy0u5eLyPi0uJzQHJkuaA0wDXjWzFyXdIOnYcJ+HgIaSFgKXA1enI5C+uzXgxkFdmfLJKm55qXDe5JwrkE1Dnl0FkRf3pMVlPzObA/Qo4vnrEu5vBk7KRDyD+7ZhwZfreeitxXRqWpuT+7Qu+SDnKhmvaXEpF48FHytPWpwrnWuP2ov9OzTi989/yLQlKes641yF4UmLSzlvHnKubPLiMUYO6Umr+jU4/9EZLF29MeqQnMsqnrS4lPOOuM6VXd0a+Tx4Vm+2bN/BeeOms+GHbVGH5FzW8KTFpZz3aXGufNo3rsXIIT359OsNXDZ+lk8f4FzIkxaXcrGC5qHUjgx1rlI5YI/G/OGovfjP/K/468sfRx2Oc1nBRw+5lPvfNP4+34Rz5XHWvu349OsN3P/GZ3RoUosTe7WKOiTnIuU1LS7ldvZp2e41Lc6VhyRGHNuF/To05Jpn5/D+Yh9R5Co3T1pcyhUkLTu8eci5csuPx7h3SC9a16/Brx6dzhff+ogiV3l50uJSLs9HDzmXUnVr5PPQ0D7sMDjnkWm+RpGrtDxpcSkXi/noIedSbbdGNbn/9F4s+eZ7Ln7iA7b6GkWuEvKkxaVcnictzqXFgPYN+dPxe/Pmp98wYuJcUrx2o3NZz0cPuZSLe9LiXNqc3Kc1n32zgQfeWMRujWpy7s92jzok5zLGkxaXcj4jrnPp9dvD9uTzbzZy86T5tG1Yk4Gdm0YdknMZ4c1DLuW8psW59IrFxO2ndKdby7r8+smZfLhsbdQhOZcRGU1aJB0u6WNJCyVdXcT2NpImS5opaY6kIzMZn0uNPF/l2WU5Sa3Dsma+pLmSLi1inwMlrZU0K7xdF0Wsu1K9SpzRZ/WmQc0qDHtkGiu+2xR1SM6lXcaSFklxYCRwBNAZOFVS50K7XQs8ZWY9gMHAvZmKz6VOWNHiSYtLO0k1w7KltLYBV5jZXkB/4KIiyiOAN82se3i7oVzBpkGT2tUYM7QPm7Zs55yx01i/2YdCu4otkzUtfYGFZrbIzLYA44FBhfYxoE54vy6wIoPxuRSRRDwmT1pcykmKSRoi6V+SvgYWACvD2pJbJXVM5jxmttLMPgjvrwfmAy3TF3n6dGpWm5GnBYsrXvTETB8K7Sq0TCYtLYGlCY+X8dNCYgRwuqRlwCTgkqJOJGm4pOmSpq9atSodsbpyisfkHXFdOkwG2gPXAM3MrLWZNQF+BkwF/izp9NKcUFI7oAfwXhGbB0iaLeklSV2KOUekZdIBezTm5uO6MuWTVVz3wkc+FNpVWJkcPaQiniv8P+tUYKyZ/V3SAOBRSV3N7Ec/HcxsFDAKoHfv3v6/MwvFJZ/G36XDoWb2kzYQM1sNTAAmSMpP9mSSaoXHXWZm6wpt/gBoa2Ybwv51zwNF1uRkQ5k0uG8blq7ZyMjJn9G6QQ0uPLBDFGE4l1aZrGlZBrROeNyKnzb/DAOeAjCzd4FqQKOMROdSKi8mXzDRpVxRCUtZ9gEIk5sJwONm9mwR51lnZhvC+5OAfElZXR795hedGNS9BX/998e8MGt51OE4l3KZTFqmAR0l7SapCkFH24mF9vkCOARA0l4ESYu3/+SgeFxs3+Ft6y49JJ0kqXZ4/1pJz0rqWYrjBTwEzDez23axT7NwPyT1JSgvvy1/9Okjib+e2I2+uzXgyqfnMHVRVofrXKllLGkxs23AxcDLBJ3enjKzuZJukHRsuNsVwHmSZgNPAkPNG2dzUlxiu791Ln3+YGbrJe0PHAY8AtxXiuP3A84ADk4Y0nykpPMlnR/ucyLwUVge3QUMzoXyqGpenNFn9KZNwxoMHzedT75aH3VIzqVMqfu0SKoJbDaz7aU9NqxinVTouesS7s8jKExcjvPRQy7NCsqfo4D7zOwFSSOSPdjM3qLofnaJ+9wD3FPmCCNUt0Y+Y8/uw/H3vsPQMe/z3EX70bROtajDcq7cSqxpSdUQQ1e5eJ8Wl2bLJT0AnAxMklQVn+H7R1rVr8HDQ/uwdtNWhj7sc7i4iiGZ/+QpH2LoKr5YzJuHXFqdTNDUfLiZfQfUB66MNqTs07VlXe47vReffrWeCx77gC3bvJ+Zy23JJC2HmtmNZjYnceixma02swlmdgLwj/SF6HJRnjcPufQ6CnjVzD6VdC3B7NnfRBxTVjpgj8b8+YRuvLXwG656ZjY7/P+ly2ElJi0FwwcL9db/Q2Jv/WSHGLrKI+aTy7n0Km9H3ErlxF6tuPKwTjw/awV//veCqMNxrsxK0wacWEj8Ai8kXDHyYvJfdC6dftIRF6gSYTxZ78ID23PmgLaMmrKIB99cFHU4zpVJaZIWLyRc0uKxmNe0uHTyjrilJInrj+nCEV2bcdO/5vvkcy4nleY/uRcSLmnxmK/y7NKqcEfcBnhH3BLFY+L2U7rTb7cG/Obp2bz5qc/d6XJLaZIO763vkhaPxTxpcWljZhuBz4DDJF0MNDGzVyIOKydUy48z+qzedGhSm189OoPZS7+LOiTnklaapMV767uk+eghl06SLgUeB5qEt8ckFbkqvPupOtXyeeTsPjSsVYWzx07js1Ubog7JuaSUtSOu99Z3xYrLkxaXVsOAfmZ2XTirdn/gvIhjyilN6lRj3Dn9iAnOfOh9Vq7dFHVIzpXIO+K6tPBp/F2aif+VSYT3i52W3/3Ubo1qMvbsvqzbtJUzH3qfNd9viTok54rlHXFdWuTFxTZf5dmlz8PAe5JGhGsOTSVYtdmVUteWdRl9Vm8+X72RoWOn8f0P26IOybldKk9HXO+t73YpJuFLD7l0MbPbgHOA1cAa4GwzuyPaqHJX/90bMnJITz5avpbhj05n89ZSr4frXEYknbR4b31XGkFHXK9pceljZjPM7C4zu9PMZkYdT64b2Lkpfz2hG28v/JZfPzmTbdv9/6/LPkknLd5b35VG3Fd5dmkgab2kdUXc1ktaV8pztZY0WdL8cNX6S4vYR5LukrRQ0pyCpUsqqhN6teL6YzrzyryvuGrCHJ/V2mWdvFLsW9Bb/3sASX8B3gXuTkdgLrfFY2KHr/LsUszMaqfwdNuAK8zsg3BdtRmSXjWzeQn7HAF0DG/9CEZM9kthDFnn7P12Y/3mbdz26ifUrprHiGO7IHkfZ5cdSpO0eG99l7S4L5jospyZrQRWhvfXS5oPtAQSk5ZBwDgzM2CqpHqSmofHVliXHNyB9Zu3MvrNxdSsmsdVh+8ZdUjOAaVLWgp66z8XPj4O763vdsGHPLtcIqkd0AN4r9CmlsDShMfLwud+lLRIGg4MB2jTpk26wswYSfzuyL34fst27n39M2pWzeOigzpEHZZzpeqIW67e+pIOl/Rx2DZ89S72OVnSvLB9+Ylkz+2yjyctLldIqgVMAC4zs8L9YoqqTf7JB9vMRplZbzPr3bhx43SEmXGSuGlQV47v0ZJbX/6YMW8tjjok50pV04KZzQBmlPYikuLASGAgwS+VaZImJrYdS+oIXAPsZ2ZrJDUp7XVc9vBp/F06SXoEuDScfgFJ9YG/m9k5pTxPPkHC8riZPVvELsuA1gmPWwEryhZ17onFxK0ndmPz1u3c8OI8quXHGdIv92uSXO4qsaYlRb31+wILzWyRmW0BxhO0FSc6DxhpZmsAzOzr0rwQl128T4tLs24FCQtAWG70KM0JFPQufQiYH9YkF2UicGY4iqg/sLai92cpLC8e487BPTioU2N+//yHPDNjWdQhuUqsxJqWFPXWL6pduHAP/D0AJL0NxIERZvbvok5W0dqPK6J4TD5c0qVTTFL9gh85khpQyppjYD/gDOBDSbPC534HtAEws/uBScCRwEJgI3B2CmLPOVXyYtx3ei/OfWQ6Vz0zm/y4GNS9ZdRhuUqotP/JyyqZduE8gmGFBxJUwb4pqWvir6mdB5qNAkYB9O7d278Zs1BeLOY1LS6d/g68I+kZgrLkZODm0pzAzN6ihBGQ4aihi8oaZEVSLT/O6DN7M/Th97n8qdnkx2McuXfzqMNylUym1g5Kpl14GfCCmW01s8XAxwRJjMtBMXlNi0sfMxsHnAh8BawCfmlmj0YbVcVXvUqcMUP70KN1PX795Exenvtl1CG5SiZTScs0oKOk3SRVAQYTtBUneh44CEBSI4LmokUZis+lWLBgoictLn3MbK6Z3WNmdxeaEM6lUc2qeTx8dh/2blWXix7/gFfnfRV1SK4SyUjSYmbbgIsJFlycDzxlZnMl3SDp2HC3l4FvJc0DJgNXmtm3mYjPpZ4PeXbpIOmt8N/CAwRKPY2/K7va1fJ55Jy+dGlZlwsfn8F/PHFxGVKupEVS0p3SzGySme1hZu3N7ObwuevMbGJ438zscjPrbGZ7m9n48sTmohWX2O7T+LsUM7P9w39rm1mdhFttM6sTdXyVSZ1q+Yw7py+dm9fhgsdneI2Ly4jy1rT8MSVRuAqnoKbFPHFxaRCufVbicy696lbPZ9ywfnRuEdS4eB8Xl27JzNMyZxe3D4GmGYjR5aC8WDAow5uIXJoMLOK5IzIehaNu9XweHdaXLi2CPi4vfVipprFxGZbMkOemwGEEU/cnEvBOyiNyFUKsIGkxy9i4elfxSboAuBBoL2lOwqbaeHkUmTrVgsTlrDHvc/GTM7ljh3HMPi2iDstVQMl8n7wI1DKzWYU3SHo95RG5CsFrWlyaPAG8BNwCJK5htt7MVkcTkoOgc+64Yf045+FpXDp+Jlu37+CXPVtFHZarYEpsHjKzYeEkTDtJahZuG5KuwFxui4dJiw97dqlkZmvNbAnwLLDazD4nmNX2QUmlmsbfpV6tqnmMPacP/XdvyBVPz2b8+19EHZKrYMraEXdSSqNwFU5B0uITzLk0+YOZrZe0P0Hz9SPA/RHH5IAaVfIYM7QPB3RszNXPfsjYt311aJc6ZU1aip362rk8r2lx6bU9/Pco4D4zewGoEmE8LkG1/DijzuzFLzo3ZcQ/5zFy8sKoQ3IVRFmTltEpjcJVODHv0+LSa7mkBwjWHJokqSqZm+HbJaFqXpyRp/VkUPcW3Pryx/z13wt8CgRXbmUa2GFm96Y6EFexeEdcl2YnA4cDfzOz7yQ1B66MOCZXSH48xm0nd6dGlTzuff0z1m/exh+P7bLzR41zpZVU0iJpT2AQ0JJgRdUVwEQzm5/G2FwOi8eCH72etLh0MLONBJ1xCx6vBHyCkCwUj4k/Hd+VOtXyeGDKItZv3sqtJ+1DftwrxlzpJTO53G+B8QT9WN4nWPxQwJOSri7uWFd5FZRH3qfFpVIq1x6SNEbS15I+2sX2AyWtlTQrvF2XitdQGUni6iP25MrDOvH8rBWc/+gMNm/dXvKBzhWSTE3LMKCLmW1NfFLSbcBc4M/pCMzlNq9pcemQuPZQCk43FrgHGFfMPm+a2dEpuFalJ4mLDupAner5XPfCR5zx0Hs8eFYf6lbPjzo0l0OSqZ/bARQ1tWHzcJtzP+F9Wly2M7MpgE9Il2Fn9G/L3af2YNbS7zjlgXf5at3mqENyOSSZmpbLgP9K+hRYGj7XBugAXJKuwFxui8mTFpc+ki4v4um1wIyiZu8uhwGSZhP04/uNmc3dRTzDgeEAbdq0SeHlK6aju7WgXvUq/OrR6fzy3ncYN6wv7RvXijoslwOSqWl5GdiDYEXnl4FXgBFAJzN7CUCSdwV3P+I1LS7NegPnEwwOaEmQMBwIjJZ0VYqu8QHQ1sz2Ae4Gnt/VjmY2ysx6m1nvxo0bp+jyFdv+HRsxfvgANm/dzon3vcMHXxRe3s65n0omaZkMXASsMLMJZvaMmU0F4pIOlvQIcFZao3Q5Jx4vmFzOWxBdWjQEeprZFWZ2BUES0xg4ABiaiguY2Toz2xDenwTkS2qUinO7wN6t6jLhgn2pUz2fIaOn8uq8r6IOyWW5ZJKWwwlmn3xS0gpJ8yQtAj4FTgVuN7OxaYzR5aB4WPm2wyeTcunRBtiS8HgrQa3IJuCHVFxAUrOCWmRJfQnKy29TcW73P+0a1WTCBfvSqWltfvXodB6d+nnUIbksVmKfFjPbDNwL3CspH2gEbDKz79IdnMtdO6fx3+5Ji0uLJ4Cpkl4gmILhaIIfVjWBecmcQNKTBE1KjSQtA64H8gHM7H7gROACSduATcBg8yld06JRrao8Obw/lzwxkz88/xHL1mzkt4ft6ZPQuZ8o1Yy44bDnMk/gJOlw4E4gDjxoZkUOl5Z0IvA00MfMppf1ei46ce/T4tLIzG6UNAnYnyBpOT+hrDgtyXOcWsL2ewiGRLsMqFEljwfO6MWIf87lgTcWsWz1Jv5+8j5Uy49HHZrLImWaxr8sJMWBkcBAYBkwTdJEM5tXaL/awK+B9zIVm0u9nUmL/zB16bONYNoFI2gecjkuLx7jxkFdadOgBre8tIAVazcx6ozeNK5dNerQXJbI5DzKfYGFZrbIzLYQzLI7qIj9bgT+Cvjg/RwW91WeXRpJuhR4nKC5ugnwmCSfgqECkMTwA9pz32k9mb9yHceNfJuPv1wfdVguS2QyaWnJ/+Z5gaC2pWXiDpJ6AK3N7MXiTiRpuKTpkqavWrUq9ZG6cttZ0+J9Wlx6DAP6mdn1ZnYd0B84L+KYXAod3rU5T/1qAFu37+CE+95h8oKvow7JZYFyJy3h2kRJ7VrEczu/0STFgNuBK0o6kc+JkP28ecilmQhGNRbYTtFljMth3VrV44WL96NtwxoMe2QaD765CO8LXbmVuk+LpKcSHwLdgb8kcegyoOZsYccAABTQSURBVHXC41YEs0wWqA10BV4PRxk2AyZKOtY74+aePF97yKXXw8B7kp4jKIeOA8ZEG5JLh+Z1q/P0+QO44qnZ3PSv+Sz4cj03H9+VqnneQbcyKktH3HVmdm7BA0n3JXncNKCjpN2A5cBgYEjBRjNbS9A+XXDe1wmmzfaEJQf5Ks8unczstrCM2I8gaTkrxdP3uyxSo0oeI4f05M7/fsqd//2URas2cP/pvWhSp1rUobkMK0vScnOhx79P5iAz2ybpYoKlAOLAGDObK+kGYLqZTSxDLC5LFazyvMOTFpdCktaT0KxMQpOQJDOzOpmPymVCLCb+b+AedGpWmyuems3Rd7/F/Wf0omeb+lGH5jKo1EmLmS0u9DjpVVLDqbAnFXruul3se2BpY3PZI89HD7k0MLPaUcfgonXk3s3ZrVFNhj86ncEPTGXEsV0Y0s8XqawsMjl6yFUiBTNZek2LS6VkFmf1BVwrvr2a1+GfF+9P//YN+d1zH/LbZ+aweev2kg90Oa/MSYsvHOaK4zUtLk0mS7pE0o9+Wkuq4gu4Vi71alTh4aF9uOig9vxj+lJOuv9dlq7eGHVYLs3KU9PiPfXdLv1vGn9f5dmllC/g6naKx8SVh+3J6DN7s+Tb7zn67rd4bYGvFF2RlSdp8SpYt0sFqzz7kGeXSma22czuNbP9gLbAIUBPM2trZuf5CKLKaWDnprx4yf60rFedc8ZO588vLWDbdv/BVBGVJ2nxbyO3S/G4Nw+59DKzrWa20lecdwBtG9bk2Qv35dS+bbj/jc84dfRUVq7dFHVYLsW8psWlRZ6v8uycy7Bq+XFu+eXe3HFKd+auWMcRd77Jf+Z5c1FFUp6k5ZqUReEqnJh8Gn+X3SSNkfS1pI92sV2S7pK0UNIcST0zHaMrm+N6tOTFS/anRd3qnDtuOiMmzvXRRRVEmZMWMyvyP7pzkFDT4gsmujSTVLeMh44l6Ni7K0cAHcPbcCDZ2b9dFti9cS2eu2hfzt6vHWPfWcJxI9/mk698tehcV+qkRdIQSeMlPSbpCUmnpiMwl9viPuTZZc4LkiZIGinpXElVkjnIzKYAxU2OOQgYZ4GpQD1JzVMRsMuMqnlxrj+mC2OG9mbV+h845u63GPv2Yl90MYeVpabl52Y22MxON7MhwP6pDsrlPknEBDu8cHDp97aZnQBcBfQEbkrReVsCSxMeLwuf+wlJwyVNlzR91apVKbq8S5WD92zKvy87gH3bN2TEP+dx5pj3+XLt5qjDcmVQlqSlqqSjJHWTdCRQPdVBuYohLxbzmhaXCfUl9Qa2AnVI3cjGogYbFHluMxtlZr3NrHfjxo1TdHmXSo1rV2XM0D7cdFxXpi1ZzWF3TOGFWcu91iXHlCVpuRCoDxwJNAAuTmlErsKIxXz0kMuIywhqfO8HXgRS1d9uGdA64XErYEWKzu0iIInT+7dl0q9/xu6Na3Lp+Flc9MQHfLPhh6hDc0kqy4KJG4HHJO1lZvPTEJOrIPJiMU9aXNqZ2RbgjjSceiJwsaTxQD9grZmtTMN1XIbt3rgWz5y/L6OmLOL2Vz9h6qIp3DCoC0ft3Rxfuiq7JVXTIulKSe9I6pDw9DJJ56cpLlcBxGPypMWlnaQbJT0taaykTqU47kngXaCTpGWShkk6P6FcmwQsAhYCowlqmV0FEY+JCw5sz4u/3p/W9atz8RMz+dWjM/h6nfd1yWbJ1rR0AP4P2DnzpJmtl3QMQZWscz/hSYvLkPpmdlI4auh24KJkDjKzYkc+WtDZIalzudy1R9PaTLhgXx56azG3vfoJh9z2Br8/ci9O6dPaa12yULJ9Wv4LDCTo6AbsXOV5v3QE5SqGeEzeEddlwg/hxG8G1Iw6GJd78uIxfvXz9vz7sgPo0qIOVz/7IaeMmsrCr31el2yTVNJiZk8Ba4GFkqZJuhnYF/g4ncG53JYXk6/y7NJK0sNAXeBQYDzwj2gjcrlst0Y1efK8/vzlhL35+Mv1HHHnm9z68gI2bfHZdLNF0qOHzOxuoA1wPRAHfgMknYZKOlzSx+GU2FcXsf3ycJn5OZL+K6ltsud22Skm4QutunQys7OBS4C3gTeB46ONyOU6SZzSpw3/veLnHNOtBSMnf8Yv7njD1zDKEqUa8mxmm8xskpldbWYHADcmc5ykODCSYFrszsCpkjoX2m0m0NvMugHPAH8tTWwu++TFvabFpZ+ZbQJWm9kdZjY86nhcxdCoVlVuO6U744f3p1penHPHTeecsdNY8s33UYdWqZVnwUTM7I0kd+0LLDSzReHwxPEEU2QnnmtyOJwaYCrBnAguh3mfFpcOPprRZVL/3Rsy6dKfce1Re/H+4tUMvP0NbnlpPus3by35YJdy5UpaSiHp6bBDw4CXdrXRp8zODXHJp/F36VDkaEbgmMgichVafjzGuT/bndd+83MGdW/JA28s4qC/vcE/pn3hIyQzLFNJS9LTYUs6HegN3Lqrk/mU2bkhHhPbfJVnl3o+mtFFokntavztpH144aL9aNewBr+d8CFH3fUmUz7xH8+ZkqmkJanpsCUdCvweONbMfF7lHBf0afGkxaWWj2Z0UdundT2ePn8AI4f05Pst2zhzzPuc/uB7fLR8bdShVXiZSlqmAR0l7RZOADWYYIrsnST1AB4gSFi+zlBcLo3iEtu9ecilQXlHMzpXXpI4qltz/nP5z/nD0Z2Zu2ItR9/9Fhc/8QGLvbNu2pR67aGyMLNtki4GXiYoYMaY2VxJNwDTzWwiQXNQLeDpcBbCL8zs2EzE59LDZ8R16RSOGpoU3pD082gjcpVR1bw4w/bfjZN6t2L0lEU8+OZiXvroS07o2ZJLDu5I6wY1og6xQslI0gJgZjsLl4Tnrku4f2imYnGZ4X1aXCaVYjSjcylXp1o+V/yiE2cOaMd9r3/GY+99znMzl3Nir9ZcfHAHWtarHnWIFUKmmodcJRSPefOQc65yaVy7Ktcd05k3rjyQwX3a8MyMpRx462SunjCHz7/1ZqPy8qTFpU1eLObNQ865Sql53erceFxX3rjyIE7t24ZnZy7noL+9zqXjZ7Lgy3VRh5ezMtY85CqfmE8u55yr5FrUq84Ng7py8UEdePCtxTw29XNemLWCgzo1ZvgB7em/ewNfTboUvKbFpU1eTOzwpMVlqSTWQxsqaZWkWeHt3CjidBVDkzrV+N2Re/HO1QdzxcA9mLNsLaeOnsqx97zNC7OWs9UXakuKJy0ubXwaf5etklwPDeAfZtY9vD2Y0SBdhVSvRhUuOaQjb199MDcf35Xvf9jGpeNnsf9fXuOe1z7l2w0+RVlxvHnIpU1cXtPistbO9dAAJBWshzYv0qhcpVEtP85p/dpyap82vP7J1zz89hL+9son3PXaQo7euzmnD2hLj9b1vOmoEE9aXNrE42Kbr/LsslNR66H1K2K/EyQdAHwC/J+ZLS1iHyQNB4YDtGnTJsWhuoosFhMH79mUg/dsysKv1/Pou58z4YPlPDtzOXs1r8OQfm0Y1L0FdarlRx1qVvDmIZc2eT65nMteyayH9k+gnZl1A/4DPLKrk/l6aC4VOjSpzR8HdWXq7w7h5uO7AvCH5z+i383/5YqnZvPeom+xSj6NhNe0uLTxafxdFitxPTQz+zbh4WjgLxmIyzlqVc3jtH5tGdK3DXOWrWX8tC/45+yVTPhgGW0b1uCXPVrxy54tK+Vsu560uLSJx8R2nxHXZaed66EBywnWQxuSuIOk5ma2Mnx4LDA/syG6yk4S+7Suxz6t6/GHozsz6cMvmTBjGbf/5xNu/88n9GlXn0HdW3Lk3s1pULNK1OFmhCctLm3y4j56yGWnJNdD+7WkY4FtwGpgaGQBu0qvRpU8TuzVihN7tWLZmo28MGsFz81czrXPf8SIiXPZr0Mjju7WnF90bkbdGhW3/4snLS5tYhI7vHnIZakk1kO7Brgm03E5V5JW9Wtw0UEduPDA9sxfuZ4XZi/nxdkrufKZOfwu/iH7dWjE4V2acWjnpjSqVTXqcFPKkxaXNnk+T4tzzqWNJDq3qEPnFnW4+vA9mbX0O/790ZdM+mglVz/7IXruQ3q1qc/Azk05ZK+mtG9cM+eHUHvS4tImHot5nxbnnMsASfRoU58ebepz9RF7Mm/lOl6Z+xWvzPuKW15awC0vLaBdwxoctGcTDurUhL67NaBafjzqsEvNkxaXNvEYPnrIOecyTBJdWtSlS4u6/N/APVi2ZiOvLfia1xZ8zRPvfcHDby+hWn6Mfrs15GcdG/Gzjo3Zo2mtnKiF8aTFpU08FvPmIeeci1ir+jU4c0A7zhzQjk1btjN18be88fEq3vx0FTf9az4wn0a1qjKgfUP2bd+Q/rs3pF3DGlmZxHjS4tImHsMnl3POuSxSvUqcgzoFTUQAy7/bxNsLv+GtT7/h3UXf8s/ZwXRFTetUpU+7BvTdrQG92zagU7PaxGPRJzGetLi0icdibN9hmFlWZuzOOVfZtaxXnZN7t+bk3q0xMz5btYGpi1bz3uLVTFu8mhfnBFMV1aqaR4829YJ+M63r0b11PepHMDdMRpMWSYcDdxLMi/Cgmf250PaqwDigF/AtcIqZLclkjC518sKsfIdB3HMW55zLapLo0KQ2HZrU5vT+bTEzlq3ZxPTPVzN9yRo++OI77nntUwoq0Ns0qEG3VnXp1qouXVsGfWjqVk/vHDEZS1oSloIfSDCF9jRJE80scVXVYcAaM+sgaTDBtNmnZCpGl1oFVYnbd1hWVCs655xLniRaN6hB6wY1OL5HKwC+/2Ebc5atZc6y75i19DtmfvHdztoYgNYNqtO5eR32Krg1q0Or+tWJpeg7IJM1LcksBT8IGBHefwa4R5Kssq8QlaMKEpWrnplNPOZrc0bhysM60axutajDcM5VEDWr5jGgfUMGtG+487lvN/zA3BXr+GjFWuauWMe8Fet4Zd5XFHxz16gSp2PT2uzRpBZXHtaJJnXKXiZlMmlJZin4nfuE02yvBRoC3yTu5MvA54Z9WtWjbcMaTFuyJupQKq1NW7dHHYJzroJrWKsqB+zRmAP2+N8K59//sI1PvlrPx1+uZ8GX6/nkq/VM/ngV1x7duVzXymTSksxS8Mnsg5mNAkYB9O7d22thstSA9g1548qDog7DOedchtWsmrdzsrtUymSdfYlLwSfuIykPqEuwUJlzzjnnKrlMJi07l4KXVIVgKfiJhfaZCJwV3j8ReM37szjnnHMOMtg8lORS8A8Bj0paSFDDMjhT8TnnnHMuu2V0npYkloLfDJyUyZicc845lxt8HKpzzjnncoJyvcuIpFXA5yXs1ohCw6ZzTC7Hn8uxQ27Hn2zsbc2sccm7uWR4mZT1cjl2yO34y10m5XzSkgxJ082sd9RxlFUux5/LsUNux5/LsVd0uf7e5HL8uRw75Hb8qYjdm4ecc845lxM8aXHOOedcTqgsScuoqAMop1yOP5djh9yOP5djr+hy/b3J5fhzOXbI7fjLHXul6NPinHPOudxXWWpanHPOOZfjPGlxzjnnXE6oNEmLpFslLZA0R9JzkupFHVOyJJ0kaa6kHZJyZqibpMMlfSxpoaSro46nNCSNkfS1pI+ijqW0JLWWNFnS/PBzc2nUMbmf8jIp87xMikYqy6RKk7QArwJdzawb8AlwTcTxlMZHwC+BKVEHkixJcWAkcATQGThVUudooyqVscDhUQdRRtuAK8xsL6A/cFGO/e0rCy+TMsjLpEilrEyqNEmLmb1iZtvCh1OBVlHGUxpmNt/MPo46jlLqCyw0s0VmtgUYDwyKOKakmdkUgkU7c46ZrTSzD8L764H5QMtoo3KFeZmUcV4mRSSVZVKlSVoKOQd4KeogKriWwNKEx8vwL86Mk9QO6AG8F20krgReJqWfl0lZoLxlUkZXeU43Sf8BmhWx6fdm9kK4z+8Jqqoez2RsJUkm9hyjIp7z8fUZJKkWMAG4zMzWRR1PZeRlUlbxMiliqSiTKlTSYmaHFrdd0lnA0cAhlmUT1JQUew5aBrROeNwKWBFRLJWOpHyCwuFxM3s26ngqKy+TsoqXSRFKVZlUaZqHJB0O/BY41sw2Rh1PJTAN6ChpN0lVgMHAxIhjqhQkCXgImG9mt0Udjyual0kZ52VSRFJZJlWapAW4B6gNvCpplqT7ow4oWZKOl7QMGAD8S9LLUcdUkrCD4cXAywSdrp4ys7nRRpU8SU8C7wKdJC2TNCzqmEphP+AM4ODwsz5L0pFRB+V+wsukDPIyKVIpK5N8Gn/nnHPO5YTKVNPinHPOuRzmSYtzzjnncoInLc4555zLCZ60OOeccy4neNLinHPOuZzgSYtzzjnncoInLc4555zLCZ60uJSS9HdJ8ySNlvRGuBx8ScdUkTRFUoVaVsI5Fz0vkyoWT1pcykjaHdjPzDoDs4BnzWx7SceFy8T/FzglzSE65yoRL5MqHk9aXKlI2lvS2wmPe0p6TVIn4A2graSZwLnACwn7TZY0MLx/k6S7Cp36eeC09L8C51xF4mVS5eLT+LtSkRQjWBm1pZltlzQZuMLMPpB0E7AEGAd8YWbNEo47ALgBGA0MIVgkbnvC9jjwpZk1ztyrcc7lOi+TKhevaXGlYmY7gLlAF0knEBQEH4Sb9wZmA42A7wodNwUQcDkwuKBwkHRjuH07sEVS7Yy8EOdcheBlUuXinYxcWUwlWLXzQuDwhOe7EBQeVYFqiQdI2htoDnxjZuvD55rx489gVWBz+sJ2zlVQXiZVEl7T4spiKnAT8JyZLQcIf41sNbONZrYGiEuqFm5rDjwODAK+l3RYeJ4eBJ3jkNQQWGVmWzP7UpxzFYCXSZWEJy2uLBYAPwB/SXiuK/BRwuNXgP0l1QCeJWhjng/cCIwI9+lOWEAABwGT0hizc67i8jKpkvCOuK7UJN0DTDOzR4rZpwdwuZmdUcw+DwHnmdkOSc8C15jZx6mP2DlXkXmZVHl4TYtLmqT2khYA1YsrHADMbCYwubiJnMxsWFg4VAGe98LBOVcaXiZVPl7T4pxzzrmc4DUtzjnnnMsJnrQ455xzLid40uKcc865nOBJi3POOedygictzjnnnMsJnrQ455xzLid40uKcc865nPD/YbIc/OI+cmIAAAAASUVORK5CYII=", - "text/plain": [ - "
" - ] - }, - "metadata": { - "needs_background": "light" - } - } + " xlabel = '$p$')\r\n" ], + "outputs": [], "metadata": { "slideshow": { "slide_type": "slide" @@ -382,15 +290,21 @@ { "cell_type": "markdown", "source": [ - "## Строим нейросеть\n", - "Рассмотрим решение нашей задачи при помощи простейшей однослойной нейросети такого вида:\n", - "\n", - "\n", - "При этом модель будет описываться как\n", - "$$\n", - "f_\\theta(x) = W\\times x + b\n", - "$$\n", - "где параметры $$\\theta = $$" + "## Neural Network Architecture\r\n", + "\r\n", + "We have generated a dataset for binary classification problem. However, let's consider it as multi-class classification right from the start, so that we can then easily switch our code to multi-class classification. In this case, our one-layer perceptron will have the following architecture:\r\n", + "\r\n", + "\r\n", + "\r\n", + "Two outputs of the network correspond to two classes, and the class with highest value among two outputs corresponds to the right solution.\r\n", + "\r\n", + "The model is defined as\r\n", + "$$\r\n", + "f_\\theta(x) = W\\times x + b\r\n", + "$$\r\n", + "where $$\\theta = \\langle W,b\\rangle$$ are parameters.\r\n", + "\r\n", + "We will define this linear layer as a Python class with a `forward` function that performs the calculation. It receives input value $x$, and produces the output of the layer. Parameters `W` and `b` are stored within the layer class, and are initialized upon creation with random values and zeroes respectively." ], "metadata": { "slideshow": { @@ -400,7 +314,7 @@ }, { "cell_type": "code", - "execution_count": 11, + "execution_count": null, "source": [ "class Linear:\r\n", " def __init__(self,nin,nout):\r\n", @@ -413,33 +327,23 @@ "net = Linear(2,2)\r\n", "net.forward(train_x[0:5])" ], - "outputs": [ - { - "output_type": "execute_result", - "data": { - "text/plain": [ - "array([[ 1.77202116, -0.25384488],\n", - " [ 0.28370828, -0.39610552],\n", - " [-0.30097433, 0.30513182],\n", - " [-0.8120485 , 0.56079421],\n", - " [-1.23519653, 0.3394973 ]])" - ] - }, - "metadata": {}, - "execution_count": 11 - } - ], + "outputs": [], "metadata": {} }, { "cell_type": "markdown", "source": [ - "## Переходим к вероятностям\n", - "Расширяем нейросетевую модель с помощью функции **softmax**: $\\sigma(\\mathbf{z}_c) = \\frac{e^{z_c}}{\\sum_{j \\in J} e^{z_j}}$ для $c \\in 1 .. |C|$\n", - "\n", - "\n", - "\n", - "Можем рассматривать $\\sigma(\\mathbf{z})$ как распределение вероятности на классах $C$: $q = \\sigma(\\mathbf{z}_c) = \\hat{p}(c | x)$\n" + "Because we use Numpy operations, we can pass a vector if input values to our network, and it will give us the vector of output values.\r\n", + "\r\n", + "## Softmax: Turning Outputs into Probabilities\r\n", + "\r\n", + "As you can see, our outputs are not probabilities - they can take any values. In order to convert them into probabilities, we need to normalize the values across all classes. This is done using **softmax** function: $$\\sigma(\\mathbf{z}_c) = \\frac{e^{z_c}}{\\sum_{j} e^{z_j}}, \\quad\\mathrm{for}\\quad c\\in 1 .. |C|$$\r\n", + "\r\n", + "\r\n", + "\r\n", + "> Output of the network $\\sigma(\\mathbf{z})$ can be interpreted as probability distribution on the set of classes $C$: $q = \\sigma(\\mathbf{z}_c) = \\hat{p}(c | x)$\r\n", + "\r\n", + "We will define the `Softmax` layer in the same manner, as a class with `forward` function: " ], "metadata": { "slideshow": { @@ -449,7 +353,7 @@ }, { "cell_type": "code", - "execution_count": 12, + "execution_count": null, "source": [ "class Softmax:\r\n", " def forward(self,z):\r\n", @@ -461,35 +365,17 @@ "softmax = Softmax()\r\n", "softmax.forward(net.forward(train_x[0:10]))" ], - "outputs": [ - { - "output_type": "execute_result", - "data": { - "text/plain": [ - "array([[0.88348621, 0.11651379],\n", - " [0.66369714, 0.33630286],\n", - " [0.35294795, 0.64705205],\n", - " [0.20216095, 0.79783905],\n", - " [0.17154828, 0.82845172],\n", - " [0.24279153, 0.75720847],\n", - " [0.18915732, 0.81084268],\n", - " [0.17282951, 0.82717049],\n", - " [0.13897531, 0.86102469],\n", - " [0.72746882, 0.27253118]])" - ] - }, - "metadata": {}, - "execution_count": 12 - } - ], + "outputs": [], "metadata": {} }, { "cell_type": "markdown", "source": [ - "## Ещё один взгляд на архитектуру сети\n", - "\n", - "![Архитектура нейросети](https://raw.githubusercontent.com/shwars/NeuroWorkshop/master/images/Cross-Entropy-Loss.PNG)\n" + "You can see that we are now getting probabilities as outputs, i.e. the sum of each output vector is exactly 1. \r\n", + "\r\n", + "In case we have more than 2 classes, softmax will normalize probabilities across all of them. Here is a diagram of network architecture that does MNIST digit classification:\r\n", + "\r\n", + "![MNIST Classifier](images/Cross-Entropy-Loss.PNG)\r\n" ], "metadata": { "slideshow": { @@ -500,20 +386,13 @@ { "cell_type": "markdown", "source": [ - "## Cross-Entropy Loss\n", - "\n", - "* Повсеместно применяется в глубоком обучении\n", - "* Основная идея:\n", - " - трактуем выход модели как распределение вероятностей появления того или иного класса\n", - " - минимизируем вероятность неправильной классификации\n", - "\n", - "Два подхода к пониманию Cross-Entropy Loss:\n", - " * Цена ошибки, которую мы платим за неправильную классификацию, т.е. $-\\log p_y$, где $y$ - правильный класс\n", - " * Разница между двумя распределениями вероятностей. Энтропия $p$ + KL-расстояние между $q$ и $p$:\n", - "$\\begin{align} H(p, q) = & ~\\color{red}{H(p)} + \\color{blue}{D_{KL}(p||q)} \\\\\n", - " = & ~\\color{red}{-\\sum_{c \\in C} p(c) \\log p(c)} + \\color{blue}{\\sum_{c \\in C} p(c) \\log \\frac{p(c)}{q(c)}} \\\\\n", - " = & ~-\\sum_{c \\in C} p(c) \\log q(c)\n", - "\\end{align}$\n" + "## Cross-Entropy Loss\r\n", + "\r\n", + "A loss function in classification is typically a logistic function, which can be generalized as **cross-entropy loss**. Cross-entropy loss is a function that can calculate similarity between two arbitrary probability distributions. You can find more detailed discussion about it [on Wikipedia](https://en.wikipedia.org/wiki/Cross_entropy).\r\n", + "\r\n", + "In our case, first distribution is the probabilistic output of our network, and the second one is so-called **one-hot** distribution, which specifies that a given class $c$ has corresponding probability 1 (all the rest being 0). In such a case cross-entropy loss can be calculated as $-\\log p_c$, where $c$ is the expected class, and $p_c$ is the corresponding probability of this class given by our neural network.\r\n", + "\r\n", + "> If the network return probability 1 for the expected class, cross-entropy loss would be 0. The closer the probability of the actual class is to 0, the higher is cross-entropy loss (and it can go up to infinity!)." ], "metadata": { "slideshow": { @@ -523,7 +402,7 @@ }, { "cell_type": "code", - "execution_count": 13, + "execution_count": null, "source": [ "def plot_cross_ent():\r\n", " p = np.linspace(0.01, 0.99, 101) # estimated probability p(y|x)\r\n", @@ -545,27 +424,14 @@ }, { "cell_type": "code", - "execution_count": 14, + "execution_count": null, "source": [ "def cross_ent(prediction, ground_truth):\r\n", " t = 1 if ground_truth > 0.5 else 0\r\n", " return -t * np.log(prediction) - (1 - t) * np.log(1 - prediction)\r\n", "plot_cross_ent()" ], - "outputs": [ - { - "output_type": "display_data", - "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfEAAADfCAYAAADm4+qPAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nO3dd5hU5dnH8e9NXbAhHXZZiqA0ARUVRVCxoGgswYbR2KImaiyxx+QlGqPGrolGsUQJisZIQIxgrLGCgrogKAIKsoBSVECalOf9455xtsIsO7tnyu9zXec6M3POzNx7dmbu8zznKRZCQERERDJPnagDEBERkW2jJC4iIpKhlMRFREQylJK4iIhIhlISFxERyVBK4iIiIhmqXtQBVFXz5s1Dhw4dog5DRESkVkydOnVZCKFFRdsyLol36NCBKVOmRB2GiIhIrTCz+ZVtU3W6iIhIhlISFxERyVBK4iIiIhkq466Ji9SWDRs2UFxczLp166IORdJMXl4eBQUF1K9fP+pQJMcpiYtUori4mB122IEOHTpgZlGHI2kihMDy5cspLi6mY8eOUYcj6SQEeOghOOgg2HXXWnnL3K5Ov/lmX0QqsG7dOpo1a6YELqWYGc2aNVMNjZS3aBGcfz68+GKtvWVuJ/F334VRo6KOQtKYErhURJ8LqVBRka979661t8ztJN67N8yaBWvXRh2JiIhkungS79Wr1t5SSXzTJpgxI+pIREQk0xUVQfv20KRJrb1lbifxPn18HT97EsliZ599Ni1btqRnz55Rh5LWJk6cyG677Ubnzp255ZZbog5HMklRUSKv1JLcTuKdOsEuu4AaqEgOOPPMM5k4cWLUYaS1TZs2ceGFFzJhwgRmzpzJ6NGjmTlzZtRhSSZYuxY++6xWr4dDrifxOnVgzhy48MKoIxGp0PTp0+nfv/+P9z/44AMGDRq0Ta81cOBAmjZtmqrQ0kqqjtN7771H586d6dSpEw0aNOCUU05h3LhxqQxVstXHH8PmzbWexNVPXCRZBx1U/rGTToILLoA1a2DIkPLbzzzTl2XL4IQTSm97/fWtvmWPHj2YO3cumzZtom7dulx++eXccccdpfYZMGAAq1atKvfc22+/nUMPPXSr75FSl14KH32U2tfs0wfuvnuLu6TqOC1cuJB27dr9uK2goIDJkyen4I+QrBdBy3RQEoeXX4aLL4YJE7xBgkgaqVOnDj169GDGjBnMnj2bwsJC9txzz1L7vPnmmxFFlz5SdZxCCOUeU3cyScpHH8H220MtDwCkJL7ddvDJJ4lWhSKV2VLJuXHjLW9v3jypkndF+vXrx9tvv839999f4TXttCqJb6XEXJNScZwKCgpYsGDBj9uKi4tp27ZtzQUt2aOoyLuW1andq9RK4rvvDmZ+FnXMMVFHI1JOv379OPPMM7nwwgvJz88vt10lcZeK47T33nsze/ZsvvjiC/Lz83nqqad48sknayJcySYhwLRp8LOf1fpb53bDNvDqj86d1c1M0lbXrl1p2LAhV199dbVeZ9iwYey3337MmjWLgoICHnnkkRRFmB5ScZzq1avHX//6VwYPHky3bt046aST6NGjRwqjlKw0bx6sXFnr3ctAJXHXuzd8+GHUUYhU6J577uHmm29mu+22q9brjB49OkURpadUHachQ4YwpKJGiiKViahRG6gk7o48EgYO9O4BImli7ty5dO3albVr13LGGWdEHU7a0nGSyBUV+WXZCAZSUkkc4OyzfRFJI7vssguffvpp1GGkPR0niVxREXTp4g2la5lK4nEheF9fERGRqigqiqQqHdIkiZtZXTP70MyejyyIzp3hsssie3sREclAK1fC55/ndhIHLgE+iTSCdu3UQl1ERKpm2jRfR9AyHdIgiZtZAXAU8HCkgfTuDdOn+9SkIjEVjeAlos+F/CjClumQBkkcuBu4Coi2aXifPn5NfO7cSMOQ9JGXl8fy5cv1gy2lhBBYvnw5eXl5UYci6aCoCJo2hQoGGKoNkbZON7OjgSUhhKlmdtAW9jsPOA+gsLCwZoKJn0UVFcGuu9bMe0hGKSgooLi4mKVLl0YdiqSZvLw8CgoKog5D0kG8UVtEY+xH3cWsP3CMmQ0B8oAdzWxUCOG0kjuFEEYAIwD69u1bM8Wi7t3h2mtht91q5OUl89SvX5+OtTyZgYhkkE2b/DLs+edHFkKkSTyEcC1wLUCsJH5F2QRea/Ly4KabInlrERHJQNOnw9q1UGbGvNqUDtfE08fGjT7T1IoVUUciIiLpbtw4n7XsiCMiCyFtkngI4fUQwtGRBjF1Khx8MIwfH2kYIiKSAcaOhf79oUWLyEJImySeFvbeG9q08X+MiIhIZb74wqewPu64SMNQEi+pTh049liYONGvc4iIiFRk3DhfH3tspGEoiZd1/PGwejW88krUkYiISLoaOxZ23x122SXSMJTEyzroINhxR5gwIepIREQkHS1bBm++GXlVOkTfTzz9NGgA776rAV9ERKRizz8Pmzcriaet7t2jjkBERNLV2LE+adYee0QdiarTK3XddXDbbVFHISIi6WTNGvjvf70UHtFQqyUpiVfmww/hgQdAk1+IiEjciy9676U0qEoHJfHKHXecT/T+8cdRRyIiIuli7FjYeWcYMCDqSAAl8codc4xXlWjgFxERAR+ae/x4OPpoqF8/6mgAJfHKtW4N++8Po0Z5K0QREcltEybAt9/6eCJpQkl8Sy6+2IdiXbUq6khERCRqd93lrdJ/8pOoI/mRuphtyUkn+SIiIrmtqAheew3+/Geolz6pUyXxZBQV+WD3IiKSm+6+Gxo3hnPPjTqSUpTEt2blSthvP7jppqgjERGRKHz9NTz5JJx1lrdMTyNK4luz447w85/DP/4BS5dGHY2IiNS2v/0NfvjB20mlGSXxZFxyCaxfDw8+GHUkIiJSm9at8yR+9NFpOaeGkngyunWDI46A++7zszEREckNo0fDkiVw6aVRR1IhJfFkXXYZfP89TJsWdSQiIlIbQvAGbb16waBBUUdTofRpJ5/uDjsMiothp52ijkRERGrDhAlecHvkkbSY7KQiKokny8wTeAg+prqIiGSvDRvgiiugSxc47bSoo6mUknhVXXyxdzlbuTLqSEREpKaMGAGffOJTUjdoEHU0lVISr6ozzvBGDuo3LiKSnb79FoYPh4MP9smw0likSdzM8szsPTMrMrMZZnZ9lPEkpW9f7zd+110axU1EJBvdeCN88w3ceWfaXguPi7okvh4YFELoDfQBjjCzfhHHtHU33eRj5159ddSRiIhIKs2eDX/5C5x9NvTpE3U0WxVpEg/u+9jd+rElRBhScvLz4aqrYMoU+O67qKMREZFUueoqaNjQS+MZIOqSOGZW18w+ApYAL4UQJkcdU1KuugpmzoQmTaKOREREUuG//4WxY+Haa6F166ijSUrkSTyEsCmE0AcoAPYxs55l9zGz88xsiplNWZou45c3agR5ebB2rfclFBGRzPXtt16F3q2bD+6VISJP4nEhhO+A14EjKtg2IoTQN4TQt0WLFrUe2xZdf71PEP/ee1FHIiIi2+rCC322slGjvJCWIaJund7CzJrEbjcCDgU+jTKmKrv2WmjbFk4/HdasiToaERGpqqee8jHShw+HPfeMOpoqibok3gZ4zcymAe/j18SfjzimqtlpJ3j8cfjsM79OLiIimWPhQvjVr6BfP7jmmqijqbJIx04PIUwD9ogyhpQ4+GC/hnLXXV61Pnhw1BGJiMjWbN4MZ53ls1OOHOldhzNM5kWcrm66CZYtgw4doo5ERESScfvt8NJLPl94ly5RR7NNlMRTJS/Pz+TAz+42bkzr8XZFRHLaCy949fmJJ8L550cdzTaL+pp49gkBhg3zrgoh/cetERHJOZ9+6r/TvXvD3/+e9kOrbomSeKqZ+QTyTzzhs9+IiEj6+PZbn9SkYUMYNw622y7qiKplm5O4mWXGyGpR+O1v4eSTvarmhReijkZERMAvcw4bBvPmwZgxUFgYdUTVVp2SePeSd8xsVpn7J1fjtTObGTz6qA+ef8opPsa6iIhEJwQf0OXFF+H+++GAA6KOKCWqk8TLXvBtWeb+Q9V47czXuDGMH+8tHjdujDoaEZHcFQJceSWMGOEDdP3iF1FHlDKpbJ1eNqlnbkuBVMnP91J4vNHEqlWwww7RxiQikmtuvBHuuAMuugj+9Keoo0mp6pTEzcy2lJHUNBsSCfyee7zBW3FxtPGIiOSSu++G//s/OOMM/x3O4JboFalOEq8LfGdmC8xsIpBnZseZWX6KYssuBxwAy5fDoYf6MH8iIlKz/vY3H01z6FB4+GGok30dsqrzF20P9AauA2YBU4F/AF+a2SIgc6aBqQ177eUt1Rct8oQ+Z07UEYmIZK9bboELLoCjj/Yuvxk4pGoytvmvCiFsBj6OLSPB69eBrsBeQGZNBVMbDjgAXnvNx1YfOBBmzdI1chGRVArBG6/9+c9w6qnw2GNQv37UUdWYbU7iZjYIODyE8OO0LyGEAHwSW0ZVP7wstNde8OabMGmSEriISCpt2uTdyB580Gcm++tfs7IKvaTq/HVXADMr2mBmw8ws83vR15Ru3XzmHICJE+Hf/442HhGRTLd6tY+D/uCDXhK/776sT+BQvSS+B/BMJdtaA3+sxmvnhhB8+tKf/hRuvlljrYuIbIsFC/xy5bhx3hr9ppuyrhV6ZaqTxBsB6yrZNgYYWI3Xzg1mMHasDwP429/Cz38O6yo7pCIiUs5778E++8DcufD883DJJVFHVKuqk8Q/xxuwlRNCmA80q8Zr545Gjbzl5I03wqhRMGiQVwuJiMiWjRoFBx7ov6PvvgtHHhl1RLWuOkl8FPAXM8sru8HMWgLfVeO1c4sZXHcdPPOMT43XuHHUEYmIpK+1a+G88+D002Hffb003qNH1FFFojpJ/C/AGmCqmf3UzOoCmFkD4DbgtRTEl1tOOMEHJzCDmTPhhhu8taWIiLjZs2G//eChh7wB28svQ/PmUUcVmW1O4iGEDcAQYCLwBLDazL4AvgX2BX6bkghz1TPPwPDhcPjhPkCMiEiuGz3au+kuWAD/+Y83YMvSQVySVa329yGE9SGEy4EC4AzgbuBUYI8QgsYWrY7hw30603ffhZ494emno45IRCQa33zj0zqfeqr/Hn70EQwZEnVUaSFVneh2A14PIdwTQhgXQlibotfNbWedBUVFsOuu/gF+8smoIxIRqV0vvgi77w7PPusNgN94A9q1izqqtLHVJG5mO5vZr8zsTDOrbP/3gN/Hhl2VVOrSBd56ywcuGDrUH1u2LNqYRERq2ooV8MtfwhFHQJMm3njtuutyvvq8rGRK4hOAvwKPAOdWtEMIYSPwQmXbpZrq1fOB/Bs2hO+/92tCJ54IixdHHZmISOqNGeMjWz70EFx+OUydCnvsEXVUaSmZJL4DsAvwDt43vDKTgfOr8uZm1s7MXjOzT8xshpnlVi/9bdGwIZx/Powf7x/yESNg8+aooxIRqb7iYjj+eK91bNUKJk+G22+HvHI9mSUmmSQ+M4QwL4QwIITw0hb26wz0MbOqTBezEbg8hNAN6AdcaGbdq/D83FO/vo/uNm2an5mefz7sv783/BARyUTr1/usY127+nwSf/6zV5/37Rt1ZGkvmSQ+3sxuSGK/y2Lr7ZJ98xDC4hDCB7Hbq/DZz/KTfX5O23VXePVVePxxaN8edt7ZH9+wIdq4RESq4oUXvOHaNdfAoYfCjBlw1VVZPX1oKm01iYcQRgJ7m9ksM7vBzA43s4rm0BwArAohbNNIbWbWAZ9UZXIF284zsylmNmXp0qXb8vLZyczHW3/6ab+9aBF06uTVTxqDXUTS2YwZcNRRvpjBhAk+l0SnTlFHllGS7WI2FJgK/A5v6PaNmU0ysz+aWXz89PXArG0Jwsy2B54FLg0hrCy7PYQwIoTQN4TQt0WLFtvyFrnhhx/8jPbKK71a6okndL1cRNLLokVw7rnQqxe8/TbcdhtMn+6t0KXKkkriIYQ1IYRTgQOAR4FFwD7AdcB7ZvYGsAxYUNUAYtfQnwWeCCGMqerzpYQOHbxq6uWXoVkzOO00H1d4/fqoIxORXPfdd/D733u32ccfh4sv9pnHrrgCGjSIOrqMVaUOdyGEd/BW6vHq7wNLLB3xIVeTFutX/gjwSQjhzqo8V7bgkEPg/fd9iMKiIm/RDl59laOTBIhIRFatgnvugTvu8ER+8sk+XKqqzVOiOmOnzwshPB5CODuEsAt+Pbuqrar6A6cDg8zso9iisfRSoU4d+NnP4NZb/f60aT5c4aBBPuKRiEhNWrXKW5l37Ogl8IED4cMP4amnlMBTKFXDrhJCKAKureJz3gohWAihVwihT2x5IVUxSQldusDdd/vsaAceCAMG+HCGIUQdmYhkk2++geuv914z11wDe+/t3cXGjYM+faKOLuukLIkDhBCmp/L1JIUaNYJLLoHPP4d774V58+CnP4Vvq3QFRESkYgsXwtVXe/L+wx+85P3ee97qfO+9o44ua6U0iUsGaNwYfv1rb1Dy6qvQtKmXxk86ycdnX7066ghFJJNMnw5nnunV5rffDkcf7Zfvxo5V8q4FSuK5qkEDb7kO3thkwQK46CIoLPRJBoqLo41PRNLX5s1ewj7iCO8q9swzPlnJ7NneoHb33aOOMGcoiYuP9vbOOz5b2sCBcMst3l3t1VejjkxE0snKlX45rmtXn8+7qAj+9CcvBNx7rxqsRUBzuokzg/79fZk3Dx5+GPbbz7c9+aS3ND31VNihosH6RCSrTZsGDzwA//iHz6TYr583Xhs6VH28I6aSuJTXoQPceKM3hgN49lmvKmvbFn71K/joo0jDE5FasGYNjBzpEyz17g2PPuozjE2eDO++C8OGKYGnASVx2bp//cu/tEOHwmOP+expv/511FGJSKqF4HN3X3CBn7SfcQYsXw533unDpY4cCfvsE3WUUoKq02XrzLz6rF8//zI/8YRfEwNvAHfttT4Ry6BBULdutLGKSNUtWeKXzR5/3Gva8vLghBPgnHN8XAmzqCOUSiiJS9U0bVq6FD59Ojz/PIwaBfn5Pl77z38O3TUtvEhaW7fOv7sjR3pL840bYa+9vKvpqadCkyZRRyhJsJBhI3b17ds3TJkyJeowpKR162D8eD+LnzjRH/vqK2je3Oc317zAIulh0yZ47TUvdT/7rLc2b9s2cfKtuRXSkplNDSH0rWibSuJSfXl5cOKJvnz9tU8v2Ly5bxsyxGdRO/lkr55r1SraWEVyzebN3qbl6ae9P/dXX3kvk6FDvXHaIYfoMlgGU8M2Sa1WrXw4V/BGMgcf7A1jLrrIz/gPOQSeey7aGEWyXTxx/+Y3PgzqAQfAiBHebfSZZ/xk++9/h8MPVwLPcEriUnPM4Le/9SlQP/44MRLcnDm+fcUKHyBCo8OJVN/GjV5VftFF0K6ddw277z7Yc09vs7J0KYwZ4zVi8e6jkvF0TVxqVwh+Xa5ePR9b+fjj/fG+feG443zp3l2tYUWSsWYNvPSSf5fGj/dar0aNfDjUoUN9HPOddoo6SqmmLV0TVxKXaM2aBf/+t09TOGmSP/bZZz516tKl3kJWDeNEEhYv9lbl48fDyy/D2rX+PTnqKD8JPvJI2G67qKOUFFISl8yweDG88oq3lAXv5vKf/3ip4qijfN2yZbQxitS2zZt9AJYXXvDkHf/9a98efvITT9wDB+pkN4updbpkhjZtEgkcfLSoxo09kf/zn/7YySfDU0/57RBU7S7ZadkyryafONH7cC9dmhh06aabPHn36KHPvyiJSxobPNiXzZt9tqQXXkgMQLFhA+y2m19LHzzYW9m2axdtvCLbasMGH5P8xRd9mTLFT1KbNfMaqCFD/DMe77opEqMkLumvTh0fr32PPRKPrVrl3dcmTvQuM+BJ/dZb4ZhjoolTJFkhwCef+DXtl16C11/32cHq1EnMEDZ4sI+gpi5gsgVK4pKZmjaFRx7xH8MZM/yH8KWXfG508K4211zj/dIHDfIpVtWtRqI0f763+Xj1VV8WL/bHO3f2y0iHHeYnpvHPsEgSlMQls5lBz56+XHZZ4vF4N7bbboObb/YpE/fd10etatMmungldyxY4CeTr7/u63nz/PFWrfzEctAgP8ns2DHKKCXDKYlLdjr0UF9WrYI33/Qf0ffegxYtfPtVV3mXtoEDfZam/faD7bePNmbJXCHA55/DG2/A//7nSzxpN23qn7Hf/MZL2mqQJimkLmaSm+6910ex+uADL7XXrev9a8eP9+2rV6uvrVRu0yafwe/NN315661E9Xjz5n5yGD9B7NXLr3WLbKO07WJmZo8CRwNLQgg9o4xFcszFF/uyahW8847/CDdokNjevbvf79/fl/33h27d9GOcq1au9Nbjb7/ty6RJ3hANvFfEwQfDgAGeuLt1U0lbak2kJXEzGwh8D4xMNomrJC41btMmuPPOxA/2smX++MUXwz33+PZXXoF99tGcy9lo82YfNXDSJJ9E5N13fez/EPwkbvfdEyd2AwZAYWHUEUuWS9uSeAjhDTPrEGUMIuXUrQtXXulLCDB7tv+Q77abb58xw7v/AHTt6l2C9t3XB+DIz48ubtk2S5fC++97STu+fPedb9tpJ///nnCCt5vYd1/Yccdo4xUpQQ3bRLbEDHbd1Ze4zp29f++kSb48/zw89pgPg5mf7wnhiSdg77196dxZ1fDp4vvvvR3E++/7gCqTJ8MXX/i2OnW8l8OJJyZOzHQJRdJc5A3bYiXx57dUnW5m5wHnARQWFu41f/782glOJBkheEvkVq18mNjHHoMLLvCJKcBLc337ekO61q398bw8XTetaWvW+Eh/U6b42ONTpvgAK5s3+/Z27TxR77OPL3vuCTvsEG3MIhVI6wlQkkniJemauGSEjRu92v39932ZNs27HTVoAJdeCiNHetIouXTposS+rVat8oT9wQeesD/4wBP2pk2+vWVLP5GK14707esnXSIZIG2viYtkrXr1oHdvX37xi9LbBg3y0vjUqd5Q7ocffACaRYt8+6OP+nqPPbyVfMOGtRt7uvv6a/joI/jww8QyZ47XiIAn57328tm9+vb12/n5OkGSrBR1F7PRwEFAczMrBoaHEB6JMiaRGnfMMYnx3X/4AWbO9MQUd++9XqoEPxno1g2GDoXhw/2x777LjVbxGzZ4K/GiotLLV18l9unQwU92Tj/d13vtpRH5JKdE3Tp9WJTvLxK5Bg2gT5/Sj33wgZcsi4q8xFlU5Nd3wa/nFhb6tfdevRJL//6wyy61H38qhOAnMdOn+2WHadP89syZsH6979OggddKHHGEH68+ffzv1jjjkuMivyZeVbomLjlt/Xq4/35P7NOn+3X39evhd7+DP/4RVqyAc89NjCffs6cn93SZCWvFCo/54499mT7d1/G++OAl6fjJSe/evu7aFerXjy5ukQjpmrhItmjYsPRELxs3eqk9PkTs4sVekv/XvxLXiPPy4PHH4aSTYMkS7xbXo4dXRddUcl+1ykvSM2aUXoqLE/tst52fZBx3nA+g0rOnJ2zNmS2SNCVxkUxWr56XUuO6dvWkvnq1t86Ol3i7d/ft//ufJ3Pw5N61q2+7/nrvz75mjZd4ky31Ll/u7xNfZs70ZcGCxD7x9znwwNI1BIWF6oMtUk1K4iLZaLvtvGV23zI1cEce6aPPzZiRSLpvv50okT/4oM/w1rmzN6jr2tVHquvXz/vCf/IJfPqpL5984iX7uEaN/DkHHugnBt26eYm/U6f0qc4XyTJK4iK5ZPvtPSH361d+26pV3up9yBCYNctHpfv3v8vvV6+eV3l36uSt7Pv1825z7durZC1Sy5TERXLJ+vU+7/Xs2d59q+QSn0oTPBl37AgHHOCjzMWHIP3f/+A///EkHx929rnnEl3kbr/dxyKPD1XbpYv321YfbZEaoSQukm1++MGrvmfP9uvj8fVnn8H8+YlhR8FL1LvuCocf7tXm8aVz54oHmRk4EH7/e280t2yZJ/Nvv01sf+cdH0t+w4bEYwMGwBtv+O0HHvBq986dPcG3aKEEL1INSuIimWjNGi9Rz53rCTq+njOnfKLecUdPmPvuC6edlighd+kCTZtu2/ubeQJu0aL042PGeIv5+fP95GH27NLjkQ8fXvo6+g47+Ih2d97p9594AgoKvFtc27aqnhfZCvUTF0lHIXi19OefJ5L13LmJ2/EhWuOaNEmUbjt39iQYT9TNm6dPabdkLUH8xKN3bzjnHD8xiXeVA2/V3rGjjzV/3nleun/xRf/bOnTwEr1IDlA/cZF0tHq1J7QvvvDl889Lr7//vvT++fmewAYP9kZlu+ySSNjbWqKubQ0alJ/aNS4vz//2kjULc+d6YzzwY/WTnyT2z8/3JH/ttd4Yb+VKHwSnY0eV4iVnKImL1JR16+DLLxOJuuy6ZLUy+FCqnTr5MmiQJ+f4/VwoecYb03XsCIcdVn57u3bePS5eIxE/2Ykn6ylT4JBD/HaDBt5avkMHuOkm72q3ZInv36GDz2qWLrUTItWgJC6yrVav9iQ9f74n5rLrkq29wQdQKSz0JHXMMYmEFV+UWLYsL6/y7nHgE6BMnJio2YifMMX7qE+YAGeemXiteJL/29/8+M+b55cp2rf3oV9VkpcMoCQuUpF46+v58xOJuuy65HjfkEjS7dv7RB0dO3qSiC9t22rQk5q0885+qaEygwfD+PGerOPL/PmJGo5Ro7zlPfj/sl07/1+OGeNtDoqKvJ1CYaFvy/aaEckISuKSm1av9qFBFyzwpFx2/eWXXh1eUuPG/qNeWOhTXsZLcu3bJ0pvStLpq3VrOProyrefdZb/X0vWqBQXJ1rX338/jBiR2L9lS7/U8c47XoPy8sve3a5dO19at9bnQWqcWqdL9lm7FhYu9IRcXJxI1iWXkn2bwX+E27RJlLIKC0sv7dt74zFVd+euRYu8VX38JG/+fD/RGznStw8Z4lX2cfXqebe+t97y+48+6iePBQWJRN+ihartZavUOl2yx6pVnpgXLvR1Rcvy5eWf17RpIjn375/4EY0n7fx8bziTFtEAAA50SURBVAwlUpm2bX2pzJNPlq/VKVnlft99PsNcSQMH+ih4AFde6ev8fE/0BQV+SaZVq9T+HZJVlMQlPWzc6K2HFy4svSxalEjYCxd6Ei+rWbNEUt5/f/8RbNcu8UNYUOBV4SI1qUkTX3r1qnj7++97O4qStUMluwa+9ppPTFPyMs4pp8Do0X570CDYaSf/fLdt6+u99vIZ4SRnKYlLzdq82UvGixZ5a+1FixJLPEkvWgRffVV6lDHw6sg2bfzHqkcPHxq0oKB0SaVtW29pLJLu6tTx6+gtW8Kee5bfPmWKN6j85ptErVKzZr5twwavKZozx0vu8ctBV1wBt93mYwq0bp1I8PHl2GN9/PsNG7x2oE0bndBmGSVx2TabN3tL3cWLSy8lk3X8sZLjaMc1a5YoTey+u69LljDy8/3HTtcLJZeY+XejWTMfyS6ufn3vPhe3Zo1/x+LV9Rs3wrnn+onx4sU+Mc3ChV4jdcABnvzjc8o3aeLfszZt4Jpr4NBD/bv86quJx1u3TgyyI2lNSVxKW7fOS8WLF/s6fjt+P377669h06byz995Z/8RaNvWJ9Jo0yZRmo6XDlq3VulZpDoaN/bR+uKaNIG77iq9TwiJ72jLlvD3vyeSfPxEO779o4+86r6k7bf37nWHHebbH3ss8X1u3drXXbrouxwxJfFcsHGjn2nHk/LXX1d8e/FiWLGi/PPj1YDxL26vXokvc3xRchZJL2Z+SQq8ZB8f6KYiBxwA06eXr1nr0MG3z5kDjzxSfijgKVP8uvzo0T4NbevWiaVVKzjjDL+Ov3Kln1TsuKN6eKSYknim2rDBG4J9/fXWl2XL/AtU1g47+BetTRuv0j7ssMT9+Nl269beDaaePioiWatRI28gV1kjuRNO8OX770uf9MfHwN9+e/+tWLzYS+3xmrqTT/YkfscdcMMNfpLfqpUvrVt78m/c2Kv/v/wysa1VK69dUMLfKvUTTxch+BcknpiXLEks8fsl1998U/HrNGpU+kw4/mWpaF1yxigRkVSJN2ht1sxr8iZP9v7y8dq/eOHi/fd9+y9+4SX9knbayRvwmflUtdOne41gq1a+zs+Hgw9OvF8Wt59J637iZnYEcA9QF3g4hHBLxCGlztq1Xo29ZEn5dUXL+vUVv06TJokPbvfu/sGNJ+j4hzq+qDGKiEStTp3Sc83vu68vlbn1VrjkktI1iOvXJ0riX3wBr7zij//wgz/WubMPvgM+pO7UqYnW/y1bQp8+8Lvf+fbXXvPXatnS42raNGtG04u0JG5mdYHPgMOAYuB9YFgIYWZlz4msJB6Cj7a0dGnyS9nrR3ENGyY+aC1aJJJx2futWvl9DUIiIuK/wytWeKFn7dpEC/4HH/SSerymculSr+ofO9a3d+0Ks2YlXqdOHTj+ePjXv/z+hRf6ukWLxLLbbonX37Qp0qSfziXxfYA5IYTPAczsKeBYoNIknlLxkvGyZYklfr/kOn677FjacQ0blv7nd+ni63hSLpmgW7Twa9G61iMiUjVmiUF1Sjr//C0/b8wYv15fska0Y8fE9smTvbRf8jLlGWd4i/wQvIazUSP//W7e3NdDh8Lpp3uCf+IJf7x580QX2VoSdRLPBxaUuF8MbKHOJcX69Ck/XSR4C8r4P6ttWz8bi9+P/wNLLkrKIiLpq3v3RD/5isRrdzdu9Gv5S5cm+uBv2uT96UsW7L74IpE7vv3WE37cMcfAuHE183dUIOokXlHmK1e/b2bnAecBFBYWpu7d77gjce0mnqCbN1f1tYhILqpXL9G+qORjw4dX/pydd/YuePEkX7aWoIZFncSLgXYl7hcAi8ruFEIYAYwAvyaesncfNixlLyUiIjmobl3YZRdfIhB1m/z3gS5m1tHMGgCnAM9FHJOIiEhGiLQkHkLYaGYXAS/iXcweDSHMiDImERGRTBF1dTohhBeAF6KOQ0REJNNEXZ0uIiIi20hJXEREJENl3NjpZrYUmF+Nl2gOLEtROLlOxzI1dBxTR8cydXQsU6e6x7J9CKFFRRsyLolXl5lNqWz4OqkaHcvU0HFMHR3L1NGxTJ2aPJaqThcREclQSuIiIiIZKheT+IioA8giOpapoeOYOjqWqaNjmTo1dixz7pq4iIhItsjFkriIiEhWyMokbmZHmNksM5tjZtdUsL2hmT0d2z7ZzDrUfpSZIYlj+Rszm2lm08zsFTNrH0WcmWBrx7LEfieYWTAztQyuRDLH0sxOin02Z5jZk7UdY6ZI4jteaGavmdmHse/5kCjiTHdm9qiZLTGzjyvZbmZ2b+w4TzOzPVPyxiGErFrwMdjnAp2ABkAR0L3MPhcAD8RunwI8HXXc6bgkeSwPBhrHbv9Kx3Lbj2Vsvx2AN4BJQN+o407HJcnPZRfgQ2Dn2P2WUcedjkuSx3IE8KvY7e7AvKjjTscFGAjsCXxcyfYhwAR8Cu5+wORUvG82lsT3AeaEED4PIfwAPAUcW2afY4HHY7f/BRxiZhXNbZ7rtnosQwivhRDWxO5OwqeTlfKS+VwC/BG4FVhXm8FlmGSO5bnAfSGEbwFCCEtqOcZMkcyxDMCOsds7UcF00QIhhDeAb7awy7HAyOAmAU3MrE113zcbk3g+sKDE/eLYYxXuE0LYCKwAmtVKdJklmWNZ0jn4maaUt9VjaWZ7AO1CCM/XZmAZKJnP5a7Armb2tplNMrMjai26zJLMsfwDcJqZFeOTVf26dkLLOlX9PU1K5LOY1YCKStRlm+Ans49U4TiZ2WlAX+DAGo0oc23xWJpZHeAu4MzaCiiDJfO5rIdXqR+E1w69aWY9Qwjf1XBsmSaZYzkMeCyEcIeZ7Qf8I3YsN9d8eFmlRvJONpbEi4F2Je4XUL7658d9zKweXkW0pWqQXJXMscTMDgWuA44JIayvpdgyzdaO5Q5AT+B1M5uHXzN7To3bKpTsd3xcCGFDCOELYBae1KW0ZI7lOcA/AUII7wJ5+FjgUjVJ/Z5WVTYm8feBLmbW0cwa4A3Xniuzz3PAGbHbJwCvhljLAyllq8cyVgX8IJ7Add2xcls8liGEFSGE5iGEDiGEDnj7gmNCCFOiCTetJfMdH4s3usTMmuPV65/XapSZIZlj+SVwCICZdcOT+NJajTI7PAf8PNZKvR+wIoSwuLovmnXV6SGEjWZ2EfAi3vLy0RDCDDO7AZgSQngOeASvEpqDl8BPiS7i9JXksbwN2B54JtY28MsQwjGRBZ2mkjyWkoQkj+WLwOFmNhPYBFwZQlgeXdTpKcljeTnwkJldhlf/nqlCT3lmNhq/fNM81n5gOFAfIITwAN6eYAgwB1gDnJWS99X/QkREJDNlY3W6iIhITlASFxERyVBK4iIiIhlKSVxERCRDKYmLiIhkKCVxERGRDKUkLpKFzGw3M/uDme0WdSwiUnPUT1wky8Rm5HsT6IVPLTlQg3OIZCeVxEWyzzlAB6AP0JHEEMMikmWUxEWySGyc8JuBs0IInwNnA7eaWdMK9u1pZhvN7LBqvN/rZvb6Nj73ODP7wcw0MYnINlJ1ukiOMrP/Ag1DCNs8fWw8gYcQDtrG508F5ocQfrqtMYjkMpXERXJQbF7ow4A7Iw7lHuB4M+sRcRwiGUlJXCQ3XQAsx2dWitIYfEanX0Ych0hGUhIXyRJmdr2ZBTMbZGajzexrM1tjZu+Z2cAS+9UDjgNeCiFsKPMaD8Reo20Fr79b7Br2PVuJo5GZFZvZl2bWsMy2h81sk5mdAhBC+B5vSX/itv/lIrlLSVwke/TB585+CmgE/B6vLu8OTDCzNrH99sLngH+vgtd4N7bep4JtdwErgT9sKYgQwlp8LuV2eIkfADO7GW85/+sQwlNl3rOVmXXd0uuKSHlK4iLZow9QF7glhHBcCGFECOF3wEVAY+Ck2H7dY+u5FbzGpNi6VBI3s6OAI4H/CyF8m0QsjwEzgGvNbHszuxS4BhgeQri/zL7xOHRdXKSKlMRFsoCZ7QwUAm+FEMo2Vnsltu4QW7eIrb8p+zohhFmxx39M4mZWHy/Rfww8mEw8IYRNeNJuAYyNPf8vIYQbKth9eWzdMpnXFpGEelEHICIpsUds/XAF2+In69/H1vF+pVbJa00C+puZxUZ6uwTYFTg0lpyTEkJ43sw+AA7Bq/gvqWTXeBzq7ypSRSqJi2SHPrH1lAq27RtbfxhbL42tyw0AEzMJ2AnYzcxa4tfWx4YQXqlk/wqZ2Ukl4lq1haFf43EsrWS7iFRCJXGR7BBPlhsr2PYbvIr8v7H7H8fWlY2UVrJx20CgIXB5VYIxs8OBfwD/BjYAZ5vZXSGETyrYvXOZuEQkSSqJi2SHeHV6qdHXzOwcvCR+Y6w7F3iJfCXQr5LXmgxsxluSnwXcHRvCNSlmti/e//tt4GfA72Kvd3MlT+kHfB27Hi8iVaCSuEiGi/XF7oon57vMrD0wDzgIGAb8E7g7vn8IYZOZjQGONbOGIYT1JV8vhLDKzGbipfCvgD9VIZZuwH+Az4DjYq8918weAX5pZv1DCG+X2H97YADwaJX/cBFRSVwkC/TET8jvBK4GTgX+ipfOLwOGVXA9+m/AzsDRlbxmvA/5tSGEVckEYWaFeJX9CuDIEMLKEptvANYCt5Z52lC8+1tSrd5FpDRNgCKS4WJV5g8DvUMI06rwvInAdiGEAWUerw98Sqyr2ZbmItcEKCLRUklcJPPtgTce+7SKz7sc2C/WCK2kK/B5yH+9pQReXWZ2HLA7XnsgIttA18RFMl8fYFYI4YeqPCmEMIPYb0BsvvHBQC/gSuDOEMKkLTy92kIIY4EGNfkeItlOSVwkg5mZ4Yn3+Wq+1GDgSWAJPkb6NdV8PRGpBbomLiIikqF0TVxERCRDKYmLiIhkKCVxERGRDKUkLiIikqGUxEVERDKUkriIiEiGUhIXERHJUEriIiIiGUpJXEREJEP9P5VmBCAoNxpqAAAAAElFTkSuQmCC", - "text/plain": [ - "
" - ] - }, - "metadata": { - "needs_background": "light" - } - } - ], + "outputs": [], "metadata": { "scrolled": true, "slideshow": { @@ -573,9 +439,16 @@ } } }, + { + "cell_type": "markdown", + "source": [ + "Cross-entropy loss will be defined again as a separate layer, but `forward` function will have two input values: output of the previous layers of the network `p`, and the expected class `y`:" + ], + "metadata": {} + }, { "cell_type": "code", - "execution_count": 15, + "execution_count": null, "source": [ "class CrossEntropyLoss:\r\n", " def forward(self,p,y):\r\n", @@ -583,69 +456,25 @@ " self.y = y\r\n", " p_of_y = p[np.arange(len(y)), y]\r\n", " log_prob = np.log(p_of_y)\r\n", - " return -log_prob.mean()\r\n", + " return -log_prob.mean() # average over all input samples\r\n", "\r\n", "cross_ent_loss = CrossEntropyLoss()\r\n", "p = softmax.forward(net.forward(train_x[0:10]))\r\n", "cross_ent_loss.forward(p,train_labels[0:10])" ], - "outputs": [ - { - "output_type": "execute_result", - "data": { - "text/plain": [ - "1.429664938969559" - ] - }, - "metadata": {}, - "execution_count": 15 - } - ], + "outputs": [], "metadata": {} }, { "cell_type": "markdown", "source": [ - "## Задача минимизации\n", - "Описав нейронную сеть как модель $f_\\theta$ и функцию ошибки $\\mathcal{L}(Y,f_\\theta(X))$, можем рассмотреть $\\mathcal{L}$ как функцию $\\theta$ на всем множестве обучающей выборки $\\mathcal{L}(\\theta) = \\mathcal{L}(Y,f_\\theta(X))$\n", - "\n", - "В этом случае задача обучения сети будет формулироваться как задача минимизации $\\mathcal{L}$ по $\\theta$:\n", - "$$\n", - "\\theta = \\mathrm{argmin}_{\\theta} \\mathcal{L}(Y,f_\\theta(X))\n", - "$$\n", - "\n", - "Минимизацию можно осуществлять разными методами, например, стохастическим градиентным спуском (stochastic gradient descent, SGD)" - ], - "metadata": { - "slideshow": { - "slide_type": "slide" - } - } - }, - { - "cell_type": "markdown", - "source": [ - "## Реализация нейронных сетей\n", - "\n", - " * Вручную\n", - " * С использованием готовых фреймворков\n", - " - PyTorch\n", - " - Tensorflow\n", - " - Chainer\n", - " - [Microsoft Cognitive Toolkit](http://cntk.ai)" - ], - "metadata": { - "slideshow": { - "slide_type": "slide" - } - } - }, - { - "cell_type": "markdown", - "source": [ - "## Вычислительный граф\n", - "\n", - "\n" + "> **IMPORTANT**: Loss function returns a number that shows how good (or bad) our network performs. It should return us one number for the whole dataset, or for the minibatch. Thus after calculating cross-entropy loss for each individual component of the input vector, we need to average (or add) all components together - which is done by the call to `.mean()`.\r\n", + "\r\n", + "## Computational Graph\r\n", + "\r\n", + "\r\n", + "\r\n", + "Up to this moment, we have defined different classes for different layers of the network. Composition of those layers can be represented as **computational graph**. Now we can compute the loss for a given training dataset (or part of it) in the following manner:" ], "metadata": { "slideshow": { @@ -655,37 +484,42 @@ }, { "cell_type": "code", - "execution_count": 16, + "execution_count": null, "source": [ "z = net.forward(train_x[0:10])\r\n", "p = softmax.forward(z)\r\n", "loss = cross_ent_loss.forward(p,train_labels[0:10])\r\n", "print(loss)" ], - "outputs": [ - { - "output_type": "stream", - "name": "stdout", - "text": [ - "1.429664938969559\n" - ] - } - ], + "outputs": [], "metadata": {} }, { "cell_type": "markdown", "source": [ - "## Обучение сети\n", - "\n", - " * Для обучения сети необходимо предъявлять ей примеры, считать ошибку и подстраивать коэффициенты\n", - " * В соответствии с принципом градиентного спуска, необходимо расчитывать изменение коэффициентов в соответствии с градиентом функции $\\nabla f_\\theta$\n", - " * Итерация обучения выглядит так:\n", - " $$\\begin{align}\\def\\L{\\mathcal{L}}\n", - " W^{i+1}&=W^i-\\eta\\frac{\\partial\\L}{\\partial W}\\cr\n", - " b^{i+1}&=b^i-\\eta\\frac{\\partial\\L}{\\partial b}\n", - " \\end{align}\n", - " $$" + "## Loss Minimization Problem and Network Training\r\n", + "\r\n", + "Once we have defined out network as $f_\\theta$, and given the loss function $\\mathcal{L}(Y,f_\\theta(X))$, we can consider $\\mathcal{L}$ as a function of $\\theta$ under our fixed training dataset: $\\mathcal{L}(\\theta) = \\mathcal{L}(Y,f_\\theta(X))$\r\n", + "\r\n", + "In this case, the network training would be a minimization problem of $\\mathcal{L}$ under argument $\\theta$:\r\n", + "$$\r\n", + "\\theta = \\mathrm{argmin}_{\\theta} \\mathcal{L}(Y,f_\\theta(X))\r\n", + "$$\r\n", + "\r\n", + "There is a well-known method of function optimization called **gradient descent**. The idea is that we can compute a derivative (in multi-dimensional case call **gradient**) of loss function with respect to parameters, and vary parameters in such a way that the error would decrease.\r\n", + "\r\n", + "Gradient descent works as follows:\r\n", + " * Initialize parameters by some random values $w^{(0)}$, $b^{(0)}$\r\n", + " * Repeat the following step many times:\r\n", + "\r\n", + " $$\\begin{align}\r\n", + " W^{(i+1)}&=W^{(i)}-\\eta\\frac{\\partial\\mathcal{L}}{\\partial W}\\\\\r\n", + " b^{(i+1)}&=b^{(i)}-\\eta\\frac{\\partial\\mathcal{L}}{\\partial b}\r\n", + " \\end{align}\r\n", + " $$\r\n", + "\r\n", + "During training, the optimization steps are supposed to be calculated considering the whole dataset (remember that loss is calculated as a sum/average through all training samples). However, in real life we take small portions of the dataset called **minibatches**, and calculate gradients based on a subset of data. Because subset is taken randomly each time, such method is called **stochastic gradient descent** (SGD).\r\n", + " " ], "metadata": { "slideshow": { @@ -696,15 +530,15 @@ { "cell_type": "markdown", "source": [ - "## Обратное распространение ошибки\n", - "\n", - "\n", - "\n", - "$$\\def\\zz#1#2{\\frac{\\partial#1}{\\partial#2}}\n", - "\\begin{align}\n", - "\\zz{\\L}{W} =& \\zz{\\L}{p}\\zz{p}{z}\\zz{z}{W}\\cr\n", - "\\zz{\\L}{b} =& \\zz{\\L}{p}\\zz{p}{z}\\zz{z}{b}\n", - "\\end{align}\n", + "## Backward Propagation\r\n", + "\r\n", + "\r\n", + "\r\n", + "$$\\def\\L{\\mathcal{L}}\\def\\zz#1#2{\\frac{\\partial#1}{\\partial#2}}\r\n", + "\\begin{align}\r\n", + "\\zz{\\L}{W} =& \\zz{\\L}{p}\\zz{p}{z}\\zz{z}{W}\\cr\r\n", + "\\zz{\\L}{b} =& \\zz{\\L}{p}\\zz{p}{z}\\zz{z}{b}\r\n", + "\\end{align}\r\n", "$$" ], "metadata": { @@ -716,14 +550,20 @@ { "cell_type": "markdown", "source": [ - "## Обратное распространение ошибки\n", - "\n", - "\n", - "\n", - " * Не повторяем одинаковые вычисления\n", - " * Вычисляем ошибку на каждом узле начиная с конца\n", - " * Обратное распространение ошибки\n", - " * Все вычисления фреймворк берёт на себя" + "To compute $\\partial\\mathcal{L}/\\partial W$ we can use the **chaining rule** for computing derivatives of a composite function, as you can see in the formulae above. It corresponds to the following idea:\r\n", + "\r\n", + "* Suppose under given input we have obtanes loss $\\Delta\\mathcal{L}$\r\n", + "* To minimize it, we would have to adjust softmax output $p$ by value $\\Delta p = (\\partial\\mathcal{L}/\\partial p)\\Delta\\mathcal{L}$ \r\n", + "* This corresponds to the changes to node $z$ by $\\Delta z = (\\partial\\mathcal{p}/\\partial z)\\Delta p$\r\n", + "* To minimize this error, we need to adjust parameters accordingly: $\\Delta W = (\\partial\\mathcal{z}/\\partial W)\\Delta z$ (and the same for $b$)\r\n", + "\r\n", + "\r\n", + "\r\n", + "This process starts distributing the loss error from the output of the network back to its parameters. Thus the process is called **back propagation**.\r\n", + "\r\n", + "One pass of the network training consists of two parts:\r\n", + "* **Forward pass**, when we calculate the value of loss function for a given input minibatch\r\n", + "* **Backward pass**, when we try to minimize this error by distributing it back to the model parameters through the computational graph." ], "metadata": { "slideshow": { @@ -734,31 +574,31 @@ { "cell_type": "markdown", "source": [ - "### Реализация обратного распространения\n", - "\n", - "* К каждому узлу добавляем функцию `backward`, которая вычисляет производную и значение ошибки\n", - "* После вычисления производных, реализуем обновление весов в соответствии с формулой выше\n", - "\n", - "Например, для линейного узла $z = x\\times W+b$:\n", - "$$\\begin{align}\n", - "\\frac{\\partial z}{\\partial W} &= x \\\\\n", - "\\frac{\\partial z}{\\partial b} &= 1 \\\\\n", - "\\end{align}$$\n", - "\n", - "Соответственно, если на вход пришла ошибка $\\Delta z$, то изменения весов вычисляются так:\n", - "$$\\begin{align}\n", - "\\Delta x &= \\Delta z \\times W \\\\\n", - "\\Delta W &= \\frac{\\partial z}{\\partial W} \\Delta z = \\Delta z \\times x \\\\\n", - "\\Delta b &= \\frac{\\partial z}{\\partial b} \\Delta z = \\Delta z \\\\\n", - "\\end{align}$$\n", - "\n", - "**ВАЖНО:** Вычисления производятся не для одного элемента обучающей выборки, а сразу для целой последовательности, называемой **minibatch**. Необходимые значения градиентов $\\Delta W$ и $\\Delta b$ вычисляются по всей выборке, а вектора имеют соответствующую размерность: $x\\in\\mathbb{R}^{\\mathrm{minibatch}\\, \\times\\, \\mathrm{nclass}}$" + "### Implementation of Back Propagation\r\n", + "\r\n", + "* Let's add `backward` function to each of our nodes that will compute the derivative and propagate the error during the backward pass.\r\n", + "* We also need to implement parameter updates according to the procedure described above\r\n", + "\r\n", + "We need to compute derivatives for each layer manually, for example for linear layer $z = x\\times W+b$:\r\n", + "$$\\begin{align}\r\n", + "\\frac{\\partial z}{\\partial W} &= x \\\\\r\n", + "\\frac{\\partial z}{\\partial b} &= 1 \\\\\r\n", + "\\end{align}$$\r\n", + "\r\n", + "If we need to compensate for the error $\\Delta z$ at the output of the layer, we need to update the weights accordingly:\r\n", + "$$\\begin{align}\r\n", + "\\Delta x &= \\Delta z \\times W \\\\\r\n", + "\\Delta W &= \\frac{\\partial z}{\\partial W} \\Delta z = \\Delta z \\times x \\\\\r\n", + "\\Delta b &= \\frac{\\partial z}{\\partial b} \\Delta z = \\Delta z \\\\\r\n", + "\\end{align}$$\r\n", + "\r\n", + "**IMPORTANT:** Calculations are done not for each training sample independently, but rather for a whole **minibatch**. Required parameter updates $\\Delta W$ and $\\Delta b$ are computed across the whole minibatch, and the respective vectors have dimensions: $x\\in\\mathbb{R}^{\\mathrm{minibatch}\\, \\times\\, \\mathrm{nclass}}$" ], "metadata": {} }, { "cell_type": "code", - "execution_count": 17, + "execution_count": null, "source": [ "class Linear:\r\n", " def __init__(self,nin,nout):\r\n", @@ -789,13 +629,13 @@ { "cell_type": "markdown", "source": [ - "Аналогичный образом функции обратного распространения `backward` добавляются к другим составляющим вычислительного графа:" + "In the same manner we can define `backward` function for the rest of our layers:" ], "metadata": {} }, { "cell_type": "code", - "execution_count": 18, + "execution_count": null, "source": [ "class Softmax:\r\n", " def forward(self,z):\r\n", @@ -827,18 +667,22 @@ { "cell_type": "markdown", "source": [ - "Теперь напишем цикл обучения модели на нашем датасете. Будем рассматривать один проход по модели - т.н. **эпоху**" + "## Training the Model\r\n", + "\r\n", + "Now we are ready to write the **training loop**, which will go through our dataset, and perform the optimization minibatch by minibatch.One complete pass through the dataset is often called **an epoch**:" ], "metadata": {} }, { "cell_type": "code", - "execution_count": 19, + "execution_count": null, "source": [ "lin = Linear(2,2)\r\n", "softmax = Softmax()\r\n", "cross_ent_loss = CrossEntropyLoss()\r\n", "\r\n", + "learning_rate = 0.1\r\n", + "\r\n", "pred = np.argmax(lin.forward(train_x),axis=1)\r\n", "acc = (pred==train_labels).mean()\r\n", "print(\"Initial accuracy: \",acc)\r\n", @@ -857,35 +701,30 @@ " dp = cross_ent_loss.backward(loss)\r\n", " dz = softmax.backward(dp)\r\n", " dx = lin.backward(dz)\r\n", - " lin.update(0.1)\r\n", + " lin.update(learning_rate)\r\n", " \r\n", "pred = np.argmax(lin.forward(train_x),axis=1)\r\n", "acc = (pred==train_labels).mean()\r\n", "print(\"Final accuracy: \",acc)\r\n", " " ], - "outputs": [ - { - "output_type": "stream", - "name": "stdout", - "text": [ - "Initial accuracy: 0.725\n", - "Final accuracy: 0.825\n" - ] - } - ], + "outputs": [], "metadata": {} }, { "cell_type": "markdown", "source": [ - "Для удобства опишем класс, который позволяет объединять узлы вычислительного графа в единую сеть, и применять функции `forward` и `backward` сразу ко всей сети последовательно:" + "Nice to see how we can increase accuracy of the model from about 50% to around 80% in one epoch.\r\n", + "\r\n", + "## Network Class\r\n", + "\r\n", + "Since in many cases neural network is just a composition of layers, we can build a class that will allow us to stack layers together and make forward and backward passes through them without explicitly programming that logic. We will store the list of layers inside the `Net` class, and use `add()` function to add new layers:" ], "metadata": {} }, { "cell_type": "code", - "execution_count": 20, + "execution_count": null, "source": [ "class Net:\r\n", " def __init__(self):\r\n", @@ -920,13 +759,13 @@ { "cell_type": "markdown", "source": [ - "Ещё раз пробуем создать и обучить нашу нейросеть:" + "With this `Net` class our model definition and training becomes more neat:" ], "metadata": {} }, { "cell_type": "code", - "execution_count": 21, + "execution_count": null, "source": [ "net = Net()\r\n", "net.add(Linear(2,2))\r\n", @@ -958,22 +797,23 @@ "print(\"Final loss={}, accuracy={}: \".format(*get_loss_acc(train_x,train_labels)))\r\n", "print(\"Test loss={}, accuracy={}: \".format(*get_loss_acc(test_x,test_labels)))" ], - "outputs": [ - { - "output_type": "stream", - "name": "stdout", - "text": [ - "Initial loss=0.6212072429381601, accuracy=0.6875: \n", - "Final loss=0.44369925927417986, accuracy=0.8: \n", - "Test loss=0.4767711377257787, accuracy=0.85: \n" - ] - } + "outputs": [], + "metadata": {} + }, + { + "cell_type": "markdown", + "source": [ + "## Plotting the Training Process\r\n", + "\r\n", + "It would be nice to see visually how the network is being trained! We will define a `train_and_plot` function for that. To visualize the state of the network we will use level map, i.e. we will represent different values of the network output using different colors.\r\n", + "\r\n", + "> Do not worry if you do not understand some of the plotting code below - it is more important to understand the underlying neural network concepts." ], "metadata": {} }, { "cell_type": "code", - "execution_count": 22, + "execution_count": null, "source": [ "def train_and_plot(n_epoch, net, loss=CrossEntropyLoss(), batch_size=4, lr=0.1):\r\n", " fig, ax = plt.subplots(2, 1)\r\n", @@ -1012,7 +852,7 @@ }, { "cell_type": "code", - "execution_count": 23, + "execution_count": null, "source": [ "import matplotlib.cm as cm\r\n", "\r\n", @@ -1062,7 +902,7 @@ }, { "cell_type": "code", - "execution_count": 24, + "execution_count": null, "source": [ "def plot_training_progress(x, y_data, fig, ax):\r\n", " styles = ['k--', 'g-']\r\n", @@ -1084,40 +924,16 @@ }, { "cell_type": "code", - "execution_count": 27, + "execution_count": null, "source": [ "%matplotlib nbagg \r\n", - "\r\n", "net = Net()\r\n", "net.add(Linear(2,2))\r\n", "net.add(Softmax())\r\n", "\r\n", "res = train_and_plot(30,net,lr=0.005)" ], - "outputs": [ - { - "output_type": "display_data", - "data": { - "application/javascript": "/* Put everything inside the global mpl namespace */\nwindow.mpl = {};\n\n\nmpl.get_websocket_type = function() {\n if (typeof(WebSocket) !== 'undefined') {\n return WebSocket;\n } else if (typeof(MozWebSocket) !== 'undefined') {\n return MozWebSocket;\n } else {\n alert('Your browser does not have WebSocket support. ' +\n 'Please try Chrome, Safari or Firefox ≥ 6. ' +\n 'Firefox 4 and 5 are also supported but you ' +\n 'have to enable WebSockets in about:config.');\n };\n}\n\nmpl.figure = function(figure_id, websocket, ondownload, parent_element) {\n this.id = figure_id;\n\n this.ws = websocket;\n\n this.supports_binary = (this.ws.binaryType != undefined);\n\n if (!this.supports_binary) {\n var warnings = document.getElementById(\"mpl-warnings\");\n if (warnings) {\n warnings.style.display = 'block';\n warnings.textContent = (\n \"This browser does not support binary websocket messages. \" +\n \"Performance may be slow.\");\n }\n }\n\n this.imageObj = new Image();\n\n this.context = undefined;\n this.message = undefined;\n this.canvas = undefined;\n this.rubberband_canvas = undefined;\n this.rubberband_context = undefined;\n this.format_dropdown = undefined;\n\n this.image_mode = 'full';\n\n this.root = $('
');\n this._root_extra_style(this.root)\n this.root.attr('style', 'display: inline-block');\n\n $(parent_element).append(this.root);\n\n this._init_header(this);\n this._init_canvas(this);\n this._init_toolbar(this);\n\n var fig = this;\n\n this.waiting = false;\n\n this.ws.onopen = function () {\n fig.send_message(\"supports_binary\", {value: fig.supports_binary});\n fig.send_message(\"send_image_mode\", {});\n if (mpl.ratio != 1) {\n fig.send_message(\"set_dpi_ratio\", {'dpi_ratio': mpl.ratio});\n }\n fig.send_message(\"refresh\", {});\n }\n\n this.imageObj.onload = function() {\n if (fig.image_mode == 'full') {\n // Full images could contain transparency (where diff images\n // almost always do), so we need to clear the canvas so that\n // there is no ghosting.\n fig.context.clearRect(0, 0, fig.canvas.width, fig.canvas.height);\n }\n fig.context.drawImage(fig.imageObj, 0, 0);\n };\n\n this.imageObj.onunload = function() {\n fig.ws.close();\n }\n\n this.ws.onmessage = this._make_on_message_function(this);\n\n this.ondownload = ondownload;\n}\n\nmpl.figure.prototype._init_header = function() {\n var titlebar = $(\n '
');\n var titletext = $(\n '
');\n titlebar.append(titletext)\n this.root.append(titlebar);\n this.header = titletext[0];\n}\n\n\n\nmpl.figure.prototype._canvas_extra_style = function(canvas_div) {\n\n}\n\n\nmpl.figure.prototype._root_extra_style = function(canvas_div) {\n\n}\n\nmpl.figure.prototype._init_canvas = function() {\n var fig = this;\n\n var canvas_div = $('
');\n\n canvas_div.attr('style', 'position: relative; clear: both; outline: 0');\n\n function canvas_keyboard_event(event) {\n return fig.key_event(event, event['data']);\n }\n\n canvas_div.keydown('key_press', canvas_keyboard_event);\n canvas_div.keyup('key_release', canvas_keyboard_event);\n this.canvas_div = canvas_div\n this._canvas_extra_style(canvas_div)\n this.root.append(canvas_div);\n\n var canvas = $('');\n canvas.addClass('mpl-canvas');\n canvas.attr('style', \"left: 0; top: 0; z-index: 0; outline: 0\")\n\n this.canvas = canvas[0];\n this.context = canvas[0].getContext(\"2d\");\n\n var backingStore = this.context.backingStorePixelRatio ||\n\tthis.context.webkitBackingStorePixelRatio ||\n\tthis.context.mozBackingStorePixelRatio ||\n\tthis.context.msBackingStorePixelRatio ||\n\tthis.context.oBackingStorePixelRatio ||\n\tthis.context.backingStorePixelRatio || 1;\n\n mpl.ratio = (window.devicePixelRatio || 1) / backingStore;\n\n var rubberband = $('');\n rubberband.attr('style', \"position: absolute; left: 0; top: 0; z-index: 1;\")\n\n var pass_mouse_events = true;\n\n canvas_div.resizable({\n start: function(event, ui) {\n pass_mouse_events = false;\n },\n resize: function(event, ui) {\n fig.request_resize(ui.size.width, ui.size.height);\n },\n stop: function(event, ui) {\n pass_mouse_events = true;\n fig.request_resize(ui.size.width, ui.size.height);\n },\n });\n\n function mouse_event_fn(event) {\n if (pass_mouse_events)\n return fig.mouse_event(event, event['data']);\n }\n\n rubberband.mousedown('button_press', mouse_event_fn);\n rubberband.mouseup('button_release', mouse_event_fn);\n // Throttle sequential mouse events to 1 every 20ms.\n rubberband.mousemove('motion_notify', mouse_event_fn);\n\n rubberband.mouseenter('figure_enter', mouse_event_fn);\n rubberband.mouseleave('figure_leave', mouse_event_fn);\n\n canvas_div.on(\"wheel\", function (event) {\n event = event.originalEvent;\n event['data'] = 'scroll'\n if (event.deltaY < 0) {\n event.step = 1;\n } else {\n event.step = -1;\n }\n mouse_event_fn(event);\n });\n\n canvas_div.append(canvas);\n canvas_div.append(rubberband);\n\n this.rubberband = rubberband;\n this.rubberband_canvas = rubberband[0];\n this.rubberband_context = rubberband[0].getContext(\"2d\");\n this.rubberband_context.strokeStyle = \"#000000\";\n\n this._resize_canvas = function(width, height) {\n // Keep the size of the canvas, canvas container, and rubber band\n // canvas in synch.\n canvas_div.css('width', width)\n canvas_div.css('height', height)\n\n canvas.attr('width', width * mpl.ratio);\n canvas.attr('height', height * mpl.ratio);\n canvas.attr('style', 'width: ' + width + 'px; height: ' + height + 'px;');\n\n rubberband.attr('width', width);\n rubberband.attr('height', height);\n }\n\n // Set the figure to an initial 600x600px, this will subsequently be updated\n // upon first draw.\n this._resize_canvas(600, 600);\n\n // Disable right mouse context menu.\n $(this.rubberband_canvas).bind(\"contextmenu\",function(e){\n return false;\n });\n\n function set_focus () {\n canvas.focus();\n canvas_div.focus();\n }\n\n window.setTimeout(set_focus, 100);\n}\n\nmpl.figure.prototype._init_toolbar = function() {\n var fig = this;\n\n var nav_element = $('
');\n nav_element.attr('style', 'width: 100%');\n this.root.append(nav_element);\n\n // Define a callback function for later on.\n function toolbar_event(event) {\n return fig.toolbar_button_onclick(event['data']);\n }\n function toolbar_mouse_event(event) {\n return fig.toolbar_button_onmouseover(event['data']);\n }\n\n for(var toolbar_ind in mpl.toolbar_items) {\n var name = mpl.toolbar_items[toolbar_ind][0];\n var tooltip = mpl.toolbar_items[toolbar_ind][1];\n var image = mpl.toolbar_items[toolbar_ind][2];\n var method_name = mpl.toolbar_items[toolbar_ind][3];\n\n if (!name) {\n // put a spacer in here.\n continue;\n }\n var button = $('');\n button.click(method_name, toolbar_event);\n button.mouseover(tooltip, toolbar_mouse_event);\n nav_element.append(button);\n }\n\n // Add the status bar.\n var status_bar = $('');\n nav_element.append(status_bar);\n this.message = status_bar[0];\n\n // Add the close button to the window.\n var buttongrp = $('
');\n var button = $('');\n button.click(function (evt) { fig.handle_close(fig, {}); } );\n button.mouseover('Stop Interaction', toolbar_mouse_event);\n buttongrp.append(button);\n var titlebar = this.root.find($('.ui-dialog-titlebar'));\n titlebar.prepend(buttongrp);\n}\n\nmpl.figure.prototype._root_extra_style = function(el){\n var fig = this\n el.on(\"remove\", function(){\n\tfig.close_ws(fig, {});\n });\n}\n\nmpl.figure.prototype._canvas_extra_style = function(el){\n // this is important to make the div 'focusable\n el.attr('tabindex', 0)\n // reach out to IPython and tell the keyboard manager to turn it's self\n // off when our div gets focus\n\n // location in version 3\n if (IPython.notebook.keyboard_manager) {\n IPython.notebook.keyboard_manager.register_events(el);\n }\n else {\n // location in version 2\n IPython.keyboard_manager.register_events(el);\n }\n\n}\n\nmpl.figure.prototype._key_event_extra = function(event, name) {\n var manager = IPython.notebook.keyboard_manager;\n if (!manager)\n manager = IPython.keyboard_manager;\n\n // Check for shift+enter\n if (event.shiftKey && event.which == 13) {\n this.canvas_div.blur();\n event.shiftKey = false;\n // Send a \"J\" for go to next cell\n event.which = 74;\n event.keyCode = 74;\n manager.command_mode();\n manager.handle_keydown(event);\n }\n}\n\nmpl.figure.prototype.handle_save = function(fig, msg) {\n fig.ondownload(fig, null);\n}\n\n\nmpl.find_output_cell = function(html_output) {\n // Return the cell and output element which can be found *uniquely* in the notebook.\n // Note - this is a bit hacky, but it is done because the \"notebook_saving.Notebook\"\n // IPython event is triggered only after the cells have been serialised, which for\n // our purposes (turning an active figure into a static one), is too late.\n var cells = IPython.notebook.get_cells();\n var ncells = cells.length;\n for (var i=0; i= 3 moved mimebundle to data attribute of output\n data = data.data;\n }\n if (data['text/html'] == html_output) {\n return [cell, data, j];\n }\n }\n }\n }\n}\n\n// Register the function which deals with the matplotlib target/channel.\n// The kernel may be null if the page has been refreshed.\nif (IPython.notebook.kernel != null) {\n IPython.notebook.kernel.comm_manager.register_target('matplotlib', mpl.mpl_figure_comm);\n}\n", - "text/plain": [ - "" - ] - }, - "metadata": {} - }, - { - "output_type": "display_data", - "data": { - "text/html": [ - "" - ], - "text/plain": [ - "" - ] - }, - "metadata": {} - } - ], + "outputs": [], "metadata": { "scrolled": false, "slideshow": { @@ -1208,15 +1032,23 @@ { "cell_type": "markdown", "source": [ - "## Важное замечание\n", - "\n", - "Простая линейная модель \n", - "* высокий training loss - \"недообучение\", выразительности модели не хватает, чтобы разделить данные\n", - "* valiadation loss и training loss примерно совпадают - модель хорошо обобщается\n", - "\n", - "Сложная многослойная модель\n", - "* низкий training loss - почти идеально приближает обучающую выборку (но может переобучиться)\n", - "* validation loss >> training loss и может возрастать - плохо обобщает данные" + "## Why Not Always Use Multi-Layered Model?\r\n", + "\r\n", + "We have seen that multi-layered model is more *powerful* and *expressive*, than one-layered one. You may be wondering why don't we always use many-layered model. The answer to this question is **overfitting**.\r\n", + "\r\n", + "We will deal with this term more in a later sections, but the idea is the following: **the more powerful the model is, the better it can approximate training data, and the more data it needs to properly generalize** for the new data it has not seen before.\r\n", + "\r\n", + "**A linear model:**\r\n", + "* We are likely to get high training loss - so-called **underfitting**, when the model does not have enough power to correctly separate all data. \r\n", + "* Valiadation loss and training loss are more or less the same. The model is likely to generalize well to test data.\r\n", + "\r\n", + "**Complex multi-layered model**\r\n", + "* Low training loss - the model can approximate training data well, because it has enough expressive power.\r\n", + "* Validation loss can be much higher than training loss and can start to increase during training - this is because the model \"memorizes\" training points, and loses the \"overall picture\"\r\n", + "\r\n", + "![Overfitting](images/overfit.png)\r\n", + "\r\n", + "> On this picture, `x` stands for training data, `o` - validation data. Left - linear model (one-layer), it approximates the nature of the data pretty well. Right - overfitted model, the model perfectly well approximates training data, but stops making sense with any other data (validation error is very high)" ], "metadata": { "slideshow": { @@ -1227,12 +1059,18 @@ { "cell_type": "markdown", "source": [ - "## Выводы\n", - "\n", - "* Простые модели с небольшим числом параметров (\"low capacity\") менее склонные к переобучению\n", - "* Более сложные модели (high capacity) могут переобучиться (надо следить за validation error)\n", - "* Для более сложных моделей необходимо иметь больше данных\n", - "* \"bias-variance trade-off\" - необходимо достичь компромисса между недообучением и переобучением (обучением на распознавание нерелевантного шума во входных данных)" + "## Takeaways\r\n", + "\r\n", + "* Simple models (fewer layers, fewer neurons) with low number of parameters (\"low capacity\") are less likely to overfit\r\n", + "* More complex models (more layers, more neurons on each layer, high capacity) are likely to overfit. We need to monitor validation error to make sure it does not start to rise with further training\r\n", + "* More complex models need more data to train on.\r\n", + "* You can solve overfitting problem by either:\r\n", + " - simplifying your model\r\n", + " - increasing the amount of training data\r\n", + "* **Bias-variance trade-off** is a term that shows that you need to get the compromise\r\n", + " - between power of the model and amount of data,\r\n", + " - between overfittig and underfitting\r\n", + "* There is not single recipe on how many layers of parameters you need - the best way is to experiment" ], "metadata": { "slideshow": { @@ -1248,19 +1086,13 @@ "This notebook is a part of [AI for Beginners Curricula](http://github.com/microsoft/ai-for-beginners), and has been prepared by [Dmitry Soshnikov](http://soshnikov.com). It is inspired by Neural Network Workshop at Microsoft Research Cambridge. Some code and illustrative materials are taken from presentations by [Katja Hoffmann](https://www.microsoft.com/en-us/research/people/kahofman/), [Matthew Johnson](https://www.microsoft.com/en-us/research/people/matjoh/) and [Ryoto Tomioka](https://www.microsoft.com/en-us/research/people/ryoto/), and from [NeuroWorkshop](http://github.com/shwars/NeuroWorkshop) repository." ], "metadata": {} - }, - { - "cell_type": "markdown", - "source": [], - "metadata": {} } ], "metadata": { "celltoolbar": "Slideshow", "kernelspec": { - "display_name": "Python 3", - "language": "python", - "name": "python3" + "name": "python3", + "display_name": "Python 3.8.8 64-bit ('base': conda)" }, "language_info": { "codemirror_mode": { @@ -1272,10 +1104,13 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.7.4" + "version": "3.8.8" }, "livereveal": { "start_slideshow_at": "selected" + }, + "interpreter": { + "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" } }, "nbformat": 4, diff --git a/3-NeuralNetworks/04-OwnFramework/README.md b/3-NeuralNetworks/04-OwnFramework/README.md index ecd5f425..de092520 100644 --- a/3-NeuralNetworks/04-OwnFramework/README.md +++ b/3-NeuralNetworks/04-OwnFramework/README.md @@ -14,14 +14,14 @@ We will also develop our own modular framework in Python that will allows us to Let's start with formalizing the Machine Learning problem. Suppose we have a training dataset **X** with labels **Y**, and we need to build a model *f* that will make most accurate predictions. The quality of predictions is measured by **Loss function** ℒ. The following loss functions are often used: -* For regression problem, when we need to predict a number, we can use **absolute error** ∑i|f(x(i))-y(i)|, or **squared error** ∑i(f(x(i))-y(i))2 +* For regression problem, when we need to predict a number, we can use **absolute error** ∑i|f(x(i))-y(i)|, or **squared error** ∑i(f(x(i))-y(i))2 * For classification, we use **0-1 loss** (which is essentially the same as **accuracy** of the model), or **logistic loss**. For one-level perceptron, function *f* was defined as a linear function *f(x)=wx+b* (here *w* is the weight matrix, *x* is the vector if input features, and *b* is bias vector). For different neural network architectures, this function can take more complex form. -> In the case of classification, it is often desirable to get probabilities of corresponding classes as network output. To convert arbitrary numbers to probabilities (eg. to normalize the output), we often use **softmax** function σ, for the function *f* becomes *f=σ(wx+b)* +> In the case of classification, it is often desirable to get probabilities of corresponding classes as network output. To convert arbitrary numbers to probabilities (eg. to normalize the output), we often use **softmax** function σ, for the function *f* becomes *f(x)=σ(wx+b)* -In the definition of *f* above, *w* and *b* are called **parameters** θ=*w,b*. Given the dataset <**X**,**Y**>, we can compute an overall error on the whole dataset as a function of parameters θ. +In the definition of *f* above, *w* and *b* are called **parameters** θ=⟨*w,b*⟩. Given the dataset ⟨**X**,**Y**⟩, we can compute an overall error on the whole dataset as a function of parameters θ. **The goal of neural network training is to minimize the error by varying parameters θ** @@ -38,15 +38,22 @@ During training, the optimization steps are supposed to be calculated considerin ## Multi-Layered Perceptrons and Back Propagation -One-layer network, as we have seen above, is capable of classifying linearly separable classes. To build reacher model, we can combine several layers of the network. Mathematically it would just mean that the function *f* would have more complex form, such as *f(x) = σ(w1α(w2x+b2)+b1)*, where α is a **non-linear activation function**, and θ=<*w1,b1,w2,b2*> are parameters. +One-layer network, as we have seen above, is capable of classifying linearly separable classes. To build reacher model, we can combine several layers of the network. Mathematically it would just mean that the function *f* would have more complex form, and will be computed in several steps: +* z1=w1x+b1 +* z2=w2α(z1)+b2 +* f = σ(z2) -The gradient descent algorithm would remain the same, but it would be more difficult to calculate gradients. Given the chain differentiation rule, we can calculate derivatives as +Here, α is a **non-linear activation function**, σ is a softmax function, and θ=<*w1,b1,w2,b2*> are parameters. -* ∂ℒ/∂w1 = (∂ℒ/∂σ)(∂σ/∂w1) -* ∂ℒ/∂w2 = (∂ℒ/∂σ)(∂σ/∂α)(∂α/∂z) +The gradient descent algorithm would remain the same, but it would be more difficult to calculate gradients. Given the + chain differentiation rule, we can calculate derivatives as: +* ∂ℒ/∂w2 = (∂ℒ/∂σ)(∂σ/∂z2)(∂z2/∂w2) +* ∂ℒ/∂w1 = (∂ℒ/∂σ)(∂σ/∂z2)(∂z2/∂α)(∂α/∂z1)(∂z1/∂w1) +Note that the beginning of all those expressions are the same, and thus we can effectively calculate derivatives starting from the loss function and going "backwards" through the computational graph. Thus the method of training multi-layered perceptron is called **back propagation**. +> We will cover back prop in much more detail in our notebook example. ## [Proceed to Notebook](OwnFramework.ipynb) -To see how we can use perceptron to solve some toy as well as real-life problems, and to continue learning - go to [OwnFramework](OwnFramework.ipynb) notebook. +In the accompanying notebook, we will implement our own framework for building and training multi-layered perceptrons. You will be able to see in detail how modern neural networks operate. Proceed to [OwnFramework](OwnFramework.ipynb) notebook. diff --git a/3-NeuralNetworks/04-OwnFramework/images/ComputeGraph.png b/3-NeuralNetworks/04-OwnFramework/images/ComputeGraph.png new file mode 100644 index 0000000000000000000000000000000000000000..4c3b5fdf13a223839edbaa16b0c723e99b8696b0 GIT binary patch literal 9655 zcmbVSWl&sAuw5jw+8@l=RUp=IFm!CU}v$sw6e(i`}^b5)8_u#+UdP5 zQ>JaH*!lSdcoSgT(E$Judk+M)nD{quvV@KHtX<}f9HOG3DW=c%VPhjYb&u4p=8c4O ztyG?rmwS16k&%%RZRT&u@e*1+1Z(7-n{POvOaml^8LH3 zKtXfb)U2}d@V9R=II5^0dtNtbZh z^~CdpnJFHldFd{wae6XKC;*9&+u>m1|7!#>jRbA0nQ_Xlvue0z^ep*os&S<8+bpWj)!vJ`T($}v zJvRo=*q=&+C#^Uhv%N7&-0%O3|B&E(y=!S7UCnuZ zW@xHUd;A0icY9Zy;Z%7wznB|L%(!g$5rL`4P?zDOGXM4C%9riOTTZuc z)$1MvOT+7TpQh#zd;t8l`|~>9E03)H!mr0Y_8ExFUeyZ?B&)8<&+EvG<48dg+k zjlPUreTx%x5J#PIZ;9mHWA46+4^U~#cKr%edDgi|+HG71WJ^OzQp#Pugt81TBkl07 z#&)U`6F_}`RuD-VWrmoz`fI?nsEy{dXb zxbib-1c_c5eRZdRs&`ZkYIW}>v$>aln#dKY7jFQWyC#A@P$_C%1%EHw)1*DGC0usB)jmlRq&XIuGrZ@s?6@+1; zeMIfcGSbn}vy0ov;KkuVkE}AeSt?f5n4fX`7is<-KhmVM`6qFH8s6q)*W$8JddJ$&%9O=WN;;GCdsnsJ0834Pn*fP$f zOGtuIlvVq2qR6U}2_31@R}RN4G6F|v9jPVB#XCljCY9)B$e`gSn3bgWESYhFKmS1c z+$YM+dI(Rh>x%*p$l)ay6u%-dEW?~GG0?<;5K2T@BAXK=A^Y1SUM!B1gh+lsyBBB9 z8{3Q01$*_0uQpzHAP|M(pe%@_D1tPaGc}yXsmqKNISEG)S%JeK6>}E^Ew~k0<4)H= zAUGkqFx`r<31ar?5os5c^FGL|?)a|n+ESeSPrxBkoX%wx@(U`V1NNUjKfo8<_=myN zJtp7yY*Y+AapqrRw!LzDC2{w=Up8TbnUu#9stu(B&=}@B)^Ud412(Gl8S0hI!sEoT zC#SO#ha2I^VTa$tHmcW-gXzR%1-+-G(;#JWR44zY{ed^}`;0D&({?faWIq4(8#Fgu z(!z|CRRhCV^o37tazO&m#vRO7A*_V;(9AF{ge*2%I$mF7(Jcsy3pO|!|C|@Svf60k z7g?ol5F7h;F03c3;-F%OaVo|pGZ4#0m*wj!@E7i;PryHP9mBV%F7ejL~ZJ0s5|C z;7GeRf_fuEn5Ol__b72_&rvg)a-++%mC zXak+wt1o=jQ8O0wdB41=v<_X?<8~6~otreRTjlVxH1)pgWHGj9F8SJHgkP;Ra8hkj z{-*r{2SO-MwXCpz@kd;L+ zf_bD`2_)*jagF~(ZP!z0?JFtCAR_jZ961vk?6DepFBr+2nZoHOY_iTA=w1n)u*^k7qyXj{H;Yn(v85z;b_xAkwvpysF0`FqH^cAY_CEY}A zcLEGrUz1WlE!bdu=PFb^@VVsd+G#?>d@SksL!bwuqa-mjd1>g((9&kjFJoiNi!c{& z@t3v7KCu=Rcys20oKQdvns;IzeX^?}jqN@bv;I_%CChHHB4J8+&pj3vl*MKmP}EAM zZyKL=9*dO4bNWa)QT7>qQ2*dFw<>yYmXv`aY}t`0CUa~Y@QRc*6H^H5U=E?GcLG)( zUPQkjs&mIjk?v=Ye7J^0z(gVlZypc_ObhQw%6c?Y<3%q!gJ?-%=op#UabVDfRJ{kJ zU{`z>>bp?Zkn|~b<<>>%3d~EyW%1@|36gTjIypQGjZOimGgZW%Q*X@5^*V_Fw3s%4 z&v2p-39Tc;)?HXAK%%sa379h{dRdpS0i>?$Y)I@DF#j|zdDaCZ&3UFn!e z8y=84l~>QMEFoKlLNHl}Bi&XKts-zxe|fCKHdg;-gs&Y^sp68(FtS(cYnM_ZRjFR7 zzPTH|jR+ZsSU#jgKwZma>>}kqU~3sut|eQE->WRN3;k*SOyJ6{QRi>1U?b|3mV+BY zV7w+^Dp?0Xy(#m{oo@AW-qS&7nFybO3KN1RE>B5uBXt*vF#E>P`JTIhPMABrK;S4U z|3JAMNom+69T(wUs6RpB=c7`ODcxlr*tO{8UMD`uU*=pwXNbPWo{s~DXjSxf1b^$r z%;>pPtfJTH16$SW1Wx_N^7ya))KhrP1x=AMTSF-n^2-e~WKwjYQJ@>kq?meq0|9Mn zuIVkLHpY4E&Tv_Ml!+;qq#%8-mj6`NS7Dp~l8)xMaqNQ2 z>Sha~bcGT_H?)gR+Hhmt_n&}wqC)hf>2UW*szEf@ZT(w$OvMVtEMO{dfDh3%GG+yA z8335OMBLw({g6C0|1M-2ZM|i6v$*_1R*AI1?5vumV90Fy?2dfbi$FE;)b@DA@$W_J zH>{C@P~D9R0dTxHJz5pUcwZ9_HYzrHkFX`fFjcBZ^R>+!FajWDH%n6F>X{tUxmhvy z-y;aqK;gD^K>eH&Fw_=}2ua%WC5z%blCTT62jhlUImuXys@FLg8$|Y+hX_`DgzH6T%!Qg@qERSn-fW0 UCP{#*=-o6r$;0 zRtX)E{shY>c!LWJ0ZA_A`U7tnA{p=fbN@#6LXcL`6?e~WU`qs`Bw&%@7qH}vn;O%E z{i%{i0y3oFu*7W)wrC1O8P8tR>QOzThZKxAA5a;g}d~1lA0mrI1X}eB#u`n%qShgz5+l5i8H6-n@6>3<=z7QOS z-&DKGM1EN{&=xUFFuq}!_nvqd61=Gsr zp1$$}%Ues01Xgs!c_lM@MXBL=C;>QV^*XS-7On#!AyQLXjXarNNPOMVy@_)3ht3~rz`b{R8YOpJdd8z8lREUBH^=) zose5>l@nzDin$iv{C*|~vN`R`b@8d97IXL^%)7rZZ06tbtcoRuQL;cp;PV!Q3zeyt z(*?6=Z;vM4v`rB@NUYy;HmreP#fp3>Of9ijLIcmFk_Vm)v9UXEWyyavI^}wrleZ=B zJ}N~^E=JP0*SKJ}Cv?0{w*W#2DXp9*hql-4fSiNgEVVZTt>H*U^SfB#^FXDBVFPRU z4$)0Sqq1hBb$)6bAH~wtXsm$g?C|S*@LT@!%;*dqL3&$(7ryG-pF0{y4kZbx+kgF?bm>|kbJt-e|$>kt=n`A}|5whf>2rShHceIC_ zoTdyRM7lVkDJM;%)kO#%{==>F8G1(p?eRpq5Gi1rD2Ui`EdkGhG%E+9;8p39 zGxPT9Ja*Y0W8dTXgSk%2ENy_S-%Mzg>hAV|KZNuPQtSl7@6+kb2-IKV5}Q5eV;j?` zYVX(MZ&1Jcovw#UwG6apN9j~ZRWuhVdC~K#LwyV&d(ZeO6?Yk)zGS}W938}FUp)x;+TeU7&;Z0SF^GxR$gh6&nl#C z*XoaVMByn|=ktm7sU9N3kH?BhgZeE(LRszoeGb0+yQNv1Ie=*MSGz+J^*(qu5EwX zR?DI(@(K#B>gA(yDTsU%nZ6C8O?`u{WSWV3AxmSCV-{4$0r^A*XIPo!w45;2AOmN8 zIa+;@gqSJ7HnA`{TvWMjfHG3n~L}BreTGKCWg8jD`ekrbiMGfG1wjbCu)KDT1a`7DU+*gR^30n!>JB^90Ri~=xQ^yF@##e{ z&`5~IpeLk>q^F8BiMdPm2l|!1?}%_-nI_eakr31(iH&aL10VB_`e5%Xk(#tgTt8~>^4F%JJg2zX+@}imR;$aGoL?U+! z(ENkezc4r$ZlglK)AIs0{z6%)!m#D(XtP~?hf%aqW#gt_^X2gy#WY_G-!AXP8+dAeLhA2n7=9kz%PIZ2v*KT^xVd!IHnzo-xPR-J3B!sm78q5EF z+qA?$4PIy~>)I=SBPXobYqz@}MjQgt!g;dJ1RCrm1{G7hYbQE5+^~;DsX1tE_sslZ zV@B*-!8b}*hQFNJGA)>#%B2~Ud2WmmQ;?kEDmnN+Vqk%V4 zzA{h2*oi^g1rK0Gc3fAR^i06EpGe_?tqsehYq!a>PVR2PtCy!a%1ciG-?}>8Z{b-N`Z`t!P)-#QH3s z1M#+0##Q07lUV|4qkY}N%8bSOSx1_ms)Zz9*J+0gqxHNs*u|-j6yj$hHK4ou5 zj(EZ=aGuv@Y>_8yb}BF->MGQRKqm;SoiWP%t{>REMUxTx{P={x|(@LtD1WYJvv zcD21ysks7a(}-Bl<=V`h*LZ#Dk5SQM;Pq+g^rCxEn(Tfl z*TdQTOSO8+QJZ!~MzeTB?AJ@*WhoE?I)jm<-~T$;c4GzG3H)up#!QgAnrY3-xu~Qr zE-5n}o2(ug>+#7%_{acq4i^US(u!%1+S@5FQ|hX@P5w^G{wN%73@CN12rW9Ei!?Om zi@M~(bwAk=3bh2Ttwr#rtgp7VQ$CbAF^7% zEHOs-{+DhD?>ss~Cjzf|)l=;HaDk;tcHb)D<;Fr;vR1=01JCgbN$Mu3JTs|vT7I~z z9|B;a*mE>5wC_U_2n!%`-KHB=7m9}!6bASXA6ve|W!Ev^6v3gO0!Yy_p)|S6A;=&} zpjTC%Ya&}&Msk|ig`qptPrU2}?<##s$N*#gnVHjt+T9QK{+M7O6C#MRN{iry;E|5h zJ*5vKk|usy#;}>haGP!b*I)Hl;x|4-QX~kJ%&gg|Mn%&@#sSx(t)tvz~o3<}R>Q;!LB$0yTXD`Jb( z?5^>*I?Q4^ON9`q4fq-!+gXR+nJa6{Mxk)MT>$5Bm(!HV)=y*&RxVKSDxI4w0J=Na zu4cGgNyRpJzAgc(^aB;Kcqf(=mjXYzu(Kjr>|o_H4)4{D!p;#5*P~I#k?$lH_lB;5 zOC-NO z);2Gb1MCjEO6YP=n>jf7PRvh>g@}AQ6GC7{>FWD&8NmJD>JMBhA$cvJHXI@VI`BnJdmHj-Q)f2 zZHU1Z2BT1CFt=qx$-#qzh7Yl$E)SBuB|)H}2N!C@{NWMlvNc8?4i63ut)QheCA;s&hI0ZY1_sL2Gx+eK*NJmVR!AJin+Uk5A2Yt@W07`@ruTI zJ&c{AU`s#xTVLmp_YHaLF&P-m3WvQdd&LY!oBbUnU!!`vjsTNL`}Wg8dq;c_Jo8)x z#Q%DbGSrY+5!vz51%MBOvZ0sLJsegADOZRaIl@VV0Z|LdwBS^(fD*6v0? zmG^7?osWMNGx|sV(fK7Rm8~=Ynmez5-V@>_eeL2&9PIa$AVLRp>Kq;}-gloF@+12} zGK*(jREz`9UY+>)_U^!ib}rZ2u@L47XI~8%BH+B}LT)`W(tceC`|*sct^U|M14HVh z&dwjK3Tj$x_xavPRly8M=Fm0W{GB zR*-?E!$#p!{;g9v;0!+u!Vd&PXz^Y#*&&1#sWeZv`}xe2!q_38ZH-lV4=OP<`@Rz?o{{X7okd#?6n((07}ct$@}$Loof+Ql05BMlcwmI5FH{!4gR zsEG$oKXHg)*SZQOA_kF0;RE?@p$E_*Q-<96_~xA|9%W3l9T!fjqfDJ*7WiJjkXn`7 zm0FrxdAUFuW>TKzs7`L$kd@>`U}?8b`fyGMG4KUMD%iU$evNr~_d>m`Kv%f5;N!jD zD{j*MNP!mSO&IqZl3 zjP7;3JaM1DFrz)z0V8$_=-%+;jpY+9{@lItT>;;N;Ot+?s-)6CMa;%GE6u|gU|=x8 ziLlG(Yo$&d>98j*gsCaNmYGHrQs}sV3Arg_Z0>FKIiftH9sN}}TvA~tNJEk{F??dj zDzi)c@v6G$_>Q65fm38smmLx)JJp+<607(C!HBB57earZw54XDKv}B+0S^w7GEn1g z^lJk-g=hZ11NW>n(jIrx0bOxC`%;~}2_@3O_30{;6%97CePFz1aRu>Xs!yX2Un^{! zI;o8QEokTjI$yKv3tc ziK|Y#9e$s2X)q(0j$O>U5fRZ0+n;7Y5bJ6%tIel1o%b9w`X#AYe_A@Bohe^}vS1eYU(<2E zU-;X@R_?f|Mh5)Y2~hCI>@wd?Gr7!CHdr>^`_&}(N4O>5g<<$i=??^ZUX9lGBTv^> z3kgv--qKr27h`>|P}bCkgMg)CsVXVFU5<2rRexVwr9gjAV6iFH`74054hn^)L<`!84T|TspS&%RNf{!6fTpqlL zDmnC}Y?nFIDh6v%A`y+}{s`q^%7=i`KjwWBQ0CPx-fw=S&1O`Ez_(tOSa%DM0aVsV#6=S6I7pu+fYp4|($d?}dF9t*5ttZC z86DZWcEY5|2W5pryBdSX^WC>xe5G)~Khwi&y+152b5^|BKDlRa{%HTvzv>wx1Hcbb z+MlXUe~c>GM$i99Kqc$4v|CY{;wi`B*+u15vCT(xVkhKNJ=Sip%x{3+5f5nJvN&4cjM+UncUD{xA!=dN}J3$Yl`lrJKNrr^*zJY5^o!e-zd2qJnIF$ z{$8ODrSlrxrMtS9U+)B=zkh4IxLUlG$N(S(QN^g_+UCK3fRYJDglC=@zE#zafpnpO z7K@9KR4XTS(L45r__h4}e;!fImV0U(-%RMEe_V(9zdl~R?|@&+29DCndoLe22@zT0 JGN7){{{Y#tWLf|K literal 0 HcmV?d00001 diff --git a/3-NeuralNetworks/04-OwnFramework/images/ComputeGraphGrad.png b/3-NeuralNetworks/04-OwnFramework/images/ComputeGraphGrad.png new file mode 100644 index 0000000000000000000000000000000000000000..58a55be65cabc4604ea4048a5bfa4984139a09e6 GIT binary patch literal 16389 zcmb_@1ydzGu=c@afyLe3WpQ_RcUb)3u-M}6a?r)yeQ|epcXxM(1upNszu>F-Qk9xY zr@Pagu6cTrNy3#BBoX0p;Q;^uqO_El3IG5B1OUL+VIjUKi3slEUk^wNVR>Nypa%5q zpApno8_Y>XQUp*niGT9dfOC-2b^-u!7XPbY*fXOSU&I~T^thl(ifPjGgxVXvY%`H#Q1=at`>D1bF zZ?B}J6bloJcY{{_SI;x8?elq%L;A$!)JSBZg4Ads3lM2CHZs-|K0n7|5F7oA zc-o|YrvJUVgTgOV&K&>ENccSg?#sFVzeN-oD=L1Md3L46T`7dn$Lx9{1qy12o){D4 z@GAxA5rFLzJtl~KwAJ=PJa`YH31d3mH;(>dxouK%L8-+`u0uiN4?%g5iElHt+ zFfwI=f)2T2*?-I1ybg2pE-iU4#@DW^oi~v+TQ*N7DH&)3j{6ScW2ZZ3?qQEVJPjuW z(z>2g7?Q8*1~kb4RT_2iTCqwys@)IP)=p6)#tc=(%HuKEFQp0(^u=>kv+d)>icW;h z3}%g{H+rDryE_!X65}(=%xD=Sp9CvM+d#T)7sDcz>~Y$|@tkb=N(+>w_@2yM>gU*; zvzH-Srpx2duy59=Da#MyRW=Q1e!KeC3L-E}H1o zYd`fj$IVo34;Hxaz#R7C1yh5>9yS@5s`U#!BSSUo^(HKPPvt<&%5zJLya)tdj^b~z zOFC$oU#@7PBB!3lsQ2yzK{q=LxNg`G-c)XTM;jNX5?7mM@j-9#>eEI1V~enU<>D-H zvb?GyEpzGLlXTt?LWLn2ZZG6q;_6fUTi!L-{U#F{`PTs zS@NRm!)=4isceUCJ$r0dd6LXEtMwV~6q^gKn5R$u}8!lVd) zs-J zW?t}Z%yqW$-J;*mrq!8E@0On&5hJNXe6Xqq;JjD11#uqw0)kT(RzxN&x|7-2$#uz0 z5`g<5h4Z|3`3H%T%2&Ga@Fhb~)ACs^YrZ2^=(OBYu#F!f8M#D7^=_mTmiv0Iw@ubT zD>#}T;dRLgjuRc-sVI+PXIbGXhIYi^~uN25%z*iFJZ zV|@YNaz_NQM2?7}O%iu=rc?rel|2)d`;eG68Y{4Q-f?$O8J}isL9+5+jz_cTHVo)7 z7CVq7&=a)k78N7@fR4m8rE25hZ{6O}JZjpeb2I$Cj(+d5V9Rw2=@RDG&3}-PG8KF z*c>cr3t~jzJLEifM>jggy!q>p?!|J$^A}QBWMxd&X5=EDHN6skr8>06#!EE@_&kf2tGP_jmYN}*v+y_L{ zM-rO-2wT@O;#q4L`KDn*0XVix%~%+83R~!M77t# zeQIF107xWaOm(?t)Qv42Dy}hsq9(@TPl4Ne2jr+DQo~hBlgD`uB&y!&1vOpFxuW_Bcjqrw75T5*O14nlo$*i~d;p>yb6523yqtbc#f; z8rApmdmj`0bb;vHT9X@-q{xbTr{fYbG4nC5w@gm5GgzNO3rAUVQ>aRl7ndo{G4|H0 zhsom-lf|&)QjTw4h3!xj-{=U7`5R3iGG3$mnsOU1-Uz-Q(2%R&!2k+{-RF zv<}`(!#mPSFTAw$nc%QhCt<6!1@PG73e2a0U6NslWK|W*&}}K(=ZIrv4MfmbTGjQo zPLuyBm9ryj8Q$IR70Q3US2oM!K>ou( zOY_W6_A_PByFyTZoi<&yngD^yqPsBTfOB2c+96fQyI~70Hv4bfw{Mz4ytf6~6X{mg~#hX^sYm^&l(+{uzINn1#LCb5XCB!)FEA%gvy#mvL-yYvI zW=2Ufb6WaZ{*pWnwkNSsqyf_!G6@Y`_K{N z4tCB*yI-pp6Lb1}@-R`z>TPtBnpI`i@5|2){ho4Vg3jKX|1QW0FKbq=nwO3%S_Z+U zeJgg$^Z%3)D4dVZ02Nu=$_BE9t%yvmuNKvv8*6M~p|JZM7^ zVg;Qo`GAW<1fm?1#^;=Oarg%`S+s0Ra>fFydhileTf__kVOeewHUgQe zm;QGdnS$|#H>FAmycEjBYH->H>-M^($JhFUINW;#ghI=rWcN7Nl#ipA4zgNH%M^>6 ztU9B$w*kwx4JKlwpIS3-wMVjfsa2MBp0vu)5v1Yk!6+uC0< z0GOHc)Q6#0zPbngat((7m>r@*s{i{`#ZJewl@R=Rf4Msn1owG=8Un!xFqnyLvV0Fv zVMBi$BOV5A_Fx}WKf12kaeO+uHSmv~3aS0fZebb_vr^|)oPX!WvJc+_h5t$M9%^H;mixAKb(l4s4v zfWFK3`n(xub?hf;4SV)?*|)B9*`T*i%q8yDfBL~w*`AeCwF|e04U7fOm;O;;Yj5a! zYjX>lcQdCIMZP;E&WtlRdow&A1DH>Jp2Q~I92EzKF|IpZv~=qo5Ujcz@^z!9Zd+?@ zwV1^q+Ozs+S%U(0SHW_6Hv``C3kVoW{G=6*$=_T}zHUun(eF38mj;@i(?(&|o{^)g z7eqh0cZxSU#cgyChb9)@Vo-0%Hs#wHy#zb0JAEUUx!iud(%C{R68@*Aa!It^?sNA` zNeyR}IyKs%2uo0H5KP|Z*h|t*$>2$>v>?QS%{kbkP>E zq`IH~M(1|Flh*$*lz&$^8pHp&6-#zl*^bb>s zC#_xbZjwZ1P-*4ipXNnYmQfMQ+QPS@4orO$y6W!Vo|*$ST7fkrP6Wt^gx1OaW+=&r zSWyrgtlm-%CWZ9;Ap-XsQmDKsd&mkpVgUuOVvAdKk zz8dt@Y$02_Nh_*EKka|zEc3mu?SvPd=;Kzxp#+*C9vHFCc_1Xb`_RhY@TT0!|d zss$YI#!$+aEl6h=rSc^a%LzD2k+_0tA}8~LD!TZF;SF99vwLmgRwgnQl?{3}N4Qb* z)0g58$OAq{xkxdvuii6nvi~NpJlfQcU-O1c9a?%TvS1}EsMvg^va-hi0~ZtoSx8|#u|-uhDxe5{xG7-h(;s2XhEQ#Pm19kk|h zbYv50SMtpyNEj!E7_5x*FBQL*5O8!bvltBPJrp)1l?q5`1F6HR&%60%W98$G3@?(W z<|!RFm*%R2#Smmgxm1x2U;4fI1iGw45Dpv{JxF-=NE21vqzvwgGiXKdd{6JV2JK z=*ugL&-5zhM-Memz{zP;fxATQ4w(9|K?|5*i*+0^+~}!Jj4VfqSo1BAc2wg1^t0Rn z@L`OsAeFTq(!w*?LN9=1LM5gS+fqW!$GY675nEWN8q(6TW}~<*q*0Z7(zqbW3IvUK zQ0c0^hk$H}>#zt!zG7jEMI_%Hkh#$ge`=o4HMy9x#o9J=3Jb%s@?6Y^z9CeE(v`y< z{S1b*$l9$#J=41gxwseF&4K8g!YYMXDQSeRj9^P0pwIUFecA~DM-egHpiCwjxxYu# zw5qC@Fu*>$zds{_cHBAyH}k-*B^nlA>8Yn_t&(esad00j(orLosAqsQJAIqv;bJ1f zk6S@Wc{d(3}GuIL&f{2DC(Fvd_ zF65V+8|LNsog(30TzRlAXp_QW4lIqoi|AQ_4rUxjhRhZ{8$ zw)cD1oC-xOV|4%+r1uaPGbA3!6N{|n;fPSohr~sG3^&JNX1tE75f$kS1(Zv{WViZB z-SnsvH~m)!7ZqlN6L|J`;Tk(?qFcNdrpZ~%_v3_OIFqbXVF*PKGtvtZI6Q5P&c5{9 zzeV66j{4eT0p2SZAn{$e`M#|9K}XB_q=w05E0`%osF{4fVqsEs2c8n)ppvpRN26Td zRuiy|ho|uU(lD(U<*?>A3#F^N)qU9UcAux#mMp^=aU1-am zXpkiC&pj*Gs_B>Hmz|VskR<8vJ`2fNFT|O3XnoGEyb5c)lma1cSwJb~#1&@a6=!z| zTpUD;ny%k!P+1IOPdrlf(%wLu3`xtYM;1fvPs9XYj}QRl4kT(g^8h#rvjMOSMfD-d zqCId7MQjiD7AdyVNKx0z(8TpYvk{eqrQk?2GJT=M^`42)mOvJ-S%F%#liYd;rI>Gr zJP^nt`AaK__)UK<@pt1Qg4U$X1L2tPa!kC*b!Si029#@y@T8Og9N z?T+=zL9j%p!h&2VTnV3+J1iPPqP)?3Dwhr=MK<2soJ*KFmm$)g1 zVM{pF3D-6+6^A8wK*$AnRyf{ig4EXcc^ak0v?~7^;tTM^%A!QqO6k}^cc)eVMorHT z2p}gY?vU6PDsAl)7wwsBTC$5#7#A<^1ZGvM@)p%E#L)<#ck)=tGBy#D^WGSe}d*g!n(`qJo_=P9uO<|%q5 z?liC-Ww;|Uq@7@hwzKuI_%=-8OZ?=~6@kIS7_+~JbTTKULB0>0_f5;lY?sNcWFcd$ zc0cw302EmBi@zf2|DaAE(e+47f4ZKvLhr?cQOr9oFA^tF++Y_-}Yo8`ko=c8{&5HM!%J&yMxbJ7>blRue}< z$0>SZ=sTz^i9p&IS_-(ObZoF#~%7iiiEu!x3 zL@tgp2#?5M%l;|NJvkVZdI`=js6FPg!8dyh@L%+4x!1ud7zNF4tPlDv!Y*4{6lmv9 zSJN?w;S}1xO-P{v6XaVPcWpH|HZJMwqs%eq#9az>>|M3F$*4z?0b@{>5B*<)nP9SQ+1s-2?>oBa_LtZeheS>dw>sYH z@=={zZDctnpW877L^~i=S^j)z4G@aNx#%tbBl6-80@==8<8meWX<(CcO~rCD2G2j+ z4bdeDB`n{`;SY9Oo)?&_T0`RM#~b!HsS!D{PU(G$2F&DTG?VOqTLb_nCLgL8?sYX+0w3bcP?7yVI222);+ zJ)MCIur<+CCeRID$1)Z;R&5YmdT#hvV!WVSHOyJS^$i~nJ<~`p$=#My zbRhDrK>ASkgD)!FnWn$F&}Nr+7hWS}F8s7Wq_p{Cp!AHC^~upgeD4eoV4_F}tr|%p z!tI;f^DCHkjxB}cjfh%EES4~((Ho$zeuG_B$VNd)RT9ZV0~38+Itv{I@~L=&eNW2*;``L+dZco653 zV~<$OZ~l-^`569~;b4fJyi??gyU|u}GnZftRKCB><~D96lP)C_!c&1oufzv@A*1F6 zl_|R!_w3ZF8#75I9TA#_GUe!Q)|UK^jqI{Eq3o?>SDK$>3ag=J$=o;->zlVO!32v> z`|s_)1?~6+Ps=jmLi#;J$SGp{qX_e=ETova^h-&WScB#bq-8<@{G!kUh_Ss5p$Bl* z{$;yl_xnYzj;i(zStJIIRSBA_;-A7s(!bO!r;x*=TvT?O6O^7YTzk@PJOx8}~J$Qie#+OPKoOAvNqkOarrql=UN{aHm( z?iE(m*u{=GBcT-^lI~h2l+kldtmu#t&e7laTtSHFbS0<9{UxkH-bgaxcMSBlFDi+u z8U*=23VJTBX3!#tx!3m?;hT$v;5gzFRqAYk$}T=X&9LGD!qceIBHM>vR?5m!NNt|+ z8~TUF)l)FKIFac_Xp%TfE(PkM(;sxK@2uN0XIP}TwCwu>9h#-OOF=RNVZdV0_-FAg z)<%KMKu25O7z{5COfV4@=tq^BT>7n?2HeOZJN22XjJoINe?s}IL91_EL=5I>?z!Xf zmMRzTMvcFF;>kBNmZzUnA)N*!QVt#X6>^x4 zw_QGIFz$Qb-|LiT`WQrJ1bNB|AAOl{`Fe7^uqV$aaHKk_JU+WF^>oU%F(DS)x}!1K zps+v842GRIb$=1`Zus?(CE0*R)XcMQbdiAUY#5zSa&eh8|ZXKdCAl6T)0q zvY1)PYV}qO5w=KT1YB05iXSA0s_0r`c^}x3UC@Il;7+rLC zY_{}wbr|pfs)CD1B~D4w0IC$`gGHVL0T~!YjIg3Hv+2Lmcn%-&!l51L;NFqZfCS3X z9LpT0?gDXW+lnQggP$`k_LE|f2&>Yhfkj~HX8vF`(1`YhyY2D+7;RdRisZJMeZ+q* zlwF|xqd}=w3!91J46pZgW)c19-HT5Uo}0ehCRl=5Qb-i;qqLoPXls8(#O1I&LS1U% z3pPKYfZCx?|LUn6dfj;e~X~}mHZM{I4OF?-t`u0tLt%Z5t!X7rA zd*9g=TlmIP>l^t^d4Gi?hdP43r9n_j}tY^HKDY9V)B4Y2<^UC1RX-gk8>aM{eqjjmzv|lNsID} zE^P_2WER?MAEMhfb?X+c-@^u@1rfgCz-s)Wj=}l795mNSuu(-J5c`Yd}O@TB^VFPxd+!_vL}VYHgUK8HL7K5~eUE|&5`-Y4tk zXBu;srDKx4t0S)Mj$|V&@m$P!74&QLqPn5e)MK97<%$O5kCMOOi|iP3ba7)kX5~RC zR_0~N3%&EcKLh{>XZ|}2;Ak1N6;_4o4svzzpx4d}8{G?CH8GQh46$%IA@1)N&J?D? zP~78DIVJ8d(cjdPB5m$VD78d1);aJ7)x`)AkPp?1Vn{-^M7vaEAEY!&=yQ1*SVEcWq>IKq4RW-EovJUG{%D_7zq+!zkMvX`{$pdnX|*QrQuQO&YemN5{WPecM zNZ!x5O#5+%La@&}#NOr)6L#t&R9c3;v0a>acl8j~IhQUZeJ>zm2~4x1ELwTfD3YV} zK5Z9c-wP9bnuBzM2NK%_Jlb}Ov4)gyf-~XuGfYyKV$MbFls(CUhEjgO1b(pcswP+u zM`2w09VTD#dU~8TO|9$=+Jm49ewC z++GT_m1_`eF}E`@!JL&-_!Q0VSF6*wbgII!vjJ4zh}kb^KrE$L*&l%fB`)@OPryB( zbP~YKT~{0LxR;>&+*gs+{*)_Js7d9BD)AlNZws|s(_Q(2aptBiJZ71JQL1un z`MIlYro@<-fMt38fPWy;G8iUYr-$|;lKAbw>iK;wYKPWT+v%{7MW0(y`tbK$4r1M9 z8jBtjmmy-ME%x}KZ_T;t;XdR~KGHQxa_t1Z@5i#v#aVY@rurT1bx8C?aTm^=jQ)!v ze$(GSUdMlj7YM$*K<#(=F~`FN)^0$f*ww9wlcdKS*+FEvbet!QA1lPrRfBqe6kk=w zBq*O{UGGe%j4@O58pb|4wq)eTPS}i>bnvYj8YwN-q3qbSVFlX+LPzTa=3}<~RMcGp zA*}TyrgBU`(FJ<99pwdphlYwVCAXRW`$_Q${q1y18F9S%5k^G`Na3)mY@9#aLG;JW z)zMEv3DNePumrC-N66m>xAM{)bH3g#FPL0Sm(;k7w>U?Mu*4jdu`2+a)z}QzrLZ%9 z)gsi%Ng67+(Z7Td~bR}Lw>L5t*$a(*wgT+d+vI}c@Cru?Fwo<*Br0iME>CSV$b3$hp74A7l*z4>{ zQEMPPyf4I(y?@UiLR%VFOfrIkS`$IOtbSW-!5LEdf6@nhwABBinQEs|ra{zb6N zN;(Jp_ijNY+m;fis+}i(G8f*bQvq6PzJYetX>C3S2WcOe$e1u-esFL&%%#9*p9K%A+@%ePN`O5H>zKp?kh1cl5 z7jqE%e0|3#^Mn#%nicjv%#=)mD9gDf7Qo{}!C!*XEpx_7ri9jdr@{-$!&lPf2EoH? zr})CfDEjp_3(&CsuPV}?JuLYn5GoE{(EUR+?ozgUnn2=!(kmxhQe|qcO{t!n(yf*l zv})C}Qj{96yaQ41AhsNbQ4?D1djv68s&`PnVIL0?h4)i>d zM9GNKGk@@veDSw@3?~LY!8;EexS8OKFB8~(W#D2g`Czs#4>E@q_x%VNWG8w)!Bw?< zB07ZW|dHZVRy<0!4hP@u~ywo_Kc6Y*s}zBbb~|KLw6 zyUH8nCNaxNu{GymIHRdi5fWRE6&i9*M&kK`sr%gzi&l<+MceseZTSQqTxiC^^^n#x z?VUiL8I;@XG+C7}atSyRuEjw_*!%v=&@E`%Qgy16ho$=KD_TjpO*UWCo^FKQ)7-9h zgB%<{R+V%*BOV0c1|i*~3*r3|>N`r`qOkkqYI%>DeOn76!lxC#h4kts9npWHFV=ln zA80tBh8p5Ma!mRTu$VYpA@6*jAz)FGP)b~!R!^uCZyso{w=2PHI_x!L|Jz+UnRG4Z zP~f8n(f)N7LVV)!Ik%r__@UhC4+Yuk$SdU_nrPGhawVNY$lsPt(CQItz)7Qo&*@gzvkU6A; zs1cSm=|PK!r*e^-p)*XO5YF4Ck{RE*GE1X?hkdbIWh4Z&0r5Jb*_3%dHA=_M3=IN7 zRh#L{c~L?ryjxo%GYOqB3_%Ws%)uhxYSl9O91|*$xZ@9;8NMv<&u*vwHFD8xlwvKP zAh%*TKkGOc*4B#Lsv-e_)*;{x1>tDMj;ejv2U;R}x`hfQ+UfE!g!W1z(J};EHKvLJ z^1qfl2LH!f$Lh;!p-|Vg zQIL2(uY2*$gFu8cy&HwRFvE1^?NQdjE3>@$Vz#7US;v@L>dL1T^m-p$-7fu`A$IqV z?dHg<9P@%$!6)*P?JZz?`D0ngME~F8-~G6DpRKK1=MLfO&xCSv=&67g<2vn(lyMgw z2Q~$Mp(6$={mFSu+_T-G3$WnoEyIiyzIwE3U;Cm%r`)#$<;te@#>#DK;V-9JJcJMt zs{Aw@+85~>2;6O!AaFy3k-zpOoK0gA{!Tgi3BfENf@{8V*wq&j%9V0*tt&XR3lRz@ z5GpM>7BnH&&O7Qru1==-Ab|U`MKJ#Bk=AdGt%dF1n$0P1$!B!4+(28^P^9I`;)yhW z*o>s)&$*r`PHJGb{!mh$Q?U$W1k+R;+CzhS-;WR^_J>C0x<+io;gG}n5URJWY{J)a zCg;-lVH(c2$g~F&j13h(DUnDm*9O(KvabmA{*5SH7ST235|GB{+_^$2 zh$hO*fY>)iN60=`g_pj;q_2Y3IVPu{Mjf)w!(0}SGgmDQL8kfcgD&o~<)uQ$6_a1i zWN{kCXRrbRKhhqYSR%A30n$s^%OJHR4#sypuX}*ZIWU8eG%|iR&E~DgX2_^2po9sk zTNdXr_Qx#gH^VQ*rmO9S+Y+w@^9oA_>JzxlbZW!s*mIXV^(g0!%hb7M5>6x75A%gr z6)U8m8)0gS`Noz9{#7cB{Sd|oMWbMeypcNc zZEdX~DqhsR#0<#)&3jsCx%A}M8dUOa10~ZI^QHvfi#%dpn2QciVtieY+iiW9uP9AK zF8b}-wlFpbfpJQQd|F8)-~Ufy&>@cstidcpHv2$=wo?VikD~;<4OXqR-6Y(S_R9mg z=;8b{jcQ^Li~b$48{|1&tF)aGF=`xblBK(DMViPtWzocWkqYd>1HXcd@jV>X*TGiH zx;)6WC)-x(SSaU(PiO62-h|fY6K)R-e5Tl;CAtx^{cuLzn|S3H)pwz>RFobG0rQz< z52k&o^X&#apr&GY;*Zm( z`T%LQTb8k6yJ+ro{#AxT$VUeS`$$ zkB8Ci-An?4hBi^-q~V`La7@{8u$ zNn!zX2&lB65fei>iUC>^8#CHzcm>xF)+D}P$E+8@0Do?2x+N~^+WF}AC~tad4DVs^ z=P+B|=%F&+$xXo+?B5`5{Hf+CoIxoq-Y)H58KHi2f@+SpeN7L!ExY zC4gUsBzQCOW_qIeo1V%W&Mq%p490)C@1rK_YYSLf>eYVc zN-CoDy!X*jnGEVu^{)U>d$K*)GFzBGCzh6k+$-Lqrb65BA*>-niYqTuxjslsZ^4;= z^8!@tsx5iOAPF^O&=1Jj^c1f8_1XzDUx^eR`Wr^h(g>4Bd_&w`mmnDJ_P2hw@UpMm zGT7RH5#PlV?_|v$7lH{I2O>B_1>k!D0i=yk0 zyg-&PMnGC_D9y)nhyH!*DZs>Vw%Frp;5&bgor{$5e$;-e(J7SUMd)OCJ~L6~K9P^k zc?_Cw&CbxJ#(6_6>)5wUH72t*P=eU2 ze}T{XdJe8{?f)d_L~pG7PwbYu*s2#gG}*KfGmN8BZ2|hDswK}&+Tr{PsL}D%OB*+| z^z_uT3H$mYys6snQmKI{4Bdxj`MY~2qCIwu75Kdmyi)1+INlV7A3l<%X(%J3MH$>z zI%rTUyM2XqyMoso{$G)Sl!(26+8`vRnCa(~9Mc=pqv&5oMxcQFXW!T4;k`97J*w=x z-mH0>nJhGpX!y(eZ64unaQPb+5^V$+Tnu~*H#Ge6{hNYVNxi|(%}s`Xk14fL2;-O+ za3_1GHuWZCxdIkX%|HM z3QxW#OZF^DJ|Y-}V-I1C4y^8N|2+izf>@_6eBf}3^?sbt+)K)M*l^CnpY+ZC{{^7P z6z%WFxkZ7FQ;s190x7r9)TSutFvo;J^%q4+oE^fcpwBjM~LhT!qWco;HYy7 zFvT~m8&ZbSUy*Z-&iz^UG>VgJG^g?oL}W9ft2Kipv$aDpWfp=>e`dg9Qp!;#LUZX3m z;r&&NgALRWMk+DN0qX#T52IW?uF0`!ZmBx#yQh7R7?Hm6SlV7jXSpr=NJ@TJpg>D31&0+of5e~x_Ry>Sd%Znl)h^bY6;!~T|uhSz1bkl{E8sqk- zdfG3<3Cv+3y;rj#PeCp+U70^45F*&&|L#@9lSrGYXb4;;=w=oE(f0CdbPCd7q8W!z zCDtrg)3NF^#Y{9A1e-a{kFOGHt<^gopiXuQ!v44%Hjtz(4U;X zfZX5JOnSFhKK7th8r)5?^xu&1t#$TDcg{qj!ax*amyt0AQZnT>r$BAK4I}C^n6}qZba<0qDnNuVFw`3M*n4u+WKUf z5_3i69XUzVi&t)2YQ+wbYq~>JEA^o-6;g+j2eq0@B5Sn<%sF_-1JckLs97sLV~*Zb zt-+_KCpIA5W(3I$6e{3^`pgt+>!SmeYN@T1(frB|etui{afiobtxyIhWIrd4`Ob*x zp}d|)2}qJJj`63f+v{)>2Um_v1Kii->zPbq4bSV$L*|@vk8|i>hYQ;-Fj#?4V=;Fg zvs#pQX2_uVb$ZwKA^wvM~0 z+QBKSf18R11Xt3eXTK|9>015InUIggz&dqa2$mU&SM7f7seXyf%X%JI6+Wfz znO}PEM4I>P5(DFCpFDq3cL?u##k#rRAsk5;!4I4@b}b zvaDXoC+jaV_V+T?S2ed_!*V#GVBYb0ySLva?(zmsVX9SdLYaK2jh(cLs`JmI{-55@ zcd2{IvaM`N2bL<_tNvgFA8a4#xh>xxR9bb`d~2VtC;au`;`3$9=#nZ0E**Fy>X z@AL%k>}mzEWSjVGPrm_akAW)k0Qgr^3&Xy#^3QRISONE~DgR-a)<^I!QTZY`A<5^} zTG@0qgI_~tgBNWB3KYM3W0xe`72HM=eiAKbYXg%8is}W z$^|`O%U1`}o-)l*23sE4`;F<5)HL<(1a5`-`nRa+egw|obl>cAX&GY#d-o{SZj3W5z*%CE3OatZL!B3}XW;A-H&ZwdZ^j`s3Z~ zsGBfSMMePmkT_P68lOLrUuq(ES9NEu_T6SXlmp*(ZTov&sBLRh$91bJ*>OrgLCl{Y zgZB?ct>%R0ta|w14RJURjWabFpB|q|*SE69^RC(X8^tQ$TSmxYpcha&@GeyifSeSo z*0t2?%Q%88chFGNi zSOC_i01BdxV?DPOpRbwMzX<`s#1z*`T-!aMi9wu1H1tBaNWRbGZD_H!fFSymS)40@ zdd%Y%``HAxzGO+v{lRrz@@IRt?ZYL#RG>)Z6Oaoxqjx6DGEbKx90c8!aA>vQ8bXa9 zok`*lmLEM!BnP+gT6-xOxCPHT6yZtW#*6DokSnm8dHV$Y`PawYb*n}&EVNc_y`zga zuf~(8j7;#Y!TFYBPxCS<-D_NH;SR`45Pb(!B1mu{=-l`N1dUFFoAi=Ueccax0%rUT zIUT+_((-s&bC(&@Fn)2LGS4IfL}(#`dK*7x;_8%SoU3 zcHgdr^7(>G(MUl;X_NEZv$cVcNg4vaMgi1~V&?+uW17+O?5|<7DamiHEg461Y5*BF zV7^r3ZnoDQz;|MJz5ofmmdYn)u*xpIu4qsMB8Ps~)IH7dOyxL>x5!Ao)^Bkk)#ze2 zsr1mjCV%R>E(T|DTIce&-u2gd#euUJkU)ONpjQ)e% zmOT`2&g>ih{{WH&zpgCZdT_Gri|P0b-B;@R@4NjMaQR5J$!Yie2W_+-oSNDJ1qB<3=^+W61GkKJL;OaUP#%AN60snPKak9BC5N@pyT$TR^&~%zb zWg16M_sIXjF4?>dLev}K=6`tXU9of^n`9|%ejK$~8 zi>u3q61W^8x9$gZ!ZXE!FLbIf@+r+nX=AknS8}X5;B_1+x;Riv130JbY7qQlDtyIA z1IYYNyNX<`uWecS8ik-wZMy7j-YMnl@(s9>P6NXt^|2mFx}7qbQ26-_QX!$~SB}aF zVo_Lq?Bl|yKlyTidam4TJB3pH?C@Lb-a-GD^7nuiBZtgKdu80CIJ8dQI!XY8Q;bXW zTRc5lgTjq28yIzDUW#Cf*-mYgOZegUuMBc%WwdIxILOq6^(vSPCxj=Y_t9K46#K*C zBZ?A0(dQjuj)01HA=0k6j{liR3D7TF3?1s$;ry^!GIs{`G_teSe;HDOlUkRyDv_T7{{#JDmpuRg literal 0 HcmV?d00001 diff --git a/3-NeuralNetworks/04-OwnFramework/images/Cross-Entropy-Loss.png b/3-NeuralNetworks/04-OwnFramework/images/Cross-Entropy-Loss.png new file mode 100644 index 0000000000000000000000000000000000000000..c5a67eec7bed7bf42434341631c039213b56c2bb GIT binary patch literal 19022 zcmbUIbyQT}`v;B=An}$|38h6TDd`wN6afM0&H<5*q4N+*Dcv9;A`QYw4iW>>-HkAe zbk5K-C{^CR(E)*o0T77L ziHr!}^I@@~3;%P+?v?s05U4tV{K|?LPZN0RD8B|(jWBQHiT~W*8he63k7oY&C3rYK zu#X?`UhAEn;_dD2^78UQkT8K;dpoFI0i^9#sr-gO4?oH6`FDP3tYVuhYi6`4B<8jV zw6nXzlKc5;gQKgft9U^qXHFUhgL#bVO?NM{v$HFZ7@nP+#TeFHU0schkN=G9i;Rp^ zqcVB(<_$b!B*Fgc?}pzW^{_bp?{P6qJ1ZpZMDDVXr-Me=$w$*8QY&%rB;ti*p z3`pP3;X|6u@AUL^$v6JKzP<(q2B-~WHwrE)DjG)}m!OmI4fO3M{bPQ8coWKPRNue5-l*dP zEV+K;s4RW*tBZ*5Z9>BB^XIqn@=^No&co_sFrhy=qK?a^X&$ei6+~5dQWbvu{5w_R zMVmfend~5yH?Q>k7WQ%nU3X+?<*h#Tk(H%z5Py=_|IEmKUyt^oNT??2_IBkyIKibi zU!J=4#Z1=8Q=d|TA!fk7>Z|!Pc^6q14$eV?FI*Homh;InL&x@m&s5s=U6*I9HW5Hs8||0brTc{ z+$I5&O3<}r*!U+PNeFawy?I3GxXy(6d#i_V@UiZlBw;@i;6Sc)tG6t(E;U~~=zRbo zYWV~xS8pm(W&m;NUITX?Jig&-S3-bb><}W)hA@I7MFgmi0mlIW$sT;VBgsc&7y`%rlg`er?9u6HejTpZJz|<#?Lr~5 z*d9ULjy6{NGXNWN-WdlP6l6|OV}0qZ^?UVD-6hfe!uj;d){{L0uY}8e$ID(f?`<%+=>OHKL7d-5gV7oo}>(2TJpWY?`^5BrFRTpVz_+kpG2Kwzb6B@tS zjLc)X1Qyy_zlO-%hB}OJ@1J2RUL2O8MX%n&5(1TeJ3AX4E|H0b{cZgXLR#y3B?%yX z{Uxom)e?_hb0QCa1k)~4lg?C5rZ{FQ0kTaJd`G26(+9a4aG$-3wF*Xf+RQQmZaKYJ zq|Elo33gmyl)4KCQ}GWLp^@BSV@khJR?T0oY6kz!2kHYKysWhrkyv92hy*;(FA!{t z_w~)au{)c-F|j=b`h(=kJv9xkq9@3=4*o-$iQS#8aTV58TC+}jXG-my5sk8O=Ju<|?`6V^X^3vnWkWEV#Ib~Px^Z7Y96sO<(B zsw~-u7R)q`exjp9)@#ZNvo<(MI*K0TM%a(&mp~N|NEwy!+pG*Kq}b{Hbhp|!ocSO; zmXsmQ+&Nav!M=_1*yi>fB4VY-y%PQU$%lqXn&5pODuzDa59=)VpUedl zR|{t@C0aAR_bt$=^k`Gt>@h^P?&~=2tFOgN0c4o~MBzIJ-}sY#H?`%zNBWN)d-vB~ zalUp{#ZkG`$4J}7iD{3ommy(5-25+*2Ff+^TF092N4-Jb5vDrLTt7IFlLd`Rh``m` zPiynDmRCv$r=&FYqgKxAhOgW-6IWGdtOhw(oG0b$i*QqLL#zM#8T%1s_&dJA{!csa zj34bAtnbc7W|yp+f}_>eDvB?-MDmWN^o4@cp%oV2i-3N7(1<0C)q7)6dL9K<$_-yz z+9H6AhM$slcNW|-j?(_PC?d3NP@j^FD+l0?yT(#?b&^w#Dp|S}kK*2_b_si}?IsJ= zvbe@%RX?u=q};$qmC4>?atAYCz2CPD?@k%9uxcO>Ph#A@0G&Jyh6G3*1e)WS=C{CP zp(J#{*wnE{ROG$3^}FJH4j8L%i>c-nNBX|3Z<#M58B)>Bt?k&EBmI(p6P>#X{~q@g61d4o5^{)QIyTsAeU82| z#UXyLUv`UzxZ`%zAOGWFHq`g$!^Ow6bROHW1al&(lF)1ukJi*a9rLzGrlAw(si--S z*9k}7EpDbck0DN0M|pjQnx`yw#!=^+H_t+zAnP+XoB0fzA0xBZStBEZZi@gFkrzLK zu%<^1J_QOX;=03AZF{HmcJIV5_g^5d+LDt-WS(l?=-((IJa(g&`guG0&e@V&rpB(d z5%CPJDU3KDqp`EGqqM<}XGbjh+=KC5*+0vH!LJQxMgM0x^7RUmUt=$o@M~D`AS+q^ z+Aam2746a=wf+Z9>8Ml-tbY^ql&|<^X2@ryUoH^{S~PGkP(q-m*E>8OyV*AzJMY?) z4%6WTM;N<{j`Sx3?Jn&gCO`yG^lG{RZ0f(A@4U!?G@XaL?PdhZxVAN6Duq5RmIIPM zxq_eL1|+qg_?ro2rq9ufHh7ZltB-X>ZoC{Yj6|FkH0mJ&tV~btt}h9u!8oTpI#nG0 zwcH<{3VMR2Tl!n6S`gnl!nGkM?H_xk&5e|9&Xmmp2JO9b-YOfcGx=4idnza(*3a5R zs~6FIIrdEY#aaECzk_MH*P7i9p5*U*LcE?qdEn@?Rm=I&KI=N}7Q}dPNPWyIcGu_Xb{42lz7*f>9 z^4kE@8+X~o{R;>5-%F~XX`*?2_sR=mf!e!J2k{2SN7$>>s>}Vy*vpw=3a}U-LC_-+pFZEQ1__WWK&7Jaw;Es&=M*wOa#^!M#T@+TOB9?; z7w7(_-J#w*9#g=s6%(x70Gx8fu0?6*B*Ll;_sZV8zHcXYV(ybnfoU6Bq}TT?SJxkv z_oM@}dO|nm#a73>(S+VMpB@W6IWZ512mEm}H2n?ZCS~v(H8V}$?r^~kti#vCgS-z+ z5FfQEEy6zw;?6iczqoOv3-aL7L$*vk+14U%`0LgerJQI>)|w^D8B8ZaZeSt;Va88g zR z+VAhku^e-XJ;jy!^td#>C7-@5aDv1SbBH5{u0mIc)9E3+kPH(6^1LR>qE%Qmx^U`W zkI}aV+Gv(khwuFJi7wrfbuShxOY5J!%$NPTx*V8+evs1AKJgDSz;KhC@Oxt0&4ILX zSSZcp;YcN2_w~mCcCRSN=9jwTvX@q(%vxQ9xSNE%R>Mp@5E3M!bF2L#in{w4K)B^c z+~u=R{+C0-IoNmqmq((JDJ@+S@7Ksf0d&l-8dFYo2xu+P^T(`_3Gf(f{e%UCWy8n{ z1=~t0Zu{H^hUr^L_@4ehCDZ?ZUt2MVRYq+3tI!_tu%w{;Rli`BW!bas zSh{vKqn*&0jx!J?;8ZqH3yjIi}TE zF}nQV^yZdfkOC4GH`(~7m*_=;3*+TcH|})!>v>DmYb^cInf6U?4O);9c9)tVnyM)6 zo6tHP8K1`rs(E1-(TBJmYTEk|(=7@&JePG@OjF0S%P{+3>31lok7dHW=T-bssQ1Bp z@5b}@xF+}Z1GA~Xz--Og?$y1VhtaUR|uF z>M+C}#Msrw!6JAr+q5s57Q zW+7X0BJO~GZcV%`4wZd9MyxmpE51P)F#nuRt^f9XxtAd8vQYJg!uOj z&vS<0jQ`qeWUio$+0UC&jFv{pMYM4Qx;z&f14{#Z^=@)pmd&+~hbmi6=yWNNHW$I$ zO8w@7GWPl%ESlz;?Z<-YauonZ z4_I6VmY<-=-I`I^VBQv17W4m*sPC@TpaxdNZdEwizkXp)?h-8CXdM!oiODUe9L>NjGMh+;CgKUNtm6|p1jq^n_Ldnqp7oRr z?Zc=q>>qp&aYcb%cS=q&GE6ouHpGGmP|d}r`w1EXPvxDlR7b1Sxz+V|K9eJT+R85} z*<}DHhqkfn;wy3*U7xomt)5Fz&^RjFB_t}v=4#=J@J53!2e5VDH_B|BlkqmD#( zqM*rj(bRu6nBrf8IwU!O*;q5V|HzOSOm1A8ef+a!kUX|00&4Ak<^2APh8^haIQJTx zy(Ynr$bwGn6xmV}bV$koDfI{NDJB9k4X` zRGlLJdyij^>6}zB?@l=M*V)+KV?3RmDSL+u=_bC!5Q=GyX;P3hUxzMe-4KUE6AfU` z1N#mbXb|qxQK*(Prz>K1Kw-EE8M(|MkBxNXKGjCXrQl~-%hpDLP)xAxBNcrAu0cf; z5>L8V{vXTr!>oki=%qDi!Sg^K{a{cyH1+klX=gJj8B%H$&myv(Iut`$Dfcu@yH*CX zffvZE)#%8@{|YaqpF)NvVTYu}8I8s{6FXoXx4Frdl^g9#OM}pEiZ~!&g5ss~a5zis z#WyH*a%4aJkHyug$m6ShKL`Pe$w4B}elG|w%lEyX(4fL|N*aW3+8=VF|6A5fwi&3Y zxWxD6+pY>-{GT$l0{yRiCk`J{(IBjBqW+CcNoq2zjSavM$g=O8)t}Sx6)2YHjC)!B1Q2?O>wiCqROP%sqspUXHAo z_6_m!{=X6@Gy89?_}db^CV0hG%x}aqn_f0v$u!1xl6pi&z=D}GCDw?=}_N396x zuLX?RGy(o>Z?!g0K+0q93wh(7l0iD^Z8cY75g?RW;quSZjv0nCDZsliZOl)~{fdCf zeRJ0}3T*gDyqPK3>=GBr%o-^YaD3k8eB-xjQ0sBg8@9?6^uo_+_)NWZc1t|iX${t@ zvjrnKA2@2X*K?j|@jI`(N+-sv*yM(gdXTjc4p<)oU6{_O$+CWP{O~sOAjR zWnMx$VsYe4W)@YRoAK~05u9ZLF+Or`gqQF9qW(U@_ddc0(QN1=6?}f?i?i9}1NH&6 zONn65xZV!B}j}KnGlYS&tRb#`#YYCJ}j<8PY2;QT3IJz+R}*?1areCHdTigPt=Ya4M{km3KJ5DKU;oGB2#>pv zf{~Ap0Tckg58h*DX1*G)e>BVT0pYJ72!$aSf+>;z;W>0e;KGGS{;K{1cJF}+kVh@p z3;ArG4qQ43;-MQ~s-C3%^)2NA2v935{~2Y86#Y) zhenU|1xw5uFlk*9E7`y0l+!i}DvZD7LQA{jummfntefDKQ9rr3xqKaBpf(DvdGEDh z?Oz$>^P`#8hgDCG1q$8-aXS7-p%+8x5zz_rZe|J0rthPH-wa-9UR|15gA*o@{ zG@JktDwJx7tum--v?PK^AzfNxAqfd>B!YuM8TLPku&?37cu*78uvLkBNvTgcvjT734D^Vsl_TAUqRyu&=e}L$&TjDF{alF;sWbdp!^rx@$fR zg7B2Q-Y~rzbhBLoeVh^GB~Aq7x2S%Txrla*`X__zvtb~Bv@;Bp1Wb#Xw%pk?)%zOf ztwFd*Ac6!`b;QzQYrrQE!PcEuDihr;B1q4JkP z3V6gAQ1yN;mct~K`ysY0P& z#2&`=tVZqwDaBL=bNT-lWME$f!H7q0*$(u2vNs+F5zwOAUr#{1gJ3)PkdYJ39}6gn zO{%zO$mE}bTkqh;Op?m{LyFcwOuJ>tG7egF$>;n^GD$Pm{z7= zt-dk(n-w@w(m`y1O0(sM*HOc^DOmaa=Lz`e%#Q}JIdRWBU79hnWQHGCn#tNFyC~Fp zpu(9CUjCD+Qoa~`w%g~6<2D>l2#&|@I343ZKH2X3=fzQF>E-1mGB)N^9aC{x({}qv*?(Tlxh{q0CpqUzbHvFO7W>RAP>*@D z;sWLL2tw^-_tBn%e6ylD9^gx+f;=NY;5@55!1K*M0ZDJ7+z`@VSPmj9=DRe2}+C+XgN8CCl~$~P@_V>t~y}JPA>TVb`^Ke~RRo9*6_QwmYp51CZH=90dd}YTt z;15UWcOL|j2;r|ARx5U|^rdgy$OHll!r`1Km z$Drn=Dd@nvTo?<^M3yyE&$Jy!580?Lc6#?!Hg@5{XlY$BLmV&-g>D2~CXlrTb$1j~ z4lDeW*7i~#`WbU-O(Q#EDE4vFYKh;-Yo(F;k7wXDk66UXn@-p(ZVkasbzt&e7y?%mz2VB{utcL>}hAH3pX3nBv49tT_IUH5OQF=F;l z?|eY4kF6h0PkWG?J+eUzCK#%NSJNnFc<*rsM3{1mU`={viICq9*Ryjtj5kU8TZYvm zA0)pG@4zu8p(oB6B=ILl#`Ed~=ZF{oL=B@g=SCSY!%;&r z$lUg{;gk2lr`5}Kv1XzJCN-a6Hh?VN_OMxxyVu3`c9frRu!L?^HLmZ^B95os!P&DL zKRoYRJ|BHzy2h0lV_MQf(4TF$iMqH;FG z5w6&g*O5(rtMQ*_9xmP&Zo0=caHlU9mZT_@)2{HI5YnGxv4>rphJRzKlJ5Jm0{-`% ze+oU`@~m~n_D3p2ulsnezO^PS*VqXWIgVY(H=2u@!Y(2vJ@Gfdwjj!ZPycO zpvvZAfSC@P{z{Fg-QqHFHXS_r4G2&*P7m*74t~CE@bN^!) z^VNoH%LibFSk0JT_G~L61$^gz3OhKekR#bvfMXa}uE1lb;v3Y9-^W4zy;I_Q(7Tw* zycx~T2B!3D%cq=agviiiyj^y}V2rbiN@5AujkTZZgn{cAJCSG>5pM)Qr>as}?Fs5n zE%{L67$ryy;JFISM*M2ywLy^jUNfyOQ@Il(_;X>_fH`LR{UYDH7q!fjL&*P6Ex_MQ z8KiuzxWqKoybOd#;MAb@Zz;oFJ8EQ~BkA^PeH;=z%O-r$qoZ8X+j}sK_$C)HaKQFO z0AoXjOgl9HV;DDTd^-91<=HnC0sl3`7FY1z5*_PoS-quo{hEtUp|dE2howGM{Q~?} zKt-k^dW0~`NrGg=aJIsi?~?qYCYI%gZR7BKH&yE#AtVx`Z+NeQ5|Z2N{(vbKH|nH8 zuES2epR`IDpgwT>gQu_P_jSiFM9msI+0&(_?j2ogG-g-^RZK#)lVL4R8 zYBa~924TDV?2KpUEEX>!RpHg~o`sb{7^z%ZOZpN^#Ln9k|3lf(0^ zebA<<*OwIafx#isTg$%>qIkQg(snKaVrFOGvP7=~I4j)c9E>V6j?9JI1)iwe|(d4gju0;8J&Tq3j z7<3an?;<1N?S|05nXEu-(H#~i!_q^#ik9Vxj9oatr)$mwZ1Fpf``lg#av@;aL|`Uj zvi*FyH88JFh%*!&cG=u~&h~G3XUikt)puTXVd9{dNW;v499je^cask=oAQ7JL)WrL z_j#t|9SGv9{*@T%AMT%vkSTBVsXMH)hWv+(1Iw_e=i zK(!6v7E@xbrYziZF639TF^UOJ9yO4`>h3{Sw_Q*S=V?v8h9Wu3gmxs?mf8*}LIZZ|+LZ(r{DA2U_^dt z(7e(`wr{|`;>uB}|A``dO?WqVIqE#TcnPs29J<3-S)-_IyN!xx(%=5=k^~6N1pFBs z0zy9SXB>`bYNrN#JcsaZGJT|5X`bP`{U9R6_L&~czs0yw1uKDt^pDMY&PhEIG78AQ z)Gx|`#U)l_1tZg6HnOVuF8(=`GQ3MY{CMN5Y`=4J3n)|2ALT9`wL~J6@0K)-eq-8c zAJdAS-kQ|Oaf}jMs=gro)m9-Ecz(ctx?1-ilk!{r0bIbtZz{}6>(p6~qG0OsEMY>3 zU{_M_`y`NmQL7!0_Y5bW3jA9+#p#M(WA@&&1R*u+JD+q12({k`PHtBe6r`jS^s>PS z5NV;Xxe>U^FGiT9J$|)RM7-4sXXo=Qok?~~VGJ$8S-r!N&&=_N$@?(C^PP{ampHwl zl!V8y>K;+pj7iBBFfmW1CLc^*+3VDC&T9t_Ju+Nfn^uzf`sK<#U~0NSf$RMzlM~YS zw#Mz*-*7sEP9il&$^XSMnR2)P?yZETXd4CQxTn$~Di-@CPICNZAP)Jb5BG%H;T#vU ziv^oZ+j?73){X90XFmdv(q9)A#HKzM{%cVmf(NLmdkV!Ma^M-gT;w zm;g}O_Og*Z)}c`Sm6%&A?orK|8(L<>!$#URSJ4Cg-~YJ^Mxc|6FoYV`&H;fEHN(++ zL~2oGx1QN>nSfSW_NUGa}?I$rmbTW!3a2el3-LV605!tAFnW zuGIDN1=owFsZ9SE1`uLBE(57YjYFYDE6ojEzK9pD-{)U3=efi$2&_2yy>fVysF2ui zZ((yrK!rRRk+8}Us%@je35G$NK#1v3=%XB=0$45+Q2A?$KV0Z^X8JA52_6b1Io%Km zUgWI|)gf!0F}`I1<2XKN7B@s-N`Jcpw>7CXANGDC+~4eHgtIT7)>_4xeyqw%R`Anz z=xnxdPegy9|JmOC0HN1G*K9MIC!%MRyL{~oIcV1I_Ok9z^OgslJ(xJv0YYr@T}ZEB zNYU4Yuh;?UNVKAuksZqG(6mExcAjd!Cu92N)3wdTb;6?ymW7xKWwD8bdd15cfWEo={HTMd(PlR}NC^Uy$$3)Ci(YYfyI^mmV z52;kt6cdc~Sg8`#Hptz)-~ZnC*i=&><<7kS*LrNGeB9j3sr!A@8jZ@hC~~*C^ptWB zCUzsOkehyoaPzW`L*Zea1eA;FO3=eUu)3biF!@cR%b*1TQbHD;Xf=Ojpq@QleXquF z^|^=M=4DvBH-h(^V;io$yZC1jNv*84uO)&-RBCKEz3KFLA^MJqeuR%U4E;jRi2DJe znEvl4B@~m7Yh8s0rkfcb);pszIqtq#~L&F!9B#&;O}SW zXYIFfwY3#fediP#xM<$%#lhi|>|dDa!xZ%7CLv&-yd~sTcGh3~!UCm+KD^l& z<35m&GR5VfBk#}RZZ3~YZi59^Do#iP-qv?2BDNO50E1Uh=+nq~suh5|5FJuQjw8g< zqw&4Cy^R$v3<&sxqsQs!j4`wuKHt3DrimE^Ay$A_McN@#TfRGVUI@vkxj_5_n@#d( zF*^^s&%rDXXE*B07#BKni80=KZ6XbhxRk0-8 zevZjO3o-4GJJ*&wB!+-9b}@W&M%IS=09yS}$6 zKyq3tI8E9`I~#RGdx^R^m^nvTN8VV31u?IfK9b%zaQa*@AED#0$&@h=J$lrJ`merG z1v}6g@T-d%9XkCAd*t3|5F%VTX95WspvjTPV;9}FXGVf9goDz51l3_^v{`{X`d9mR zA1X%(W$EwaIkvKuqQ>6vDu5t(ur!0k?BNF1v|OHiNc&JX^H>_mz1#XW z3@W+i^zttE#mOfXOJCuyB{=7sNki_k0P?SRjAnFPC~c+E^PW`5!pw(Rz}QlcyJ69q zQvoe^5V>2=6w|@{1&RKZVPaf1G(=gNm{qq<7u4ky8C4?zz_%B8&k^wZ z+tvOz(mb1LC)->#ZNE{o)E-NESXC8l@_jL5Ia};L+k+J!kB>sBS@@ZJlZwj=XajZA zy_CUO!24-E;BY2uhu!HO^)Z^g*&so_nnN~p;Eu{833#v8Hw+(7o%$xQfDj<$-HN!s zlt1n$;3w`D{3(6O`ciRE%jxq?-iOLwT&`Cw{ftE6AIWrFVo z!VqH(j6K68p0V#fetID+X)e<^Bst+7PfT4|s#YnNw2DcK4}0O4^Ok|D+5>rbu#RW2 zW}Cd-m*{ujl6{cpey69REqy|S2rug3f0pPqO(|IZ^(x0F&k|m8F7^}I z`_Y*mKQB6b);jd})s+(59nxbD#y#5O!Ue(EQ>1Md_$+g#Cnui&!_Xe^mJ+ zG4I2f?r_QRh|Fm&8>rn-s{YD~%7#D98rCDj{+!uCJkx>nYD&uSP+WHK8(5vt(n8pl zQ_IoAW7oi^@wsvzO|W$iUu8>CPKIj+SxiWt{n1DYd2!YkTgT!(**L0fj(mw;p2+7q z{y*fw9L$Afue#b9gynHoq5}9g)FWSGp?tMxv+~(#%Cu+i3wM79ryZVR8B25$mW4@${c})-ov*ay{Siu-j?+9 zcHfhvnLJ&x`LZwcPW-_zP6r9z4m&m%E8KVMC;vqD;jKj?-aka>c=NApEAeYZuR)qlk^?&{=v)WO|cIK44Y*+~?bApS*`? zu`*bAIYMD||~po1%R-$&r`h}oubHT%biu25@COdF=TFZwF zDTq)F%t{fK(n8nNzP}ZoWqS6e-krmK7*jlU9 zfN;ok_jGRWscIMHUk>0ezb3}gb3h_3qABy#;|DA7g~m(Mtj5@vfp&}kc+$HM3zzEz z`3dq>OycGnUn#E3Fy$8ZRT!r$-7Tkgw6ZE?+`0iBd$hd_%xkfT9al7EA$o4K*3Q@0JFO&G-*4- zQ!%8#H)WcWZ%>!bV*4=mExQx$h_xpJJ;w{NweE~e+O&&RIaSZ+Y;2=2?g(9a?7?Zx zZ!P#kWF_pCc_#}8M2AIN_HFNysi_)u{9l5P zRTb#ADKe5>Gv(EYts;GHQ+nJtrZb@NKJM1iD5X*}4Wcd^i)d2KUW z+yIr(>{H{7R$8s`#j1%W{}08xWv>+Zt;T8~l2q6uQ%ePe?^}~KjJckRX6mo~y3PzC zrlj++MRYyMY=L=_zfFDapQeas3hV7LM7B`j8eJZ_N6f$jS2mfl`DF>f z>QAKr=X)sySLP*`8N=)@=IJW%1@~o|}$-9{iDaow#rRnuOL>+(%C$->r&A zuKm3DS)xIp8~OXLD2_&|Pl3K#XjH|JzfE!*DY7W9ahzZd~M9$f%!4?Hu;=<15?I79QtET*~iqK zd{UJLXZ;kn_xud$+Rq)|)3xsAU!hGL-<3q=vmf;hWhuMu*{k#)|bzBdlP2yQFq1or-37ua%bMG{wrIUiJ{lvA^5x?(e+j z%7ft^V(cf@7e|y8E8zU!E5G^&E|Irql_nr$v2he?!BVM%efYksEc=*3r{MrjkUV<^ zD07JkTc)E7N!Z?mIG!9sFb85oV2WuH-O=99kG|&G)PJ9%n8H8n9Kb9^&wQcdOE&r} zz4VyvQ=}x;$->RY2pNL2J7vqDN=(r~wIQu@#?f3)5hoT?RsVYewJ3B`SP)>!oexxk6)MJ$E>x-`3tD0Hm$VejkI01xh25x z=d(w;6v_k+_$q%HSFIbk}v%-dL8?vYYr$O_o-&ngyYyQ zW>2ezk=`wSri=RoKz8|@8o_K>ozbs4BKImW>4-1F)KnpwkawQtcWNi(8Fo&Ox3`Oy zF#hP@r-vVHLCB4O5trvtvufG}sSlO6p2$*41*%JGjowV%(RcEwOi&dyoqI6Fa_tww z5zs6EbVmC2fRIXzAUUGm z{E@iGT{Z=Gc+_)|8g{}>DjfY0(ww625LqOz$t5~K*&hdNC~k`sdyf~;eVnGfc@LM#&QCJBosK?uVVjnnM39;+IEfHS)OH;i^q-%!$0+{vu~{TSpV3+)~t8Vuy)DOB5zy zLS=(e1=MRT1@iFOz{Rp9Y0IZtWI`|x?1>~|kHaDMn17b}gUz{trP zwPh7msMa+}lj_!@NBQ})$L*sKr;{MLA-GFRKVN_(hF8uDS;bfc_RW}yQncd^{zCUR z!t)~&Ld$J=a0@OiJJubr+o^JMclrhQMQgpp>$F@$2--;wEcwMtG#IzO4T^ zm46ar*H5=Uwg+zWPoRap?zG)rCGEe=ct53CaHhqs$Nv&HAnLpD5sL?Qmv~chgrtly zaG`#e=d+9_V!6Kegx-;^_+1pTy?POfN66w{Z{C#1&f9JM~9@CFs_Gt!X27M zA6IuY3T?kw==o75t3988tX%O3gAi^}%YDEnaJlyc$X|FIvtaAxMMNw8S2LdZJ8MX% zmW1Z;Mv?VXmsp-||NaP-nz56|Bw|8B+4Nk$pTs*cv$DZfKHHmfsVyqa9Og` z(U)^Ur+gA~T^)wQ1i5dwp(rpi+){O1wuoEohCWVOn-#1^ZkG(VYZwjo{N^eK=rawl zG(8a(KB~{#e5r?(6MCa4lzQx=vXx42p<}mk_p3A_P8F-%y|T3!|0c%@ie#1N;}HGA*6X#OYRmp=hcEuC5I zkR)Ils=BLCTi#2dSe_BUZ|1F6CO#K?TjmEs-q~++9R)1!i2G1^OBDp z8FTHBaf(_BNC?w=F|s_RkP04D`d6JX-r7MY!=jrTG~@Nn#V_RH9p~rGyv><0^#wnz zGylyr;DXxjDBQmwk;qyQjP)HQU3>&=kFhY1+RpVTf7zOklR|hWMD;mH&~@14 z#3z&};{W^iaoJBFJ>9Msj{abn^R#*6mZQKs=mtA*a*fUlxjM9)&byV>aJH{%5M?(# zk5E0eAR#vvkPuZ4{}$0NF}0T2SMc9L=!i&xYsS2%bH%#ieMy%jNiLKd%{}55rPD;e z|49nm$VqvUUD8$WazjoDI%cnu=uYK=>WCmi%SAAULTui11Krfq4`PRQ4-qL^^H$7N zRV)`bXBcUubZ$I?@uQ3&Gm1V^W{ZE(lHT(38RIxJpTv9e@wA5`)^l8>(kDs7D@>2U z60t@{&wM*;@?0P4DkHaWoW57htezp+pYN?_4|pD!uQ#*T z%?i%ls<~*m-ni4GdlKMJ1U#Gr5m8Ne+6X&iF3@VF)$-@Cgc&>IF&d4W*H;oyi2%ax4D;5C#{X{b0~ixhlia3bm__zQX%bqBcFX)?Ol>>} z%jg(k7Og^$+qsjzPoWXX!*SNL!4JX z`~$EvZON8HbKi$NaHp~^YVEd!ub*#`DKNyJKh74BSC?O8MQ(7`zdMU=@oeA=guvo6 z^{c`B$ix3vBiH`Vblb~D;hE?p~xY7x}6VUC2cJ^6m3E_^Tk9-Bb6K; z$*PI`*XUk>xa+l^S&zZ z4llnq;pUt7EuZYcEI+{MC8heyY&kNt8gKBiamtG1o2It#-JL|)4<<-o+*SKRJXy2- z@yk;V0|2n`%=*6ot5aQ!nT0}TpAi;G74ZfwgT~K5kVA6X&wJ4(iQzFx2i`Zs9gsbi z__X0I;_$UFyOxSBF%zlB3TR_U9Sf61rIL1nHI7*(dt_=#HMfuEVsq%l1@^(;;s#d@ z%h}Te?J1SnAJy_N;)XX*=jb8{mnBVAa0E$RB-1H#d{TTLXDvK>h%gMARFJw44S$a1 zIhi)Wg%2I*VqJF)KbkJ(W*IwrEzHBE#m8}0WN{hjc)qePu$MAqMBnU zO19;9_Hkp=*`nl^t)jmA0x7!`Bdg87RF{@Lb?EIRVyUwHaNR0>Ebq3(NffGIz&D1_ zgfvlfw47)W`-(_lVp~!+nBVA7CpPNkZJlV;JHc43?>v5dXIaxKDGv(uFrCfK#&rj@ zUUVN!$Z<92lEc!u1&Cb(?Ne#L#xe3mex>^eO28*H|EGNpoLm}8V`f#uWxy$AZQBMf zl&pECH{p_sy?`%rmIqgCcE>MOc|UGy0I}hcI8pbf>0heEe3?x#Z6(mJp}R3|NL0%R zMvLk`+=#XkA@#Y~Zi5T55CmkMi?cfk2q>IOqT%%aLf1&Qy(g?&WgY!@rDtBP>`2}Q z$xf+T*3bmg12aa<2^G!XUOdBmA z3AyA{@=eN*+4yOU{80bd;7HWTuYj5GXqJTc?61OHsf`2JU-a~;p&_xqMdq8LjZf;E zR8-~svYJqfMA1(BrLzN`Uk+zO@I19{9wsQa3t5Way2~6$lHhNL_zF10Y z-s__?e{(Bs5F*r1R;98=MmG%;qhyC_%O!$})P#Y#AX(ykPNdSb&)H^lZ-OaU zcOmQLuhkzhLj*^DXLxZdl6@ElvRcNn9XCS}IVINyqr$3-VEM;jG6L!gl;nI+VR6Z$ zdR`_}1${H2l=}Izqcl_U+v|#w@$e&urLkfGy<7sO+$@faNEB2N?Q@F9m627{&5*u^ zLGh%GBgo3ncX?}5+&W4^cyGdQx0^QyGoJB1BasxNE1i?)&w=vpNS>l9^e0bKXqQU? zS`UmM)qUIBZ5_dIL&$TxUGYw+FL7RB6F_PDgEFx@0Q$S;GofsJIS$(`nY6J5OIbSB zz5r#v|F@(&NWzfIdsIU`6b5!cuKd7uUfg7g;n+!{B$*>J0RF=lFGf#eg1 z{Pf#?FD{qNfiv|}z6O3C>&!P7QfrHnzw)&4=tq1F1ID5(%=hBlTd9`hA@$!xD+@5A z{_FD%A+_LXd_4>I)yf$3Ru@vMi&W6P0b;@IQ8?K<^dK^V4LjNxxbR{pWD#q15VSR+ zwzU+~QVJwN@mdbMs!9F%SO)H)L6;gnEk!FLsqpG?C4*AG@{YStzLZO3 zb=Xd2Cfi-4f6Ld-Q=t(RfGd17xlwdvaBa``Mrv;Kkq4m}YK>-|=NtBmBN05jT@C{G z1$m8G$=7H!`-9$&NYBQ$jkb&QV=j$_d!u_TPqF#C!7~~{X``LUK)=_X)6bA@gCB}T zP3L|wX{in7?aCW^0%CWX^d^Zt<%hRIbFkO-GNZ>ZPMz03~%rnoP*^tRu`S2X3LPAVO3<7~jpsG)`Kp;Fg2!v}v zhzB5fqt#Tv8@`RahCB#V5r60M^)0~0@z7FH0G0LAuLA~=tE#>S2t+%Mec{jy^=t#6 z|8sS1rJEQ~LsKjDBX(X<82~#!Js$T2-Q3*VCZ%bz#@ToadKVR+`Sn}72le^x%t<6j zMMGyKM)GvNd2@5y+R<&HHgalu1~o7Q0;%X4oBzpo^!9(d)>qV@^x~%4STlS5W;`)* z`etbv-F0&#c?V4?* zA5(L(uz{PK2(b|SLD@Jo4)k)oH^aeyOWj5c)l?-GK8Xu5BNqw>fy9YGZ{eU;Ca+X@v@ zb%3z6JY@$VghRjE=f;?~1E7a!QWAaoGZ2JiQtLc63FjR%5G>y*mP```#CGQ$v~Em% zGrOWw;>ae1ggi6XZ6Us=v;aon7jfX#6mlg&%LNt5#EBpwx54k2UCCj%3_lbT z^u&o4a>*KRhyG`Y&0Urv1dipV|8Q5y9%7MbbwxkH&AK%(>TI zYA?2|P27<)9JtuXfmQ+?tVGYlDC~{dpFAyFcH9N69?OzNKz%>5J@6A>EGOg zgy^eE7mx$B4hvBv13^NZ!+0B=Q~OV9?;sNN@A^2xw~b0U=a=SCWI3Bq0BVt!4)U^8~T+Hf&Wfm@1O6s89Pwt*FQ*$xXJE_6U98)Q?~}FYcyE@#oYdPRSx+V zGFDC#)e1GE*>4p}1=LwMak^as>a5>lSMvaiP3V8$t^dX3jg?dWlh8qqtxWHdJG+(w zOq?ihed1r@_aP!Eq z#$-g#$7hQvuA#lIWuekT@JYZzE>yueZRPlh?vzgnQ(8o=yYuA!*Y5HU2k(~gAdpFU zS*IXl7wS#tL-hDgAu5Su2@lezno#W5)>me3%-7JQh-|&+iaQtwyj8^9<;>bdMkiZS zbv~cTKwbi}=w74!MDIuH8mA}+J_Br656rfsnsur_B=98_)38Zi$$chqiO18iYf;O^ z&MV!jWozx+WB_D0dCm_v(}1||$5X9XHhNNeP; z`&IN;azCqz;Wbxr`K!8Qz$NP}x~n$A!w6%^c=lI&-hFERsK5s~q*PmR=Hkn>Z- zGo5rzAq928;mqEV{ZeQU<53%v_>T8q=@cY`*#*uSFKpMTi~hI{j;mxAt4B7 zym#;)pD4xqN_U}u zS_6l+%aJ_t=8pH_F5l8J85)@^kVBiL@rf-3U3a-{Z5!Eyo3G_^MfP7z`B+um&qlMn z;GX2?5UaMdnn+!N&B0w-#?FrFUIm;e8zN%#ee&&9k|Z=!wfm=#*)m?C;4yVItD4N% zinZM0s**E&mKUgz)nyKb*DTJO0ZHP>s-T=OT=Po3< z%{KZ&@J*)E)0V~|a>(@c<9$AHH3YY>s8fvhtP-6hUBku2q^hc^BGtsoQ)OaZ^12dU zYoXa-<@BW-N4eB3{f^0^&an*xN%iKe3_u4fZ1pO!+LEE;*fgC{)6IL&rRcm?U3rb; zxJal@RQV-)`;R2Twc@bzH?J6&BC76t54Bp)EsH@6)8(eDKW4y=2bx{9%%;+7MPPn@ z88h)ZsUmPUq^RIR>e(;kK*s=Vd^gz8aT;aH z<0U9N(&vIuZ;^zim&IrW(!GkWUCn*amZ*$$=^1}}9FQiM^Qwkto%>iP$W@;TXuk7L zI7VZHDPsn)tyt=n=tyHYd63sIKLFQYFtks5tN!t}E%V0)g9!YbzxJ5-@vIJi2cN8P ztG0I=eS?4MP^p9GJ7fg-2FI*}tD2A~7^Lbt7H)Oe@zrOZDqVAToO=u5-^AZwgD}_H)$$>dcwBJ9l(nP_da=Z^YZ=HeFhbCB{tm|dGClxv+JCSZ zUv)o<2_MyK0zn;a>Ool_e&0%gU#+yAoEe2Xid?Za&G70pRRz#Vkqkb4Bo=l!t-!q2 zr5O*W>QtG6_Aei<)CPIER|L!5K)Eucjf~`sMOdEA#T=oI>Hc0$5WG?QPU~_!Zd}6b zlVdh!qMdM}7AylZpkb1jy3-$KRs6#J?#2VZw96AX91Xci#NYH~l0!O#!LK`xMi?Ky zpprm>e&DW^G3~9@UJC0rmaBI*JpC425f#V#jtasM74PgvJ(zJ<_KV7Sz=Zb6Rei(0 znviivDt2w)J<0T(8V1oDFl(a@d!9DL+u8FU9F`XFGq(g@v6Ifo#?fc%R#W{O`guu6 zVO#Ph_hG-K)v!#jJ)*Y>2BlWh)96UO2QiKsNMV>B5bHvK(^W-J-d}3%^1VT13y+NI zk1pn_?bAMfxpV!T=SmYEKThsZX^&wHv1X+{u^?@qUgG}o3n0g{Jcb|5un==@iw(Nu zPx1~e^+rJ`9|8aI9oeRMHG&WEfR6?etL*JWk%w%+M^s#&efBo_B)`5uh+@D3x;tF4 zR?V=oW22dv`s=htOxwOhyy{QYL39ZGH;pVAM9y=A=$yIYb)^}nthg-S)j)|8pRRK! z0j$X8G*H!YXn){(M1h&)&(JG+#;os$d!D*4zw8r3ys9bT-b1W{cCB%pOE9#MLou^P z_P(DKtkW$xy^?88*Rg6jTAMCKhGx$)#^qH{XTC-f1hRZcQtpcb`8IBZKGDrkEza6y zwxW|>6}>oQ{}`=#@q$%P`IYcp-&p4265&B)fz7wUSk>4VEfPj}MvR|sRC^@rGo;`z zUPx)>uqs4~D*{oXmu?i+HDidM&Z3L^;gp}26-5~B{58O8K#6A|HwHfQtRmr!!l#F4 zyXZ2&_vjs3|nKjyyho%VvBE26;*9tJT@9?kabtHDR52fs{Io zrHCcfNE)^qeg@&;A6{tY647#{a?h@nQiZi6(=Kdvn_pkVH`?T$$|oK$@ISmL)1|R1 zgGZ_d$m*MR`O0Pm&}LFGzG5U)z{~jinSB3j1I;l>+-_s>8UBgJt_V(W&0d~!s@^!E z9{|Po{%qMNq7tIhDS?cokh5OsMZ%wdbxDPf)7a&~BX<`4{J%PMDO}$?;In!_28n9AzD})sBOuvuN9%;M>qs%fC zs*s5Pc>Y!q78mq62-m4F`wv1zHL&$D#AP+TYFzt;eLu@L*@ZZ`kVrhq-Dmutra#?u zzp(CS$&+1(fkzjw$9`dH`*aA0l>PK{&pIcHcM#Yj3?;9C~<_Nhv*dzO^z8!Z09m%9V0Mub9WOw z{@$~rQE-&g`4W~Ad@;_ab4&A_u6HoQA;d-1gsc+uTQXsqV9`0LHaMAr^ zE0x2KTt)m&5vlkRdh1JrR5~B2jF8fYoIvmx!HlbW#aMu48tJL1(GsBz^aw``F&bcL zKqA5Yq1P9GjDQ}HJMwewHm^j&e^uBUc}^xi^v;UtMyNawY$byny1qbVPm{z`D8;!_ zX*0RDBRh0Z*@m(D?$kh%vYQWuK<3;}8D=wy(JL#`^L9U}fE{)qjh=NR+ytQYp$7dP z{aNj?E2BVh1W5%~rK;7OEwkC}!#%YA4u|BPzvY7Q%8C&)u}t$oxFsYc;`K~C^`$!> z<;yNa!K1h3C%Zk02{Y$f$|*&2_kGY_B2Nv*sv!Ja4TsuB0CBGd^;3Brmap{b=tiv(!UmXlv% zUwL&zy&V5E$+So>qavNXfd`XFEv+fs!4g6%l@@Q$cvnbl-Yzm z<%utsaV)v5BHl%gVFAJHSxJ8X^yDcpYHZA$wh+eDecWBh^?0LW@rEd8Vxp@MD%9w^MxM~m?^mb{LQ(zM%5rxw+IJl_Tt`w$hl zpR((5hkaJm!43xXu!?O|aSMlc03{`B5>nCvto$E0E zh2tzlXWFlqZluVo4+Bbqoy*t5uB^-L&F`=Pw2|q0a#e1|^b}VmA9z}P2^DU%Bsn0@^DEj&1!69c8V)c6!B zj|aOZ)82L#IJ*Z6r5lZLt&#Agay@YQ%7%dgyCc{Bn!O1Iq(ATe>-jE#03y%v(nkVd z!|zbPs0r@mw?^QPsXxP3K*4yNnIoiQLm(e(P4~#pPD)U5ncY3VDN!$F$>k;v8WO(b#B( z0zcgY;Nd0*SPsUfjpwQb8Zw4r0oo*+fA1W*;zZ)mU1D(}2C&y&kCee%IBr3v?}{? zH1n~!n5 zjlkDvO!ZRf2TJv6Cu}(t0Tfia7e;`f=-9lSjJ1S6PmpeXJX`>$tjva$qNT-&lzoHE zz!;zds`0OTV<$KPI-x6G3RP^5yF-T&+?|fV)40bQXG;gmgf^|v!SIO-=815COkQpt z@11}OmZYB6`!!es=)Chg$0nU_)~xIVnC zwZ&?x_9o`I4QjhEzuN)beCU4JmVo+^kM0KKGKv)*D1JPOKOym0K+(O-VuJwtI&CYy zD;+6+#8EV}HGcyO+T>indPlv0&VrPsb+SwEpwiLDcmoj?p@wVB%Hqr5nL_f6z%`|n z{1A=Yv|1@FFP-72#60g+V^~bIwfCU+w^bh1vVacgPY>SX4t7%Gu*;WpI|P|3$um@Y z8i4BOWavN)Ywco6F!3|KMzxx=(p@QQg!lWR9bE?Hiv_TD$suBlZ+iUEK!7_>qI&mH zh2NV;ahGv@Lb`w4%dmCtDdoNUpxl-BP@ph0vzdD7e4Z4n9gPab&_huwWa+JvJ@ z%(ISci$mWuj|#jpzPapKX!XlYtlXW85m|P+ap`p;#{PHJk4j-hvbaGWD_k&zLfz*K z^i?m>zZIoTGQw&k3WFr6fbuGYn2N=;Eq}hbI43Ehf_VK%i*fCQcGr!TQ!p_#KbF3} z%XMR#%nUCDzgNXJ<*qL3rmv}Zu+_R&;VjQu%A+IthPU!qQi zp=YJ0N7vwCgHGL^sNW}JM<)tnAZrFBFRr2o4{;#yucI?QXsa*QgZ!BZs70 zYN0n1IOe1szzSKoitz}HH364n$Nj2C9>YL&v*chFu|O`>(s{j=v1g(|g71OZe+i9Y zo5q)AtJL~V`E(rPQxY5pAKNkjMrA#Ow1ypu@!=16RL#&*lX$H00R&A3%(m^w$`z=8Qaj8A>S>&Q> zIyVOqe$m}@kgs^XEDsFWBoLRL^GfAlQRkr#X#*R^Js+`SHwi?u=d)1)UmR30w@*Ox zplj$BcA7uP-7EO}eXZ=brrW{llc4;?VDk+;XYoQ6q?=Tm($nTcqpTb37Z}%rJSb8i zy1(~(W^k?Sml5RJftTX6w+!5E zi>LEdcah^qehnPSf@Y};i1%VP-MVJZciOYen-!5vcIr{@-Iju^-D~JeV3ju&d^+lW z+T-nAy?o*FK6-Hp7^$Hgw3O3TLI-*xpHTVWlKkcu+SOeB`r?f%TURDGr9`L9%RHaZ zvwN(;xdLaR)cWaiH&QbA@E{MJnVz)CSXXO4)a%lw-_+{L2K3@c8ux&nzkV*wha+3S zgDI|?&KP=++3=5YIv%TPq@B*J4Fia z;A;tBH*=?|Q107r5unvijk^1uWR&G*m^OdHxQ7GJ&PTKkGJkgVEk0;b!Dx=vCO$2h z>_CMLE^W`lzT|;IAi1}pslE+8mPdH3a7@eOXVa=p(t@j2T^@K9`)vENSV~Dx0X%qu zuUX*Nr_kR4yf)qYBcYK}2N|X&X%8q$d8!~)Ty#f)J>i!LC@`sCPdG-iBbUMO^&4wT zE2|$!G~(+~8`9tRj?FC#(-+-bi?IZw!EbN$ZnF95a{`-eT#Og?)qhu%3HbK= zwX^6=*CiKJdfm`d6x<9C)>bN8w#s@jmr~00I2bG>k*=2`_u(f#`z7pSexLV~Hl3t)gYJ2(CosBm6V(H*{0#tNjhFbCYN3j6fSEz>` zn_0wdO?(#jsBMiP&rsG zgoMCDKKjRW5L4>W8b7QZ|vG1Ncs&!joaMh}pO64S_w)F8K?-kGVho(O;3xRnVzq!s>k^>bY zxU}@uOV95Ta701*g7>aLsZ~H&S=WaXzJbKK2GM$Fo(({!`s6Ou1>%H>{!Xv zY!c}I?s;?fkber*yA=ODyLV)eCU}%T)f}!}Xj>Yg;e1raC(9R}ZtI=p4U!aii}|yAZM9Dl}hrA~@01uwO;WGaz#=;*W^sm#>c5S0RHpDhI7rwdVf8 z{rJE$4Cw^VXLA>0dJ>9kV$xO|evne)0y`->UsU;hZ93Cq=wCd(P$PDj;cwd68x)Ww zLQ0mdFz#O7r)=xS^YZdV2RMsfw8x;r{YB1ZeBBAY?<4Q*zbEEbv5~%Jl|Jg zfL}hcYAUJNbzQrcH6S*j@YAttKeSkVF5n4e??LI4#mSrJzb$<)SqjD>cnxm6V0M>n z+S93mu*>l+0}rYnvRPi-e7sW))#GEut@pNKA78UuCQ?_GuezHvE?)Y`EK6LK?%Mq8 zsTEb56P89)CyZ%k9d~@x2pAWkKDoRhDz|l?-ODJp-gk`3$&Ed;f9`5$?4TW zbnALBZ`wOK&Qx@3qe!z)H#1_)i|vd;_^u*~lXvedZYGw1@Jf~Dh+}CMr=Ozd1{RkQep=gI40xEcet4PhbV%7a3`&-)|2uV zc6k=oM%uvx7$VH>}7Xf>}rb&+H7O*&w_Y#ImtH(zZj9LeN7cJwO4I@+G^_0S~K zT3d<{XBw{#mn&JtXMvZ81b+N%5T|@KDbbFi=Coe6K*gB13n)Lo)0aM$f`&4d-FR#l zf3Hee?utGq#~D{YX3JJX^NSK$!_DP?0Qw6J2Sh~BeU*6Pt6Dvsuux-2<`uk@`Z11s1vrn;i6 zyc+*<;%P>oIqVBsM{l(>-@8tV5M1AlXY`woG^>o4o}en&B0r+8>XyaPq&n7A#Yiz+ zamW>&CqrYY6z7cfxHyRf*NL^gCEp9Hz22hSv$Tj?LZxhOENPms``3{92^3N$gHuZi zw%RQvr_OIluj9@@+CB>mG#a?fcVy9bW;%r{-$~k?oV)8(0PeArx2k2`r+nzVB?oO0 zIAKBt*z8K;CTCni?^w+X7Ba!7o}Z24K0q#Mds`$aV1>jzAe&uo@L37${xrdd-&aN@v=8HB^Nl&8zi`)ar3K zGB!eTGP3*2a!NyP(`5gt0Oph-RsHs9_i!SVn76HEesW?FCrqPL*-9NS5 z)#T>!(BpZZi?XEmG|f_XqZo(yIIdRPldOfx!fR?7-Xg=#MC{@Vto?8Qc$~-ycdLNi zOa*E*s#EE~XX9S`3dBOE??{>oJ>pO8)$PqX7O1vXw^syfur1qTdJ9M9DxmuVKg6y1 zVQi&c^hlX;|B!CyYvI&J%VHI3q074dT;-=)mV#6#m0nWCUUr$e>I50IHVMbL3A5$h z=b~Ba2Acdw62(VllnJ*p?Wk50s#Egu;5J3-R8PHwPtYeZfF(Y1`BDA2{vzkoX*rN( zCXNkGnJ_y~br6eG+nl5H36^GuU+r|0H`~sqlQrRS9*4?a&zM&psZi~7`)RXHJJS{L zy@QOUo)s-%12g%{D;)T5hxb0CCG`mBaj-6ZYnEDmp!a)YrcX$mxs`Dus1tL{+%7O2IV^>D1s-EWw!5u48Oo-7j>o1lsMYb1O2i;AOwLTJVYMV$_}Td)Y07As#O5bX z`;oLZCVmcM!?V6r-9qL|9C)SIr+I{3GUqPYGF51yXssMTb4C;Kq~QzG^;>Awosf4{AmHX!gjvWWIY1lk`LR}C z3H6)|04LK_r|p_j_(Y^j?f^`l+u#nG5}}a;*qu{6wr`s{u$Zq-Z~x`kape3aG$P#k z7LcywQ}e{;HjvQ5D%vAMND1rfV>4aRP_-UV?3PD}Whzz96x0wgC5eH?#yW5k*)D?s+;Ox2yp1wxu^Ircm=f6iocACE8BdBmF#3D6;tooPCM~QqlAQIxEg4Rw%W>v z_Jq^+F2Dqz`5qnU&a($gNz)950ReRV_Brm#9LolLj zBpJ(X32Y6T{fhC^`uB8R=h3hu^R#@mT` zrtm}OM%)yM1?~og;Sf2qM6Cz@vQD25iw&GLZ9H#~pgc!G7q;5arCn|iN)KM0+ z_Fc}5ASGOz>wEN-R9k240$@-$RFc)yb?)J*D-=}^R5FxUzlFYkSJ^$*Nlus&Zp%Ku z8D$3sZVKKTE#jwytFpTW%`Ov%Kvgpjy1l{z2-FpXlT^xZ5Fq${A}hmWq>UmS<+F`X*2`z`o=j*j<6ZQ!TSf7fSZ%zFGpFNPZek9hF! zr-Y9ZVq<{)2zps0cX$m+b4r|mHrg`edvvJI!-(1wWdU0% zzJW(hU!m*hrZ`XmN&Oo`;FUC^xuD|{GDn}Qrt-?=OkcO$gkm<1A+(lmZSxLEouQh| zk!@n4M!#6Z$n?-ybw%HQ_93b-X7SAKGH_JIxG6{jQ^KgBRf+U98PP7Tfc4C9NfxWX zlbjA3m^Rk!CCX&&rFF`)j=lrc*eXLw2dx0^m9;_Lf!(j&a@i)Oko7$Qre?xG$)uz( z&HNY+Yhc;J*>4%}JEz{_6;bp*R7s;!$wppz9sVj=-TMpC6v#oE`J^eA*$R?JvkY0b zid8F@rGW`;j=mtDbH_b}f2FC*M!veuq9;uIp(DXokOJ$)u7X`x<;7B|V!bL}(SKl5 ztPm&HcxUbgPP^N&q$*4sh4oLeLfbcou~~fow}y`TKfP8^-IN;do(*H2d#+HpSUT}q zcr`X|szDE%ld}hYgOi4@f%|{Igo)Dd|KFGN1rBYM(DVGW1s&k0+90SB>}i>TS?K=& D5uXO2 literal 0 HcmV?d00001 diff --git a/3-NeuralNetworks/04-OwnFramework/images/overfit.png b/3-NeuralNetworks/04-OwnFramework/images/overfit.png new file mode 100644 index 0000000000000000000000000000000000000000..3f01344dc0bc6b4eb95315077ba9d3e042e1e2e7 GIT binary patch literal 18414 zcmbSzhg%a*wDyJ~C`eHuN-tuE1w}vv1*96O7CMV42uklQ5J3c`qY`>YItogMfFL!} zd+#N5LujGh#oxW(AMp7+knHTtnKS3S=RId;S3YQ{DV;vWb_xK1(<;jHS^z+c0{}{p zmJ)86tndhjKd4^Gs>%XDK`7n+b87e+;h?1?2jsVL5a5fGw#vE=0C1u2_y=*mvtcTQN z8YfXDEJojr+SVuw;@*RX+b;!;^Ww&7eD0iv%B|lB8n>#8G=MVS2X1;H{(bxT|Gu4L zoS#JXyE&Zc+dntGe~;-#v05%oZrH}2jPTlfumSwOnsJ`J>EF<|MfP6Yv~zXq^*hvk z2j^bwQ@jhvv)U$xu2ZOsBp73KLMAtDjc*%Usf&n(y=~g8m#aB-rGO8W2W?%?4H55{(07mjVfNxmYH^rw`eU)PR6mR}Dx8tqB2})bLdg()gNl7-Ex} zRu_77+(!EhdIu!o069R$9`V1wXbDF4Z^?XAn1IdMPC0l&rt4)r|6~cH5t|BVL=RF1 zjeuwU@BsdVFK7W}A!D|G3*n&xE+XMyLeeTi8n9@dGJqElPM#*sHXloZXIiBDza`MZ zdcab2{wcyl0YuOp{|an>Jv4G3b)flhQ`Wt&BzTkgxH0ADo!;vFhq}K1uK1o@Nw0w{fX3Hv(^@nR_i$S;qzw@Av#sQ5j;K zn{=#(?c){oa(qEeRLwtYmUR70lwPxUwDe3n&WM- zMjONggc&@_C?oS?zPVt#xygs^xmEIl$EM-$|^hSbYnJZe)`y>;EbJ(On>Heim@DDLZyW>wPaX0mR+_=B?im=W*9`9jJpEqxK zS*NCIzfiC}$f>4sfB*NaPsPFa!DYb*Nn{gpAye6RTv9bvHrR*o=3Ny74YrsvaZ$J% z4-4s2J)Mi0pWMum$5M$W5l8c@fXd!%cOw`Y%I^x2x} zwt16tSufI>tgT^atz>QpA(LRs4_G-T6!ajib3nJ+a87S#3Z|r%wP2%*$dacl#AM7- z3$$dKt=(j$676ZCe zn|^nytnlX!qYGZ{^HC*zZp0lMG0%6#E7*%*Ip*)_XFx48`}eMQFNwCsFmp#POkzuhODi!xU55^D_FUO9_t>P^9jz}@sHX`UYG zUvfZXij$ePz2>{5I;^WpFD*jnT@~Jlb@p$V!8R*j!o9^ykL+Ew(QhwA?PP=Jl1_S1 zi499@!Lf>UI(65x?9J=(_-DsgPqU|S-uuq3{i`tIl5n{y44#-Yr_Fz{o~y~ZyBR-J z0fRxtK2M#Y>L_B=;4+o5UY-l(MI`mZLdFrcgVm8pZu2M56;&2*=gH-K zGR5Px>EVV><9TItvgn<|Je%fvr$xFT?hvt%9^|QKjYlRw$guAQ+SrvI6x*fYG~!+( zzOA@!W|qR4DPVQ#Pku1}rXPi%@!G?C`r%T7 z|G+ML-mlI-;H^Rh23R>5H_r{HqOdWs4GJ?$*tlHrG;65PvYNwbo8 z$~3(jqR{OSy~uT%suv0H_ab|-e}zAoO*yTMVcx$D4W=nKdnu989_YTd|*z^%e@4_cBRF6d*xJ)%P$ShvH& zL~)+ImpZt-jig{7{6R(jG}h09@O( zrjl2J-#q>JCUk+$_J~eB!*0f+14+J_4~>_;H*D|catluZYc4yoDUn;vBLZLDzq{wtMkq(Uj$Y*Z3y-OsO{dXY;H7JNZ z^-4bO#b?Gj9r3w{h-Z?+;D991^-O`_=;=6rmoG4-X=M03=n6B?6bZIh`bZ390?KEG z^ZbEkma}&Gu8-mHuJN;#oo;r(%zRvPxL16w{?RU+^-l+qy%fP)VFt_xdP+q_)<0s~ zU2^gx>x()RVb6Ywj~PxcX@GdjQg}m`9vn&}wuert&lB-IMz*||&(X`+(?aIIy*F-M z>~HZsPaPf~o8JO%@X7bZ2EtJ65DOWzljUDP$^TUNsPph#eFbh=%upoIGW|3f!Nqg8 zh*&6f^|udBUfyy3YtEMD1^KyFk`P*U%HSc$rj!KQF^^u}Uo?c~{1>aMITOBu%ApT# z10`8UaW>x7M=LJ95?byE^oNfso*TTXdkYeY)zPpul|%4=LD8})l4QCB@GYQT-r(b~ z$s1Bq5D-Zx81iCBnXrmaMJ||*vS>*5 zp}Lcx-vppPgh|ypbT^i-b{TOAJ8dwdW!-zu+WOHT$Y(AyC&UWH`20;Slj7-@PJyi` zxAu<>KD6cwo)=$9U6!g`8XhfyL%i}P?Po{^nlfb_efD|!PQg&Dx>>o>w=QHa62A4Y z4om_&mr$aRMrRgMe1(6MpBK|Z+sL?tr<9^s&OaZm*rz>cAm}dfem!OZj-51x?gzLL z>C?*jO##aqcFv^_bHbiqKOVKvtTsCR$@DCh-xf0$A65T&WLy{aZCFJ~IF85q(%$SA z(cNA`Jb38fWDRAr0ZP~y+Cya75NNd-hKJ$2ZwEC)Q^6q_0qp2`r#5{VbQ=zk^twW4-0{BVi%_qd5NZn2DKMv<5ATkZ&Cn4Dt#)Ik+rQ}!hhmGy(>Tzw z7h#s#S!EO;%R#Gr1NeLi4hAQdxt&TfzP8hX>{B}S39dv96`sNb*~Q;`FDqwOt9_(R z`1ZaW#KasQZ7A@x153p};YVX{-KQe$cIRtq_W|w#9b%+4o7QjOh1-$+(XwO3GIbKD zvWUN)4L-j;7u+hfW*srIjyfWzkbUmXq~D?_sSrn8imbxmIRfq4+s9%{zm2;lf$}xGh~8n5N6+rY`>rj9mC*6V{46mDDCGYiPTZ3^@c8=R!(lKOQHPRMj zWG)>}x|UesT+%Ilb6p7%K99a?{Wy8uKn4yFH#x}2Qzx=Z{ZGxij%io;2J(LD5 z2B%6`4;JtsqY2f|y4);644jvKG0|%KeU1&O8sWGfx2*4W%<^-So+yWG(-lnRdD~r^WMdS9swh8fTQJmEm6my8Rc9)5ZDEqTB{%VD#H(G zka=a+A9G1rXylT-yDtxR(hzinv1a<*3>iU3N)McbO`P1VyJb&IKs5V>+DM%AiU2>u zkUeo|WO@w3Rq_?}?2_dvn3kS@TDcvO`<;f!mP3?qX}o|^O7F7Dw4pReP5}(5{_9U*(do^^dL1F zf!@V}`n!Jc`5*cM_Ycuch5!9tjZ52dO2mUU?g0oj;$h;2dEQ3=%MX$M^{eT}0y2 zzzWsP{%j{v^=WpOtJUY>O{S~hphel{Ie2X>=xA&&6ezE!#BysIlmkxp+IYEbmOKbO zaxYD?TdM$?{k|=rShRDtk9tg_s1|;Pnt3t3lNowqIQ^r56i>M;W?-8Z7>HQszj&O* z?pYlqx8=NO2Mxcaci+2-LqTUU7OtSG?}Ww2;L}UWOW`u*31s%vn5Cb3W+(bDY?T<{ zt!HtnU#mHEEZIGDSHHj+XZ<_w-Oyj54D`!AA+$;y6Cy|Pr91xZvSR^~-yD`= z0M)WiM4;MNAD2nV1y}+q1(u3UN>6gdJ@8_b&p{3Gaphy z=X))RhWvqPYa$t&xUtZ9?HuHsh3@X#SD;P$dIme19TDY)vzm_{oh7EhdD{p%JS=bb)ETj`<@$zXmPhA@MjY}~uL6BRS>q5Ok-vLtVqs1r?DhD(&e zG;N)#P$KnoCsx;-gXL;*=nfrP_Ce?HzL%L9vTnwPeDlmus(AHMhvcQ=opgq-tj!7% zHH&Enmdjnq%vof4CIw9G!e*OzMqM7bHwBWY-_})TY@Ym-7i~!Yxtu(8CrAZa{HhtT zlA5#W)VJ1!eR6VdvBl^;v~^sIu2-9HCoIEh-VEFErJGx@jVwuy5Pl9B-e(T8x(!Fu zx*)j=XOlj7uMoGYOHR2h%oT%nn#WTnw4}uhZg#^l_N~j>YB)Z4Hfn;1W%vI3a0ksh zLHSzLa;WO$fxJw<-8*Q4F^6e5uM2%9MBV#rHpri99;+zr6n0>4TneX) zn-oLwms4iIB?yP)6&7MR`tD-0WU(Xz_}vkIHx+ry?Fc#g>%3)CGSy75z`>GPhWIG- z#G_L57F)03o}1V@Wf86kCr@fEKGV?OiH(aESF(PkCwpRS98S)3y}v)b{j#u0Zv6}H z`{7>Jms{M3hKImu{kO`-j;MDss9R8(iOh50s6O3+jS_xStnv%;N%o<~;F1K%imhG* zkPJ~Q1TNFHWIUh$4CjdjRN$YOLz6>%9hr|dN!zHSbY1K%j_0LLGB~~)$eGH6X;gJ~ zDdjbO&dX!Z%i-tYmH%KRteTdBR=K)7!S2;n9{abrpUU0PjUc#d)Qzt!Vw3@x{+sp( z1x)d;<|5?|<)99>=}{GmH{>2%z)D!4JOQL>m_o_yFSP_=H&U7{l*@67| z{w^ghs7lpavlvOWev7m$HzOCz`yS`H0#a_cP$Nb@Bp3!5nBQ(#P&+WQJF$A-9XQz2 z#aGudFVN|ozDb%@^bSuvLuGs2Y0_Mi$s6v9c;I5Qg?xXRcA^idG)db@)Smf>f~ca~ zqs_X9O91WA(vZ zltXX-B*8)JYock`Zhln;CZ_VnKa>1qAI2cbOv0nE0m-rS? zm!mIKp&(lVsp3=RBSSddQINYjHKc8i>u_H$Lx~j9@#+lI)vA?0(NSnf_%udYlfARo4yG`jOfw+tQuPO)P*(`q`fd&1?bG^-hI4~g+kwrG)>nT*)S1ne-Znt;?^&Up>97jlP; zP`Zq(M|~=hrR%UYTeeoAygEoWZ}jPO3RF{)o$%hq-qKI!mzi1QiEU?4^C#mC?;Ox} zfj%{#_Dx4i>oz-E!n-c>*cYA6Qsi#zTk$M=-#Sy9c`QsAu8N0# zH6k}GQQY=c$j2;$ABVr;#a1mH-_#iP5RioH9#IlkS-YusPsrzsT?u7fo9@VJ^f!$t4}0hOq>J71y1{2*xZhi()oUKnYSL8z+QM(Muo#Q(cpNd;=cjw0fFz$` zKj>94Vm0zUjy2C8_k-MZtoz?8Nk!>z^Ei>~FT>pOdnV2uWQrQl6Ysl;tV+e>O?WRA%zZczM;A z-f8U=g{Fu#X<5r!X6BP?IR;E>h3K49n-{uUX@|kwKm}7q5b{f#Xw+! zq0G#~iQzZg&f;kQ4C=S$T+gH{YFzfO^7rTX1t>$Fu4|2x7UVO*I2b+)D7{1@vi<$w z97+!OL}}CvCcqA?XiD^&b*Vzn19?cY{oy2S63p*P$M*|_jUH)h!2n4H7w$-@LMHCB zEFa(A*fhPx7F;~Rc=#QbaBbz%D79Z zcXee_XC3X!Jdk^3b^;vU@fLZpRJ-Cx)1p+ifYKGY<{%WyfDO46NPKOfG>Hk?OfSa% zvUEMX*`W&FqCt0CI<}0VFye5!4PjX-WwfQ%pZf% zs6brY+3#ox>~ zyv8Z7kglTk0x95^{5bw!t`3J1R%c5|<`YSeCgx`qcCXN(ug>>cRYbm(X7q>_Iv0XT zbohfy`&;oOBV^H`eq)ySgkvCRs#?4TD<8!11RctwMbYK@H?%MF!FvC19lX1RS&cb3C#QwEyy4s9gMD%0YsAiPKQ~Q1 zuCYDE(l{!_Z!S}W3JX#8uQ_gYCO&!A#^UpQB-{u0^KDMX<91x`QnLE6TB~==`7H{m zT=42k=zYemu5qh^)G_0VTS=-{|CD82Nk37_SScoHRQ~K<*AF_WZ1Chu=%WoyR|1=H z$|0#6`z-2$5%wp}AP&3wY=SH14NTs^>-8Bx-UFq4pZU z*FRfqX1gIggeX5z3PSH5L1t? z{lPQMj1zaXyJJ=O@Dm8yGlXqSL-aiGom0KBX|tO&E<9#C>v6O40S5Qd`@>KiC;Zv| zU2Xb8*#>-aQN}m&@?!j`z$tWdS&idLNP;4C^VXw^ZbyqD2s&l`?+&@iu>M z0tQ$3NdnikXR!?r_ZLu#@H4`S=y-mBG%HzBwp&>6Yw_UaNV)#pu9&vh-;QeVp|=Fq z=*+l2Y>eB}EBdmjos@-tq*~%Gm*{!PLWNaYXW5ki`TUZTE(W*z*yBe7aOd1>^1eVF zSSf?e<+jg8a1DE8R+$lY#?qQQ87y;{Rb=E|>BQdHvL_6xN}Yx$e)_KT+v)x}0zWCe z=d5+^zTBojAbs}_I++>rp9j|MR*%~5$w7tRB&>|80&y>ubK9gV47Mo3{9kp4=Ty67 zUAW=AmIBsu+39}@?ZDDsdekl?!;E$lBM*nVAjyJ}uqyk3YM*>FI@&!qUYe~Fgqmqmv5U7|?>1xn-M zL#Pk{Z1}#kB2-Y>C~O~ELkEWpc*n+L&+@|Quc>avvL$m8sKD)T_gE1@iZh`Arpc*} z?`;1OsQAMBCON``^R@LlGr2I~?IZyB?YadSsLy$=egE|^A&6607Ha7jncydx77@9= zj)trW0A%8L3QL2EwU!I2p8MRMEsg)ty?|#U*RB4RH$lKAs#V-zrP1%D zhdu4nGcaO7GF4@YZriFwo(0gmdd`*i^c=rB({^CH)XT$(fxRarsJo@(R{RjbFvQwv zV@F$_zuBfmUoE~OE#~CJNw(h(yqSmsKV%?atOQ~6QcGnEZ$%Lir1YIs>(qT8+z9)9 zp>*WXF$fbQicx{;stk&eS4ipSCkxV7k)&DNg5B!iDCd9#1L=q zQhzvY2!nJaLBUYSL**Z0p!SdA-NDA@65KEpa#oSsuO4<_W5+{K1V0;o*k@y_ zVA{QTH<~b}0A)+9LK5l)t($9f!f@nZa3hr|mX)2|x+(={$yyOPIW9^*K&AWLOR!jY zrO{IStTfj-8J<9kQa0PE@Wgj%Jx?P744_MMDt=1*v&d}F@9c7ZIP6~v;2A$Q_}W5P z*26PFjuIN^teL8;U&^@n{UX12xhj37iG)B4177I)A6xq?_vKG8uHOvONJGn!q(=T750?g63L*Fj> zzlB6y5cd~BO-iy>LQZqAb%je7&hwnxT(>RZ!z)Bm26!SHyt-X0y#Q(=&{;JL)gJw7 zlm>or&m{-xDA^F_HtS_J)A3_5DUjDmL4)M4+bo0;b=W3j2kVT1l`uEU1XVGkFA%>C zlV_2n^r?#ZvQXScvo4_zHL#hQBGkgkSOAp+#ry5bvzQJ{`MMU`K_)Sw;Yj zXYR}EGDNZ~yV~IACfX`oT9bn4q~+Z?TU0*$#|@`sPBVC$y#@_RG%d>H;!_>njk+4kGe zWo@+++YfWMedBE)e95{&%Zet|)%y}zi>`mPVgb)x{;*vL2!9j!C{qXZkV*YJKUr&= zlQam!zOonzj60F1hFKs*@6iem5;ZLT2A%Vj z!?U;PI1KbN{_e-K)WEOZ!Ha+Ko|qnXG0y0gXp+#cSFq&~fQQr@rF@-a=Jf8-Bw6U% zt@0byqTfL2j5I{~)6D$gl@!_~Fi34P(~=^kHq;+M65|VFxZf zDcKQH3;RNqh{eU!zSvDSMXPkLiWr6zs9 zi<#<1H+S2XeG&=YY7GgO_vmLUw5!pCiuc9xm&T3^e8s)X3FaSr4V&~m{buCr&a&=> zKq~E2N_QX3(|oZI^C4;;d#e^5JIYs|N2J+0kOF~@8!DoiAc(E{ZU;};!fgDSRmNv# z4j`vo?DJBj`I!M?yYTYuLKsaiF8=;cChxMYF~z4`P6GMNR+d-G=STvEVj)+KaixDY z$E+lR9O3B{#AbBp$(dmN2y%CKMs#4~M7~6?CW1>{o;i|dYWVwGtAjTqUrDgqo9+vK4M%BrXFk3w*sf8fFbAV-K-|2(I{=%QgBP{jX%6S_qu*pPbl`f#3{YbQ*n-z0c$N?HW0S6hgiu z6%4avZ&MNWkL42UZ0cBZL}$0^=A4Q(1Dq`2Guxq?1$j%Ss}ic^NeUm@@RJrWc9;J? zC?MZ_=qUeQE${YOba-v+sMFY{-x?iUlDvQXH{HGCLJw=k!faap*_2Op-EEr!B115l zT&>I0+R)igKkFupx)ad<>|&0Kz3WgBj)O*2%NtHfErO>`F)J-6NQlJxy9LtCe3Cxv zUlxDhJTfu8TFz~h13n*%j~ci~Ap_Z;O^;l-@{<0MWzY2Dx~Qy7@N>Q8eA%Ys$?R$i<{MLkW(UuX5uO2h#SqYrif2Z`lK6M%Rm2hGk{?=-y!Q5 zJ-@?F1EHSCZmeb=G!vn3+<>TAm~Xp==+lL zWzUsrTZ?$>TANA_?OcTQO2%I;ci9buUI+5aBI@%Eke==O|>8nsr4cdnP@dv zh{T*2xK2PZN$c`!Zcj{v3v2sl2wHRB>Bf4&H{s(epGyl0Tpw=Tf9tDM8}zj#G>(8T zsX1{S#?9E7b3gj8d>0AJ9e=bI1G|4K=F{(ES`8wgE_;xkBolu%0T%h?F0H+d?l$nd zbueDANKt(nDh9z&4ylL++ijtLt191>DJy-*z2PQ>XmhuS{Xy$y=%8R+%WTk8R+1uUWp^i?7oIIYE8qt)rdPGc13C;`zXBLyhM zz2}=J(f@7miBky0wfrAB;K8J+6o2DiVCX?QxeL!dxUdv?|NMYCrqgUYP#1kC(kP@m z-EA8V_lFk`;=^@rf14Zmi3o~>V48nkfIJ}zW8%v+PKSv+URPJ=-F`&Qm@EHXW^p#? zQU&yqZNhr*;h%ps0f5e8{g48`!r<*iz}I>0Hr~JO;c}LeUG+v*d_6Hx1$_Z7UB#Zj z^wpx0V{o2q=(W2ydw<*e;OS!?KAO-+#)bQZB^e;9eHAP{-YsV-YP4!6a~GH!#6N1{e_i2`r%?WY_%kOF8ESk@#~@l^wE)$jc9jBd`pNt z#mpFJRroBX=UL5z31A&<`b7tsg;&}u_jDxhcc*%a=s=%^^BI&+QUq2tG0-U`R`2e* z-Szz0zQM142SasuD!{<%HSd=wodGY!97%&-Z}K0U@J}^5wr;Rx2CpMA^wjSMldzZV z)KwZD`@vT~9_741Z?}l6vd%4Zc87`lR8uvR8K1H+B-Rs`+5CBvCZj%iBEsZi(Ys5V zuapJuaxP(<^7#ku{k_6_UN7UliRjvE=Gd4S)TGw{zK8*_Lf6|LfdF=I8fd3)MT-%p z#7yJJ4(GRtel*YSrI(WeFYRiTac|v>3~l=y0)&B3h*^P~-zArkci+08&4A@VuH4Rw zhhsud>r;LAo`Y3yk0jbO0bnL@-%818we*8gL!!VBdE5eTw^+H_7f%%_dJl{F`@Op7 zA1VP&SOj4>B&Nsi)2%IuQtsUp+SEK3y=bTYgYVel$$&0qfH?cB`G$-AT!A20Z@-ZD z3jSQEX!zAzYY*4gs%w57L}FgN!ra}krDQ5Qp*2{PYWBLf>~=LN|NYN(BH#1Tr#{7F z7N*46x;n#osgdV!j_8eO5<4ksF|ArHRntG0{4Tj7Tf=Ou_Yx%aop7)RCx!B|tdFXN zE)Ln{W4^z;C2?BgtkaoSouXaB!B0~_5kJ6(RlPJs?rP6kRJy;@$z1<29+9o=czpjT zWn=A=7rav#jsIgJNb%r@toR+Nzgp)qpK)!o)Uc;z|EXxgUbUO}xJxZ_&O4^5=Md-}Q@O96aINyXS(H`bJu#er4^`iR^fPBW z;M0O1(8*T0o@TnNUobHS*E&UVGOsjWUg0jwa{rqRBJ|}Ro1^dw9?sq%!zSMTo`uA%OTE`SMiS^2VmnmS- z4lm^2mVeqd6dI+M@mUK$V<<~BB`xQJY3vA@G|#Nh?IsSrUQdLJ)UHOlX%{PF`Jps= z4+JoSchFiHt%+veL#3FxUYGKadsppdKNFa2O9gbzqSEiBYrIs;JP}lBy@-d z08?W9J5m2Uy)D;kx^8J~sXPZne4^*CFa`i-E!HR3Rm)~x?%3CN7>-_1u6D)a&S7tN zpGQno?Q~M*k6Tw;oOves{L0!5<2jLsk%zzDm1rz`EL`8~x?w)^>jY+dHgI|wbzy*M z|78KOCNpS2JdbskfV_XAH1u`%3TDTN9JNGQL*0<`boo7!RQ?bdY+rVN1I42tC?_xksJbG3uo z;s)W&ml;sp-u6`v1V(W;;=P3e=Wys;H<)|()Vn8+ay~E_Kr)?5=?2w&W#Gs7z>^lNN%Q9W zw*+hu8j-A_GUhoWOnc7N8DY#NraD8OmJFeB=h6N_hRkCbXqyR07a}!W&S!oBbvw)$ zZe@5iXL9;wTF$0Jam`rGs)<2r*m(jcJwuu^p_akJs}UeCkHo>$Biy;WQWb#kn;AVl zoaMF~ZoJ3)O>o80?EJdT;QgnZgEu{BYf8SeV7=^gq~NoD;MU}oJ^QweX9Yi9I;8F^ z+CDnCp{#uosx{%Lx}L|j&{6J@C0(Wap^izJkw?xTffq<;1O6)UYY2}rKDsOwx7 z{uJ?TEi;h$(V$U%w{YB9yaSCpxqCZr?tAs;i;&y{OLBd>r;2hFfXQ!qiVe*lH9o+g zluG7%_PY3W3-Uxne}9GIO1LS2g%1X?>bnu#LqnZj>|247tX^Zq)t|2y-+q7njo)Cw z=+1U5{qp8q>e+TGpb`(gLVlApaB9X*r1ph6TyFmOwgPdVwYCckSryJ0(Yg^^977NA z+_mqG+d#*9mc2hJJp5^8UYap|j`o9n52)}?cEVgG@>Vkv*kQJUpPB~RIJF$TnMKjN zO-= zx~lSTdqT_Dz03%fW~>G?!An;WjxzVUPOEtX?R%1&4@wchxh8B|2QzAf4X*?)V*k>q z;MSkOz-Uf3Ye)?q7uDo8!p#{lP=dK@^$k9{O zKUc1jTKBx&kL&`EV=QhPE&G!Fiyi1zAMl3LlJv+wu#|+)m96o_x?9Zm8j4f{HKDuR z^7*Y@<8uRo^vd^E#Y(6{qZ8B%Vzjg@D<=7b4sND;x~&dcSs%{MnxI+ z8=ZHu@%^qk>Em3_p5d7n1uPp#D7Gk16^3@{3hvR~SxZ4i0ALb?KDOJM-tHDXIpgYs z6+pGjpNuMKdFOsIUWEVhi<^^{#M!-CO>K#L3NF3~ponN1FklxFUPMAg?(fMzsFEki z2dqo9UW6KIJM-Q5$0fHE_*-E5P{2K5W;FSMms1~Mh~)H1NL^=+^omx&8Q?$;Cw9tW z->~PiqQI3faOETyA<#1w!X;>beW+dAa&ls-U-lu{G@qyA0>q$xhnnL-=`qK!@cP#Fs_h;wfVW z2aKk=e^7LRKd2EKe|`*qug*epz8K=70AA0cm=9PJe+%*C6?XMxL|(P|O;?O{<)Tp` zV0V3LCTq{q1G3FX-@NH*wd~w3(W*Uq6kDn%TZFO?0B?wzSRAH1sw=E+!rs z4nHQq4D$C+aR$L2m=5fm&9{BGB5;i=9SVb89Xc*FOQ*fM{J~iT^&4f`?i1G<3N2s)iBuv4tjjE&p(4Ok?wM>e zbSdlR(oD7l6`#T*@~rBvi8HMYKx=@z=-4;ne^~gZFgN#mrF>A4qzuN>O@y|Gj?G0G z?x2r*a^~7TP%D}n_I20A2QzqkDcJQaf`?*P?kl23v1zVB+()M9P5&)>0R$##mJZzw zK+F+dZg*(Ujapm4Zk@PeFUaY8^Jo(dxal}xPL0bNmwSp-7*%mJWl~@+d))~KNvf1m zG7WPL{WGYQ!4+d!e!w-vfKG{*{1w5a^-#IG@*)&bVEgVbHL%}l2E~nt5Mocb?&M7B zj#cwPS2z$d^2GWM@zd)}Cu7+&@Yn*|pD#xEfOjCY0QcjgrOIW9GLYqZ44w-CcuwqU zk*)K9u!jV1ryBVqFS`qLO8Yk{G4mDB*-|fX-|%?M`y&$m=W^ieV12_0;N}dz3w|-{ zDO93lGbp$! zaTTKX%z(b88-s9Cq$z-2Lt9p}YOj7ro?fzR$PDI&|=${mLE} z`~I#Tu0jn5i@5w4kNhZiYTQ3#Pn&}RFs(|w)Sp}%oW8A4Q`KanMJ7^*f%omK*;Zd~ zuS)Jx%>2j+1KU*V3|GbsFGCp5*SWZzG0~C!&EH6+f`FmV0i z*i*!VwQ9?9wj9Ei1hn)lEpAH#m+unnGVCTj-NZ|G(jE98x}IdK?Ln zAjY#yaEMHv94;%+*!mR9_ig7g$+Hg|RxZg}blX%(I7@2}G>u}cMc=Q|JS4!X0c9%H+ zn++GfKb;4Dze3K*({F*7Vqj;<#%d(i+XCT$T@JNROVW$&cg0SlzQyVhz+-jl`;e{% z{ryt^DQA(p|CiPE^j-|~pSbc5KR2l(Gu>SEj!l^1p=^VGOQXudsE$9DUFd*}%yM;= zbfx5wDR1ueSfabp)qD3pNx#|*bzVQV`p@=0*uGyTy$t`2`tE>yW^|9`_zJe&vdLj- ze8u+>4Z`|QPekDys$*uc$^prGcev{AS569mxA<)iXz15W_wCK&SN}vQE+!x8!m^@aD=thS}>O)B*Xqk!n zi!j=zopYQe6PHFV(7&FQx95J&v*HK1II{I+WIH=ITeyJ~J$Kh+a4b$NSFb}cBNl;UNMwgl~3ad$T;Xg*~o zzvgziBRu8)-rbXdVPiRe3(zZ$@%{a6BF@0_eczw|Q>WeAn5nViwR73~Q{oEqzh8c0 zHRb=|$6L04X6^b_RdZ$fo*kg{+hE~*1mwIYngOA<;0U$A%8%5wpcJ-_nYkL-8 zjSDgdMx0&sFtQ2Qef-2aqxx;`d}T>+aR0o1FPNnV-MEwh& zZV4Nlmg#r@Z~4HSy{D#e+iP~FW1p2zye^Q5?hE+I^j5B(L*ggXqocwp8~$j;H-z^A zXW?0R4m|Fb^L>2u6*q@OgZt`*g?n!7=bx6yD0h1L^BLj)@*64`XUx<;Z?yi;bB2#h z2HNpwv^TJ5$2EkU(Vn3Hh!r>x1|mJCB9X3pNISvCVZncfuwMt(-a5-#3DW85>gTe~ HDWM4f&1VJG literal 0 HcmV?d00001