AI-For-Beginners/3-NeuralNetworks/05-Frameworks/IntroPyTorch.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "celltoolbar": "Slideshow",
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3",
      "language": "python"
    },
    "language_info": {
      "mimetype": "text/x-python",
      "nbconvert_exporter": "python",
      "name": "python",
      "file_extension": ".py",
      "version": "3.7.4-final",
      "pygments_lexer": "ipython3",
      "codemirror_mode": {
        "version": 3,
        "name": "ipython"
      }
    },
    "livereveal": {
      "start_slideshow_at": "selected"
    },
    "colab": {
      "name": "IntroPyTorch.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "En2vX4FuwHlu"
      },
      "source": [
        "# Введение в нейронные сети\n",
        "\n",
        "## Эпизод 2b: Многослойный персептрон на PyTorch\n",
        "\n",
        "Дмитрий Сошников | dmitri@soshnikov.com\n",
        "\n",
        "http://github.com/shwars/NeuroWorkshop\n",
        "-> Notebooks -> IntroPyTorch.ipynb"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fVjlmWBwwHl2"
      },
      "source": [
        "## Нейросетевые фреймворки\n",
        "\n",
        "Мы видели, что для обучения нейросетей нужно:\n",
        "* Быстро умножать матрицы (тензоры)\n",
        "* Считать производные для вычисления градиента для метода обратного распространения ошибки\n",
        "\n",
        "Что позволяют делать нейросетевые фреймворки:\n",
        "* Оперировать с тензорами, как на CPU, так и на GPU\n",
        "* Автоматически вычислять производные (они вручную прописаны для всех элементарных функций)\n",
        "\n",
        "Опционально:\n",
        "* Конструктор для нейросетей (описание сети как набора слоёв)\n",
        "* Простые функции для обучения (`fit`, как в Scikit Learn)\n",
        "* Набор алгоритмов оптимизации\n",
        "* Набор абстракций для работы с данными"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8cACQoFMwHl3"
      },
      "source": [
        "## Основные фреймворки\n",
        "\n",
        "* Tensorflow 1.0 - первый, получивший широкое распространение (Google). Позволял определять статический computation graph, и затем в явном виде выполнять вычисления\n",
        "* PyTorch - Facebook\n",
        "* Keras - надстройка над Tensorflow/PyTorch для унификации (Francois Chollet)\n",
        "* Tensorflow 2.0 + Keras - динамический вычислительный граф, код получается похожим на обычные вычисления в numpy\n",
        "\n",
        "Мы рассмотрим PyTorch. Для начала рекомендуется установить PyTorch [по инструкции на сайте](https://pytorch.org/get-started/locally/). Либо можно выполнять код в [Google Colab](https://colab.research.google.com/), в котором PyTorch уже установлен."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "tags": [],
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "id": "xwqVx9-bwHl3",
        "outputId": "38564a63-0567-4406-ee1a-1d3618f27351"
      },
      "source": [
        "import torch\n",
        "torch.__version__"
      ],
      "execution_count": 168,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'1.8.0+cu101'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 168
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6tp2xGV7wHl4"
      },
      "source": [
        "## Основные понятия в PyTorch\n",
        "\n",
        "**Тензор** - это многомерный массив произвольной размерности. Удобно использовать при обучении нейросетей, например:\n",
        "* 400x400 - чёрно-белая картинка\n",
        "* 400x400x3 - цветная картинка\n",
        "* 16x400x400x3 - minibatch из 16 картинок, используемый для одного шага обучения\n",
        "* 25x400x400x3 - секунда видео\n",
        "* 8x25x400x400x3 - minibatch из 8 1-секундных видео"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qG2bsaR7wHl4"
      },
      "source": [
        "### Простые тензоры"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ybpnk08HwHl4",
        "outputId": "54e2c89b-b373-4389-b285-49b0510be931"
      },
      "source": [
        "a = torch.tensor([[1,2],[3,4]])\n",
        "print(a)\n",
        "a = torch.randn(size=(10,3))\n",
        "print(a)"
      ],
      "execution_count": 169,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "tensor([[1, 2],\n",
            "        [3, 4]])\n",
            "tensor([[ 0.1288, -0.8088, -0.2092],\n",
            "        [ 0.0789, -1.4706, -0.8880],\n",
            "        [-0.0601,  0.6659, -3.2120],\n",
            "        [ 0.8023, -0.4045, -1.0164],\n",
            "        [-1.5578, -1.3014, -0.8372],\n",
            "        [ 1.3626, -1.0825, -2.7018],\n",
            "        [ 0.2258, -0.7675, -0.4102],\n",
            "        [-1.9663,  0.0386, -2.6183],\n",
            "        [ 0.7666, -1.3057, -0.4486],\n",
            "        [-0.9006,  0.7092,  0.8018]])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AXFMsV3r09Ux"
      },
      "source": [
        "С тензорами можно производить обычные вычисления, которые производятся поэлементно (как в numpy). При этом тензоры автоматически дополняются до нужной размерности. Можно извлечь numpy-массив из тензора при помощи `.numpy()`:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "e5Nu5Xgj1DnQ",
        "outputId": "c1fbcd86-dde6-40b6-8edf-7a37f9d60901"
      },
      "source": [
        "print(a-a[0])\n",
        "print(torch.exp(a)[0].numpy())"
      ],
      "execution_count": 170,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "tensor([[ 0.0000,  0.0000,  0.0000],\n",
            "        [-0.0499, -0.6618, -0.6788],\n",
            "        [-0.1889,  1.4747, -3.0028],\n",
            "        [ 0.6734,  0.4044, -0.8072],\n",
            "        [-1.6867, -0.4926, -0.6280],\n",
            "        [ 1.2338, -0.2737, -2.4927],\n",
            "        [ 0.0970,  0.0413, -0.2010],\n",
            "        [-2.0951,  0.8475, -2.4091],\n",
            "        [ 0.6377, -0.4969, -0.2395],\n",
            "        [-1.0295,  1.5180,  1.0110]])\n",
            "[1.1375165  0.44538715 0.81125516]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uQ5zN6cVyrG7"
      },
      "source": [
        "## In-place и out-of-place операции\n",
        "\n",
        "Обычно операции с тензорами возвращают новые тензоры. Однако для большинства операций сущетвуют аналогичные варианты, которые модифицируют исходный тензор:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Mjkbcw3-ACKS",
        "outputId": "ca021008-9ab6-4b09-c5a5-bbe854cd1493"
      },
      "source": [
        "u = torch.tensor(5)\n",
        "print(\"Result when adding out-of-place:\",u.add(torch.tensor(3)))\n",
        "u.add_(torch.tensor(3))\n",
        "print(\"Result after adding in-place:\", u)"
      ],
      "execution_count": 171,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Result when adding out-of-place: tensor(8)\n",
            "Result after adding in-place: tensor(8)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DLPUcVsXACKT"
      },
      "source": [
        "Например, вот там можно \"наивно\" посчитать сумму строк тензора `a`:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "7pu0UZ-_yqfB",
        "outputId": "bd2e8c6a-39e1-4f29-990b-9591e866936c"
      },
      "source": [
        "s = torch.zeros_like(a[0])\n",
        "for i in a:\n",
        "  s.add_(i)\n",
        "\n",
        "print(s)"
      ],
      "execution_count": 172,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "tensor([ -1.1197,  -5.7273, -11.5398])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rIh1EHcezlNo"
      },
      "source": [
        "Умный способ:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "aQIdWZ1kzn6P",
        "outputId": "89000bb4-f45e-493b-a7b0-39fa4e7d92c1"
      },
      "source": [
        "torch.sum(a,axis=0)"
      ],
      "execution_count": 176,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([ -1.1197,  -5.7273, -11.5398])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 176
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5UzUmEZhACKT"
      },
      "source": [
        "Подробнее про тензоры в PyTorch смотрите [в официальном руководстве](https://pytorch.org/tutorials/beginner/basics/tensorqs_tutorial.html)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "U-auwezDwHl6"
      },
      "source": [
        "## Вычисляем производные\n",
        "\n",
        "Для обратного распространения ошибки, нам нужно уметь вычислять градиенты. Мы можем пометить любой тензор в PyTorch атрибутом `requires_grad`, и впоследствии автоматически будут вычисляться все градиенты при операциях с этим тензором. Для вычисления производной необходимо вызвать метод `backward()`:\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "m8vFOXr7wHl6",
        "outputId": "7054c2b1-0b61-4938-937d-813f75f0b195"
      },
      "source": [
        "a = torch.randn(size=(2, 2), requires_grad=True)\n",
        "b = torch.randn(size=(2, 2))\n",
        "\n",
        "c = torch.mean(torch.sqrt(torch.square(a) + torch.square(b)))  # Do some math using `a`\n",
        "c.backward() # call backward() to compute all gradients\n",
        "# What's the gradient of `c` with respect to `a`?\n",
        "print(a.grad)"
      ],
      "execution_count": 179,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "tensor([[0.0227, 0.1661],\n",
            "        [0.1980, 0.2253]])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nPj3rtrtACKU"
      },
      "source": [
        "На самом деле, PyTorch может таким образом вычислять \"накапливаемые\" градиенты. Если при вызове `backward` указать `retain_graph=True`, то граф вычислений будет сохраняться, и градиенты - накапливаться. Чтобы начать их вычислять заново, нужно в явном виде обнулить поле `grad`:  "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "z_VIw8MoACKU",
        "outputId": "36a28b11-6919-47ab-c3f9-c7f1d8500423"
      },
      "source": [
        "c = torch.mean(torch.sqrt(torch.square(a) + torch.square(b)))\n",
        "c.backward(retain_graph=True)\n",
        "c.backward(retain_graph=True)\n",
        "print(a.grad)\n",
        "a.grad.zero_()\n",
        "c.backward()\n",
        "print(a.grad)"
      ],
      "execution_count": 180,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "tensor([[0.0681, 0.4983],\n",
            "        [0.5939, 0.6758]])\n",
            "tensor([[0.0227, 0.1661],\n",
            "        [0.1980, 0.2253]])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HM9sUkVgCiG9"
      },
      "source": [
        "Для вычисления градиентов PyTorch создаёт и поддерживает **граф вычислений**. Для каждого тензора, который вычисляется с использованием тензоров с установленным флагом `requires_grad`, устанавливается специальная функция `grad_fn`, представляющая собой функцию для вычисления производной по правилу дифференциирования сложной функции:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "PcxHb-7jC7Vv",
        "outputId": "3b3fa138-6d09-4636-8a71-f4a4051c7827"
      },
      "source": [
        "print(c)"
      ],
      "execution_count": 181,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "tensor(1.0358, grad_fn=<MeanBackward0>)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rvLfNiblACKV"
      },
      "source": [
        "На самом деле, PyTorch может вычислять градиенты только для скалярных функций. Если речь идет о вычислении производной тензора по тензору, то PyTorch позволяет нам вычислять произведение якобиана на вектор.\n",
        "\n",
        "Например, пусть есть векторная функция $\\vec{y}=f(\\vec{x})$, где\n",
        "$\\vec{x}=\\langle x_1,\\dots,x_n\\rangle$ и\n",
        "$\\vec{y}=\\langle y_1,\\dots,y_m\\rangle$, тогда градиент $\\vec{y}$ по $\\vec{x}$ задаётся **якобианом**:\n",
        "\n",
        "$$\n",
        "\\begin{align}J=\\left(\\begin{array}{ccc}\n",
        "   \\frac{\\partial y_{1}}{\\partial x_{1}} & \\cdots & \\frac{\\partial y_{1}}{\\partial x_{n}}\\\\\n",
        "   \\vdots & \\ddots & \\vdots\\\\\n",
        "   \\frac{\\partial y_{m}}{\\partial x_{1}} & \\cdots & \\frac{\\partial y_{m}}{\\partial x_{n}}\n",
        "\\end{array}\\right)\\end{align}\n",
        "$$\n",
        "\n",
        "Вместо вычисления якобиана, PyTorch вычисляет произведение $v^T\\cdot J$ на некоторый вектор\n",
        "$v=(v_1 \\dots v_m)$. Для этого необходимо вызвать ``backward``, передав `v` в качестве аргумента. Размер `v` должен совпадать с размером исходного тензора, по которому мы вычисляем производную.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "VUNYiQCOACKV",
        "outputId": "e3127c21-fce6-420d-f347-ec40cc827e7e"
      },
      "source": [
        "c = torch.sqrt(torch.square(a) + torch.square(b))\n",
        "c.backward(torch.eye(2))\n",
        "print(a.grad)"
      ],
      "execution_count": 182,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "tensor([[0.1135, 0.1661],\n",
            "        [0.1980, 1.1263]])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dGHlkVlvACKV"
      },
      "source": [
        "Подробнее про вычисление градиентов в PyTorch читайте [в официальном руководстве](https://pytorch.org/tutorials/beginner/basics/autogradqs_tutorial.html)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FnVvj4LkD15r"
      },
      "source": [
        "# Пример 0: Оптимизация функции методом градиентного спуска\r\n",
        "\r\n",
        "Попробуем использовать автоматическое дифференциирования для оптимизации простой функции двух переменных $f(x_1,x_2)=(x_1-3)^2+(x_2+2)^2$. Пусть тензор `x` представляет собой текущие координато точки. Мы начнем с некоторой начальной точки $x^{(0)}=(0,0)$, и будет вычислять следующую точку по формуле:\r\n",
        "$$\r\n",
        "x^{(n+1)} = x^{(n)} - \\eta\\nabla f\r\n",
        "$$\r\n",
        "Здесь $\\eta$ - т.н. **learning rage** (назоём его `lr` в коде), а $\\nabla f = (\\frac{\\partial f}{\\partial x_1},\\frac{\\partial f}{\\partial x_2})$ - градиент функции $f$.\r\n",
        "\r\n",
        "Для начала определим стартовое значение `x` и функцию `f`:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nDw5mV9KEeOa"
      },
      "source": [
        "x = torch.zeros(2,requires_grad=True)\r\n",
        "f = lambda x : (x-torch.tensor([3,-2])).pow(2).sum()\r\n",
        "lr = 0.1"
      ],
      "execution_count": 184,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Wt815LWdEj77"
      },
      "source": [
        "Теперь проделаем 15 итераций градиентного спуска. На каждой итерации, мы будем обновлять координаты `x` и печатать их, чтобы убедиться, что мы достигаем минимума (3,-2):"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "KfwMf555EyWJ",
        "outputId": "67e2199c-61ff-4ad1-9c48-b4a646bf8bbd"
      },
      "source": [
        "for i in range(15):\r\n",
        "    y = f(x)\r\n",
        "    y.backward()\r\n",
        "    gr = x.grad\r\n",
        "    x.data.add_(-lr*gr)\r\n",
        "    x.grad.zero_()\r\n",
        "    print(\"Step {}: x[0]={}, x[1]={}\".format(i,x[0],x[1]))"
      ],
      "execution_count": 186,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Step 0: x[0]=1.2000000476837158, x[1]=-0.800000011920929\n",
            "Step 1: x[0]=1.5600000619888306, x[1]=-1.0399999618530273\n",
            "Step 2: x[0]=1.8480000495910645, x[1]=-1.2319999933242798\n",
            "Step 3: x[0]=2.078400135040283, x[1]=-1.385599970817566\n",
            "Step 4: x[0]=2.2627201080322266, x[1]=-1.5084799528121948\n",
            "Step 5: x[0]=2.4101760387420654, x[1]=-1.6067839860916138\n",
            "Step 6: x[0]=2.5281407833099365, x[1]=-1.685427188873291\n",
            "Step 7: x[0]=2.6225125789642334, x[1]=-1.7483417987823486\n",
            "Step 8: x[0]=2.698009967803955, x[1]=-1.798673391342163\n",
            "Step 9: x[0]=2.7584080696105957, x[1]=-1.8389387130737305\n",
            "Step 10: x[0]=2.8067264556884766, x[1]=-1.8711509704589844\n",
            "Step 11: x[0]=2.845381259918213, x[1]=-1.8969208002090454\n",
            "Step 12: x[0]=2.876305103302002, x[1]=-1.9175366163253784\n",
            "Step 13: x[0]=2.9010441303253174, x[1]=-1.9340293407440186\n",
            "Step 14: x[0]=2.920835256576538, x[1]=-1.947223424911499\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8sfjBMBu59B5"
      },
      "source": [
        "## Пример 1: Линейная регрессия\r\n",
        "\r\n",
        "Попробуем с помощью полученных знаний решить классическую задачу линейной регрессии. Для этого сгенерируем небольшой синтетический датасет:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "id": "j723455WwHl7"
      },
      "source": [
        "import numpy as np\n",
        "import matplotlib.pyplot as plt\n",
        "from sklearn.datasets import make_classification, make_regression\n",
        "from sklearn.model_selection import train_test_split\n",
        "import random"
      ],
      "execution_count": 187,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 282
        },
        "id": "WJNK_J6v6I-Z",
        "outputId": "09e6386e-a6d4-4b81-c8d2-153f0acf9696"
      },
      "source": [
        "np.random.seed(13) # pick the seed for reproducability - change it to explore the effects of random variations\n",
        "\n",
        "train_x = np.linspace(0, 3, 120)\n",
        "train_labels = 2 * train_x + 0.9 + np.random.randn(*train_x.shape) * 0.5\n",
        "\n",
        "plt.scatter(train_x,train_labels)"
      ],
      "execution_count": 188,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<matplotlib.collections.PathCollection at 0x7f0ffb8714d0>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 188
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWoAAAD4CAYAAADFAawfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAb2ElEQVR4nO3df4xlZXkH8O+zwwUuP8pVmVoYXZemzZIKZZedEJo1RtZYbFHcgFEb0xbbZpM2bYVQ2tE/ivYfJjFWbW1qJpQWW6pLXdhQUJFkN6FuFDvDLgKyNChuZbBllB0QZsrODE//uPcsd86c97zvuec957zvvd9PsnF25+yZ9+yV5773eZ/3eUVVQURE4drU9ACIiCgfAzURUeAYqImIAsdATUQUOAZqIqLAnVLFTc8991zdsmVLFbcmIhpKc3NzP1HV8azvVRKot2zZgtnZ2SpuTUQ0lETkmOl7TH0QEQWOgZqIKHAM1EREgWOgJiIKHAM1EVHgKqn6ICIaJfsPz+NT9z+JZxeXcX6njZuu3Ird2ye83Z+BmoiohP2H5/Gxux7F8soaAGB+cRkfu+tRAPAWrJn6ICIq4VP3P3kySCeWV9bwqfuf9PYzGKiJiEp4dnG50J8PgoGaiKiE8zvtQn8+CAZqIiJ0c807pw/ggqn7sHP6APYfnnf6ezdduRXt1ti6P2u3xnDTlVu9jY2LiUQ08sosCCbfZ9UHEVGF8hYEXQLu7u0TXgNzGlMfRDTy6lgQLMMaqEVkq4gc6fv1oohcX8fgiIjqUMeCYBnWQK2qT6rqNlXdBmAHgCUAd1c+MiKimtSxIFhG0Rz1OwF8X1WNDa6JiGJTZEGwf7v4Oe0WRIDFpZVKFhEToqruF4vcBuBhVf18xvf2ANgDAJs3b95x7BhjORENl3R1SFq7NYZbrrl4oGAtInOqOpn1PefFRBE5FcDVAP4t6/uqOqOqk6o6OT6eeewXEVHw8uqps6pD+vneOp4okvr4DXRn0//rfRRERAGw1VO7VIFUUSlSpDzvtwB8yfsIiIgCYWuw5FIFUkWliFOgFpEzAbwLwF3eR0BEFAhbPXVWdUi/qipFnFIfqvoygDd4/+lERAE5v9PGfEawTmbJ6eqQuqo+uIWciKjnpiu3bqjqSM+Sq94unoWBmoiop44GS4NgoCYi6tPEjNmGTZmIiALHQE1EFDgGaiKiwDFHTUQjob+ZUiiLhK4YqIloKOQF4kGP2kruOb+4jDERrKliooEgz0BNRIWEODPNC8QAcOOdj2At1SnUdtRW+p7J3y9ynqIvhdqcupqcnNTZ2Vnv9yWiZmW1+SzT2tOXndMHMncUdtotvLL6am7Hu4lOO/NNx3TP/r93aGpX+cH3eGlzSkRka1rUFFOPjsXlldwgLejOkBWvzZSTtqa2Lnh1nqfIQE1EzkI9BHaQjnUCIJ1PKNIpr87zFBmoiciZKTid024Zm+3XwXTm4evOaGVePyayIUgnXDrl1X2eIgM1ETnLCl6tTYKXT6waUwhVSJ/CAgC3XHMxJjptCLr541uuuRg3v/etmQH80x+4BBOWk8d3b584eU+gG9zRd29WfRBRkLKaFi2dWMXxpZV119kqKsowVXjccs3FxsU9U5VKiJ3ysrDqg4hKuWDqvsw0ggB4evoq7z/PVI0xSBVGSKWGeVUfnFETUSm2Zvu++VzQDGXGbMMcNRGVYlrIq2qxzfQGUGcVRt0YqImolP5Ft/6FvKpmqnW/MYSAqQ8iKq3OFEKop7BUySlQi0gHwK0ALkK3Rvz3VPVbVQ6MiMgkltyyL64z6s8B+Lqqvl9ETgVwRoVjIiLKFFKVRp2sgVpEzgHwdgDXAYCqngBwotphERGtN2ir0mHgsph4AYAFAP8oIodF5FYROTN9kYjsEZFZEZldWFjwPlAialZ6N2Dd28RDbQhVB5dAfQqASwH8vapuB/AygKn0Rao6o6qTqjo5Pj7ueZhE1KRkNlvnNvH+n53XcrTphlB1cMlRPwPgGVV9qPf7ryAjUBPR8DLNZj9xz+O5p6qUzSdn9b9OG+b66YQ1UKvq/4jIj0Rkq6o+CeCdAL5X/dCIKBR5/Z4Xl7t9PtKnqvjIJ2e9QfQb9vrphGvVx58AuKNX8fEDAB+pbkhEFBrTNvG0/pyxKZ9cJFDnpTWaOLuwKU6BWlWPAMhsFkJEw++mK7daUxCJvOBaNJ9seoPwfQxW6LiFnIissraJm5ryn99pO/XjcKkiGcXt4lm4hZyInKR3A5oOuk2CaN73XGqik8XI5ZU1jIlgTXWk0h39GKiJRliZygyXnhum7+XVRO/ePrEhkK+pngz0oxakAR4cQDSyTDPiOo6ZMh02AHTTKqaFy6zc9LBsK887OIA5aqIR1eROP1MOW4Dc6pL0YmSTG3HqxEBNNKJ8npRSVNYioQDGWXYiHeBHZVs5AzXRkDNVV5hmtQpU3ssjXUXSabesQTqr2qPJN5s6cTGRaIjlVVfk1Ub77kxnyiP3LxzmMVV71H1eY1MYqImGUBIYs4JYkhpIFuVs15UN1LZSvLxt4rbFzaw3m3ZrDFdcOI6d0weiX2BMMPVB5EHTLUDTY0kW2EyS1MDu7RM4NLULYrmuDFseOe9n2CpQsjbiXLtjAvvm5odqgZEzaqKSQmtob2tkBGxMDfhMIaTTHLb2pHnbxF3+/dIbcXZOH/DSZyQknFETldRE5UHeDN42C85alPO1VTurXM40W0/eBHxvE3dZYAzpE5ALzqiJSqq78sA2g8+bxY6JrHsTSWaYvk72znrTUmSX3i2dWMX+w/PeTxW3fToI7ROQCwZqopLqrjywbb/OWmBrbRJAgJW1brjMCk4+TvY2vTkpuiV4Se9qADi+tLJuDL6CpGmBMZmh2/79QsTUB1FJdXd4s83gsxbYzjr9lJNBOmFKz5RJC5jenCY6bZx52sZ5YRUpoqzn71+UjLH2mjNqopJ8f3S3cZnBp2eoF0zdl3kv05bsQdMCebPZG/YecRqDD3kz9BhrrzmjJvIgKXN7evoqHJraVelH6EFm8C79oYHyC6N5s1nXMVQtxh7XnFETRWaQGbwtb5vwkRYwzWZdx1C1uj8B+cBATRShootvrsGpyrRASAHS5+JlHdiPmihQTfRZbrJH9ajL60fNGTVRgJqq9Q1p1kuvcQrUIvJDAD8DsAZg1RT1iciPJmt9Y0sLjIIiM+orVPUnlY2EiE6KsdaXqsPUB1EA0vnozhktHF9a2XBdyLW+VB3XOmoF8A0RmRORPVkXiMgeEZkVkdmFhQV/IyQaclmNjF76v1W0xta3Mwq91peq4zqjfpuqzovIzwN4QESOquqD/Reo6gyAGaBb9eF5nERDIz17XjqxuiEfvfKqotNu4czTTuGiHrkFalWd7/3vcyJyN4DLADyY/7eIKC2rmsPkheUVHLn51+saGgXMmvoQkTNF5OzkawC/DuCxqgdGNIxcmvonmI+mhMuM+o0A7haR5Pp/VdWvVzoqoiHlWrXBfDT1swZqVf0BgEtqGAvR0DNt0WY+mvKwPI+i1MT2ah9MjYk+cfVbjePvf9Zz2i2IAItLK1E9N5XDQE3RqXN7te0NoegbRtEt2uln7T8hJeu5Y30Do3xsykTR2Tl9wHhq9aGpXd5+jq1BUR0NjEzP2i95bjZUilteUyYeHEDRqWt7ta2Jfh2nj7s8U3JNE6ehUz0YqCk6dZ0UYntDqOMNw+WZkmvYH2R4MVBTdOo6Ssn2hlDHG0bWs/brf+5Qjroi/xioKTq2U6Z9sb0h1PGGkX7WTruF153RynzuKsZT5kRy8oeLiUQ5fFd9ND3eovfi4mR98hYTGaiJIlfVm0Vd1TXUxaO4iCJRNOhWWVPOxclwMEdNlNJUXjarL/XH7no09+dXWZLHxclwMFBTVKoOooMES9f72sY9SNCtctZbV3UN2TFQUzSqCqL9qpihuo57kKBb5ay3ruoasmOOmqJRx8ncVcxQXcdt6qyXF3RNTZ58zXp5InkYGKgpGnXtBCwaLG1M45tfXMbO6QMnFw6vuHAc++bmCwXdok2eKE4M1BSNKoJoWhUzVNO4Ba8dxTW/uIx9c/O4dscEDh5dKBR0OesdfgzUFI2qP+YD+TPUQeuVs8YtANI7GJZX1nDw6EItNcqhbdShfNzwQlFpKsBk7dIDgNed0cLN781u+p/X8N/UulQAPD19VRWPsG5c3HEYHm54oaFh+5hfVSA3HUp7fGklc4NJVsP/dmsMn/ngNuzePmHc9VdHjXIdi7LkF8vzyIsQmvdUWb6Xt2CZVb5nK/NrskaZOw7j4xyoRWRMRA6LyL1VDojiU0d9swtTcLzxzkdKj8U2000HOVswbLJGmTsO41Mk9fFRAE8A+LmKxkKRCuWjtCk4rqmW7n+RtSDYrz/I7T88j00iWMtY/+m/rqlqjToWZckvpxm1iLwJwFUAbq12OBSjUD5K580Iy+4uTGbAnXZrw/f6g1zy6SIrSIcSDLnjMD6uM+rPAvhzAGebLhCRPQD2AMDmzZvLj4yiUUd9swvbrLfsG0cyA85bsDQtOo6JBBUMWXsdF2ugFpH3AHhOVedE5B2m61R1BsAM0C3P8zZCCl4oH6WTwHPjnY9Y0w5lf44pyJneDF5VZWCkgbmkPnYCuFpEfgjgywB2ici/VDoqikpIH6V3b5/Apz9wSWMVFVyooyoU2vDSm1H/maq+J+86bnihpoW0MYabScgFN7zQyGkqB8smSVQFbiGnxjTZb4K9Lig0nFFTcHye9efrnMHZY88X7lxHVAfOqKkRpl4XQHcx0jVIFskJJwE9ryFS/38NzC1TnfJm1Oz1QY3Iq2kusgXd5eis/Yfnse2T38D1e48YgzSQ3XbUxyGxRGUxUFMjbOVqrkHStisymXEvLq8UH2TO/YnqxEBNznx2yMvqHpfmEiRtdcumnYJpUvD+RHVioCYnvjvk9W+SMXEJkrZ2oS7BfqLTxocv39zYJhkiG1Z9kJMqOuT1984YdAu6rW457zSV9GLh5Ftez5I9ChIDNTmpskNe2U0ieZtbTI2aso7QYqMiChUD9RCoY/NG1R3yqgqS3ClIw4CBOnI+N47kCaVD3iA4U6bYcTExci51xD6E1CGPaNRwRh25Ok9XqWNm2p/GOafdgkj3pO+x3tFWRXYtEg0LBurIhXK6io1LHj2dxunfpJIcBFBVaocoZEx9RM5WRxwC1xps180p3NpNo4aBOnIx5I5NefTr9x5Zt8OxSLqGW7tplDD1MQRCr2pwacAE5G9OSQsttUNUJQZq8i6dj+6c0cLxJXNTpCSVYTtFPBFaaoeoagzU5FVWXXdrk6A1JlhZM/c+f3ZxecPmFFZ9EHUxUJNXWfnolVcVnXYLZ552ijG1kaQy6k7j8EguigEXE8krUz76heUVHJrahc9+cFswVSq+OwISVcUaqEXkdBH5jog8IiKPi8gn6xgYvcZnH+iq2fpDh1SlUteuTqKyXFIfrwDYpaoviUgLwDdF5Guq+u2Kx0aor5eHLy49QUKpUqlzVydRGdZArd3Tb1/q/bbV++X/RFzK5LMPdB352Ji61cWyq5PI6RRyERkDMAfglwD8nar+RcY1ewDsAYDNmzfvOHbsmOehjqYLpu7LfFcUAE9PX+V8n6zm/Mmp21mVFKOwyFbkBHOiquWdQu5U9aGqawC2iUgHwN0icpGqPpa6ZgbADABMTk5yxu2Jr1lf1sw8eZHS6RRbuiWrcdLi0kp0AT2m2T+NtkLleaq6KCIHAbwbwGO264dVnbNNX32gbXnX/nSKbZHN1Dgp9Px5llDy5UR5XKo+xnszaYhIG8C7ABytemChqruky1eVhMsMPAnmeYtstsZJrJog8s9lRn0egNt7eepNAO5U1XurHVa4qjjk1cbHrM9le3YSzE3plk0iTr04WDVB5JdL1cd3AWyvYSxRiLGkK0nVLK+sndyKnSwkJvrTKaagnvX3siQBfxQWJInqwJ2JBdk2dISmP1UDdINtuzWGD1++2ZhOSdItYyIb7qfoVouYJAGfu/6I/GGvj4JCOuTVZcZqStUcPLqAQ1O7jPfevX0CN+w9kvm9pKQvr+pj5/SB2lNERMOKgbqgUEq6XHcslknVmHLVE512bpAv+3OJaD0G6gGEUNLluqhZpg67zKcH7voj8oc56opV1VDJdcZa5kzFMqWBMZzlSBQLzqgrVGVDJdcZa9lUzaCfHkJJERENA6deH0VNTk7q7Oys9/vGZuf0gcxgOiaCT3/gklJBi30qiIZL6V4fNBhTemJNtfTMmjNWotHBQF2hvFO1fZSq+V7U5AYVojBxMbFCWQtq/UIqVeMGFaJwMVBXKG+HHxBWqRqPpSIKF1MfFUtSB3XvZiyaxuAGFaJwMVDXwGXhz2d+eJCyQG5QIQoXA3VN8hb+fNdbD9KKNaQeJkS0HnPUAfCZH95/eN5YaZKXxvB1QAER+ccZdQB85YeTmbmJLY0RQg8TItqIM+oA+OpxnXdMFtMYRPHijLokH4uApvzwFReOY+f0gdKVGwCYxiCKGAN1Cb4WAbOqQq64cBz75ua9VG5MdNoM0kQRG9pAXXamW+b0FNet4Xk/Y5ATUli5QTScrIFaRN4M4IsA3ojuKUwzqvq5qgdWRtmZbh2np9h+xiD3HqRRE/t7EIXPZUa9CuBGVX1YRM4GMCciD6jq9yoe28DKznTrOD3F9jMGvXeRyo0q+2UTkT/Wqg9V/bGqPtz7+mcAngAQ9H/FZcvd6jg9xfYz6jghhf09iOJQqDxPRLYA2A7goYzv7RGRWRGZXVhY8DO6AZUtd3P9+2U2idh+Rh0bUNjfgygOzouJInIWgH0ArlfVF9PfV9UZADNA94QXbyN0kM6zpismgGKz0SKLcoNuEnH5GVVvQGF/D6I4OM2oRaSFbpC+Q1XvqnZIxWT1Ud43N49rd0wMPBsdZDZb9BDbELZs8wBaojhYz0wUEQFwO4DnVfV6l5vWeWai6VzCiU4bh6Z2lb6/S1VEzOcXsuqDKAx5Zya6BOq3AfgPAI8CeLX3xx9X1a+a/k6VgTodWEwNiATA09NXFb5ff6DKCsCCbo3iRN+1Vb9ZENHwK3W4rap+E9341LiscrIkcKa55Flt5WlZVRHJz+q/lotyRFSlqJoymQJn+l3ENc9qK0+zBdrkWl9NlYiIskQVqE2BM0lFCIBOu4XTW5tww94j1kU920zYJdA+u7jMRTkiqlRUgdoUOJNc8Gc+uA2vrL6K40srTidp22bCtlPEk2tDqOAgouEVVVMmW+1x0a3jtvv1987Iyoenr2VgJqIqRBWobU2Hii7quTQx6g/AIZSyhTAGIqqXtTxvEHXWUfcb9jK5mOu1iShfXnleVDlqm1gW9YruYkywiRLRaIoq9WEzSD/mupVpLcp6baLRFHWgNuVrBw3MvvK/efcp0yubTZSIRlO0gdp303tf9yt7cktekOdRW0SjKdocte98ra/72e6TV7ud1Qmwvw6c9dpEoym4GbVr+sF3vtbX/VxObjHNil3SIqzXJho9QQVqU9pg9tjzOHh0YV3w9p2v9XU/233yFjxv2Hsk855cLCQabUEFatOM8o5v//eGrnXX7pgodYpLmq/8b5mTW7hYSERZgspR5zVd6re8soaDRxe85mt95X/L3CeWOnAiqldQOxNNOwuzuB4MEBtuEScaTaUODqhTVtqgzMEAMeJiIRGlBZX6yEobfPjyzUwHENFIC2pGDWTPKCff8nqmA4hoZAUXqLMwHUBEo8waqEXkNgDvAfCcql5U/ZCqw4U6IoqRy4z6nwB8HsAXqx3KRj4Dq+/eIEREdbEuJqrqgwCer2Es69j6XhTFXs5EFKugqj76+Q6s7OVMRLHyFqhFZI+IzIrI7MLCQun7+Q6sthPHiYhC5S1Qq+qMqk6q6uT4+Hjp+/kOrNyeTUSxCjb14TOwJouSyytrGBMBAHTaLZze2oQb9h4pdG4hEVHdrIFaRL4E4FsAtorIMyLy+9UPy1+TpP5FSQBYU0Vrk+DlE6s4vrTiZaGSiKhKQTVlqkKRRk8TnTYOTe2qeERERBvlNWUKNvXhS5HFR1aAEFGIhj5QF1l8ZAUIEYVo6AN11qJka5OgNSbr/owVIEQUqiiaMpVhOqMw68+4lZyIQjT0i4lERDGI4oQXdrYjIsoWRKBmZzsiIrMgFhPZ2Y6IyCyIQM3OdkREZkEEana2IyIyCyJQs7MdEZFZEIuJplpnLiQSEQUSqAGeNE5EZBJE6oOIiMwYqImIAsdATUQUOAZqIqLAMVATEQWuku55IrIA4NiAf/1cAD/xOJwmDcuzDMtzAHyWEA3LcwDlnuUtqjqe9Y1KAnUZIjJravUXm2F5lmF5DoDPEqJheQ6gumdh6oOIKHAM1EREgQsxUM80PQCPhuVZhuU5AD5LiIblOYCKniW4HDUREa0X4oyaiIj6MFATEQWusUAtIu8WkSdF5CkRmcr4/mkisrf3/YdEZEv9o7RzeI7rRGRBRI70fv1BE+O0EZHbROQ5EXnM8H0Rkb/pPed3ReTSusfoyuFZ3iEiL/S9Jn9Z9xhdicibReSgiHxPRB4XkY9mXBP8a+P4HFG8LiJyuoh8R0Qe6T3LJzOu8Ru/VLX2XwDGAHwfwC8COBXAIwB+JXXNHwH4Qu/rDwHY28RYPTzHdQA+3/RYHZ7l7QAuBfCY4fu/CeBrAATA5QAeanrMJZ7lHQDubXqcjs9yHoBLe1+fDeC/Mv4/Fvxr4/gcUbwuvX/ns3pftwA8BODy1DVe41dTM+rLADylqj9Q1RMAvgzgfalr3gfg9t7XXwHwThGRGsfowuU5oqCqDwJ4PueS9wH4onZ9G0BHRM6rZ3TFODxLNFT1x6r6cO/rnwF4AkC6cXvwr43jc0Sh9+/8Uu+3rd6vdFWG1/jVVKCeAPCjvt8/g40v2slrVHUVwAsA3lDL6Ny5PAcAXNv7SPoVEXlzPUPzzvVZY/FrvY+uXxORtzY9GBe9j8/b0Z3B9Yvqtcl5DiCS10VExkTkCIDnADygqsbXxEf84mJi9f4dwBZV/VUAD+C1d1lqzsPo9lW4BMDfAtjf8HisROQsAPsAXK+qLzY9nkFZniOa10VV11R1G4A3AbhMRC6q8uc1FajnAfTPLN/U+7PMa0TkFADnAPhpLaNzZ30OVf2pqr7S++2tAHbUNDbfXF6zKKjqi8lHV1X9KoCWiJzb8LCMRKSFbnC7Q1XvyrgkitfG9hyxvS4AoKqLAA4CeHfqW17jV1OB+j8B/LKIXCAip6KbbL8ndc09AH639/X7ARzQXmY+INbnSOUKr0Y3NxejewD8Tq/C4HIAL6jqj5se1CBE5BeSfKGIXIbufwehTQIAdCs6APwDgCdU9a8NlwX/2rg8Ryyvi4iMi0in93UbwLsAHE1d5jV+NXK4raquisgfA7gf3cqJ21T1cRH5KwCzqnoPui/qP4vIU+guDH2oibHmcXyOPxWRqwGsovsc1zU24Bwi8iV0V93PFZFnANyM7iIJVPULAL6KbnXBUwCWAHykmZHaOTzL+wH8oYisAlgG8KEAJwGJnQB+G8CjvZwoAHwcwGYgqtfG5TlieV3OA3C7iIyh+2Zyp6reW2X84hZyIqLAcTGRiChwDNRERIFjoCYiChwDNRFR4BioiYgCx0BNRBQ4BmoiosD9P3fFq1WlCzEwAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ng4rZmGc6oxk"
      },
      "source": [
        "Линейная регрессия вычисляется как $f_{W,b}(x) = Wx+b$, где $W, b$ - параметры модели, которые необходимо найти. Функция ошибки на наборе данных $\\{x_i,y_u\\}_{i=1}^N$ может быть определена как среднеевадратичное отклонение\r\n",
        "$$\r\n",
        "\\mathcal{L}(W,b) = {1\\over N}\\sum_{i=1}^N (f_{W,b}(x_i)-y_i)^2\r\n",
        "$$\r\n",
        "\r\n",
        "Опишем модель и функцию ошибки:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QxhI4GlB6aiH"
      },
      "source": [
        "input_dim = 1\r\n",
        "output_dim = 1\r\n",
        "learning_rate = 0.1\r\n",
        "\r\n",
        "# This is our weight matrix\r\n",
        "w = torch.tensor([100.0],requires_grad=True,dtype=torch.float32)\r\n",
        "# This is our bias vector\r\n",
        "b = torch.zeros(size=(output_dim,),requires_grad=True)\r\n",
        "\r\n",
        "def f(x):\r\n",
        "  return torch.matmul(x,w) + b\r\n",
        "\r\n",
        "def compute_loss(labels, predictions):\r\n",
        "  return torch.mean(torch.square(labels - predictions))"
      ],
      "execution_count": 189,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JUxwj3367gD2"
      },
      "source": [
        "Обучать модель будем на сериях примеров - minibatches. Для обучения используем градиентный спуск, подстраивая парметры в соответствии с формулой:\r\n",
        "$$\r\n",
        "\\begin{array}{l}\r\n",
        "W^{(n+1)}=W^{(n)}-\\eta\\frac{\\partial\\mathcal{L}}{\\partial W} \\\\\r\n",
        "b^{(n+1)}=b^{(n)}-\\eta\\frac{\\partial\\mathcal{L}}{\\partial b} \\\\\r\n",
        "\\end{array}\r\n",
        "$$"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-991PErM7fJU"
      },
      "source": [
        "def train_on_batch(x, y):\r\n",
        "  predictions = f(x)\r\n",
        "  loss = compute_loss(y, predictions)\r\n",
        "  loss.backward()\r\n",
        "  w.data.sub_(learning_rate * w.grad)\r\n",
        "  b.data.sub_(learning_rate * b.grad)\r\n",
        "  w.grad.zero_()\r\n",
        "  b.grad.zero_()\r\n",
        "  return loss"
      ],
      "execution_count": 190,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "idr2VEWb9rr0"
      },
      "source": [
        "Теперь приступаем к обучению: делаем несколько проходов по всему датасету (эпох), разбиваем его на minibatches, и вызываем функцию обучения:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nOuu0qpx-wAp"
      },
      "source": [
        "# Shuffle the data.\r\n",
        "indices = np.random.permutation(len(train_x))\r\n",
        "features = torch.tensor(train_x[indices],dtype=torch.float32)\r\n",
        "labels = torch.tensor(train_labels[indices],dtype=torch.float32)"
      ],
      "execution_count": 191,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "3zdIf6c_85Ht",
        "outputId": "6520288c-da59-4a9f-c37e-cd99779c3073"
      },
      "source": [
        "batch_size = 4\r\n",
        "for epoch in range(10):\r\n",
        "  for i in range(0,len(features),batch_size):\r\n",
        "    loss = train_on_batch(features[i:i+batch_size].view(-1,1),labels[i:i+batch_size])\r\n",
        "  print('Epoch %d: last batch loss = %.4f' % (epoch, float(loss)))"
      ],
      "execution_count": 192,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch 0: last batch loss = 94.5247\n",
            "Epoch 1: last batch loss = 9.3428\n",
            "Epoch 2: last batch loss = 1.4166\n",
            "Epoch 3: last batch loss = 0.5224\n",
            "Epoch 4: last batch loss = 0.3807\n",
            "Epoch 5: last batch loss = 0.3495\n",
            "Epoch 6: last batch loss = 0.3413\n",
            "Epoch 7: last batch loss = 0.3390\n",
            "Epoch 8: last batch loss = 0.3384\n",
            "Epoch 9: last batch loss = 0.3382\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "US6q0nCBD-LL",
        "outputId": "c804b779-3231-4f6f-c854-032d211b2853"
      },
      "source": [
        "w,b"
      ],
      "execution_count": 193,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(tensor([1.8617], requires_grad=True), tensor([1.0711], requires_grad=True))"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 193
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 282
        },
        "id": "_e6xRMZFDnyI",
        "outputId": "79e6c360-265a-401d-ce39-8f211917a13d"
      },
      "source": [
        "plt.scatter(train_x,train_labels)\r\n",
        "x = np.array([min(train_x),max(train_x)])\r\n",
        "with torch.no_grad():\r\n",
        "  y = w.numpy()*x+b.numpy()\r\n",
        "plt.plot(x,y)"
      ],
      "execution_count": 194,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[<matplotlib.lines.Line2D at 0x7f0ffb65ba10>]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 194
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWoAAAD4CAYAAADFAawfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deZhU1Z3/8ffp7uqFrRulQWhoQFZFZOsxGowKiBhN3KNJRo3JTJiJGbcYlGQWZ/L8ZgTcMyYTiZplYhKNInFiZFFQXFFWUeluEGRp9qWbpffq8/uju9qmum7Vra5b1beaz+t5eNJ03br3XMp876lzvud7jLUWERHxr4zOboCIiESnQC0i4nMK1CIiPqdALSLicwrUIiI+l5WMk/bp08cOGTIkGacWEemSVq9efcBaWxjptaQE6iFDhrBq1apknFpEpEsyxmxzek1DHyIiPqdALSLicwrUIiI+p0AtIuJzCtQiIj6XlKwPEZGTycK1FTywuIxdlTUMKMhj1oxRXDWhyLPzK1CLiCRg4doKfrRgAzUNQQAqKmv40YINAJ4Faw19iIgk4IHFZa1BOqSmIcgDi8s8u4YCtYhIAnZV1sT1+45QoBYRScCAgry4ft8RCtQiIjSPNU+es4yhs19m8pxlLFxb4ep9s2aMIi+QecLv8gKZzJoxyrO2aTJRRE56iUwIhl5X1oeISBJFmxB0E3CvmlDkaWAOp6EPETnppWJCMBExA7UxZpQxZl2bP0eMMXemonEiIqmQignBRMQM1NbaMmvteGvteGASUA28mPSWiYikSComBBMR7xj1NOBTa61jgWsRkXQTz4Rg2+Xi+XkBjIHK6oakTCKGGGut+4ONeRpYY619PMJrM4GZAMXFxZO2bVMsF5GuJTw7JFxeIJP7rxnboWBtjFltrS2J9JrryURjTDZwBfCnSK9ba+dba0ustSWFhRG3/RIR8b1o+dSRskPa8nrpeEg8Qx9fprk3vdfzVoiI+ECsfGo3WSDJyBSJJz3vG8AfPG+BiIhPxCqw5CYLJBmZIq4CtTGmOzAdWOB5C0REfCJWPnWk7JC2kpUp4mrow1p7HDjV86uLiPjIgII8KiIE61AvOTw7JFVZH1pCLiLSYtaMUe2yOsJ7ycleLh6JArWISItUFFjqCAVqEZE2OqPHHIuKMomIeOCjiiqeWZmchX7qUYuIJOCzA8d5cEkZf/lwN3175nDNhIHkZTtnhnSEArWISAfsO1rLf7+2mT+8v51AZga3TR3Ody843fMgDQrUInKSaFtMKZFJwqO1DcxfsYWn3tpKfWMTXz9nELdPG0HfnrlJaHUzBWoR6RKiBeKObrUVOmdFZQ0ZQBOQYaDJwlfO7s8PLxnFkD7dk31rCtQiEh+veqZet8kpEAPc/dx6gmGVQmNttRV+zqaW3zdZyMnK4OIz+qUkSEOcZU7dKikpsatWrfL8vCLSuSKV+UyktKdXJs9ZFnFFYUFegLrGpqgV74oK8iI+dL54/2vsqqqN+r63Z09NvPEtopU5VY9aRFxLdBPYZHGq0VFZ0xD1fQZaA3zbXnjxqd2iBulo10wGBWoRcc2vm8A61eiIxgDh4wk1DUFmL/iQ2oam1rHoaNdMFS14ERHXnIJTfl7Asdh+Kjjtedi7WyDi8ZnGtAvSIbUNTdw9fST3Xz3WsVJeqvdTVKAWEdciBcRAhuF4fSMVlTVYPh9CSGawDt+FBeD+a8ZSVJCHoXn8+P5rxnLfV8dEDOAPXT+OIoeHzmm9crlt2ghuOKe49ZzQHNxpc+5UDvVo6ENEXItUtKi6vpHD1SeOBSdz3Nopw+P+a8Y6Tu6FZ6nMGHMar27c2264JC+Qyewvj279u1/qfijrQ0QSMnT2yxGHEQywdc7lnl/PKcPDTRZGY7CJP63eyaOvlrP3SB1jBvRi/9E69h+t6/RUQ2V9iEjSxCq277WOTGhaa1n88R7mLS5jy/7jTCwu4L+/MZFzhp6SlDZ6TYFaRBLipti+l+J9MLz76UHmLipl3Y5KRvTtwfybJjH9zH6YljHndKBALSIJSXWxfbcPhk92HWHuolLeKN9P//xc5l13NtdOHEhmRvoE6BAFahFJWCon3WI9GHYcquahJWX8ef0ueuUG+PFlo7n5vCHkRtmU1u9cBWpjTAHwJHAWzTni37HWvpvMhomIOIn0YDhwrI7Hl23mmZXbyMww/OOFw/jHC4eRnxc5lzqduO1RPwYsstZeZ4zJBrolsU0iIhFFKgh18Zn9ePLNLfxyxRZqG5u4vmQQd148gn69kld2NNViBmpjTD5wAXALgLW2HqhPbrNERE4UKX961p/WkxPI5FhdI18+6zR+OGMUwwp7dHJLveemRz0U2A/8yhgzDlgN3GGtPd72IGPMTGAmQHFxsdftFJFO1tnlTSMVhGposphgEwu/P5nxgwpS1pZUc7OEPAuYCPyPtXYCcByYHX6QtXa+tbbEWltSWFjocTNFpDOFerOpXCbe9tpOi1wA6hubunSQBnc96p3ATmvtypa/P0+EQC0iXZdTedN/f+njqLuqJNoDj1T/OpxTzY6uJGagttbuMcbsMMaMstaWAdOAT5LfNBHxi2j1nkM1n8N3VenI1lfh/uuvG6MG6VRXsessbrM+bgOeacn42AJ8O3lNEhG/cVvvOVSMKfRzpNfcBOq9R2p59NVN7Dta53hMkU+2AUsFV4HaWrsOiFgsRES6vkirAZ1Eq7kRa4OBqpoGnnjjU55+eyvBJkv37EyO17e/ptfbYPmdViaKSExuy5vC5zU3YtXjaDuG3T8/l5Ihp/BG+X6O1DZw5bgB/GD6KNZsP5zSOiJ+pUAtIq6ErwZ02ug2FESjvRb+3l1Vtby0fhejT+vJ77/7BcYMyG8N5DUNQTKNIWjtSTXc0ZYCtchJLJHMDDfFmJxem7eoNOIwytHaxtYg3TaQB61tDfQnW5AGbRwgctJy6hEne5up97ce4vonnEsFFUWZuIw0Nt3ZC3G8oo0DRKQdp9zoZG2hVbrnCA8sKuO10n2OO3wbIo9th4RPRjptywXxpQH6nQK1yEmqIzuldMTOw9U8vLScF9dW0CMni3svHc2p3bO576WPT3hQGHDcGTwkfHOAVD9sOosCtUgX5zQ04JQbbWnelzDRIYRDx+v52fLN/O+728DAzC+dzvcuGkZBt2wAsrMyWtuVnxdoXTjjJFK2R6oeNp1NgVqkC4s2NBAtNzqRIYTq+kaeenMr81ds4Xh9I9dNGsgZ/XvxZMvv2j4srppQ1NrGaJyyPVK9X2NnUaAW6YJCvehIQSw0NBCalIt1nNtA3RBs4o8f7OCnr21i/9E6LjmzH7NmjOLjXUeijiNHGr4IiTW56bQt15TRhUyesyztJxhDFKhFPOCnzAM3hYxCQwOhXu3Q2S9HHB92M4TQ1GR5ecNuHlpSxmcHqzlnyCn84sZJTBrcG4BbfvVB1HHkaNeIlYESKUVwyuhCXlhd0aUmGBWoRRLkt8yDaD3UkPChgY4OIby16QBzF5WyoaKK0af15OlbSqiqbuD2P6xtDZxOWRyhAO10TFFBnqt/v/CFOJPnLOtyE4xu6lGLSBTRMg+SJVSjeejsl5k8Z9kJdaFj9YIjTcrNmjGKvLDNX6Mt1d6ws4obn1zJjU+t5NDxeh6+fhwv3/4ljtQ08uMXPzqhbrXTnt+hh0C8147FzQRjtH8/P1KPWiRBqc48iNWDj9aLzTTmhIdIqIfpZpUhwGcHjvPgkjL+8uFuencL8K9fOZMbzy0mJyuz9f3hDy1L5NS76vpGFq6tcH1tt2J9O/DbNyA3FKhFEpTqzINYucORJtgCGQYMNASbw2Wk4BRpZ++QfUdr+elrm/jj+zsIZGZw29ThfPeC0+mVe+IO304PJwsUhKXgHa5uOKENXgVJpwnGUA89HXOvFahFEhQrMHgtVg/ebaU7p+DUdmL0tPxcxhbl8+amAzQEm/jGOcXcNm04fXtG3uE72ngz0C5XOhkBMlYPPR1zrxWoRRLk9Vf3WNz04MN7qENnvxzxXLGWZO+uqmV3VS0TBhXwyA3jGdKne9S2RXto3fXsOldt8EK0Hno65l5rMlHEA1dNKOLt2VPZOudy3p49NalfoTsy+eYUhMJ/71TVbt/RuphBGpr/He6/ZixFBXkYmnvSoRQ7t21INq8nL1NBPWqRNNORHnys4RlrLcvL9rGrqjbi++Pp9Tr1ZlM9ROQk1d+AvKBALZKG4p18ixacVm87zNxXSnn/s0NkZhiCEcraedHr9VOA9HLyMhVUj1rEp5K92nHT3qPMW1zG0k/2UtgzhzumjSAvkMm/LPwo5TWqRfWoRdJOMnN9d1XW8Oir5Ty/eifds7P44SUj+c75Q+mW3RwOMjOML3q98jlXgdoY8xlwFAgCjU5RX0S8kYxc38rqen7++qf8+p3PwMJ3Jg/l1inDOaV79gnHpduwwMkgnh71FGvtgaS1RERaeZnrW1Mf5FfvbOV/Xv+UY3WNXDNhIHdNH8HA3t0SbaakiIY+RHwgfDy6oFug3QIViG9SrzHYxHOrdvLYa+XsPVLHtNF9mXXpKEaf1svLpksKuA3UFlhijLHAE9ba+eEHGGNmAjMBiouLvWuhSBcXaTw6kGEIZJrWJd/gPpXNWsuij/bwwJIytuw/zqTBvXn8mxP5myGnJO0eJLncBurzrbUVxpi+wFJjTKm1dkXbA1qC93xozvrwuJ0iXUZ477m6vrHdeHRDk6UgL0D3nKy4JvXe+fQAcxeVsX5HJSP69uCXN5dw8Rl9Mcaphp2kA1eB2lpb0fK/+4wxLwLnACuiv0tEwkXqPTupqmlg3X2XuDrvx7uqmLeojDfK9zMgP5d5153NtRMHkpmhAN0VxAzUxpjuQIa19mjLz5cAP0l6y0S6IDdF/UPcjEdvP1jNQ0vL+PO6XeTnBfjny87gpvMGkxu2RFrSm5sedT/gxZavTlnA7621i5LaKpEuym3WRqzx6APH6nh82WaeWbmNzAzDrRcN4x8uHEZ+XsDxPZK+YgZqa+0WYFwK2iLS5TlVbnM7Hn2srpFfrtjCk29uobaxietLBnHnxSPo1yty2VHpGpSeJ2nJT5vJxsOpMNG/XzHGsf0L11Ywb1Epu6pqT9gppW/PHL4w9BQF6ZOAArWknVRupRTrgRDvAyPewkQL1uzk3hc+bE3Ta5tOte9oXbv7TtcHmESnokySdibPWea4i8jbs6d6dp3wBwKcWKAo1uuJsNbyRvl+vvvbVSfkUkcSuu9ktkeSL1pRJm0cIGknVVspxdpdPFm7j6/bUck3fvket/zqg5hBGj6/787YDV1SQ4Fa0k6qdgqJ9UDw+oHx6f5jfO93q7nqZ2+zed8x/uOKMQzIjz3+HLrvdNwLUNxRoJa0k6qtlGI9ELx6YOypquVHCz7kkkdWsKJ8P3ddPJLXZ03hW18cwj2Xjm53r221vW+/bHUl3lOglrQTbV8+L8V6ICT6wKiqaWDuolIuenA5z6/eyU3nDuaNe6Zwx8Uj6JHTPM8ffq8FeQF6dwtEvO9kPMAWrq1g8pxlDJ39MpPnLGPh2ooOn0s6TpOJIlF4nfUBUNsQ5DfvfMbPX/+UI7UNXDW+iB9MH8mgUxIvO+pl1ocmJ1Mr2mSiArVIijQGm1iwpoJHXi1nd1UtF40q5J4ZozlzQGJlR5OVkpeq7Bpppq24RDqRtZYln+zlgcVlbN53jHGDCnj4+vGcN+zUdsfGG3STvWVXPL+X5FGgFgnjZQ/1/a2HmPPKRtZsr+T0wu784saJzBhzWsSyox0JusnYsivEabm7JidTT5OJklaSPbkVCpYVlTVYPg+W8V6ndM8RvvPrD7j+iXepqKzhhpJB1NYH+d7v1nD+3OURz9eRPOhk9npTlV0jsalHLWkjFUvHE+2h7jxczcNLy3lxbQU9c7K499LRnNo9m/te+jhmuzsSdJPZ6413ubskjwK1pI1kfs0P6WgP9dDxeh5ftpnfvbcNY2DmBadz64XDye8WYPKcZa7a3ZGg61Tkyater3Yk9wcFakkbqZjcijdYVtc38tSbW3lixRaq6xv52qRB3Dl9BP3zPz/eqX0VlTVMnrOstbc6ZXQhL6yuiCvoqtd7clCglrSRiskttz3UhmATf3x/O4+9tpkDx+q45Mx+3HPpKIb37em63YbPt+KqqKzhhdUVXDupiOWl++MKuur1dn0K1JI2kv01H6L3UNvWhc7MMASbLOcMPYUnbprEpMG942p327rSITUNQZaX7k9JjrLKoaYXBWpJG6n6mh+ph7pwbQX3PP8h9cEmAIJNzWG2fM8Rdhyqjhio2wbD/LwAuYEMKqsbHHvYkJoc5VTW8xZvKFBLWon1NT8ZPcUPd1Zy7wufB+m2KmsaIwa58GBYWdNAXiCTR24Yz1UTihxX/aUiRzkVk7LiLeVRiyf8ULzHqxzokK0HjvP936/hisffpq6xfZAOiZTrHCsnujNzlLXiMP247lEbYzKBVUCFtfYryWuSpBu/fJV2Co53P7c+rrbsO1rLT1/bxB/f30F2Vga3Tx3On1bvZHdVreN7woNcrGDYmdkaWnGYfuIZ+rgD2AgkVkFGuhy/fJV2Co5Ba109OI7UNjD/jS089dZWGoJNfPMLxdw2dQSFPXM4vbBHuwnBttoGuYVrK8gwhmCEgmdtj+usbI1UTMqKt1wFamPMQOBy4D+BHyS1RZJ2/PJVOtokXbQHR21DkN+9t42fLd/M4eoGvjpuAHdPH8mQPt1bjwm9799f+pjKmoYT3t82yIW+XUQK0n4Jhsq9Tj9ue9SPAvcA7ZNEWxhjZgIzAYqLixNvmaQNv3yVjtRTbCv8wRFssry4toJHlpZTUVnDl0b04d5LR3NWUX7E94d6wNEmLCN9uwDINMZXdZyVe51eYgZqY8xXgH3W2tXGmIucjrPWzgfmQ3M9as9aKL7nl6/SocBz93Prow47WGtZVrqPeYvKKNt7lLMH5jPvurOZPLyP6+s4BTmnbxFN1iowSoe56VFPBq4wxlwG5AK9jDG/s9bemNymSbrw01fp0DWdHhyrtx1iziulfPDZYYac2o2ffXMil42NXHa0I/zy7UK6lrh2eGnpUf8wVtaHdniRzhY+PHHjucWs3lbJqxv3UtgzhzumjeCGvxlEINPbDFVtXyUdpR1e5KQTGp7YVVnDI0vLeWBxGd2zs5g1YxTfnjyEbtnJ+U/fT98upOvQnonSaZJZb6Kyup6fv/4pv37nM7Bw83mD+f6U4fTunp30a4t0hHrU4jteLpJpG3RPy89lYnFvVmzaz7G6Rq6dOJC7po+kKCzPOdK1V207FHflOpFUUI9aOoVTrQto3uXabZCMNCYMMGZALx6+fjyjTut5wrEPLC5zvG54RTuNLUsqqUctvhNtMUw8vet5i0oj5i1XVje0BumFaysiLlQJF6nsqAoViR+oKJN0iljparE2dQV459MD7HKovxF6EIR63LGCtBMVKhI/UI9aXPNyAi7WKkJwDpIfVVQxb3EZK8r3kxmjpobTSsFwkQr5tz2PSGdSoBZXvK6Q1zaNzWnMODxIbj9YzYNLynhp/S4KugX458vOoKBbgH/788eOqyLd9IiLOrhfoUiqKFCLK8mokNe2dka0Jej7j9bx+LJN/P797WRmGL4/ZRgzLxhGfl4AgEBmhmNPP1qhpvDJwpLBpyhlT3xJgVpcSWaFPKdFIhef2Y+Hl5bz5JtbqGts4oa/GcQd00bQr1duu/c7BVSnIZbe3QLc99UxJ7xPhYrErxSou4BULN5Idg2LtkGyrjHI71du58J5yzl4vJ7Lx/bn7ktGcnphjw6dF7RSUNKbAnWaS9XuKqmokNfUZHlp/S4eXFLGzsM1fHHYqdx76WjGDSpI6LzqKUu6U6BOc6naXSWZPVNrLa+X72feojI27j7Cmf178dvvjOVLI/p4VtVOJJ0pUKe5VO6ukoye6drth5nzSikrtx6i+JRu3HTuYF7buJdvPf0++XkBjIHD1Q2taXjxrFoU6SoUqNNcutQ/Dh9Hv/m8wazdXsmij/fQp0c2P7lyDN0Cmfxrm1S7totUQrnSnbVxrkhn0srENDdrxijyApkn/M5v+b+hcfSKyhoszcH2/ldKWV62j7suHskbs6Zw83lDeOTVTa4Wp7hZtSjSlShQp7mrJhRx/zVjKSrIw9C8eMNvhYScVgfWNTbx3KodLP1kLxDfcI2WdsvJREMfXYCfsxpqG4KOC07gxKGMaItTwvltaEckmRSoxXML11Ywb1Epu6pqyXCRtBEaynBT/wP8N7QjkmwK1OKpF9fs5N4XNlAfbAKgyUKmgYwMQ0PQufb5rsqadimAyvoQaaZALZ5ZueXgCUE6JGihZ3YW3XOyYhZgSvUwjrbkknSgQC0J27j7CPMWlbK8bL/jMVU1Day775KYBZhSKVWrOkUSFTNQG2NygRVATsvxz1tr70t2w+Rzfu317ThUzSNLy3lxXQU9c7KY/eXR/PadzyIW82/bYwZ/1N5I1apOkUS56VHXAVOttceMMQHgLWPMK9ba95LcNsGfvb6Dx+p4fPlmnnlvO8bAzAtO59YLh5PfLcBpvXJj9pj9kqWSylWdIomIGaht8+63x1r+Gmj54/2OuBKRl72+RHvmx+saeeqtrcxfsYXq+kauLxnEHRePoH/+56lyfuoxx5IuqzpFXO1CbozJBFYDw4GfWWvvjXDMTGAmQHFx8aRt27Z53NST09DZL0d8Khpg65zLXZ8n0thwaPupSJkUbYN6//xczht2Km+UH+DAsTpmjOnHrBmjGN63Z/sLpRGn8XK/LRiSk0PCu5Bba4PAeGNMAfCiMeYsa+1HYcfMB+YDlJSUqMftEa96fZF65qEPKXw4JTyA7aqq5YU1FQwr7M78myex/WA133r6gxNS6CqrG3zde44knXr/cnKLK+vDWltpjFkOXAp8FOv4riqVk3te1YGONe7adjjFacl3TX2Q7QerT2hP28JJfhg/j5dfxstFoolZ68MYU9jSk8YYkwdMB0qT3TC/ilRg6EcLNrBwbUVSrudVLQ83PfBdlTV8uLPSMdd5d1VtzF29VTBJxHsxx6iNMWcDvwEyaQ7sz1lrfxLtPSUlJXbVqlWeNdJPJs9ZFjGQFRXk8fbsqZ3QIncijceGywtkUtMQJMM0rygMF1odGEu84+cikuAYtbX2Q2CC561KU+mY0hUaqqlpCLYG29BEYltBa7l92ggG5OfyH//3Sbug7vS+cKHeu1/zv0XSjVYmxindUrrCe9JBa8kLZPLVcf3564Y9HKtrBOD84X145IbxFPbMASA3kMndz61v14O2EDVYh8bP/Zj/LZKuVI86Tn4q1L9wbQWT5yxj6OyXmTxnWcRxcqc87D+t3smxukauGDeAN2ZdxO/+/gutQRqag2mTwzBHKKXPAAV5AXp3C7QbP4+W/y0i8VGPOk5+Sely22N1GpKxFv5y2/mcVZTveA2nbw9uxuPTcYhIxK8UqDvADyldblcs9s/PjVh7o6ggL2qQhsRSA9NtiEjEzzT0kWRuhic6wk2PddVnh8gJG6YB98E2kdRAPw0RiaQ79aiTKJkTatF6rOV7jzJvURmvbtxL3545fG3SQN7efIDdVbVxD9V09NuDX4aIRLoCV7U+4tWV86jj4ZRznWkMD10/LqGgFSkvOicrg3EDC1i17RDds7P4x4uG8e3JQ+iWreexiN8lXOtDOsZpeCJobcI967Y91orKGnrkZFHbEGTdzkr+7vyh3HrRcHp3z+5Yw0XEVxSokyjartpeFKi/ZEw/Kipr+MXrn3K8vpFrJg7krukjKerghJ0WqIj4kwJ1EsXaVbujqWoNwSaeW7WDx17dxL6jdVx8Rj/uuXQUI/t1vOyoFqiI+JcCdRKFAlykFX4Qf6qatZa/btjDg0vK2HrgOCWDe/Pzv51IyZBTEm6rtqUS8S8F6iQLBblES5W+s/kAcxeVsn5nFSP79eDJm0uYdkZfjDERj493GEMLVET8S4E6BdykqjkF1o8qqpi7qJQ3Nx1gQH4uD35tHFdPKCIzI3KADp0r3mEMLVAR8S+l5/mAU6rdmAG9WLO9koJuAf5pynBuPHcwuREWsITrSClWbUsl0rmUnudzkcaH6xqbWLu9ku9PGcY/XDiMXrkBV+dauLbCMdMk2jCGFqiI+JcCtQ84Fk4CZs0Y7fo8oV6xk1jDGH6oYSIi7anWRyerawzSKy9ybznefOho22SpzoZI+lKPOkEdXSTS1GT58/oKHlpSTlVNQ7vtr/ICmUwZXcjkOcsSztwANNYsksYUqBPQkewKay2vl+1n7qJSSvccZcyAXvzX1WM5eKyOB5eUtwblKaMLeWF1hSeZG0UFeQrSImmsywbqRJdDu3l/vItE1mw/zNxXSlm59RCDT+3GzecN5tVP9vKtp99vd43Jc5bFvQAlkfrRIuJfMQO1MWYQ8FugH83zW/OttY8lu2GJSHQ5dKK7p4T/fvO+YzywuJTFH++lT49sfnLlGLoFMvnXP3/seI2OLEDpSOaG6nuI+J+bHnUjcLe1do0xpiew2hiz1Fr7SZLb1mGJLod2+/5Yi0T2VNXy6KvlPLdqB92ys/jB9JH83flD6Z6TFbPH3NEFKPFkbqi+h0h6iJn1Ya3dba1d0/LzUWAj4Ov/Fye6HNrt+512Mfn+lGHc/8pGLnxgOS+s2cm3vjiEN2ZdxO3TRtA9J8vVNVKxQ4o2oBVJD3GNURtjhgATgJURXpsJzAQoLi72oGkdl+hyaLfvDx9q6J+fy6TBvZnzSilH6xq5enwRd00fyaBTusV9jVQsQFF9D5H04DpQG2N6AC8Ad1prj4S/bq2dD8yH5iXknrXQhfBx1vCMCYivNxrPpNxVE4r4ytn9eX71Th59dRP/9+Fupowq5J5LR3NG/14JXSPZC1BU30MkPbha8GKMCdAcpJ+x1i5IbpPiExpnraiswdI8zvrC6gqunVTUoU1Zwf2mrtZaFn20hxmPrmD2gg0cPF4HQPneY5TtOerJNZJJG9CKpIeYRZlMcx3N3wCHrLV3ujlpKosydaQAUTycsiLe23KQuYtKWbu9kr49c6isbjgHR4kAAAn7SURBVKA+2NT6vnQpaKSsDxF/iFaUyU2gPh94E9gAhCLRj621f3V6TzIDdXhgcSpAZICtcy6P+3xtA1WkinJt5ecF+PFlo3ns1U3sqqpt97pXDwsR6foSqp5nrX2L5rjX6SKlkxmak7vDuRlnjZWeFq12BkBdQ5CcrEx2RwjSoEk5EfFGWhVlihQ4Le2fIm7HWWOlp8UKtLWNTTywuMzxoaBJORHxQloF6mjlQEOTcgV5AXIDGdz17Domz1nGwrUVcZ9vV2UNx+sa6ZEbOylmV2WNJuVEJKnSKlA79VBDY8GP3DCeusYmDlc3tGaA/GjBBsdg7XS+XnkBLnxgOUdrG4my41XrOfyQwSEiXVdaBepYPdd4V9pFOp8BqmoaGFbYgwW3fpGHrx/fWhc62hDLVROKeHv2VLbOuZy3Z09VkBYRz6RV9bxYq/XiXWl31YQirLX8v5c3cvB4PQD983P5z2vGctHIQowxTCzufUIWSGensvmhDSKSWmkVqCH6ar14V9qt31HJn1bv5ODxegb2zuPuS0Zy5bgiMhzGOzp7qyoVURI5OaVdoI7G7dLvLfuP8dCScl7esJtTumdz31fP5JtfKCYnK/YO317oaK840aqAIpKeulSgjjU0su9ILY++tolnP9hBTlYGt08bwXe/NJSeLnf49kIivWIVURI5OaV1oHbqmYYHvCO1DTzxxqc89dZWgk2WG79QzD9NHUFhzxxX5/OqXZBYr1hFlEROTmkbqN30TGsbgvzvu9v42eubqaxu4IpxA7j7kpEMPrV7h87nRbti9YqjBXlttSVyckrbQB2tZ/rVcQNYsGYnjywtZ1dVLReMLOSeGaM4qyi/Q+eLJ1DHOk+0XnGsIJ+KGtUi4j++C9Ruhx+ceqYVlTV8+bEVlO89xriB+Tz4tXF8cXifmNf1avzXzc4tTr1iNw+Lzs48EZHU81WgdupRrtp2iOWl+08I3tEq5zUGLT//24l8+azTaK7SGptX47+J7Nxy17PrIp5Tk4UiJzdfBWqnHuUz721vrZAXCt7XTipqt4sLwPUlA/nPq8cSyIxv0aVX47+J7NyiyUIRicRXS8ijFV1qq6YhyNJP9jK2zZhzr9ws5l17NvOuGxd3kAbvdlxJ5Dwq7iQikcTcOKAjOrpxgNNuLU6yszK45YtD+N6Fw+jdPTvu6/mRloiLnJwS2jgglSINGzhtDNAtO5OlP7iwtWBSV6HJQhEJ56uhj0jDBt84ZxCBsNobOVkZ/NfVY7tckBYRicRXPWr4vEdpreXlDbt5aEk5DU2W7MwM6oNNFGk4QEROMr4L1ABvbz7A3EWlfLizipH9evDkzSVMO6Ov61Q7EZGuJGagNsY8DXwF2GetPSuZjfmoooq5i0p5c9MBigryePBr47h6QhGZsbZZcUkTdSKSjtz0qH8NPA78NpkNqapp4Gu/eJecQAb/cvkZ3HjuYBZ9tIcL5i33JLCqlrOIpKuYgdpau8IYMyTZDcnPC/DETZMYX1xAr9yA54FVtZxFJF35KuvjgpGF9GqpDR3v/oexqJaziKQrzwK1MWamMWaVMWbV/v37Ez6f14HVaRm2lmeLiN95FqittfOttSXW2pLCwsKEz+d1YNXybBFJV74a+mjLy8AayvaoaQiS2ZLiV5AXIDeQwV3PrmPynGUsXFvhSbtFRLwWM1AbY/4AvAuMMsbsNMb8XfKb5V2RpNCkZKiGSNBaAhmG4/WNHK5uwPL5RKWCtYj4ka+KMiVDPIWeigryeHv21CS3SESkvWhFmXw79OGVeCYflQEiIn7U5QN1PJOPygARET/q8oE60qRkIMMQyDxxWboyQETEr3xZlMlLTnsURvqdViiKiB91+clEEZF0kBY7vKiynYhIZL4I1KpsJyLizBeTiV4XYBIR6Up8EahV2U5ExJkvArUq24mIOPNFoFZlOxERZ76YTHTKddZEooiITwI1NAdrBWYRkfZ8MfQhIiLOFKhFRHxOgVpExOcUqEVEfE6BWkTE55JSPc8Ysx/Y1sG39wEOeNicztRV7qWr3AfoXvyoq9wHJHYvg621hZFeSEqgToQxZpVTqb9001XupavcB+he/Kir3Ack71409CEi4nMK1CIiPufHQD2/sxvgoa5yL13lPkD34kdd5T4gSffiuzFqERE5kR971CIi0oYCtYiIz3VaoDbGXGqMKTPGbDbGzI7weo4x5tmW11caY4akvpWxubiPW4wx+40x61r+/H1ntDMWY8zTxph9xpiPHF43xpifttznh8aYialuo1su7uUiY0xVm8/k31LdRreMMYOMMcuNMZ8YYz42xtwR4RjffzYu7yMtPhdjTK4x5n1jzPqWe/mPCMd4G7+stSn/A2QCnwKnA9nAeuDMsGNuBX7R8vPXgWc7o60e3MctwOOd3VYX93IBMBH4yOH1y4BXAAOcC6zs7DYncC8XAX/p7Ha6vJf+wMSWn3sC5RH+G/P9Z+PyPtLic2n5d+7R8nMAWAmcG3aMp/Grs3rU5wCbrbVbrLX1wB+BK8OOuRL4TcvPzwPTjDEmhW10w819pAVr7QrgUJRDrgR+a5u9BxQYY/qnpnXxcXEvacNau9tau6bl56PARiC8cLvvPxuX95EWWv6dj7X8NdDyJzwrw9P41VmBugjY0ebvO2n/obUeY61tBKqAU1PSOvfc3AfAtS1fSZ83xgxKTdM85/Ze08V5LV9dXzHGjOnsxrjR8vV5As09uLbS6rOJch+QJp+LMSbTGLMO2AcstdY6fiZexC9NJibf/wFDrLVnA0v5/CkrnWcNzXUVxgH/DSzs5PbEZIzpAbwA3GmtPdLZ7emoGPeRNp+LtTZorR0PDATOMcaclczrdVagrgDa9iwHtvwu4jHGmCwgHziYkta5F/M+rLUHrbV1LX99EpiUorZ5zc1nlhastUdCX12ttX8FAsaYPp3cLEfGmADNwe0Za+2CCIekxWcT6z7S7XMBsNZWAsuBS8Ne8jR+dVag/gAYYYwZaozJpnmw/aWwY14CvtXy83XAMtsyMu8jMe8jbKzwCprH5tLRS8DNLRkG5wJV1trdnd2ojjDGnBYaLzTGnEPz/w/81gkAmjM6gKeAjdbahx0O8/1n4+Y+0uVzMcYUGmMKWn7OA6YDpWGHeRq/OmVzW2ttozHmn4DFNGdOPG2t/dgY8xNglbX2JZo/1P81xmymeWLo653R1mhc3sftxpgrgEaa7+OWTmtwFMaYP9A8697HGLMTuI/mSRKstb8A/kpzdsFmoBr4due0NDYX93Id8D1jTCNQA3zdh52AkMnATcCGljFRgB8DxZBWn42b+0iXz6U/8BtjTCbND5PnrLV/SWb80hJyERGf02SiiIjPKVCLiPicArWIiM8pUIuI+JwCtYiIzylQi4j4nAK1iIjP/X8W5zmilIONHQAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0giuwC9GHzi8"
      },
      "source": [
        "## Вычислиения на GPU\r\n",
        "\r\n",
        "Для проведения вычислений на GPU PyTorch поддерживает перемещение тензоров на GPU и автоматическое построение вычислетельного графа там. Традиционный способ вычислений состоит в том, что вначале мы определяем доступное вычислительное устройство `device` (CPU или GPU), и затем перемещаем туда все необходимые тензоры по мере необходимости с помощью вызова `.to(device)`. Мы также можем заранее создавать тензоры на нужно устройстве, указывая параметр `device=...`. Такой код работает без изменений как на GPU, так и на CPU: "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HK7HPLz3Hyrl",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "7e14cccb-d376-4e59-be66-4ab3f5c3f6f4"
      },
      "source": [
        "device = 'cuda' if torch.cuda.is_available() else 'cpu'\r\n",
        "\r\n",
        "print('Doing computations on '+device)\r\n",
        "\r\n",
        "### Changes here: indicate device\r\n",
        "w = torch.tensor([100.0],requires_grad=True,dtype=torch.float32,device=device)\r\n",
        "b = torch.zeros(size=(output_dim,),requires_grad=True,device=device)\r\n",
        "\r\n",
        "def f(x):\r\n",
        "  return torch.matmul(x,w) + b\r\n",
        "\r\n",
        "def compute_loss(labels, predictions):\r\n",
        "  return torch.mean(torch.square(labels - predictions))\r\n",
        "\r\n",
        "def train_on_batch(x, y):\r\n",
        "  predictions = f(x)\r\n",
        "  loss = compute_loss(y, predictions)\r\n",
        "  loss.backward()\r\n",
        "  w.data.sub_(learning_rate * w.grad)\r\n",
        "  b.data.sub_(learning_rate * b.grad)\r\n",
        "  w.grad.zero_()\r\n",
        "  b.grad.zero_()\r\n",
        "  return loss\r\n",
        "\r\n",
        "batch_size = 4\r\n",
        "for epoch in range(10):\r\n",
        "  for i in range(0,len(features),batch_size):\r\n",
        "    ### Changes here: move data to required device\r\n",
        "    loss = train_on_batch(features[i:i+batch_size].view(-1,1).to(device),labels[i:i+batch_size].to(device))\r\n",
        "  print('Epoch %d: last batch loss = %.4f' % (epoch, float(loss)))"
      ],
      "execution_count": 195,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Doing computations on cuda\n",
            "Epoch 0: last batch loss = 94.5247\n",
            "Epoch 1: last batch loss = 9.3428\n",
            "Epoch 2: last batch loss = 1.4166\n",
            "Epoch 3: last batch loss = 0.5224\n",
            "Epoch 4: last batch loss = 0.3807\n",
            "Epoch 5: last batch loss = 0.3495\n",
            "Epoch 6: last batch loss = 0.3413\n",
            "Epoch 7: last batch loss = 0.3390\n",
            "Epoch 8: last batch loss = 0.3384\n",
            "Epoch 9: last batch loss = 0.3382\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "A10prCPowHl7"
      },
      "source": [
        "## Пример 2: Задача классификации\n",
        "\n",
        "Рассмотрим пример двухмерной задачи классификации на 2 класса. Примером такой задачи может быть классификация опухоли на 2 типа - доброкачественная и злокачественная, в зависимости от её размера и возраста.\n",
        "\n",
        "Сгенерируем тестовые данные случайным образом:\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "scrolled": false,
        "trusted": true,
        "id": "j0OTPkGpwHl7"
      },
      "source": [
        "np.random.seed(0) # pick the seed for reproducability - change it to explore the effects of random variations\n",
        "\n",
        "n = 100\n",
        "X, Y = make_classification(n_samples = n, n_features=2,\n",
        "                           n_redundant=0, n_informative=2, flip_y=0.2,class_sep=1)\n",
        "X = X.astype(np.float32)\n",
        "Y = Y.astype(np.int32)\n",
        "\n",
        "split = [ 70*n//100, (15+70)*n//100 ]\n",
        "train_x, valid_x, test_x = np.split(X, split)\n",
        "train_labels, valid_labels, test_labels = np.split(Y, split)"
      ],
      "execution_count": 196,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "scrolled": false,
        "trusted": true,
        "id": "c-_BjSHPwHl8"
      },
      "source": [
        "def plot_dataset(features, labels, W=None, b=None):\n",
        "    # prepare the plot\n",
        "    fig, ax = plt.subplots(1, 1)\n",
        "    ax.set_xlabel('$x_i[0]$ -- (feature 1)')\n",
        "    ax.set_ylabel('$x_i[1]$ -- (feature 2)')\n",
        "    colors = ['r' if l else 'b' for l in labels]\n",
        "    ax.scatter(features[:, 0], features[:, 1], marker='o', c=colors, s=100, alpha = 0.5)\n",
        "    if W is not None:\n",
        "        min_x = min(features[:,0])\n",
        "        max_x = max(features[:,1])\n",
        "        min_y = min(features[:,1])*(1-.1)\n",
        "        max_y = max(features[:,1])*(1+.1)\n",
        "        cx = np.array([min_x,max_x],dtype=np.float32)\n",
        "        cy = (0.5-W[0]*cx-b)/W[1]\n",
        "        ax.plot(cx,cy,'g')\n",
        "        ax.set_ylim(min_y,max_y)\n",
        "    fig.show()"
      ],
      "execution_count": 197,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "scrolled": false,
        "trusted": true,
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 283
        },
        "id": "tq0vFchQwHl8",
        "outputId": "919f1922-f789-4779-cbdc-4f9e742c358b"
      },
      "source": [
        "plot_dataset(train_x, train_labels)"
      ],
      "execution_count": 198,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYQAAAEKCAYAAAASByJ7AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO2dd5iU5dX/v2dmtrINdpfOsoBKV9AFsQRFNIgx1thiNJZojDH6mkSNKSbRn0l8TaKmGDUxRo2vFbEiYgsIEaWIIFUEgaXsLmzvOzP374/vPM6WqbvT93yua67dnXn2mTPzzJxz36eKMQaKoiiKYou3AIqiKEpioAZBURRFAaAGQVEURfGgBkFRFEUBoAZBURRF8aAGQVEURQEAOOItQF8oKioypaWl8RZDURQlqVizZs1BY0xx9/uT2iCUlpZi9erV8RZDURQlqRCRXb7uV5eRoiiKAkANgqIoqYgxQHs70NERb0mSiqR2GSmK0gecTmDjRuDNN4GdOwG7HZg+HZgzBxgzBhCJt4Th09YGrFoFLFoE7NvH+w47DDjjDOCoo/gaFb+oQVCiijFAYyN/5uQANt2TJgatrcCDDwKffALk5gKFhYDbDXz4IfD++8BZZwHnnptcRqGpCbj/fmDrVqCoCBg9mvdXVgL33QccfzxwzTWAQ9WeP/SdUaJCRwfwwQdcqFVUUK/k5wPz5wNf+QqQlRVvCfs5TzwBbNgAlJZ2VfrDh3PnsHAhlers2XETMWwefxz4/POeu5tBg4CBA4GVK4Fhw4BzzomfjAlOwqzXRGSUiLwnIptEZKOI3BRvmZTe0d4O/PnPwN//zt9LSnhzOIB//xv43e+AhoZ4S9mPqayktR41yvcOwOEAhg4FXnoJcLliL19vqKqiq2jkSN+vSYTGbvFi7o4UnySMQQDgBPAjY8wkALMAfF9EJsVZJqUXLFwIrF8PjB0L5OV5v58DBvC+8nLgscfiK2O/5uOP+TOQ/27AAKCmhrGFZGDDBvolA72mjAyuUD77LHZyJRkJ4zIyxuwHsN/ze4OIbAYwAsCmuAqmhEVTE/D228CIEf7dzyNHUidVVABDhsRWPgVAbS2Qlhb8OJuNFzRW1NRwlb9vH5CZCUydCkyYEFoguKkp9ICx7hD8kjAGoTMiUgpgOoAP4yuJEi7bt9MFHUjfiHAxt3EjMHgwvRJ2e3LFL5OavLzQ0jGNiU2wx+Wie+r11/mcmZn8EC1ezA/ID35A91YgCgpCc29Z2Q2KTxLOIIhIDoAFAP7HGFPv4/FrAVwLACUlJTGWTglGW1toit3lApYtA155hQtWhwM49ljg1FMZE1SiyLRpwLPPUjn6u1gtLcw+Gjs2+vIsWAC8+iqzgrpnAB06xKDTHXcE3k5OncodjbW68IX1mg4/PHKypxiJFEOAiKSBxuApY8yLvo4xxjxijCkzxpQVF/doxaHEmbw86plANDQAa9YAH30EpKcz0WXoUHoLfv1r4N13YyJq/2XYMODooxnM8XWx3G66bc48M/opmlVVwBtv8EPg67kKC7nKeP11733G0O3TeUdQUADMnQvs2uX7NTmdfE3nnqtppwFImHdGRATAowA2G2P+GG95lN5x2GFML21uBrKzez7ucgHLl/P3adNoEAB+R4cPZ8zvX//i7xMmxEzs/sfVVzNnf9s2pmValvzgQRaOfPWrVLDR5oMPuLIP5P8fOhRYsQI47TQGn5YsYcxABJg5k/ePGwdceCFXGx98wA9fQQFf06FDdJGdcw5w8snRf01JjJhgy7kYISInAngfwAYAbs/dPzXGLPL3P2VlZUab2yUe778PPPIIU027xxLKy4H33gPKyoCJE33/f1UVPRU/+lH0Ze3XtLcDq1dzhV5eTgU7dSqNwaRJsQnqPPAAA0+FhYGP27yZsQW7nXGFzEyuLioruYO47DIaMLebRu6tt1iTIAIceSSrr0eP1kCVBxFZY4wp635/wuwQjDHLAejVSgFOPBGorwdeeIGLv4ED+T2srfWmo44f7///CwsZcG5ooMtXiRLp6azePf54KlKR2CtMh4PPHQinE/j0U+4GOscN7Xa6v9rbWZQ2ciQ/WBMm6PaylySMQVBSBxHga1+jm3r5choBY2goLAUfKF3cZuM5rBigEgPi1VNk+nQGjwLFA3ftotHwl2mUns7MoUWLAq80lKCoQVCixrBhwAUX8Gbxl78AmzbRZe0Pt5sGxFcMQkkxpk/nhW5s9J0Oagw/MIcfHthoFRVx5eHvPEpIJFSWkZL6nHQSA86BqKwEjjlGv9f9gqws4PrrGfg9eLBrhlBbG/DFF/Q5jhsX+DzWtrKtLaripjpqEJSYMnEid/779/t+vKWFGYVnnBFbuZQ4MmUK8POf84OxaxewZw+wezeDTmefzQ9De3vgc7hcNAi6rewT6jJSYorDAdx8M/CHP3DxV1DAtjkdHVwg2mzADTdocVrCcegQW0s4HOxLEkrri3AYNw647Tb2M6mu5vOUlLD/0Cef0B0UKM5QWQnMmqVtdPuIGgQl5gwaBPziF8C6dUwpr6hgFuGZZzLwPHhwvCVUvmTXLuDFF6mQbTZvcOf004F58yJvGIYM6VmRPHmyd1s5bFjP/2lq4opi3rzIytIPSZg6hN6gdQiKEkW2bgXuvZdKv7jYG9RtaaFynjaNfYYibRR8UV3NbWV5eddt5aFD3E18//uciKaERMLXISiKkkC0tXGoRV5ez5SwrCz69NatA955h7uFaDNoEPsZffIJi84qK7mtPO881lEMGhT6uZqaaNDcbm5HCwqiJ3eSoQZBiQvxqoNSQmTdOirOoiLfj4uwpcQbb7AjYSz6A2VksDht5sze/X9DA/Dyy8DSpV37IM2cyR5H2otdDYISO+rr2ZLmrbfoAcjMZMxgzhzGKZUE4uOPgwdos7OZCXDgAKuEE5n6enZN3b+fjbIsN5fLxU6Ln34K/PSnfKwfo2mnSkzYu5eB5Oee42Jy9Gi2qFi6lPevXBlvCZUutLeHNnBGJDnGbD7zDLMXRo/uGvOw27kaMQZ46KHgrXpTHDUIStRpbWU80OXi9zE7m3okLY3fxcGDgYcfZhqqkiCUlASfluZyUYEOHBgbmXpLbS1XHIFW/0VFrH9IlpGhUUINghJ11q6li8ifOzozk+1oliyJrVxh4XJxbu8//gHcdx/w5JPsppmqK8rjjuNrC7T6r6hg7n+gPiSJgLXSCLTjsYJZ27ZFXZxERmMIStRZtowzEgIxeDDw4YfAt7/N2GFCUVHBNs3WrN/0dPbXeecd5sh/73up12djyBDOGVi8mNu67sq0poZK9Otfj4984RCqS8tmC220aAqjBkGJOrW1wZW83c4FaUtLghmE+nrgnnvo9yot7fqYMezT/6c/AbfemnqTuC66iEryzTep/LOyqFxbWxkAuv1234ViiUZhobdjYqC0Nrc7OV5PFEmxT7CSiAwcyKByoDYzTqdX5yQUy5ZxNTx6dM/HRFhBu3UrdwxHHhl7+aKJ3Q5cfDHTSleu5EXMyGCH0smTk8cAjh7N61Rb6z/e0drK1zZ1amxlSzCS5IoqycycOcCDD/qvHWpsZNq7zcaF9ogRHNo1ZUqcdY7bzcBGoF4aInQXvfVW6hkEi6Ii9hVJVkSAb30L+O1vmcnQ3b3X1kZj953vJNj2NPZoUFmJOkcdRZ1aWdnzsV276JHYvZuT1LKzmexx333A//4vjUXcaG1lMVOwbUtuLuMLSuIyfjzwwx/SJ7lrF2snKioYcD54ELjySmD27HhLGXd0h6BEnYwMfhfvvZffv8JCKv59+zh/ecAA7iKGDvUeP2gQR+0+9BBnK8elotnanrjdgYezuFwMNCuJzZFHAn/8I9tfbNrE6zp2LAd8p1pSQC9Rg6DEhKFDgTvv5LTEt97i4mzzZg7CmjKFRqEzlnv+009pROLSDjs9ncLt2BG49fKhQ+zb3wusrFVt4REj+tr+IsVRg6DEjAEDgJNP5q2qivGCkhL/ytAqXvvgg14ahMZGYONGZgoNGABMmhR+I7PTT2eW0cCBvgMaLS3cPZxwQsindDpZ0vDGG9wFiXBw0Lx5jNXGa7yxoqhBUOJCYyOTWIKtjDMz6eINC6eTPfyXLOHvFjYbZ3hefHHowcNJk7j6f+kl+rry8ym0MRSssZF1CP6q7rrR1gb87W8s1svP97YA2rmTLrXZs4ErrkieBB4ltdCPnRIXMjO9nQ8CGYX29jALYd1u4LHHGJwoKemqWV0u4L336OK58cbQtK4IWyyXlACvvMLot83G57GMxRFHhCzes88yo2rMmK6vu7iY9mbZMv4899wwXrOiRAg1CEpcGDqUrWXq6wNXMbe3A8ceG8aJt24Fli9nEVl334vdzpz0devYzXPGjNDOKcJjy8qYKtXWxiBkOD34wTT4//yHsRFfRtBm445h8WJg/nwaTUWJJeqtVOKCCHDWWYwldPbqdMbqqjx+fBgnfvttpjD5c8SLMB6waFHYMkOELR1KSsI2BoA3sSVQS530dBrBLVvCF09R+ooaBCVuzJxJb8zu3Vx4u928v6mJmUW5ucBNN4UZZN26NXj3zYICOu1j3La5oSH0bKKWlujKoii+SCiXkYj8E8CZACqNMVPiLY8SXUToK584kcVpn3zC+wsKgG9+kxk3tbVAXR3dSyG1tQhV48YhzzMvL/TmqIHafChKtEgogwDgXwD+AuCJOMuhxAgr5XLiRC7YnU4agZdf5kwTK6HH4QBOOYUdFALWEE2ezBiBVeXmi+pqFkCEMgAmgkyezKd0Ov3Hs9vamAAVlptMUSJEQrmMjDHLAFTHWw4lPtjtTAC66y4WsI0YwQBsSQmzOhcvZklAQ0OAk8ydS3+LP3eQMdxyzJ8fldcQiLw8GrU9e3zvFNxuoLwc+NrXNKCsxIeEMghK/8YY5ugbQ2PQeQGfns7Eob17geefD3CSww5jH/8vvuByuzMdHYwdzJoVt0Z0F17I2MnOnd6AutPJyu1du2jPzjgjLqIpSsK5jIIiItcCuBYASkpK4iyNEkk+/5wr5ECXdfhwYMUK4Bvf8FOfIMIAxMCBwKuvdjUKaWlMbTr77Ji7izqLcN11LEB7803gs88o8qRJtGMTJmgbCyV+JJ1BMMY8AuARACgrK0vR+YX9EyvVMpBCdDi4g/jiiwCLfJuNfpdTT+VJGxsZpR0/PiGitXY72+7389b7SgKSdAZBSV06OkJfuPurXehCRgZ7byuKEhIJFUMQkacBfABgvIiUi8jV8ZZJiR3DhwdX9MYw+FpYGBuZFKU/kVA7BGPMJfGWQYkM9fXeRqO5uUy5DNSiAqALKCPDm3oJAHZXO4bsX4fDPn8TefXlaOlw4OCYGShxnwygNMqvQlH6FwllEJTkp6ODWUDvvMPMT6uOwGYDvlpWjfOG/Rfp2zfz4IkTgeOP/7INRFYW57o/9hjTTXPQiGM/uB+F1Z+hNbMANWmD0druwpyMFZBfvceD58/XKKyiRAg1CErEcLmAv/8d+PBDKvQvi6+MQen2tzD8vmewbSAw6dhctqPYvJltqi+6iEOURTBnDt1Gzz1rcOT6h5HZuBPl2aVwtgnS0oCZJ9pRMGQELc/TT3M2Z1lZPF+2oqQMahCUiLFpE7ByZc/WziW738f0DU+idsgobKtPR2E7MGwYmBra3g78+9/cHsyeDRHahuNH7EbDDzdg/9DRyLYJiotZfPylkUlLY7XaSy8BxxyjuwRFiQAJFVRWkpslSxgv6Kybba4OTN74PBpyh8M40pGZySlhX5Kezmjyc89x1e8hZ/MqDBtpx9HHCKZPZ9fTHu0e8vI4mFkH3CtKRFCDoESMbdt6NhotOrgF6e0NcKaxM11WFlsJdWndkJXFWoHNm733VVcH798gwuBEU1PfhW9sZN+M1ta+nyuWVFez7HnfPm+7WEXpJeoyUiKGzdazR09WSzXS25swuGID0tsb4LKlQWwjANcQwNGt6KC21vt7fn7P1hPdMYaBi74Um23eDLz+Ov1dIrwdfzwHHI8Y0fvzRpvPPwcWLmQql/XGFxUBX/865zvrYGalF6hBUCLGpEnUq0OG8G+HsxVHbH0FQw98jLaMPLhtaXB2uDHJsRfydiZ7CnUeep+W5v195kxOoQ80Y7OhgU/WW8W9eDHwf/9HP5c1xszpZCDkv/8FfvhDvqhEY+1a4M9/piEcNcqr/BsagEceAXbsAC67TI2CEjb6iVHC4uBB4LXX2ITu0Uepm9rb+dhpp9F743YDMAbT1/wduQ170Z6eg/b0XHQ4stAiA5A9vICKfsWKTv8AtqS2KC1lYx9/8QGnk93hzjmndwHlrVuZpTRqFAcaW+dwOBjTyM8H/vQndkZNJOrqgIceoszFxV2Vfm4uI/rvvAOsWRM/GZWkRXcISki4XMALL3BRLcLFqcvFWfb5+cAPfsBWQaecArz7LjA1eweG71+D6sLxyGyrx4D6/ahHPvLzgQEDAEgWq9a2b2fZ8YwZdHlYiADf+x7wxz/SR15YSIXndtMQtLRw3FpYA5c78eabfBGddyUADdWhQ3yOigrm0V53XZAhDDHkww9pDP25yWw274jQsjLNvlLCQg1CClBVxWZvbjfT8ktLI68Hnn+eOmb06J79hurqgP/9X+AXv6CnorAQqP7jMtS1pKMWghr7UTjc2YiitFoMyM6B9bFz2jNg1m1Ey/zzkP3Ny3t+GPPzgdtvB1avpvto924+eVkZ+0QfdljvXmhbG/Dxx9wddH8hq1YxwGyzMevpySfZIO+882JfBFdfT3k++ojbsJISYMMGP21eO1FQwF7aDQ3Bj1WUTqhBSGIOHmQKvzV6UoRGYeRI4PLLgSOOiNzzLF7s2xgA1NstLSwJ+MEPGNfsWF2OA5tysP4LoL4lA58POxGNTTswuPxzHNrTBIcDaHc50Jxegservov2X+Vi/nzq+S6L9sxM4MQTebMi1n1Vyu3t3gwli8ZGYPly3m/FNTo6+JxDhtC95HZzZFssWLeOfrn2dr7BdjvwwQeMbwwezPiLw0GlX1/P/8nJoQGwguMhdQBUFC9qEJKUQ4eA3/yGeqxzXNEYJuv89rfALbdEJia6ciX1S6BOpIMHc9FdU0OPRYdkYOd2F9w2FpSJpKO1aAK2NR2OAztb4GwDikZmYnTaXhSNyYVUb8MX/+9DvP9wDWafPRCOE2b13AFEanWelcX6h/Z2/gS4C3C5uq6oOzqojNPSuDp/8UUaps6B8GiwfTvwwAN0oQ0Y4L0/N5dWeetW73tRVcWfVo+QgQN50dPSeLwSHk4nd2GLF3t3pEcfDcyZE52td4KhQeUk5bnnuDgcPrzrQleEOmHgQODhhyOzSNyzJ3hmp83GW7VnAOp/mmfC1ljfpVDN7QbK99vhzMyBLS8HTZWNaEgvQtlHf8Npq3+Dsub34dz8GfY9swy4+25avGgEdR0ObkUOHODfra0cxdY9TtDeDowdy9/T0qhwP/oo8vJ058UX+YZ3NgYWpaWUf+1aoLKSBquggD/z8xmkf/ttDlvoHh9RAtPcDPzhD8D99zOZobCQC4SVK4Ff/5oDl3zNPk0h1CAkITU1dC0HmiOfl0ddunFj358vPT20mie32+vFePXADKTnZSK9vfHLx5uavAPm7eJGfkcV9h1MQ379btQVlKIxbziksBAba0bAPWo0AyN/+EN0isXmzKE7qqaGiqC7C6m+ngp28GDvfdnZDHBHk4MHWRvROcDemUGDuJOxZm/62kFlZHAUW4orr4jzz39yp1haSmPgcHgr6UeOZCBt5cp4SxlVwjYIIjJAROIzf1ABwMUsEHyYjMPRrU1EL5k+nTGCQLS0cEE7fDiTc9ocA7Dq2BuQ2VKDnMYDEOP+Mlab7azHsNad2JAxA1vaxqAxz1tHkJbGhXlDo7C+YNcuroYjTVERcOut/H3vXhqd9nZardpaWtTjjuv6JlsWL5rU1fFN8ueaaGykIcvP57H19TRojY2UOy2Nk+JqayNz8fsL+/YxVbekxPd773BwcbBwYUpXhAc1CCJiE5FvisjrIlIJYAuA/SKySUTuFZHDoi+m0plAtVqdsYLMfWXqVK/+8SfP/v0s7rW8FCJA1eDJWHrSHagYPAV59eUY2FSOYR274LSl45Vh38VHOBbt6b7TOb+Ue+BA+nOjQWkp8LvfATfc4A3GFhXREMye3bN1RktL9OdepqcHXtnX11PO4mLGCsaO5Wp2+HDKPXeuNw6ya1d0ZU0l1q71BuP9kZvLmM3u3bGTK8aEstx5D8DbAG4H8Kkxxg0AIjIIwBwA94jIQmPMv6MnptKZoUOpMN3uwMWoHR3UeX0lLQ248UbgnnuoEwcP9j5vSwuNwZFHskupJZ+V5FJXMBqrZt6AjNY6lG+qx9YdaXAVDoaBIG/3Cgwb2rUPkaULv9TFVgO7aJGVRffRLbfQR+zvDWts5BYo2iM5hw9nTKCpyXcMwVoNuFwMund2afk6VgmN6mpvgkEgItU7K0EJxWV0qjHmLmPMessYAIAxptoYs8AYcz6AZ6MnotKd4mIq4IoK/8c0N9Pl7XcQfZiMGwfccQcnn5WX87Z7N5/nwgtpMKzvU04OF9id9XhbZj6yx4/CQcdQuIwNHU6Bw+7C+OzyLs/T1ESDkpXlucPp9BkcdbkivHM/4wyutnft6tJ1FcbQr3/oEHD99d5RbtHCbmfe7oEDvl9gTg6tcHa2/ziDRSL3Yko08vO7Xnd/GNPpw5l6BN0hGGM6RGQCgBEAPjTGfBklFJHTjTGLjTEhvJNKJLnoIuCuu7iDLSrqutNtaqKx+MEPIqu/Ro7kOWtquKCyujz4SmY56yxm7+3dy9kHNhsXvOPGMWbqcABnT6jFoOZ9aEwfDoD1Ym43O1Z8SVUVm7V5Hl+1ijVqe/fynFOncmcyaVIfMwKzsoAf/5g+4v/8hxYHoECHHQZcfDF/xoLZs9m8bulSXlzLndXeTpdRXh6LTPxtD+vr6UYaPz428qYCRx/Nax/IH9vUxN3b6NGxlS2GiAmyrRSRGwF8H8BmANMA3GSMednz2FpjzNFRl9IPZWVlZvXq1fF6+rhTXs7OCpar2EpFz89nxXC8B4nV1ABPPMEaKwur84TdDkzK2olzN/waNVkj0OZOQ0YGe9oVFnoO7uig5v/1r9EwaDTuu496ctAg6kRr8d7YyPjFJZdEqJ9bUxO3P04nhRk2LPb55243M1oWLfJaP4eD7q0jjmDRWnZ2z37j9fV843/8Y27nlNAwhm1SNm/myqc7Lhe/aFddBZx8cszFizQissYY00NDhGIQNgA4zhjTKCKlAF4A8KQx5gER+dgYMz0aAodCfzcIAD/HO3eywaXLRd01aVL0k2HCobLSK19hIXvYHTzIwltZ/AYmfPw08scMRNG4AjjSxFtdV10NXHopzLzTcd99TKHt3m0C4Hm/+AK48kr2UkoprPfC5WJQ09ry7djB1cCBA11XtYMGUWmpMQifhgb4XHVUVdE3evrp3CmmQHFaXwzCRmPM5E5/54BGYROAU4wx0yItbKioQUgBjOEW4qWXuCq32bg6Li0Fzj4bmDYN5eXAz3/Onbq/72JzMz0qv/99YhnDqOJ2M7V0927+PmIE/W3B8pEV/7S1eXtn7d3LD9y0aWzlO2FCShgDwL9BCOWrUyEi04wx6wDAs1M4E8A/AUQ5B09JeURY6DBtGrcSVjR88OAvv3zr1gVOzQf4LwcPcqcQK1d/3LHZ6D6KVNMqhTuwE07gze0OnoqaYoRiEC4H0KUBgjHGCeByEXk4KlIp/Q8R72SdbtTVhdaFQSR4AZ2ihEw/HDAUSpZReYDHVkRWHEXpycCBoWUEut2+U/eTjpoaNrBra6Mfe+LE4POlFSUC9Bdvq5LEHH00m/kFyghsbGTAOhKFeHGjuZkjPf/73641CJmZnAw3b16/cl8osUcNgpKQGMPEGrudhWqzZnFYmK9WM04n6y6uuy6Jd/ltbcxw2b6daY+dA8NtbcBTT9HqnX++GgUlaoRsEEREAFwKYKwx5k4RKQEw1BgTsX7AInI6gAcA2AH8wxjzu0idW0kO6uuZjvrGG8y2dDhYmzB7NvXhhg3MvszP5yL64EFmF11wAXD88fGWvg+sWEE30ZgxPRV+RgZTrF57jZbRV568okSAcHYIDwJwAzgFwJ0AGgAsADAjEoJ4Oqj+FcBpAMoBrBKRV4wxmyJxfiXxOXCAozhratieo7SUu4Q1a6gvL7mE2X9vvslsIrud/dxOOSVwSmrC43azAK1TZlUPHA7eli4FLr00tvIp/YZwDMKxxpijReRjADDG1IhICN2gQmYmgO3GmB0AICLPADgbrHdQUpyODhaKtrV17Qxgt7PYrr2d7vWf/MTbtTplaGxkEV5JSeDjCgpYSasoUSIcj2uHZxVvAEBEisEdQ6QYAWBPp7/LPfd1QUSuFZHVIrK6yhofqCQcLhdX+jU13rZAgfj0U5YhFBf7fjw9nQk3r70WWTkVRfESzg7hTwAWAhgsIncD+AaAn0dFqgAYYx4B8AjASuVYP78SmNZW4P336QGx5ifk5wPz5/seMWCxfHnwlNHCQg60qqvjOVOGnBy2Smhs7DnGszO1tZx3kAg0NbGid8MGWvxx4+i/+7IRVQRoaGDDLrebNSrBursqfSYkg+AJKC8DsAbAXAAC4BxjTCT3r3sBdO5UM9Jzn5IkNDXR7fPZZ/z+Wh6Qpibg3/9mltAPf+hb8dfXB+/MahWNtrSkmEGw2YCvfQ147DG+Ob7iCE4nFe9XvhJ7+bqzZg3wyCP07+XkUP5PPgEWLGB67Fln9S2gU1fHc63oVObkdnMWxYUXssWuEhVCchkZNjxaZIzZYoz5qzHmLxE2BgCwCsDhIjLGE5u4GMArEX4OJYo8/TQb7Y0d21XpDxjA+3buZPakLwYNCl5l7HYzHbUvxWetrRxt0NDQ+3NEheOPZ6+c3bt7+tja2thp88wz459htHEj8Oc/0yKXlnLVPmgQrf+IEcALLzBFrLfU1gK/+Q23jMOGsZuhdduyBbjzTmDPnuDnUXpFOC6jtSIywxizKhqCGGOcInIDgDfBtNN/GmMiMAuADNAAACAASURBVCJeiQU1NaynCjSTZcQIppRecEHPrs2zZtHN1NLCZJqiop7upaoqtjzKzQ1fvv37mZ20YgX1rTEcF/C1rwFTpiRAhlJGBnDzzbSqy5d7C9NEOKvhW9/yjqSLF8ZQvvx8No/qjsNBw7BwIXDSSb2z3M8+y1zi7gF2m40FKYcOAQ89xGEgSVt0kriElWUE4FIR2QWgCXQbGWNMhGZyAcaYRQAWRep8SuzYupU6LFCjTbudOmXrVhoAgP+zZAl1yJ49fCw7m3qwpITKOi2NhqKlhQo8XLZtYxdUY+jKcjj4+969wL33starr16OsLDmjhrDKLo1Azkri62rzzuPb0R7O63fxInRn9QWCrt3800LlA2VlsaUsY8/Bk48Mbzz19TQrxhoVVFYyN3S9u3a1C8KhGMQ5kVNiiTDGCqvigqv4go02rY/0N4e/rHG0FX8yiv0hMybxx1EfT31ys6d1BFjx9KYfO97jF2GQ1MT8MADdHVbehfgdSss5GJ3wQKed8qUEE/qcvHEDofvlbI/GhuZJvXee4wJWMyaRd+7lWJVUAAce2zo540VNTXB284CvHgHDoR/fmvSU7D23SK08moQIk7IBsEYsyuagiQLO3dyCtjOnd4gpzGcXfytb/lPm0x1OivbQIh4jy0vB15/na5oSwfMnk1d8vnn3krkuXPZlsJPM9SArFrFFkH+rovDQXkWLQrBINTVsTBsyRKe1BhO+znjDAY8AynKhgZW3e3Zw6CoNYDa5eKqeP164Kc/pd88UbG2VsFwuXrXjM9qNx0Mmy20XGYlbMJpXXGHr/uNMXdGTpzEZvt24He/486+c2Ws2816obvvBn72s/5pFCZO5GK5tdW/Lmht5Xs3cSL/XrqUi8nOC0KHg7sFK3ba0MDFeG/f0xUrgmckFRby+gXM+qys5MWvqfGmQBpD188f/kD//qWX+vdrP/cc3S3du+/Z7XyxlZX0jf/qVwkQ0PDDmDGUt6PDfz9yY/iFsC5yOBQXewM8gd4Dp5Nb8i1b+OHIzuYQjFB6pCsBCcdl1NTp90wAZ4JzlvsFLhfw8MNUGAUFXR+z2bjo27eP1bQ33RQfGeNJRgZd348/TmPZfWqZ00mPwDHHMLvS7WbsNFhqeW4uXdfNzYFT9P3R0hJcT4jwGvp1e7nd9Du1tHQto7b8TgMHctcwejS3ON2pqwsecS8u5htkpWklIgMGMFj89tv+28pWVfF96M1rGDmSRufgQf/1DM3NDCw/8QSzryyys5mF9dWvarC5D4TjMvpD579F5PdgRlC/YOtWftYDtVceOpTTvQ4e7J81NHPncpX98sv8TlqGs7aWC2mnkxXJVvLJxo3UtdOn++7pFgmGDgU2bQqc8NLRwYWv32O2bvW9urew2bhife01BlK7K6SdO7nqDeQbt/yPW7dGxiBYw7bffZd1A04nUzdPP52pWum97Dpz/vlsJPXZZ3RvWdvBjg76+nJyGOzpzcUUod/1N7+hFe/uh2xpoWEdMICPZWV5H2ttZU5zZSVw2WWJu8tKcPrS/jobLB7rF+zYEXzhYcXb9uwJbBA6Orjbra/n9/KII1Kj0EqEsdFjj+Xqf5OnC9XgwXytY8d2/Q6PG8dYwbp11JWdF98WDQ18L8OJ3XZmzhwW1AbyQlRUACefHCCR56OPgitQayuzb1/PWoHqan4oNm3iqjYjg2/GqFFd3xC7PbzovD+MoXFasIByFxfzw3noEPDXvzLu8T//07stV1YW8KMfAW+9xTzeqiqvMTvpJKaB9aVaedw44JZbgAcfpOFJT+e529poEHJygLKynlvQzEyuKt5+m49PmtR7Gfox4cQQNsDTxwisEygGcFc0hEpEgrk1Lawgs79zLF3K2p2mJu85RViAevHFXfVDsjJsGGsNAO4AbruNmVjdX9vo0TS0AwZw59B9DIAx1DdXXtl7L8CECdR/O3bw/N2vYV0dn/O00wKcpLExtBW1zdbVjQFwJf3oo1RuhYV8UYcOccuUnc1WrVaAxOmMTFD5ww8Zs+juuxs4kNu2HTtYaXzzzb1bSWdlMU/39NO5Ine7+doiNa5uwgTGZTZu5I7J5eLu7KOPuJLqbgwsbDYajLfeUoPQS8LZIZzZ6XcngArPbOV+wahRwRMbrKEuQ4f6fvy11/g9HT686w7C5QKWLaOO+NGPEiPlPFJ8/jldaL5W/3l5VNZbt/K9q6z06kOXi1lIEyb0bc6B3Q7ceCPwpz9RN2dlUW91dHDXkpPDBWnADKYhQ5hXHwjr4neumquupmIrLqby//xzCmSz8fjmZuD55+n7HjKEF/7IPpb1uN3ASy9xW+ZLcYrww7x+Pd/gUaN6HhMq6enRq5xOS6Nra9o0731PPBF891FYyP5Koa7glC6Es+663hizy3Pb66ksvidqkiUYkyfzu97U5P+YqioqMF+tVg4c4A7e10rZbuf9W7bQ1ZJKHDrk/3spwoXckUfSU7JjBxfSX3xBl/1XvsJFbF8NZG6ut2321Kk0CMOGsQbsnnuYoBKQY4/l6j1QymV1Na1b54KU5cv5whoaqPxtNiq69HS+KGtF/e67XA1ffHHfZyeXl9MHFqicW4QfutWr+/ZcscbtDr5VFOk6flQJi3B2CKcBuK3bffN93JeSpKVRgdx/Pxd83d2v1dVcIH7zm77/f/lyrz7whQh1yaJF9HunSqJEsNR1ESpky202YQL15YQJPbO5fFFdzZvDwSQef++v3U7j0ytPwsiR9EuvXu17Ek9LC7cb3/9+1/vffZd9fpYu5cXNy+M2sKOj6wVuaOAH6qSTeiFcNyzDE4yMDL5xycS4cTR4gXYJdXXRy1DoBwQ1CCLyPQDXAxgrIus7PZQL4L/REiwRmT6dK9bHHuMq1mbzLkaGD2cnT39V/Z9+2rN/T3dycph5WF8fmjJMBqyEGWs+si+sBJwzzgjdA7F7N9tdfPKJ9zpkZ9Ot/dWv9j6JxiciwNVXU5GvW8cnys31Dn2w2ZhZM2FC1/+rr6el6ujgbiAtjT+t4grAuyNIT+fFHzOmb7JmZ4e2Qm5vD/6BTDTmzWM73UGDfCt8Y3g9Lrss9rKlCKHsEP4PwBsAfgvgJ53ubzDGJNkSo+9Mm0a38KZNTCgRodIbNy4yq/pUW9gMGsTODB995N9dXVFBXRooTb8z27ax6DctjQbEet9bWxmj2bKFcYOIGoWsLBaYbNvGlf+uXVS+c+b4nwOQl0ffV+eLarMxpcxKK+vooGGx2Rhs6atBGDmSW9iGBv9uI2PoAisr69tzxZopU7gq++QTrrw6f+HcbmZyTZ3KqnGlVwQ1CMaYOgB1AC4RkYEADgcL0yAiMMYsi66IiYfDQb93OPG/CROYERfItdvURB3Sm26eicwll1B/7tpF372lqDs66EEpKAC+853QjGFbG7sv5+X1TFO3Mg/Xr+d7fcYZEX4hNhsvZPedgD9OOYVpnoF8Zs3NPF+wOoUQaGgAVq2yocl+DiYu/xuyJmRhyHBHz9jy3r1Urn0JKMcDh4M7saee8gbb7HZvtsesWcDll2vFch8IJ+30OwBuAmsP1gGYBeADAKdER7TUYvZsYPFiLsz8Zc1VVLAup496IeHIywNuv53xkXff9fZ1E2H+/5lncicRCuvXU/H5qxETodF54w2mksZVN5xwAvDMM1y5+sp6aWvjxR4xgtH3QFWPATCGmZbPPUfdmJV5HMpzKzB12UvYkZGJ8ScWY/BQO/3rVrfA7343ObejmZl0351zDt13dXX8gB11VP/sGRNhwgkq3wRgBoCVxpg5IjIBwG+iI1bqMXw48PWvMyNw5MiumTNuN2NlY8cmxkCsaJCbC1x0EdPXKyu9nZ/DTV3/+OPgRWpZWcz4OnAgzovgwkLgl79kj6OKCv7tcNAiNjfTGBx3HEu5Z8wI3Sp24513mJFZUmLtvgQVg89BxxFTMHzzO3D/Zw0ypriQP2UUM5mmT0/+3ObCwsQZJ5pChGMQWo0xrSICEckwxmwRkfFRkywFOfdcKrOXXmJMz/IkiFAfXH554hemNTQwHvDppzRkhx/OOoFQdZnVGLC3dE/Q8YfN1rXDdNwYP54Nnm65hZFwh4OrXCs/uaaGRSn+0tOC0NzMncGoUd1iJiKoLjwc1ScejkOHgKVFBr+4Q5JyU6DEjnAMQrmIFAB4CcBbIlIDQFtih4HNxmHzc+Yw7byujgu18eOTo/fRihXAv/5FRZuTQ0O2YQPw4otscXPGGdH3QowaxdY8gXC5aKx6ueCOPJMnMzVtwQL247YmBdXV0Zd47rm97l3yySc0koEW/IMGATu/EOzdy6detYq7tJwcNhuMVEKEkvyE09zuXM+vvxKR9wDkA1gcFalSnMxMfhGTibVr2e11+PCutVODBlEhPfMMldKpp0ZXjuOOY7ppoDTWigruuBKqP9SQIcD113NHYE1WGj68zxkEe/f6j0lZWLVaDz/McIbNxmvY3s64VmkpcMMNybEoUaJLyOsCId8SkTuMMUvBwPK0YP+nJD9uN0fpFhf7LqS10j+ff56pn5GkqYktJ7Zt44K6uJh1Bl984buVSG0tFeBZZ0VWjogxcCDdRePHRySdLCMjtJYqW7dyNzF6tHfC38iRNAYVFazYrq/vszhKkhOOy+hBAG4wq+hOAA0AFoCBZiWFCdSPyCIjg8bg008jk95eX8/RmkuXUuFZTQNnzfLOP37zTe8MepeLzz9wIJvphVrTkOyMHx+8Dq2ykklMJ5/s26U3ZAgN7LJlzPhS+i/hGIRjjTFHi8jHAGCMqRGRSJb+KAlKTU1ox9ntzO7pK3V1HE5WUcEUUit11OWi/3vDBk6mO/VUYOVKuk0yMlg0OHlylFJNnU5WI65fz3TRUaPol4pzte9hh9H4BZrBsW4ddwOB2iQNGUIDO39+6qU9K6ETjkHoEBE7PC2wRaQY3DEoKY7DEVqw2O2OTHXwM8/QsHRvA2Kl7FdW0h9+xx0xWtHu3An85S/s/ZOeTkGWL6egZ57JnPg4RWWtrhm/+Q0r54cM8Sr09nYW/tlswYt3rVTd5ubUK4xUQiccg/AnAAsBDBaRuwF8A8DPoyKVklCMG0eDEKwfkTF0YfSFmhqu+gP1NCoupo7etavXtVyhs28ftyuZmT19Zk4nI9wA54fGiZEjaRwXLuQOyuqubbez/U9xcXC3knX9dHfQvwmlud2TxpjLABQBuBXAXAAC4BxjTL+Zqdyfyc9nrcGKFf6b9x04wLnqffXd79zJn8GmTQIMNEfdILz0En/6cg05HDQSr71GB30c81yHDuVO4ZJLuNK35nxnZfH28suBB6RZBcwh1cE0NzPt7N13GcXPy+PrnzEjckNylLgQyj73GBEZDuAqABUAngYb3lWISKJkeitR5uKL6TbfubNrJlF7O1fqubnANdf0vQ4hWMaMRUwKz2pr2fI60PQcK+dz5cooCxMaBQUsFhw3zqvcTziB18VfBpjbTYMwf34I12/vXuDnPwf+8Q+vC62ujsV3P/0pPwxK0hKKQXgIwDsAJgBYA2C152b9rvQDBgxg9s5557Faec8ettuoqmIg97jjqDu3bOnbfBIrMBqoHxzA5/A3mS5iWNN9gsUHsrISWhEWFQFXXEHvV01N1/e2qYlGfs4c4Oijg5yosRH4/e+5CigtpfXJzOQWcvRonvjee2lIlaQklG6nfwLwJxH5mzHme9EQQkQuAPArABMBzDTGqKFJQLKzgbPP5kqyqor1AS++yFTTjRt5jNWj6NpruVINl9JSZhbV1fmfCdHSQlmmTOn1SwmNUB3qbnfCd9j8ylf4fr74oneWhzHU5VdcEeJQpo8+orL3l388aBDbc6xYAXztaxF+BUosCCWGIIb4NQbWMX2Q41MA5wF4uA/nUGJEejoXi48/TiXT3Y9fW8s47O23hzCeshsi7PhqzTvo7pJua2PmzDXXRHjegS+ssuzW1sA5my0tXWf/JihTp9KI7t/PXV5GBt2AIQeSlywJHicpLmbr1VAMwr59dLXt2cP3t6yMQkb9wir+CCXL6D0RWQDgZWPMbutOTw3CiQC+DeA9AP/qrRBWcFq081ZS4HYDjz5KY+CrPURBAY/517+Au+4KP64waRIn0z38MPPrMzK8PnCHg6NMZ85kRs3771O5DRzICZSTJwdv5RAy6elM01mwgFbP1wupr2dQNZzhGHHE6pjRK2pqus6M9kVmJgtIAs0/7ugAnnySF89mo9V3Omkc8vM53cgatafElFC+OqeDAeWnRWQMgFpwQI4dwBIA9xtjPo6eiF0RkWsBXAsAJf5SXpSosn07XUaBKpcHDqT3YOfO3n23jzoKuO8+FlVt3kz3xtixXETW1TGuWVXFzJmMDOqgtWuZgnnzzYHH7obFvHksSNu8mZrU6iLndtNatbcDt97aP1a1Awbw9QZKRerooFHwZwyMYa/upUtpZLsfV1vLPhq/+hV9h0pMCSWG0Aq2rXhQRNLA9NMWY0xYkSMReRuArzDgz4wxL4d6HmPMIwAeAYCysrK+uKmUXrJ/f/Cgr7WY3r+/94u9jAzg2GN5s6ivpzupo6Orq8oqptq3j7rkqqt4n3dGQC/JyKCFWbSIrpDWVioxl4v+l298o2/9vJOJk05i/mqghVhlZeA5BXv3cmfgyxgA3F42NzOV95pr+iyyEh5hba6NMR0A9vfmiYwxUe6DqcSKcFxAkfYCrljBHYIvHVxVxdqE8nLuTAYP5mJ2/nwu9Hsd983IYIvqM87gyZ1ObkH6W3vQE04AXn+dASRfRQ3NzVwpnHyy/3OsWEGfXqAI9pAhdB9dckng4gkl4kTK25o0VFdz1QrQAxDnVjRJiTWFzNdUSAur8jWSE8uMYb8dX5MS9+5lEkxmJnV1YyPdS62tHCCzYwdw/QVVcHz0X7b9dLmYBjV7NoUMxXJlZDDBv79SVAT84AfAAw/QKg8eTCvrdNJn53IB3/9+YFfPnj3Bi9esKHdtrRqEGJMQBkFEzgXwZwDFAF4XkXXGmHmRfI6KCrZnXrvW+903hsWVF1zQ/xZ7faG0lCv0Q4f8v29VVdS3gVpQhIvLRR3RPauprY1Dc3JyqJ9cLubXAzQQY0oNml99F1VvPEVdVVDAD8HSpcDbb7NL3iWXaN+GUJg6FbjzTs7tXLaMxsBu5+7h1FODrwBC6dcNMEYTsewAJVR69Y6LyFBjzIFICWGMWQj2SYoKBw4Ad99NxTFypHe36nLRQGzdyu6ZOqM7NEQ45/zuu+kyLi72Gtn2dhantbQAJ57IwHJJSWRcR3a7dyRxZ11RXt61FKB7WcCIfR/h6L3/wua8URg6M90rS24uPwRvvknf0vnn913IRMUYrs63bOG2afBgZkYFG1Dti+HDgcsu49jP1lYq+VCV94wZtN6BVmCNjdzmBctoUiJOb03wIgDB6hoTAmOYIul09ky3s7pn7t/PnPof/zg+MiYjJSXM9HnySfrtAb6PO3dSzx5xBNPWlyxhLcK11/b9+y3CiugPP+x6Lffv7zpCsrmZzw8AYtyYvOl5tBcMQUNL+pdFbV9it3O7s2gRJ++kYqvPgweBRx7hhbIqr10uRtvPO48Blt5YbLs9/N5F06ZxK2el63bHGG7nr7pK53rGgd6+40lTMFBezoraQMpo6FBW2x6I2J6nfzBqFIvP7r6b33Oru+bXv86up5Zrqbwc+O1vGb/pK3PnchfS3u69z+326rOODv5uJcIMrNmB7OZDaE+nL9pndpTDQQX5ySd9FzDRqK3lm797Ny/G6NG8cKWlXKU/9RTw6quxkyczk3UG9fX8wnXuc9LUxGDP8ccztqPEnN4ahL9HVIoosnNn4OAnwMdEvJ02ldAR4Yp7/Xpg+vSe+f8iNLh1dUxQCURzM/XWnj1dFX5nSkuBb3+bRqaykvpk4EC6AxsbqVPKyrwL14zWOhgRuFxccPodRu9wRGa6T6KxeDEt8dChPb8E6ek0EAsXchcRK444AvjlLxmPKC/nBd+9m0b5yiu5ndR4TlzorcvoxYhKEUWs8Yuh0JembP0Zq9FnoLTOYcMYgzz//J5u6+pqpp2//773GmRkAKedxh1H9+PnzKHL6PXXOT0tLY3G4PDDeeucOeaypwPGoLGRriu/rm6Xq3f+9ESmtRV4773AWT9WCugHH3BrFytGjWJGUl0ddzEOB42WGoK4kvIxhECdiztjjMawestnnwV3JaeleYt7O9c1VVZy2ldDA/WBZVRaWzmKYP16xna6n3/8eN4aG7mzWLiQKe7d3dLVgw5Dc3sasgZ0YOxYPxbLypGNere8GHPoEINnwSrzcnJ4EeNBfr7v/idKXEj5GML48VwxNjb6P6aujoajP6eY9wW7PXjlskXn3ZoxwEMPUfmPGtV1h5GZCYwZw67Szz/v/3w5OTTkV18NnH466xF276Z7urwc2L43C+UTT8VXxpQjK9OPkPv2sQlSr5v8JCjhbI21j5iCfhBDsNuByy/3zovtTmMje3ZdfrkmNfSWyZO9ef/+aGujG6jzLmzXLsYQA+3ihg/3NrALhMPBLMh77wUuugiYNYvupp/+FLjwqXOQNesoBok6n6ilhb2gi4oiM90n0SguZjqtv8k4Fo2NSdOcT4kuvXIZGWMejLQg0WT6dCY2PPooXRTWDrq9nSvMm2+mUlN6x4wZwP/9H/Wrv75n+/dzlkLnoO6WLd6Avj8cDu4kduwIPigeYFB7Xo+SxnRW2P73v0wv3e1p2jtgAHsRnXxyalbEpqVx2/T889xu+cJqITtzZmxlUxKSflMKeMwxdBGvX0/lAjAAOWVK/2hUGU0GDAC+8x3gr39lu/zOLmGnkx6Z0aN7Kuq2ttB3ZX0el5mWxuZsX/kKUx6NYc1BqlfDzp3L6sudO1l00/n11tczqHPddalZf6GETYp/G7qSkcHV7IwZ8ZYk9Zg5k37/p5+mK8ha9YuwYvmii3om8QwZEryLgRXvjdj8epvN/yi2VCQri1H5F15gmpeVdmcML8DNN3MLrSgIbWJaKF9Fd7jtsJXQsDoObN7MFXVxMV0niZgheeSRTC3fuZOppA4HW1/7Kki1jk9Pp+vO3y6tvp5Zk937FylhkJ3NINm553J73NHBTIsxYzRwpnQhlB3CPs8tUMTNDkCn1USYgwc5Neyzz7p2HEhLY8eB009PvDioCI1AKDMQsrPpwn/ySaaidq9jaGmhYfnxjxPvdSYlubmhBWJiSGOjN5anrtv4E4pB2GyMCbinFJGYTUzrL1gdB5qa6H/vrBDb2xnEbWsDzjknfjJGgtNO4+t50VPqOGAAd0WNjVQQN9zAXYeSOhjDWN7rr3OxY7MxG/Ckk/h50Hqg+CEmSAK5iGR6pqb16ZhoUFZWZlavXh3rp40JzzzDJpz+hnE5ncy5v/fe1GjdXV3Niuft26kcpkzp2oJCSQ2MofF/+WUmHwwaxMVORwd72jkcwG23+U+KUiKDiKwxxpR1vz/UEZr+TnqlMeaxeBiDVCbUjgMizKQ866zYyRYtBg3iQDIltfn4Y1agl5Z27VKRlsbW9LW1wP33c6xyZmbcxOy39DWi9OuISKF0IdSOA7m58es4oCjhYgx7Vg0c6L9lUUEBOwesWxdb2RQSSpbRen8PAQixU5ASDqEmfhijSSJK8lBbywy0kiDpJ7m53PnOmhUbuZKKxkbOtWhvp8/t8MMjWksTypmGAJgHoKbb/QLgvxGTRPmSoiJm4ASq/AXYhUE7DijJQns7FzDBMsbS0oK3Qul3tLay4nzp0q61JHl5TNU78cSIpOKFYhBeA5BjjOmxiROR//RZAqUHaWms6tWOA0oqYRVDu1yBu1w3NQETJ8ZGpqSgvR144AH2eulebd7cDPz971wdRiAIF9ThYIy52hiz3M9j3+yzBIpPTj2Vu8Fdu3q2baivZzuIK6/UjgNK8pCdTTdQRYX/Y4xhOrUOTOvEihXAxo30tXV3D2Vns1Xw888HfmNDRD3QCUpmJvDDH3IYzIED7Me2ezcNRHo6Ow4cf3y8pVSU8Jg/n922fXWvtaryJ07kYkgB36zXX2dxhj+XUFoafXErVvT56UIJKq81xgQchhPKMUr4WB0HzjsP+Pxz7hQKCrTjgJK8jBzJyvMHHmA2XUEBF71NTfR+TJzIQWr6+fZQX88inWCR+IICjg8877w+PV0oMYSJATKNAAaXdeRRFMnJSbiOA0osaWqiYkhLY3/vJO/jMWECCypXrWI2UWsrW53MmcNxyzpFsxOhTp4CIjIDOBSDMCGEY4L0rFQUJWz27wfeeINa0xh+4YcNA848k874JF5G5+TQAMyZE29JEpy8PAYKm5sDd7Ssq2Npfx8JpVJ5V+e/ReROz/+tA7DOGLOtz1IoitKVHTtYrut2c9i0NSmovh7429+YcXLFFUltFJQQsNvZxfKZZ/ynHLpc9CdHIBIf9qfJGHMHgAcA1AE4V0SSZpymoiQFbW3s35CV1TXNUITFSGPGsLfJBx/EV04lNpx0EmMI5eU93UIdHRwDO39+RGaCh2wQROQBETovjTEVxpg3jTH3GGOu6asQInKviGwRkfUislBE+tEEE0XpxiefcCfgb5CPzcbBGK++Gp6PWUlOsrOBW29lFeru3TQA1s+qKuDCC4ELLohZYZpFA4BXRORiY0yTiMwDcIcx5oQ+SwG8BeB2Y4xTRO4BcDuA2yJwXkVJPj78MPiM59xcKoWqKu0X3R/IzeVg+IoKTstqbWVHyClTIjotK2SDYIz5uYh8E8B/RKQdQCOAn0RCCGPMkk5/rgTwjUicV1GSEqsMPRDWxKT29tjIpCQGQ4bwFiXCcRnNBXANgCYARQBuNMa8HwWZrgLwRgA5rhWR1SKyuqqqKgpPryhxZsQINjELhDWMOl8zvpXIEU5Q+WcAfmGMORlcwT8rIqeE+s8i8raIfOrjdnanY34GWx6JlAAADjtJREFUwAngKX/nMcY8YowpM8aUFRcXhyG+oiQJJ5zArJFA8YGKCuDYY7V3iRJRwnEZndLp9w0iMh/AAgAhNVAwxpwa6HERuQLAmQDmmmBj3BQllSkpYdfCDz/kJJnuwcL6ehoLnSikRJheN9I2xuz3uJH6jIicDuBWACcZY5ojcU5FSVpEgKuvptJftYoVyjk53DU0NHCu6C23sA+EokSQPk1WMMa0REiOvwDIAPCWJ7N1pTHmugidW1GSj4wM4Prr2c3w/fc5QDsjgzuHo48OPChDUXpJ5Ebt9AFjzGHxlkFREg4RuoxKS+MtidJP0Lp3RVEUBYAaBEVRFMVDQriMFCVROXCALvyNGxnjnTCBPcSGD0/6LtSK0gM1CIriA2M4qGrBAir+gQN5/zvvAG++yYzPb3xDm40qqYUaBEXxwbJlwLPPsiQgLc17f24ui4RffZWZoFoKoKQSur5RlG50dAAvvEC3UGdjYGG3c675yy8DLZFKvFaUBEANgqJ0Y9s2thIKlOqfns6xBRs3xk4uRYk26jJSlG7U1YV2nAhQUxNdWRKGtjZg3ToGUaqr6S876SRgxozgrbqVpEENghJzqqqArVupYwoKgMmTgczMeEvlJT09tOOM6ScFw5WVwO9/z5/5+bxYdXXA448z6v6jH/kf76gkFWoQlJjR0AA88QSwejUnAYrwlp4OnHsuMG9eYqRyHnEEs4ecTv9jCSz5J0yIrWwxp6WFxqChoWvFdFYWjUNtLXDvvcBddwGFhXETU4kMGkNQYkJzM/XK2rUMyI4ZQ/0yejQHPz31FLBwYbylJHl5rDUoL/ffgXrvXqCsDCgqiq1sMWftWu4M/A1lKSjgQJ9ly2IrlxIV1CAoMWHpUo6AHTWqZ+5+RgYNwyuvsBAsEbjoIq7+d+7sOqumuRnYsYOv49vfjp98MeOdd/zPdrYYMgR46y2d75wCqEFQoo7TCSxeDAwd6v8Yh4PpnO9HYwZfL8jMBG6+GbjiCv69ezdv7e3At74F3HZbP4mlVlUFn9mbkUHXko7zTHo0hqBEnbo6uqCtal9/5OcDW7bERqZQyMgATjkFOPlkzqQBWJhmt8dVrNiSk0NFHyjS7nTyTfFVtKEkFbpDUKJOIgSK+4LNRq9JQUE/MwYAgymHDgU+prISOP547eORAugVVKJOfj5vTU2Bj6utZQqqkkAcdxy3Sg0Nvh9vbWVp96kBJ+QqSYIaBCXq2O3A/PmcC+8v7tjRwcdOPDG2silBKCgAbrqJPrPycl4ogA2d9u1jFsB3vsOmT0rSozEEJSbMns3xwNu3cxRwZ9dLSwuwfz8zewYPjp+Mih8mTgTuvBN4912mizmd9AMeeyxw2mlalJZCiEniVLGysjKzevXqeIuhhEhzM/DMM8CKFSzsssjJYSvp2bOTP96Q8jiddBNlZGgQOYkRkTXGmLLu9+sOQYkZ2dnAVVcB55/PBnLt7YwtjB+vuiVpcDj6Sb5t/0QNghJz8vPZE01RlMRCg8qKoigKADUIiqIoigc1CIqiKAoANQiKoiiKBzUIiqIoCoAEMQgicpeIrBeRdSKyRESGx1smRVGU/kZCGAQA9xpjjjTGTAPwGoA74i2QoihKfyMhDIIxpr7TnwMAJG/5tKIoSpKSMIVpInI3gMsB1AGYE+C4awFcCwAl2lBLURQlYsSsl5GIvA3A18ysnxljXu503O0AMo0xvwx2Tu1lpCiKEj5x72VkjAm1YfpTABYBCGoQFEVRlMiREDEEETm8059nA0igQYqKoij9g0SJIfxORMYDcAPYBeC6OMujKIrS70gIg2CMOT/eMiiKovR3EsJlpCiKosQfNQiKoigKADUIiqIoigc1CIqiKAoANQiKoiiKBzUIiqIoCgA1CIqiKIoHNQiKoigKADUIiqIoioeEqFRWoo/TCezYATQ1AVlZwLhxQFpavKVSFCWRUIOQ4hgDLF0KvPQSUFcHiPC+7GzgzDOBr34VsNvjLaWiKImAGoQUxhjgxRdpDIYNA0aP9j7W2go8/TRw4ADw7W8DNnUeKkq/R9VACrNzJ/DqqzQE2dldH8vMBEpLgffeAzZujIt4iqIkGGoQUpj33gPS0wGHn32gzQbk5gJvvhlbuRRFSUzUIKQw69YBhYWBjykq4g7B7Y6NTIqiJC5qEFIYtzu02IAxvCmK0r9Rg5DCjBsH1NYGPqa+Hhg1SjONFEVRg5DSnHYa0NDgf/VvDHDoEDB/fmzlUhQlMVGDkMJMmgSUlQG7dvWMERgD7NkDjB/PYxRFUbQOIYWx24Hvfhd46ing/fd5n8PBqmVjaAiuvJKZSIqiKGoQUpyMDOCqq4Czz2bWUU0NkJcHHHUUMGRIvKVTFCWRUIPQTygsBObOjbcUiqIkMhpDUBRFUQCoQVAURVE8qEFQFEVRAABikrhEVUSqAOyK4lMUATgYxfNHk2SVXeWOPckqe7LKDcRf9tHGmOLudya1QYg2IrLaGJOUWfrJKrvKHXuSVfZklRtIXNnVZaQoiqIAUIOgKIqieFCDEJhH4i1AH0hW2VXu2JOssier3ECCyq4xBEVRFAWA7hAURVEUD2oQFEVRFABqEIIiIneJyHoRWSciS0RkeLxlCgURuVdEtnhkXygiBfGWKVRE5AIR2SgibhFJuNS87ojI6SKyVUS2i8hP4i1PqIjIP0WkUkQ+jbcs4SAio0TkPRHZ5Pmc3BRvmUJBRDJF5CMR+cQj96/jLVN3NIYQBBHJM8bUe36/EcAkY8x1cRYrKCLyVQDvGmOcInIPABhjbouzWCEhIhMBuAE8DODHxpjVcRbJLyJiB7ANwGkAygGsAnCJMWZTXAULARGZDaARwBPGmCnxlidURGQYgGHGmLUikgtgDYBzEv09FxEBMMAY0ygiaQCWA7jJGLMyzqJ9ie4QgmAZAw8DACSFBTXGLDHGOD1/rgQwMp7yhIMxZrMxZmu85QiRmQC2G2N2GGPaATwD4Ow4yxQSxphlAKrjLUe4GGP2G2PWen5vALAZwIj4ShUcQxo9f6Z5bgmlT9QghICI3C0iewBcCuCOeMvTC64C8Ea8hUhRRgDY0+nvciSBckoVRKQUwHQAH8ZXktAQEbuIrANQCeAtY0xCya0GAYCIvC0in/q4nQ0AxpifGWNGAXgKwA3xldZLMLk9x/wMgBOUPWEIRXZFCYSI5ABYAOB/uu3kExZjjMsYMw3csc8UkYRy1emAHADGmFNDPPQpAIsA/DKK4oRMMLlF5AoAZwKYaxIsWBTGe57o7AUwqtPfIz33KVHE44NfAOApY8yL8ZYnXIwxtSLyHoDTASRMUF93CEEQkcM7/Xk2gC3xkiUcROR0ALcCOMsY0xxveVKYVQAOF5ExIpIO4GIAr8RZppTGE5x9FMBmY8wf4y1PqIhIsZXtJyJZYCJCQukTzTIKgogsADAezHrZBeA6Y0zCrwBFZDuADACHPHetTIbsKAAQkXMB/BlAMYBaAOuMMfPiK5V/ROQMAPcDsAP4pzHm7jiLFBIi8jSAk8FWzBUAfmmMeTSuQoWAiJwI4H0AG8DvJQD81BizKH5SBUdEjgTwOPg5sQF4zhhzZ3yl6ooaBEVRFAWAuowURVEUD2oQFEVRFABqEBRFURQPahAURVEUAGoQFEVRFA9qEBRFURQAahCUFEFESkWkxdMnxrqvR1tqEcnytDJvF5GiPj5nlogs9XQ8hYjcKCKbRSSsNiEiUiAi1/dFlhCeo0eraxFJF5FlIqIdCxQAahCU1OJzT58Yqy31XwHMBzAJwCUiMskY0+I5Zl8Enu8qAC8aY1yev68HcJox5tIwz1Pg+d+wEBLqd/hfYJuEL/F0Z30HwEXhPreSmqhBUJIGz1CU0zy//z8R+XOAw2PRlvpSAC975HkIwFgAb4jIzSLyLc8wlHUi8nCnXcRLIrLGMyDlWs95fgdgnOfYez27nc4r+R+LyK88v5d6dj1PgD1wRvl7rs4EaHX9kud1KIoaBCWp+CWAn4nIpWDL4/8JcGxU21J7+haNNcZ8AQCetiD7AMwBsBhcdZ/g2Y244FW6VxljjgFQBuBGESkE8BN4djfGmFtCePrDATxojJkMIDvAc4XCpwBmhHG8ksKo71BJGowxyzyNzX4I4GTLVSMid4HNzmJJEdhnyRdzARwDYBXFRRbY/x6gETjX8/soULkfCPO5d3WashXouYJijHF54im5nmEzSj9GDYKSNIjIVADDAByylJeIDAUnT3Un7LbUIvJ9ANd4/jwDwLmd/zbGdI47tADI9HcqAI8bY27vdv6TAZwK4DhjTLOI/MfPOZzounvvfkxTsOcKkwwArX34fyVFUJeRkhQI5+g+BcYBGj3tvQFgGoB1Pv4l7LbUxpi/etw204wx+7r/3e3YGgB2EfGl0N8B8A0RGeyRfZCIjAaQD6DGYwwmAJjlOb4BQG6n/68AMFhECkUkA5xp4Q9/zxUSHpfVQWNMR6j/o6QuahCUhEdEsgG8COBHxpjNAO6Cd0iRT4PgmSd9A4A3wZm7zxljNkZYtCUATvTx3JsA/BzAEhFZD+AtcGezGIBDRDaDgeSVnuMPAVjhmRh3r0c53wngI8//+u2ZH+C5uuBpdf0BgPEiUi4iV3semgPg9d68eCX10PbXSlIjIo+Cbp0SAK8ZY0IaSSgiXwAoM8Yc7MNzHw3gZmPMZb09R7wRkRcB/MQYsy3esijxR3cISlJjjLnaGOMGs2vyOxem+cIqTAPjDu5Ax4bw3GsBvOcrzTMZ8LjSXlJjoFjoDkFRFEUBoDsERVEUxYMaBEVRFAWAGgRFURTFgxoERVEUBYAaBEVRFMWDGgRFURQFgBoERVEUxYMaBEVRFAUA8P8BBYGhJSJXix0AAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SjPlpf2-wHl8"
      },
      "source": [
        "## Обучение простейшего одноуровневого персептрона вручную\r\n",
        "\r\n",
        "Используем возможности tensorflow по вычислению градиента для обучения одноуровневого персептрона.\r\n",
        "\r\n",
        "Для начала, задаём архитектуру сети, в которой будет 2 входа и один выход. Соответственно, матрица весов $W$ будет иметь размерность $2\\times1$, а вектор сдвига $b$ -- $1$.\r\n",
        "\r\n",
        "Для удобства сгруппируем все параметры в отдельный класс:\r\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "J1KaixW-cMWJ"
      },
      "source": [
        "class Network():\r\n",
        "  def __init__(self):\r\n",
        "     self.W = torch.randn(size=(2,1),requires_grad=True)\r\n",
        "     self.b = torch.zeros(size=(1,),requires_grad=True)\r\n",
        "\r\n",
        "  def forward(self,x):\r\n",
        "    return torch.matmul(x,self.W)+self.b\r\n",
        "\r\n",
        "  def zero_grad(self):\r\n",
        "    self.W.data.zero_()\r\n",
        "    self.b.data.zero_()\r\n",
        "\r\n",
        "  def update(self,lr=0.1):\r\n",
        "    self.W.data.sub_(lr*self.W.grad)\r\n",
        "    self.b.data.sub_(lr*self.b)\r\n",
        "\r\n",
        "net = Network()"
      ],
      "execution_count": 199,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rQ7W6TOacIAI"
      },
      "source": [
        "> Обратите внимание, что мы используем `W.data.zero_()` вместо `W.zero_()`. Это делается потому, что к тензору, состояние которого отслеживается через механизм *Autograd*, нельзя обращаться напрямую.\r\n",
        "\r\n",
        "Функция обучение будет такая же, как в прошлом примере, но функция ошибки будет представлять собой логистическую функцию ошибки. Для этого нам нужно получить на выходе сети значение **вероятности** класса 1, т.е. необходимо привести выход сети $z$ к диапазону [0,1] с помощью передаточной функции `sigmoid`: $p=\\sigma(z)$.\r\n",
        "Далее, если для примера с номером класса $y_i\\in\\{0,1\\}$ был получен выход сети $p_i$, то ошибка вычисляется как $\\mathcal{L_i}=-(y_i\\log p_i + (1-y_i)log(1-p_i))$. \r\n",
        "\r\n",
        "В PyTorch оба эти этапа (применение сигмоиды и взятие логистической функции ошибки) делается одним вызовом `binary_cross_entropy_with_logits`. Поскольку мы делаем обучение по минибатчам, то необходимо усреднить ошибку по всем компонентам минибатча - это функция `binary_cross_entropy_with_logits` делает сама, возвращая одно число. \r\n",
        "\r\n",
        "> Следует отметить, что вызов `binary_crossentropy_with_logits` эквивалентен последовательному применению `sigmoid` и `binary_crossentropy`"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "id": "kdDxWeCqwHl8"
      },
      "source": [
        "def train_on_batch(net, x, y):\n",
        "  z = net.forward(x).flatten()\n",
        "  loss = torch.nn.functional.binary_cross_entropy_with_logits(input=z,target=y)\n",
        "  net.zero_grad()\n",
        "  loss.backward()\n",
        "  net.update()\n",
        "  return loss"
      ],
      "execution_count": 200,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zAAgw0h6KzUd"
      },
      "source": [
        "Для чтения данных воспользуемся встроенными функциями PyTorch по организации датасетов. Концепция датасетов основана на двух понятиях:\r\n",
        "* **Dataset** - собественно источник данных, может быть **Iterable** и **Map-style**\r\n",
        "* **Dataloader** отвечает за загрузку данных и разбиение на батчи.\r\n",
        "\r\n",
        "В нашем случае мы определяем датасет на основе тензора, и далее разбиваем входные данные на минибатчи по 16 элементов. Каждый минибатч включает в себя два тензора, входные данные (размером 16x2) и выходные (вектор длины 16 целого типа - номер класса)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "PfyqjVb2wHl8",
        "outputId": "b3a685a9-304c-4e7e-adf9-2858cc47c3a5"
      },
      "source": [
        "# Create a tf.data.Dataset object for easy batched iteration\n",
        "dataset = torch.utils.data.TensorDataset(torch.tensor(train_x),torch.tensor(train_labels,dtype=torch.float32))\n",
        "dataloader = torch.utils.data.DataLoader(dataset,batch_size=16)\n",
        "\n",
        "list(dataloader)[0]"
      ],
      "execution_count": 201,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[tensor([[ 1.3383, -0.9861],\n",
              "         [ 0.5128,  0.4330],\n",
              "         [-0.4474, -0.2681],\n",
              "         [-0.9866, -0.2869],\n",
              "         [-1.0694,  0.4172],\n",
              "         [-0.8694, -0.8607],\n",
              "         [-0.9894,  0.3734],\n",
              "         [-0.9951,  1.2320],\n",
              "         [-1.3667, -1.0529],\n",
              "         [ 0.5239, -1.9724],\n",
              "         [-1.2474,  0.7092],\n",
              "         [ 1.8675, -0.2057],\n",
              "         [-1.1899,  0.4243],\n",
              "         [ 0.7644, -0.6796],\n",
              "         [-2.0532,  0.4223],\n",
              "         [-0.5973, -1.2011]]),\n",
              " tensor([1., 1., 0., 0., 0., 0., 1., 0., 0., 1., 0., 1., 0., 0., 0., 0.])]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 201
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xrwgkbQjhkEp"
      },
      "source": [
        "Теперь мы можем пройтись по всему датасету и организовать процесс обучения:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "QGchp9D6gVJa",
        "outputId": "b4c4751d-cb56-4104-d5b5-f1ae9d3d858d"
      },
      "source": [
        "for epoch in range(15):\r\n",
        "  for (x, y) in dataloader:\r\n",
        "    loss = train_on_batch(net,x,y)\r\n",
        "  print('Epoch %d: last batch loss = %.4f' % (epoch, float(loss)))"
      ],
      "execution_count": 202,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch 0: last batch loss = 0.6447\n",
            "Epoch 1: last batch loss = 0.6046\n",
            "Epoch 2: last batch loss = 0.5817\n",
            "Epoch 3: last batch loss = 0.5680\n",
            "Epoch 4: last batch loss = 0.5596\n",
            "Epoch 5: last batch loss = 0.5542\n",
            "Epoch 6: last batch loss = 0.5507\n",
            "Epoch 7: last batch loss = 0.5485\n",
            "Epoch 8: last batch loss = 0.5470\n",
            "Epoch 9: last batch loss = 0.5459\n",
            "Epoch 10: last batch loss = 0.5453\n",
            "Epoch 11: last batch loss = 0.5448\n",
            "Epoch 12: last batch loss = 0.5445\n",
            "Epoch 13: last batch loss = 0.5443\n",
            "Epoch 14: last batch loss = 0.5442\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "5QaDiCQUkFOT",
        "outputId": "45b4a66b-1222-40f4-c758-d58f1c7daf8c"
      },
      "source": [
        "print(net.W,net.b)"
      ],
      "execution_count": 203,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "tensor([[1.0563],\n",
            "        [0.4450]], requires_grad=True) tensor([0.], requires_grad=True)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "s4_Atvn5K4K9"
      },
      "source": [
        "Для демонстрации того, как сработало обучение, построим граничную прямую $W\\times x + b = 0.5$"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 283
        },
        "id": "PgRTHttLwHl9",
        "outputId": "d9abf92f-cb70-4c56-ccd0-5e027239da58"
      },
      "source": [
        "plot_dataset(train_x,train_labels,net.W.detach().numpy(),net.b.detach().numpy())"
      ],
      "execution_count": 204,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYQAAAEKCAYAAAASByJ7AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOydd3iUVdrG7zOT3nsIISGh92YA6QgIglhZP+xlLWtvKEp1V0AF17oWLKyu3V0bKCKCUqVIly4ISYBUEtLLZDLn++NmnJSpybQk53ddcxFm3nnfZ9p5ztOFlBIKhUKhUGg8LYBCoVAovAOlEBQKhUIBQCkEhUKhUJxHKQSFQqFQAFAKQaFQKBTnUQpBoVAoFAAAH08L0BxiYmJkSkqKp8VQtCCOFx5Hma4MfeP7Qiu0nhZHofAIu3btOiuljG14f4tWCCkpKdi5c6enxVC0IHZm7cTgdwbj/8b9H2aNmuVpcRQKjyCEyDB3f4tWCAqFo6S1T8OUrlPwwtYX8MDQBxDiF+JpkTzL2bPA1q3AiROAVgv07QukpQHBwZ6WrGlICWRkABs2AEePAhoN0L8/MGoU0K6dp6XzekRLrlROS0uTykJQOMq209swbNkwLJmwBI+PeNzT4ngGgwFYvhxYsQIQAggJ4X3l5YCfH/C3vwGDBnlaSseorQU++ghYtw7w9QUiIqggzp3ja7vmGmDyZL7eNo4QYpeUMq3R/UohKFxFdTWwdy9w+DB/l506ec/mc+KHE7Evdx9OPnQSQb5BnhbH/Xz7LfDf/wIdOwI+DRwFFRVAXh7wxBNAz56eka8pfPkllVxKCi2DutTUAJmZVHQjR3pEPG/CkkJQWUYKl7BvH/DII8CbbwLbtgG//gq89x7w8MO05j29D5k/Zj7yyvPw9q63PSuIJygt5cKZnNxYGQBAUBAQHg58/rnnPyh7KS0Fvv+er6mhMgBoMbRrR6VRW+t++VoIXqMQhBBJQoh1QohDQoiDQoiHPC2TomkcOgS89BItgZQU/g7j4/l3TAywbBnwyy+elXFk8khclHIRFv+yGJU1lZ4Vxt3s2cNF0dfX8jEREUB6OnD6tNvEaha//cbXZE7BGQkOBoqKgOPH3SdXC8NrFAIAPYAZUspeAC4EcJ8QopeHZVI4iJTAxx9zPTHnGvL3BxISgE8+AXQ698tXl/lj5iOnLAfL9izzrCDuJjvbujIA6GfXaOh/bwkUFjIobgshgJIS18vTQvEahSClzJZS7j7/dymAwwASPSuVwlHS04GsLHocLBEYCFRWAgcOuE0ss4zpOAajkkfhuc3PoVpf7Vlh3ElAgP1uE2s7bmeTlQX87380L19/Hdi+nYEoewgKsu81ScldicIsXqMQ6iKESAEwEMB2M4/dJYTYKYTYmZ+f727RFDY4e5abMFuJHEIAOTnukcmyDALzRs/DmdIzeG/ve54Vxp307s3F01p8oKaGysAdhZ81NcC//w3Mng2sXg2cPEm/49KlwGOPAceO2T5Hr/POBFuvydcX6NLFOXK3QrxOIQghQgB8CeBhKWUj205K+baUMk1KmRYb26jQTuFhtFr74pB6PfDHH8AzzwAPPcSElhUrgIIC18tYlwmdJuDCDhfi2c3PQlfrYR+Wu+jcmcHXs2fNPy4lcOYMMG4cd96u5qOPgPXrKVOHDkBUFBAba8qAWrLEdiwjIQEYOJBym0NKnmPCBPe8phaKVykEIYQvqAw+llJ+5Wl5FI5j3FBas94rKpiFtH49rYSgIFNa/JNPArt3u0NSIoTA/NHzkVmciQ/3fei+C3sSIYB776X2PnWK2tlIZSV36N26AVde6XpZsrKAjRvNp4oC9D1qNNwt2OK226gYTp5kPQVARVBSwsK7AQPc85paMF5ThyCEEAD+A6BQSvmwPc9RdQjeyZtvAjt3AklJjR8zGIBVq6gwLrussWupooIb1/nz3eOtAAApJYa8OwQFFQU4ev9R+GptBFxbC2fPMlVz40YunFIyE2DSJO6kAwJcL8MXX/ALYe7LYqS2ljv/l16iwvr1V+4kgoNZPNe9u0mZVFQAmzfznMXFfE0JCcCUKcDQobaD6W0Ery9ME0KMBLAJwH4AhvN3z5ZSfm/pOUoheCfFxcCzz7K2KSHB9BusrWWR2sGD/H2Ghpp/fk4Ouw3cc4/7ZF5xdAWu+OwKvH/F+7hlwC3uu7A3UF7OLB2NBoiLc++i+fLL3L1HRVk/LiODRXKHDtGy8fenZVNdTTfTAw/Ub01hMABlZaYqbFWdXA+vVwhNQSkE76WkBPjmG2DTJv42heC/lZX8LVvb/dfW0pPw6qvuq2qWUmLgWwNRUVOBw/cdhlajOqG6haVLWUMQF2f5GCmBH38EEhMZPG7oWsrLY7uNp54CIiNdK28rQVUqK9xKWBhw883cAD75JDBzJvDii9zkRUdbf64xndzoBnYHQgjMHzMfxwqP4fODn7vvwm2dCy6gm8caWVncYfTsaT7OEBfHx3/+2TUytiGUQlC4lOBgoGtXxigjIugmspVaLiWtCXe4sOtyZY8r0SeuDxZuXIhag2pv4Bb69+fuobjY/ONS0seYmmq98Cw+Hli7lqmliiajFILCrYwcSdeuNQoK6BkIC3OPTEY0QoN5o+fh8NnD+PLwl+69eFvFz495x+XltATqpqeVlTG+EBHBHYU1/P2501BVyM1CKQSFW+nXjy4jSynwNTXsU3bppe6Vy8i0ntPQI6YHFmxcAIM02H6Covl07kz/f1oalcKpU+xMCjCV9MILbRe3GLOk3FlZ3QpR757Crfj7A48+CixezN98XJypk0JeHjd5N9xgKjx1N1qNFnNHzcWNX9+I5UeW46qeV3lGkLZGYiJw113AddexAZ2PD78cWi13CZ99Zr0fSkkJs43cbVa2MpSFoHA7iYnA008DV19Nr0BGBtPMBwwA5s4FJk70rHzT+0xH16iueHrj02jJWXgtktBQ1iQkJJhiBkOH8u9KC11ppaTJeemlKr20mSgLQeERIiKAqVP5G66uZuq7Pc0q3YGPxgdzRs3BrctvxXe/f4fLul/maZE8h7F4ZO1atn7w8+MCPWIEe5m7g4gIDrZ57TVaAFFRpoW/ooLdW0eNAoYMcY88rRhVh6BQmKGmtgY9Xu+BqMAo/HrHrxBtcedZXs5F+NAhFneFhFBBGIvY7riD/n13ceQIB9wcP87dg8FABXHppey75C07ihaApToEZSEoFGbw1fpi1shZuPPbO/HD8R8wuetkT4vkXqQE3n6bg+pTUuq7YkJC6L55800WgnXv7h6ZevRgR9S8PKap+vkxbqACyU5DxRAUCgvc3P9mJIcnt81YQkYGOxAmJZn3ywcG0t+/fLl75RKCNQfdulFROaoMpGTzuw0bgHXr6A4zqGwyI0q1KtyOlKZ2++YKT70FP60fZo2chXtW3oOfTv6ECZ0meFok97F1Kz8ga66y6GguqAUFtsvPvYFTpzi/NT2dX0Lja4uJYXpr794eFc8b8OKfo6K1ce4cuxg/8ABjhHfdxbkoGRmelswytw24DYmhifjHhn+0LSshL49WgDWMYzZLS90jU3M4cwZYtAjIz+echdRUWhgpKbQQ/vlPz4/w8wKUQlC4hcxMYN484Ouv6YLu2JHNKbdtY03Spk2eltA8/j7+eGLEE9icuRkbMjZ4Whz3ERpquw2EscdISxhJ+fHHVGCxsY2tHmPm0rvv1p8N0QZRCkHhciorgRde4GayY0dTjyIfH6B9e6acL1vGCWreyB2D7kC7kHZYsHGBp0VxH0OGAFVV1o8pKeGHV7fttDeSnc1MKWsdVUNDWRB3+LD75PJClEJQuJxdu7h2WGp57+9PJbF6tXvlspdA30DMHD4TP5/8GZu3fEbXQkaGfbNCWyo9e1Jb5+WZf7y2lsVg5qYceRvZ2dyN2JJTq/Vu/6UbUApB4XI2brTdUSA2llPWbG1KPYLBgL8VdUacIRALVszg5K6//50pkK21DkarZdO5gAAuksYqYWNVcEYG8//dWYfgalqzgrcTpRAULqekxLab2VhTZKs1ttsxGID330fQJ//DY/oh+DEwC9uSBAfCV1dzis+PP3paStcQH88Az7RprAjOyOCtY0fg8ceB6dO93zoA2CvFGO+whsHAYHMbRqWdKlxOdDSDykFBlo/R62nV20pscTv79gHr1wOpqbhHk4TF8lcswEasFNfT7AkIAD79FOjbl/701kZ4OHuMTJlCK8HHp2UEkesSHw/06QMcO2Y53lFczC9qjx7ulc3LUBaCwuWMHWs7MzE3l+1xvK7odNUqLooaDULgh0cxDN+LY9iFLD7u50dNtqGVZyBpNJx21NKUgZEbbuCXKze3vmtISuZDFxczD7qNt79QCkHhcvr358YsN9f846dPMwnkxx+Bv/4VeOIJbsptTVZ0OdXVwO+/14uG348hiJABWICNpuOio4E9ezwgoMJu2rUD5syhuyszk8VpRhdYSAgwa5b7WnB4Md62H1O0Qvz8gBkzWPuTns7mlcHBXG/37eNvsl8/oEsXbkTLyoD332fW0eOPW85OcjkGA33kdfzkYfDHI7gQT4n12CtzMADtKLQa3ej9JCRwwPeZM1QKUtKd1KlTy4iFuAFlISjcQmwsE3PuvJP90IqLmdGo19NF3acPrXUhmBKeksKOCP/6lwdbzQQEUNjy8np3P4ihCJP+WGi0EoqLKbCiZZCYCAwbBgwfzmltShn8iVIICrcRGMg4wVNPcaFPTTXNWDdH+/bsQ3bsmHvl/BMhgMmTG+XiRyAAD2IIvhSHcUDm0rc1oQ31OVK0WpRCUHiE3Fxa7damIgJ0N23f3owLSQnodE3PMR8xgi2Wz5ypd46HcSFCpB8WVfzAWcAO+p9rahh2+Ppr4JtvgP3723zXBIUXoGIICo9QWWlyEVnD358dBRwmP5+ZP2vXstrN15fpThddRNPDXoKCGMhYupQDWjQawM8P0Tod7g/vjsURB/DUtDHo4UDb1t272dSvrIyJL1Ky8DciArj77jaf+ajwIEohKDxCcDAXwbpdiM1RVdWEzsq//87mSXo9+9f4+3NLvm4db/fdBwwcaP/5IiKY+pSZyW19aSkQHY1Hez+GVz8ejEXbn8eHV31o16n27gVeeYViNZxAWVICLFnChJeuXR14vQqFk1AuI4VHiI1lcse5c5aPMc5NcKg7QlERW0sEB3O4izFv3teXrp+oKOD119nfxhGEYMrilVcCN90ETJmC2I69cE/aPfhk/yc4VmA70KHXM3sqNpbiNSQsjAH1//xHdVFQeAalEBQeQQjgiiuoEHQ688ecOUP3SadODpx4yxbms1qKVBvLpdevd0Rcizw2/DH4af3w7OZnbR575AgTkkJCLB8TEcG6jPR0p4inUDiEUggKj9GvH3Drrdysnz7NdVyvp5I4eZIb+nvvdTArcP162z6m+Hge54RteLuQdvjbBX/DB/s+wMlzJ60em5Nj+5LGmTOWivgUCleiFILCo4wfDyxYwHhvWRljwdHRdPPPnm27S2ojSkuZmmQNX19qn9rapopdj5kjZsJH42PTStBo7NNBUnr3aFFF68WrgspCiH8DmAogT0rZx9PyKNxDhw7AjTfy1hDjAmq3lRAVZcoqskR1NV1HTupb0z60Pe4YdAfe3vU25oyag44RHc0el5LC12EtkC4lb8nJThFNoXAIb9uHvA/gEk8LofAsNTUcM7BgAXD77axufuUV9juyucMePx4oLLR+TF4ecPHFTq1QfWLEEwCAxb8stnhMairj0mfPWj5PTg4bp3r7EDJF68SrFIKUciMAG79mRWumshJ4+WWOGTh7ltZDQgJw9CiweDHw4Yc2WlkMGcJqt4IC848XF9N6GD3aqXInhSfhtgG3YdmeZThTcsbsMUIAd9xB+XNy6r8Og4FB9IAAJjEpFJ7AqxSCQvHhh7QEUlPZRkijoWcnLo6767VrgTVrrJwgJAR47DE+KT2dyf01NQxQZGTQXfTYY00obrDNrFGzYJAGLPllicVjkpKAuXPZyC8z03Q7dYqWwdy51kf/KhSuREgvS3gWQqQA+M5SDEEIcReAuwAgOTn5gow2PgO1NZGfD8ycyUXTUlC1spKtg1580XqYAGVlwI4dwE8/MW0pLIxVykOH2u6X0QxuX347PjnwCU48eAIJodYH5uTm0lIQgv3WXKCjFAqzCCF2SSnTGt7vVUFle5BSvg3gbQBIS0vzLm2maBb79vFfaxk2gYEMARw/zjnwFgkJoQK46CKnymiLWaNm4f197+OfW/6JFya9YPXY+HjeFApvQbmMFF5DSYl9E9OE8ILhORboEtUFN/S9AUt3LUVeeZ7tJygUXoRXKQQhxKcAtgLoLoQ4LYS43dMyKdxHeLj9c2bMtX7wFuaMmoPKmkq8uPVFT4uiUDiEVykEKeV1UsoEKaWvlLKDlHKZp2VSuI/+/bn7t5ZFVFlJZdC5s/vkcpTuMd1xbZ9r8dqvr+FshZUcU4XCy/AqhaBoXZw7x5YUtsoCjMTEcPyAcbqhkaCKs+h29FtcsO11dNvwNm7svgO+hmrXCO0k5oyag4qaCry87WVPi6JQ2E2LCyorvJ8jR4AVK4DDh5n9WVvLds5XXAH0TigEsrJ4YPv2jQYm33gjYwl79wIRobUYnv0luv6xClXVAhUyCKOSDOiyYwtwJAR44AGvHYzeO643pvWahle3v4oZw2YgMjDS0yIpFDbxurRTR0hLS5M7d+70tBiKOmzYACxbxizP6GhTqwbdmXx0/+1/uDhyJxI7nK8QNhiACy4Apk9nT+jz6PXAgQPAiSX/Q8KOFSgMTUFcghadOlF/CAFqjZISYN48Fih4Ifty9mHAWwPw9zF/x1Njn/K0OArFn7SatFOF93LmDPv9JyaaxhAAQEh5LkbtXwSNrhI7czoguKcGERGgQti3jwNt5sz5MwfTxwcYkFyIAQGrgGtTzPccCgtjQOGrr4BHHnHHy3OY/u3648oeV+Ll7S/j4QsfRniA6+ofFApnoGIICqexfj3X7rrKAFJi4N73oK3VoSIiERofDU4au0RrNNQeOh3w3nv1AwfGQcrWGtDFxgK//Wa5TYUXMG/0PBRVFeG1X1/ztCgKhU2UQlA4ja1b63l+AAChpVmIPnsE5cHc/QcHM9Bcj/h4Bh6MsQWAvRwCAqxfUKPhzd6otQcYlDAIU7tNxYvbXkRpdamnxVEorKIUgsJp6HSNC8siitL5x/nOokKYZin/ibHraN0xYf7+9s0rkNK+ajZbVFQwLara+dlL80bPQ2FlId7Y8YbTz43CQr5vWVk2uv4pFLZRMQSF02jXjvNp6rYKEtIAH30Vos7+Dj9dGaqlL/xD20EgBkCD9tN1F7QBA4B166xfsKqKvSwSE5su9NGjwKpVjGUIQRfVqFFsj51gvReRvQxJHIJJnSfhha0v4P4h9yPYzwlVdX/8AXzzDaPvWi3fu9hYYOpU5u6qCTuKJqC+NQqnMWlSfe+NprYGSZmb0T5rJ2LPHkZoWTbCC9PRt/gX4OefqT2MCFHf39S7N1OKzp0zfzEpOXtz0iTbE9IssWYNsGgRg9pJSZxKEx8PbNwIzJ9PN5aTmD9mPvIr8vHWrreaf7K9e4GFC6kUkpLYIzw5melZb78NfPSRU8aDKtoeSiEoHEJKZhPt38821XV7Cl1wAdemrCwe2H/vfxCbfwgVQbHQ+wSgHMGoDQpFUPsI7u43b2amUEkJc1S7dTOdzMeHdQZVVTxhXfdRZSWHLvfpA1zSxHlKx45x4ezQgf2mjTtqHx/eFxbGqTwlJU07fwOGJw3H+NTxWPLLElTWVDb9RMXFwBtvsIqvrtwAEBrKvuFr1gC7djVfaEWbQykEhd0cP84N9dy5wEsvAS+8ADz0EPDxx1QMAQEcNZCYCJQezETs75uRE5iKE2EDYKiohL9Gh+Tk84lDwcEMOhw8yCyhW25p7OZITQWeegoYNIha6PRpBpsrKoD/+z9evKnWwQ8/cIymuedLyfuLixkpdxLzRs9Dbnku3tn9TtNPsn07LQFLzZw0GlpWK1cqK0HhMA4XpgkhggFUSSmdM6G8GajCNPdx4ABnEAQH1ykOA5vRZWVx7X78cbr0DQYg+/kPUfLdRhT4JyIwEEgNyEZE+h5oanSmhb+mhge/9x4nnVmjuJj+KK2Wvn2rwxBsUF0N3H1348ELRvPn+HGgqIgLb0AAMHs2MG4cWDzRPMa8PwbHC4/jjwf/QICPjSwqgJZRfj5liYykJi4osD7TQUr2/3jlFVo6CkUDmlyYJoTQALgWwA0ABgOoBuAvhDgLYCWAt6SUx50sr8JOioqAbdvoVtbrOYlr1Ch2hXDWyOCqKuDNN6kIQkLqP+bry0LhEycYm7366vPlBbp0JA4ORbmWa9OerAQYtHFI8M1FUug5BAVKFBgiUVlSg9+PpqBzFBvWWZQ5PNx5g210Ol6ooTLYu5euqMBAXkuvp9L67jvGFZ58stnDjuePno8JH07Ae3vewz2D77F8oF7Pa65cyTiKUdbcXLq0rL0Xxtem1zdLVkXbw54so3UA1gKYBeCAlNIAAEKIKAAXAVgshPhaSvmR68RUmOOXX7i5rq3l+qDRcMTk6tWcNX/dddbruuxl3z6gvLxxjUFdEhKAH38ELr30fGGary/OZNZi5x98PDAQEFot0qvbY/ex9tDpgKhIibjqU/hulRalP3HDfu+9TkvusUxgIF1COp3JZZSeTmUQGVnf/AkPZ8A2L4+DnhcsaNabOi51HIYnDcezm5/F7YNuh5/WjMtKr6cG/vVXBrmTk033Hz3KYPLEidTO2dm0GKSkxm7fnn/7+jbW3gr7qa3l1D2tlmaxs3ZXXo49CmGClLJRl3opZSGALwF8KYRohv2uaAq//Qa89RZ//3Xrt8LC+F1evZoL8zXXNP9a+/dzDbWGvz/X1+xsICUFOBmdhlN7PkZwQmS9MoGSEv7OhABQVgqZGIf41CD0zdmG2LUbsP/7IoROiUDI1LHsh22rOK0p+PhwktqPP1ILGQzMNAoJqf/D1+mATp34d1wcZzL//ruNUW3WEUJg/uj5uOTjS/Cfvf/BnRfc2fign36iMujUqb48Pj5Mx127ljdjrYbxDc7I4BcjPh647bamx1faMqWlbMi1ejVjVQYDFfKUKUBamnN2WF6MzaCylLJGCNFDCDFeCFFvyyGEuMR4jKsEVDRGSuDzz7khNLdearV046xaRdd7c9Hr7UtrN84ykBL47MQQaAN9ESBNaUh6Pd3hAQFAgL+Eb1E+jkYOw7h1czF411IkGM7AUFOL9C1ngKVL2bguO7v5L8Ac48ZxF11UxEWgqqr+AlpaSu1ad+K9ry/ghJjVxM4TMbj9YDyz+RnU1Db46ej1wPff00wytyuNiuItI4MWTEQEFVlICP+Wkv47FTtwnIICWoBffMGEA2MqcnEx8NprwLvv2lcs2YKx+TMXQjwIYDmABwAcEEJcUefhZ1wlmMIymZkM5FpzI/v4cHHevbv510tNtT2y0lh9HBPDwfFHs8Owf9jfEFyeh8AKujSMGZwBshLtq0/iQNBgFBzJh09NFYojUlAVGAW/iCD8URiF6nYdedHnn6dJ4WxiY4GZMyl4ZiatAZ2O1ywq4gI7bFj9HaGvL31nzUQIgflj5iO9KB0f7/+4/oNZWVRGQUGWTyAlU0zPnaPJVVnJ96ioiKbcRRcB335r//g5Bd/TN97g4p+SYjKJhaCi7dSJPtoff/SomK7GnrTTOwFcIKW8EsBYAPOEEA+df6xtONa8jOJifk9tuTX9/Oj6bi5Dh/Ja1taXnBzgwgu5MS0t5Tqak3gBNo+ajdLQBIQXZyL43Cm0r82Ev6zC2rjr8XPQVJRWalERbApOGC2Rigpw0T53ju4TV9CpE7B4MVNe/f158chIvpAxYxr7yaqq6lsMzeDSrpdiYLuBWLRpEfSGOsFfvd76B3vuHN+cxEQOmejUiTK3b08FNn48XUZlZUzpVdjHyZOMzVhKGhCCVtv337dqRWtPDEEjpSwDAClluhBiLIAvhBAdoRSCR7A341Kvt77RtJeICGYPff45LeiG1z97lsrnivO2o7+/KQW+ILobNo+ajeCyXBzfVYycAl+URSahVvgg9PA2+ISb93P/uSZGR9OfO25c81+IOYKCgMsvZ15tdjavZw4paU1ceKFTLmu0Eq76/Cp8duAz3NjvRj4QEUHTzmAw76czmmp6PZVTjx6WLuCc3UBbYe9e7mKsKePAQPo8MzKYztcKscdCyBVCDDD+57xymAogBkBfVwmmsExqKhdlnc7yMVJyTenTxznXvPRS4PrrmfVo7KV26hT/Dg0FZs36c5wBOnTgulbX01MeEg99p27I0KSiVvhQwJoadA6v37par+fv8s8EmaAg97S3njaNpk2lmSpiKbkIXHghd+JO4vLul6NvXF8s3LgQtYbzvumoKKBfPy485jAuWFLyjbaEMdNIYR9lZfa9X0K4pAGit2CPQrgZQE7dO6SUeinlzQBGu0QqhVUCAtjC58wZy8WoubnM609Jcc41hQAmTwZefhm4/XZu2C+9lIpg0aL6a5NWy013bm79fnXt2jG2odMBZeUC8QElSPar99VCWRnl/jMzSadzTaZRQ7p3B+67j8VvGRn0zZeX02rIyGBfjttuc+olNUKDeaPn4WjBUXxx6AvTA1ddxdddaqZddng47+/Y0XJaqfFL0bmzU+Vt1cTGWt9hAaZdVisO2Nt0GUkpG3avr/vYL84VR2EvU6dyd75nD3fmxrWhupr+/Kgo4J57nJ8+HRLCwjdbjB7NWO3atVzDoqK4AevThy2MwsOBYQOrEJKVh9LwDjAYuAZHRDSwxvPzqf3qUFREN5VWyw17vYE8zWHwYPrlt24FduzgApGWBowdS6FckIs+rdc09IrthQUbF+Ca3tdAIzTU4o8/zrqHggK+6VotFZRez1iBtRedl0d5k5KcLm+rZfBg4L//teyqAxi869DBumXWwlEzlVswNTXApk2McxUWcr3y9WXn5gkTnFfY21SkZFHb99+zlxzAzVXPnlRmVSezcfWeuSjzj4HeN2cw7loAACAASURBVBAdO/KxP7M/Kyu5IC5cCLRrh+xsdnzescO0Nhtf75QptmslvJVP93+K67+6Hl9c8wWm9ZpmeqCykhp/504qp44d2do6MBB49llqxfbtTeZUbS0tmuBgttsw+vAU9vH++2y5npLSWPlXV9MknzGD9TEtHEutK5RCaAUYDEw+MRioBLyxHqmqyhTk1mioLE6fBio2/IrYL95EaJQffBPjuBOureUuV6ejGyctDZmZXANra7nOGbNBq6sZz+jcmY31nBFEdze1hlr0eqMXAnwCsOdve2gl2KK4GFi+nO0tjL9hKWk9XHUV838VjqHTAf/+Ny3EgADuXow/LiGAm29m9lkroNkKQQghwH5GnaSUTwshkgG0k1K6KCfQNkohtBLS05nfvX07FzUhGMCdOBHo2BEGA9sIVVZaTgJKT6dn6brr3Cm48/hg3we45Ztb8M30b3BFjytsP8FIWRmtAimpKT1tFrZ0pGT66fr1TEX18WG33REjWpWSdYZCeBOAAcA4KWVPIUQkgB+llIOdK6r9KIXQyqiu5qofGFjPR374MMsFrAXIa2oYbnj55ZZpJegNenR/rTsiAyKx484dEG2kd47CM1hSCI7MQxgqpbwPQBUASCnPAfBC54SixeLvz6hyg4DpwYO2xyb7+tIllZHhQvlciI/GB3NGzcGu7F1YdXyVp8VRtFEcUQg1QggtAAkAQohY0GJQKFyKTmdfTzEhWnarmZv63YSO4R3x9Ian0ZJje4qWiyMK4VUAXwOIE0IsArAZqpeRwg0kJtqfIm4pxtAS8NX6Yvao2dh+ZjvWnFjjaXEUbRC7FML5gPJGADMBPAsgG8CVUsr/uVA2hQIAY3parfUWMgUFLCFw+SwFF3NLypXo4BeDpz+4HfLRR4DnnjPVRCgULsYuhSBpv34vpTwipXxdSvmalPKwi2VTKACwNcbVV7PQzZxSKC1li5/p090vm1M5dAj+c+bjyezO+EVzGusDcphT+/rrwD/+4Z4WHoo2jSMuo91CCJdmFAkhLhFCHBVCHBdCPOnKaym8G73e1NkZYNuM6dNNnSTy8019laqrWdjbojs1ZGVxXnJwMG6PmoAEGYKn/baxxDslhUVoL73UqjttKjyPPd1OjQwFcIMQIgNAOdjpVEop+zlDkPMB69cBXAzgNIAdQogVUspDzji/omVQUAD8/DOHhul0jA307k2FcOmlwMiR7IZtTBHv25eFo05rX+Ep1qz5c85BAIAnMAIPi9XYKDMwGh3pC0tPZ1fWgQM9La2ileKIQphk+5BmMQTAcSnlCQAQQnwG4AoASiG0ETIzgSVL6P5p144V11Jy8V+8mAW4V17JerVWhU7HHiR1evHfiQvwjNyMBdiINbiJd4aEsLWCUggKF2G3y0hKmWHu5kRZEgGcqvP/0+fvq4cQ4i4hxE4hxM58Sy2CFS2O6moWlWk0nLlgbL8hBBtRJicDX3/N3kitDuPs3jrFFkHwxeMYjrXiBLYafxZBQZbbYisUTsBuhSCEmG/u5krhzCGlfFtKmSalTIuNjbX9BEWLYN8+toyJijL/uI8Pa9a++85yy+8Wi3GikKF+Wc/dSEOMDMICbOQdOp3lltcKhRNwxGVUd5hsADgkx5mZRmcA1O3X2+H8fYoWRmkpMyWPHOH/e/Rgd+HQUMvP2bKFTTqtERnJNjNFRfy71RAYSDfQ4cP1OpSGwA8zMAyzxE/YIc9gcLGeg3y8BZ2OkX0pWQBi6wNUeD12KwQp5Qt1/y+E+CeA1U6UZQeArkKIVFARXAvgeieeX+EGNm4EPviAFcPGnkI7dgCffALcdJPlZpHl5ba7tApBl1KrHFh1ySXArl1cZOu8EfdhMJ6XW7Cg5iesCLmUg3o8TWUlg+CrV/Nv44DvUaM4qMMZ1YGFhdwl/PYbv0zduvH8TpxYp2iMIxZCQ4LAXbxTkFLqhRD3g0pGC+DfUko1JbwFsW0b8M47nB/SMOunuhp4913eb24scVwcR3JaG0ZlbEthzdJosXTrBtxyC7Wpvz/fEK0WodXAw1U9MT9iN/bcMgkDPd25r6ICePFFDrhISGCAB2Ce8MaNwO7dHKNnaVi9Paxfz/cBYPdWIaiAfviBLW3/7/8sD7FRNAtHYgj7hRC/nb8dBHAUwCvOFEZK+b2UspuUsrOUcpEzz61wLXo98OmnXAfMpYD6+/OxTz/lsQ0ZPZozE6zFB/Ly6HpqqmdCSs44OXSIrievS+kfNw6YNw8YMICCZmQARUV4YNhDCPcLw8IT73laQuCrr/jmpabWH23q48MJbTU1wJtvNj3Qs3MnZxLExzOTIDycuwTjpLKVK4EVK5zzWhSNcMRCmFrnbz2AXCmlmZ+2oi1y9CgLyTp2tHxMcDCTZI4c4SjNusTHU2ls3syNZ3x8fUugrIyKZMqUpsl34ADwxRdcY40DeoKDeb6JE+1rnucWOnfm7NM77uDi6u+PCK0WD637A09vfBr7c/ejb3xfz8hWXg5s2GDdbRMbyzf5xAnHKwUNBo6xjI01v6vQaqkkvvuOIwFVgN3pOGJ33Vsn3fTMeRfPYpdJpmhRFBTYvyksLDT9rddzDZgxg4t+cTFdTytXcu3Jz2d9QlkZ8PDDXA8cZcsW1jecO8fnJyXx36AgWixvv+3mLql6PQdfZ2XRLDKHry8FPK+pHrrwIYT6hWLhpoVuFLQBJ0/yjfL1tXyMMdBzqAnlQydO8AO35hM09jlvlfnHnscRC+FiAE80uG+ymfvaBBUVXASFoLvXG8dWuhNb8wrMHSsl8NFHrExOTub9nTuzS8OJE2xTsX8/R2OOGdO0YWCFhfRAtG9f38MBMLknNZUKY+BA87ENp1JVBaxdy2Bs+fmkPV9f4KKLGFSOiLD41KjAKNw/5H48t/k5/H3M39EztqeLhTWDXt941rA5tNqmRf6Li+07v48P/YcKp2PzZyyEuAfAvQA6CSF+q/NQKIAtrhLMWyks5ND4DRtMaeP+/ox1TZzYcge9Nxejd8BgsBzvM75fxmNPnmT8MCXF9ByNhgo2Ls50TEBA0ydDbtnC6zZUBkaEYFLMypXA0KE21qPaWvq7fvqJZoufH580fLgpuGqJykoGY48epU/MmImj01FB7NjBYKyVMY2PDnsUr25/FYs2LcJHV39k/XquIDKSb6ZxzKklamqaFlS2d1dVN4VN4VTscRl9AuAyACvO/2u8XSClvMGFsnkdeXnAggXc0cbF0fWQlMSY11dfmdoutEXi47nLzs62fEx2NvsOGVPt16/nGmAtYSQuDli1qlHNlt3s3m114w2Ayub0adZPWKSyks3lliyhUtBquZgvX86Bz1ts7I2++AI4fpwmSd1dg58fAy/l5fRdWfG7xQTF4N7B9+LTA5/i94LfrV/PFSQnM7BbVGT5mJoaWj1Naa/RubPtPudSmhpcKZyOTYUgpSyWUqZLKa8DUAIgHkBHAH2EEKNdLaC3ICXw1ltcF5KT67tRAwL4O09PB/7XhidE3HwzN5EN21TX1PC+yEjWIhQU0C109KjtnX9wMD0J5eXWj7OEXm87Q9Ho9rYYR5CS+bQHD9KciYvjoh4Swi9DbCywdCkLy8xRVkaTMjHR8s46Ph74/Xfm3lphxrAZ8Nf645lNHphNJQRw7bX8QMztfPR6ftBXXdW0VLCgIGD8eGZYWVKM2dn8sR06RIvqnnuAJ56glVVc7Pg1FfVwJO30DnBIzmoA/zj/799dI5b3kZHBbLs6haSNSExkKrbVnWYrJjISmDMHuPhixgZPneItP58xgGHDgEWL2Kp65kx6Xo4ftz77xbghbOrM+ZQUZj9Zo6qKSt1iLPP0aZoaycnmBTH6tL76yvxCduIEtY21QIuxuMuSUjlPfEg87k67Gx/99hH+KPzD6rEAuHAfOUJllptr+3hb9O4NPPAAF9+MDGr2ggIqgqws4Jpr2Jq2qVx1Fa9x4gQVqZGqKu64NBp+Hp9+yp1GdDTf288+A2bP5vMUTcaRoPJDAAYD2CalvEgI0QNtaITm0aOm36wlfHy4Hpw4QddIWyQ8HLjuOnYlNcb9QkO5gf7pJ1N6OcB15MAB/jtihPlMw5ISKtqm1h6MHWuK91iyFHJzKa/F9XrbNj5o7cOPimKxVn6+KQBipKaGz62sNLlUzAWbtFrLWUd1eHz443hjxxt4dvOzePfyd80fVFFBd9bPP5v8bQYD0LMnC7tSUmxexyIXXED32a5dpkriTp0YS7HUjMpe/P2Bhx5i99dVq6hohKDSnTiRfsagoPq9S0JCeCsqAv75T+46WlVvE/fhiEKoklJWCSEghPCXUh4RQnR3mWReRlWV/cWR5gqv6lJYyN9SXh6/2/370wpu6i7YGwkMNNUk/Oc/XCsbvsYuXbjJLCkB9uxpnOUjJTegf/lL09+b1FSuU7/8Uj94bTx/bi7XsLFjrZwkL89yVNqIEFQapaWNFUJ+PhdOo//KYGDwuHv3+sHo2trGzzVDQmgC7hx0J5buWop5o+ehY0SD4o+KCuD557mjbt/e5N+UkvctXEg3S9euNq9lkaAgtpIYNarp57CEnx9dRxddxFxhKbnTWLuWCtVSHUREBBXIL7+whYbCYRypQzgthIgA8A2ANUKI5QCc2f7aq4mPt52rbmxYaWlzotcDH3/MnPtPPqF76bvvgKef5m/07Fnny+1pSkr4Os25z0ND6R2oraUXoK6HQKfj2pWWxkSepiIE8Ne/sgj41CmuF7m5vF5GBpNhnnzSRiwjLMy+mcbm0pl+/pkfusHAhbm2lgGR48fp9z52jMfpdFQodpqWT4x8AhqhwXObn2v84Lff8s3r2LF+sMuYIx0aCrz2mheWajdAo6FLKCaGr+Onn6xmYQGggl271j3ytUIcaW531fk//y6EWAcgHMAPLpHKC+nXjxuXBr3H6lFSwozC1NTGj0nJnfKGDfyd1q2MNbZUWLyYnQus9fNpaRw71qjVfz26dKGXYPt2uo+Mmz9fX+Dyy3lzpMbBHL6+wK23cuLajh3c8AcHMxGmc2c7rI+0NC5G1igt5WKVkGC6LzMT+PBDZuYUFgJbt/LFGF9QRQX78+j1/FJdd53d6ZQdwjrgrwP+imV7lmH2qNlICj/fKLiykkqorhwNCQ9vmdPXioqsvy6ACjkvr3mBpzaMI0FlIYS4UQgxX0q5AcBeAANcJ5p3ERTEQe+nTpnfWFVW8jd/3XXmv4cZGdwpp6Q0bpMgBHeq+fl0kbYmbG1ChWBMYfBgxiPvuw945BEOy5k2zXpRrKPExrJVxa238lpduti5ZnTvzvxiS0HZ2louQpddVt8ntW4dF//sbH5xkpL44RsMvLCvLxeudevol3cwGPvkyCchIbHklyWmO0+f5ptuK6ff35+B5pZEaKhtS02no7ZXyqBJOOIyegPAMADXnf9/KTgDuc1gbLSYlcXNnzF9MiODrs777qMlYQ57cu7btaMXwVYMoiXhSEFZv35cF/v29bICP40GePBB7goyMqj9AVOQIyODi/nIkfWft20b/YcHDtDsi4ykSRIfz51sQAD/jolpUgZQx4iOuKX/LXhn9zvILj1fAGItet7wNXm7y6ghF11ke2JcXh79g4om4YgxPlRKOUgIsQcApJTnhBBtqmGDENwEjhhB6/+PP/i76tOHXgVrvbZOnrS9OBqt3dLS1pMk0bUrX3d5ueVMIZ2OG9aGDe+sodczFX3tWm6+jUXDI0faFZd1nNhY4KmnmP2yerUphapHDyqDvn0b70qrq7no6vUmN5GPDz9c4wcsJX2N2dlNagg3a+QsvL/3fTy/5Xm8OOlFymkw2FYMVVXWOxF6IyNGsKS8tNR8jnBZGS0wVwS62wiOKIQaIYQWgAQAIUQsgCbWj7ZsoqLoj3YEo6fAGsac+9bU6t3Hh1lCb71Fj0lDT4ZezwX9+uvttwrKyxkTPXSISjg0lF6blSt5++tfuXY4nbAwfvCTJ9NK0GqtZx+1a0chrbkvjGMxhaDP0UGF0DmqM27sdyOW7lyKJ0Y8gfioeAamDx+23D6ipoZfsrQ0h67lcaKj2eHwpZcYT4iL407COLlNo2HKqhqt22QcWXpeBfA1gDghxCIAm9GG6hCay6BB1iv+AVNQujUFlQEuzjfdxAafmZmsaSopoSI4fZo1AJdcYt+5pGSHh6NHGY+JjeWaHBxMhRMbS+Vjo76reWg0vKCtVNRJkxg4ttYGtqLCFMxoYg/u2aNmo7q2Gi9sPT/U8C9/4TXPnWt8sE7HD2HatJb5RevZkyl5U6bwB5WRQUU6aRLvd8TMVDTCnuZ2H0opbwIQA2AmgPEABIArpZSu/Nm1KoYNYyGrsSq2IUZ39N/+1vriYUKwpmjQILb82b+f1tKwYbTuHemDlpEB7N1LZWCtaPjrr7l2eJQhQ0zFFsbJX3UpLeWiHB9PF1SnTk26TLfobpje61q8tv0NBO+dibz0RMRWPYmr019HSkg6QiN9IbQaKgOtluaYvRrYG4mLo0K7+mqahlpt6/vReAh7XEYXCCHaA/grgA8AfGp8QAgRJaUstPhMxZ9ERwO33caWOFFR9deHqiq6kIcM4SLZWomJMaWSNpVt25icY2/RsEe9B4GBLDI5fJiZCKGh9KHp9byFh7MaLz+f/9rqwmeB6mogOWMOKvWfYkXJS7gkYhF04al43XcxYvIP47Kwgxg6sAYiqYPtYFdLwlgMqHAa9rybSwH8BKATgF117hdgPKFp25o2yKhRXBO++IJWu3FyV0AArfzJk71ocpeXkp9vO9ZgbFZXVuYF7uSYGFYhzpzJoLG/PxfkpCR+GXJyWKdw3XW2z2WBjz4Ccvb3Qq+Ua3Ag6F+YgBkIRBQSOmihb9cHb2T2QXUy+0kpFNawqRCklK8CeFUI8aaU8h43yNSqGTCAMb9Tp+hH9/Oj+6OlDNipqWEW5ZEjdPt07szXZMud7izsKRo2Voyb643kERISWFjx7becEQrQLDTOBJ0ypcm79rNnecrkZCAQc3FI/Bfb5Cu4CP8AwA10u3Z0oY0cqTYcCuvYE0MQklhUBsZjnCta68VYjNXSOHCAAduyMpPbZs0aLry33MKeQa5myBDWcVmjtJSLoK2iVrcSHc2KuL/8hWaOsRqxmZp0717+q9EA8eiLnvJqbMcrGIZHEAC6oIKCTFPoUlPZNaO0lJ9bly5q1ozChD0uo3VCiC8BLJdSZhrvPF+DMBLALQDWAXjfJRIqvIKjR4EXXqB/vmH6emUlu5lqNK4fQ2ksGs7JMR+Mrq3lenvvvV4aZzR25nQS587Vd6OPwlwcFl9hu/wXxmDen/dLyUr51183TaqUkor94ouBK65oOVaqwnXYk3Z6CYBaAJ8KIbKEEIeEECcAHAOrll+WUr7vQhkVHkZKtuQJCzNfDxQYyMX5o49cX/xqLBoOCWE7HuOcFoOBiiAjg8WDLp+P7CWEhdVvupiAgegmL8M2vIRqmAZznDrFbti+vlToycn8NyaGDRbfeKN1VcgrmoY9E9OqpJRvSClHgJPSxgMYJKXsKKW8U0q5x+VSKjxKejrrBawlwQQF0ZXkzPY4paXs0/bqq8Arr7DorLDQVDQ8fTozbDIyuOB16sTY7TXXeKl14AL69zeNOTYyGvNQJc5hx/nOMnl5bJ7Yq1fjanFjDGvXLjYYVLRtHMrZklLWALAyNVfRGjG6vG0tshoN02cHOKHl4ZYtwHvvcddqLOTdt48ZWn/5C+OwkyezHqm6msHStujyaNeOmaS7d9OVJgSQiMHoIidjK15AmuF+7N8fguRky+EKIWgpfP8940BtRZkqGqOSeBU2sTczxWBwThbLnj2MSbRvX38Ri4qiS+qzz7j4X3wxlZBbGuHl5XHwyq+/Ms0pKQmYMIHVbx5O3bntNsYFjh411biMkvPwnnY41pxbivYRj6G7jVFWYWFMha6oaPp0OkXLpxV1zVG4CuO0RWu9mIwuiy5dmnctg4HjcmNizO9ofX2Ztv/FF3ZNm3QOmzdzis6339Jk8fUFfv+dU8leesn8wHk3EhzMOdV3380YT0YGIE8NQ1fNBKQnPo9uvSts9scyWoAqV7Bt0ySFIIRwoNmAoqUTHc22E9lWnIVnzzJIaW44kCOcOEEXlbU2O/7+dBPt39+8a9nFwYNsnhQXx0issYdRXBw15cGDwLJlHl9J/fzo7lmwgOIsWwYsu3k+CqvzkJXwjs0+WuXljBGpFNS2TVMthO+dKoXC67nxRroiTp2qn9ViMJgUxV13Nd//XFho3zk0GlMHapchJSu6IiLMmyvGgpJduxh19xK0Wr4/ozqOwtiUsfi5ejEqdFVWLby8PDZytavTrsHAysT33uNQ+3ffZVdXWzNmFV5PU2MIKuzUxoiMBObMYYO+LVtM9xsMtB6uucaxJnWWMA4Rs4VbKpHz8jj0wloVobGfzrZtjCt4GfNHz8e4D8ZBXLAM6TvuQ3Jy/boF4/jWlBQ7CwuLipj2ZWzDERDASrfNm/n6H36YJqWiRdJUhfCOU6VQtAgiI4Hbb+fif+YMF5P4eAYyKyvpSg8MbJ6V0Lkzd6nGJpbmMCoMW4HSZlNaSmFsvaCAANuTvDzE2JSxGJE0AuuLnsOSqXdg7Wr/P99bY7rqoEEMTNsMzldXszoxJ8cUWKpLTg6wZAlzgpXvqUXSVIXwlTOFEEJcA+DvAHoCGCKl3OnM8yucS1iYqafQtm1MV8zJ4bqZkMCU0KFDmzYPOSyM8xOM/XnMkZtLZdChQ/Neh00CAuwzV3Q68xV7XoAQAvPHzMekjyahtPN/8PLLd+G33+iaCwri+ID4eDtPtns3U5EsBYratWPRyvbtHHepaHF4SwzhAICrAWx08nkVLqKyEnjxRbqPdTpuGJOTef9bb7GQrLq6aeeePp3nOnmyfiZRTQ3Xo+Bg4M473ZAv3749051KS60fp9OxyZKXcnGnizE0cSie2fQMAoJqMGIEq7nHj3dAGQAcHWprtmt0NPDDD44JqNfbHieocAtNVQhO/SlKKQ9LKY8685wK1/LJJ8x7T001ZQQJwcBzp05shPff/zbt3MHBrDi+6ip2hM3MZDA7P5+1B/PmcZ024rIEH42Gwxvy8iwHTHNyqL26dnWREM3HaCVkFGfgw98+bPqJsrNtW0IhITThbC3wZWXsjDhjBnDHHbwtXcp4hMJjtLgYghDiLgB3AUByS2wZ2go4d441Wh06mN+lC8HHNmzgeMymeFOCgvjcKVOoCKTk5tPo587P57z7n3/m2hIRwTqxESNsb2IdYsQIaqQffqDmi47mC6yooKKIjQUeeMDrB2FP7jIZFyRcgEWbFuHm/jfDR9OEn76/P3fz1grxamttTzDKz2eswTjBKCWFz9uzB9i6lSbi5MmqZNoDNOlbLKV8w9HnCCHWCiEOmLld4eC135ZSpkkp02I9Pv2kbWKchWBtXfDx4W/8yJHmXcvPD0hMpIIxKoMDB5jxtHIlN6QdO3Kt+uorYO5cupqchhAcXvPYY7QEMjNZ+VVVBfzf/wHz53vBFB7bCCEwb/Q8nDh3Ap/s/6RpJxk61HbwPC+Px1lazGtrORuitJSKwFgWrdUyANWhA0vR9+1rmoyKZuG21hVSygnuupbCtVRW2neclE2PI1giN5fxifDw+l2kAwO5XhcVAc89x/U7Koq3Zm80hQD69eOtupqLWkCA11sFDbm8++XoH98fizYtwg19b4BW42DLjbFjgbVr+R6Yy/mtqeFj48dbPsfhw6Y8V3P4+tLcW7GCnfuUleBWWtY3WuEVWKsirosQzk++Wb+e1om5kQKlpUyPX7eOXpzHH2cG5K+/OjHO4O9Pf1YLUwaAyUr4veB3/PdgEwI8iYmchHTmDEvTjW+qlExbysykNWWtXH3rVtv5rZGRNPMKChyXUdEs7JmYFmXHeQxSShvF8VavcRWAfwGIBbBSCLFXSjmpqedTuJbevenK0eksdxitrubvvkcP511XSi725jJjCgpMBXNRUYwrJCWx6du//gVccglw/XUS4vSp881+zhdRdO3aIhf3pnJVz6vQO7Y3FmxcgOl9pkMjHHztY8eybceKFcwq0GppMXXuzNSvvn2tP7+42HZbWuNQbHtNUYXTsMdllHX+Zs120wJocoRXSvk1gK+b+nx7MFbbr1nD7zHA/vATJjCnXVmm9hMYyLTF//6Xln/DWEJtLTs53HCDc6uJ9XoqmobrSU0N6yH8/Hi92lq6+IWg9yE0FNj2VRbGbFqGDlV/mJ4oBP3/t97KL0MbQCM0mDt6Lq778jp8eehLXNP7GsdP0qsXb4WFbIIUGGgKttsiOho4dsz6Mcah2E6cLKewD3sUwmEp5UBrBwghvHpIjl4P/PvfzIwJDjZloRw8COzYwRqam27yeBfjFsWllzIldM0aun2jovg7PneOC/TUqcDEic69po8PXfcNLZPsbF7TGJ+sra2viMIrsjHx2CJkZUgkTukIoamzcBUXs2vpjBms0moDXNPrGvx9/d+xcNNCTOs1zXErwYgxSOMIw4cz/cwaBQVsK+7UdDGFPdjzTRjmpGM8xldfsfK1Y0dau76+vMXH876ff2bGisJ+NBrg+uvpozf2wDEYmCQSF8f5vY89Bnz+ufUuqY4gBDBuHAPLdTl9ur4CqKhgLYSRvvs/RoBvLbL0caiqbrCLDQ/novbuu617hmRODvC//wEzZkD7wIOYe64vfsv9DSsOfuleObp25c1SM8CqKu40rnAo+VDhJOwaoWnpMSHEbbaO8TRlZcCPPzIDxZyrWKPhIrZypRv767cShGD88LbbmO4ZEcGgblUV11lfX773c+ZQQTiDMWNoydUtHtbrTZ9tZaUpVRUAgstyEZd3AOUh8RDCQn1ZaCgthUOHnCOkt7F5MzB7NmspfHyAiAhcmx+PLjWhePqLByFt9cZ2JhoNcP/9/IDSQR12+gAAGq5JREFU000mZWUlqw/z8tg21+WNqhTmaG407R9OkcKFGLvy+lhxjhkDpIcPu0+u1oSUHNJ+8iRjClFRfE8DAxnYjY83dUhuLnFxwCOPUNFnZnIdCQ6mK7u4mLIMH26yGEJLsyCFBgYpIISVmIZGw2Bza+PoUeCdd/jGJSXxQ/H1hU9UDOb4jMMekYOV/3rAva0jwsO5S7j3XsZwior4A5w0CVi0CBg50n2yKOphT5bRb5YeAuBIJxSPUFZmf8pheblrZWmtnDhBZdqxo/m4or8/14BvvrEeu9XpuDZoNHQfW4rp9OwJPPMMY0Lr1nGNq61lgktSkvnRBcaso6Y03GvRrFjB4KyZN+UG9MU/5AY8XbkKlx47BuHOXbmfH/s/eXEPqLaIPUHleACTAJxrcL8AsKXx4d6FI1143TKbtxWydSt/39aSTKKimFxy9mz9PkQA3T9r1zJAXV1NBR4Rwe4FY8aYz1KMjmabocsv5/Evv8zi1oYWQGlYIvTVBogAA7p0sWIQGwyWi6VaKufO0Syz0OLFF1rMxkjc5f8dfvzpbUzq/oKbBVR4G/a4jL4DECKlzGhwSwew3qXSOYFevbjTtBYvrKmhS6lnT/fJ1ZooKDC/K6+LEPwcysrq319YCCxcCCxfTisiKYnrl1YLfPghRxbbiu0IwXnCAwbQ65OVRfdRQQFwMD8Op6P6YXT3PMtFciUl1ECt7QtQVmZznsMtGICk2hD8o+BLSDVQuc1jT1D5dinlZguPXe98kZxLWBhraU6dMu86kpKPXXyxmunRVMLDbbeokJJunbpWmJR0b587x8153d19cDAD1ocPc4qlLQIDgYceYifU4cPpmk5NpaKY8tH1CI/2YcCy4ZegqIgC3HGH9UBTSyQwkK/XykLvBy1mlQ3EVkMGfj75sxuFU3gjrewXYJ7p07kT3bmTvmljenNhIXeSw4ax1bKiaQwdypYS1igu5s4/Ls503+nTptiDOYRgMsrPP9M1ZKwzsIQQLJjt3LnhI+0YxPz3v01FUUJwoWzXjlrDmSXV3kJ0NLXi2bNWc/r/ei4Fi9odxdMbn8b4Tlb6EClaPfYElXdLKQc19xhP4ucH3Hcf8NtvwKpVppbr3btzJGTfvqoorTl0787FPifH/FxlvZ7um5tuqu+9OHyY/7cWe/D15fP/+IO95ZpMQgJTL0+fpkloMDD9yTizszUiBEvKX3iBgWVzEfXcXPgndMDMUbPw0I+PYEP6BoxJGeN+WRVegT0WQk8rmUYAg8vhTpLHZfj4cHbsoEFcC2wtRAr70WiABx9kwW96Ot01wcF8n/PymBr6l7/wva9LZaV9ilgIxnmajRAMUiQlOeFkLYT+/YFrr2VRWkAAPxytlil1xgj/ww/jzshQPPPLc1iwcYFSCG0YexSCPba0hXFS3klr3RB6kpgY+u+3baMVZkzpT0tjv6hu3Ror4NhY+4qDpWScQtEEhGCfke7dmcplbP0aGcnOpMOHAyEhCAQwc8RMzPhxBn7J/AUjkkd4WnKFBxCOZhYIIZ4GFcleAHullL+7QjB7SEtLkzt37vTU5RUWkNI0WMua8i0rAx5+mJ4bS/UBpaU8z3PPKUXuFAwGU6VmAw1dritH6iupGJQwCD/c6OBcZEWLQgixS0qZ1vB+h39iUsr5AF4BUAzgKiGEx8ZpthXy89naecMGxkGc4j5xIUJwgbe1gIeEMFicmWm+pUR1NV1O//d/Shk4DY3G4ojLYL9gzBg2A6v/WI1fz/zqAeEUnsZuC0EI8QqAh6UXJSu3dguhuBj44ANg9+76mYMhIVwkR41q+XEQgwH44gvg++9pCYSF8bWWlPC13Xwzi9MU7qG0uhQpr6RgeNJwfHvdt54WR+EiLFkIjqSdlgJYIYS4VkpZLoSYBGC+lFI5G11AWRmweDF3yElJ9XfIlZXsDVRV5fwW0+5GozEpt82bmRWq0fB1DRumOiC7m1D/UDx64aOYu24udmfvxqAE1yQPFhWxwn3LFn6P27Xj5M0+fVpfOUhLwqEYghDiegCPANABKAOwUEq5yUWy2aQ1WwhffcU2NJa6Keh0TPN84QW1aCqcS3FVMVJeScHYlLH4errz51b99hvw+uv8DkdFUQGUlzNe1Lkz40r2jmlVNI1mxxCEEOMB3AmgHEAMgAc9qQxaMzod+/okJFg+xs+PrpVt29wnl6JtEB4QjoeHPoxvjnyDfTn7nHrujAz2nQoLY0FiaCgLqmNiWEOXmQm8+qp7m68qTDgSqpsDYJ6UciyAvwD4XAgxziVStXHOnqVSsDV+MjRUtexWuIYHhz6IUL9QLNy00KnnXbWKMW1LVeeJiSwcNY65VbgXuxWClHKcsaeRlHI/gMkAnPttUSgUXkFkYCQeHPogvjz0JQ7mHXTKOcvLWQZRt31JQ4Rg/ZyzBiopHKPJyXxSymwAqvGJC4iJoXVgq2FcaWnra9Cp8B4eufARBPkGYdGmRU45X1mZqeutNYKCmEyhMEN2NvDTT8B33zEi37B9cDNpVjxfSlnpLEEUJvz82H11+XLrQWWNhpk4CoUriA6Kxv1D7seSX5Zg/pj56BHTvAaA/v6MDUhpPV1ap2NqtaIOxcVszrjvfExHqzUVGF56KQt6nNCQTZX7eCkXX0x/6unTjQNsFRW8//rr2cZfoXAVjw57FAE+AXhm0zPNPldEBNC1K7sMW6OsDBihktlNlJezUdihQ4zEp6QwFz0lhWX+X30FfP65/aMhraAUgpcSEgI88QQwcCAzL9LTTbfKSs4hH6dC+goXExcch3vS7sEn+z/B8cLjzT7fZZdxs2uph1VhIVNR+/dv9qVaDxs2sENvYmJj08rXl4rhxx+BM2eafSmHexl5E625DqEuZ88y86KmhjUH3bu3wdnACo+RXZqNTq92wvV9rseyK5Y161xSAitXsvlqcDDjZRoN42W5uYwfPPFE22pIa5XaWuDRRxlptzbj9/Rp4KKLgBtusOu0zqhUVniImJjGc4gVCneREJqAuwbdhTd2voG5o+ciNTK1yecyNl/t3BlYvZoucSEYX5g6ldMNo6KcJ3uLp6SEPjRb1acREcDvze8zqhSCQqGwycwRM7F011I8t/k5vHXZW806lxDMjuvZk5ZBTQ03v2pIlRmMk/1sIaVTOkCqGIJC4c3U1HAbvWED0ww9lI+ZGJaI2wfejvf2vofM4kynndffn/EypQwsEBZG90BpqfXjioo4+rGZKAtBofBGpAQ2bWL2SHl5/VzNQYOAW25x+9SgJ0c+iXd3v4slvyzBa1Nec+u12ywaDTBlClNOQ0LM5+vqdPx+jBzZ/Ms1+wwKhcL5rFkDvPMOF4GUFDb6SUnh8Or9+9kK18lFSbZIDk/GrQNuxTu738GZkuZntCjsZPhw7v4zMhoPQykvZwbS9OnWS8DtRCkEhcLbOHcO+Owzpto0zCzRaJh+mJXFVEM3M2vkLNQaavH8lufdfu02i58fh5ZPmkSXYUaG6abXA3ff7bQ++F7hMhJCPA/gMrCt9h8AbpNSFnlWKoXCQ2zdyn/9/Cwfk5DAGclTp1o/zsmkRqbi5v43461db+HJkU+iXUg7t127TePvzxnYl1/OHHSdjvGFLl2cGoDxFgthDYA+Usp+AH4HMMvD8igUnuPIEbaytYax2dXZs+6RqQ6zR82GrlaHf275p9uv3eYJDmbV3uDBLEhycjTeKxSClPJHKaWxdnEbgA6elEeh8DheXDDaJaoLru97Pd7c+Sbyy/M9LY7CiXiFQmjAXwGssvSgEOIuIcROIcTO/Hz1ZVS0Qnr2tJ1mWF1NK8FDFYtzRs1BZU0lXtz6okeur3ANblMIQoi1QogDZm5X1DlmDgA9gI8tnUdK+baUMk1KmRYbG+sO0RUK9zJsGIPHOp3lY7Ky2AHRjfGDuvSI6YHpfabjtR2voaCiwCMyKJyP2xSClHKClLKPmdtyABBC3ApgKoAbZEtusKRQNJeICLayPXWKrW3rYjDw/qQkp2WWNJU5o+agTFeGl7e97FE5FM7DW7KMLgEwE8AYKWWFreMVilbPuHF0Cf3vf0B+fv3CtMGDgZtusjyH0k30ieuDaT2n4dVfX8WM4TMQEaB6sbd0vKLbqRDiOAB/AEbbc5uU8m5bz2sr3U4VbRi9nllHRUUchtK1KxAd7Wmp/mRvzl4MfGsg/jH2H5g/Zr6nxVHYiaVup16hEJqKUggKhee58rMrsSFjAzIezkCYf5inxVHYgSWF4I1ZRgqFogUxb/Q8FFUV4bVfVX+jlo5SCAqFollc0P4CTOk6BS9ufRFlOvf2V1I4F6UQFApFs5k3eh4KKgvw5o43PS2KohkohaBQ2KCqipmemZmcZ61ozIUdLsTEzhPx/JbnUVGjEgVbKl6RdqpQeCNlZcCqVewhZ+w67OPD0bWTJ7NcQGFi/uj5GPneSLy18y08MuwRT4ujaALKQlAozFBSAjz7LAfCR0VxDEFyMhAby67TixYBhYWeltK7GJE8AuNSx2HJliWorFGmVEtEKQSFwgwffwzk5HAmjb+/6X5fX6BjR6C4GFi2zGPieS3zRs9DTlkO3t39rqdFUTQBpRAUigYUFgI7dnAOjSUSEoBDh9hSSGFiTMcxGJU8Cot/WYxqfbWnxVE4iFIICkUD/viDnSI0Vn4dQvCY48fdJ1dLQAiB+WPm40zpGby39z1Pi6NwEKUQFIoGNBxbaw1rDUlbHTk5wHffAe+/D3z9NUc4mul0MD51PIZ1GIZnNz8LXW1beoNaPirLSOERqqu5mAYGMnPHm4iKsv9YD40jcC9VVVQC27bRbPLzo9ZcvpyzG+6+GwgP//NwIQTmjZ6HKZ9MwQf7PsAdg+7wnOwKh/Cyn6KitXPkCLB6NbB3L90ufn7A+PFM5fSWxbVrVyAykmmnISHmj6msBIKCgF693Cub26mtBd58E9i3j9F0Y8dVgNbBsWPAP/8JzJ5N7X6eS7pcgrT2aXhm0zO4pf8t8NX6ekB4haMol5HCbXz/PVM5jx5lO//kZO7Gf/gBeOopeiC8Aa2W4wjy8rg5bohOx2Dytdd6bD6N+zh8mNq7oTIA+P8OHVixt21bg4cE5o+ej5NFJ/HJ/k/cKLCiOSiFoHALBw8Cn33G9SMuzhSw9fOjcvDxAV56yfwC7AnS0oC//Q0oKADS0znLvqCASis3F7jlFmDUKE9L6QbWrKGZ1FAZ1CU2lhV8DeIJU7tNxYB2A7Bo0yLoDXoLT1Z4E8plpHALK1cCYWHM4zdHZCQX3r17gQsvdKtoFhkxAujXjymohw5xveveHRg6tJ7LvHWTns4PzhohITxOp6tXtGG0Eq7+79X4/MDnuKHfDS4VVdF8lEJQuJzSUnoekpOtHxcWBmza5D0KAQBCQzm8bNw4T0viIXx8OLbTGsZpblpto4eu6HEF+sT1wcJNC3Ftn2uh1TQ+RuE9KJeRwuVUVdFFZM3rANB6KC93j0wKOxk0iL4ya5w9y+i6mXQxjdBg3uh5OHL2CL48/KWLhFQ4C6UQFC7HmKlTW2v9uMpKuqMVXsSYMfzgLBVnGAw0AS+5xOIppvWchp4xPbFg4wIYpA1rQ+FRlEJQuJzAQLqBcnOtH1dRwfVH4UV06ABMn85MopKS+oHj8nLg5ElgwgSgTx+Lp9BqtJg7ei4O5B3AN0e+cYPQiqaiFILCLUyezI1mmYWBWmfOAJ06sc5J4WVMmgQ8+CADxpmZHA6RkUGr4bbbgBtvtOkPnN57OrpGdcWCjQvQkue4t3ZES/5w0tLS5M6dOz0thsJODh4E/vUvVilHRdHlXF5OJZGSAjzySBvK3mmJSEmFUF4OBASwNsFMINkS/9n7H9y6/FasuHYFLut+mQsFVdhCCLFLSpnW6H6lEBTupLgY2L4d2LyZMYN27Vip3Lu35ZRUReugprYGPV7vgajAKPx6x68QtrIMFC7DkkJQaacKtxIeDkycyJuibeGr9cXskbNxx7d34IfjP2By18meFknRABVDUCgUbuOm/v/f3v3HVlXecRx/fwaUFoqiCMqQ4VyMm3OGiTqZZNEozpltzrhfhpnMHyOIRFCZq2MZE9S4kCwmxmUu0UwNbiHC1Dh0oGMgy1DRoKL114hMZCoyyWRqHPDdH+epXmvbeyv2Pue2n1fSpOf23Pt82rTn2+ecc7/POUzYdwJXrr7S1xJKyAXBzOqmaVATbVPaeOjlh7h/0/2541gnLghmVlfnTjyXcSPGsWDNAs8SSsYFwczqaujgobRNaWPtP9eyevPq3HGsgguCmdXdBUdfwNjWsSxYvSB3FKvggmBmddc8uJnLT7icVS+u4sHND+aOY4kLgpllMX3SdMYMH8PCNQtzR7HEBcHMshg2ZBhzJ89l5aaVrNuyrvoTrM+VoiBIWijpCUkbJK2Q9Mncmcys71147IWMahnlWUJJlKIgAIsi4qiImAjcA/w8dyAz63utTa1cNvkylj+/nPVb3YYmt1IUhIj4T8XmcMA3J5sNEBcddxH7Ne/nWUIJlKIgAEi6WtJLwDR6mCFImi5pvaT127Ztq19AM+sT+wzdhznHz+HuZ+9mwysbcscZ0OrW7VTS/cBBXXxpXkTcVbHfFUBzRMyv9prudmrWP+x4ZwcTrpvA1EOncsd378gdp9/rrttp3WYIEXFKRBzZxcddnXZdDJxVr1xmlt/I5pHM/tJslrYvZeNrG3PHGbBKccpI0mEVm2cAz+TKYmZ5zDl+Dq1NrVy15qrcUQasUhQE4FpJGyU9AZwKzM4dyMzqa/+W/Zl17CyWPLWE9m3tueMMSKUoCBFxVjp9dFREfCMiXs6dyczq79LJl9IypIVr1l6TO8qAVIqCYGYGMHr4aGYeM5Pbn7yd57c/nzvOgOOCYGalMvfLc2ka1ORZQgYuCGZWKge2HsiMSTO47fHb2PTGptxxBpS6vQ+hL0jaBmzuwyEOAF7vw9fvS42a3bnrr1GzN2puyJ99QkSM7vxgQxeEviZpfVdv3mgEjZrdueuvUbM3am4ob3afMjIzM8AFwczMEheEnv02d4C90KjZnbv+GjV7o+aGkmb3NQQzMwM8QzAzs8QFwczMABeEqhp1vWdJiyQ9k7L/UdLI3JlqJek7kp6StEdS6W7N60zSaZKelfSCpLbceWol6WZJr0lqqH7TksZLWiXp6fR70hDNMCU1S3pY0uMp95W5M3XmawhVSNqnY4lPSRcDR0TEjMyxqpJ0KvCXiNgl6ZcAEfGTzLFqIulzwB7gRmBuRJR2FSRJg4DngKnAFuAR4OyIeDprsBpI+gqwE7g1Io7MnadWksYCYyPiMUkjgEeBb5X9Zy5JwPCI2ClpCLAWmB0R6zJHe49nCFU06nrPEbEiInalzXXAwTnz9EZEtEfEs7lz1Og44IWI2BQR7wJ/oFjTo/QiYg3w79w5eisi/hURj6XP3wTagXF5U1UXhZ1pc0j6KNXxxAWhBrWu91xi5wH35g7RT40DXqrY3kIDHJz6C0mHAF8EHsqbpDaSBknaALwGrIyIUuV2QaBY7zkt0NP54wyAiJgXEeMplveclTft+6rlTvvMA3ZRZC+NWrKb9URSK7AUmNNpJl9aEbE7IiZSzNiPk1SqU3WDcwcog4g4pcZdFwPLgfl9GKdm1XJL+iHwdeDkKNnFol78zMvuZWB8xfbB6THrQ+kc/FJgcUQsy52ntyJih6RVwGlAaS7qe4ZQRaOu9yzpNOBy4JsR8VbuPP3YI8Bhkj4tqQn4PnB35kz9Wro4exPQHhG/yp2nVpJGd9ztJ6mF4kaEUh1PfJdRFZKWAodT3PWyGZjRCEt8SnoBGApsTw+ta4S7owAknQlcD4wGdgAbIuKreVN1T9LpwHXAIODmiLg6c6SaSPo9cCJFK+ZXgfkRcVPWUDWQNAV4EHiS4u8S4KcRsTxfquokHQXcQvF78glgSUQsyJvqg1wQzMwM8CkjMzNLXBDMzAxwQTAzs8QFwczMABcEMzNLXBDMzAxwQbB+QtIhkt5OfWI6HvtQW2pJLamV+buSDtjLMVskrU4dT5F0saR2Sb1qEyJppKSZe5OlhjE+1OpaUpOkNZLcscAAFwTrX/6R+sR0tKW+AfgacARwtqQjIuLttM/Wj2G884BlEbE7bc8EpkbEtF6+zsj03F5Roda/4d9RtEl4T+rO+gDwvd6Obf2TC4I1jLQoytT0+VWSru9h93q0pZ4G3JXy/AY4FLhX0iWSfpAWQ9kg6caKWcSdkh5NC6RMT69zLfCZtO+iNNup/E9+rqRfpM8PSbOeWyl64IzvbqxKPbS6vjN9H2YuCNZQ5gPzJE2jaHk8p4d9+7QtdepbdGhEvAiQ2oJsBU4C7qP4r/uENBvZzfsH3fMiYhJwDHCxpFFAG2l2ExE/rmH4w4BfR8TngWE9jFWLjcCxvdjf+jGfO7SGERFrUmOzS4ETO07VSFpI0eysng6g6LPUlZOBScAjRVxaKPrfQ1EEzkyfj6c4uL/Sy7E3V6yy1dNYVUXE7nQ9ZURabMYGMBcEaxiSvgCMBbZ3HLwkHUSx8lRnvW5LLeki4Edp83TgzMrtiKi87vA20NzdSwG3RMQVnV7/ROAUYHJEvCXpr928xi4+OHvvvM9/q43VS0OBd/bi+dZP+JSRNQQV6+guprgOsDO19waYCGzo4im9bksdETek0zYTI2Jr5+1O+74BDJLU1QH9AeDbksak7PtLmgDsC7yRisFngePT/m8CIyqe/yowRtIoSUMp1rToTndj1SSdsno9Iv5X63Os/3JBsNKTNAxYBlwWEe3AQt5fpKjLgpDWk54F/Jlizd0lEfHUxxxtBTCli7GfBn4GrJD0BLCSYmZzHzBYUjvFheR1af/twN/SinGL0sF5AfBwem63PfN7GOsDUqvrvwOHS9oi6fz0pZOAP32Ub976H7e/toYm6SaK0zqfAu6JiJqWJJT0InBMRLy+F2MfDVwSEed81NfITdIyoC0insudxfLzDMEaWkScHxF7KO6u2bfyjWld6XhjGsV1hz097VvD2I8Bq7q6zbMRpFNpd7oYWAfPEMzMDPAMwczMEhcEMzMDXBDMzCxxQTAzM8AFwczMEhcEMzMDXBDMzCxxQTAzMwD+D0I+ZtV3h/8cAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1W4TZfXOmIlS"
      },
      "source": [
        "Посчитаем точность на тестовом датасете:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "HUjdeIefsIsg",
        "outputId": "a1a363d4-a307-4769-9ccf-fe8a857b62af"
      },
      "source": [
        "pred = torch.sigmoid(net.forward(torch.tensor(valid_x)))\r\n",
        "torch.mean(((pred.view(-1)>0.5)==(torch.tensor(valid_labels)>0.5)).type(torch.float32))"
      ],
      "execution_count": 205,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor(0.8667)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 205
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_95qF9lY2kHp"
      },
      "source": [
        "## Нейросети и оптимизаторы\r\n",
        "\r\n",
        "В PyTorch реализованы специальные модули `torch.nn.Module` для реализации нейросетей в виде набора слоёв. При этом есть два способа описания нейросетей:\r\n",
        "* **Sequential**, в виде перечисления набора слоёв\r\n",
        "* В виде **класса**, унаследованного от `Module`\r\n",
        "\r\n",
        "Первый способ проще позволяет описывать стандартные сети, второй - более гибкий, позволяет описывать произвольные сложные конфигурации вычислений.\r\n",
        "\r\n",
        "Внутри модулей можно использовать стандартные слои, среди которых:\r\n",
        "* `Linear` - линейный слой, аналогичный нашей нейросети, приведенной выше\r\n",
        "* `Softmax`, `Sigmoid`, `ReLU` - слои для передаточных функций\r\n",
        "\r\n",
        "> Обратите внимание, что большинство передаточных функций и функций потерь в PyTorch доступны в двух вариантах: **функциональный** (внутри `torch.nn.functional`) и **в виде класса** (внутри `torch.nn`). Для передаточных функций удобно использовать функции из `torch.nn.functional`, не создавая отдельного слоя.\r\n",
        "\r\n",
        "Для однослойной сети мы можем просто взять экземпляр слоя `Linear`:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "D77pXPR6oFRs",
        "outputId": "efa49e5c-72d4-4781-89d4-4ab6597d2b0e"
      },
      "source": [
        "net = torch.nn.Linear(2,1) # 2 inputs, 1 output\r\n",
        "\r\n",
        "print(list(net.parameters()))"
      ],
      "execution_count": 206,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[Parameter containing:\n",
            "tensor([[-0.1892, -0.5979]], requires_grad=True), Parameter containing:\n",
            "tensor([0.1538], requires_grad=True)]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0tbe0Et_oiNo"
      },
      "source": [
        "Мы видим, что у сети есть метод `parameters()`, возвращающий все параметры, которые необходимо подстраивать.\r\n",
        "\r\n",
        "Методы обучения, такие, как градиентный спуск, реализованы в виде отдельных объектов, которым передается список параметров:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "B4AxyrFMozh0"
      },
      "source": [
        "optim = torch.optim.SGD(net.parameters(),lr=0.05)"
      ],
      "execution_count": 207,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6eB8v58eo9pp"
      },
      "source": [
        "С использованием оптимизатора, наш процесс обучения будет выглядеть так:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ups7nlV22ofp",
        "outputId": "503d8ae9-35f3-4ecb-e2ff-4da2ec2914eb"
      },
      "source": [
        "val_x = torch.tensor(valid_x)\r\n",
        "val_lab = torch.tensor(valid_labels)\r\n",
        "\r\n",
        "for ep in range(10):\r\n",
        "  for (x,y) in dataloader:\r\n",
        "    z = net(x).flatten()\r\n",
        "    loss = torch.nn.functional.binary_cross_entropy_with_logits(z,y)\r\n",
        "    optim.zero_grad()\r\n",
        "    loss.backward()\r\n",
        "    optim.step()\r\n",
        "  acc = ((torch.sigmoid(net(val_x).flatten())>0.5).float()==val_lab).float().mean()\r\n",
        "  print(f\"Epoch {ep}: last batch loss = {loss}, val acc = {acc}\")"
      ],
      "execution_count": 208,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch 0: last batch loss = 1.1270943880081177, val acc = 0.6666666865348816\n",
            "Epoch 1: last batch loss = 1.0163036584854126, val acc = 0.7333333492279053\n",
            "Epoch 2: last batch loss = 0.9239333271980286, val acc = 0.800000011920929\n",
            "Epoch 3: last batch loss = 0.8493780493736267, val acc = 0.9333333373069763\n",
            "Epoch 4: last batch loss = 0.7907424569129944, val acc = 0.8666666746139526\n",
            "Epoch 5: last batch loss = 0.7453570365905762, val acc = 0.8666666746139526\n",
            "Epoch 6: last batch loss = 0.7104158401489258, val acc = 0.8666666746139526\n",
            "Epoch 7: last batch loss = 0.6834256649017334, val acc = 0.8666666746139526\n",
            "Epoch 8: last batch loss = 0.6623864769935608, val acc = 0.8666666746139526\n",
            "Epoch 9: last batch loss = 0.645784318447113, val acc = 0.8666666746139526\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vRLXEQ4Qrcvx"
      },
      "source": [
        "> Обратите внимание, что для вызова (применения) нейросети к входному аргументу мы используем синтаксис `net(x)`, вместо `net.forward(x)`, поскольку `nn.Module` реализует метод `__call__()`\r\n",
        "\r\n",
        "С учётом описанного, можем описать универсальную обучающую функцию:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "5c6WsBhlrlIs",
        "outputId": "54de8404-4170-4a15-abba-039d06d5e946"
      },
      "source": [
        "def train(net, dataloader, val_x, val_lab, epochs=10, lr=0.05):\r\n",
        "  optim = torch.optim.Adam(net.parameters(),lr=lr)\r\n",
        "  for ep in range(epochs):\r\n",
        "    for (x,y) in dataloader:\r\n",
        "      z = net(x).flatten()\r\n",
        "      loss = torch.nn.functional.binary_cross_entropy_with_logits(z,y)\r\n",
        "      optim.zero_grad()\r\n",
        "      loss.backward()\r\n",
        "      optim.step()\r\n",
        "    acc = ((torch.sigmoid(net(val_x).flatten())>0.5).float()==val_lab).float().mean()\r\n",
        "    print(f\"Epoch {ep}: last batch loss = {loss}, val acc = {acc}\")\r\n",
        "\r\n",
        "net = torch.nn.Linear(2,1)\r\n",
        "\r\n",
        "train(net,dataloader,val_x,val_lab,lr=0.03)"
      ],
      "execution_count": 214,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch 0: last batch loss = 0.6797666549682617, val acc = 0.06666667014360428\n",
            "Epoch 1: last batch loss = 0.6262387037277222, val acc = 0.06666667014360428\n",
            "Epoch 2: last batch loss = 0.5835054516792297, val acc = 0.20000000298023224\n",
            "Epoch 3: last batch loss = 0.5517064332962036, val acc = 0.2666666805744171\n",
            "Epoch 4: last batch loss = 0.5291333794593811, val acc = 0.2666666805744171\n",
            "Epoch 5: last batch loss = 0.5140607357025146, val acc = 0.3333333432674408\n",
            "Epoch 6: last batch loss = 0.5050028562545776, val acc = 0.4000000059604645\n",
            "Epoch 7: last batch loss = 0.5006228089332581, val acc = 0.6000000238418579\n",
            "Epoch 8: last batch loss = 0.4996645152568817, val acc = 0.6000000238418579\n",
            "Epoch 9: last batch loss = 0.5009827017784119, val acc = 0.7333333492279053\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KzuIDqJ8sFYm"
      },
      "source": [
        "## Описание сети в виде набора слоёв\r\n",
        "\r\n",
        "Простейшую многослойную нейросеть можно описать в виде набора последовательно-применяемых слоёв. При этом такая сеть будет обладать всеми характеристиками вышеописанных - она автоматически соберёт в методе `parameters` параметры всех промежуточных слоёв"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "tBtytmEAsq-O",
        "outputId": "06ad840b-c2b7-409e-e01e-a9170548151d"
      },
      "source": [
        "net = torch.nn.Sequential(torch.nn.Linear(2,5),torch.nn.Sigmoid(),torch.nn.Linear(5,1))\r\n",
        "print(net)"
      ],
      "execution_count": 215,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Sequential(\n",
            "  (0): Linear(in_features=2, out_features=5, bias=True)\n",
            "  (1): Sigmoid()\n",
            "  (2): Linear(in_features=5, out_features=1, bias=True)\n",
            ")\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5r5RbLB1s6YB"
      },
      "source": [
        "Обучать такую сеть мы можем с помощью описанного ранее метода `train`:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ogXKdcfIs_ND",
        "outputId": "957ccd8d-0076-4e9b-89f1-edc1de75f18e"
      },
      "source": [
        "train(net,dataloader,val_x,val_lab)"
      ],
      "execution_count": 216,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch 0: last batch loss = 0.6622872352600098, val acc = 0.6666666865348816\n",
            "Epoch 1: last batch loss = 0.6727332472801208, val acc = 0.800000011920929\n",
            "Epoch 2: last batch loss = 0.6326467990875244, val acc = 0.800000011920929\n",
            "Epoch 3: last batch loss = 0.5876082181930542, val acc = 0.7333333492279053\n",
            "Epoch 4: last batch loss = 0.5610458254814148, val acc = 0.7333333492279053\n",
            "Epoch 5: last batch loss = 0.5546905398368835, val acc = 0.7333333492279053\n",
            "Epoch 6: last batch loss = 0.5626044273376465, val acc = 0.7333333492279053\n",
            "Epoch 7: last batch loss = 0.5754281878471375, val acc = 0.800000011920929\n",
            "Epoch 8: last batch loss = 0.5850474238395691, val acc = 0.800000011920929\n",
            "Epoch 9: last batch loss = 0.5874522924423218, val acc = 0.800000011920929\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jY4R1XEGtEzJ"
      },
      "source": [
        "## Описание нейросети в виде класса\r\n",
        "\r\n",
        "Это более гибкий способ описания нейросетей, поскольку он позволяет выполнять произвольные вычисления."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "SlsJmGu0tMsZ",
        "outputId": "240d5c89-096c-4392-99cd-1ade5ff3e3e1"
      },
      "source": [
        "class MyNet(torch.nn.Module):\r\n",
        "  def __init__(self,hidden_size=10,func=torch.nn.Sigmoid()):\r\n",
        "    super().__init__()\r\n",
        "    self.fc1 = torch.nn.Linear(2,hidden_size)\r\n",
        "    self.func = func\r\n",
        "    self.fc2 = torch.nn.Linear(hidden_size,1)\r\n",
        "\r\n",
        "  def forward(self,x):\r\n",
        "    x = self.fc1(x)\r\n",
        "    x = self.func(x)\r\n",
        "    x = self.fc2(x)\r\n",
        "    return x\r\n",
        "  \r\n",
        "net = MyNet(func=torch.nn.ReLU())\r\n",
        "print(net)"
      ],
      "execution_count": 230,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "MyNet(\n",
            "  (fc1): Linear(in_features=2, out_features=10, bias=True)\n",
            "  (func): ReLU()\n",
            "  (fc2): Linear(in_features=10, out_features=1, bias=True)\n",
            ")\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "HwdapRxft-7M",
        "outputId": "6eb900cf-4902-4a04-c62b-497b68455406"
      },
      "source": [
        "train(net,dataloader,val_x,val_lab,lr=0.005)"
      ],
      "execution_count": 231,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch 0: last batch loss = 0.6930208206176758, val acc = 0.8666666746139526\n",
            "Epoch 1: last batch loss = 0.6779811382293701, val acc = 0.9333333373069763\n",
            "Epoch 2: last batch loss = 0.6627538800239563, val acc = 0.9333333373069763\n",
            "Epoch 3: last batch loss = 0.6472628712654114, val acc = 0.9333333373069763\n",
            "Epoch 4: last batch loss = 0.6321150660514832, val acc = 0.9333333373069763\n",
            "Epoch 5: last batch loss = 0.618022620677948, val acc = 0.9333333373069763\n",
            "Epoch 6: last batch loss = 0.6052160859107971, val acc = 0.9333333373069763\n",
            "Epoch 7: last batch loss = 0.5925578474998474, val acc = 0.9333333373069763\n",
            "Epoch 8: last batch loss = 0.5803073048591614, val acc = 0.9333333373069763\n",
            "Epoch 9: last batch loss = 0.569009006023407, val acc = 0.9333333373069763\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dvAiaj_JndyP"
      },
      "source": [
        "**Задание 1**: Постройте графики ошибок на обучающей и тестовой выборке в процессе обучения\r\n",
        "\r\n",
        "**Задание 2**: Попробуйте решить задачу классификации на датасете MNIST с помощью этого кода. Подсказка: используйте `crossentropy_with_logits` в качестве функции ошибки. При этом в первом случае на выход сети необходимо подавать целевые значения в формате *one hot encoding*, а во втором - в виде целочисленного номера класса."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gZ-kWx84bMDH"
      },
      "source": [
        "**Задание 3**: \r\n",
        "Используйте PyTorch для обучения классификатора на сети MNIST. При этом:\r\n",
        "* Обратите внимание, что в keras заложены типовые датасеты, включая MNIST. Для обращения к нему достаточно пары строчек кода с использованием библиотеки `torchvision` (см, например, [тут](https://pytorch.org/vision/0.8/datasets.html))\r\n",
        "* Попробуйте несколько конфигураций сети с несколькими полносвязными слоями, передаточными функциями, и разным количеством нейронов\r\n",
        "\r\n",
        "Какой точности вам удалось достичь?\r\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "id": "MG64NKzawHl-"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}