1457 lines
154 KiB
Plaintext
1457 lines
154 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "En2vX4FuwHlu"
|
|
},
|
|
"source": [
|
|
"## Introduction to Tensorflow and Keras\n",
|
|
"\n",
|
|
"> This notebook is a part of [AI for Beginners Curricula](http://github.com/microsoft/ai-for-beginners). Visit the repository for complete set of learning materials.\n",
|
|
"\n",
|
|
"### Neural Frameworks\n",
|
|
"\n",
|
|
"We have learnt that to train neural networks you need:\n",
|
|
"* Quickly multiply matrices (tensors)\n",
|
|
"* Compute gradients to perform gradient descent optimization\n",
|
|
"\n",
|
|
"What neural network frameworks allow you to do:\n",
|
|
"* Operate with tensors on whatever compute is available, CPU or GPU, or even TPU\n",
|
|
"* Automatically compute gradients (they are explicitly programmed for all built-in tensor functions)\n",
|
|
"\n",
|
|
"Optionally:\n",
|
|
"* Neural Network constructor / higher level API (describe network as a sequence of layers)\n",
|
|
"* Simple training functions (`fit`, as in Scikit Learn)\n",
|
|
"* A number of optimization algorithms in addition to gradient descent\n",
|
|
"* Data handling abstractions (that will ideally work on GPU, too)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "8cACQoFMwHl3"
|
|
},
|
|
"source": [
|
|
"### Most Popular Frameworks\n",
|
|
"\n",
|
|
"* Tensorflow 1.x - first widely available framework (Google). Allowed to define static computation graph, push it to GPU, and explicitly evaluate it\n",
|
|
"* PyTorch - a framework from Facebook that is growing in popularity\n",
|
|
"* Keras - higher level API on top of Tensorflow/PyTorch to unify and simplify using neural networks (Francois Chollet)\n",
|
|
"* Tensorflow 2.x + Keras - new version of Tensorflow with integrated Keras functionality, which supports **dynamic computation graph**, allowing to perform tensor operations very similar to numpy (and PyTorch)\n",
|
|
"\n",
|
|
"We will consider Tensorflow 2.x and Keras. Make sure you have version 2.x.x of Tensorflow installed:\n",
|
|
"```\n",
|
|
"pip install tensorflow\n",
|
|
"```\n",
|
|
"or\n",
|
|
"```\n",
|
|
"conda install tensorflow\n",
|
|
"```"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 1,
|
|
"metadata": {
|
|
"colab": {
|
|
"base_uri": "https://localhost:8080/"
|
|
},
|
|
"id": "xwqVx9-bwHl3",
|
|
"outputId": "2aa591b4-b647-441f-9c8e-4e0da2d517a0",
|
|
"tags": []
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"2.3.0\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"import tensorflow as tf\n",
|
|
"import numpy as np\n",
|
|
"print(tf.__version__)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "6tp2xGV7wHl4"
|
|
},
|
|
"source": [
|
|
"## Basic Concepts: Tensor\n",
|
|
"\n",
|
|
"**Tensor** is a multi-dimensional array. It is very convenient to use tensors to represent different types of data:\n",
|
|
"* 400x400 - black-and-white picture\n",
|
|
"* 400x400x3 - color picture \n",
|
|
"* 16x400x400x3 - minibatch of 16 color pictures\n",
|
|
"* 25x400x400x3 - one second of 25-fps video\n",
|
|
"* 8x25x400x400x3 - minibatch of 8 1-second videos"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "qG2bsaR7wHl4"
|
|
},
|
|
"source": [
|
|
"### Simple Tensors\n",
|
|
"\n",
|
|
"You can easily create simple tensors from lists of np-arrays, or generate random ones:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 2,
|
|
"metadata": {
|
|
"colab": {
|
|
"base_uri": "https://localhost:8080/"
|
|
},
|
|
"id": "ybpnk08HwHl4",
|
|
"outputId": "fad9ed4a-df82-44a0-84ea-324bc71ea46f",
|
|
"trusted": true
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"tf.Tensor(\n",
|
|
"[[1 2]\n",
|
|
" [3 4]], shape=(2, 2), dtype=int32)\n",
|
|
"tf.Tensor(\n",
|
|
"[[-0.50882405 1.1335199 -0.78524745]\n",
|
|
" [ 0.5572137 0.8736796 -0.05827063]\n",
|
|
" [-0.4501901 0.2969733 -1.660203 ]\n",
|
|
" [-0.13222459 0.23592396 -0.32563084]\n",
|
|
" [-0.9020366 -0.24209827 0.72447866]\n",
|
|
" [ 0.22193347 1.3671101 -0.13040254]\n",
|
|
" [-0.35891932 -2.0387182 0.7902264 ]\n",
|
|
" [ 2.590151 -0.40071017 -0.7124725 ]\n",
|
|
" [ 1.1319299 0.34047043 1.0180026 ]\n",
|
|
" [ 0.57838875 -1.5080605 0.8308813 ]], shape=(10, 3), dtype=float32)\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"a = tf.constant([[1,2],[3,4]])\n",
|
|
"print(a)\n",
|
|
"a = tf.random.normal(shape=(10,3))\n",
|
|
"print(a)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "AXFMsV3r09Ux"
|
|
},
|
|
"source": [
|
|
"You can use arithmetic operations on tensors, which are performed element-wise, as in numpy. Tensors are automatically expanded to required dimension, if needed. To extract numpy-array from tensor, use `.numpy()`:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 3,
|
|
"metadata": {
|
|
"colab": {
|
|
"base_uri": "https://localhost:8080/"
|
|
},
|
|
"id": "e5Nu5Xgj1DnQ",
|
|
"outputId": "0dfc8758-4ffd-4968-c7bf-6ba8d435df2e"
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"tf.Tensor(\n",
|
|
"[[ 0. 0. 0. ]\n",
|
|
" [ 1.0660378 -0.2598403 0.7269768 ]\n",
|
|
" [ 0.05863395 -0.8365466 -0.87495553]\n",
|
|
" [ 0.37659946 -0.89759594 0.4596166 ]\n",
|
|
" [-0.39321256 -1.3756182 1.509726 ]\n",
|
|
" [ 0.73075753 0.23359025 0.6548449 ]\n",
|
|
" [ 0.14990473 -3.172238 1.5754738 ]\n",
|
|
" [ 3.0989752 -1.53423 0.07277495]\n",
|
|
" [ 1.640754 -0.79304945 1.8032501 ]\n",
|
|
" [ 1.0872128 -2.6415803 1.6161287 ]], shape=(10, 3), dtype=float32)\n",
|
|
"[0.60120213 3.1065722 0.45600685]\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"print(a-a[0])\n",
|
|
"print(tf.exp(a)[0].numpy())"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "uQ5zN6cVyrG7"
|
|
},
|
|
"source": [
|
|
"## Variables\n",
|
|
"\n",
|
|
"Variables are useful to represent tensor values that can be modified using `assign` and `assign_add`. They are often used to represent neural network weights.\n",
|
|
"\n",
|
|
"As an example, here is a silly way to get a sum of all rows of tensor `a`:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 4,
|
|
"metadata": {
|
|
"colab": {
|
|
"base_uri": "https://localhost:8080/"
|
|
},
|
|
"id": "7pu0UZ-_yqfB",
|
|
"outputId": "6708c83e-02e6-4442-8757-45918eb1fbc2"
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"<tf.Variable 'Variable:0' shape=(3,) dtype=float32, numpy=array([ 2.7274222 , 0.05809033, -0.3086378 ], dtype=float32)>\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"s = tf.Variable(tf.zeros_like(a[0]))\n",
|
|
"for i in a:\n",
|
|
" s.assign_add(i)\n",
|
|
"\n",
|
|
"print(s)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "rIh1EHcezlNo"
|
|
},
|
|
"source": [
|
|
"Much better way to do it:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 5,
|
|
"metadata": {
|
|
"colab": {
|
|
"base_uri": "https://localhost:8080/"
|
|
},
|
|
"id": "aQIdWZ1kzn6P",
|
|
"outputId": "1c123d9a-ecd2-4f2e-828e-5ade85ac8f63"
|
|
},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"<tf.Tensor: shape=(3,), dtype=float32, numpy=array([ 0.69976234, -1.5164013 , -0.19107199], dtype=float32)>"
|
|
]
|
|
},
|
|
"execution_count": 5,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"tf.reduce_sum(a,axis=0)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "U-auwezDwHl6"
|
|
},
|
|
"source": [
|
|
"## Computing Gradients\n",
|
|
"\n",
|
|
"For back propagation, you need to compute gradients. This is done using `tf.GradientTape()` idiom:\n",
|
|
" * Add `with tf.GradientTape` block around our computations\n",
|
|
" * Mark those tensors with respect to which we need to compute gradients by calling `tape.watch` (all variables are watched automatically)\n",
|
|
" * Compute whatever we need (build computational graph)\n",
|
|
" * Obtain gradients using `tape.gradient` "
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 5,
|
|
"metadata": {
|
|
"colab": {
|
|
"base_uri": "https://localhost:8080/"
|
|
},
|
|
"id": "m8vFOXr7wHl6",
|
|
"outputId": "860ac72e-50c7-4ff2-f258-747f27194f90",
|
|
"trusted": true
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"tf.Tensor(\n",
|
|
"[[-0.9959172 -0.79481506]\n",
|
|
" [ 0.74432266 0.7713953 ]], shape=(2, 2), dtype=float32)\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"a = tf.random.normal(shape=(2, 2))\n",
|
|
"b = tf.random.normal(shape=(2, 2))\n",
|
|
"\n",
|
|
"with tf.GradientTape() as tape:\n",
|
|
" tape.watch(a) # Start recording the history of operations applied to `a`\n",
|
|
" c = tf.sqrt(tf.square(a) + tf.square(b)) # Do some math using `a`\n",
|
|
" # What's the gradient of `c` with respect to `a`?\n",
|
|
" dc_da = tape.gradient(c, a)\n",
|
|
" print(dc_da)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "8sfjBMBu59B5"
|
|
},
|
|
"source": [
|
|
"## Example 1: Linear Regression\n",
|
|
"\n",
|
|
"Now we know enough to solve the classical problem of **Linear regression**. Let's generate small synthetic dataset:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 6,
|
|
"metadata": {
|
|
"id": "j723455WwHl7",
|
|
"trusted": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"import matplotlib.pyplot as plt\n",
|
|
"from sklearn.datasets import make_classification, make_regression\n",
|
|
"from sklearn.model_selection import train_test_split\n",
|
|
"import random"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 7,
|
|
"metadata": {
|
|
"colab": {
|
|
"base_uri": "https://localhost:8080/",
|
|
"height": 282
|
|
},
|
|
"id": "WJNK_J6v6I-Z",
|
|
"outputId": "eb4a66a6-6b9a-4c8a-bc24-d81eeb2d3f27"
|
|
},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"<matplotlib.collections.PathCollection at 0x203d1780f40>"
|
|
]
|
|
},
|
|
"execution_count": 7,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
},
|
|
{
|
|
"data": {
|
|
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAWoAAAD4CAYAAADFAawfAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAAbVklEQVR4nO3df4ylVXkH8O+zw0XuAjJaphVGx8XULKkQdtmJoZnGyGrFij82YNTGtKltsmmTNkIo7egfRf9iEmPVxqbNxNJqSnVpFzYGVDRZDHWj2Bl2EZClUXErs7aMZQfEmbqzw9M/7n2XO++85z3nve857z3n3u8n2bC795075903PPfMc57zHFFVEBFRvLYNegBERFSOgZqIKHIM1EREkWOgJiKKHAM1EVHkzgnxphdffLHu2LEjxFsTEQ2lxcXFn6nqRNFrQQL1jh07sLCwEOKtiYiGkoicML3G1AcRUeQYqImIIsdATUQUOQZqIqLIMVATEUUuSNUHEdEoOXR0CZ+4/0mcXFnDpeNt3HrdTuzbPent/RmoiYhqOHR0CR+5+1GsrW8AAJZW1vCRux8FAG/BmqkPIqIaPnH/k2eDdGZtfQOfuP9Jb9+DgZqIqIaTK2uV/r4fDNRERDVcOt6u9Pf9YKAmIkIn1zwzdxiXzd6HmbnDOHR0yenrbr1uJ9qtsU1/126N4dbrdnobGxcTiWjk1VkQzF5n1QcRUUBlC4IuAXff7kmvgTmPqQ8iGnlNLAjWYQ3UIrJTRI71/HpeRG5qYGxERI1oYkGwDmugVtUnVXWXqu4CsAfAKoB7Qg+MiKgpTSwI1lE1R/0WAD9UVWODayKi1FRZEOzdLn5RuwURYGV1PcgiYkZU1f1ikTsAPKyqny14bT+A/QAwNTW158QJxnIiGi756pC8dmsMt99wZV/BWkQWVXW66DXnxUQRORfAuwH8a9HrqjqvqtOqOj0xUXjsFxFR9MrqqYuqQ3r53jqeqZL6+B10ZtP/430UREQRsNVTu1SBhKgUqVKe97sAvuh9BEREkbA1WHKpAglRKeIUqEVkO4DfBnC39xEQEUXCVk9dVB3SK1SliFPqQ1VXAfyK9+9ORBSRS8fbWCoI1tksOV8d0lTVB7eQExF13Xrdzi1VHflZcujt4kUYqImIupposNQPBmoioh6DmDHbsCkTEVHkGKiJiCLHQE1EFDnmqIloJPQ2U4plkdAVAzURDYWyQNzvUVvZey6trGFMBBuqmBxAkGegJqJKYpyZlgViALjlrkewkesUajtqK/+e2ddXOU/Rl0ptTl1NT0/rwsKC9/closEqavNZp7WnLzNzhwt3FI63W/jlmRdLO95NjrcLP3RM79n7dUdm99YffJeXNqdERLamRYNi6tGxsrZeGqQFnRmy4qWZctbW1NYFr8nzFBmoichZrIfA9tOxTgDk8wlVOuU1eZ4iAzUROTMFp4vaLWOz/SaYzjx8xfZW4fVjIluCdMalU17T5ykyUBORs6Lg1dom+MXpM8YUQgj5U1gA4PYbrsTkeBuCTv749huuxG3vekNhAP/k+67CpOXk8X27J8++J9AJ7uh5b1Z9EFGUipoWrZ4+g1Or65uus1VU1GGq8Lj9hiuNi3umKpUYO+UVYdUHEdVy2ex9hWkEAfDU3PXev5+pGqOfKoyYSg3Lqj44oyaiWmzN9n3zuaAZy4zZhjlqIqrFtJAXarHN9AHQZBVG0xioiaiW3kW33oW8UDPVpj8YYsDUBxHV1mQKIdZTWEJyCtQiMg7gcwCuQKdG/A9V9dsBx0VEZJRKbtkX1xn1ZwB8TVXfKyLnAtgecExERIViqtJokjVQi8jLAbwJwB8AgKqeBnA67LCIiDbrt1XpMHBZTHwdgGUA/ygiR0XkcyJyfv4iEdkvIgsisrC8vOx9oEQ0WPndgE1vE4+1IVQTXAL1OQCuBvB3qrobwC8AzOYvUtV5VZ1W1emJiQnPwySiQcpms01uE+/93mUtRwfdEKoJLjnqpwE8raoPdf/8bygI1EQ0vEyz2Y99+fHSU1Xq5pOL+l/nDXP9dMYaqFX1v0XkJyKyU1WfBPAWAN8PPzQiikVZv+eVtU6fj/ypKj7yyUUfEL2GvX4641r18WcA7uxWfPwIwIfCDYmIYmPaJp7XmzM25ZOrBOqytMYgzi4cFKdArarHABQ2CyGi4XfrdTutKYhMWXCtmk82fUD4PgYrdtxCTkRWRdvETU35Lx1vO/XjcKkiGcXt4kW4hZyInOR3A5oOus2CaNlrLjXR2WLk2voGxkSwoTpS6Y5eDNREI6xOZYZLzw3Ta2U10ft2T24J5BuqZwP9qAVpgAcHEI0s04y4iWOmTIcNAJ20imnhsig3PSzbyssODmCOmmhEDXKnnymHLUBpdUl+MXKQG3GaxEBNNKJ8npRSVdEioQDGWXYmH+BHZVs5AzXRkDNVV5hmtQoE7+WRryIZb7esQbqo2mOQHzZN4mIi0RArq64oq4323ZnOlEfuXTgsY6r2aPq8xkFhoCYaQllgLApiWWogW5SzXVc3UNtK8cq2idsWN4s+bNqtMVx7+QRm5g4nv8CYYeqDyINBtwDNjyVbYDPJUgP7dk/iyOxeiOW6Omx55LLvYatAKdqIc+OeSRxcXBqqBUbOqIlqiq2hva2REbA1NeAzhZBPc9jak5ZtE3f598tvxJmZO+ylz0hMOKMmqmkQlQdlM3jbLLhoUc7XVu2icjnTbD37EPC9TdxlgTGmn4BccEZNVFPTlQe2GXzZLHZMZNOHSDbD9HWyd9GHlqK49G719BkcOrrk/VRx208Hsf0E5IKBmqimpisPbNuvixbYWtsEEGB9oxMui4KTj5O9TR9Oik4JXta7GgBOra5vGoOvIGlaYMxm6LZ/vxgx9UFUU9Md3mwz+KIFtgvOO+dskM6Y0jN10gKmD6fJ8TbOf9nWeWGIFFHR/fcuSqZYe80ZNVFNvn90t3GZwednqJfN3lf4XqYt2f2mBcpmszcfOOY0Bh/KZugp1l5zRk3kQVbm9tTc9Tgyuzfoj9D9zOBd+kMD9RdGy2azrmMILcUe15xREyWmnxm8LW+b8ZEWMM1mXccQWtM/AfnAQE2UoKqLb67BKWRaIKYA6XPxsgnsR00UqUH0WR5kj+pRV9aPmjNqoggNqtY3plkvvcQpUIvIjwH8HMAGgDOmqE9Efgyy1je1tMAoqDKjvlZVfxZsJER0Voq1vhQOUx9EEcjno8e3t3BqdX3LdTHX+lI4rnXUCuDrIrIoIvuLLhCR/SKyICILy8vL/kZINOSKGhm98H9n0Brb3M4o9lpfCsd1Rj2jqidF5FcBfENEjqvqg70XqOo8gHmgU/XheZxEQyM/e149fWZLPnr9RcV4u4XzX3YOF/XILVCr6snuf58RkXsAvBHAg+VfRUR5RdUcJs+trePYbW9ramgUMWvqQ0TOF5ELs98DeBuAx0IPjGgYuTT1zzAfTRmXGfWvAbhHRLLr/0VVvxZ0VERDyrVqg/lo6mUN1Kr6IwBXNTAWoqFn2qLNfDSVYXkeJWkQ26t9MDUm+ti732Acf++9XtRuQQRYWV1P6r6pHgZqSk6T26ttHwhVPzCqbtHO32vvCSlF953qBxiVY1MmSs7M3GHjqdVHZvd6+z62BkVNNDAy3Wuv7L7ZUCltZU2ZeHAAJaep7dW2JvpNnD7uck/ZNYM4DZ2awUBNyWnqpBDbB0ITHxgu95Rdw/4gw4uBmpLT1FFKtg+EJj4wiu61V+99x3LUFfnHQE3JsZ0y7YvtA6GJD4z8vY63W3jF9lbhfYcYT50TyckfLiYSlfBd9THo8VZ9Ly5ONqdsMZGBmihxoT4smqquoQ4exUWUiKpBN2RNORcn48EcNVHOoPKyRX2pP3L3o6XfP2RJHhcn48FATUkJHUT7CZau72sbdz9BN+Sst6nqGrJjoKZkhAqivULMUF3H3U/QDTnrbaq6huyYo6ZkNHEyd4gZquu4TZ31yoKuqcmTr1kvTySPAwM1JaOpnYBVg6WNaXxLK2uYmTt8duHw2ssncHBxqVLQrdrkidLEQE3JCBFE80LMUE3jFrx0FNfSyhoOLi7hxj2TeOD4cqWgy1nv8GOgpmSE/jEfKJ+h9luvXDRuAZDfwbC2voEHji83UqMc20YdKscNL5SUQQWYol16APCK7S3c9q7ipv9lDf9NrUsFwFNz14e4hU3j4o7D+HDDCw0N24/5oQK56VDaU6vrhRtMihr+t1tj+NT7d2Hf7knjrr8mapSbWJQlv1ieR17E0LwnZPle2YJlUfmercxvkDXK3HGYHudALSJjInJURO4NOSBKTxP1zS5MwfGWux6pPRbbTDcf5GzBcJA1ytxxmJ4qqY8PA3gCwMsDjYUSFcuP0qbguKFau/9F0YJgr94gd+joEraJYKNg/af3ukFVazSxKEt+Oc2oReTVAK4H8Lmww6EUxfKjdNmMsO7uwmwGPN5ubXmtN8hlP10UBelYgiF3HKbHdUb9aQB/AeBC0wUish/AfgCYmpqqPTBKRxP1zS5ss966HxzZDLhswdK06DgmElUwZO11WqyBWkTeCeAZVV0UkTebrlPVeQDzQKc8z9cAKX6x/CidBZ5b7nrEmnao+31MQc70YfCiKgMj9c0l9TED4N0i8mMAXwKwV0T+OeioKCkx/Si9b/ckPvm+qwZWUcGFOgqh0oaX7oz6z1X1nWXXccMLDVpMG2O4mYRccMMLjZxB5WDZJIlC4BZyGphB9ptgrwuKDWfUFB2fZ/35Omdw4cSzlTvXETWBM2oaCFOvC6CzGOkaJKvkhLOAXtYQqff/BuaWqUllM2r2+qCBKKtprrIF3eXorENHl7Dr41/HTQeOGYM0UNx21MchsUR1MVDTQNjK1VyDpG1XZDbjXllbrz7IkvcnahIDNTnz2SGvqHtcnkuQtNUtm3YK5knF9ydqEgM1OfHdIa93k4yJS5C0tQt1CfaT42188JqpgW2SIbJh1Qc5CdEhr7d3Rr9b0G11y2WnqeQXC6df+0qW7FGUGKjJScgOeXU3iZRtbjE1aio6QouNiihWDNRDoInNG6E75IUKktwpSMOAgTpxPjeOlImlQ14/OFOm1HExMXEudcQ+xNQhj2jUcEaduCZPV2liZtqbxrmo3YJI56Tvse7RVlV2LRINCwbqxMVyuoqNSx49n8bp3aSSHQQQKrVDFDOmPhJnqyOOgWsNtuvmFG7tplHDQJ24FHLHpjz6TQeObdrhWCVdw63dNEqY+hgCsVc1uDRgAso3p+TFltohComBmrzL56PHt7dwatXcFClLZdhOEc/EltohCo2BmrwqqutubRO0xgTrG+be5ydX1rZsTmHVB1EHAzV5VZSPXn9RMd5u4fyXnWNMbWSpjKbTODySi1LAxUTyypSPfm5tHUdm9+LT798VTZWK746ARKFYA7WInCci3xWRR0TkcRH5eBMDo5f47AMdmq0/dExVKk3t6iSqyyX18UsAe1X1BRFpAfiWiHxVVb8TeGyE5np5+OLSEySWKpUmd3US1WEN1No5/faF7h9b3V/+T8SlQj77QDeRj02pW10quzqJnE4hF5ExAIsAfh3A36rqXxZcsx/AfgCYmprac+LECc9DHU2Xzd5X+KkoAJ6au975fYqa82enbhdVUozCIluVE8yJQis7hdyp6kNVNwDsEpFxAPeIyBWq+ljumnkA8wAwPT3NGbcnvmZ9RTPz7CHl0ym2dEtR46SV1fXkAnpKs38abZXK81R1RUS+CeDtAB6zXD60mpxt+uoDbcu79qZTbItspsZJsefPi8SSLycq41L1MdGdSUNE2gDeCuB44HFFq+mSLl9VEi4z8CyYly2y2RonsWqCyD+XGfUlAD7fzVNvA3CXqt4bdljxCnHIq42PWZ/L9uwsmJvSLdtEnHpxsGqCyC+Xqo/vAdjdwFiSkGJJV5aqWVvfOLsVO1tIzPSmU0xBvejrimQBfxQWJImawJ2JFdk2dMSmN1UDdIJtuzWGD14zZUynZOmWMZEt76foVIuYZAGfu/6I/GGvj4piOuTVZcZqStU8cHwZR2b3Gt973+5J3HzgWOFrWUlfWdXHzNzhxlNERMOKgbqiWEq6XHcs1knVmHLVk+Pt0iBf9/sS0WYM1H2IoaTLdVGzTh12nZ8euOuPyB/mqAML1VDJdcZa50zFOqWBKZzlSJQKzqgDCtlQyXXGWjdV0+9PD7GkiIiGgVOvj6qmp6d1YWHB+/umZmbucGEwHRPBJ993Va2gxT4VRMOldq8P6o8pPbGhWntmzRkr0ehgoA6o7FRtH6Vqvhc1uUGFKE5cTAyoaEGtV0ylatygQhQvBuqAynb4AXGVqvFYKqJ4MfURWJY6aHo3Y9U0BjeoEMWLgboBLgt/PvPD/ZQFcoMKUbwYqBtStvDnu966n1asMfUwIaLNmKOOgM/88KGjS8ZKk7I0hq8DCojIP86oI+ArP5zNzE1saYwYepgQ0VacUUfAV4/rsmOymMYgShdn1DX5WAQ05YevvXwCM3OHa1duAGAagyhhDNQ1+FoELKoKufbyCRxcXPJSuTE53maQJkrY0AbqujPdOqenuG4NL/se/ZyQwsoNouFkDdQi8hoAXwDwKgAvAphX1c+EHlgddWe6TZyeYvse/bx3P42a2N+DKH4uM+ozAG5R1YdF5EIAiyLyDVX9fuCx9a3uTLeJ01Ns36Pf965SuRGyXzYR+WOt+lDVn6rqw93f/xzAEwCi/r+4brlbE6en2L5HEyeksL8HURoqleeJyA4AuwE8VPDafhFZEJGF5eVlT8PrT91yN9evr7NJxPY9mtiAwv4eRGlwXkwUkQsAHARwk6o+n39dVecBzAOdE168jdBBPs+ar5gAqs1GqyzK9btJxOV7hN6Awv4eRGlwmlGLSAudIH2nqt4ddkjVFPVRPri4hBv3TPY9G+1nNlv1ENsYtmzzAFqiNFjPTBQRAfB5AM+q6k0ub9rkmYmmcwknx9s4Mru39vu7VEWkfH4hqz6I4lB2ZqJLoP4tAP8O4FF0yvMA4KOq+hXT14QM1PnAYmpAJACemru+8vv1BqqiACwAFJ0Pguza0B8WRDT8ah1uq6rfQic+DVxROVkWOPNc8qy28rSiqojse/Vey0U5IgopqaZMpsCZ/xRxzbPaytNsgTa71ldTJSKiIkkFalPgzFIRAmC83cJ5rW24+cAx66KebSbsEmhPrqxxUY6IgkoqUJsCZ5YL/tT7d+GXZ17EqdV1p5O0bTNh2yni2bUxVHAQ0fBKqimTrfa46tZx2/v19s4oyofnr2VgJqIQkgrUtqZDVRf1XJoY9QbgGErZYhgDETXLWp7XjybrqHsNe5lcyvXaRFSurDwvqRy1TSqLelV3MWbYRIloNCWV+rDppx9z0+q0FmW9NtFoSjpQm/K1/QZmX/nfsvep0yubTZSIRlOygdp303tf71f35JayIM+jtohGU7I5at/5Wl/vZ3ufstrtok6AvXXgrNcmGk3Rzahd0w++87W+3s/l5BbTrNglLcJ6baLRE1WgNqUNFk48iweOL28K3r7ztb7ez/Y+ZQueNx84VvieXCwkGm1RBWrTjPLO7/zXlq51N+6ZrHWKS56v/G+dk1u4WEhERaLKUZc1Xeq1tr6BB44ve83X+sr/1nmfVOrAiahZUe1MNO0sLOJ6MEBquEWcaDTVOjigSUVpgzoHA6SIi4VElBdV6qMobfDBa6aYDiCikRbVjBoonlFOv/aVTAcQ0ciKLlAXYTqAiEaZNVCLyB0A3gngGVW9IvyQwuFCHRGlyGVG/U8APgvgC2GHspXPwOq7NwgRUVOsi4mq+iCAZxsYyya2vhdVsZczEaUqqqqPXr4DK3s5E1GqvAVqEdkvIgsisrC8vFz7/XwHVtuJ40REsfIWqFV1XlWnVXV6YmKi9vv5Dqzcnk1EqYo29eEzsGaLkmvrGxgTAQCMt1s4r7UNNx84VuncQiKiplkDtYh8EcC3AewUkadF5I/CD8tfk6TeRUkA2FBFa5vgF6fP4NTqupeFSiKikKJqyhRClUZPk+NtHJndG3hERERblTVlijb14UuVxUdWgBBRjIY+UFdZfGQFCBHFaOgDddGiZGuboDUmm/6OFSBEFKskmjLVYTqjsOjvuJWciGI09IuJREQpSOKEF3a2IyIqFkWgZmc7IiKzKBYT2dmOiMgsikDNznZERGZRBGp2tiMiMosiULOzHRGRWRSLiaZaZy4kEhFFEqgBnjRORGQSReqDiIjMGKiJiCLHQE1EFDkGaiKiyDFQExFFLkj3PBFZBnCizy+/GMDPPA5nkIblXoblPgDeS4yG5T6AevfyWlWdKHohSKCuQ0QWTK3+UjMs9zIs9wHwXmI0LPcBhLsXpj6IiCLHQE1EFLkYA/X8oAfg0bDcy7DcB8B7idGw3AcQ6F6iy1ETEdFmMc6oiYioBwM1EVHkBhKoReTtIvKkiPxARGYLXhcR+Zvu698TkasHMU4XDvfyZhF5TkSOdX/91SDGaSMid4jIMyLymOH1lJ6J7V5SeSavEZEHROQJEXlcRD5ccE0Sz8XxXlJ5LueJyHdF5JHuvXy84Bq/z0VVG/0FYAzADwG8DsC5AB4B8Bu5a94B4KsABMA1AB5qepwe7+XNAO4d9Fgd7uVNAK4G8Jjh9SSeieO9pPJMLgFwdff3FwL4z4T/X3G5l1SeiwC4oPv7FoCHAFwT8rkMYkb9RgA/UNUfqeppAF8C8J7cNe8B8AXt+A6AcRG5pOmBOnC5lySo6oMAni25JJVn4nIvSVDVn6rqw93f/xzAEwDyTduTeC6O95KE7r/1C90/trq/8lUZXp/LIAL1JICf9Pz5aWx9YC7XxMB1nL/Z/THpqyLyhmaG5l0qz8RVUs9ERHYA2I3O7K1Xcs+l5F6ARJ6LiIyJyDEAzwD4hqoGfS6DOOFFCv4u/2nkck0MXMb5MDp7+F8QkXcAOATg9aEHFkAqz8RFUs9ERC4AcBDATar6fP7lgi+J9rlY7iWZ56KqGwB2icg4gHtE5ApV7V0T8fpcBjGjfhrAa3r+/GoAJ/u4JgbWcarq89mPSar6FQAtEbm4uSF6k8ozsUrpmYhIC53Adqeq3l1wSTLPxXYvKT2XjKquAPgmgLfnXvL6XAYRqP8DwOtF5DIRORfABwB8OXfNlwH8fnfl9BoAz6nqT5seqAPrvYjIq0REur9/Izr/5v/b+EjrS+WZWKXyTLpj/AcAT6jqXxsuS+K5uNxLQs9lojuThoi0AbwVwPHcZV6fS+OpD1U9IyJ/CuB+dKom7lDVx0Xkj7uv/z2Ar6CzavoDAKsAPtT0OF043st7AfyJiJwBsAbgA9pdFo6JiHwRnVX3i0XkaQC3obNIktQzAZzuJYlnAmAGwO8BeLSbDwWAjwKYApJ7Li73kspzuQTA50VkDJ0Pk7tU9d6QMYxbyImIIsediUREkWOgJiKKHAM1EVHkGKiJiCLHQE1EFDkGaiKiyDFQExFF7v8B4SC4LI9GLoEAAAAASUVORK5CYII=",
|
|
"text/plain": [
|
|
"<Figure size 432x288 with 1 Axes>"
|
|
]
|
|
},
|
|
"metadata": {
|
|
"needs_background": "light"
|
|
},
|
|
"output_type": "display_data"
|
|
}
|
|
],
|
|
"source": [
|
|
"np.random.seed(13) # pick the seed for reproducability - change it to explore the effects of random variations\n",
|
|
"\n",
|
|
"train_x = np.linspace(0, 3, 120)\n",
|
|
"train_labels = 2 * train_x + 0.9 + np.random.randn(*train_x.shape) * 0.5\n",
|
|
"\n",
|
|
"plt.scatter(train_x,train_labels)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "Ng4rZmGc6oxk"
|
|
},
|
|
"source": [
|
|
"Linear regression is defined by a straight line $f_{W,b}(x) = Wx+b$, where $W, b$ are model parameters that we need to find. An error on our dataset $\\{x_i,y_u\\}_{i=1}^N$ (also called **loss function**) can be defined as mean square error:\n",
|
|
"$$\n",
|
|
"\\mathcal{L}(W,b) = {1\\over N}\\sum_{i=1}^N (f_{W,b}(x_i)-y_i)^2\n",
|
|
"$$\n",
|
|
"\n",
|
|
"Let's define our model and loss function:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 8,
|
|
"metadata": {
|
|
"id": "QxhI4GlB6aiH"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"input_dim = 1\n",
|
|
"output_dim = 1\n",
|
|
"learning_rate = 0.1\n",
|
|
"\n",
|
|
"# This is our weight matrix\n",
|
|
"w = tf.Variable([[100.0]])\n",
|
|
"# This is our bias vector\n",
|
|
"b = tf.Variable(tf.zeros(shape=(output_dim,)))\n",
|
|
"\n",
|
|
"def f(x):\n",
|
|
" return tf.matmul(x,w) + b\n",
|
|
"\n",
|
|
"def compute_loss(labels, predictions):\n",
|
|
" return tf.reduce_mean(tf.square(labels - predictions))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "JUxwj3367gD2"
|
|
},
|
|
"source": [
|
|
"We will train the model on a series of minibatches. We will use gradient descent, adjusting model parameters using the following formulae:\n",
|
|
"$$\n",
|
|
"\\begin{array}{l}\n",
|
|
"W^{(n+1)}=W^{(n)}-\\eta\\frac{\\partial\\mathcal{L}}{\\partial W} \\\\\n",
|
|
"b^{(n+1)}=b^{(n)}-\\eta\\frac{\\partial\\mathcal{L}}{\\partial b} \\\\\n",
|
|
"\\end{array}\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 11,
|
|
"metadata": {
|
|
"id": "-991PErM7fJU"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def train_on_batch(x, y):\n",
|
|
" with tf.GradientTape() as tape:\n",
|
|
" predictions = f(x)\n",
|
|
" loss = compute_loss(y, predictions)\n",
|
|
" # Note that `tape.gradient` works with a list as well (w, b).\n",
|
|
" dloss_dw, dloss_db = tape.gradient(loss, [w, b])\n",
|
|
" w.assign_sub(learning_rate * dloss_dw)\n",
|
|
" b.assign_sub(learning_rate * dloss_db)\n",
|
|
" return loss"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "idr2VEWb9rr0"
|
|
},
|
|
"source": [
|
|
"Let's do the training. We will do several passes through the dataset (so-called **epochs**), divide it into minibatches and call the function defined above:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 12,
|
|
"metadata": {
|
|
"id": "nOuu0qpx-wAp"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Shuffle the data.\n",
|
|
"indices = np.random.permutation(len(train_x))\n",
|
|
"features = tf.constant(train_x[indices],dtype=tf.float32)\n",
|
|
"labels = tf.constant(train_labels[indices],dtype=tf.float32)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 13,
|
|
"metadata": {
|
|
"colab": {
|
|
"base_uri": "https://localhost:8080/"
|
|
},
|
|
"id": "3zdIf6c_85Ht",
|
|
"outputId": "43b04684-8b90-4c65-d5ff-20ebac61c73c"
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 0: last batch loss = 63.5879\n",
|
|
"Epoch 1: last batch loss = 6.0149\n",
|
|
"Epoch 2: last batch loss = 0.9082\n",
|
|
"Epoch 3: last batch loss = 0.3712\n",
|
|
"Epoch 4: last batch loss = 0.2928\n",
|
|
"Epoch 5: last batch loss = 0.2765\n",
|
|
"Epoch 6: last batch loss = 0.2724\n",
|
|
"Epoch 7: last batch loss = 0.2712\n",
|
|
"Epoch 8: last batch loss = 0.2709\n",
|
|
"Epoch 9: last batch loss = 0.2708\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"batch_size = 4\n",
|
|
"for epoch in range(10):\n",
|
|
" for i in range(0,len(features),batch_size):\n",
|
|
" loss = train_on_batch(tf.reshape(features[i:i+batch_size],(-1,1)),tf.reshape(labels[i:i+batch_size],(-1,1)))\n",
|
|
" print('Epoch %d: last batch loss = %.4f' % (epoch, float(loss)))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"We now have obtained optimized parameters $W$ and $b$. Note that their values are similar to the original values used when generating the dataset ($W=2, b=1$)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 14,
|
|
"metadata": {
|
|
"colab": {
|
|
"base_uri": "https://localhost:8080/"
|
|
},
|
|
"id": "US6q0nCBD-LL",
|
|
"outputId": "65a79620-a3eb-445b-aafb-60a60575ab0e"
|
|
},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"(<tf.Variable 'Variable:0' shape=(1, 1) dtype=float32, numpy=array([[2.00624]], dtype=float32)>,\n",
|
|
" <tf.Variable 'Variable:0' shape=(1,) dtype=float32, numpy=array([1.0982299], dtype=float32)>)"
|
|
]
|
|
},
|
|
"execution_count": 14,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"w,b"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 15,
|
|
"metadata": {
|
|
"colab": {
|
|
"base_uri": "https://localhost:8080/",
|
|
"height": 282
|
|
},
|
|
"id": "_e6xRMZFDnyI",
|
|
"outputId": "d202b7fe-4383-4d82-b98e-a20f3180093e"
|
|
},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"[<matplotlib.lines.Line2D at 0x203d1cfa340>]"
|
|
]
|
|
},
|
|
"execution_count": 15,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
},
|
|
{
|
|
"data": {
|
|
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAWoAAAD4CAYAAADFAawfAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAArjUlEQVR4nO3deXxU1fnH8c8hBAiCRCQiRCloMViQRSg/K3UBq6gsplpbcalaFZe6gCgGtCxuoLhXqwXFFQUtiIoiRRZxqwoCigLWBZQABYSwJUASzu+Pm8QwzHInc2dyb/J9v168BObOzLnOi2dOnvOc5xhrLSIi4l91qnsAIiISnQK1iIjPKVCLiPicArWIiM8pUIuI+FzdZLxos2bNbOvWrZPx0iIiNdKiRYs2WWuzwj2WlEDdunVrFi5cmIyXFhGpkYwxqyM9ptSHiIjPKVCLiPicArWIiM8pUIuI+JwCtYiIzyWl6kNEpDaZvjifcbNWsragiJaZGdzcO4fcLtmevb4CtYhIAqYvzmfYtC8oKi4FIL+giGHTvgDwLFgr9SEikoBxs1ZWBOlyRcWljJu10rP3UKAWEUnA2oKiuP6+KhSoRUQS0DIzI66/rwoFahERnFxzj7FzaZP3Jj3GzmX64nxXz7u5dw4Z6Wn7/F1Geho3987xbGxaTBSRWi+RBcHyx1X1ISKSRNEWBN0E3Nwu2c51hYXQsKHn41PqQ0RqvYQXBNesgQsugB49oLQ09vVxihmojTE5xpgllX5tM8YM8nwkIiLVpMoLgkVFcMcdkJMDU6dC375QUuL5+GIGamvtSmttZ2ttZ6ArUAi86vlIRESqSdwLgtbCK69Au3YwYgSceSasWOEE7fr1PR9fvKmPU4BvrbURG1yLiARNbpdsxpx9DNmZGRggOzODMWcfEzY/Pe+lWSw5ohP88Y+s3F2Xyy8dR5sjL6HH5O9cV4rEy1hr3V9szETgM2vto2EeGwgMBGjVqlXX1asVy0WkBtmwgVVXDabVqy+xJaMx9534Z6Z0PJW9dX6eiWekp0UM8LEYYxZZa7uFfcxtoDbG1APWAu2ttf+Ldm23bt2sjuISkSAKbbA0tFcbznr/VRg9mpIdO3nm2L480mMA2xo0Cvv87MwMPsjrFff7RgvU8ZTnnYEzm44apEVEgiq0nrrtogV0HPdn2JwPp59O78PO4tuDD4/6Gl5uHS8XT456APCS5yMQEfGJ8nrqI3/6kadfGckz/xrNXuCmi++GmTPZdeRRMV/Dy63j5VwFamNMQ+BUYJrnIxAR8Ynt6zdy25wJvD3xWrquWc4dPS/j9L88ytRDOwLhq0Mq83rreDlXqQ9rbSFwsOfvLiLiB6Wl8NRTvPvkUJrs3MbkTqdx/wkX8dMBmYCTd4b9t4s3yUjHGCgoLE7K1vFy2kIuIrXbu+/CDTfA0qWUdvk//tD1Qj47uE3Fw6Gz5Irt4imkLeQiUjutWgXnngsnnwxbtsCUKTRb9BF/vub3ruqpU0kzahGpXXbuhLFjYdw4qFMHRo+Gm26qaKZUHTPmWBSoRaR2sBZefBFuuQXy8+H8852AfXj0cjs/UOpDRGq+Tz91OttdeCEceii8/z5MmhSIIA0K1CJSk61bB5deCt27w3ffwcSJ8MknTtAOEAVqEal5du1y0hpHHeXMnIcOZcYr8+nxv1/QZvjMuI7a8gPlqEWkRpi+OJ9xb6+g/afzGPnuRLI3r4P+/eG++5i+o2GVjtoq7/uRX1BEmjGUWkt2EuulI1GgFpG4hDYtSnXQijSmiU+8wT2znuC3q5fy9cGtuGzAnfS7+RLYAUNeXkppSAO6WEdthfb9KH9+POcpeiWuNqduqXueSM0UGrwgsdaenvjpJ6b2vYzcj99gR70MHjjhQiZ1PoOStLpkZqSzu2TvfuchVpadmRH2S6fH2LnkR2mwVNUueZFE656nHLWIuBbtENiUKymBRx+Ftm3J/fgNXuhyBiddOYFnu/ajJM1JFhQUFUcN0gZnhmz5eaZcnruO1QUvGV3yIlHqQ0RcS/gQWK+88w4MGgRffgm9enHJMQN4r0GLuF7CAKH5hMrpkJaZGVFn1MnokheJZtQi4lqk4NQkI50eY+fSJu/N5FZUfPMN5ObCqac6B8u++iq88w7nXHxG2DMPD2qYHvZl0ozZL0iXK//SidYpL1ld8iJRoBYR18IFr/Q6hp17SiKmEDyxfTvk5UH79vDOO3x5XR49L3mMNv9Jp8c98wDCnnk4sl/7sAH8/j92quiIF6plpU555a8JTnCH6un/odSHiLgW2uazZWYGhXtK2FJYvM91sSoqXNu7F557DoYNg/Xr4eKLefv86xj83kaKdu5bajfm7GMiLu5FqlIJtzBa3Z3ywlHVh4gkpE3em2HTCAb4fmyfqr/whx867UcXLoTjjoOHH4bu3SNWY1SlCsNPpYZenZkoIrKfSItuVV5sW7PGaZz04ovQsiW88AIMGOB0usPbBU2/zJhjUY5aRBISLm9dpcW2oiK44w7IyYGpU+G222DlSrjggoogDZG/AFJZhZFqCtQikpDKi25VarZvLbzyChx9NIwYAWeeCcuXO0G7UaP9LvfsiyFAlPoQkYRVOYWwZImTh16wADp2hHnznBNXYrwXRF4grIlcBWpjTCbwJNABp0b8L9baj5I4LhGpyTZudFIbEyZA06bwxBNw+eWQFvmE78qCklv2itsZ9cPA29baPxhj6gENkzgmEamp9uyBxx5zjr/audOZTY8YAQcd5OrpfqrSSKWYgdoYcyBwInAJgLV2D7AnucMSkRpn5kwYPNhZIOzdGx580MlLuxTaEKo6uthVFzeLiUcAG4GnjTGLjTFPGmMOCL3IGDPQGLPQGLNw48aNng9URKrX9MX5VdsmvnIl9OnjLBLu3QszZjhBO44gDT5rCJVibgJ1XeBY4HFrbRdgJ5AXepG1dry1tpu1tltWVpbHwxSR6lQ+m41rm3hBAdx4I3To4JxReN99sGyZE7TLtmO7fe9oLUdT3hCqGrjJUa8B1lhrPy77878IE6hFpOaKNJsd9fqX++eMOx4KTz3F7luGkb51C1M6nsYLfa/gil7HkVuvXlzvG67/daiaXD9dLmagttauN8b8aIzJsdauBE4Bvkr+0ETELyLNWguKiikocvp85BcUMe3BSfT8+FmafP0VXxzegZG5I/my+ZFQQpXyyeG+ICqr6fXT5dxWfVwHTCqr+PgOuDR5QxIRv4nVmzl76waGzZtI35Xvs77JIdw34G88f3j3fVIcVWnUFC2tUR1nF1YXV4HaWrsECNssRERqvpt754RNQWTs2cVVH/+LKz+ZhsXwwG8vYEL337MrvUHY14k3nxzpC8LrY7D8TjsTRSSm/XYDNmnAiYtmc/2sJ2mx4ydeO/okxp58CesOzKro3xyrUZObmuhwXxC1Jd1RmQK1iLhSsRtw4UJno8qHH/Llob/kurOGsvCw9sC+QTRagHVTE10eyIuKS0kzhlJra1W6ozIFapFaLK6dfuvXw/Dh8PTTcMgh8NRT/LfTqayb/V9MhOdHeu1oNdG5XbL3C+Sl1lYE+toWpEEHB4jUWuFK3zLS0/bvfLd7Nzz0ENx5p/P7QYOcPh0HHljl94502AA4+edIC5fhctM1ZVt5tIMD1OZUpJaKudPPWnjtNeecwrw86NnTOfX73nsTCtIQufbZED63XS50MbJKG3ECSIFapJaKelLKsmVw2mnOid/16sGsWfD669C2rSfvHa6ntIGIs+xyoQG+tmwrV45apIaLlBoIV/rWpGg7g9+fRMm4t9h7QGPqPfIIXHUVpKd7OqbQKpImGekVG2ciCVft4eWxXH6mQC1Sg0Wrrqhc+pa2t5Tzl8zkxvcmceDunUzqfAaP97yIW377W3I9CNKRviwqLxxGE6naw/PzGn1KgVqkBioPjOGCWHlqoHxRbt5jL3LNa4+Ss+kHPmzVkdG/G8jKrNYAce8kjDSWaKV40baJh13crCRSnXXPdln0GDs38AuM5RSoRTzgp8oDN42M1hYUwbffkjt6CLmvvcYPTZpz5e+HM6vtb/bZ9u1FCiFWKV6094h19mK4Y7l6tsti6qL8GtW3WoFaJEF+a2gfq5HRAbsLyVs8DR6c5uSe776bi0u68P3O/Z9TlRRC6JdWrPak0baJu/n/F3osV4+xc6N+MQSRqj5EElQdlQfRmvhHmqEau5c/fPEO85+8kovenQznnQdffw3DhnFD32M8Odk7XLlcpM7T5V8CXp8q7maBscqHIFQTzahFEpTqyoNYM/hwM9Rj85cz8p3xdFr/Xxa3yGHYn++k7+VnkduyZcXzIPGTvcN9aVnCl94V7ilh+uJ8z08Vj7XA6LefgNxQoBZJUKorD2LlfCsvsB26bRO3vPsMv/9qPusbHcygvkN47VcnYU0dPggJTl6c7B3py8kCmSEleFsKi/cJkF4FyViNnGL9//MjBWqRBKW6w1usGXxul2zq7Cpi/ci7uXD+i6TZvUw4cQAPdj2bwno/f3lECk6JLIxGyzcD+9VKJyNAxpqhB7H2WoFaJEFe/+geS9QZvLUwdSr9b7oJVq+Gc86BceO4+59fhd31F2lLdlXTAtG+tAZPWRL2OckIkNFm6EGsvdZioogHcrtk80FeL74f24cP8nol9UfoSItvd7YucfpxnHsuNGkCc+fCv/4FbdpEDEJeb8nO7ZLNmLOPITszA4Mzky4vsXM7hmTzevEyFTSjFgmY0Bn8r+ru5tGvptHm7pfgoIPg8cfh8suh7s//vN2mZ7xIC0SazfrlEIBU/wTkBQVqkQDK7ZJNbvsseOwxGD0adu6E66+HESOcYB3meogdnJKZFvBTgPRy8TIV1I9axKeiLurNnAmDB8PKldC7Nzz4IBx9tCfv6apHtXguWj9qzahFfCjSol6jVd/yuyfvgbfeclqOzpgBZ565z7bvRPhp1is/cxWojTGrgO1AKVASKeqLiDdCF/UO3LWD6+ZO5uS734BGB8B998F11zm9oj0WtLRAbRDPjLqntXZT0kYiIhXKF+/q7C3lj5/P5qb3nqdp4TZe7ngq5/37OWjevJpHKKmk1IeID4TmozMbptN25WJGvjOe9hu+45PDfsXF546moN0xnKcgXeu4DdQW+LcxxgL/tNaOD73AGDMQGAjQqlUr70YoUsPtt4C3ejV3vvs0fZa/R37jLK7tP5QZ7U4go15dxvi41leSx22g7mGtXWuMOQSYbYxZYa1dUPmCsuA9HpyqD4/HKVJjhM6eC/eUUFRcSsaeXVz18VSu/GQqFsPjJ1/EKz3P4/tCG/GEE6kdXAVqa+3asv9uMMa8CnQHFkR/loiEClfNgbX0X/4uefOfoeX2Tbx+9ImMPfkS1h14CN+POLOaRyx+EDNQG2MOAOpYa7eX/f404Pakj0ykBgqt5jhm3X8ZOWc83fKX80XzI7m+/80sPKw98HMjIxE3M+rmwKvGqdOsC7xorX07qaMSqaHKqzmydmzh5gXP8ocv5vBTwybcfMb1TO1wCnvrOD0o/N57QlIrZqC21n4HdErBWERqvF80SqP3nClc++EU6pcUM77773n0+POom9mEFvXrapOJhKXyPAkkPx0m64q18MYbzPjn9TRas5rZv+zOXT0vY1XTbKfzXf/2Ecdf+V6bZKRjDBQUFgfjvsUTCtQSOKk8SinWF4KrL4wvv3T6csyeTaOjj+bDR19g1PYWrC0oilnNEXqvlRvvh7vvwH2BiSsK1BI4qTpKKdYXQswvjM2bYdQo+Mc/oHFjePhhuPpqjk9P5wOXY4h1onjl+w7iWYDijg4OkMBJ1VFKsZroR3r8gbe+ctqPtm3r/HfgQPjvf502pOnpcY3BzT2VX1Mdp6FLamhGLYGTqqOUYn0hhHv8+FVLGDlnPGz6wTlt5aGHoGPHKo8h0r2GXuNmvBJcmlFL4KTqKKVYR0dVfrzVlnX8c9qdvDjlNhqX7oGpU2HOnISCNIS/18oq37dfjroS7ylQS+BEO5fPS7G+EG7uncPBe3cz9N1nmP3U1fx21RIe6HkJC996H84+25Me0aH3mpmRzkEN08PedzK+wKYvzqfH2Lm0yXuTHmPnMn1xfgJ3I1WlE15EoohYRbF3Lzz/PLtuGkqDTRuY2qEXz/W7kkvP7VGtC3deVn3otJfUinbCiwK1SLw++ghuuAE+/RS6d4dHHoH/+79qG06ySvJ6jJ0bNj+enZnBB3m9En592ZeO4hLxQn4+5OXBCy9Aixbw3HNwwQVQx7sMYrxBN5kleVqc9A8FapEQocHylpNa0X/OZBgzBkpLYfhwGDYMGjXy/H3jDbrJrClPVXWNxKZALYGS7J13+wRLa+n4n9l0HTsRtm5wFgjHjYMjjkjKuKsSdJM56725d07YHLWaRaWeArUERip23pUHy6M3fMfId8Zz3I/LWJ7VmrGX38ffJwxJ6rirEnSTOevVieT+oUAtgZGKreO71q7nrvee57yl/2Zrg0bceto1TO7Um7110vh7FV/T7birEnSTPevVieT+oEAtgZHUxa3iYnjsMeZPuI2M3UU8e2xfHvrt+Wxr4OShE2niH2l8+QVF9Bg7t2K22rNdFlMX5ccVdDXrrR0UqCUwkvZj/ttvO93tVqyg8LiTGNBpAMsyD6t4ONEZaqRxG6j4+/yCIqYuyuecrtnMW7ExrqCrWW/Np0AtgeH5j/lffw033ghvvgm//CW88QbN+/Th8iVrw85Qq7qQGW7cBgjdwVBUXMq8FRtTUqOsdqjBog0vEiieBJitW+GOO5y2oxkZMGKE09muXr2o7xsabAEOapjOyH7hm/5Ha/gfqdGSAb4f2ye++4mTdhz6kza8SI0R68f8qIG8tBQmToRbb4VNm+Avf4G77oLmzWO+b6S+0FsKi8NWcIRr+J+RnsaDf+pMbpfsiLv+UlGjnKp+3uIdNWUST/iheU95cMwvKMLycxnc9MX5sGABdOvm9IbOyYGFC+HJJ10FaYi+YBmu53Os3tCp6gAYjnYcBo/rQG2MSTPGLDbGzEjmgCR4ogbIFAoXHJtuWkf6+QPgpJPgp59g8mQnaB97bFyvHWumGxrkYgXDVHUADEftUIMnntTHDcBy4MAkjUUCyi8/SlcOjhl7dnHVx1O58pOpWAx/P/FCWo8ZSb/jf1ml1w63IFhZ5SA3fXE+dYyhNMz6T+XrqqtaQzsOg8dVoDbGHAb0Ae4CbkzqiCRw/PKjdMvMDPK3FNJ/+QLy5j9Ny+2beP3oExl78iWsPfAQshf8UOVAXR5QR73+5T4HzMK+Qa78p4twQdovwVC118Hjdkb9EDAUaBzpAmPMQGAgQKtWrRIemASHX5r33Hn4bg587Ba6rvmKZc2P5IZ+N/Hp4R0qHk/0i6N8BhxtwTLSomOaMb6qqlDtdbDEDNTGmL7ABmvtImPMyZGus9aOB8aDU57n1QDF/6r9R+n16+HWW+n59NPsOuhg8s64npc7nMLeOvsu1nn1xREtyEX6MthrrQKjVJmbxcQeQH9jzCpgMtDLGPNCUkclgVJtC2O7dzvd7I46Cp5/HoYMocF333DcXUOpX3/fmuhUfXFooU6SIa4NL2Uz6pustX2jXacNL5JU1sKMGc6uwm++gb594f77nYBdprp23mkziVSVNrxIzfHVV05fjn//G9q1g5kz4fTT97usunKwWqiTZIgrUFtr5wPzkzISqXXimvVu3gyjRsE//gGNGzvbv6++GtLTk//ecdJCnXhNM2qpFq4PASgpgfHj4W9/g4ICuPJKuP12aNZsn9fy4pzBhas3x925TiQV1JRJqkWkXhfgLEbe3DuH3C0rndO+ly2Dk092ZtEdO+5zbTw54fKAHq0hUuV/DcotSypFy1Gr14dUi2g1zXVWfc8BA/4Ip5wCO3bA1Kkwd+5+QRpi99QAJ0B3Hv1vBk1ZEjFIQ/i2o6E9PESqg1IfUi3CbZI5YHch1/znFS7/9FVK6tTln6f9hStfewwaNIj4OrF2RUZqT+qWGhWJHyhQi2teLsBV3iRj7F7OXjaPoQuepfmOzUxt35N7T7qYDY2bcWWUIA2xd0VG2ikYKlwj/8qvI1KdFKjFFa9PAC9/zlsTXuWa6Y/Qed1/WdLiKK7KHc7i7HaAu3MKY+2KdDMjzq7ieYUiqaJALa543iEvP5/cB/LIfeEFipo1J6//EKa0OwlrnGUTt0EyVt1ytNNUQhcLu/2iqeqfxZcUqMUVzzrkFRXBAw/A3Xc7pXfDhpExbBjHfbON96oYJKPVLUdqTxruCC3VP4tfKVDXAKnYLp1whzxrYdo0uOkmWLUKzj7b6dNxxBEA5HZpnJQgqZ2CUhMoUAec17njSBLqkLd0KQwaBPPnQ4cOMGcO9Er+SdvlNFOWoFMddcC5qSP2QpU65G3c6GzzPvZY+PxzZ/v34sUpDdIiNYFm1AGXytNVXM9Mi4udoDxqFGzfDtdeCyNHQtOmMZ9aOY3TJCMdY5yTvtPKjrbKVupCaiEF6oDzy+kqFd5+2+lut2IFnHoqPPQQ/OpXTgAePzdqnjg0jVP5yKvyo62SldoR8TOlPgLu5t45ZKTve5JJtdT/fv210xf6jDOcao7XX4dZsyqCtJtTyt1uTtHWbqltFKgDrtpOVym3datTydGhAyxY4FRyLFsG/fqBMUDkPPqgKUvoMXZuRcCOJ12jrd1Smyj1UQNUS1VDaSk8/TQMHw6bNsFf/gJ33QXNm+93abSgWjmVEW1zSiht7ZbaRIFa4vfee0770cWLoUcP55SVrl0rHg6t685smM6WwuKIL1eeyoi0OSWUtnZLbaNALe798AMMHQpTpsBhh8FLL8Gf/lSR4oDwdd3pdQzpaYbi0si9z9cWFO23OUVVHyIOBWqJrbAQ7r0X7rnH+fPIkU7Abthwv0vD5aOL91oyM9I5oH7diKmN8lRGqtM41XUIrkg8tJgokVkLkydDTg6MHg1nnQUrVzr10WGCNETOR28tKuaDvF489KfO/qhSAdfVKCLVLWagNsY0MMZ8YoxZaoz50hgzOhUDk59NX5xPj7FzaZP35j5VEkm1aBGccAIMGABZWU5Fx+TJ0KpV1KdFWuSrPGOu1iqVSlK1q1MkUW5SH7uBXtbaHcaYdOB9Y8xMa+1/kjw2IXW9PCqsXw+33upUdGRlwYQJcOmlkJYW+7m46wnil94bqdzVKZKImIHaOqff7ij7Y3rZL+9PxJWwvOwDHTUfu3s3PPII3HEH7NoFQ4bAbbdBkyZxvUeQutX5blenSASuTiE3xqQBi4BfAo9Za28Jc81AYCBAq1atuq5evdrjodZObfLeDPutaIDvx/Zx/Trhzg40gLWWP65fwoh5T9Hox1XO7sL772f6zgMCEWwTEc8J5iLJFu0UcldVH9baUqCzMSYTeNUY08FauyzkmvHAeIBu3bppxu0Rr2Z94WbmR276gRFzJnDiqsV8e/DhfP735zn+2gtjplvCNU4qKCwOXEAP0uxfare4yvOstQXGmPnA6cCyGJfXWKks6UqoD3QllfOuB+7aweD3J3HRZ29SWC+D0adcwfNd+tB8R2M+IPYiW6TGSUFsmOSXfLlINDEDtTEmCyguC9IZwO+Ae5I+Mp9K9eKeV7O+lpkZrN+8gwFLZ3Hjey/QZNcOXurUmwdOuJDNDZ08dHkwj7bIFqtxUkLnKIpIWG5m1C2AZ8vy1HWAl621M5I7LP/y/JBXF7yY9d1z0CayHryFnA2r+KjVMdx+yhUsP+SIfa4pT6dESrfUMcZVLw5VTYh4y03Vx+dAlxSMJRACV9L13XesveJafjt3Jj82ac41ucN566jfYCpt+4Z90ymRem6UWussQMZ4y/KAr11/It7QFvI4Baaka8cOGDOG0vvuJ9PCuBMu4slf57I7vT4Z6Wmc0zWbeSs2hg2i5f8d8vLSiob95SxEDdblAT/l9d8iNZgCdZy8WtzzQtgZa6cW8MILkJcH69bx7y6nMuo3F/C/xs0qnldUXMq8FRv5IC/y2YW5XbIZPGVJ2Mcszo7CaFUfPcbOTXmKSKSmUqCOk19KusLNWF/6+yuc+OmzNF22BLp3h2nTuGb6T2Fnv25SNZF+esjOzIga5KO9vm9TRCI+pkBdBX4o6aq8qHnI9p+45d1nOOfLeWxq3BSefRYuvBDq1KHl/LlVTtUk8tNDYFJEIgGg7nlJlqyGSmsLiqhfsodrPnqZeROupO+K93jsuHM5+bIn4M9/hjrOR5vImYqJNFDyzVmOIjWAZtRJlLQFNWs5f82nXDXjCQ7f+j/ePuo33NXzMn7MPJTskBlroqmaqv704JcUkUhN4KrXR7y6detmFy5c6PnrBk2PseHTDmnGcP8fO1UtaH3+uXMM1vz5fJ3VmlG9LufD1p0B9akQCbJovT6U+kiiSAtnpdbG36B+0ya4+mro0sUJ1o89xvI357O682+qva+ziCSXUh9JFO1UbdelasXF8I9/OKeqbN8Of/2r8/umTTkLOOvXv/BsvNqgIuJPmlEnUbgFtcpilqrNmgUdO8KgQfDrX8PSpU7P6KZNvR0oOpZKxM8UqJOovGoiLWS7drmIpWpffw39+sHppzsz6tdec4J2+/ZJG6uOpRLxL6U+kqw8deCqHnnrVrjzTnj4YWjQwDn5+/rroX79uN833jSGNqiI+JcCdQrELFUrLWXx7Q/yiwfuInPHVt7s1pu6d9/NGadWrRdWVcoCtUFFxL8UqFMkYj3y++9TcMU1dFnxBQuzj2b02SP5okVbMhb8j93N8qu0mFeVVqx+6mEiIvtSoK4uP/wAQ4fClCnsbpLF9f1u5vWjT4SyfHYiB9hGqjSJlsbQBhUR/1KgTrXCQif3fO+9YC2MHEnPbcdQWK/BfpfGmx8uT3lEEiuN4YceJiKyP1V9pIq1MHkytGsHo0dD//6wYgWMGsVBhxwU9ileHGBbTmkMkeDSjDpBrqorFi1ytn1/8IGzs3DSJDjhhIqHI+WHe7bLosfYuQlXbgDatSgSYArUCYhZXfG//8Gtt8LEidCsGUyYAJdeCmn7boIJlx/u2S6LqYvyPancyM7MUJAWCbAaG6gT3Q7t5vmRqisefHMZuXNegttvh6IiuPFG+NvfoEkT1+9RlRNSVLkhUjPFDNTGmMOB54BDgb3AeGvtw8keWCISbS/q9vn7pRqspde3n/K3uU/ClrXQpw/cfz/k7B8oY71HVTagVKVyQ/09RPzPzYy6BBhirf3MGNMYWGSMmW2t/SrJY6uyqtQRV+X5lVMNR276kRFzJ3DS95+xOutweOstOOOMKr9HVTegxFO5oQNoRYIhZtWHtXadtfazst9vB5YDvv5XnOh2aLfPv7l3Ds1LChnxznhmTfwrXdauZMypA1kyY0HUIO3mPVJxQor6e4gEQ1w5amNMa6AL8HGYxwYCAwFatWrlxdiqLNHt0K6eX1JC7n9e54ynbqXu1q1M7nQaL/a5givO+T/OcjEbjfUeqdiAov4eIsHgOlAbYxoBU4FB1tptoY9ba8cD48E54cWzEboQmmcNrZiA+GajMRfl5s1zyu2++IL6J50EDz/MBZ06cUEcY3az8JfsDSjq7yESDK42vBhj0nGC9CRr7bTkDik+4fooT12Uzzlds6t0KCtEOdQ1cw+ccw706gXbtsG//uUE7U6d4j7ENpGDY72iA2hFgiHmmYnGGAM8C2y21g5y86KpPDMx0rmE2ZkZfJDXK+HXn744n0dfX8LvZz3H5Z++SlrdutS97Van5C4jo+KacLPjIGwyUdWHiD9EOzPRTeqjB3AR8IUxZknZ3w231r7l0fjiEhpYqtKAKNrrVQ5U0xf9yEe3P8KkuRNpvmMz09r35N6TLmb99mZkP/xRxbWJVplUJ/X3EPG/mIHaWvs+EP6IkhQLV05mgHA/E7jJs0YtT9uzhrbnXkruj8tZ0qItV+cO47PsoyueW/laLcqJSDIFamdiuJmrhf2Ctds8a7jXa7x5A/UvuxQWzybrgIMYcuZgpnXoiTX7p/PLZ81alBORZApU97xIM1QLFYtymRnpNEivw+ApS2Iu6lV+vfole7jmo5eZN+FKen0+D/LyuODm55h6zClhg3Tl19CinIgkU6ACdaQZavnC4YN/6szukr1sKSx2dZJ2y8wMsJbeX3/I7CevZuiC53i/dWcuGvw0jBnDX/t3iXqKePlr+KGCQ0RqrkClPmLVHse7qHfHEXvJuOU2frNqKSubteKCP93JZ227MubsY4B9N52Ey4dXfm8tyolIsgQqUMfared6UW/TJhgxgl7//Cd7Gh/Iff2v54mcU2jetBFjQsrTKgdgP5Sy+WEMIpJaMeuoqyKVddSVxaypLi6Gxx+HkSNh+3a45hoYNQqaNk35WKsiyPXaIhJdtDrqQOWoY4m6qDdrFnTq5Gz97tYNli6FRx6pliAd7y7GcmqiJFI7BSr1EUu41Mioo9M5dcRVMGMGHHkkvPYa9OtXcdp3qiXSWlT12iK1U6ADdaR8bW6XbNi6Fe68E859GOrXh3vucWbT9evH/XpejQsS65Wtem2R2imwgTrizLS0lNyls2H4cNiwwTmj8O674dBDq/Z6xNdEP9GTW6IFeR21JVI7BTZHHW5m2v77z2l31u/g8sudNMcnnzgHy8YI0pFeryr531ivE2n22zIzI2wnwMp14KrXFqmdfDejdpt+qDwzbbltA3nzn6H/8gWsa3QwTJoEAwbElYf2Kv/r5uSWSLNiN2kR1WuL1D6+CtSR0gYLV29m3oqN+wTvlpkZ/LRxC1d+PI2rPp6KwfLw8efxeu8LmXN+n7jf26v8byIntwyesiTsa2qxUKR281WgjjSjnPSfHyp2BOYXFDFs6ueM3LWME58cR8ttG5nR7gTGnHwpm5u1YEy/Y6r03l7lfxM5uUWLhSISjq8CdbSmS+Xar/+GkXPG033NVxTktOea84Yz86C2tMzM2G9XYTy8OqMwkdfRYqGIhOOrnYmRdhYCNNu5hSELnudPn89mc8MDuf+Eixjz5kOQFr1pUtBoi7hI7ZToCS8pE25GWa+0mIsXvsF1H04mo2Q3T/36LP5+/Hk0PjSrxgVp0GKhiOzPV4F6n7TBlkL+sH4pt8yeQLN1q5lz5K+5q+dlfHfwYUoHiEit4qtADWUzygbbYPBgpz9HTg4fDn+OETuczSLZSgeISC3jr0C9ZQuMHg2PPgqNGsGDD8Jf/8rx6el8UN1jExGpJjEDtTFmItAX2GCt7ZC0kRQUQE6O0yt64EC44w7IyvL0LbRQJyJB5GZG/QzwKPBcUkeSmQm33AKnnAKdOwPeBlavenmIiKRazF4f1toFwOYUjAWGDNknSEfrexEv9XIWkaDybVMmrwOrejmLSFB5FqiNMQONMQuNMQs3btyY8Ot5HVijda0TEfEzzwK1tXa8tbabtbZblgeLgF4H1qjHdImI+JhvUx9eBtbyRcmi4lLSylqfZmak0yC9DoOnLInr3EIRkVSLGaiNMS8BHwE5xpg1xpjLkj8s75rkV16UBCi1lvQ6hp17SthSWOzJQqWISDL5qilTMkRr9BQqOzODD/J6JXlEIiL7i9aUybepD6/Es/ioChAR8aMaH6jjWXxUBYiI+FGND9ThFiXT6xjS0/Y9T1EVICLiV/5qypQEkU5cCfd32kouIn5U4xcTRUSCIBAnvKiznYhIeL4I1OpsJyISmS8WE9XZTkQkMl8EanW2ExGJzBeBWp3tREQi80WgVmc7EZHIfLGYGKnWWQuJIiI+CdTgBGsFZhGR/fki9SEiIpEpUIuI+JwCtYiIzylQi4j4nAK1iIjPJaV7njFmI7C6ik9vBmzycDjVqabcS025D9C9+FFNuQ9I7F5+Ya3NCvdAUgJ1IowxCyO1+guamnIvNeU+QPfiRzXlPiB596LUh4iIzylQi4j4nB8D9fjqHoCHasq91JT7AN2LH9WU+4Ak3YvvctQiIrIvP86oRUSkEgVqERGfq5ZAbYw53Riz0hjzjTEmL8zjxhjzSNnjnxtjjq2Ocbrh4l5ONsZsNcYsKfs1ojrGGYsxZqIxZoMxZlmEx4P0mcS6l6B8JocbY+YZY5YbY740xtwQ5ppAfC4u7yUon0sDY8wnxpilZfcyOsw13n4u1tqU/gLSgG+BI4B6wFLgVyHXnAnMBAxwHPBxqsfp4b2cDMyo7rG6uJcTgWOBZREeD8Rn4vJegvKZtACOLft9Y+DrAP9bcXMvQflcDNCo7PfpwMfAccn8XKpjRt0d+MZa+521dg8wGTgr5JqzgOes4z9ApjGmRaoH6oKbewkEa+0CYHOUS4Lymbi5l0Cw1q6z1n5W9vvtwHIgtGl7ID4Xl/cSCGX/r3eU/TG97FdoVYann0t1BOps4MdKf17D/h+Ym2v8wO04f1P2Y9JMY0z71AzNc0H5TNwK1GdijGkNdMGZvVUWuM8lyr1AQD4XY0yaMWYJsAGYba1N6udSHSe8mDB/F/pt5OYaP3Azzs9w9vDvMMacCUwH2iZ7YEkQlM/EjUB9JsaYRsBUYJC1dlvow2Ge4tvPJca9BOZzsdaWAp2NMZnAq8aYDtbaymsinn4u1TGjXgMcXunPhwFrq3CNH8Qcp7V2W/mPSdbat4B0Y0yz1A3RM0H5TGIK0mdijEnHCWyTrLXTwlwSmM8l1r0E6XMpZ60tAOYDp4c85OnnUh2B+lOgrTGmjTGmHnAe8HrINa8Dfy5bOT0O2GqtXZfqgboQ816MMYcaY0zZ77vj/D//KeUjTVxQPpOYgvKZlI3xKWC5tfaBCJcF4nNxcy8B+lyyymbSGGMygN8BK0Iu8/RzSXnqw1pbYoy5FpiFUzUx0Vr7pTHmqrLHnwDewlk1/QYoBC5N9TjdcHkvfwCuNsaUAEXAebZsWdhPjDEv4ay6NzPGrAFG4iySBOozAVf3EojPBOgBXAR8UZYPBRgOtILAfS5u7iUon0sL4FljTBrOl8nL1toZyYxh2kIuIuJz2pkoIuJzCtQiIj6nQC0i4nMK1CIiPqdALSLicwrUIiI+p0AtIuJz/w+0sBpb5YETRAAAAABJRU5ErkJggg==",
|
|
"text/plain": [
|
|
"<Figure size 432x288 with 1 Axes>"
|
|
]
|
|
},
|
|
"metadata": {
|
|
"needs_background": "light"
|
|
},
|
|
"output_type": "display_data"
|
|
}
|
|
],
|
|
"source": [
|
|
"plt.scatter(train_x,train_labels)\n",
|
|
"x = np.array([min(train_x),max(train_x)])\n",
|
|
"y = w.numpy()[0,0]*x+b.numpy()[0]\n",
|
|
"plt.plot(x,y,color='red')"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "0giuwC9GHzi8"
|
|
},
|
|
"source": [
|
|
"## Computational Graph and GPU Computations\n",
|
|
"\n",
|
|
"Whenever we compute tensor expression, Tensorflow builds a computational graph that can be computed on the available computing device, e.g. CPU or GPU. Since we were using arbitrary Python function in our code, they cannot be included as part of computational graph, and thus when running our code on GPU we would need to pass the data between CPU and GPU back and forth, and compute custom function on CPU.\n",
|
|
"\n",
|
|
"Tensorflow allows us to mark our Python function using `@tf.function` decorator, which will make this function a part of the same computational graph. This decorator can be applied to functions that use standard Tensorflow tensor operations. "
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 16,
|
|
"metadata": {
|
|
"id": "HK7HPLz3Hyrl"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"@tf.function\n",
|
|
"def train_on_batch(x, y):\n",
|
|
" with tf.GradientTape() as tape:\n",
|
|
" predictions = f(x)\n",
|
|
" loss = compute_loss(y, predictions)\n",
|
|
" # Note that `tape.gradient` works with a list as well (w, b).\n",
|
|
" dloss_dw, dloss_db = tape.gradient(loss, [w, b])\n",
|
|
" w.assign_sub(learning_rate * dloss_dw)\n",
|
|
" b.assign_sub(learning_rate * dloss_db)\n",
|
|
" return loss"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "J7HusxWkGjLX"
|
|
},
|
|
"source": [
|
|
"The code has not changed, but if you were running this code on GPU and on larger dataset - you would have noticed the difference in speed. \n",
|
|
"\n",
|
|
"## Dataset API\n",
|
|
"\n",
|
|
"Tensorflow contains a convenient API to work with data. Let's try to use it. We will also train our model from scratch."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 17,
|
|
"metadata": {
|
|
"colab": {
|
|
"base_uri": "https://localhost:8080/"
|
|
},
|
|
"id": "oYro9Lbr8q0M",
|
|
"outputId": "78c0a6de-71bd-4eef-8819-439495b28672"
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 0: last batch loss = 173.4585\n",
|
|
"Epoch 1: last batch loss = 13.8459\n",
|
|
"Epoch 2: last batch loss = 4.5407\n",
|
|
"Epoch 3: last batch loss = 3.7364\n",
|
|
"Epoch 4: last batch loss = 3.4334\n",
|
|
"Epoch 5: last batch loss = 3.1790\n",
|
|
"Epoch 6: last batch loss = 2.9458\n",
|
|
"Epoch 7: last batch loss = 2.7311\n",
|
|
"Epoch 8: last batch loss = 2.5332\n",
|
|
"Epoch 9: last batch loss = 2.3507\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"w.assign([[10.0]])\n",
|
|
"b.assign([0.0])\n",
|
|
"\n",
|
|
"# Create a tf.data.Dataset object for easy batched iteration\n",
|
|
"dataset = tf.data.Dataset.from_tensor_slices((train_x.astype(np.float32), train_labels.astype(np.float32)))\n",
|
|
"dataset = dataset.shuffle(buffer_size=1024).batch(256)\n",
|
|
"\n",
|
|
"for epoch in range(10):\n",
|
|
" for step, (x, y) in enumerate(dataset):\n",
|
|
" loss = train_on_batch(tf.reshape(x,(-1,1)), tf.reshape(y,(-1,1)))\n",
|
|
" print('Epoch %d: last batch loss = %.4f' % (epoch, float(loss)))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "A10prCPowHl7"
|
|
},
|
|
"source": [
|
|
"## Example 2: Classification\n",
|
|
"\n",
|
|
"Now we will consider binary classification problem. A good example of such a problem would be a tumour classification between malignant and benign based on it's size and age.\n",
|
|
"\n",
|
|
"The core model is similar to regression, but we need to use different loss function. Let's start by generating sample data:\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 18,
|
|
"metadata": {
|
|
"id": "j0OTPkGpwHl7",
|
|
"scrolled": false,
|
|
"trusted": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"np.random.seed(0) # pick the seed for reproducibility - change it to explore the effects of random variations\n",
|
|
"\n",
|
|
"n = 100\n",
|
|
"X, Y = make_classification(n_samples = n, n_features=2,\n",
|
|
" n_redundant=0, n_informative=2, flip_y=0.2,class_sep=1)\n",
|
|
"X = X.astype(np.float32)\n",
|
|
"Y = Y.astype(np.int32)\n",
|
|
"\n",
|
|
"split = [ 70*n//100, (15+70)*n//100 ]\n",
|
|
"train_x, valid_x, test_x = np.split(X, split)\n",
|
|
"train_labels, valid_labels, test_labels = np.split(Y, split)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 19,
|
|
"metadata": {
|
|
"id": "c-_BjSHPwHl8",
|
|
"scrolled": false,
|
|
"trusted": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def plot_dataset(features, labels, W=None, b=None):\n",
|
|
" # prepare the plot\n",
|
|
" fig, ax = plt.subplots(1, 1)\n",
|
|
" ax.set_xlabel('$x_i[0]$ -- (feature 1)')\n",
|
|
" ax.set_ylabel('$x_i[1]$ -- (feature 2)')\n",
|
|
" colors = ['r' if l else 'b' for l in labels]\n",
|
|
" ax.scatter(features[:, 0], features[:, 1], marker='o', c=colors, s=100, alpha = 0.5)\n",
|
|
" if W is not None:\n",
|
|
" min_x = min(features[:,0])\n",
|
|
" max_x = max(features[:,1])\n",
|
|
" min_y = min(features[:,1])*(1-.1)\n",
|
|
" max_y = max(features[:,1])*(1+.1)\n",
|
|
" cx = np.array([min_x,max_x],dtype=np.float32)\n",
|
|
" cy = (0.5-W[0]*cx-b)/W[1]\n",
|
|
" ax.plot(cx,cy,'g')\n",
|
|
" ax.set_ylim(min_y,max_y)\n",
|
|
" fig.show()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 20,
|
|
"metadata": {
|
|
"colab": {
|
|
"base_uri": "https://localhost:8080/",
|
|
"height": 283
|
|
},
|
|
"id": "tq0vFchQwHl8",
|
|
"outputId": "9a5aa6a0-c92f-4d72-9e78-c0f615804bff",
|
|
"scrolled": false,
|
|
"trusted": true
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stderr",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"C:\\Users\\dmitryso\\AppData\\Local\\Temp/ipykernel_66332/2721537645.py:17: UserWarning: Matplotlib is currently using module://matplotlib_inline.backend_inline, which is a non-GUI backend, so cannot show the figure.\n",
|
|
" fig.show()\n"
|
|
]
|
|
},
|
|
{
|
|
"data": {
|
|
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAYQAAAEKCAYAAAASByJ7AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAABOTElEQVR4nO2dd5iU5dX/v2dmtrINdpfOsoBKV9AFsQRFLGCMNbYYNWo0xhh9NTHGFJPoz8S8JrEkMWpeY9QYK4IdsQWEiFJEkKYIAkvZXdjed2bu3x/feZwtU3en7/lc11y7O/PsM2fmmTnnvk8VYwwURVEUxRZvARRFUZTEQA2CoiiKAkANgqIoiuJBDYKiKIoCQA2CoiiK4kENgqIoigIAcMRbgL5QVFRkSktL4y2GoihKUrFmzZoDxpji7vcntUEoLS3F6tWr4y2GoihKUiEiO33dry4jRVEUBYAaBEVRUhFjgPZ2oKMj3pIkFUntMlIUpQ84ncDGjcCbbwI7dgB2OzB9OjBnDjBmDCASbwnDp60NWLUKeP11YO9e3nfIIcDppwNHHMHXqPhFDYISVYwBGhv5MycHsOmeNDFobQUefBD45BMgNxcoLATcbuDDD4H33wfOPBM455zkMgpNTcB99wFbtwJFRcDo0by/shK4917g2GOBq68GHKr2/KHvjBIVOjqADz7gQq2ignolPx+YPx/42teArKx4S9jPeeIJYMMGoLS0q9IfPpw7h4ULqVRnz46biGHz+OPAF1/03N0MGgQMHAisXAkMGwacfXbcREx0Ema9JiKjROQ9EdksIhtF5MZ4y6T0jvZ24M9/Bv7+d/5eUsKbwwH861/A3XcDDQ3xlrIfU1lJaz1qlO8dgMMBDB0KLFoEuFwxF69XVFXRVTRypO/XJEJjt3gxd0eKTxLGIABwAviRMWYigFkAfiAik+Isk9ILFi4E1q8Hxo4F8vK8388BA3hfeTnw2GPxlbFf8/HH/BnIfzdgAFBTw9hCMrBhA/2SgV5TRgZXKJ9/Hju5koyEcRkZY/YB2Of5vUFENgMYAWBTXAVTwqKpCXj7bWDECP/u55EjqZMqKoAhQ2IrnwKgthZISwt+nM3GCxoramq4yt+7F8jMBKZOBSZMCC0Q3NQUesBYdwh+SRiD0BkRKQUwHcCHcRZFCZNt2+iCDqRvRLiY27gRGDyYXgm7Pbnil0lNXl5o6ZjGxCbY43LRPfXaa3zOzEx+iBYv5gfkhz+keysQBQWhubes7AbFJwlnEEQkB8ACAP9jjKn38fg1AK4BgJKSkhhLpwSjrS00xe5yAcuWAS+/zAWrwwEcfTRw8smMCSpRZNo04NlnqRz9XayWFmYfjR0bfXkWLABeeYVZQd0zgA4eZNDp9tsDbyenTuWOxlpd+MJ6TYceGjnZU4xEiiFARNJAY/CUMeZFX8cYYx4xxpQZY8qKi3u04lDiTF4e9UwgGhqANWuAjz4C0tOZ6DJ0KL0Fv/kN8O67MRG1/zJsGHDkkQzm+LpYbjfdNmecEf0Uzaoq4I03+CHw9VyFhVxlvPaa9z5j6PbpvCMoKADmzgV27vT9mpxOvqZzztG00wAkzDsjIgLgUQCbjTF/irc8Su845BCmlzY3A9nZPR93uYDly/n7tGk0CAC/o8OHM+b3z3/y9wkTYiV1P+Sqq5iz/9lnTMu0LPmBAywcOfVUKtho88EHXNkH8v8PHQqsWAGccgqDT0uWMGYgAsycyfvHjQMuuICrjQ8+4IevoICv6eBBusjOPhs48cTov6YkRkyw5VyMEJHjAbwPYAMAt+funxljXvf3P2VlZUab2yUe778PPPIIU027xxLKy4H33gPKyoCJE33/f1UVPRU/+lH0Ze3XtLcDq1dzhV5eTgU7dSqNwaRJsQnq3H8/A0+FhYGP27yZsQW7nXGFzEyuLioruYO49FIaMLebRu6tt1iTIAIcfjirr0eP1kCVBxFZY4wp635/wuwQjDHLAejVSgGOPx6orwdeeIGLv4ED+T2srfWmo44f7///CwsZcG5ooMtXiRLp6azePfZYKlKR2CtMh4PPHQinE/j0U+4GOscN7Xa6v9rbWZQ2ciQ/WBMm6PaylySMQVBSBxHg61+nm3r5choBY2goLAUfKF3cZuM5rBigEgPi1VNk+nQGjwLFA3fupNHwl2mUns7ModdfD7zSUIKiBkGJGsOGAeefz5vFX/4CbNpEl7U/3G4aEF8xCCXFmD6dF7qx0Xc6qDH8wBx6aGCjVVTElYe/8yghkVBZRkrqc8IJDDgHorISOOoo/V73C7KygOuuY+D3wIGuGUJtbcCXX9LnOG5c4PNY28q2tqiKm+qoQVBiysSJ3Pnv2+f78ZYWZhSefnps5VLiyJQpwC9+wQ/Gzp3A7t3Arl0MOp11Fj8M7e2Bz+Fy0SDotrJPqMtIiSkOB3DTTcAf/8jFX0EB2+Z0dHCBaLMB11+vxWkJx8GDbC3hcLAvSSitL8Jh3Djg1lvZz6S6ms9TUsL+Q598QndQoDhDZSUwa5a20e0jahCUmDNoEPDLXwLr1jGlvKKCWYRnnMHA8+DB8ZZQ+YqdO4EXX6RCttm8wZ1584DTTou8YRgypGdF8uTJ3m3lsGE9/6epiSuK006LrCz9kISpQ+gNWoegKFFk61bgnnuo9IuLvUHdlhYq52nT2Gco0kbBF9XV3FaWl3fdVh48yN3ED37AiWhKSCR8HYKiKAlEWxuHWuTl9UwJy8qiT2/dOuCdd7hbiDaDBrGf0SefsOisspLbynPPZR3FoEGhn6upiQbN7eZ2tKAgamInG2oQlLgQrzooJUTWraPiLCry/bgIW0q88QY7EsaiP1BGBovTZs7s3f83NAAvvQQsXdq1D9LMmexxpL3Y1SAosaO+ni1p3nqLHoDMTMYM5sxhnFJJID7+OHiANjubmQD797NKOJGpr2fX1H372CjLcnO5XOy0+OmnwM9+xsf6MZp2qsSEPXsYSH7uOS4mR49mi4qlS3n/ypXxllDpQnt7aANnRJJjzOYzzzB7YfTorjEPu52rEWOAhx4K3qo3xVGDoESd1lbGA10ufh+zs6lH0tL4XRw8GHj4YaahKglCSUnwaWkuFxXowIGxkam31NZyxRFo9V9UxPqHZBkZGiXUIChRZ+1auoj8uaMzM9mOZsmS2MoVFi4X5/b+3/8B994LPPkku2mm6orymGP42gKt/isqmPsfqA9JImCtNALteKxg1mefRV2cREZjCErUWbaMMxICMXgw8OGHwOWXM3aYUFRUsE2zNes3PZ39dd55hzny3/9+6vXZGDKEcwYWL+a2rrsyramhEv3GN+IjXziE6tKy2UIbLZrCqEFQok5tbXAlb7dzQdrSkmAGob4e+P3v6fcqLe36mDHs0//AA8BPfpJ6k7guvJBK8s03qfyzsqhcW1sZALrtNt+FYolGYaG3Y2KgtDa3OzleTxRJsU+wkogMHMigcqA2M06nV+ckFMuWcTU8enTPx0RYQbt1K3cMhx8ee/miid0OXHQR00pXruRFzMhgh9LJk5PHAI4ezetUW+s/3tHaytc2dWpMRUs0kuSKKsnMnDnAgw/6rx1qbGTau83GhfaIERzaNWVKnHWO283ARqBeGiJ0F731VuoZBIuiIvYVSVZEgG9/G/jd75jJ0N2919ZGY/fd7ybY9jT2aFBZiTpHHEGdWlnZ87GdO+mR2LWLk9Sys5nsce+9wP/+L41F3GhtZTFTsG1Lbi7jC0riMn48cPPN9Enu3MnaiYoKBpwPHACuuAKYPTveUsYd3SEoUScjg9/Fe+7h96+wkIp/717OXx4wgLuIoUO9xw8axFG7Dz3E2cpxqWi2tidud+DhLC4XA81KYnP44cCf/sT2F5s28bqOHcsB36mWFNBL1CAoMWHoUOCOOzgt8a23uDjbvJmDsKZMoVHojOWe//RTGpG4tMNOT6dw27cHbr188CD79vcCK2tVW3jEiL62v0hx1CAoMWPAAODEE3mrqmK8oKTEvzK0itc++KCXBqGxEdi4kZlCAwYAkyaF38hs3jxmGQ0c6Dug0dLC3cNxx4V8SqeTJQ1vvMFdkAgHB512GmO18RpvrChqEJS40NjIJJZgK+PMTLp4w8LpZA//JUv4u4XNxhmeF10UevBw0iSu/hctoq8rP59CG0PBGhtZh+Cv6q4bbW3A3/7GYr38fG8LoB076FKbPRv4zneSJ4FHSS30Y6fEhcxMb+eDQEahvT3MQli3G3jsMQYnSkq6alaXC3jvPbp4brghNK0rwhbLJSXAyy8z+m2z8XksY3HYYSGL9+yzzKgaM6br6y4upr1Ztow/zzkn9JesKJFCDYISF4YOZWuZ+vrAVczt7cDRR4dx4q1bgeXLWUTW3fditzMnfd06dvOcMSO0c4rw2LIypkq1tTEIGU4PfjAN/j//YWzElxG02bhjWLwYmD+fRlNRYol6K5W4IAKceSZjCZ29Op2xuiqPHx/Gid9+mylM/hzxIowHvP562DJDhC0dSkrCNgaAN7ElUEud9HQawS1bwhdPUfqKGgQlbsycSW/Mrl1ceLvdvL+piZlFubnAjTeGGWTdujV4982CAjrtY9y2uaEh9GyilpboyqIovkgol5GI/APAGQAqjTFT4i2PEl1E6CufOJHFaZ98wvsLCoBvfYsZN7W1QF0d3UshtbUIVePGIc8zLy/05qiB2nwoSrRIKIMA4J8A/gLgiTjLocQIK+Vy4kQu2J1OGoGXXuJMEyuhx+EATjqJHRQC1hBNnswYgVXl5ovqahZAhDIAJoJMnsyndDr9x7Pb2pgAFZabTFEiREK5jIwxywBUx1sOJT7Y7UwAuvNOFrCNGMEAbEkJszoXL2ZJQENDgJPMnUt/iz93kDHccsyfH5XXEIi8PBq13bt97xTcbqC8HPj61zWgrMSHhDIISv/GGOboG0Nj0HkBn57OxKE9e4Dnnw9wkkMOYR//L7/kcrszHR2MHcyaFbdGdBdcwNjJjh3egLrTycrtnTtpz04/PS6iKUrCuYyCIiLXALgGAEpKSuIsjRJJvviCK+RAl3X4cGDFCuCb3/RTnyDCAMTAgcArr3Q1CmlpTG0666yYu4s6i3DttSxAe/NN4PPPKfKkSbRjEyZoGwslfiSdQTDGPALgEQAoKytL0fmF/RMr1TKQQnQ4uIP48ssAi3ybjX6Xk0/mSRsbGaUdPz4horV2O9vu9/PW+0oCknQGQUldOjpCX7j7q13oQkYGe28rihISCRVDEJGnAXwAYLyIlIvIVfGWSYkdw4cHV/TGMPhaWBgbmRSlP5FQOwRjzMXxlkGJDPX13kajublMuQzUogKgCygjw5t6CQB2VzuG7FuHQ754E3n15WjpcODAmBkocZ8IoDTKr0JR+hcJZRCU5Kejg1lA77zDzE+rjsBmA04tq8a5w/6L9G2befDEicCxx37VBiIri3PdH3uM6aY5aMTRH9yHwurP0ZpZgJq0wWhtd2FOxgrIr9/jwfPnaxRWUSKEGgQlYrhcwN//Dnz4IRX6V8VXxqB021sYfu8z+GwgMOnoXLaj2LyZbaovvJBDlEUwZw7dRs89a3D4+oeR2bgD5dmlcLYJ0tKAmcfbUTBkBC3P009zNmdZWTxftqKkDGoQlIixaROwcmXP1s4lu97H9A1PonbIKHxWn47CdmDYMDA1tL0d+Ne/uD2YPRsitA3HjtiFhps3YN/Q0ci2CYqLWXz8lZFJS2O12qJFwFFH6S5BUSJAQgWVleRmyRLGCzrrZpurA5M3Po+G3OEwjnRkZnJK2FekpzOa/NxzXPV7yNm8CsNG2nHkUYLp09n1tEe7h7w8DmbWAfeKEhHUICgR47PPejYaLTqwBentDXCmsTNdVhZbCXVp3ZCVxVqBzZu991VXB+/fIMLgRFNT34VvbGTfjNbWvp8rllRXs+x5715vu1hF6SXqMlIihs3Ws0dPVks10tubMLhiA9LbG+CypUFsIwDXEMDRreigttb7e35+z9YT3TGGgYu+FJtt3gy89hr9XSK8HXssBxyPGNH780abL74AFi5kKpf1xhcVAd/4Buc762BmpReoQVAixqRJ1KtDhvBvh7MVh219GUP3f4y2jDy4bWlwdrgxybEH8nYmewp1Hnqflub9feZMTqEPNGOzoYFP1lvFvXgx8O9/089ljTFzOhkI+e9/gZtv5otKNNauBf78ZxrCUaO8yr+hAXjkEWD7duDSS9UoKGGjnxglLA4cAF59lU3oHn2Uuqm9nY+dcgq9N243AGMwfc3fkduwB+3pOWhPz0WHIwstMgDZwwuo6Fes6PQPYEtqi9JSNvbxFx9wOtkd7uyzexdQ3rqVWUqjRnGgsXUOh4Mxjfx84IEH2Bk1kairAx56iDIXF3dV+rm5jOi/8w6wZk38ZFSSFt0hKCHhcgEvvMBFtQgXpy4XZ9nn5wM//CFbBZ10EvDuu8DU7O0Yvm8NqgvHI7OtHgPq96Ee+cjPBwYMACBZrFrbto1lxzNm0OVhIQJ8//vAn/5EH3lhIRWe201D0NLCcWthDVzuxJtv8kV03pUANFQHD/I5KiqYR3vttUGGMMSQDz+kMfTnJrPZvCNCy8o0+0oJCzUIKUBVFZu9ud1Myy8tjbweeP556pjRo3v2G6qrA/73f4Ff/pKeisJCoPpPy1DXko5aCGrsR+BQZyOK0moxIDsH1sfOac+AWbcRLfPPRfa3Luv5YczPB267DVi9mu6jXbv45GVl7BN9yCG9e6FtbcDHH3N30P2FrFrFALPNxqynJ59kg7xzz419EVx9PeX56CNuw0pKgA0b/LR57URBAXtpNzQEP1ZROqEGIYk5cIAp/NboSREahZEjgcsuAw47LHLPs3ixb2MAUG+3tLAk4Ic/ZFyzY3U59m/KwfovgfqWDHwx7Hg0Nm3H4PIvcHB3ExwOoN3lQHN6CR6v+h7af52L+fOp57ss2jMzgeOP582KWPdVKbe3ezOULBobgeXLeb8V1+jo4HMOGUL3ktvNkW2xYN06+uXa2/kG2+3ABx8wvjF4MOMvDgeVfn09/ycnhwbACo6H1AFQUbyoQUhSDh4Efvtb6rHOcUVjmKzzu98Bt9wSmZjoypXUL4E6kQ4ezEV3TQ09Fh2SgR3bXHDbWFAmko7Wogn4rOlQ7N/RAmcbUDQyE6PT9qBoTC6k+jN8+f8+xPsP12D2WQPhOG5Wzx1ApFbnWVmsf2hv50+AuwCXq+uKuqODyjgtjavzF1+kYeocCI8G27YB999PF9qAAd77c3Nplbdu9b4XVVX8afUIGTiQFz0tjccr4eF0che2eLF3R3rkkcCcOdHZeicYGlROUp57jovD4cO7LnRFqBMGDgQefjgyi8Tdu4NndtpsvFV7BqD+p3kmbI31XQrV3G6gfJ8dzswc2PJy0FTZiIb0IpR99Decsvq3KGt+H87Nn2PvM8uAu+6ixYtGUNfh4FZk/37+3drKUWzd4wTt7cDYsfw9LY0K96OPIi9Pd158kW94Z2NgUVpK+deuBSorabAKCvgzP59B+rff5rCF7vERJTDNzcAf/wjcdx+TGQoLuUBYuRL4zW84cMnX7NMUQg1CElJTQ9dyoDnyeXnUpRs39v350tNDq3lyu71ejFf2z0B6XibS2xu/erypyTtg3i5u5HdUYe+BNOTX70JdQSka84ZDCguxsWYE3KNGMzDyxz9Gp1hszhy6o2pqqAi6u5Dq66lgBw/23pedzQB3NDlwgLURnQPsnRk0iDsZa/amrx1URgZHsaW48oo4//gHd4qlpTQGDoe3kn7kSAbSVq6Mt5RRJWyDICIDRCQ+8wcVAFzMAsGHyTgc3dpE9JLp0xkjCERLCxe0w4czOafNMQCrjr4emS01yGncDzHur2K12c56DGvdgQ0ZM7ClbQwa87x1BGlpXJg3NArrC3bu5Go40hQVAT/5CX/fs4dGp72dVqu2lhb1mGO6vsmWxYsmdXV8k/y5Jhobacjy83lsfT0NWmMj5U5L46S42trIXPz+wt69TNUtKfH93jscXBwsXJjSFeFBDYKI2ETkWyLymohUAtgCYJ+IbBSRe0Tk0GDnUCJLoFqtzlhB5r4ydapX//iTZ98+FvdaXgoRoGrwZCw94XZUDJ6CvPpyDGwqx7COnXDa0vHysO/hIxyN9nTf6ZxfyT1wIP250aC0FLj7buD6673B2KIiGoLZs3u2zmhpif7cy/T0wCv7+nrKWVzMWMHYsVzNDh9OuefO9cZBdu6MrqypxNq13mC8P3JzGbPZtSt2csWYUJY77wF4G8BtAD41xrgBQEQGAZgD4G4RWWiM+Vf0xFQ6M3QoFabbHbgYtaODOq+vpKUBN9wA/P731ImDB3uft6WFxuDww9ml1JLPSnKpKxiNVTOvR0ZrHco31WPr9jS4CgfDQJC3awWGDe3ah8jShV/pYquBXbTIyqL76JZb6CP294Y1NnILFO2RnMOHMybQ1OQ7hmCtBlwuBt07u7R8HauERnW1N8EgEJHqnZWghOIyOtkYc6cxZr1lDADAGFNtjFlgjDkPwLPRE1HpTnExFXBFhf9jmpvp8vY7iD5Mxo0Dbr+dk8/Ky3nbtYvPc8EFNBjW9yknhwvsznq8LTMf2eNH4YBjKFzGhg6nwGF3YXx2eZfnaWqiQcnK8tzhdPoMjrpcEd65n346V9s7d3bpugpj6Nc/eBC47jrvKLdoYbczb3f/ft8vMCeHVjg723+cwSKRezElGvn5Xa+7P4zp9OFMPYLuEIwxHSIyAcAIAB8aY76KEorIPGPMYmNMCO+kEkkuvBC4807uYIuKuu50m5poLH74w8jqr5Ejec6aGi6orC4PvpJZzjyT2Xt79nD2gc3GBe+4cYyZOhzAWRNqMah5LxrThwNgvZjbzY4VX1FVxWZtnsdXrWKN2p49POfUqdyZTJrUx4zArCzgxz+mj/g//6HFASjQIYcAF13En7Fg9mw2r1u6lBfXcme1t9NllJfHIhN/28P6erqRxo+PjbypwJFH8toH8sc2NXH3Nnp0TEWLJWKCbCtF5AYAPwCwGcA0ADcaY17yPLbWGHNktIX0R1lZmVm9enW8nj7ulJezs4LlKrZS0fPzWTEc70FiNTXAE0+wxsrC6jxhtwOTsnbgnA2/QU3WCLS505CRwZ52hYWegzs6qPl/8xs0DBqNe++lnhw0iDrRWrw3NjJ+cfHFEern1tTE7Y/TSWGGDYt9/rnbzYyW11/3Wj+Hg+6tww5j0Vp2ds9+4/X1fON//GNu55TQMIZtUjZv5sqnOy4Xv2hXXgmceGLMxYs0IrLGGNNDQ4RiEDYAOMYY0ygipQBeAPCkMeZ+EfnYGDM9KhKHQH83CAA/xzt2sMGly0XdNWlS9JNhwqGy0itfYSF72B04wMJbWfwGJnz8NPLHDETRuAI40sRbXVddDVxyCcxp83DvvUyh7d5tAuB5v/wSuOIK9lJKKaz3wuViUNPa8m3fztXA/v1dV7WDBlFpqTEIn4YG+Fx1VFXRNzpvHneKKVCc1heDsMkYM6nT3zmgUdgE4CRjzLQIyxoyahBSAGO4hVi0iKtym42r49JS4KyzgGnTUF4O/OIX3Kn7+y42N9Oj8oc/JJYxjCpuN1NLd+3i7yNG0N8WLB9Z8U9bm7d31p49/MBNm8ZWvhMmpIQxAPwbhFC+OvtFZJoxZh0AeHYKZwD4B4Ao5+ApKY8ICx2mTeNWwoqGDx781Zdv3brAqfkA/+XAAe4UYuXqjzs2G91HkWpapXAHdtxxvLndwVNRU4xQDMJlALo0QDDGOAFcJiIPR0Uqpf8h4p2s0426utC6MIgEL6BTlJDphwOGQskyKg/w2IrIiqMoPRk4MLSMQLfbd+p+0lFTwwZ2bW30Y0+cGHy+tKJEgP7ibVWSmCOPZDO/QBmBjY0MWEeiEC9uNDdzpOd//9u1BiEzk5PhTjutX7kvlNijBkFJSIxhYo3dzkK1WbM4LMxXqxmnk3UX116bxLv8tjZmuGzbxrTHzoHhtjbgqado9c47T42CEjVCNggiIgAuATDWGHOHiJQAGGqMiVg/YBGZB+B+AHYA/2eMuTtS51aSg/p6pqO+8QazLR0O1ibMnk19uGEDsy/z87mIPnCA2UXnnw8ce2y8pe8DK1bQTTRmTE+Fn5HBFKtXX6Vl9JUnrygRIJwdwoMA3ABOAnAHgAYACwDMiIQgng6qfwVwCoByAKtE5GVjzKZInF9JfPbv5yjOmhq25ygt5S5hzRrqy4svZvbfm28ym8huZz+3k04KnJKa8LjdLEDrlFnVA4eDt6VLgUsuia18Sr8hHINwtDHmSBH5GACMMTUiEkI3qJCZCWCbMWY7AIjIMwDOAusdlBSno4OFom1tXTsD2O0stmtvp3v9pz/1dq1OGRobWYRXUhL4uIICVtIqSpQIx+Pa4VnFGwAQkWJwxxApRgDY3envcs99XRCRa0RktYisrrLGByoJh8vFlX5NjbctUCA+/ZRlCMXFvh9PT2fCzauvRlZORVG8hLNDeADAQgCDReQuAN8E8IsIyuJrr9yjjNoY8wiARwBWKkfw+ZUI0NoKvP8+PSDW/IT8fGD+fN8jBiyWLw+eMlpYyIFWdXU8Z8qQk8NWCY2NPcd4dqa2lvMOEoGmJlb0bthAiz9uHP13XzWiigANDWzY5XazRiVYd1elz4RkEDwB5WUA1gCYCyrvs40xkdy/lgPo3KlmJIAoNsJXIk1TE90+n3/O76/lAWlqAv71L2YJ3Xyzb8VfXx+8M6tVNNrSkmIGwWYDvv514LHH+Ob4iiM4nVS8X/ta7OXrzpo1wCOP0L+Xk0P5P/kEWLCA6bFnntm3gE5dHc+1olOZk9vNWRQXXMAWu0pUCMllZNjwaJExZosx5q/GmL9E2BgAwCoAh4rIGE9s4iIAL0f4OZQo8vTTbLQ3dmxXpT9gAO/bsYPZk74YNCh4lbHbzXTUvhSftbZytEFDQ+/PERWOPZa9cnbt6ulja2tjp80zzoh/htHGjcCf/0yLXFrKVfugQbT+I0YAL7zAFLHeUlsL/Pa33DIOG8ZuhtZtyxbgjjuA3buDnkbpHeG4jFaKyAxjzKpoCGKMcYrI9QDeBNNO/2GMicCIeCUW1NSwnirQTJYRI5hSev75Pbs2z5pFN1NLC5Npiop6upeqqtjyKDc3fPn27WN20ooV1LfGcFzA178OTJmSABlKGRnATTfRqi5f7i1ME+Gshm9/2zuSLl4YQ/ny89k8qjsOBw3DwoXACSf0znI/+yxzibsH2G02FqQcPAg89BCHgSRt0UniEo5BmAPgeyKyE0AT6DYyxpgIzeQCjDGvA3g9UudTYsfWrdRhgRpt2u3UKVu30gAA/J8lS6hDdu/mY9nZ1IMlJVTWaWk0FC0tVODh8tln7IJqDF1ZDgd/37MHuOce1nr11csRFtbcUWMYRbdmIGdlsXX1uefyjWhvp/WbODH6k9pCYdcuvmmBsqHS0pgy9vHHwPHHh3f+mhr6FQOtKgoLuVvatk2b+kWBcAzC/KhJkWQYQ+VVUeFVXIFG2/YH2tvDP9YYuopffpmekNNO4w6ivp56ZccO6oixY2lMvv99xi7DoakJuP9+urotvQvwuhUWcrG7YAHPO2VKiCd1uXhih8P3StkfjY1Mk3rvPcYELGbNou/dSrEqKACOPjr088aKmprgbWcBXrz9+8M/vzXpKVj7bhFaeTUIESdkg2CM2RlNQZKFHTs4BWzHDm+Q0xjOLv72t/2nTaY6nZVtIES8x5aXA6+9Rle0pQNmz6Yu+eILbyXy3LlsS+GnGWpAVq1iiyB/18XhoDyvvx6CQairY2HYkiU8qTGc9nP66Qx4BlKUDQ2sutu9m0FRawC1y8VV8fr1wM9+Rr95omJtrYLhcvWuGZ/VbjoYNltoucxK2ITTuuJ2X/cbY+6InDiJzbZtwN13c2ffuTLW7Wa90F13AT//ef80ChMncrHc2upfF7S28r2bOJF/L13KxWTnBaHDwd2CFTttaOBivLfv6YoVwTOSCgt5/QJmfVZW8uLX1HhTII2h6+ePf6R//5JL/Pu1n3uO7pbu3ffsdr7Yykr6xn/96wQIaPhhzBjK29Hhvx+5MfxCWBc5HIqLvQGeQO+B08kt+ZYt/HBkZ3MIRig90pWAhOMyaur0eyaAM8A5y/0Clwt4+GEqjIKCro/ZbFz07d3Latobb4yLiHElI4Ou78cfp7HsPrXM6aRH4KijmF3pdjN2Giy1PDeXruvm5sAp+v5oaQmuJ0R4Df26vdxu+p1aWrqWUVt+p4EDuWsYPZpbnO7U1QWPuBcX8w2y0rQSkQEDGCx++23/bWWrqvg+9OY1jBxJo3PggP96huZmBpafeILZVxbZ2czCOvVUDTb3gXBcRn/s/LeI/AH9KC1061Z+1gO1Vx46lNO9DhzonzU0c+dylf3SS/xOWoaztpYLaaeTFclW8snGjdS106f77ukWCYYOBTZtCpzw0tHBha/fY7Zu9b26t7DZuGJ99VUGUrsrpB07uOoN5Bu3/I9bt0bGIFjDtt99l3UDTidTN+fNY6qW5bIKl/POYyOpzz+ne8vaDnZ00NeXk8NgT28upgj9rr/9La14dz9kSwsN64ABfCwry/tYaytzmisrgUsvTdxdVoLTl/bX2QASdCkTebZvD77wsOJtu3cHNggdHdzt1tfze3nYYalRaCXC2OjRR3P1v8nThWrwYL7WsWO7fofHjWOsYN066srOi2+Lhga+l+HEbjszZw4LagN5ISoqgBNPDJDI89FHwRWotZXZu7dnrUB1NT8UmzZxVZuRwTdj1Kiub4jdHl503h/G0DgtWEC5i4v54Tx4EPjrXxn3+J//6d2WKysL+NGPgLfeYh5vVZXXmJ1wAtPA+lKtPG4ccMstwIMP0vCkp/PcbW00CDk5QFlZzy1oZiZXFW+/zccnTfJ5eiUw4cQQNsDbSsIOoBjAndEQKhEJ5ta0sILM/s6xdClrd5qavOcUYQHqRRd11Q/JyrBhrDUAuAO49VZmYnV/baNH09AOGMCdQ/cxAMZQ31xxRe+9ABMmUP9t387zd7+GdXV8zlNOCXCSxsbQVtQ2W1c3BsCV9KOPUrkVFvJFHTzILVN2Nlu1WgESpzMyQeUPP2TMorvvbuBAbtu2b2el8U039W4lnZXFPN1587gid7v52iI1rm7CBMZlNm7kjsnl4u7so4+4kupuDCxsNhqMt95Sg9BLwtkhnNHpdyeACs9s5X7BqFHBExusoS5Dh/p+/NVX+T0dPrzrDsLlApYto4740Y8SI+U8UnzxBV1ovlb/eXlU1lu38r2rrPTqQ5eLWUgTJvRtzoHdDtxwA/DAA9TNWVnUWx0d3LXk5HBBGjCDacgQ5tUHwrr4navmqqup2IqLqfy/+IIC2Ww8vrkZeP55+r6HDOGFP7yPZT1uN7BoEbdlvhSnCD/M69fzDR41qucxoZKeHr3K6bQ0uramTfPe98QTwXcfhYXsrxTqCk7pQjjrruuMMTs9tz2eyuLfR02yBGPyZH7Xm5r8H1NVRQXmq9XK/v3cwftaKdvtvH/LFrpaUomDB/1/L0W4kDv8cHpKtm/nQvrLL+my/9rXuIjtq4HMzfW2zZ46lQZh2DDWgP3+90xQCcjRR3P1Hijlsrqa1q1zQcry5XxhDQ1U/jYbFV16Ol+UtaJ+912uhi+6qO+zk8vL6QMLVM4twg/d6tV9e65Y43YH3yqKdB0/qoRFODuEUwDc2u2++T7uS0nS0qhA7ruPC77u7tfqai4Qv/Ut3/+/fLlXH/hChLrk9dfp906VRIlgqesiVMiW22zCBOrLCRN6ZnP5orqaN4eDSTz+3l+7ncanV56EkSPpl1692vcknpYWbjd+8IOu97/7Lvv8LF3Ki5uXx21gR0fXC9zQwA/UCSf0QrhuWIYnGBkZfOOSiXHjaPAC7RLq6qKXodAPCGoQROT7AK4DMFZE1nd6KBfAf6MlWCIyfTpXrI89xlWszeZdjAwfzk6e/qr6P/20Z/+e7uTkMPOwvj40ZZgMWAkz1nxkX1gJOKefHroHYtcutrv45BPvdcjOplv71FN7n0TjExHgqquoyNet4xPl5nqHPthszKyZMKHr/9XX01J1dHA3kJbGn1ZxBeDdEaSn8+KPGdM3WbOzQ1sht7cH/0AmGqedxna6gwb5VvjG8HpcemnsZUsRQtkh/BvAGwB+B+Cnne5vMMYk2RKj70ybRrfwpk1MKBGh0hs3LjKr+lRb2AwaxM4MH33k311dUUFdGihNvzOffcai37Q0GhDrfW9tZYxmyxbGDSJqFLKyWGDy2Wdc+e/cSeU7Z47/OQB5efR9db6oNhtTyqy0so4OGhabjcGWvhqEkSO5hW1o8O82MoYusLKyvj1XrJkyhauyTz7hyqvzF87tZibX1KmsGld6RVCDYIypA1AH4GIRGQjgULAwDSICY8yy6IqYeDgc9HuHE/+bMIEZcYFcu01N1CG96eaZyFx8MfXnzp303VuKuqODHpSCAuC73w3NGLa1sftyXl7PNHUr83D9er7Xp58e4Rdis/FCdt8J+OOkk5jmGchn1tzM8wWrUwiBhgZg1SobmuxnY+LyvyFrQhaGDHf0jC3v2UPl2peAcjxwOLgTe+opb7DNbvdme8yaBVx2mVYs94Fw0k6/C+BGcHDNOgCzAHwA4KSoSJZizJ4NLF7MhZm/rLmKCtbl9FEvJBx5ecBttzE+8u673r5uIsz/P+MM7iRCYf16Kj5/NWIiNDpvvMFU0rjqhuOOA555hitXX1kvbW282CNGMPoeqOoxAMYw0/K556gbszKPQXluBaYuW4TtGZkYf3wxBg+1079udQv83veSczuamUn33dln031XV8cP2BFH9M+eMREmnKDyjQBmAFhpjJkjIhMA/CY6YqUew4cD3/gGMwJHjuyaOeN2M1Y2dmxiDMSKBrm5wIUXMn29stLb+Tnc1PWPPw5epJaVxYyv/fvjvAguLAR+9Sv2OKqo4N8OBy1iczONwTHHsJR7xozQrWI33nmHGZklJdbuS1Ax+Gx0HDYFwze/A/d/1iBjigv5U0Yxk2n69OTPbS4sTJxxoilEOAah1RjTKiIQkQxjzBYRGR81yVKQc86hMlu0iDE9y5MgQn1w2WWJX5jW0MB4wKef0pAdeijrBELVZVZjwN7SPUHHHzZb1w7TcWP8eDZ4uuUWRsIdDq5yrfzkmhoWpfhLTwtCczN3BqNGdYuZiKC68FBUH38oDh4ElhYZ/PJ2ScpNgRI7wjEI5SJSAGARgLdEpAY68zgsbDYOm58zh2nndXVcqI0fnxy9j1asAP75TyranBwasg0bgBdfZIub00+Pvhdi1Ci25gmEy0Vj1csFd+SZPJmpaQsWsB+3NSmoro6+xHPO6XXvkk8+oZEMtOAfNAjY8aVgzx4+9apV3KXl5LDZYKQSIpTkJ5zmdud4fv21iLwHIB/A4qhIleJkZvKLmEysXctur8OHd62dGjSICumZZ6iUTj45unIccwzTTQOlsVZUcMeVUP2hhgwBrruOOwJrstLw4X3OINizx39MysKq1Xr4YYYzbDZew/Z2xrVKS4Hrr0+ORYkSXUJeFwj5tojcboxZCgaWp0VLMCVxcLs5Sre42HchrZX++fzzTP2MJE1NbDnx2WdcUBcXs87gyy99txKpraUCPPPMyMoRMQYOpLto/PiIpJNlZITWUmXrVu4mRo/2TvgbOZLGoKKCFdv19X0WR0lywnEZPQjADWYV3QGgAcACMNCspDCB+hFZZGTQGHz6aWTS2+vrOVpz6VIqPKtp4KxZ3vnHb77pnUHvcvH5Bw5kM71QaxqSnfHjg9ehVVYyienEE3279IYMoYFdtowZX0r/JRyDcLQx5kgR+RgAjDE1IhLJ0h8lQampCe04u53ZPX2lro7DySoqmEJqpY66XPR/b9jAyXQnnwysXEm3SUYGiwYnT45SqqnTyWrE9euZLjpqFP1Sca72PeQQGr9AMzjWreNuIFCbpCFDaGDnz0+9tGcldMIxCB0iYoenBbaIFIM7BiXFcThCCxa73ZGpDn7mGRqW7m1ArJT9ykr6w2+/PUYr2h07gL/8hb1/0tMpyPLlFPSMM5gTH6eorNU147e/ZeX8kCFehd7ezsI/my148a6VqtvcnHqFkUrohGMQHgCwEMBgEbkLwDcB/CIqUikJxbhxNAjB+hEZQxdGX6ip4ao/UE+j4mLq6J07e13LFTp793K7kpnZ02fmdDLCDXB+aJwYOZLGceFC7qCs7tp2O9v/FBcHdytZ1093B/2bUJrbPWmMuRRAEYCfAJgLQACcbYzpNzOV+zP5+aw1WLHCf/O+/fs5V72vvvsdO/gz2LRJgIHmqBuERYv405dryOGgkXj1VTro45jnOnQodwoXX8yVvjXnOyuLt5deCjwgzSpgDqkOprmZaWfvvssofl4eX/+MGZEbkqPEhVD2uUeJyGgAVwKoAPA02PCuQkQSJdNbiTIXXUS3+Y4dXTOJ2tu5Us/NBa6+uu91CMEyZixiUnhWW8uW14Gm51g5nytXRlmY0CgoYLHguHFe5X7ccbwu/jLA3G4ahPnzQ7h+e/YAv/gF8H//53Wh1dWx+O5nP+OHQUlaQjEID4H1BhMArAGw2nOzflf6AQMGMHvn3HNZrbx7N9ttVFUxkHvMMdSdW7b0bT6JFRgN1A8O4HP4m0wXMazpPsHiA1lZCa0Ii4qA73yH3q+amq7vbVMTjfycOcCRRwY5UWMj8Ic/cBVQWkrrk5nJLeTo0TzxPffQkCpJSSjdTh8A8ICI/M0Y8/1oCCEi5wP4NYCJAGYaY9TQJCDZ2cBZZ3ElWVXF+oAXX2Sq6caNPMbqUXTNNVyphktpKTOL6ur8z4RoaaEsU6b09pWESKgOdbc74Ttsfu1rfD9ffNE7y8MY6vLvfCfEoUwffURl7y//eNAgtudYsQL4+tcjKr8SG0KJIYghfo2BdUwf5PgUwLkAHu7DOZQYkZ7OxeLjj1PJdPfj19YyDnvbbSGMp+yGCDu+WvMOuruk29qYOXP11RGed+ALqyy7tTVwzmZLS9fZvwnK1Kk0ovv2cZeXkUE3YMiB5CVLgsdJiovZejUUg7B3L11tu3fz/S0ro5BRv7CKP0LJMnpPRBYAeMkYs8u601ODcDyAywG8B+CfvRXCCk6Ldt5KCtxu4NFHaQx8tYcoKOAx//wncOed4ccVJk3iZLqHH2Z+fUaG1wfucHCU6cyZzKh5/30qt4EDOYFy8uTgrRxCJj2daToLFtDq+Xoh9fUMqoYzHCOOWB0zekVNTdeZ0b7IzGQBSaD5xx0dwJNP8uLZbLT6TieNQ34+pxtZo/aUmBLKV2ceGFB+WkTGAKgFB+TYASwBcK8xZl20BOyOiFwD4BoAKPGX8qJElW3b6DIKVLk8cCC9Bzt29O67fcQRwL33sqhq82a6N8aO5SKyro5xzaoqZs5kZFAHrV3LFMybbgo8djcsTjuNBWmbN1OTWl3k3G5aq/Z24Cc/6R+r2gED+HoDpSJ1dNAo+DMGxrBX99KlNLLdj6utZR+NX/+avkMlpoQSQ2gF21Y8KCJpYPppizGmNpwnEpG3AfgKA/7cGPNSqOcxxjwC4BEAKCsr64ubSukl+/YFD/pai+l9+3q/2MvIAI4+mjeL+nq6kzo6urqqrGKqvXupS668kvd5ZwT0kowMWpjXX6crpLWVSszlov/lm9/sWz/vZOKEE5i/GmghVlkZeE7Bnj3cGfgyBgC3l83NTOW9+uq+SqyESViba2NMB4B9vXkiY0yU+2AqsSIcF1CkvYArVnCH4EsHV1WxNqG8nDuTwYO5mJ0/nwv9Xsd9MzLYovr003lyp5NbkP7WHvS444DXXmMAyVdRQ3MzVwonnuj/HCtW0KcXKII9ZAjdRxdfHLh4Qok4kfK2Jg3V1Vy1AvQAxLkVTVJiTSHzNRXSwqp8jeTEMmPYb8fXpMQ9e5gEk5lJXd3YSPdSaysHyGzfDlx3fhUcH/2XbT9dLqZBzZ5NIUOxXBkZTPDvrxQVAT/8IXD//bTKgwfTyjqd9Nm5XMAPfhDY1bN7d/DiNSvKXVurBiHGJIRBEJFzAPwZQDGA10RknTHmtEg+R0UF2zOvXev97hvD4srzz+9/i72+UFrKFfrBg/7ft6oq6ttALSjCxeWijuie1dTWxqE5OTnUTy4X8+sBGogxpQbNr7yLqjeeoq4qKOCHYOlS4O232SXv4ou1b0MoTJ0K3HEH53YuW0ZjYLdz93DyycFXAKH06wYYo4lYdoASKr16x0VkqDFmf6SEMMYsBPskRYX9+4G77qLiGDnSu1t1uWggtm5l90yd0R0aIpxzftdddBkXF3uNbHs7i9NaWoDjj2dguaQkMq4ju907krizrigv71oK0L0sYMTej3Dknn9ic94oDJ2Z7pUlN5cfgjffpG/pvPP6LmSiYgxX51u2cNs0eDAzo4INqPbF8OHApZdy7GdrK5V8qMp7xgxa70ArsMZGbvOCZTQpEae3Jvh1AMHqGhMCY5gi6XT2TLezumfu28ec+h//OD4yJiMlJcz0efJJ+u0Bvo87dlDPHnYY09aXLGEtwjXX9P37LcKK6A8/7Hot9+3rOkKyuZnPDwBi3Ji86Xm0FwxBQ0v6V0VtX2G3c7vz+uucvJOKrT4PHAAeeYQXyqq8drkYbT/3XAZYemOx7fbwexdNm8atnJWu2x1juJ2/8kqd6xkHevuOJ03BQHk5K2oDKaOhQ1ltuz9ie57+wahRLD676y5+z63umt/4BrueWq6l8nLgd79j/KavzJ3LXUh7u/c+t9urzzo6+LuVCDOwZjuymw+iPZ2+aJ/ZUQ4HFeQnn/RdwESjtpZv/q5dvBijR/PClZZylf7UU8Arr8ROnsxM1hnU1/ML17nPSVMTgz3HHsvYjhJzemsQ/h5RKaLIjh2Bg58AHxPxdtpUQkeEK+7164Hp03vm/4vQ4NbVMUElEM3N1Fu7d3dV+J0pLQUuv5xGprKS+mTgQLoDGxupU8rKvAvXjNY6GBG4XFxw+h1G73BEZrpPorF4MS3x0KE9vwTp6TQQCxdyFxErDjsM+NWvGI8oL+cF37WLRvmKK7id1HhOXOity+jFiEoRRazxi6HQl6Zs/Rmr0WegtM5hwxiDPO+8nm7r6mqmnb//vvcaZGQAp5zCHUf34+fMocvotdc4PS0tjcbg0EN565w55rKnA8agsZGuK7+ubperd/70RKa1FXjvvcBZP1YK6AcfcGsXK0aNYkZSXR13MQ4HjZYagriS8jGEQJ2LO2OMxrB6y+efB3clp6V5i3s71zVVVnLaV0MD9YFlVFpbOYpg/XrGdrqff/x43hobubNYuJAp7t3d0tWDDkFzexqyBnRg7Fg/FsvKkY16t7wYc/Agg2fBKvNycngR40F+vu/+J0pcSPkYwvjxXDE2Nvo/pq6OhqM/p5j3Bbs9eOWyRefdmjHAQw9R+Y8a1XWHkZkJjBnDrtLPP+//fDk5NORXXQXMm8d6hF276J4uLwe27clC+cST8bUx5cjK9CPk3r1sgtTrJj8JSjhbY+0jpqAfxBDsduCyy7zzYrvT2MieXZddpkkNvWXyZG/evz/a2ugG6rwL27mTMcRAu7jhw70N7ALhcDAL8p57gAsvBGbNorvpZz8DLnjqbGTNOoJBos4namlhL+iioshM90k0iouZTutvMo5FY2PSNOdTokuvXEbGmAcjLUg0mT6diQ2PPkoXhbWDbm/nCvOmm6jUlN4xYwbw739Tv/rre7ZvH2cpdA7qbtniDej7w+HgTmL79uCD4gEGtU/rUdKYzgrb//6X6aW7PE17BwxgL6ITT0zNiti0NG6bnn+e2y1fWC1kZ86MrWxKQtJvSgGPOoou4vXrqVwABiCnTOkfjSqjyYABwHe/C/z1r2yX39kl7HTSIzN6dE9F3dYW+q6sz+My09LYnO1rX2PKozGsOUj1ati5c1l9uWMHi246v976egZ1rr02NesvlLBJ8W9DVzIyuJqdMSPekqQeM2fS7//003QFWat+EVYsX3hhzySeIUOCdzGw4r0Rm19vs/kfxZaKZGUxKv/CC0zzstLujOEFuOkmbqEVBaFNTAvlq+gOtx22EhpWx4HNm7miLi6m6yQRMyQPP5yp5Tt2MJXU4WDra18Fqdbx6el03fnbpdXXM2uye/8iJQyysxkkO+ccbo87OphpMWaMBs6ULoSyQ9jruQWKuNkB6LSaCHPgAKeGff55144DaWnsODBvXuLFQUVoBEKZgZCdTRf+k08yFbV7HUNLCw3Lj3+ceK8zKcnNDS0QE0MaG72xPHXdxp9QDMJmY0zAPaWIfBwheRQPVseBpib63zsrxPZ2BnHb2oCzz46XhJHhlFP4el70lDoOGMBdUWMjFcT113PXoaQOxjCW99prXOzYbMwGPOEEfh60Hih+iAmSQC4imZ6paX06JhqUlZWZ1atXx/ppY8Izz7AJp79hXE4nc+7vuSc1WndXV7Pieds2KocpU7q2oFBSA2No/F96ickHgwZxsdPRwZ52Dgdw663+k6KUyCAia4wxZd3vD3WEpr+TXmGMeSwexiCVCbXjgAgzKc88M3ayRYtBgziQTEltPv6YFeilpV27VKSlsTV9bS1w330cq5yZGR8Z+zN9jSj9JiJSKF0IteNAbm78Og4oSrgYw55VAwf6b1lUUMDOAevWxVIyxSKULKP1/h4CEGKnICUcQk38MEaTRJTkobaWGWglQdJPcnO58501KyZiJReNjZxr0d5On9uhh0a0liaUMw0BcBqAmm73C4D/RkwS5SuKipiBE6jyF2AXBu04oCQL7e1cwATLGEtLC94Kpd/R2sqK86VLu9aS5OUxVe/44yOSiheKQXgVQI4xZl33B0TkP32WQOlBWhqrerXjgJJKWMXQLlfgLtdNTcDEibGRKSlobwfuv5+9XrpXmzc3A3//O1eHEQjCBXU4GGOuMsYs9/PYt/osgeKTk0/mbnDnzp5tG+rr2Q7iiiu044CSPGRn0w1UUeH/GGOYTq0D0zqxYgWwcSN9bd3dQ9nZbBX8/POB39gQUQ90gpKZCdx8M4fB7N/Pfmy7dtFApKez48Cxx8ZbSkUJj/nz2W3bV/daqyp/4kQuhhTwzXrtNRZn+HMJpaXRF7diRZ+fLpSg8lpjTMBhOKEco4SP1XHg3HOBL77gTqGgQDsOKMnLyJGsPL//fmbTFRRw0dvURO/HxIkcpKafbw/19SzSCRaJLyjg+MBzz+3T04USQ5gYINMIYHBZRx5FkZychOs4oMSSpiYqhrQ09vdO8j4eEyawoHLVKmYTtbay1cmcORy3rFM0OxHq5CkgIjOAQzEIE0I4JkjPSkVRwmbfPuCNN6g1jeEXftgw4Iwz6IxP4mV0Tg4NwJw58ZYkwcnLY6CwuTlwR8u6Opb295FQKpV3dv5bRO4Am9mtA7DOGKOlUYoSabZvZ7mu281h09akoPp64G9/Y8bJd76T1EZBCQG7nV0sn3nGf8qhy0V/cgQi8WF/mowxtwN4AEADgPNEJGnGaSpKUtDWxv4NWVld0wxFWIw0Zgx7m3zwQVzFVGLECScwhlBe3tMt1NHBMbDz50dkJnjIBkFE7hOh89IYU2GMWWyMudsYc3VfhRCRe0Rki4isF5GFIlLQ13MqStLyySfcCfgb5GOzcTDGK6+E52NWkpPsbOAnP2EV6q5dNADWz6oq4IILgPPPj1lhmkUjgJdF5CJjTJOInArgV8aY4/osBfAWgNuMMU4R+T2A2wDcGoHzKkry8eGHwWc85+ZSKVRVab/o/kBuLgfDV1RwWlZrKztCTpkS0WlZIRsEY8wvRORbAP4jIm0AmgD8NBJCGGOWdPpzJYBvRuK8ipKUWGXogbAmJrW3x0YmJTEYMoS3KBGOy2gugKtBQ1AM4AZjzPtRkOlKAG8EkOMaEVktIqurqqqi8PSKEmdGjGATs0BYw6jzNeNbiRzhBJV/DuCXxpgTwRX8syJyUqj/LCJvi8inPm5ndTrm5wCcAJ7ydx5jzCPGmDJjTFlxcXEY4itKknDcccwaCRQfqKgAjj5ae5coESUcl9FJnX7fICLzASwAEFIDBWPMyYEeF5HLAZwBYK4JNsZNUVKZkhJ2LfzwQ06S6R4srK+nsdCJQkqE6XUjbWPMPo8bqc+IyDwwiHyCMaY5EudUlKRFBLjqKir9VatYoZyTw11DQwPnit5yC/tAKEoE6dNkBWNMS4Tk+AuADABveTJbVxpjro3QuRUl+cjIAK67jt0M33+fA7QzMrhzOPLIwIMyFKWXRG7UTh8wxhwSbxkUJeEQocuotDTekij9BK17VxRFUQCoQVAURVE8JITLSFESlf376cLfuJEx3gkT2ENs+PCk70KtKD1Qg6AoPjCGg6oWLKDiHziQ97/zDvDmm8z4/OY3tdmoklqoQVAUHyxbBjz7LEsC0tK89+fmskj4lVeYCaqlAEoqoesbRelGRwfwwgt0C3U2BhZ2O+eav/QS0BKpxGtFSQDUIChKNz77jK2EAqX6p6dzbMHGjbGTS1GijbqMFKUbdXWhHScC1NREV5aEoa0NWLeOQZTqavrLTjgBmDEjeKtuJWlQg6DEnKoqYOtW6piCAmDyZCAzM95SeUlPD+04Y/pJwXBlJfCHP/Bnfj4vVl0d8PjjjLr/6Ef+xzsqSYUaBCVmNDQATzwBrF7NSYAivKWnA+ecA5x2WmKkch52GLOHnE7/Ywks+SdMiK1sMaelhcagoaFrxXRWFo1DbS1wzz3AnXcChYXxklKJEBpDUGJCczP1ytq1DMiOGUP9Mno0Bz899RSwcGG8pSR5eaw1KC/334F6zx6grAwoKoqtbDFn7VruDPwNZSko4ECfZctiKpYSHdQgKDFh6VKOgB01qmfufkYGDcPLL7MQLBG48EKu/nfs6DqrprkZ2L6dr+Pyy+MnX8x45x3/s50thgwB3npL5zunAGoQlKjjdAKLFwNDh/o/xuFgOuf70ZjB1wsyM4GbbgK+8x3+vWsXb+3twLe/Ddx6az+JpVZVBZ/Zm5FB15KO80x6NIagRJ26OrqgrWpff+TnA1u2xEamUMjIAE46CTjxRM6kAViYZrfHVazYkpNDRR8o0u508k3xVbShJBW6Q1CiTiIEivuCzUavSUFBPzMGAIMpBw8GPqayEjj2WO3jkQLoFVSiTn4+b01NgY+rrWUKqpJAHHMMt0oNDb4fb21laffJASfkKkmCGgQl6tjtwPz5nAvvL+7Y0cHHjj8+trIpQSgoAG68kT6z8nJeKIANnfbuZRbAd7/Lpk9K0qMxBCUmzJ7N8cDbtnEUcGfXS0sLsG8fM3sGD46fjIofJk4E7rgDePddpos5nfQDHn00cMopWpSWQohJ4lSxsrIys3r16niLoYRIczPwzDPAihUs7LLIyWEr6dmzkz/ekPI4nXQTZWRoEDmJEZE1xpiy7vfrDkGJGdnZwJVXAuedxwZy7e2MLYwfr7olaXA4+km+bf9EDYISc/Lz2RNNUZTEQoPKiqIoCgA1CIqiKIoHNQiKoigKADUIiqIoigc1CIqiKAqABDEIInKniKwXkXUiskREhsdbJkVRlP5GQhgEAPcYYw43xkwD8CqA2+Msj6IoSr8jIQyCMaa+058DACRv+bSiKEqSkjCFaSJyF4DLANQBmBPguGsAXAMAJdpQS1EUJWLErJeRiLwNwNfMrJ8bY17qdNxtADKNMb8Kdk7tZaQoihI+ce9lZIwJtWH6vwG8BiCoQVAURVEiR0LEEETk0E5/ngkggQYpKoqi9A8SJYZwt4iMB+AGsBPAtXGWR1EUpd+REAbBGHNevGVQFEXp7ySEy0hRFEWJP2oQFEVRFABqEBRFURQPahAURVEUAGoQFEVRFA9qEBRFURQAahAURVEUD2oQFEVRFABqEBRFURQPCVGprEQfpxPYvh1oagKysoBx44C0tHhLpShKIqEGIcUxBli6FFi0CKirA0R4X3Y2cMYZwKmnAnZ7vKVUFCURUIOQwhgDvPgijcGwYcDo0d7HWluBp58G9u8HLr8csKnzUFH6PaoGUpgdO4BXXqEhyM7u+lhmJlBaCrz3HrBxY1zEUxQlwVCDkMK89x6Qng44/OwDbTYgNxd4883YyqUoSmKiBiGFWbcOKCwMfExREXcIbndMRFIUJYFRg5DCuN2hxQaM4U1RlP6NGoQUZtw4oLY28DH19cCoUZpppCiKGoSU5pRTgIYG/6t/Y4CDB4H582Mrl6IoiYkahBRm0iSgrAzYubNnjMAYYPduYPx4HqMoiqJ1CCmM3Q5873vAU08B77/P+xwOVi0bQ0NwxRXMRFIURVGDkOJkZABXXgmcdRazjmpqgLw84IgjgCFD4i2doiiJhBqEfkJhITB3brylUBQlkdEYgqIoigJADYKiKIriQQ2CoiiKAgAQk8QlqiJSBWBnFJ+iCMCBKJ4/miSr7Cp37ElW2ZNVbiD+so82xhR3vzOpDUK0EZHVxpikzNJPVtlV7tiTrLInq9xA4squLiNFURQFgBoERVEUxYMahMA8Em8B+kCyyq5yx55klT1Z5QYSVHaNISiKoigAdIegKIqieFCDoCiKogBQgxAUEblTRNaLyDoRWSIiw+MtUyiIyD0issUj+0IRKYi3TKEiIueLyEYRcYtIwqXmdUdE5onIVhHZJiI/jbc8oSIi/xCRShH5NN6yhIOIjBKR90Rks+dzcmO8ZQoFEckUkY9E5BOP3L+Jt0zd0RhCEEQkzxhT7/n9BgCTjDHXxlmsoIjIqQDeNcY4ReT3AGCMuTXOYoWEiEwE4AbwMIAfG2NWx1kkv4iIHcBnAE4BUA5gFYCLjTGb4ipYCIjIbACNAJ4wxkyJtzyhIiLDAAwzxqwVkVwAawCcnejvuYgIgAHGmEYRSQOwHMCNxpiVcRbtK3SHEATLGHgYACApLKgxZokxxun5cyWAkfGUJxyMMZuNMVvjLUeIzASwzRiz3RjTDuAZAGfFWaaQMMYsA1AdbznCxRizzxiz1vN7A4DNAEbEV6rgGNLo+TPNc0sofaIGIQRE5C4R2Q3gEgC3x1ueXnAlgDfiLUSKMgLA7k5/lyMJlFOqICKlAKYD+DDOooSEiNhFZB2ASgBvGWMSSm41CABE5G0R+dTH7SwAMMb83BgzCsBTAK6Pr7RegsntOebnAJyg7AlDKLInCeLjvoRa9aUqIpIDYAGA/+m2k09YjDEuY8w0cMc+U0QSylWnA3IAGGNODvHQfwN4DcCvoihOyASTW0QuB3AGgLkmwYJFYbzniU45gFGd/h4JYG+cZOk3eHzwCwA8ZYx5Md7yhIsxplZE/gNgHoCECerrDiEIInJopz/PBLAlXrKEg4jMA3ArgDONMc3xlieFWQXgUBEZIyLpAC4C8HKcZUppPMHZRwFsNsb8Kd7yhIqIFFvZfiKSBeBkJJg+0SyjIIjIAgDjwayXnQCuNcbsia9UwRGRbQAyABz03LUyGbKjAEBEzgHwZwDFAGoBrDPGnBZXoQIgIqcDuA+AHcA/jDF3xVei0BCRpwGcCLZirgDwK2PMo3EVKgRE5HgA7wPYAH4vAeBnxpjX4ydVcETkcACPg58TG4DnjDF3xFeqrqhBUBRFUQCoy0hRFEXxoAZBURRFAaAGQVEURfGgBkFRFEUBoAZBURRF8aAGQVEURQGgBkFJEUSkVERaPH1irPt6tKUWkSxPK/N2ESnq43NmichST8dTiMgNnpbMYbUJEZECEbmuL7KE8Bw9Wl2LSLqILBMR7VigAFCDoKQWX3j6xFhtqf8KYD6ASQAuFpFJxpgWzzGRaC9xJYAXjTEuz9/XATjdGHNJmOcp8PxvWAgJ9Tv8T7BNwld4urO+A+DCcJ9bSU3UIChJg2coyime3/+fiDwQ4PBYtKW+BMBLHnkeAjAWwMsicpOIfNszDGWdiDzcaRexSETWeAakXOM5z90AxnmOvcez2+m8kv+xiPza83upZxfyIIC1AEb5e67OBGh1vcjzOhRFDYKSVPwKwM9F5BKw5fFNAY6NaltqT9+iscaYLwHA0xZkL4A5ABaDq+7jPLsRF7xK90pjzFEAygDcICKFAH4Kz+7GGHNLCE8/HhxqMx1AdoDnCoVPAcwI43glhVHfoZI0GGOWeRqb3QzgRMtVIyJ3gs3OOhPtttRFYJ8lX8wFcBSAVRQXWWD/e4BG4BzP76MAHApgf5jPvbPTlK1AzxUUY4zLE0/J9QybUfoxahCUpEFEpgIYBuCApbxEZCh8f47DbkstIj8AcLXnz9MBnNP5b2NM5/9vAZDp71QAHjfG3Nbt/CeCHS6PMcY0e9of+zqHE113792PaQr2XGGSAaC1D/+vpAjqMlKSAuEc3afAOECTiFjdT6cDWOfjX8JuS22M+avHbTPNGLO3+9/djq0BYBcRXwr9HQDfFJHBHtkHichoAPkAajzGYAKAWZ7jGwDkdvr/CgCDRaRQRDLAmRb+8PdcIeFxWVUZYzpC/R8ldVGDoCQ8IpIN4EUAPzLGbAZwJ4Bfex6eBh8GwTNP+noAb4Izd58zxmyMsGhLABzv47k3AfgFgCUish7AW+DOZjEAh+e+O8FZ1zDGHASwwjMx7h6Pcr4DHAv5KgL0zA/wXF3wtLr+AMB4ESkXkas8D80BkNBto5XYoe2vlaRGRB4F3TolAF41xoQ0klBEvgRQZow50Ifnng7gZmPMpb09R7wRkRcB3GaM2RpvWZT4ozsEJakxxlxljHGD2TX5nQvTfGEVpgFIg3e4Sm+f+2MA7/lK80wGPK60RWoMFAvdISiKoigAdIegKIqieFCDoCiKogBQg6AoiqJ4UIOgKIqiAFCDoCiKonhQg6AoiqIAUIOgKIqieFCDoCiKogAA/j9ckqQf6WEScQAAAABJRU5ErkJggg==",
|
|
"text/plain": [
|
|
"<Figure size 432x288 with 1 Axes>"
|
|
]
|
|
},
|
|
"metadata": {
|
|
"needs_background": "light"
|
|
},
|
|
"output_type": "display_data"
|
|
}
|
|
],
|
|
"source": [
|
|
"plot_dataset(train_x, train_labels)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "SjPlpf2-wHl8"
|
|
},
|
|
"source": [
|
|
"## Training One-Layer Perceptron\n",
|
|
"\n",
|
|
"Let's use Tensorflow gradient computing machinery to train one-layer perceptron.\n",
|
|
"\n",
|
|
"Our neural network will have 2 inputs and 1 output. The weight matrix $W$ will have size $2\\times1$, and bias vector $b$ -- $1$.\n",
|
|
"\n",
|
|
"Core model will be the same as in previous example, but loss function will be a logistic loss. To apply logistic loss, we need to get the value of **probability** as the output of our network, i.e. we need to bring the output $z$ to the range [0,1] using `sigmoid` activation function: $p=\\sigma(z)$.\n",
|
|
"\n",
|
|
"If we get the probability $p_i$ for the i-th input value corresponding to the actual class $y_i\\in\\{0,1\\}$, we compute the loss as $\\mathcal{L_i}=-(y_i\\log p_i + (1-y_i)log(1-p_i))$. \n",
|
|
"\n",
|
|
"In Tensorflow, both those steps (applying sigmoid and then logistic loss) can be done using one call to `sigmoid_cross_entropy_with_logits` function. Since we are training our network in minibatches, we need to average out the loss across all elements of a minibatch using `reduce_mean`: "
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 57,
|
|
"metadata": {
|
|
"id": "kdDxWeCqwHl8",
|
|
"trusted": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"W = tf.Variable(tf.random.normal(shape=(2,1)),dtype=tf.float32)\n",
|
|
"b = tf.Variable(tf.zeros(shape=(1,),dtype=tf.float32))\n",
|
|
"\n",
|
|
"learning_rate = 0.01\n",
|
|
"\n",
|
|
"@tf.function\n",
|
|
"def train_on_batch(x, y):\n",
|
|
" with tf.GradientTape() as tape:\n",
|
|
" z = tf.matmul(x, W) + b\n",
|
|
" loss = tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(labels=y,logits=z))\n",
|
|
" dloss_dw, dloss_db = tape.gradient(loss, [W, b])\n",
|
|
" W.assign_sub(learning_rate * dloss_dw)\n",
|
|
" b.assign_sub(learning_rate * dloss_db)\n",
|
|
" return loss"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "zAAgw0h6KzUd"
|
|
},
|
|
"source": [
|
|
"We will use minibatches of 16 elements, and do a few epochs of training:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 58,
|
|
"metadata": {
|
|
"colab": {
|
|
"base_uri": "https://localhost:8080/"
|
|
},
|
|
"id": "PfyqjVb2wHl8",
|
|
"outputId": "308850b8-fe17-4cda-ac27-8bcda210f113",
|
|
"trusted": true
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 0: last batch loss = 0.8960\n",
|
|
"Epoch 1: last batch loss = 0.7991\n",
|
|
"Epoch 2: last batch loss = 0.8121\n",
|
|
"Epoch 3: last batch loss = 0.9656\n",
|
|
"Epoch 4: last batch loss = 0.7388\n",
|
|
"Epoch 5: last batch loss = 1.0532\n",
|
|
"Epoch 6: last batch loss = 0.7494\n",
|
|
"Epoch 7: last batch loss = 0.8559\n",
|
|
"Epoch 8: last batch loss = 0.8226\n",
|
|
"Epoch 9: last batch loss = 0.7524\n",
|
|
"Epoch 10: last batch loss = 0.9427\n",
|
|
"Epoch 11: last batch loss = 0.6479\n",
|
|
"Epoch 12: last batch loss = 0.7756\n",
|
|
"Epoch 13: last batch loss = 0.7771\n",
|
|
"Epoch 14: last batch loss = 0.7996\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"# Create a tf.data.Dataset object for easy batched iteration\n",
|
|
"dataset = tf.data.Dataset.from_tensor_slices((train_x, train_labels.astype(np.float32)))\n",
|
|
"dataset = dataset.shuffle(128).batch(16)\n",
|
|
"\n",
|
|
"for epoch in range(15):\n",
|
|
" for step, (x, y) in enumerate(dataset):\n",
|
|
" loss = train_on_batch(x, tf.expand_dims(y,1))\n",
|
|
" print('Epoch %d: last batch loss = %.4f' % (epoch, float(loss)))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "s4_Atvn5K4K9"
|
|
},
|
|
"source": [
|
|
"To make sure our training worked, let's plot the line that separates two classes. Separation line is defined by the equation $W\\times x + b = 0.5$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 49,
|
|
"metadata": {
|
|
"colab": {
|
|
"base_uri": "https://localhost:8080/",
|
|
"height": 283
|
|
},
|
|
"id": "PgRTHttLwHl9",
|
|
"outputId": "e4407e1b-edf5-48e5-fdc2-da28120a3c6b",
|
|
"trusted": true
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stderr",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"C:\\Users\\dmitryso\\AppData\\Local\\Temp/ipykernel_66332/2721537645.py:17: UserWarning: Matplotlib is currently using module://matplotlib_inline.backend_inline, which is a non-GUI backend, so cannot show the figure.\n",
|
|
" fig.show()\n"
|
|
]
|
|
},
|
|
{
|
|
"data": {
|
|
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAYQAAAEKCAYAAAASByJ7AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAABSZklEQVR4nO2dd5jU5fX27zPbe2ELy1KWJioIFhCxgh00lkRji0lMMUYTjYmxJjFvEvOLMYnGWKKJpqjRJIolKlJUQBRUqoCAIrBLX2Bn+2yZ3ef9456vs2X6Tt/zua69YKc+MzvznOe0+4gxBoqiKIpii/UCFEVRlPhADYKiKIoCQA2CoiiK4kINgqIoigJADYKiKIriQg2CoiiKAgBIjfUCBkJJSYmpqqqK9TIUJSx8tP8j5Gfko6qwKtZLUZKcVatWHTTGlPa9PKENQlVVFVauXBnrZShKWMi+Jxtfn/Z1/O7s38V6KUqSIyLVni5PaIOgKMlCu7MdDqcDRZlF0X3igweB5cuBbduAlBTgqKOAqVOBnJzoriNcGANUVwNLlgBbtgA2GzBlCnDKKcDQobFeXdyjBkFR4gB7mx0AUJQVJYPQ3Q28/DLwyiuACJCby8tWrQL+9S/gO98Bjj02OmsJF11dwNNPA2+/DaSlAYWFNBBvvAG8/jpw6aXA7Nl8vYpH1CAoEaO9HVi7Fti0id/LMWMS+/AZSewOl0GIlofw2mvA3LnAqFFAao9toKQEaG0FHnwQuO024IgjorOecPDSS8CbbwJVVfQMLHJzgc5O4Nlngfx84OSTY7XCuEerjJSIsG4dcPPNwKOPAitWAB98APztb8APfkBvXiW0ehNVD6Gpid7ByJG9jYFFdjZQUAD8+9+J84dqaqIXMHJkb2NgkZbGkNELL9CTUDwSNwZBREaIyNsisklENorITbFekxIaH38M3H8/PYGqKn4Py8v5/5IS4IkngHffjfUq44uoeghr1nBTTEvzfpvCQmDHDmDXrsivJxx89BFfkycDZ5GTA9TXA1u3Rm1ZiUbcGAQATgA/MsYcAeAEADeIyJExXpMSJMYAzzzD/cRTaCgjA6ioYJi6oyPqy4tbouoh7N3r2xgAjLPbbIDdHvn1hIO6OibF/SECNDZGfj0JStzkEIwxewHsdf2/SUQ2AagE8HFMF6YExY4dwJ499Ny9kZUF1NYCGzYkXt4yUkTVQ8jMDDxs4uvEHW727KHruGsXkJ7OhNPRR/MU4Y/s7MBekzGBPd4gJW4MQk9EpArAMQDe93DdtQCuBYCRvnYdJSYcPMhDmL9CDhFg377orCkRsDyEwszCyD/ZxInAiy9yc/T2h+rspDGIRuNnZyfw1FPA0qV8TmtzX7mSCeEbbwTGj/f9GEe6ggn+XlNaGjBuXHjXn0TEU8gIACAiuQBeAPADY0w/384Y87gxZqoxZmppab9GOyXGpKQElod0OoHPPgN+/WvgpptY0PLKK8ChQ5FfYzxid9iRm56LtBQ/oZxwMHYsXbiDBz1fbwywezdw+uncnCPN008DixdzTcOHA8XFQGmpuwLqt7/1n8uoqACOOYbr9oQxfIwzz4zOa0pQ4sogiEgaaAyeMcbMjfV6lOCxDpS+vPfWVlYhLV5MLyE7210Wf/vtwOrV0VhpfGFvs0ev5FQEuP56Wu+dO2mdLRwOYPt24LDDgIsuivxa9uyhZ9C3VNSioICXv/KK/8e65hoahu3bgZYWXmYMcwbbtjH8FI3XlMDETchIRATAEwA2GWP+EOv1KKFRXAwcfzy9/REj+l/f3c2+oZQUYNIkt3efkcEkdGsr8NBDwM9+Fp1oRbxgb7NHrykNYOnXz37GUs2lS7lxGsM/wpe/zJN0Zmbk1/Hee9zwPRkDi7IyfqAaGmiwPviAJ4mcHCahJkzg/fPzgTvuAJYtA+bNA2pq+JoqKthoN326/2T6ICduDAKAkwBcDWC9iKx1XXanMeb12C1JCYUrr6R6QE0Nv4vWd7Cri01qzc3AnDmeQ73Z2cwnzpsHfPe70V13LLE7oughWJSUAF/9KvClL7FKx2bj5hvNTXPXLv+dilYc8rHHWNOcksIThNMJLFjAMNP3v08jl50NnH02DVpzs7sLW7uTAyJuDIIxZhkA/aslAQUFwJ13snH0nXfoFYjw37Q0HtTy8rzfv7SUB8KWlsHT1Wxvs2NccYySnTk5sXujMzN7h6w8YQyweTObz448sr83UVsL/OY3wN13A0Uuo2p5DEpQxI1BUJKL/Hz34XPPHn6ny8qoiNDQ4Pu+Vjn5oDIIsfAQ4oHjjmMruy/27GEe4MwzPYeWysrojr71Fj9wSsjEVVJZST5yclgxeNhhbFbLy6PGkS+MoTcRjRB2vBDVpHI8MWUKTw/eTgnGABs3AqNH+248Ky8HFi1iaakSMmoQlKhy8skM7fri0CFGBgaLx9/R1YHWztboJpXjhfR01h23tNAT6Fme1tzM6qDCQp4ofJGRwZOGdiEPCDUISlSZPBkYMsR7CXxnJ0PF550X3XXFkqgrncYbY8cy/j91Ko3Czp0MAQEsJT3hBP/NLVaVVDQ7q5MQffeUqJKRAfzwh8C99/I7X1bmVlKoreUh76qr3I2ng4Goz0KIRyorgWuvBa64ggJ0qan8cKSk8JTw3HOsVvBGYyOrjQaLWxkh1CAoUaeyEvjFL1gu/sYbwP79rEKaNg0466zBpyww6D2EnuTl9S9Bmz4d+O9/2YOQldX/PsbQ5bzsMi0vHSBqEJSYUFgInH8+Q0Pt7SxHDUSsMhmJaw/Bah5ZtMgtOjd9OnDSSexjiAaFhWwse+ghegDFxe6Nv7WV6q2nnMKOSGVAqEFQYorI4Kom8kTceggtLdyEP/6YzV25udQsf/ll4H//A771Lcb3o8G0aexCfuEFzjNISWEpmlXffPrpvrudlYBQg6AoMSYuPQRjgMcf56D6qqreoZjcXIZvHn2UjWATJkRnTYcfzo7H2lqWqaanM2+gieSwoe+kosSYuPQQqqupQDhqlOe4fFYWY/0vvwzcemv01iXCnoPy8tDubwyHdtTU0MMYOtSthaSoQVCijzFuuX39HtJDyEnLiY70daAsX84/kK8k7ZAhzC8cOsT/xzs7d3J+644dvecmlJSwvHXixJguLx5Qg6BEDbud2kYLFjA8nZICnHgicMYZPIgOVqKudBoItbWeK3p6Yo3ZbGqKf4Owezdwzz380PX1ehobgd/9DvjRjyjBO4jR85kSFWpqgJ/+lIO6cnP5nRw6lDI2d99NQzFYsTvsKM4qjvUyepOX518GwtIYSYSRlM88QyNQWtrf67Eql/76V/9Ce0mOGgQl4jgcwO9/z8PkqFHuqqLUVGDYMEpkP/EEJ6gNRuocdfGVPwBYwtnW5vs2jY384w0dGp01hcrevayUKivzfpu8PDbEbdoUtWXFI2oQlIizahX3jmIvh+CMDBqJ+fOju66gcTpZ8rhhA5OugcwKDYC4DBkdcQStdW2t5+u7utgM9oUvxH8z2N69PI34W2dKCv+ugxjNISgRZ+lS/4oC1gyEtrY47Evo7mZj1muvMV5uDXcYOpRyy1OnDujh41L6OiWFonP33stNsqyMOQVjmERubGRXYbT6EKJBmAx8IqMGQYk4jY3+w8xWl3J7e5wZhO5u4O9/59zPigr3ABZjaBwefBD4ylc4pStE4lb6urzcneCZP5/egjGsxjn33N4zUOOZykp3vsNXWVt3N2W2BzFqEJSIM2QIk8rZ2d5v43Tyu+qvsCXqrFsHLF7MjaLnZiJCtyczE3j2WeCoo2gwgiTupa8LCqgxMmcOk0GpqYmRRO5JeTmN16efes93NDTwg3r44dFdW5yhOQQl4sycycO0L/bvpzxO3DWdzpvHTdHbyTI9ndctWRLSw8dlU5onbDZOO0o0Y2Bx1VX8cO3f3zs0ZAzroRsaqLY6WAW1XMTb109JQqZM4cFs/37PDaa7drG4o7mZ+2p5OTB7NgtdfHkVEae9HfjkE/9NEkOGAGvWAJdfHvRTxKVsRTIydChw113AP/5BOQ6rMc0YhpS+9z2O9hvkqEFQIk56Ont+fvc7NokWFvKw2d7OiEx1NQfnjBvHg2hzM8P28+cDP/6x9+qkiNPdzU3DX5zcZgt5dGPCeAjJQEUFcPvtbFKrqaExKC8HxoxJjFxIFFCDoESF0lLg5z8HVq9mwc6BA0w2O50MUfesQrIk8ffsAf70Jza0xUTiIjOTSeSWFlowbzQ0hDzRRz2EGFBZyR+lH5pDUKJGVhbzBHffzY1+9Gj3jHVPDBsGbN/OXGBMEGHsylstPsBTZmsrcOaZIT2FeghKPKEGQYkJ+/fTa/c1FRFguOn99wfwRMZQwz/UGvOTTqLE8u7d/R+ju5vxrqlTg5aA7uxk2mHx+zQIe7cXDXbVBCUO0JCREhMcDhZ0+AvdZmRQUSBoDhxghnrRIna7paWx3GnWLLoegZKdzUTGn/8MbN7M2FV6utvInHwycPXVQcW0Vq8GnnySuZL1xXagAHjy4SK8VAhcd92gr3xUYogaBCUm5ORQ/aCnCrEn2tpCENL85BOKJzmd7LDNyOCR/O23+XPDDcAxxwT+eIWFwG230aVZs8at7jl1qm99HA+sXQv88Y+8W0kJsAV2pJkcjKlKQ2Mj8NvfcjCYFrwosUANghITSktZ3HHggPcqImtuQlDqCPX1wP330+L0TE6kpTH009oKPPww8MtfBtdIJsLy0wHodDudrJ4qLXXnqNtgRxaYP8jPp5H8xz+4PC18UaKN5hCUmCACXHghe4I6OjzfZvduhk/GjAnigd97j/Ws3jLVVmPD4sXBLDcsbN7MgqTcXPdlDtiRCXdCubCQfRk7dkR9eYqiBkGJHZMnA1//OsUod+3iPu500khs384D/fXXB3lSXrzYf4ypvJy3i7KY2b59/Z+yp4cAuGfO7N8f1aUpCgANGSkx5owzWKCzeDGriTo6WCJ+1VXA0UczfxsUTU3+O9nS0mh9urqiqpVhs/U3CA7YUYTeLpAxOlpUiQ1xZRBE5EkA5wOoNcYM7ll2g4jhwykY+pWv9L/O2kAD9hKKi91VRd5ob2foKMq6NVVVbrUE6/X09RCM4c/IkVFdmqIAiDODAODvAB4C8M8Yr0OJIZ2dlLSYN4+hI5uNYqJnncW5LT6NwxlnAE89xVZnb9TWsj06ylnb0aOZkz54kIlloH8OYd8+vtZ4H0KmJCdx5ZgaY5YCqIv1OpTY4XAADzzAMQMHD9J7qKigHtm993Kv7+728QDHH89ut0OHPF/f0EDv4dRTI7F8n4gA3/oW179vH9DZ3YlOaUEmitDdzSR6ZibbGhQlFsSVQVCUp57i+NvRoykjZLMxslNWxtP1okXAwoU+HiA3F7jlFt5pxw4KJnV2sgusuprholtuCaG5ITyMGAH85CcU8tu6i13KDnsRdu6kZ/CTnwTd2qAoYSPeQkZ+EZFrAVwLACM10JpUHDgALF/OTdNTNMdmo7fw6qvA6af7SBMMHw786lfAhx8Cb77JsqX8fOCKK4Dp0/3rZUSYykrapHe32PGf54ALzy7CtSfEzEYpyucknEEwxjwO4HEAmDp1qg5BTSLWreO/vipssrKYAti6lfkEr+TmUqZi1qywrjGcpOTQQ5hyWJEaAyUu0JCREjc0NgZWBSrChuNER5VOlXgjrgyCiDwLYDmACSKyS0S+Ges1KdGjoCDwOTO+xhMkCjoLQYk34sogGGOuMMZUGGPSjDHDjTFPxHpNSvSYMoWnf19VRA4HjcHYsdFbV6RQD0GJNxIuh6AkDnY7h41lZwc2BrOkhOMHli1jRZGVWM5uPYjhO5cjz16D2vo0TPrKMUjrngwgQQe+u1APQYk31CAoYWfzZuCVV4BNm1j92dVFOecLLwQmVtRxNibAuQR9LMVXvsJcwtq1QGFeF07c+wLGfzYPbe2CVpONU0Z0Y9yH7wGbc4Hvfz/owTTxRJ2jDjlpOUhPCVafQ1EigxoEJawsWQI88QSrPEeOdEs1NG07gA+v+S8Ki1aicrjr6N/dDRx3HHDZZZ+37mZmAjfeCGzYAGz77VxUrP8f9uVVoWxsCg4fQ/shAlqN++7jwOUBSFLHEnubXb0DJa5Qg6CEjd27qfdfWcmZNBa5Lftxyvp7YOtwYOW+4cg5wobCQtAgrFvHgTZ33UUVUrDS6OiRdTg6cx5weZVnzaH8fCYU5s4Fbr45Cq8u/Ngdds0fKHFFXCWVlcRm8WLu3T2NAYzBMWv/hpSuDrQWVsKWasP27a7rbDZaj44O4G9/6y0Fag1S9iVAV1oKfPSRd5mKOEc9BCXeUIOghI3ly92ibRZ5TXsw5OBmtOTw9J+Tw9kHvSgvZ+LByi0AwM6djB/5wmbjT11iyl+ph6DEG2oQlLDR0dG/saywfgf/4yoZEnHPUv4cq5yo55iwjAze0B/GhGemQWsry6La2wf+WAESFg+hro7v2549flT/FMU/mkNQwsbQoZxP01MqSEw3Up1tKD74CdI7mtFu0pCRNxSCEgB9BIt6bmhHHw28/bbvJ2xro5ZFZWXoi96yhTrb69bRMKWkAKecQq3tYGYuh8CAPITPPgNeeonZ95QUvnelpZT1PukknbCjhIQaBCVsnHMO8PjjboNg6+rEiJplGLZnJTrS89BtS0VmezfGdH8GvJVHqWprboFI73jTxIksKbLbKXvaF2M4e/NLXwphrJqLhQspr5qbS0U9m40zPJcuBd55B/jRjzjUOQJ0dnWipbMlNIOwdi3wxz/SGFrrBmiNH3+cQySuvjrq8x6UxEePEUpQGMNqovXrKVPdU1PouOMoNLpnD284Ze0/UHrgY7Rml8KZmokW5KArOw/Zwwp5ul+2jJVCjY2U+jzsMPeDpaayz6CtjQ/YM3zkcHDTmzQJOPfc0F7Ip58CTz/NBZeVuTfV1FRelp/PTbexMbTH90PITWkNDcAjj7CLr+e6ARrX0aNp6FatCuNqlcGCeghKwGzdCjz3HKMVIu6B8DNnAhdfzI7kW27hcJvG9TUo/WQZ9hWORkN+Lqr2LUdGtmDYyHQWDuXkcHPbuJHG4JZb+oc5Ro8G7r6bXW4ffOAeSpydDXz5y8DZZ4fuHbzxBh/H0/2N4eV79zJTfs45oT2HD0KWrXj/fXox3sScbDZ6Vq+9RgutXoISBEEbBBHJAdBmjAkg46ckCxs2AH/4A/chq+EMoBjdokU0Ej/+MaM7P/0psPe+pWjcn4buDBuyysqRN3E6Cnesga2l1b3xO5086f/sZ5wO44lhw4DrruMsg7o6xssrKnzPTPZHezuwejXDLT2x3J+tW4H6eq7vrruov3H66WDzRHgI2EPo6uKgCKeTb+4HH/if51BYyERzUxM9HUUJEL8GQURsAC4HcBWAaQDaAWSIyAEArwN43BjzaURXqXilvh5YsYJhZaeTk7hOOYX7aLgOh21twKOP8uCZm9v7urQ0Ngpv28bc7Be/6Gov6NiByml5aEkBamqANXsq0J1Shoq0/RiRZ0d2lsGh7iI4GjvxyZYqjC2mYJ3XNRcUhG+wTUeH272xMIZv4vbtjM0XFPAN7ezkRJ6lS4Hbbw/bsGO/HoKVy3jtNeZRrLXu38+Qlq/3wnptTmdY1qoMHgLxEN4GsAjAHQA2GGO6AUBEigHMAvAbEXnRGPN05JapeOLdd9nP1dXF/cFm42l9/nzOmr/iCt99XYGybh0PyX17DHpSUQEsWACcd56rMS0tDbtrurDyM16flQVISgp2tA/D6k+HoaMDKC4yKGvfiVfnpaDpTR7Yr78+4sU9XEx6Og2DFTLasYPGoKiot/tTUECXqLaWsbBf/jIsb6pPD8HppAX+4AP2aFiTAZ1OVkV99hnDZbm5DGsdOkSDVlzMk4AxtNR9rbcSOF1dHLuakkK3eJCE3gIxCGcaY/qp1Btj6gC8AOAFERmA/66EwkcfAY89xu9/z/6t/Hx+lufP58Z86aUDf67167mH+iIjg/vr3r1AVRWwfchU7FzzDHIqinq1CTQ28nsmAqC5CaayDOWjs3HUvhUoXbQE61+vR96cQuSeP5N62P6a00IhNZWT1BYsoBXq7qZ8Rm5u7y9+RwcwZgz/X1bGmcyffOJnVFtg+PQQ3nyTxmDMmN7rSU1lOe6iRfyxejWsN7i6mh+M8nLgmmtCz68MZpqaKMg1fz4rJrq7aZDnzAGmTg3PCSuO8VtlZIzpFJHDReQMEel15BCRc63bRGqBSn+MAf79bx4IPe2XKSkM48ybx7ztQHE6Aytrt2YZGAM8t+14pGSlIdO4y5CcTobDMzOBzAyDtPoD2FI0A6e//RNMW/VnVHTvRndnF3a8txv4859dyYi9A38BnrCGMtfXcxNoa+u9gVrx954T79PSgJUrw/L0Xj0EpxN4/XW6SZ5OpcXF/KmupgdTWEhDlpvL/xvD+J3mDoLn0CF6gM8/z4KDESNoDBoagIceAv7618CaJRMYv19zEbkRwMsAvg9gg4hc2OPqX0dqYYp3ampYiekrjJyays159eqBP9/o0f5HVlrdxyUlwL59wJa9+Vg/4zvIaalFVitDGlYFZ6ZxYFj7dmzInoZDmw8gtbMNDYVVaMsqRnphNj6rK0b70FF80vvuo0sRbkpLgVtv5cJraugNdHTwOevrucHOmNH7RJiWxthZGLA77MhOy+4vfb1nD41Rdrb3OxvDElO7nS6Xw8H3qL6ertysWcD//hf4+DmF7+kjj3Dzr6pyu8QiNLRjxjBGu2BBLFcZcQLpQ/g2gOOMMRcBmAngpyJyk+u6wRFYizMaGtxln75IT2foe6BMn87n8rW/7NsHnHACD6ZNTdxH91Ueh2Wn3ImmvAoUNNQgx74Tw7pqkGHasKjsSryVfT6aHClozXEnJyxPpLUV3LTtdoZPIsGYMcC99wJf+xrDLzYbcwgnnACcdlr/OFlbW2+PYQDY27x0KTudvv+wdjvfnMpKDpkYM4ZrHjaMBuyMMxgyam5mSa8SGNu3MzfjrWhAhF7b668ntaENJIeQYoxpBgBjzA4RmQngeREZBTUIMSHQikun0/dBM1AKC1k99O9/04Pu+/wHD9L4XOjyHTMy3FpFh4YchmWn3Imc5v3YuqoB+w6lobloBLokFXmbViC1wHOc+/M9ccgQxnNPP33gL8QT2dnABRewrnbvXj6fJ4yhN3HCCWF5Wq86RoWFdO26uz3H6SxXzemkcfLWSS0SntPAYGHtWp5ifBnjrCzGPKurWc6XhATiIewTkaOtX1zG4XwAJQC8FI8rkWT0aG7KHR3eb2MM95RJk8LznOedB1x5JaseLS21nTv5/7w84I47Ph9ngOHDua/1jPS05JbDOeYwVNtGo0tSucDOTowt6C1d7XTye/l5gUx2dnTkrb/0Jbo2Dkf/64zhJnDCCTyJhwGvOkbFxcDkydx4PGFtWMbwjfaGVWmkBEZzc2Dvl0hUBRCjTSAewlcB9CpoNsY4AXxVRB6LyKoUn2Rmsnn25ZcZ7vR0qNm/n3X9VVXheU4RYPZs9jisWcP+rfR0Sg6NH9/7MJuSwkP3E09wP7euGzqUuQ2G6wXjMhsxMn0fDNxeQnMzp2J+XpnU0RGZSqO+TJgA3HADtYCsJrCUFMbo29vZ9XvNNWF7OnubHVWFVZ6vvPhiJjebmtxaTxYFBbx80iTvZaWWezZ2bNjWm/SUlvo+YQHuU1YSJ+z9GgRjTF/1+p7XvRve5SiBcv75PJ2vWcOTubU3tLcznl9cDHz3u+Evn87NpVHwx6mnMle7aBH3sOJiHsAmTaKEUUEBMOOYNuTuqUVTwXB0d3PvLSzs440fONBPOqK+nmGqlBQe2HsN5BkI06bRui1fDnz4ITeIqVOpzTFuXFjfTLvDjmOGHuP5yqoqtn0/+CC9o9xcvtiWFhqrGTN8v+jaWq63bye24p1p04D//Md7qA5g8m74cN+eWYKjWkYJSno6td/eeYd5rpoa7ldpaTQWZ54ZvsbeULDZKLg5eTLX96mrl72ggLJFO3YA67efiPEtr6PV6YAzLQujR7PE/3PP3eHgizr1VAAM8b/0Evdqa29OS6NS9Zw5/nslAqKwkK7Q7NlheDDveE0qWxx+OPD739Pir1xJ4zRqFKWts7KA//s//tGHDXO7U11dfJNycoBvf3vQNFOFhZISFhK8/bZnt7u9ndIp11yT1O+rGoQEJi2NudaZM1l80t3NDTde+pFE2Ed19NEs0LGS3JZG3a5dFWhd8h1Mev5R5BWnI62yjCfhri6ecjs6GMYZOhQ1NdwDu7pYYGNVg7a3U/tu40YamnAk0SNNZ1cnmjua/esYZWUBJ57In77cdRdjhkuXukNExtB7uPhibnBKcFx5JT+oy5czTJmfzy+V3c4P8ze/yWbJJCZggyAiAuoZjTHG/EJERgIYaoyJUE2gEig2m/fimHihbxpAxBXR+MrxwMllrO9+/31uaiJM4J59NjBqFLq72ReUmupOXFtkZDDJvm0b98crrojaSwqZ+rZ6ACEonfakoAD46ldZ/rV3L9+38vLYuoWJTno68J3v0L1evJilqKmp9FBPOmlQGNlgPIRHAHQDOB3ALwA0gdIV0yKwLmUwUVUFXHst+wEcDp6Me8TIt2yhw+ArQV5ZSW//wgvj30sIeRaCJ3JzmfdQwoMI8y9JWlbqj2AG5Ew3xtwAoA0AjDF2AHESnFCSgowMxvD7JEw3bvQ/NjktjSGp6urILS9chDwLQVEiTDAGoVNEUgAYABCRUtBjUJSI0tERmKaYSGJIzVgeQnFWcYxXoii9CcYgPAjgRQBlInIPgGVQLSMlClRWBl4iHu+5FACoc9QBCFPISFHCSEA5BFdCeSmAVQDOACUrLjLGbIrg2hQFAHDsscBTT1FCxlsz6aFDDKVHfJZCGPAZMmpo4MSjd96hTEVZGfWJpkyJn/IxJWkJyCAYY4yIvGSMOQ7A5givSVF6kZfHYppnn/WspdTUxL3zsstis75g8ZpU/vhjNqO1t7OTLz2dGiEPP0w36Yc/TAwXSElYggkZrRCRiFYUici5IrJFRLaKyO2RfC4lvnE63crOAPvELruMFZbV1WxgtnSV2tvZ2JsoSg0epa/37AHuv59NZaNG0QpmZtIwVFWxNfv++5NaaVOJPcGUnc4C8B0RqQbQAoaNjDFmcjgW4kpYPwzgLAC7AHwoIq8YYz4Ox+MricGhQ8Bbb3FoWEcHcwMTJ9IgnHcecPLJVMO2SsSPOorRlLDJV0QBj13KCxe65xx4oqKC1m/DBuAYL5IXijJAgjEIke3lB44HsNUYsw0AROQ5ABcCUIMwSKipAX77W4Z/hg5lxMQYbv733ssG3IsuYr9aItNP+rqjgzkDb1r8Frm5bLZQg6BEiIANgjEm0hXelQB29vh9F4DpfW8kItcCuBYARlrDx5WEp70deOABdl33/LOKUIiyqAh48UV2JR99dKxWGR76SV9bs3v9NVtkZ3uXxVaUMBCMdMXPPF1ujPlFmNbiSTHKeHi+xwE8DgBTp07td72SmKxbR8mYUaM8X5+ayp61V19liCiR9cX6SV9bE4V8KW0C9CRKS71frygDJJiQUc9hspngkJxwlp3uAtBTr3c4gD1hfHwlSjQ1UZF0s6se7fDDqS7sLTwOAO+9x3yqL4qKOOWwvp7/T1T6SV9nZTEMtGlTf7GmnjQ0cJBPvNDRwcy+Max+8vcHVOKeYEJGv+/5u4j8DsArYVzLhwDGi8hoALsBXA7gyjA+vhIFli4F/vlPdgxbmkIffgj861+Uwz7tNM/3a2nxX2YvwgN0og+s8phUPvdcYNUqbrKe3oj6euYQjjsuKmv0icPBJPj8+W6JchEOyjj//PCUxtbV8ZTw0Uf8MB12GB8/TBPrFM8MRP46G8CYcC3EGOMUke8BmA8gBcCTxhidEp5ArFgB/OUvnB/St+qnvR346195uaexxGVlHMnpaxiVJUvhy9OId7xKXx92GMX9/vlPvkllLinw9nYq+6WnA7feGnvlvtZW4A9/4ICLigp3CMvp5Glg9WrOU/WXIPfF4sV8HwCqt4rQAL3xBoclffnLvkNrSsgEk0NYD3dMPwVAKYBfhnMxxpjXAbwezsdUooPTycaxoUM9l4BmZPC6Z5/lELK++dNTTwXefdetfu2J2lqGnkKNTBjDcv+GBq7HU5NbpPEpfX366UyiLFhAt8oY9iLMmUPXKh7kl+fOZdxu9Ojel6emUs+8thZ49FHg5z8PLdGzciXw5JNsxOv5QcrP54ngtdf4nlx00UBeheKFYDyE83v83wlgv2u2sqJgyxY2knlLCgPcyA8cYG5h0qTe15WX8/u/bBkPnuXlvT2B5mYanTlzQlvfhg3A88+zqc0a0JOTw8c7++zAxPPCgV/p67FjOfv0W99iE1pGRvQW54+WFmDJEt9hm9JSvsnbtgXfKdjdzTGWpaWeTxUpKbTir77KmQXeZkorIROMQbjeGHNbzwtE5N6+lymDk0OH3IO7/FFX5/6/08lD5/z5HFbV0MD9JDWVhuHIIxmmTk8HfvCD3iWpgfLee8Cf/8xE9MiR7oOrw0GPZccOjmOIxr77uY6RA3RXiov7Tw8C6LpE233xx/btPKX7WpeV6Pn44+ANwrZtPDH4OlVYOufr1nFojRJWgjEIZwHou/nP9nDZoKC1lZugCMO9g113zF8JvafbGgM8/TQ7k0eO5OVjx1KlYds2ylSsX8/RmKedFtowsLo6RiCGDeu/72ZlMfLx3nss8vGU2wgrbW2wL3kDAFD092eB9re4wc2axaRyYWGEFzBAnM7AwkBW7iNYGhoCe/zUVIamlLDj92ssIt8FcD2AMSLyUY+r8gC8F6mFxSt1dRwav2QJPVyA3u055zD0EJZB7wmIdRj0VUpvvV/WbbdvZ/6wqsp9H5uNBraszH2bzMzQJ0O+9x6f19MhHOD+M2QIQ9PTp/vZj7q6GO968022Vaen804nnui/P8DhAP7wB9h3LwLKgaLyKgClrCqaP585gzvuiI88gTeKivhm+kr0AAx1hZJUDvRU1bOETQkrgaTq/wXgC2CJ6Rd6/BxnjLkqgmuLO2prgV/+kifasjLm0EaMYL5r7ly37MJgpLycp+y9e73fZu9eNpVZpfaLF3MP8FUwUlYGzJvnNibBsnq1/4N3QQGwaxf7J7zicFBc7re/pVFISeFm/vLLwO230/L44vnnga1bYS9lYqQIrpNDejpDJC0twOOPBx53iwUjR7KErL7e+20sjfJQ5DXGjuX76kvAzxi3wJUSdvwaBGNMgzFmhzHmCgCNAMoBjAIwSUROjfQC4wVjgMce477QtzolM5Ohhx07gP/+N2ZLjDlf/SoPkTU1vb/TnZ28rKiIvQiHDjEstGWL/5N/Tg4jCS0tvm/nDafTf4WiFfb2Om3NGNbTbtxId6asjK5gbi4/DKWlTFJs8tKn2dxMl7KyEnZpAwAUoY/LUl4OfPIJa2/jFRHg8sv5B/F08nE6+Ye++OLQSsGyszn7Yfdu74Zx715+2T7+mB7Vd78L3HYbvayGhuCfU+lFwMW8IvItcEjOfAD/z/XvzyOzrPijuprVdr4aSSsrWYrt86SZxBQVAXfdBZx1FnODO3fy58AB5gBmzADuuYdS1bfeysjL1q2+p6FZB8JQpSqqqlj95Iu2Nhp1r/0Nu3bR1eiZke6JFdOaO9fzRrZtG61NairsaEOWSUVG32it1dzlzaiESmsrPZqNG9lVPFAmTgS+/3139v/gQVr4mhomyS+9lNK0oXLxxXyObdtoSC3a2njistn493j2WZ40hgzhe/vcc8Cdd/J+SsgEk1S+CcA0ACuMMbNE5HDQMAwKtmxxf2e9kZrK/WDbNoZGBiMFBcAVV7BM3Mr75eXxAP3mmzSoVqXQoUMsBz10iAUjnioNGxtpaEPtPZg5053v8eYp7N/P9XpNjK9YwSt9/fGLi9msdeCAOwFi0dnJ+zocsKc2oig107NyV0oKN75w0NrKcNZbb7njbd3dwBFHsLGrqir0xz7uOIbPVq1ydxKPGcNcSvEA50RnZAA33UT113nzaGhEaHTPPptxxuzs3tolubn8qa8Hfvc7njoSWdskhgRjENqMMW0iAhHJMMZsFpEJEVtZnNHWFnhzpNNPd0ZdHb9LtbX8bE+ZQi84kQXb+pKV5a4e/Mc/uFf2fY3jxvGQ2dgIrFnTv8rHGB5AL7kk9Pdm9GjuU+++2zt5bT3+/v3cw2bO9PEgtbXes9IWIjQaTU39DcKBA9w4nU7Yp+9CUa4TWLcMmDChdzK6q6v/fUOhtRW47z6eqIcNc8c3jeFlv/oVwyzjx4f+HNnZlJI45ZSBr7cv6ekMHc2aRcVDY3jSWLSIxtVbH0RhIQ3Iu+9SQkMJmmAMwi4RKQTwEoCFImLHIBKfKy/3EWN2YQlWejucOJ3Av//NzzXAz73TyYOc1Y8Uz0UmodDYyDBaZWX/TT0vj9GBjz5iFKC52d1r1NHBUPLUqSzkCRUR4Bvf4MFzyRL+npHBfcXppJG44QY/uYz8fN9xLQtP5UxvvQU88wyvS0uDPc2JojYwVrZjB1/c+PF8/NTU8LiW//sfH7tvPb9VI93QADz0EE/T8dbr0BObrbcu0ptv+v+ClJbyC6YGISSCEbe72PXfn4vI2wAKALwRkVXFIZMncwP3pj0GcPOrqOjf1Q/QWPzjH9yURo3q3QRlDDe/e+8FfvpT33o+icann/qW+h83jhv0++8zfGQd/tLSgAsu4E8wPQ6eSEsDvv51Tlz78EMe+HNyWAgzdmwA3sfUqdyMfNHUxM2qosJ9WU0N8NRTrMypqwOWL4f9hDaMbEnlB6m1lfo8Tic/VFdcMfBySoeDRqjnOvpSUJCY09fq632/LoAGubZ2YImnQUwwWkYC4CoAY4wxvxCRkQCOBvBBhNYWV2Rnc9D7M8941sBxOPid/9GPPH8Oq6t5Uu4btgB4+6FD+R1dvJibYLLgbwSwCN/P7m5GH44+mnvjuHHh7+koLQ1R+mLCBNYX79/vuaqgq4ub0LXX9v7jvv02rdnevcyujxgBe8YGTDnkSkZZXbdvv82E6ECSsRa7dvFN91fTn5HBRHMiGYS8PBpSXx+Mjg5aezUGIRGMZOAjAGYAuML1exM4A3nQYAkt7tnDw59VPlldzVDnDTfQk/BEIDX3Q4eyes5fDiKRCKahbPJk5iuPOirOGvxsNuDGG3kqqK6m9QfcSY7qam7mJ5/c+34rVjB+uGED3b6iItizBEUpuTzJZmbSwJSUhKcCCPA/ZKfna/JnreONWbP8T4yrraVIoBISwTjj040xx4rIGgAwxthFZFAJNogAX/gCK2KWL2cZqs1GobapU31rbW3f7n9ztLzdpqbkKZIYP56vu6XFe6VQRwcPrH0F73zhdLIUfdEiHr6tpuGTTw5PXrYfpaXA3Xez+mX+fHcJ1eGH0xgcdVT/U2l7uztZkZoKpxg0pXahKCWHHgdAo9LYSC8iFEE4T+vs7vZvGNrafGsGxSMnncSW8qYmzzXCzc2MxUYi0T1ICMYgdIpIClwS2CJSCiDE/tHEpriY8ehgSEnx321r1dwnk9R7aiqrhB57jHtg30iG08kN/corA/cKWlqYE/34YxrhvDy3MvJrrzGJHBHds/x8/uFnz6aXkJLiu/po6FAu0mUo6lPp+hV39og3dnTwRYgw5jhQg1BczMT0pk3e5SM6O/khmzp1YM8VbYYMocLh/fczn1BWxpOENbnNZmPJqo4ZDZlgtp4HAbwIoExE7gGwDMCvI7KqJOTYY313/APupHQyJZUBbs5XXw3s28dQW0MDX+vOnQx5X3QRtd0CwRgqPGzZwnxMaSn35BzXobu0lMYn3P1dvbDZ+IT+SlHPOYeJY1ezWl0qQzRFzh7nsNZWJkxEwie3esklfE67vf91HR38I3zpS4n5QTviCJbNzpnDL1R1NQ3pOefw8mDcTKUfgYjbPWWMuRpACYBbAZwBttVcZIyJ5NcuqZgxg42sVldsX6xw9He+k3z5MBH2FB17LCV/1q+ntzRjBr37YHTQqquBtWtpDHw1Db/4IveOmHL88e5mi4IC2NPoIXxuEJqauCmXlzMENSb0AYRdXcwRL1gAbN9eidK22/HFHQ+jKncH8orSICk2GoOUFLpjgVrgeKSsjAbti1/kC09JSb4vTYwIJGR0nIiMAvANAP8E8Kx1hYgUG2PqvN5T+ZwhQ4BrrqEkTnGxezIgQCOxdy/3jxkzYrvOSFJS4i4lDZUVK1icE2jTcEyjB1lZwC9+QXdlzx7YsxkzLGrs5Om2oIDdeAcO8N8Q5a/b2zmkbM0aRp8KC4GOgtF4OO1elBzYhC/kb8T0YzohI4b7T3YlElYzoBI2Ank3/wz2G4wBsKrH5QLmE8I2VznZOeUUxruff55euzW5KzOTXv7s2fEzHCteOXDAf67BEqtrbo6DcHJJCfCvfwG33gp76/sAgKKiCuCwI/lh2LePfQpXXOHngbzz9NOevaaK4SlwDp2ER2omoX0k9aQUxRd+DYIx5kEAD4rIo8aY70ZhTUnN0Ucz57dzJ+Po6en8IifKgJ3OTlZRbt7MsM/YsXxN/sLp4SKQpmGrY9yTNlJMqKgAHngA9n/dDNR9iqLCCrqF1kzQOXNCPrUfPMixo95091JTGZJ78UVWYOmBQ/FFIDkEMcSrMbBuE96lJS9WM1aisWEDE7bNze6wzcKF3Hi/9jVqBkWa449nH5cvmpq4Cfprao0qQ4bAPmUC8DZQdOvPgNRMLnKAlnTtWv7rqzItO9s9hW70aKpmNDXx7zZunM6aUdwEEjJ6W0ReAPCyMabGutDVg3AygK8BeBvA3yOyQiUu2LIF+P3vGZ/vW77ucFDN1GaL/BhKq2l43z7PyeiuLoaVrr8+/vKM9jY7slKzkDHu8PA9pj2wMLox7JR/+GH3pEpjaNjPOgu48MLE8VKVyBFI2em5ALoAPCsie0TkYxHZBuBTsGv5fmPM3yO4RiXGGENJnvx8z/1AWVncnJ9+OvLNr1bTcG4upT6sOS3d3TQE1dVsHoz4fOQQsDvsKMoKb8dhfr5/0UWAIcqXX6YBGDWKHuqoUUxxvPoq8MgjydUhr4RGIDmENlC24hERSQPLTx3GmPoIr02JE3bsYL+Ar8bW7GxWTm7cyJxCOGhqohjdhg00SuPGsQrLahpetozacJaawaRJLEefODH+vAOAHkJRZngNwpQpnBXjS8uttpbiiRde2D9CZeWwVq2iwGBEGvqUhCGomi1jTCcAH1NzlWTkwAH/w4EAnt737g2PQXjvPeBvf+Op1WrkXbeOFVqXXMI87OzZNADt7UyWxnvIw94Wfg9h6FBWkq5ezVBa379Rdzf7PkaO9J6uEKGn8PrrzAPFozFVooMW8Sp+CbQypbs7PFUsa9YwJzFsWO9NrLiYIannnuPmf9ZZNEJREcKrreXglQ8+YJnTiBHAmWey+y3AF2132DGyIPzVBNdcw7zAli3uHhdjKL7Y1MS+hAl+Rlnl57MUurU19Ol0SuKjBkHxizVt0ZdemlVjNm7cwJ6ru5shkJISzyfatDSW7T//PPs6olLuumwZ8OST/P+QIVzEJ5/Qch11FDPYAZTq2NvsmDI0/LNVc3I4p3rlSk6drK7mKX/iRDYkP/20fx0tywPUWsHBTUgGQUSGGmP2hXsxSnwyZAhlJz76iJPPPHHwIHMMnoYDBcO2bQxR+cpXZGQwTLR+PTBt2sCezy8bN1I8qa+7kpnJ3XPjRuCJJ4Dvfc9vrMXuCH8OwSI9neGeE09kktlqzgOAww5jh7cvm9XSQk9CS1AHN6Hqar4e1lUocc9XvsJQxM6dvatauruZNwA4H2ag8ee6usAew2ZzK1BHDGPY0VVY6NkVsRpKVq1i1t0Hzm4nmjqaImYQepKS0tuTmzWLBtSXl1BbSyHXgJR2u7vZmfi3v3EM51//SlXXQMqdlLgm1JCRpp0GGUVFwF13UaDvvffcl3d303u49NLgROq8kZYWWNgiKp3ItbUceuGri9DS01mxwj3jwAP1bfUAEPakciBUVXFmzJtv8qX07FuwxrdWVQXYWFhfDzz4IF25jAwayq1bGVYbMYLy1D3nICsJRagG4S9hXYWSEBQVAd/8Jjf/3bu5mZSXM5HpcDAhmZU1MC9h7FieUi0RS09YBsNfonTANDVxMf5eUGam30ledgelqKPhIfRFhB5eTg5zDNZ7293N9/LYY5mY9pucb29nd+K+fe7EUk/27QN++1vWBGvsKSEJ1SDMDeciRORSAD8HcASA440xK8P5+Ep4yc93awqtWMFyxX37uPFUVLAkdPr0/nOnA33sk05y6/N4Yv9+GoPhwwf2Ovxi5Qn80dHhuWOvB/Y2l0GIgYcA0ABccgnLdD/6iKG57Gz2bngaE+2R1atZiuQtUWQNBn//fcaplIQjVIPwOoBjw7iODQC+COCxMD6mEkEcDuCPf2TouKSEB0ZrGuRjj9FQfP/7oYV1LruM+8727TQwVvi+s5P5iqIi4NvfjkK9/LBhfHHeRjZadHRQZMkHsfQQepKXN4Dms/nz/c92HTKE3YLBGASnk55YMo0KTFDiIodgDdoR7YhJGP71L9a9jx7t3phFmHjOz2d38X/+w0lpwZKTA9x6K4e9LFjAUL41UOyss9iQ1nN0gK8u3QFhs3F4w2OP8TjtKYa1bx9dmfHjfT5UrD2EsLB3r/+B1bm5rHv1N9O5uZmDyd94gw0TNhuN6plnDrx2WQmZhMshiMi1AK4FgJGJKBmaBNjt7NEaPtzzRizC65Ys4XhMP9EUj2Rn875z5jA8bwwPn1ac+8ABzrt/6y3uLYWF3EtOOsn/ITYoTjqJ7sobb9DSDRnCF9jaSktVWkpXyM/pNl48hAGRkcHTvK9GvK4u/xOMDhxgrsGaYFRVxfutWUMjcdlltPp6QIw6IRkEY8wjwd5HRBYB8FSHcpcx5uUgnvtxAI8DwNSpU7WNJgZYsxB87QupqfyOb948sF6B9PT+vQ8bNrDQpauL+8mQIRwvMHcuk6a33DLwfojPEeHwmkmTaBSsYc15ecCXv+yeeuSHpPAQpk+n9riv5E1tLW/nbTPv6gIeeIBhuJ6J6ZQUxgetVvRhw8IniqUETNQ6lY0xZ0bruZTI4nAEdjtjWJgSTvbvZ+6ioKD3TJmsLEZu6uuB3/yGRqG4mD8DPmiKAJMn86e9nZtaZmZQMW+7w47M1ExkpkZpklAkmDkTWLSI74Gn5FBnJ6874wzvj7Fpk7vO1RNpaXT3XnmFyn3qJUQVla5QgiY/P7DbiYQWLvLF4sX0TjwNGGtqYnn85s0MYw8bxsPs+efTSwnL3hJi80MklE6jTmUlJyE9+WTv8JkxjCPa7cBVV/l2z5Yv91/fWlTEioJDh5jUV6JGIBPTigN4nO6ByGGLyMUA/gSgFMBrIrLWGHNOqI+nRJaJExnK6ejwrjDa3s7v/eHhmwUDYxix8FQmeeiQu2GuuJh5hREjKPr2pz9R0+fKKwxk105aC6uJYvz4qFS3RELpNCbMnMnE8iuvsKogJYUe09ixLP066ijf929o8C9La+luBOqKKmEjEA9hj+vH1/kqBUDIGV5jzIsAXgz1/oFgddsvXMjPMQAceSQTkRMmqGcaDFlZHELzn//Q8++bS+jqopLDVVeFt5vY6aSh6bufdHayzDU9nc/X1cWcggijD3l5wIq5e3DaO09geNtn7juKMAnx9a/zwxBBIqljFHWOPJI/dXUUQcrKcnsL/hgyBPj0U9+3sYZihzhnWgmdQAzCJmPMMb5uICJrwrSeiOB00st9912WNFpVKBs3cgDLrFksj9QB5IFz3nnsOVi4kGHf4mJ35KCzk2Gas88O73OmpjJ039cz2buXz2nJNnd19TZEBa17cfan92BPtUHlnFEQW4+Nq6EBuO8+4Ec/YuI4Qtjb7BieH+lOuihjJWmC4cQTWX7mi0OHKCse1nIxJRACMQgzwnSbmDF3Ljtfq6p6RwfKy3kQeestfq4vuCBmS0w4bDbgyiv5/V682F15NHw4D41Ll7Jhdfp04NRTwzPwXoSaPG+80Vs2aNeu3gagtZX5X4uj1j+DzLQu7HYORVt7nxB2QQFfzF//SqG2QAYUh4DdYcfk8sn+bxgp9u1jne6KFXSfhg2jxZ4yJbqThcaP58/27Z6rldraeNK48MLorUn5HL/BU9cITY+IyDX+bhNrmpvZ3DRypOdQsc3Gz+Vrr/GzqASOCPOH11wD/OQnDM9s28b3saCAnsOCBRTFW7o0PM952mn05Jqa3JdZja4Aw849S1VzmvejrHYDWnLLIeJFkDMvj57Cxx+HZ5EeqHPUxS5ktGwZcOedtKSpqfxD7d8PPPww8Otf87VHC5uNUuGVlZS5sFxKh4NSurW1lM2NuFCV4omBZtP+X1hWEUEsVV5fBz8rQWqVmCvBYQyHtG/fTi+suJjvaVYWT/Ll5W6F5IFSVgbcfDMNfU0N95GcHHolDQ1cy4knuj2GvKY9MGJDtxGI+Mhp2GxMNkeAaEpf92PLFuAvf+EbN2IE/yhWjK+qiu7VQw/5n6ATTgoKeEq4/nrmcOrr+QU85xzgnnuAk0+O3lqUXgRSZfSRt6sABCqLFTOamwOfAtXSEtm1JCvbttGYjhrlOa+YkcE94KWXfOduOzq4N9hsDB97y+kccQQPtu++y6qjrCwa/aOO4p7naXSBVXUUiuDeQIml9DVeeYXJWW+j5SormeTdupWTdKJFejqlKvxoQCnRJZCAaTmAcwDY+1wuAN7rf/P4IhgV3qjM5k1Cli/n99tXkUlxMfedgwf7l5Y3NbHfaeFCVhEZw6jG7NkMEXkKcQ8ZwpzPBRfw9g88AKxb198DaMqvhLO9G5LZjXHjfDjE3d3em6UGSMxkK+x2umX+5jlkZNC6RtMgKHFJIAbhVQC5xpi1fa8QkcXhXlC4OfJInjSdTu9ho85OXnfEEdFdW7Jw6JD/2caWOF1zc2+DUFcH3HsvQ9oVFe4NvaUFeOopKi7fdJPvxxcBrruOGnRr19KA5OTwb76jqQxjiyfjjBGfIC/PywSfxkZaoAh9AGImW9HcHPg8h4MHo7MmJa4JJKn8TWPMMi/XXRn+JYWX/Hz20uzc6Tl0ZAyvO+ssnekRKgUF/iUqjGFYp6cXZgzD23Y7D+c9T/c5OUxYb9rEKZb+yMqi4fjpT5lDKC3l/a+7Dpjz9JUoGJLKhGXfD0F9PRfwrW9FtMIIiIGHkJXF1+svZtrRwT+iMugZFNIVl13Gk+jKlYxNW+XNdXVMRM6YAVx8cWzXmMhMn87SU180NDBy0VM9edcud+7BEyIMcb/1FkNDVp+BN0TYMDt2bN9rhjKJ+eST7qYoS3Jh6FBajXC2VPchZh7CkCG0igcP+q7pb2sLcH6mkuwEklRebYzxOQwnkNvEkvR04IYbOClq3jzmzwBWtn3zm0xGalNa6EyYwM1+3z7Pc5WdToaVrr66d/Ri0yb+7iuikZbG+3/2We/egqCpqGDp5a5ddAm7u1n+ZM3sjCAx8xBE2FL++98zsewpo27F6jReqiAwD+EIH5VGAJPLce9vpqZyduyxx3Iv8LcRKYFjswE33siG3x07GK7JyeH7XFvL0tBLLuF73xOHIzBDLMI8z4ARYalRz662KBBT6espU4DLLwf++1/mCkpL+aa3tLgz/D/4gZ6IFACBGYRAfGlP7T5xi07qCz8lJYzfr1hBL8wq6Z86lXpRhx3W3wCXlvL07w9jEjvEHVPpaxHqjEyYwFKuDz7gG1pUxDkPJ56omkHK5/g1CMaYXt06IvILUMxuLYC1xhg/SlXKYCE3l5v/GWe4B2v5Mr6TJ9Nz6+z03h/Q1ETDMWZMZNYcDeJC+nrcOP5ce627U1NdZKUPQZ+VjTE/A/AggCYAXxKRmI3THCwcOEBp5yVLmAcJS/gkgohwg/fnieXmMllcU+NZUqK9nSGnL385sb26uJK+ttn8j7hUBi0BVxmJyAMAbjZkP4A3XD9KhGhoAP75T9bi96wczM11T29M9O/1+eezyOX11+lR5OfztTY28rV961sMOyUySSV9rSQ1wZSdNgN4RUQuN8a0iMjZAO42xpwUobUNapqb2bBVW8scaM8TssNBbaC2tvBLTEcbm81t3JYtY1WozcbXNWNGciggJ6X09QCpr2eH+3vv8XM8dChDjZMmRawdRAmAgN96Y8xPRORKAItFpB1AC4DbI7ayQc6CBcCePZ7VFLKyqND63HMcDZkMm2ZFBXDppbFeRWSwO+w4qszPJLFBxEcfUWi1o4OSJqmpFEa8/35WAf/gB4GPaVXCS8CRWRE5A8C3QUNQCuBGY8w7kVrYYKajg7o+vmYIpKcztLJiRfTWpYRGXCSV44TqaupO5eezITEvjweckhL20NXUAA8+GF3xVcVNMKm6uwD81BgzE8AlAP4tIqdHZFWDnIMHaRT8jZ/My1PJ7ninq7sLje2N8ZNUjjHz5jGn7a3rvLKSjaPWmFslugRsEIwxp1uaRsaY9QBmA/hVpBamKMnA59LX6iGgpYVtED3lS/oiwv65cA1UUoIj5PSNMWavK4ykhJmSEnoH7e2+vYSmJlUciHdi2qUcZzQ3u1VvfZGdzWIKxQN791LS3OFgAmby5LA2Fg4on2+McYRrIYqb9HSqr778sneJ/o4OVuPMiOtp1krMdIzikIwM5gaM8V0u3dGhzdP9aGigOOO6dfw9JcXdYHjeeWzoCYP8iBZ4xSlnnUV11l27OA+9Z9lpayuF5L72Ncr4K/GLeghuCguB8eN5yB0yxPvtmpuBk7SY3U1LC4XC9u/vP5awsxOYO5cewxVXDLgxKYH7P5Ob3FzgttuAY45h5cWOHe4fh4MKBKdrSj/uUQ+hN1/4Ag+73jSs6uoYCZkyJbrrimuWLKFCb2Vl/w0/LY1hhAULgN27B/xU6iHEMfn5lO0+eJCVF52d7DmYMCE2s4GV4FEPoTeTJrER8b//ZaVRSQm93/Z2HoCzs4Gbb/ZfYTdo6OoC5s/3rCtvkZLC0NGSJcBVVw3o6dQgJAAlJf3nECuJgXoIvbHEV8eO5T63bp17rPP553O6YXFxrFcZRzQ2Mobmr/u0sBD45JMBP50aBEWJIPY2OzJSMpCVluX/xoMEEVbHHXEEPYPOTjan6UgGD1iT/fxhTFgUINUgKEoEsTsGqHTa2ckyw/p6xgnHjfNdyJ9gZGRoeMgn+fkMDzQ1sRPVG/X1YcnEq0FQlAhib7OjOCuEGIgxwDvvAP/+N6tMetZqHnssS8wSeWqQEhg2GzBnDktOc3M9VxF1dPDzcfLJA3+6AT+CoiheqXPUhZY/WLgQ+MtfuAlUVVHop6qKw6vXr6cUbnNzuJerxCMnnsjB79XV/YehtLSwAumyy8LiOapBUJQIEtJwHLudUrYjRjC43hObjeWHe/aw1FBJftLTObT8nHPYwl1d7f5xOoHrrgubDn5chIxE5D4AXwDQAeAzANcYY+pjuihFCQMhSV8vX85/09O936aigjOSzz/f9+2U5CAjg41nF1zAGvSODuYXxo0LazY+XjyEhQAmGWMmA/gEwB0xXo+ihIWQpK83b/adQATcYlcHD4a+OCXxyMlh1960aWxICnNpVlwYBGPMAmOM1bu4AoCOl1ISngFJXwdSaqgoYSYuDEIfvgFgnrcrReRaEVkpIisPHDgQxWUpSnCELH19xBEsM/SFJYWrHYtKGImaQRCRRSKywcPPhT1ucxcAJ4BnvD2OMeZxY8xUY8zU0tLSaCxdUUIiZNmKGTOYPO7o8H6bPXuogKj5AyWMRC2pbIw509f1IvI1AOcDOMMY9ZeVxCdk2YrCQuDKK4G//53J4+xs93Xd3RQxGzEibJUlimIRL1VG5wK4DcBpxpjWWK9HUcLBgITtTj+dIaH//hc4cKB3Y9q0acDVV3ufQ6koIRIXBgHAQwAyACwUfuhXGGOui+2SFGVgDEjYToSdpyecwKqj+noqWo4f73uYgKIMgLgwCMaYcbFeg6KEm7BIX6emUjNaUaJAPFYZKUpSoNLXSqKhBkFRIoRKXyuJhhoERYkQA5a+VpQoExc5BEWJZ9ra3IU+paX99ea8EZJshaLEEDUIiuKF5mZg3jxqyFmqw6mpwKxZwOzZbBfwRUhKp4oSQ9QgKIoHGhs5cmD3bvaGWVO9OjupOr16NXDHHb7n/9oddgzLGxadBStKGNAcgqJ44JlngH37OJOm54jHtDRg1CigoQF44gnfj6EegpJoqEFQlD7U1QEffsg5NN6oqOCo4z17vN/G7tAcgpJYqEFQlD589hkTyDYf3w4R3mbrVs/Xd3V3oaG9QQ2CklBoDkFR+tB3bK0vvAmSNrQ3ABhgl3K8sW8fsHIlh/IUFADHHssZz54GvysJiRoEJSa0t3Mzzcpi5U484StR3Bdv4wiSqku5rY3KqytW0G1KT6fVfPllzm647joaCCXhibOvopLsbN4MzJ8PrF3Lg2V6OnDGGSzljJdZL+PHA0VFLDvNzfV8G4eDqtRHHun5+rDoGMUDXV3Ao48C69Yxm97TGzAG+PRT4He/A+68M/AGDSVu0RyCEjVefx34v/8DtmyhnP/IkTyNv/EGcPfdQHV1rFdIUlI4jqC2lofjvnR0MJl8+eXe59MkjYewaROtd19jAPD34cOBmhp6D0rCowZBiQobNwLPPcf9o6zMnbBNT6dxSE0F7r/f8wYcC6ZOBb7zHeDQIWDHDobNDx2i0dq/H/ja14BTTvF+/6TxEBYupJvkK09QWsoOPp1rlfBoyEiJCq+9BuTns47fE0VF3HjXruUIgHjgpJOAyZNZgvrxx9zvJkwApk/3HzJPGg9hxw7+4XyRm8vbdXT0btpQEg41CErEaWpi5GHkSN+3y88H3nknfgwCAOTlcXjZ6acHdz/LQyjOCiJDHY+kpnJspy+saW4pKdFZkxIxNGSkRJy2NoaI/FUnpqUBLS3RWVOkqXPUJYf09bHHMlbmi4MHmV2Pt3IxJWjUICgRx6rU6eryfTuHg+HoZCBppK9PO41/OG/NGd3ddAHPPTe661IighoEJeJkZTEMtH+/79u1tnL/SQaSRvp6+HDgsstYSdTY2Dtx3NICbN8OnHmmjvlMEtTHU6LC7NmsTPRW2797NzBmDPuckoGkErY75xxgyBBg7lwaBpuNnkFBAXDNNcDMmdqtnCSoQVCiwvDhwA9/CPzpTwxJFxcz5NzSQiNRVQXcdFPy5CWTSvpaBJg2jbW4NTX8o2VmsjchWf5gCgA1CEoUmTiRMwbefx9Ytow5g9Gj2ak8caL3ktRExN5mx8SyibFeRngRoRFQkhY1CEpUKSgAzj6bP8mMSl8riYgmlRUlzKj0tZKoqEFQlDCTlNLXyqBADYKihJmkka1QBh1qEBQlzCSNsJ0y6FCDoChhRj0EJVFRg6AoYUY9BCVRUYOgKGFGPQQlUVGDoChhRj0EJVGJC4MgIr8UkY9EZK2ILBCRJOn5VwYjdocd6SnpyEpNcOlrZdARFwYBwH3GmMnGmKMBvArgZzFej6KEjKV0Kir4piQYcWEQjDGNPX7NAaDDWZWEJamUTpVBRdxoGYnIPQC+CqABwCwft7sWwLUAMNLfTEZFiQGqY6QkKlHzEERkkYhs8PBzIQAYY+4yxowA8AyA73l7HGPM48aYqcaYqaXJMl5LSSrUQ1ASlah5CMaYMwO86b8AvAbg7gguR1Eiht1hxxElSTLpRxlUxEUOQUTG9/j1AgCbY7UWRRkoSTM+Uxl0xEsO4TciMgFAN4BqANfFeD2KEhLdphsNbQ0aMlISkrgwCMaYL8V6DYoSDhraGmBg1ENQEpK4CBkpSrKgXcpKIqMGQVHCiOoYKYmMGgRFCSPqISiJjBiTuE3BInIATEJHihIAByP4+JEkUdeu644+ibr2RF03EPu1jzLG9GvkSmiDEGlEZKUxZmqs1xEKibp2XXf0SdS1J+q6gfhdu4aMFEVRFABqEBRFURQXahB883isFzAAEnXtuu7ok6hrT9R1A3G6ds0hKIqiKADUQ1AURVFcqEFQFEVRAKhB8EuiznsWkftEZLNr7S+KSGGs1xQoInKpiGwUkW4RibvSvL6IyLkiskVEtorI7bFeT6CIyJMiUisiG2K9lmAQkREi8raIbHJ9Tm6K9ZoCQUQyReQDEVnnWvf/i/Wa+qI5BD+ISL414lNEbgRwpDEm7tVYReRsAG8ZY5wici8AGGNui/GyAkJEjgCVbx8DcIsxZmWMl+QVEUkB8AmAswDsAvAhgCuMMR/HdGEBICKnAmgG8E9jzKRYrydQRKQCQIUxZrWI5AFYBeCieH/PhUO2c4wxzSKSBmAZgJuMMStivLTPUQ/BD4k679kYs8AY43T9ugLA8FiuJxiMMZuMMVtivY4AOR7AVmPMNmNMB4DnAFwY4zUFhDFmKYC6WK8jWIwxe40xq13/bwKwCUBlbFflH0OaXb+muX7iaj9RgxAAInKPiOwEcBWAn8V6PSHwDQDzYr2IJKUSwM4ev+9CAmxOyYKIVAE4BsD7MV5KQIhIioisBVALYKExJq7WrQYB4Zv3HG38rdt1m7sAOMG1xw2BrD1BEA+XxdWpL1kRkVwALwD4QR9PPm4xxnQZY44GPfbjRSSuQnVxMSAn1iTqvGd/6xaRrwE4H8AZJs6SRUG85/HOLgAjevw+HMCeGK1l0OCKwb8A4BljzNxYrydYjDH1IrIYwLkA4iaprx6CHxJ13rOInAvgNgAXGGNaY72eJOZDAONFZLSIpAO4HMArMV5TUuNKzj4BYJMx5g+xXk+giEipVe0nIlkAzkSc7SdaZeQHEXkBQK95z8aY3bFdlX9EZCuADACHXBetSITqKAAQkYsB/AlAKYB6AGuNMefEdFE+EJE5AB4AkALgSWPMPbFdUWCIyLMAZoJSzPsB3G2MeSKmiwoAETkZwDsA1oPfSwC40xjzeuxW5R8RmQzgH+DnxAbgP8aYX8R2Vb1Rg6AoiqIA0JCRoiiK4kINgqIoigJADYKiKIriQg2CoiiKAkANgqIoiuJCDYKiKIoCQA2CkiSISJWIOFw6MdZl/WSpRSTLJWXeISIlA3zOLBFZ4lI8hYjc6JJkDkomREQKReT6gawlgOfoJ3UtIukislREVLFAAaAGQUkuPnPpxFiy1A8DmA3gSABXiMiRxhiH6zbhkJf4BoC5xpgu1+/XA5hjjLkqyMcpdN03KIQE+h3+OyiT8DkuddY3AVwW7HMryYkaBCVhcA1FOcv1/1+JyIM+bh4NWeqrALzsWs+fAYwB8IqI3CwiX3ENQ1krIo/18CJeEpFVrgEp17oe5zcAxrpue5/L2+l5kr9FRH7u+n+Vywt5BMBqACO8PVdPfEhdv+R6HYqiBkFJKO4GcJeIXAVKHt/s47YRlaV26RaNMcbsAACXLMgeALMAvAGeuk9yeSNdcG+63zDGHAdgKoAbRWQIgNvh8m6MMT8O4OkngENtjgGQ7eO5AmEDgGlB3F5JYjR2qCQMxpilLmGzHwKYaYVqROSXoNhZTyItS10C6ix54gwAxwH4kMtFFqh/D9AIXOz6/wgA4wHsC/K5q3tM2fL1XH4xxnS58il5rmEzyiBGDYKSMIjIUQAqABy0Ni8RGQrPn+OgZalF5AYA33b9OgfAxT1/N8b0vL8DQKa3hwLwD2PMHX0efyaocDnDGNPqkj/29BhO9Pbe+96mxd9zBUkGgLYB3F9JEjRkpCQEwjm6z4B5gBYRsdRPjwGw1sNdgpalNsY87ArbHG2M2dP39z63tQNIERFPG/qbAC4RkTLX2otFZBSAAgB2lzE4HMAJrts3Acjrcf/9AMpEZIiIZIAzLbzh7bkCwhWyOmCM6Qz0PkryogZBiXtEJBvAXAA/MsZsAvBLAD93XX00PBgE1zzp7wGYD87c/Y8xZmOYl7YAwMkenvtjAD8BsEBEPgKwEPRs3gCQ6rrsl+CsaxhjDgF41zUx7j7X5vwLcCzkq/Chme/juXrhkrpeDmCCiOwSkW+6rpoFIK5lo5XoofLXSkIjIk+AYZ2RAF41xgQ0klBEdgCYaow5OIDnPgbAD40xV4f6GLFGROYCuMMYsyXWa1Fij3oISkJjjPmmMaYbrK4p6NmY5gmrMQ1AGtzDVUJ97jUA3vZU5pkIuEJpL6kxUCzUQ1AURVEAqIegKIqiuFCDoCiKogBQg6AoiqK4UIOgKIqiAFCDoCiKorhQg6AoiqIAUIOgKIqiuFCDoCiKogAA/j/oQ7KLrYAZXwAAAABJRU5ErkJggg==",
|
|
"text/plain": [
|
|
"<Figure size 432x288 with 1 Axes>"
|
|
]
|
|
},
|
|
"metadata": {
|
|
"needs_background": "light"
|
|
},
|
|
"output_type": "display_data"
|
|
}
|
|
],
|
|
"source": [
|
|
"plot_dataset(train_x,train_labels,W.numpy(),b.numpy())"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Let's see how our model behaves on the validation data."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 23,
|
|
"metadata": {
|
|
"colab": {
|
|
"base_uri": "https://localhost:8080/",
|
|
"height": 282
|
|
},
|
|
"id": "oEQswfCGrmHw",
|
|
"outputId": "3cf61882-60e1-4baa-8e51-0c31ea80875c"
|
|
},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"<matplotlib.collections.PathCollection at 0x25c717e8970>"
|
|
]
|
|
},
|
|
"execution_count": 23,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
},
|
|
{
|
|
"data": {
|
|
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAXwAAAD4CAYAAADvsV2wAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAAixElEQVR4nO3deZxcVZn/8c9TVd3VSzohG4QQlgAJENSItMEIGKIsAREEhQmOiqBmQED05/yQRXCE0UHRcWMzI7LMT9lGkQiBsGpABdJggISwRJZJSEyakK3X6qp6fn9UA013ddJJ3apbVff7fr361V3n3r7nCTz15ObUueeYuyMiItUvFnYAIiJSGir4IiIRoYIvIhIRKvgiIhGhgi8iEhGJsAPYkjFjxvgee+wRdhhSpZ588sk33H1sqftVXksxbSmvy7rg77HHHrS0tIQdhlQpM3stjH6V11JMW8prDemIiESECr6ISESo4IuIRIQKvohIRKjgAx2bO3n2kWWseOH1sEMRCdTql9fw7CPL2Ly+LexQpAyU9SydUrj9R/O4/qKbcRzPOhP22YUf3H8xo8aNDDs0ke3WtqGdi4+7nGWPvYTFDM86J557DF/+wecws7DDk5BE+g5/0YLF/PL8X9OTSpNOZciks7y2dAXnHvytsEMTKcilJ/2IJY8+TyadIZ1Kk0lnuP1Hf+DOK+8NOzQJUaQL/vUX3Uw2kx3Q/o9X1vLS314OISKRwm16czOLH1qS99gNl9xS4miknES64K9+Zc2gx1oWLC5dICIBatvQzmD7XLRv7ChxNFJOIl3wd9p98Kfqx+81roSRiARnx93GDHosnoj0Wz7yIv1//7TLZkOez69qkjUc9PEDSx+QSAASiQT7H7xv3mOHfOpDJY5GykmkC/5BHz+QT339WOKJGGZGLB6jpq6Gy+Z9k7qGZNjhiWy3S39/HuMm7tib25CoibPbfhP42jVzwg5NQhT5aZln/PBUjv/KLFoWLKausY4PH99M44jGsMMSKcjw0U3c+NLPeeqBZ1n5wip2mzKB98/cn1gs0vd4kRf5gg+w85478Ykzjwo7DJFAxWIxmo+cSvORU8MORcqE/roXEYkIFXwRkYhQwRcRiQgVfBGRiFDBFxGJCBV8EZGICKTgm9mvzGytmeVdscnMDjOzjWa2uPfrkiD6FSkm5bVUm6Dm4d8AXAnctIVzHnH3YwPqT6QUbkB5LVUkkDt8d18IvBnEtUTKhfJaqk0px/Cnm9nTZnaPme0/2ElmNsfMWsyspbW1tYThiWwX5bVUjFIV/KeA3d19KvBz4PeDnejuc9292d2bx44dfPlikTKgvJaKUpKC7+6b3L2t9+f5QI2ZDb5ot0gFUF5LpSlJwTezcda7c7KZTevtd10p+hYpFuW1VJpAZumY2c3AYcAYM1sJfBuoAXD3a4FPA2eaWRroBGb7YHuwiZQJ5bVUm0AKvrufspXjV5Kb3iZSMZTXUm30pK2ISERUTcHf9OZmli9+hfaN7WGHIhIYd2flS6t5dekKstls2OFIhav4Ha/SPWl+cuZcHvr1o9QkE6RTaY47axZf/v5ntZ2bVLT/ff51/u3EK1j7WisWMxqGN3Dhb85l6oxBp/uLbFHFV8TrLvwNf7z5z/R099CxqZNUVw9/uOY+7vjZ/LBDE9luPakevnHYJax84XW6O1N0tXfz5ur1fOvY/2Dd6vVhhycVqqILfjab5a5r76O7M/Wu9u6Obm7/0R9CikqkcE/M/xupzh76z/nJZrLcf9MfQ4lJKl9FF/x0T4ZUv2L/ls3rNpc4GpHgvLl6PZl0ZkB7qquHtSs01V+2T0UX/NpkDeP3Hpf32D4f3LvE0YgE5z2H7As2sL1+WB0HzHxP6QOSqlDRBR/gnCu/RLKhlt4HHonFjLrGJGf856khRyay/Sa+d3c+dGwzdY3Jt9tq62uZsM94ph/XHGJkUskqfpbOBw5/H//5p0v5zfd+x2vPrWTygXvxmQtPYPcpu4YdmkhBLvj1V1lw/R+5+xf30ZNK89HPHMonzzmaRE3Fv20lJFbOT4I3Nzd7S0tL2GFIlTKzJ9295LfLymsppi3ldcUP6YiIyNCo4IuIRIQKvohIRKjgi4hEhAq+iEhEBFLwzexXZrbWzJYMctzM7GdmttzMnjGzDwTRr0gxKa+l2gR1h38DMGsLx48GJvV+zQGuCahfkWK6AeW1VJFACr67LwTe3MIpxwM3ec5jwA5mtnMQfYsUi/Jaqk2pxvB3AVb0eb2yt02kkimvpaKUquDnWQaKvI/4mtkcM2sxs5bW1tYihyVSEOW1VJRSFfyVQN/FbSYAq/Kd6O5z3b3Z3ZvHjh1bkuBEtpPyWipKqQr+PODzvbMaPgRsdPfVxe501d//wV//0MKKF14vdlcSTaHkdSaT4ZmFz/H4/Kdo39RR7O6kigSy7J6Z3QwcBowxs5XAt4EaAHe/FpgPHAMsBzqA04LodzCp7h6+O/vHtCxYTKI2Qbonw3sP3Y9/+93/pa4hufULiFB+eQ3w96df5YKjv0tXexdmRronw1k/PY1jvnR4sbuWKhBIwXf3U7Zy3IGzguhrKG769q203Pc0qa4eUl09ADy78Dmu/caNfO2aOaUKQypcueV1Jp3h/CMvY0Prpne1X/2165ncvBd7v39iqUKRClWVT9re/V8PDNj6MNXVw/03/pFyXg5aZEv+9tASUt09A9p7utPcPff+ECKSSlOVBb+7oztve093mmw2W+JoRILRvrEj7xygbCbLpnVtpQ9IKk5VFvz3fWQKlmfC3L4H7U08Hi99QCIBeN+MKaR70gPa6xqTHHLCQSFEJJWmKgv+V356Og3DG6hJ1gCQqE1QP6yOr1795ZAjE9l+I3ccwWcvOYlkQ/LtG5q6xiR7vX8ih35KBV+2rio3x9xt31247rmfMO+qe3lh0XL2nLoHnzx7FjvupvnPUtlOOf8EpkyfzN1zH6B9YwczTprOzFMO1j63MiRVmyWjdx7Jaf++xUkWIhVp6oz9mTpj/7DDkApUlUM6IiIykAq+iEhEqOCLiESECr6ISESo4IuIRIQKvohIRKjgi4hEhAq+iEhEqOCLiESECr6ISEQEUvDNbJaZvWBmy83s/DzHDzOzjWa2uPfrkiD6FSk25bZUk4LX0jGzOHAVcAS5TZ0Xmdk8d3+u36mPuPuxhfYnUirKbak2QdzhTwOWu/vL7p4CbgGOD+C6ImFTbktVCaLg7wKs6PN6ZW9bf9PN7Gkzu8fMBl3qz8zmmFmLmbW0trYGEJ7Idgsst5XXUg6CKPh59pYasBHbU8Du7j4V+Dnw+8Eu5u5z3b3Z3ZvHjtX69RKqwHJbeS3lIIiCvxLYtc/rCcCqvie4+yZ3b+v9eT5QY2ZjAuhbpJiU21JVgij4i4BJZjbRzGqB2cC8vieY2Tiz3KZsZjatt991AfQtUkzKbakqBc/Scfe0mZ0NLADiwK/cfamZndF7/Frg08CZZpYGOoHZ7t7/n8ah6En18Luf3M09v3yQdDrDzNkHc8oFJ9LQVB92aBKySs/tlS+t5qZv38qzjyxj9PiRnHLBiRz8yWlhhyUhsjLJzbyam5u9paWlaNd3dy6Y9e8sefR5ujtTANQka9hl0jiuefIH2ie0ypnZk+7eXOp+i53XAKv+/g/OPPA8utq6yGZz7/G6hiSnfe8UTvzqx4vat4RrS3kd6Sdtn39iOUv/8sLbxR6gp7uHNa+28pc7F4UYmUhh/vvS2+lq73672AN0dXRzw7duIdXdE2JkEqZoF/zHXyKTzg5o72zrYsmfnw8hIpFgLHn0ebKZgbmNweqX15Q+ICkLkS74Y3cdTaI2PqA9WV/LuIk7hhCRSDB22j3/1M90T4aRO40ocTRSLiJd8A/6+AeoH1aPxd493TpeE+fwf/5ISFGJFO6UC04g2ZB8V1ttXQ0HH/9Bho9qCikqCVukC35NbQ0/eeQyJn1gIjXJGmrrapiwz3iuePDbDB+tN4VUrgOPmMo5V36RYSMbqWtMUpOs4ZATD+Ib130l7NAkRJGfhrLznjtx1RPfZ/2aDaR7MoydMDrskEQCcdQXZnL4Zz/C2hVvMHx0E43DG8IOSUIW+YL/lpE77RB2CCKBiyfi7Dxxp7DDkDIR6SEdEZEoUcEXEYkIFXwRkYhQwRcRiQgVfBGRiFDBFxGJCE3LLKHH7nqSO6+8h80b2jn0Uwdx3JlHUT9MyzBLZfPsZrzjv6HrQYiNxBpPxZKHhh2W5KGCXyLXX3ILv/vxXXS1dwPwyrP/y303/ImrFl1OXb9H4EUqhWfb8HWfhMxaIJfbnlqEDzuL2LA5ocYmA2lIpwTWr9nA7T+c93axB0h1pljzWiv33/SnECMTKYx33AqZVt4q9jmd0PZzPLsprLBkEIEUfDObZWYvmNlyMzs/z3Ezs5/1Hn/GzD4QRL+V4rm/vkhN7cB/THV3dPPE/KdCiEiGSrm9Fd0PAV0D260Gep4teTiyZQUXfDOLA1cBRwNTgFPMbEq/044GJvV+zQGuKbTfvtwd73kW73oYz7QGeelAjBg7HM8O3FksFo8xaueRIUQkQ1EOud3Z3sWie//G4oeXkO5JB3npYMR3BCzPgQzElNvlJogx/GnAcnd/GcDMbgGOB57rc87xwE29e30+ZmY7mNnO7r660M49swZ/8zTIrgJi4Cm84bNY0zfp3Vs6dFOmT2b4mCa6OrrfVfhrkgmO+8pRIUYmWxFqbv/ptr/wwy9eTSyeuy+L18S57M7z2f/D+xR66cBYw+fxrgd5911+DGK7QGK/sMKSQQQxpLMLsKLP65W9bdt6DgBmNsfMWsyspbV163frvuEcyLwC3gHeBqSg42bounub/hDFFIvFuOKBbzNh8njqGpM0DK+noamer889g72m7hF2eDK4wHJ7W/N69ctr+MFpV9HV3k3Hpk46NnWyeV0bFx7zXTrb8wyhhMRqD4DhF4PVgw0D6iExCRt1XdnccMk7grjDz/d/tf/4xVDOyTW6zwXmQm6z5y117JnV0LMMyPQ70ol33ITVH7ulXy+pnffcieuW/pjXnltJx6YO9v7AntQmawK5tmc3Qs8zEBsFiSl6owUnsNzelrwGuO+mP5JN98/r3PDl43c9yWH/dPDWLlEysYaT8PpPQM9zEBuBJfYK7NqeXg6ZVZDYD4vn38VLhi6Igr8S2LXP6wnAqu04Z9tl28Di+f/qKMMZAmbGHvvvuvUTt0G2bS60/Tz3IZlnIb4zjLoOi48PtJ+ICi23N7/ZTrpnYMHPZrK0b+wo9PKBM6uD2uA+r/bsRnz9nNwNndWAd+MN/4Q1fUs3NAUIYkhnETDJzCaaWS0wG5jX75x5wOd7ZzR8CNgYxBgniYlAbZ4DtVB3RMGXL3fe/Qi0XQV09w5ndUDmFXz9v4QdWrUILbcPOuYA6obVDWj3rHPAx95b6OXLnm84D3qWAF3gm8kN1f4P3nFb2KFVtIILvrungbOBBcAy4DZ3X2pmZ5jZGb2nzQdeBpYD/wUEss+aWQKGfxeoA97ajLwOYmOwxi8G0UVZ8/Ybgc5+rVlIv5b7p7AUJMzcPvDIqbzvI/tR1/jOQ3l1jUk+8ZWjGL/XuCC6KFue3QSpR4Gefkc6oePGMEKqGoE8aevu88klft+2a/v87MBZQfTVX6z+CDxxO97x/yDzOtQegjWcjMWGFaO78pJdn7/dEmU5pFWJwsrtWCzGpXd+k4W3P8ZDNz9CbbKGWV/8GM1HTg26q/Lj7Qx6L+obSxpKtamKpRWsZh9sxGVhh1F6dYdD24u8+ylHgCzU9J8uXhj3Hsi8BjYSi2vf31KIx+PMnH0wM2eXzwe0JREbB7EdILum34E4JA8LvDvProfsOojvRm7krnppaYUKZg2fg/g4ckNakJswUgdN38p9iBaQbMed+Nrp+LpP460zyL75JT02L0VjZtiI7/HuodpasBHYsHMC68e9i+yGr+FrD83l9tqDyLbfFNj1y1FV3OFHlcWGweg7ch9kdT8M8R2xhs9jtcH9s99Ti2DTxbzrwZrUX/EN52CjNJ4qxWHJQ2H0b/GOGyD9GtQehDX+MxYbFVgfvvHC3AqfpMBTucbNP8Lj47G6wwPrp5yo4Fc4iw3Dhp0Ow04vyvW9/ToGrpXSA6mn8MzrWDzv83MiBbOaSdiI7xbl2p7dDF33Aal+Rzrx9murtuBX5ZCOZzfi6ZW4Z8MOpfJlBplSbjWQeaO0sUScu7PmtVY2vqHhtIJl1/POcFE/mf6fHVSPqrrD9+zG3Pzd1J/JrecxDIb/O1b30bBDq1y1H4b03xkwRc7TEOATlbJlix9ewg++cCWb3thMNptlyvR9uPA35zJqnBYo2y7x8b0PdPWf1hyD2g+GElIpVNUdvq8/s3f+bgroguwb+Iav4T3Pbe1XZRDWeHrvGil97w3qYdjZ0Zj6WgZWv7yGiz9xOa0r1tHdmaKnO82SR5/nvCMuJTcrVLaVWQKaLuCdCQ8AMbAGbNi5YYVVdFVT8D39au+Tef0f1kjh7b8KIaLqYPEdsTF3Qv1JEN8Vag7AdrhCuxmV0J1X3ztgaeRMOsPa195g2eMvhRRV5Ys1fAobeTXUTIP4BKg7Dht9B5bYPezQiqaihnQ88w+849e59TVq3oM1fAaL75g7mFnd+0+0/h8wZiGzYsC1ZOgsPg4b8Z2ww6hqT/9xKfN/+SBd7V3MOPnDzDhpOvFEbox59fI1edfVMTNaV6yDD5U62uphyUOw5CFhh1EyFVPwvedF/M3ZvdOnUpB6LLdx8uhbscTeULPvO1Or3qUWavWOkPJ103du47Yr5pHq7MYdnnrgGRbc8DDfm38h8Xic9390f5584Gm6O96d3+meNJOb9wwpaqlEFTOk45u+885695D77m34ptwTthYbCQ2fBer7/FYcrBFr+HyJoxUZmtaV67j1+7+nuyNX7AG62rt57q8v8vhdue0vj/zCTIaPbiJR886skmRDksNmH8zOE3cKI2ypUBVR8N0dep7MdwRST7z9yprOw0b8GyT2yT2eXf8pbMydWgpAytbih5cQSwycHtjV1sVf5i0CoHF4A1e3fJ9jzziSsbuOZrf9JjDnis/xjV+eWepwpcJVxJCOmeEkGbgyJNBnCQEzg/oTsPoTShecSAEahzcQy7O+ezwRo2nUO7Ogdhg7grN+ejpn/bQ4D9hJNFTEHT4A9ScCyX6NydzsEZEK1XzU1Lf3rO0rXpNg1mkzQ4hIqlnFFHwbfl7vAxF1vfPCk7n1NZq+EXZoItuttq6W/7j3IoaPaaKhqZ6G4fUk62v56lVfYvcpwe6OJlIRQzoAZvXYqF/lNvZIvwKJvbCEZihI5dt32iRufX0uzz6yjO6OFO+bMYWGpvqt/6LINiqo4JvZKOBWYA/gVeBkdx+wK4eZvQpsJrfbeNrdm7e7z8TekNh7e39dZEhKnduJmgQHfLT6ty6UcBU6pHM+8KC7TwIe7H09mJnu/v5Cir1ICSm3peoUWvCPB95aFP1G4JMFXk+kXCi3peoUWvB3cvfVAL3fdxzkPAfuM7MnzWyLi7CY2RwzazGzltbW1gLDE9lugea28lrKwVbH8M3sAWBcnkMXbUM/B7v7KjPbEbjfzJ5394X5TnT3ucBcgObmZi0FKMU02cyW5GkPPLeV11IOtlrw3X3QrV/MbI2Z7ezuq81sZ2DtINdY1ft9rZndAUwD8hZ8kRJ6cbBxd+W2VKNCh3TmAaf2/nwqcGf/E8ys0cya3voZOBLId1clUk6U21J1Ci34lwNHmNlLwBG9rzGz8WY2v/ecnYBHzexp4Angbne/t8B+RYpNuS1Vp6B5+O6+DvhYnvZVwDG9P78MTC2kH5FSU25LNaqYpRVERKQwKvgiIhGhgi8iEhEq+CIiEaGCLyISESr4IiIRoYIvIhIRKvgiIhGhgi8iEhEq+CIiEaGCLyISESr4IiIRoYIvIhIRKvhSdN79F7LrPk12zYFk3zgB79b+IFL53FNk264ku/ZQsms+SHbDeXhmTdhhbZEKvhSVdy/E158BPc+Ab4b0Unz92WQ77w87NJGC+IZzoW0uZNeAb4SuP+DrTsSzm8MObVAFFXwzO8nMlppZ1szybhXXe94sM3vBzJab2fmF9CmVxTddDnT1a+2CtsvDCGfIlNuyJZ5eDt1/5t25nYHsZrzzt2GFtVWF3uEvAU5kC3t4mlkcuAo4GpgCnGJmUwrsVypF5pVB2lfgni1tLNtGuS2D61kGFs9zoAtST5U8nKEqqOC7+zJ3f2Erp00Dlrv7y+6eAm4Bji+kX6kgsbH5220UZuU7oqjcli2K7wp4ngO1kNir1NEMWSnecbsAK/q8XtnblpeZzTGzFjNraW1tLXpwUmTDzgKr79dYD8POCCWcgA05t5XXVaZmKsR3Y8AusZbAGv4plJCGYqsF38weMLMleb6Geidjedry/dWYO+A+192b3b157NhB7g6lYlj9yTDs62BNQG3u+7CzsIZTww4NYHKpclt5XV3MDBt1IyRnADVAAhKTsVH/jcXHhR3eoLa6ibm7H15gHyuBXfu8ngCsKvCaUiHMDGv8At7wOfBNYE2YbTXtSuVFdx/0A9khUG5HmMVGYiOvwb0TPI3FmsIOaatKMaSzCJhkZhPNrBaYDcwrQb9SRsziuTdI+RT7ICi3BbP6iij2UPi0zBPMbCUwHbjbzBb0to83s/kA7p4GzgYWAMuA29x9aWFhixSXcluqUUG3W+5+B3BHnvZVwDF9Xs8H5hfSl0gpKbelGpXvvDgREQmUCr6ISERU1SdoMnSe+Qd0LQDvhuRhWM3ksEMSKZi78+wjy3j2kWWM3HEEM06eTuOIxrDDKhsq+BGU7bwbNr617EsG2q7EGz5DbLiWgpHKle5Jc8nx3+fZR5aR6kxRW1/LL/71Jr5//8XsO21S2OGVBQ3pRIxnN8HGC4Du3q800AWdN+NlvAaIyNbcc91DPLNwGV3t3WSzTld7Nx2bO/nOp3+I+6DPekaKCn7UdC/Mv+iTd+GdmkIulWvB9Q/T3dE9oL1tQwevLvnfECIqPyr4kaM7HYka5fxbVPCjJvkR8PTAdqvD6j9R+nhEAnLUaTNJNiQHtA8b0cge79kthIjKjwp+xFhsBIz4DyDZ+xUH6qB+NlZ7YLjBiRTg6C9+lPcesi91jUksZtQ1JmloqueS//lXzPKtcxc9mqUTQbH6Y/HaZui6t8+0zH3CDkukIImaBN+75yKeWfgcSx55npE7aVpmfyr4EWXxcdD4hbDDEAmUmTF1xv5MnbF/2KGUJQ3piIhEhAq+iEhEqOCLiESECr6ISESo4IuIREShO16dZGZLzSxrZoPuDWpmr5rZs2a22MxaCulTpBSU21KNCp2WuQQ4EfjFEM6d6e5vFNifSKkot6XqFLrF4TJAT7FJ1VFuSzUq1Ri+A/eZ2ZNmNmdLJ5rZHDNrMbOW1tbWEoUnst2GlNvKaykHW73DN7MHgHF5Dl3k7ncOsZ+D3X2Vme0I3G9mz7v7wnwnuvtcYC5Ac3OzlrmTYppsZkvytAee28prKQdbLfjufnihnbj7qt7va83sDmAakLfgS2Vwd0g/D56CmimY1YQd0vZ40d0H/UB2KJTb1cezbZB+EWJjsER1rbJZ9LV0zKwRiLn75t6fjwQuLXa/Ujze8yK+/l/A1wMGJGCHH2LJGWGHVlLK7eqTbbsO2n4KlgDvwWv2x0ZejcVGhR1aIAqdlnmCma0EpgN3m9mC3vbxZja/97SdgEfN7GngCeBud7+3kH4lPO4p/M3PQfZ18A7wdvCN+Ppz8MzrYYcXGOV29HjXw9D2M6ALvA3ohp5n8PXnhB1aYAqdpXMHcEee9lXAMb0/vwxMLaQfKSPdC4FUngMZvOO3WNNXSx1RUSi3o8c7rgc6+7Wmc0U/sxqL7xxGWIHSk7aybbLrwLN5DvRAVrNPpIJlBslfS0B2fWljKRIVfNk2tdOAPAXfGrDkISUPRyQwyRnAIJMPEnuXNJRiUcGXbWKJiVB/PFDfp7UOEpMh+bGwwhIpmDV+GWIjgNo+rXXQdBFmtYP9WkXRjleyzWz4ZVA7He+4BeiCuuOwhpMxUzpJ5bL4aBhzF95+Y+6zqvg4rPF0rPaDYYcWGL1DZZuZGdR/HKv/eNihiATKYqOwpq9D09fDDqUoNKQjIhIRKvgiIhGhgi8iEhEq+CIiEaGCLyISESr4IiIRoYIvIhIRKvgiIhGhgi8iEhEq+CIiEaGCLyISEYXueHWFmT1vZs+Y2R1mtsMg580ysxfMbLmZnV9InyKloNyWalToHf79wHvc/X3Ai8AF/U8wszhwFXA0MAU4xcymFNivSLEpt6XqFFTw3f0+d0/3vnwMmJDntGnAcnd/2d1TwC3A8YX0K1Jsym2pRkGO4Z8O3JOnfRdgRZ/XK3vb8jKzOWbWYmYtra3aMk/KQsG5rbyWcrDVgm9mD5jZkjxfx/c55yIgDfw63yXytPlg/bn7XHdvdvfmsWPHDuXPILK9Jpcqt5XXUg62ugGKux++peNmdipwLPAxd8+X7CuBXfu8ngCs2pYgRYrkRXdvHuygcluqTaGzdGYB3wSOc/eOQU5bBEwys4mW2xhyNjCvkH5Fik25LdWo0DH8K4Em4H4zW2xm1wKY2Xgzmw/Q+8HX2cACYBlwm7svLbBfkWJTbkvVKWhPW3ffe5D2VcAxfV7PB+YX0pdIKSm3pRppE3OpOu5ZSD2Kdz0EseFY/SexxJ5hhyVSMM9uwjvvgPRySOyP1X8CizUO+fdV8KWquGfwDWdB6jHwDiCBt9+AD7+MWIOmyEvl8vQr+LqTwVNAJ1CPt18Jo/8Hi48b0jW0lo5Ul+4H+hR7yM2o7IJNF+PZtjAjEymIb/wW+CZyxZ7c9+w6fNPlQ76GCr5UFe+8q0+x78Piub8IRCqQew/0PMnAxzwykHp4yNdRwZfqYnXkfx7KwJKljkYkIDEGL9c123QVkaph9Z8G6vIciUHtQaUORyQQZnFIfoyBH7vWQt3QP5tSwZeqYsmDoPELQBKoB2sEa8RGXkvu2SiRymQjvgPxPXI5TR1YA9RMwZr+z5CvoVk6UnViTV/H60+C1F9yb47kTCzWEHZYIgWx2CgYcxekHofMa5CYDDUHYJZvCDM/FXypSpaYAImTww5DJFBmMUhOB6Zv1+9rSEdEJCJU8EVEIkIFX0QkIlTwRUQiQgVfRCQiLP9GPuXBzFqB14rYxRjgjSJePwiVECNUZpy7u3vJ9xtUXr9NcQZnSHld1gW/2MysZUtb3JWDSogRFGc5qZQ/o+IMzlBj1JCOiEhEqOCLiERE1Av+3LADGIJKiBEUZzmplD+j4gzOkGKM9Bi+iEiURP0OX0QkMlTwRUQiItIF38xOMrOlZpY1s7KbdmVms8zsBTNbbmbnhx1PPmb2KzNba2ZLwo5lMGa2q5k9bGbLev9/nxt2TMVWzrldCXkN1ZnbkS74wBLgRGBh2IH0Z2Zx4CrgaGAKcIqZTQk3qrxuAGaFHcRWpIFvuPt+wIeAs8r0v2WQyjK3KyivoQpzO9IF392XufsLYccxiGnAcnd/2d1TwC3A0PcyKxF3Xwi8GXYcW+Luq939qd6fNwPLgF3Cjaq4yji3KyKvoTpzO9IFv8ztAqzo83olVV6kSsHM9gAOAB4POZSoUl4XyVByu+p3vDKzB4BxeQ5d5O53ljqebZBv3zLNoS2AmQ0Dfgt8zd03hR1PoSo0t5XXRTDU3K76gu/uh4cdw3ZaCeza5/UEYFVIsVQ8M6sh94b4tbv/Lux4glChua28Dti25LaGdMrXImCSmU00s1pgNjAv5JgqkuV2eb4OWObu/xl2PBGnvA7QtuZ2pAu+mZ1gZivJ7Qh8t5ktCDumt7h7GjgbWEDug5jb3H1puFENZGY3A38F9jGzlWb2xbBjyuNg4HPAR81sce/XMWEHVUzlmtuVktdQnbmtpRVERCIi0nf4IiJRooIvIhIRKvgiIhGhgi8iEhEq+CIiEaGCLyISESr4IiIR8f8B/BEmpoDipmoAAAAASUVORK5CYII=",
|
|
"text/plain": [
|
|
"<Figure size 432x288 with 2 Axes>"
|
|
]
|
|
},
|
|
"metadata": {
|
|
"needs_background": "light"
|
|
},
|
|
"output_type": "display_data"
|
|
}
|
|
],
|
|
"source": [
|
|
"pred = tf.matmul(valid_x,W)+b\n",
|
|
"fig,ax = plt.subplots(1,2)\n",
|
|
"ax[0].scatter(valid_x[:,0],valid_x[:,1],c=pred[:,0]>0.5)\n",
|
|
"ax[1].scatter(valid_x[:,0],valid_x[:,1],c=valid_labels)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"To compute the accuracy on the validation data, we can cast boolean type to float, and compute the mean:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 24,
|
|
"metadata": {
|
|
"colab": {
|
|
"base_uri": "https://localhost:8080/"
|
|
},
|
|
"id": "HUjdeIefsIsg",
|
|
"outputId": "f267f505-8ba4-43ef-9ebe-df124c3c05a1"
|
|
},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"<tf.Tensor: shape=(), dtype=float32, numpy=0.4>"
|
|
]
|
|
},
|
|
"execution_count": 24,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"tf.reduce_mean(tf.cast(((pred[0]>0.5)==valid_labels),tf.float32))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Let's explain what goes on here:\n",
|
|
"* `pred` is the values predicted by the network. They are not quite probabilities, because we have not used an activation function, but values greater than 0.5 correspond to class 1, and smaller - to class 0.\n",
|
|
"* `pred[0]>0.5` creates a boolean tensor of results, where `True` corresponds to class 1, and `False` - to class 0\n",
|
|
"* We compare that tensor to expected labels `valid_labels`, getting the boolean vector or correct predictions, where `True` corresponds to the correct prediction, and `False` - to incorrect one.\n",
|
|
"* We convert that tensor to floating point using `tf.cast`\n",
|
|
"* We then compute the mean value using `tf.reduce_mean` - that is exactly our desired accuracy "
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "_95qF9lY2kHp"
|
|
},
|
|
"source": [
|
|
"## Using TensorFlow/Keras Optimizers\n",
|
|
"\n",
|
|
"Tensorflow is closely integrated with Keras, which contains a lot of useful functionality. For example, we can use different **optimization algorithms**. Let's do that, and also print obtained accuracy during training."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 52,
|
|
"metadata": {
|
|
"colab": {
|
|
"base_uri": "https://localhost:8080/"
|
|
},
|
|
"id": "ups7nlV22ofp",
|
|
"outputId": "aa4dff06-82b9-4b2f-ca00-33970ea2b989"
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 0: last batch loss = 1.4725, acc = 1.0000\n",
|
|
"Epoch 1: last batch loss = 5.4883, acc = 0.6667\n",
|
|
"Epoch 2: last batch loss = 3.4839, acc = 0.8333\n",
|
|
"Epoch 3: last batch loss = 6.0828, acc = 0.6667\n",
|
|
"Epoch 4: last batch loss = 7.4671, acc = 0.3333\n",
|
|
"Epoch 5: last batch loss = 4.8935, acc = 0.8333\n",
|
|
"Epoch 6: last batch loss = 4.5684, acc = 0.8333\n",
|
|
"Epoch 7: last batch loss = 4.8099, acc = 0.6667\n",
|
|
"Epoch 8: last batch loss = 6.8351, acc = 0.5000\n",
|
|
"Epoch 9: last batch loss = 4.1826, acc = 0.8333\n",
|
|
"Epoch 10: last batch loss = 4.8398, acc = 0.8333\n",
|
|
"Epoch 11: last batch loss = 5.0882, acc = 0.8333\n",
|
|
"Epoch 12: last batch loss = 3.6824, acc = 1.0000\n",
|
|
"Epoch 13: last batch loss = 2.9054, acc = 1.0000\n",
|
|
"Epoch 14: last batch loss = 3.8756, acc = 1.0000\n",
|
|
"Epoch 15: last batch loss = 5.0647, acc = 1.0000\n",
|
|
"Epoch 16: last batch loss = 3.7667, acc = 1.0000\n",
|
|
"Epoch 17: last batch loss = 2.2130, acc = 1.0000\n",
|
|
"Epoch 18: last batch loss = 3.9063, acc = 0.8333\n",
|
|
"Epoch 19: last batch loss = 5.5129, acc = 0.6667\n",
|
|
"Epoch 20: last batch loss = 6.1930, acc = 0.6667\n",
|
|
"Epoch 21: last batch loss = 5.8449, acc = 0.6667\n",
|
|
"Epoch 22: last batch loss = 2.4126, acc = 1.0000\n",
|
|
"Epoch 23: last batch loss = 2.6007, acc = 0.8333\n",
|
|
"Epoch 24: last batch loss = 2.1278, acc = 0.8333\n",
|
|
"Epoch 25: last batch loss = 2.1106, acc = 1.0000\n",
|
|
"Epoch 26: last batch loss = 5.0858, acc = 0.5000\n",
|
|
"Epoch 27: last batch loss = 5.0222, acc = 0.6667\n",
|
|
"Epoch 28: last batch loss = 5.5569, acc = 0.6667\n",
|
|
"Epoch 29: last batch loss = 6.2620, acc = 0.6667\n",
|
|
"Epoch 30: last batch loss = 2.6355, acc = 0.8333\n",
|
|
"Epoch 31: last batch loss = 3.6728, acc = 0.8333\n",
|
|
"Epoch 32: last batch loss = 4.3967, acc = 0.6667\n",
|
|
"Epoch 33: last batch loss = 4.1097, acc = 0.8333\n",
|
|
"Epoch 34: last batch loss = 0.6812, acc = 1.0000\n",
|
|
"Epoch 35: last batch loss = 4.7926, acc = 0.6667\n",
|
|
"Epoch 36: last batch loss = 3.1601, acc = 0.8333\n",
|
|
"Epoch 37: last batch loss = 1.8902, acc = 1.0000\n",
|
|
"Epoch 38: last batch loss = 7.0491, acc = 0.5000\n",
|
|
"Epoch 39: last batch loss = 1.2064, acc = 1.0000\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"optimizer = tf.keras.optimizers.Adam(0.01)\n",
|
|
"\n",
|
|
"learning_rate = 0.05\n",
|
|
"\n",
|
|
"W = tf.Variable(tf.random.normal(shape=(2,1)))\n",
|
|
"b = tf.Variable(tf.zeros(shape=(1,),dtype=tf.float32))\n",
|
|
"\n",
|
|
"@tf.function\n",
|
|
"def train_on_batch(x, y):\n",
|
|
" vars = [W, b]\n",
|
|
" with tf.GradientTape() as tape:\n",
|
|
" z = tf.sigmoid(tf.matmul(x, W) + b)\n",
|
|
" loss = tf.reduce_mean(tf.keras.losses.binary_crossentropy(z,y))\n",
|
|
" correct_prediction = tf.equal(tf.round(y), tf.round(z))\n",
|
|
" acc = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))\n",
|
|
" grads = tape.gradient(loss, vars)\n",
|
|
" optimizer.apply_gradients(zip(grads,vars))\n",
|
|
" return loss,acc\n",
|
|
"\n",
|
|
"for epoch in range(40):\n",
|
|
" for step, (x, y) in enumerate(dataset):\n",
|
|
" loss,acc = train_on_batch(tf.reshape(x,(-1,2)), tf.reshape(y,(-1,1)))\n",
|
|
" print('Epoch %d: last batch loss = %.4f, acc = %.4f' % (epoch, float(loss),acc))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "dvAiaj_JndyP"
|
|
},
|
|
"source": [
|
|
"**Task 1**: Plot the graphs of loss function and accuracy on training and validation data during training\n",
|
|
"\n",
|
|
"**Task 2**: Try to solve MNIST classificiation problem using this code. Hint: use `softmax_crossentropy_with_logits` or `sparse_softmax_cross_entropy_with_logits` as loss function. In the first case you need to feed expected output values in *one hot encoding*, and in the second case - as integer class number."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "995iCprDrgYQ"
|
|
},
|
|
"source": [
|
|
"## Keras\n",
|
|
"### Deep Learning for Humans\n",
|
|
"\n",
|
|
"* Keras is a library originally developed by Francois Chollet to work on top of Tensorflow, CNTK and Theano, to unify all lower-level frameworks. You can still install Keras as a separate library, but it is not advised to do so. \n",
|
|
"* Now Keras is included as part of Tensorflow library\n",
|
|
"* You can easily construct neural networks from layers\n",
|
|
"* Contains `fit` function to do all training, plus a lot of functions to work with typical data (pictures, text, etc.)\n",
|
|
"* A lot of samples\n",
|
|
"* Functional API vs. Sequential API\n",
|
|
"\n",
|
|
"Keras provides higher level abstractions for neural networks, allowing us to operate in terms of layers, models and optimizers, and not in terms of tensors and gradients. \n",
|
|
"\n",
|
|
"Classical Deep Learning book from the creator of Keras: [Deep Learning with Python](https://www.manning.com/books/deep-learning-with-python)\n",
|
|
"\n",
|
|
"### Functional API\n",
|
|
"\n",
|
|
"When using functional API, we define the **input** to the network as `keras.Input`, and then compute the **output** by passing it through a series of computations. Finally, we define **model** as an object that transforms input into output.\n",
|
|
"\n",
|
|
"Once we obtained **model** object, we need to:\n",
|
|
"* **Compile it**, by specifying loss function and the optimizer that we want to use with our model\n",
|
|
"* **Train it** by calling `fit` function with the training (and possibly validation) data"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 26,
|
|
"metadata": {
|
|
"colab": {
|
|
"base_uri": "https://localhost:8080/"
|
|
},
|
|
"id": "QJWplVfy34Eo",
|
|
"outputId": "9be976f2-4f9a-495c-bddc-a7f9ec30989a"
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Model: \"functional_1\"\n",
|
|
"_________________________________________________________________\n",
|
|
"Layer (type) Output Shape Param # \n",
|
|
"=================================================================\n",
|
|
"input_1 (InputLayer) [(None, 2)] 0 \n",
|
|
"_________________________________________________________________\n",
|
|
"dense (Dense) (None, 1) 3 \n",
|
|
"=================================================================\n",
|
|
"Total params: 3\n",
|
|
"Trainable params: 3\n",
|
|
"Non-trainable params: 0\n",
|
|
"_________________________________________________________________\n",
|
|
"Epoch 1/15\n",
|
|
"9/9 [==============================] - 0s 5ms/step - loss: 1.2681 - accuracy: 0.1857\n",
|
|
"Epoch 2/15\n",
|
|
"9/9 [==============================] - 0s 1ms/step - loss: 0.7420 - accuracy: 0.5286\n",
|
|
"Epoch 3/15\n",
|
|
"9/9 [==============================] - 0s 2ms/step - loss: 0.5167 - accuracy: 0.7714\n",
|
|
"Epoch 4/15\n",
|
|
"9/9 [==============================] - 0s 1ms/step - loss: 0.4352 - accuracy: 0.8429\n",
|
|
"Epoch 5/15\n",
|
|
"9/9 [==============================] - 0s 2ms/step - loss: 0.4198 - accuracy: 0.8571\n",
|
|
"Epoch 6/15\n",
|
|
"9/9 [==============================] - 0s 2ms/step - loss: 0.4091 - accuracy: 0.8429\n",
|
|
"Epoch 7/15\n",
|
|
"9/9 [==============================] - 0s 1ms/step - loss: 0.4118 - accuracy: 0.8286\n",
|
|
"Epoch 8/15\n",
|
|
"9/9 [==============================] - 0s 1ms/step - loss: 0.4058 - accuracy: 0.8429\n",
|
|
"Epoch 9/15\n",
|
|
"9/9 [==============================] - 0s 1ms/step - loss: 0.4072 - accuracy: 0.8571\n",
|
|
"Epoch 10/15\n",
|
|
"9/9 [==============================] - 0s 1ms/step - loss: 0.4125 - accuracy: 0.8714\n",
|
|
"Epoch 11/15\n",
|
|
"9/9 [==============================] - 0s 1ms/step - loss: 0.4186 - accuracy: 0.8429\n",
|
|
"Epoch 12/15\n",
|
|
"9/9 [==============================] - 0s 1ms/step - loss: 0.4092 - accuracy: 0.8429\n",
|
|
"Epoch 13/15\n",
|
|
"9/9 [==============================] - 0s 2ms/step - loss: 0.4078 - accuracy: 0.8429\n",
|
|
"Epoch 14/15\n",
|
|
"9/9 [==============================] - 0s 1ms/step - loss: 0.4095 - accuracy: 0.8429\n",
|
|
"Epoch 15/15\n",
|
|
"9/9 [==============================] - 0s 1ms/step - loss: 0.4087 - accuracy: 0.8429\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"inputs = tf.keras.Input(shape=(2,))\n",
|
|
"z = tf.keras.layers.Dense(1,kernel_initializer='glorot_uniform',activation='sigmoid')(inputs)\n",
|
|
"model = tf.keras.models.Model(inputs,z)\n",
|
|
"\n",
|
|
"train_x_norm = train_x-np.min(train_x) / (np.max(train_x)-np.min(train_x))\n",
|
|
"\n",
|
|
"model.compile(tf.keras.optimizers.Adam(0.1),'binary_crossentropy',['accuracy'])\n",
|
|
"model.summary()\n",
|
|
"h = model.fit(train_x_norm,train_labels,batch_size=8,epochs=15)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 27,
|
|
"metadata": {
|
|
"colab": {
|
|
"base_uri": "https://localhost:8080/",
|
|
"height": 282
|
|
},
|
|
"id": "K2Kf60IrZcqs",
|
|
"outputId": "b60b868d-3562-4715-f5d5-1f9764e45f09"
|
|
},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"[<matplotlib.lines.Line2D at 0x25c76445310>]"
|
|
]
|
|
},
|
|
"execution_count": 27,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
},
|
|
{
|
|
"data": {
|
|
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD7CAYAAAB68m/qAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAAfbUlEQVR4nO3deXCU953n8fdXFwJdXBKgg8NYNgjbHJbtOKfXHhw8yYQc9g6e3JMU8e44m92a3QnZqczWbqp2k83s7sxsnCVU4nhmkwoVJ86YyRAb1jmcSRwPQsLYEmBjDJIQEgJ0g+7v/tGNaISEGmjp6X7686pS0U/3j9aHLvTR079+fs9j7o6IiKS+jKADiIhIYqjQRURCQoUuIhISKnQRkZBQoYuIhIQKXUQkJOIqdDPbZGZHzOyomW2b4PF5ZvYTMztoZv9sZrclPqqIiFzNlIVuZpnAE8BDQBXwqJlVjRv2H4ED7n4H8AngrxMdVEREri4rjjF3A0fd/RiAme0ENgMNMWOqgP8G4O6HzWy5mS1y97bJnnThwoW+fPny6w4uIpKO9u/ff8bdiyd6LJ5CLwOaYrabgXvGjXkF+DDwT2Z2N7AMKAcmLfTly5dTU1MTx7cXEZGLzOzEZI/FM4duE9w3/nwBXwXmmdkB4PNAHTA8QZCtZlZjZjXt7e1xfGsREYlXPHvozUBFzHY50BI7wN27gU8DmJkBb0W/GDduB7ADoLq6WieRERFJoHj20PcBlWa2wsxygC3ArtgBZjY3+hjAZ4EXoyUvIiIzZMo9dHcfNrPHgeeBTOBJd683s8eij28HVgN/Z2YjRD4s/cw0ZhYRkQnEM+WCu+8Gdo+7b3vM7ZeAysRGExGRa6GVoiIiIaFCFxEJibimXEQkuZwfHObQqR4aTnUzKzODTbcvpjA3O+hYEjAVukiSO9M7QH1LNw0t3dS3dNFwqpu3zvQRe/XILz/7GptuW8wjd1bw9pULyMiYaPmIhJ0KXSRJjI46jefO03AqWtwt3dS3dHO6Z2BsTPm82VQtKWTz2jKqSgupKi3kbO8AT9c08+yBkzx7oIWyubP5yIYyHr6zgqUL5gT4L5KZZkFdJLq6utq19F/S1cDwCG+09dLQ0j1W4IdO9dA7EFlgnZlhVJbkR0p7SSFrSouoWlJI0ZzJp1X6h0bY29DG0/ub+fUb7bjDPSvm80h1Bb9/+2Lm5Gj/LQzMbL+7V0/4mApdAHr6hzh0qodTXRdYWZxP5aJ8ZmVlBh0rFLr7hyLFHd3jbjjVzRttPQyPRn728nIyWb0ksre9prSQqiVFVC7KJzf7+l//U10XeKb2JE/XNHH87HnycjJ53x1LeKS6gupl84gs6JZUpEKXMe7O6Z6By97SN5zq5sTZ85eNy840bi4piO4dFo69vdcHb5Nzd1q7+y+9ri3d1J/qounchbExxQWzLntN15QWsWz+nGmb83Z3ak508HRNE/948BR9gyOsWJjHw3eW8+ENZSwpmj0t31emjwo9TY2MOm+d6btsTrahpZuzfYNjY5YtmBPdK4yUy5K5uRw93RvzIVw3Z3ovzeFWzJ/NmiVFl/YmSwtZXJibdnt8I6POsfbe6GvbPTZ1ci7mtV2xMC9myiTyWpUU5AaWuW9gmJ+91srTNU28/NY5MgzeWVnMI3eWs7Fq0Q29I5CZo0JPA/1DIxxp7YnucXdR39LN4VM9XBgaASJ73Lcsit3jLmL1kgIK4tjjPt3Tf6m0osX11pm+scfn5+WM2+ssZMXCfDJDcqTFhcERDrdeejdT39LNkdZu+odGAcjJzODWxdHXtixS4KuWFJI/K3nnrE+c7ePH+5v5ce1JTnZeoGh2Nh9YW8oj1eXcXlaUdr+gU4kKPYRqGzvYf7xj7DC2N9v7GInOyRbMymL12HxsZM/75pJ8crISt46sd2CYw6cun1p4vbWXwZFIyeVmZ7Bqcey8cCGrFhcyOye59wLP9Q1eMR11rL2X6EtLYW5WdK+7iDWlkQJfWZxPdmZqrtEbHXV+++ZZnt7fxHOvtTIwPMqtiwp4pLqcWxcXBB0vtMrnzWHFwrzr+rsq9BBxd775yzf5+vNHAFhcmHvZnnHVkiIq5s8OZA9raGSUo6d7x8rw4i+bnv7IkRsZBjcV51/2i6aqtJD5eTlTPHPiuTtN5y6MvZu5mLm1u39sTGlRbvSzg6KxdyDl84J5bWdC14UhfnqwhadrmjnQ1Bl0nFB77D0r2fbQquv6uyr0kHB3vvbcEbb/6k0+uK6UL7+/igX5s4KOdVXuTnPHhWhpdo1NWZzqulScS4pyr/igMJHFOTQyyhttvWO/YOpbujnU0k3PwKVfNCsv/qKJ/lIM6hdNsjh+pu+yz04ksRYV5lIx//rWCKjQQ2B01PlPu+r5v787wUfvWcpXNt+W0qsBz/UNRufjL01vvBkztVGQm0XVkksFX7WkkMpFU09t9PQPcbi1h/qTl8r7jbZLU0GzszNZtaTgsncIqxYX6ANBSRkq9BQ3PDLKn/3oIM/UneRz776JbQ+tCuXb/v6hEQ639ly2xP3Qqcs/fKxcdGnKpqq0iL6B4cuO4jkec/jlgrycscMtLxb4ioV5ofmwVtKTCj2FDQyP8IUfHOC5+lb+dOMtPH7/zaEs88lcPPTyYsFf3JuPPTwQYOn8yOGXsdM2JQWz0uq1kvRwtUJP3uOqhAuDI3zue/t58fV2vvz+Kj7zzhVBR5pxmRnGzSX53FySz+Z1ZUBkXr6te4BDp7qZk5PJai14EgFU6Emrp3+IzzxVw74T5/jaR27nD+9aGnSkpGFmLC7KZXFRcIt0RJKRCj0JdfQN8snv/jMNLd38zZb1/MHa0qAjiUgKUKEnmdPd/XzsOy9z/Ox5vvXxO3lg9aKgI4lIiohreZuZbTKzI2Z21My2TfB4kZn9g5m9Ymb1ZvbpxEcNv+aO8zzyrZdo7rjAU5++S2UuItdkykI3s0zgCeAhoAp41Myqxg37E6DB3dcC9wH/w8zSd1XGdTjW3ssj21+io2+Q7332Ht6+cmHQkUQkxcSzh343cNTdj7n7ILAT2DxujAMFFjlGLB84BwwnNGmINbR08y+/9RKDw6Ps3HovG5bOCzqSiKSgeAq9DGiK2W6O3hfrG8BqoAV4FfiCu48mJGHI1TZ2sGXHS2RnZvDDx+6lqrQw6EgikqLiKfSJVmaMX430XuAAUAqsA75hZlc0k5ltNbMaM6tpb2+/xqjh89s3z/Cxb7/MvLwcfvi5e1lZnB90JBFJYfEUejNQEbNdTmRPPNangWc84ijwFnDFqcTcfYe7V7t7dXFx8fVmDoWfH27j09/dR/m82Tz9uXuv+0Q9IiIXxVPo+4BKM1sR/aBzC7Br3JhG4AEAM1sE3AocS2TQMPnpwRa2/t1+bllUwM6t91JSqAUyInLjpjwO3d2Hzexx4HkgE3jS3evN7LHo49uBrwBPmdmrRKZovujuZ6Yxd8r64b4mtj1zkDuXzeM7n7pLS9ZFJGHiWljk7ruB3ePu2x5zuwV4MLHRwue7v3mL//wPDbyrciE7Pl6d9FfvEZHUopWiM8DdeeIXR/nLPa/z3jWL+JtH1zMrS2UuIomlQp9m7s5XnzvMt351jA+vL+O/P3wHWSl6/UkRSW4q9Gn21Z8d5lsvHuNjb1vKf/lAal9lSESSmwp9GvUODPPd3xznQ+vL+Mrm23SxBRGZVnrvP41+daSdwZFRttxVoTIXkWmnQp9GexpamZ+Xw53LdG4WEZl+KvRpMjQyys8Pn+b+VSX6EFREZoSaZpq8fOwcPf3DPFilc5qLyMxQoU+TvQ2t5GZn8K7K9D5njYjMHBX6NHB39ja08a7KYq0GFZEZo0KfBvUt3bR09bNR0y0iMoNU6NNgT30rGQYPrCoJOoqIpBEV+jTY09BG9bL5LMifFXQUEUkjKvQEazp3nsOtPTy4RtMtIjKzVOgJtqehDUDz5yIy41ToCbanvpVbFxWwbEFe0FFEJM2o0BOoo2+QfcfPae9cRAKhQk+gFw6fZtTR/LmIBEKFnkB7G1pZXJjL7WVFQUcRkTSkQk+Q/qERXnz9DBurFulUuSISiLgK3cw2mdkRMztqZtsmePw/mNmB6NdrZjZiZvMTHzd5/dMbZ7gwNKL5cxEJzJSFbmaZwBPAQ0AV8KiZVcWOcfevu/s6d18HfAn4lbufm4a8SWtPQysFs7J4200Lgo4iImkqnj30u4Gj7n7M3QeBncDmq4x/FPhBIsKlipFR54VDp7lvVQk5WZrFEpFgxNM+ZUBTzHZz9L4rmNkcYBPw4xuPljpqGzs42zeoc5+LSKDiKfSJPuHzScb+AfCbyaZbzGyrmdWYWU17e3u8GZPe3oY2sjON+27Vuc9FJDjxFHozUBGzXQ60TDJ2C1eZbnH3He5e7e7VxcXhKD93Z099K/euXEhBbnbQcUQkjcVT6PuASjNbYWY5REp71/hBZlYEvAd4NrERk9vR070cP3teR7eISOCyphrg7sNm9jjwPJAJPOnu9Wb2WPTx7dGhHwL2uHvftKVNQmMn41qtQheRYE1Z6ADuvhvYPe6+7eO2nwKeSlSwVLGnoY215UUsLsoNOoqIpDkdY3cD2rr7eaWpkwfXLA46ioiICv1G7NW5z0UkiajQb8CehjaWL5hDZUl+0FFERFTo16unf4iX3tTJuEQkeajQr9Mvj7QzNOKaPxeRpKFCv057G9pYkJfDhqXzgo4iIgKo0K/L4PAovzh8mgdWl5CZoekWEUkOKvTr8PJbZ+kZGObBKk23iEjyUKFfhz31bczOzuSdlQuDjiIiMkaFfo3cnb0Nbbz7loXkZmcGHUdEZIwK/Rq9erKL1u5+Nmq6RUSSjAr9Gu1taCPD4IFVJUFHERG5jAr9Gu2pb+Ou5fOZl5cTdBQRkcuo0K/BibN9HGnr0WIiEUlKKvRrcPFkXLp2qIgkIxX6NdjT0MaqxQVUzJ8TdBQRkSuo0ON0rm+QmuPntHcuIklLhR6nFw61Mepo/lxEkpYKPU57GtooLcplTWlh0FFERCakQo/DhcERfv1Gu859LiJJLa5CN7NNZnbEzI6a2bZJxtxnZgfMrN7MfpXYmMH69Rvt9A+NanWoiCS1rKkGmFkm8ASwEWgG9pnZLndviBkzF/gmsMndG80sVMso9za0UZCbxT03zQ86iojIpOLZQ78bOOrux9x9ENgJbB435o+AZ9y9EcDdTyc2ZnBGRp0XDp/m/lUlZGdqhkpEklc8DVUGNMVsN0fvi3ULMM/Mfmlm+83sExM9kZltNbMaM6tpb2+/vsQzbP+JDs71Derc5yKS9OIp9Ik+BfRx21nAncD7gPcCXzazW674S+473L3a3auLi4uvOWwQ9tS3kpOZwXtuTY28IpK+ppxDJ7JHXhGzXQ60TDDmjLv3AX1m9iKwFng9ISkD4u7sPdTG229eQP6seF4qEZHgxLOHvg+oNLMVZpYDbAF2jRvzLPAuM8sysznAPcChxEadea+39XLi7Hk2anWoiKSAKXc73X3YzB4HngcygSfdvd7MHos+vt3dD5nZc8BBYBT4tru/Np3BZ8LehlYANq5WoYtI8otrHsHddwO7x923fdz214GvJy5a8PY0tLGuYi4lhblBRxERmZKOw5vEqa4LHGzu4sE12jsXkdSgQp/E/9O5z0UkxajQJ7GnoY2bFuaxsjg/6CgiInFRoU+gu3+I3x07q5NxiUhKUaFP4JdH2hkacc2fi0hKUaFPYE99Kwvzc1hXMS/oKCIicVOhjzMwPMIvj7Tze6sXkZmh6RYRSR0q9HF+d+wcvQPDWh0qIilHhT7O3oZW5uRk8o6bFwYdRUTkmqjQY4yOOnsb2nh3ZTG52ZlBxxERuSYq9BivnuyirXtAR7eISEpSocfY09BKZoZx/6pQXUFPRNKECj3G3oY27l4+n7lzcoKOIiJyzVToUcfP9PF6W6+ObhGRlKVCj9obPRmXCl1EUpUKPeqlY2e5uSSfivlzgo4iInJdVOhErh1a19jBhqVzg44iInLdVOjA8bPn6Tg/xIalOneLiKQuFTpQ19gBwHoVuoikMBU6UNfYSf6sLG4u0cUsRCR1xVXoZrbJzI6Y2VEz2zbB4/eZWZeZHYh+/UXio06f2sYO1lXM1dkVRSSlZU01wMwygSeAjUAzsM/Mdrl7w7ihv3b3909Dxml1fnCYw609/Ov7VgYdRUTkhsSzh343cNTdj7n7ILAT2Dy9sWbOq81djIw663WEi4ikuHgKvQxoitlujt433r1m9oqZ/czM1kz0RGa21cxqzKymvb39OuImXm1jJwDrdXUiEUlx8RT6RBPLPm67Fljm7muB/w38/URP5O473L3a3auLi4uvKeh0qWvsYMXCPObl6fwtIpLa4in0ZqAiZrscaIkd4O7d7t4bvb0byDazpL9ChLtT19TJ+oq5QUcREblh8RT6PqDSzFaYWQ6wBdgVO8DMFpuZRW/fHX3es4kOm2jNHRdo7xlg/TJNt4hI6pvyKBd3Hzazx4HngUzgSXevN7PHoo9vBx4G/pWZDQMXgC3uPn5aJunUNXUCaA9dREJhykKHsWmU3ePu2x5z+xvANxIbbfrVNXaQm53BqsUFQUcREblhab1StLaxkzvK55KVmdYvg4iERNo2Wf/QCA0tXTohl4iERtoWen1LN0MjWlAkIuGRtoV+6QyLc4MNIiKSIGlc6J2Uz5tNSUFu0FFERBIijQu9Q+c/F5FQSctCb+3qp6WrX8efi0iopGWhX5w/36AVoiISIulZ6E2d5GRlULWkMOgoIiIJk56F3tjBbaWF5GSl5T9fREIq7RptcHiUg81aUCQi4ZN2hX64tZuB4VEd4SIioZN2hV538QpFWlAkIiGTdoVe29jB4sJcSufODjqKiEhCpV2h1zV2au9cREIprQr9TO8AjefOq9BFJJTSqtAvzZ/rA1ERCZ80K/QOsjKM28uKgo4iIpJwaVbonVSVFpKbnRl0FBGRhIur0M1sk5kdMbOjZrbtKuPuMrMRM3s4cRETY3hklFeaO3VCLhEJrSkL3cwygSeAh4Aq4FEzq5pk3NeA5xMdMhFeb+vl/OCITsglIqEVzx763cBRdz/m7oPATmDzBOM+D/wYOJ3AfAlT1xS9QlGFCl1EwimeQi8DmmK2m6P3jTGzMuBDwPbERUus2hOdLMjLoWK+FhSJSDjFU+g2wX0+bvuvgC+6+8hVn8hsq5nVmFlNe3t7nBETo64pcoUis4n+OSIiqS+eQm8GKmK2y4GWcWOqgZ1mdhx4GPimmX1w/BO5+w53r3b36uLi4utLfB06zw9yrL1PC4pEJNSy4hizD6g0sxXASWAL8EexA9x9xcXbZvYU8FN3//vExbwxB5o6AZ2QS0TCbcpCd/dhM3ucyNErmcCT7l5vZo9FH0/aefOLahs7yTBYWz436CgiItMmnj103H03sHvcfRMWubt/6sZjJVZdYwe3Li4kb1Zc/1wRkZQU+pWio6POgSadYVFEwi/0hf5mey89/cO65JyIhF7oC11XKBKRdBH+Qm/qoGh2NisW5AUdRURkWoW+0GtPdLKuYi4ZGVpQJCLhFupC7+kf4vXTPZo/F5G0EOpCP9jchbvmz0UkPYS60GtPRM6wuFbnQBeRNBDqQq9r6qSyJJ+i2dlBRxERmXahLXR3p66xQ9MtIpI2Qlvox8+ep+P8EOv1gaiIpInQFnpdY2T+XEe4iEi6CHGhd5I/K4ubS/KDjiIiMiPCW+hNHaytKCJTC4pEJE2EstDPDw5z6JQWFIlIegllob/a3MXIqOsIFxFJK6Es9LroJefWVWgPXUTSRygLvfZEB8sXzGF+Xk7QUUREZkzoCt3dqWvq1Py5iKSd0BX6yc4LtPcMaP5cRNJOXIVuZpvM7IiZHTWzbRM8vtnMDprZATOrMbN3Jj5qfGrHrlCkPXQRSS9ZUw0ws0zgCWAj0AzsM7Nd7t4QM+wFYJe7u5ndAfwQWDUdgadS19hBbnYGqxYXBPHtRUQCE88e+t3AUXc/5u6DwE5gc+wAd+91d49u5gFOQOoaO7mjfC5ZmaGbTRIRuap4Wq8MaIrZbo7edxkz+5CZHQb+EfjjxMS7Nv1DI9S3dGn+XETSUjyFPtHa+Sv2wN39J+6+Cvgg8JUJn8hsa3SOvaa9vf2agsajvqWboRHXES4ikpbiKfRmoCJmuxxomWywu78IrDSzhRM8tsPdq929uri4+JrDTuXiGRbX6wpFIpKG4in0fUClma0wsxxgC7ArdoCZ3WxmFr29AcgBziY67FTqGjspmzubksLcmf7WIiKBm/IoF3cfNrPHgeeBTOBJd683s8eij28HPgJ8wsyGgAvAH8Z8SDpj6ho7uHP5/Jn+tiIiSWHKQgdw993A7nH3bY+5/TXga4mNdm1au/pp6erns5puEZE0FZpj+8bmz3WEi4ikqfAUelMnOVkZrCktCjqKiEggwlPojR3cVlpITlZo/kkiItckFO03NDLKweYunb9FRNJaKAr90KluBoZHNX8uImktFIVeFz3DolaIikg6C0mhd7CocBZLirSgSETSVygKvbaxk/UV84guVhURSUspX+hnegdoPHeeDcvmBh1FRCRQKV/oB3SFIhERIASFXtvYQVaGcXuZFhSJSHpL+UKva+ykqrSQ3OzMoKOIiAQqpQt9ZNR5pblT5z8XESHFC/1Iaw/nB0c0fy4iQooXel1T5AyLWlAkIpLqhd7YyYK8HCrmzw46iohI4FK60GsbO1i/dK4WFImIkMKF3nl+kGPtfZo/FxGJStlCP9DUCegKRSIiF6Vsodc1dpJhcEf53KCjiIgkhbgK3cw2mdkRMztqZtsmePyjZnYw+vVbM1ub+KiXq23s4JZFBeTPius61yIioTdloZtZJvAE8BBQBTxqZlXjhr0FvMfd7wC+AuxIdNBYo6POgaZONizT/LmIyEXx7KHfDRx192PuPgjsBDbHDnD337p7R3Tzd0B5YmNe7tiZXnr6h7VCVEQkRjyFXgY0xWw3R++bzGeAn91IqKnUnugEdIZFEZFY8UxAT3SQt0840OxfECn0d07y+FZgK8DSpUvjjHiluqYOimZnc9PCvOt+DhGRsIlnD70ZqIjZLgdaxg8yszuAbwOb3f3sRE/k7jvcvdrdq4uLi68nLxA5wmVdxVwyMrSgSETkongKfR9QaWYrzCwH2ALsih1gZkuBZ4CPu/vriY95SU//EEfaenT8uYjIOFNOubj7sJk9DjwPZAJPunu9mT0WfXw78BfAAuCb0WX4w+5ePR2BDzZ34a4TcomIjBfXQdzuvhvYPe6+7TG3Pwt8NrHRJpaTlcH9q0pYqyNcREQuk3Krcu5aPp+7PjU/6BgiIkknZZf+i4jI5VToIiIhoUIXEQkJFbqISEio0EVEQkKFLiISEip0EZGQUKGLiISEuU944sTp/8Zm7cCJ6/zrC4EzCYwz3VIpbyplhdTKm0pZIbXyplJWuLG8y9x9wrMbBlboN8LMaqbrXDHTIZXyplJWSK28qZQVUitvKmWF6curKRcRkZBQoYuIhESqFvq0XoR6GqRS3lTKCqmVN5WyQmrlTaWsME15U3IOXURErpSqe+giIjJOyhW6mW0ysyNmdtTMtgWdZzJmVmFmvzCzQ2ZWb2ZfCDpTPMws08zqzOynQWe5GjOba2Y/MrPD0df43qAzXY2Z/bvo/4PXzOwHZpYbdKZYZvakmZ02s9di7ptvZnvN7I3on0lxmbBJsn49+n/hoJn9xMzmBhjxMhPljXns35uZm9nCRHyvlCp0M8sEngAeAqqAR82sKthUkxoG/tTdVwNvA/4kibPG+gJwKOgQcfhr4Dl3XwWsJYkzm1kZ8G+Aane/jcilHLcEm+oKTwGbxt23DXjB3SuBF6LbyeAprsy6F7jN3e8AXge+NNOhruIprsyLmVUAG4HGRH2jlCp04G7gqLsfc/dBYCewOeBME3L3U+5eG73dQ6RwyoJNdXVmVg68D/h20FmuxswKgXcD3wFw90F37ww01NSygNlmlgXMAVoCznMZd38RODfu7s3A30Zv/y3wwZnMNJmJsrr7Hncfjm7+Diif8WCTmOS1BfhfwJ8BCfsgM9UKvQxoitluJslLEsDMlgPrgZcDjjKVvyLyH2w04BxTuQloB74bnR76tpnlBR1qMu5+EvhLIntip4Aud98TbKq4LHL3UxDZQQFKAs4Trz8GfhZ0iKsxsw8AJ939lUQ+b6oVuk1wX1IfpmNm+cCPgX/r7t1B55mMmb0fOO3u+4POEocsYAPwf9x9PdBH8kwHXCE697wZWAGUAnlm9rFgU4WTmf05kenO7wedZTJmNgf4c+AvEv3cqVbozUBFzHY5SfbWNZaZZRMp8++7+zNB55nCO4APmNlxIlNZ95vZ94KNNKlmoNndL77j+RGRgk9Wvwe85e7t7j4EPAO8PeBM8WgzsyUA0T9PB5znqszsk8D7gY96ch+PvZLIL/dXoj9v5UCtmS2+0SdOtULfB1Sa2QozyyHywdKugDNNyMyMyBzvIXf/n0HnmYq7f8ndy919OZHX9efunpR7ke7eCjSZ2a3Rux4AGgKMNJVG4G1mNif6/+IBkvhD3Bi7gE9Gb38SeDbALFdlZpuALwIfcPfzQee5Gnd/1d1L3H159OetGdgQ/X99Q1Kq0KMfejwOPE/kB+KH7l4fbKpJvQP4OJE93QPRr98POlSIfB74vpkdBNYB/zXYOJOLvpP4EVALvErk5y6pVjaa2Q+Al4BbzazZzD4DfBXYaGZvEDka46tBZrxokqzfAAqAvdGfte2BhowxSd7p+V7J/c5ERETilVJ76CIiMjkVuohISKjQRURCQoUuIhISKnQRkZBQoYuIhIQKXUQkJFToIiIh8f8BXPL7vZXYSYQAAAAASUVORK5CYII=",
|
|
"text/plain": [
|
|
"<Figure size 432x288 with 1 Axes>"
|
|
]
|
|
},
|
|
"metadata": {
|
|
"needs_background": "light"
|
|
},
|
|
"output_type": "display_data"
|
|
}
|
|
],
|
|
"source": [
|
|
"plt.plot(h.history['accuracy'])"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "iJruFXmb_dur"
|
|
},
|
|
"source": [
|
|
"### Sequential API\n",
|
|
"\n",
|
|
"Alternatively, we can start thinking of a model as of a **sequence of layers**, and just specify those layers by adding them to the `model` object:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 28,
|
|
"metadata": {
|
|
"colab": {
|
|
"base_uri": "https://localhost:8080/"
|
|
},
|
|
"id": "iWc_kSr8_YXt",
|
|
"outputId": "345dbe65-629d-468f-ed75-1d412c966340"
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Model: \"sequential\"\n",
|
|
"_________________________________________________________________\n",
|
|
"Layer (type) Output Shape Param # \n",
|
|
"=================================================================\n",
|
|
"dense_1 (Dense) (None, 5) 15 \n",
|
|
"_________________________________________________________________\n",
|
|
"dense_2 (Dense) (None, 1) 6 \n",
|
|
"=================================================================\n",
|
|
"Total params: 21\n",
|
|
"Trainable params: 21\n",
|
|
"Non-trainable params: 0\n",
|
|
"_________________________________________________________________\n",
|
|
"Epoch 1/15\n",
|
|
"9/9 [==============================] - 0s 26ms/step - loss: nan - accuracy: 0.4571 - val_loss: nan - val_accuracy: 0.4000\n",
|
|
"Epoch 2/15\n",
|
|
"9/9 [==============================] - 0s 4ms/step - loss: nan - accuracy: 0.5000 - val_loss: nan - val_accuracy: 0.4000\n",
|
|
"Epoch 3/15\n",
|
|
"9/9 [==============================] - 0s 4ms/step - loss: nan - accuracy: 0.5000 - val_loss: nan - val_accuracy: 0.4000\n",
|
|
"Epoch 4/15\n",
|
|
"9/9 [==============================] - 0s 4ms/step - loss: nan - accuracy: 0.5000 - val_loss: nan - val_accuracy: 0.4000\n",
|
|
"Epoch 5/15\n",
|
|
"9/9 [==============================] - 0s 4ms/step - loss: nan - accuracy: 0.5000 - val_loss: nan - val_accuracy: 0.4000\n",
|
|
"Epoch 6/15\n",
|
|
"9/9 [==============================] - 0s 4ms/step - loss: nan - accuracy: 0.5000 - val_loss: nan - val_accuracy: 0.4000\n",
|
|
"Epoch 7/15\n",
|
|
"9/9 [==============================] - 0s 4ms/step - loss: nan - accuracy: 0.5000 - val_loss: nan - val_accuracy: 0.4000\n",
|
|
"Epoch 8/15\n",
|
|
"9/9 [==============================] - 0s 4ms/step - loss: nan - accuracy: 0.5000 - val_loss: nan - val_accuracy: 0.4000\n",
|
|
"Epoch 9/15\n",
|
|
"9/9 [==============================] - 0s 4ms/step - loss: nan - accuracy: 0.5000 - val_loss: nan - val_accuracy: 0.4000\n",
|
|
"Epoch 10/15\n",
|
|
"9/9 [==============================] - 0s 4ms/step - loss: nan - accuracy: 0.5000 - val_loss: nan - val_accuracy: 0.4000\n",
|
|
"Epoch 11/15\n",
|
|
"9/9 [==============================] - 0s 4ms/step - loss: nan - accuracy: 0.5000 - val_loss: nan - val_accuracy: 0.4000\n",
|
|
"Epoch 12/15\n",
|
|
"9/9 [==============================] - 0s 4ms/step - loss: nan - accuracy: 0.5000 - val_loss: nan - val_accuracy: 0.4000\n",
|
|
"Epoch 13/15\n",
|
|
"9/9 [==============================] - 0s 4ms/step - loss: nan - accuracy: 0.5000 - val_loss: nan - val_accuracy: 0.4000\n",
|
|
"Epoch 14/15\n",
|
|
"9/9 [==============================] - 0s 4ms/step - loss: nan - accuracy: 0.5000 - val_loss: nan - val_accuracy: 0.4000\n",
|
|
"Epoch 15/15\n",
|
|
"9/9 [==============================] - 0s 3ms/step - loss: nan - accuracy: 0.5000 - val_loss: nan - val_accuracy: 0.4000\n"
|
|
]
|
|
},
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"<tensorflow.python.keras.callbacks.History at 0x25c7749a0d0>"
|
|
]
|
|
},
|
|
"execution_count": 28,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"model = tf.keras.models.Sequential()\n",
|
|
"model.add(tf.keras.layers.Dense(5,activation='sigmoid',input_shape=(2,)))\n",
|
|
"model.add(tf.keras.layers.Dense(1,activation='sigmoid'))\n",
|
|
"\n",
|
|
"test_x_norm = test_x-np.min(train_x) / (np.max(train_x)-np.min(train_x))\n",
|
|
"# This is not an error: we use min/man on training data when normalizing our test data\n",
|
|
"# Because we want to use the same normalization during both training and testing\n",
|
|
"\n",
|
|
"model.compile(tf.keras.optimizers.Adam(0.1),'binary_crossentropy',['accuracy'])\n",
|
|
"model.summary()\n",
|
|
"model.fit(train_x_norm,train_labels,validation_data=(test_x_norm,test_labels),batch_size=8,epochs=15)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "BmHNhUU8bqEX"
|
|
},
|
|
"source": [
|
|
"## Classification Loss Functions\n",
|
|
"\n",
|
|
"It is important to correctly specify loss function and activation function on the last layer of the network. The main rules are the following:\n",
|
|
"* If the network has one output (**binary classification**), we use **sigmoid** activation function, for **multiclass classification** - **softmax**\n",
|
|
"* If the output class is represented as one-hot-encoding, the loss function will be **cross entropy loss** (categorical cross-entropy), if the output contains class number - **sparse categorical cross-entropy**. For **binary classification** - use **binary cross-entropy** (same as **log loss**)\n",
|
|
"* **Multi-label classification** is when we can have an object belonging to several classes at the same time. In this case, we need to encode labels using one-hot encoding, and use **sigmoid** as activation function, so that each class probability is between 0 and 1.\n",
|
|
"\n",
|
|
"| Classification | Label Format | Activation Function | Loss |\n",
|
|
"|---------------|-----------------------|-----------------|----------|\n",
|
|
"| Binary | Probability of 1st class | sigmoid | binary crossentropy |\n",
|
|
"| Binary | One-hot encoding (2 outputs) | softmax | categorical crossentropy |\n",
|
|
"| Multiclass | One-hot encoding | softmax | categorical crossentropy |\n",
|
|
"| Multiclass | Class Number | softmax | sparse categorical crossentropy |\n",
|
|
"| Multilabel | One-hot encoding | sigmoid | categorical crossentropy |\n",
|
|
"\n",
|
|
"> Binary classification can also be handled as a special case of multi-class classification with two outputs. In this case, we need to use **softmax**.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "gZ-kWx84bMDH"
|
|
},
|
|
"source": [
|
|
"**Task 3**: \n",
|
|
"Use Keras to train MNIST classifier:\n",
|
|
"* Notice that Keras contains some standard datasets, including MNIST. To use MNIST from Keras, you only need a couple of lines of code (more information [here](https://www.tensorflow.org/api_docs/python/tf/keras/datasets/mnist))\n",
|
|
"* Try several network configuration, with different number of layers/neurons, activation functions.\n",
|
|
"\n",
|
|
"What is the best accuracy you were able to achieve?"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "yX6hqiafwHl9"
|
|
},
|
|
"source": [
|
|
"## Takeaways\n",
|
|
"\n",
|
|
"* Tensorflow allows you to operate on tensors at low level, you have most flexibility.\n",
|
|
"* There are convenient tools to work with data (`td.Data`) and layers (`tf.layers`)\n",
|
|
"* For beginners/typical tasks, it is recommended to use **Keras**, which allows to construct networks from layers\n",
|
|
"* If non-standard architecture is needed, you can implement your own Keras layer, and then use it in Keras models\n",
|
|
"* It is a good idea to look at PyTorch as well and compare approaches. \n",
|
|
"\n",
|
|
"A good sample notebook from the creator of Keras on Keras and Tensorflow 2.0 can be found [here](https://t.co/k694J95PI8)."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": []
|
|
}
|
|
],
|
|
"metadata": {
|
|
"celltoolbar": "Slideshow",
|
|
"colab": {
|
|
"collapsed_sections": [],
|
|
"name": "IntroKerasTF.ipynb",
|
|
"provenance": []
|
|
},
|
|
"interpreter": {
|
|
"hash": "0cb620c6d4b9f7a635928804c26cf22403d89d98d79684e4529119355ee6d5a5"
|
|
},
|
|
"kernelspec": {
|
|
"display_name": "Python 3.8.12 64-bit (conda)",
|
|
"name": "python3"
|
|
},
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3",
|
|
"version": "3.8.12"
|
|
},
|
|
"livereveal": {
|
|
"start_slideshow_at": "selected"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 0
|
|
}
|