diff --git a/3-NeuralNetworks/05-Frameworks/Overfitting.md b/3-NeuralNetworks/05-Frameworks/Overfitting.md new file mode 100644 index 00000000..9ca971de --- /dev/null +++ b/3-NeuralNetworks/05-Frameworks/Overfitting.md @@ -0,0 +1,44 @@ +# Overfitting + +Overfitting is an extremely important concept in machine learning, and it is very important to get it right! + +Consider the following simple problem of approximating 5 dots (represented by `x` on the graphs below): + +![](images/overfit1.jpg) | ![](images/overfit2.jpg) +-------------------------|-------------------------- +**Linear model, 2 parameters** | **Non-linear model, 7 parameters** +Training error = 5.3 | Training error = 0 +Validation error = 5.1 | Validation error = 20 + +* On the left, we see a good straight line approximation. Because the number of parameters is adequate, the model gets the idea behind point distribution right. +* On the right, the model is too powerful. Because we only have 5 points and model has 7 parameters, it can adjust in such a way as to pass through all points, making training error to be 0. However, it prevents the model from understanding the correct pattern behind data, thus validation error is very high. + +Thus it is very important to strike a correct balance between richness of the model (number of parameters) and the number of training samples. + +## Why overfitting occurs + + * Not enough training data + * Too powerful model + * Too much noise in input data + +## How to detect overfitting + +As you can see from the graph above, overfitting can be detected by very low training error, and high validation error. Normally during training we will see both training and validation errors starting to decrease, and then at some point validaton error might stop decreasing and start rising. This will be a sign of overfitting, and the indicator that we should probably stop training at this point (or at least make a snapshot of the model). + + + +## How to prevent overfitting + +If you can see that overfitting occurs, you can do one of the following: + + * Increase the amount of training data + * Decrease the complexity of the model + * Use some [regularization technique](../4-ComputerVision/08-TransferLearning/TrainingTricks.md), such as [Dropout](../4-ComputerVision/08-TransferLearning/TrainingTricks.md#Dropout), which we will consider later. + +## Overfitting and Bias-Variance Tradeoff + +Overfitting is actually a case of more generic problem in statistics, called [Bias-Variance Tradeoff](https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff). If we consider possible sources of error in our model, we can see two types of errors: +* **Bias errors** are caused by our algorithm not being able to capture the relationship between training data correctly. It can result from the fact that our model is not powerful enough (**underfitting**). +* **Variance errors**, which are caused by the model approximating noise in the input data instead of meaningful relationship (**overfitting**). + +During training, bias error decreases (as our model learns to approximate the data), and variance error increases. It is important to stop training - either manually (when we detect overfitting) or automatically (by introducing regularization) to prevent overfitting. diff --git a/3-NeuralNetworks/images/Overtitting.png b/3-NeuralNetworks/images/Overtitting.png new file mode 100644 index 00000000..fedba18c Binary files /dev/null and b/3-NeuralNetworks/images/Overtitting.png differ diff --git a/3-NeuralNetworks/images/overfit1.jpg b/3-NeuralNetworks/images/overfit1.jpg new file mode 100644 index 00000000..d1362521 Binary files /dev/null and b/3-NeuralNetworks/images/overfit1.jpg differ diff --git a/3-NeuralNetworks/images/overfit2.jpg b/3-NeuralNetworks/images/overfit2.jpg new file mode 100644 index 00000000..ab9a379b Binary files /dev/null and b/3-NeuralNetworks/images/overfit2.jpg differ diff --git a/4-ComputerVision/08-TransferLearning/Dropout.ipynb b/4-ComputerVision/08-TransferLearning/Dropout.ipynb new file mode 100644 index 00000000..a0b7cdef --- /dev/null +++ b/4-ComputerVision/08-TransferLearning/Dropout.ipynb @@ -0,0 +1,195 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## The Effect of Dropout\n", + "\n", + "Let's see for ourselves how dropout actually affects training. We will use MNIST dataset and a simple convolutional network to do that:" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": {}, + "outputs": [], + "source": [ + "from tensorflow import keras\n", + "import numpy as np\n", + "import matplotlib.pyplot as plt\n", + "\n", + "(x_train, y_train), (x_test, y_test) = keras.datasets.mnist.load_data()\n", + "\n", + "x_train = x_train.astype(\"float32\") / 255\n", + "x_test = x_test.astype(\"float32\") / 255\n", + "\n", + "x_train = np.expand_dims(x_train, -1)\n", + "x_test = np.expand_dims(x_test, -1)\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We will define `train` function that will take care of all training process, including:\n", + "* Defining the neural network architecture with a given dropout rate `d`\n", + "* Specifying suitable training parameters (optimizer and loss function)\n", + "* Doing the training and collecting the history\n", + "\n", + "We will then run this function for a bunch of different dropout values:" + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Training with dropout = 0\n", + "Epoch 1/5\n", + "938/938 [==============================] - 26s 27ms/step - loss: 0.1949 - acc: 0.9435 - val_loss: 0.0596 - val_acc: 0.9802\n", + "Epoch 2/5\n", + "938/938 [==============================] - 27s 29ms/step - loss: 0.0592 - acc: 0.9816 - val_loss: 0.0433 - val_acc: 0.9857\n", + "Epoch 3/5\n", + "938/938 [==============================] - 26s 28ms/step - loss: 0.0438 - acc: 0.9867 - val_loss: 0.0472 - val_acc: 0.9849\n", + "Epoch 4/5\n", + "938/938 [==============================] - 27s 28ms/step - loss: 0.0355 - acc: 0.9890 - val_loss: 0.0353 - val_acc: 0.9882\n", + "Epoch 5/5\n", + "938/938 [==============================] - 26s 28ms/step - loss: 0.0294 - acc: 0.9910 - val_loss: 0.0305 - val_acc: 0.9894\n", + "Training with dropout = 0.2\n", + "Epoch 1/5\n", + "938/938 [==============================] - 29s 31ms/step - loss: 0.2097 - acc: 0.9377 - val_loss: 0.0655 - val_acc: 0.9781\n", + "Epoch 2/5\n", + "938/938 [==============================] - 31s 33ms/step - loss: 0.0676 - acc: 0.9792 - val_loss: 0.0409 - val_acc: 0.9852\n", + "Epoch 3/5\n", + "938/938 [==============================] - 28s 30ms/step - loss: 0.0514 - acc: 0.9837 - val_loss: 0.0384 - val_acc: 0.9871\n", + "Epoch 4/5\n", + "938/938 [==============================] - 28s 29ms/step - loss: 0.0424 - acc: 0.9871 - val_loss: 0.0343 - val_acc: 0.9889\n", + "Epoch 5/5\n", + "938/938 [==============================] - 30s 32ms/step - loss: 0.0356 - acc: 0.9893 - val_loss: 0.0343 - val_acc: 0.9885\n", + "Training with dropout = 0.5\n", + "Epoch 1/5\n", + "938/938 [==============================] - 30s 31ms/step - loss: 0.2586 - acc: 0.9212 - val_loss: 0.0666 - val_acc: 0.9797\n", + "Epoch 2/5\n", + "938/938 [==============================] - 28s 30ms/step - loss: 0.0860 - acc: 0.9734 - val_loss: 0.0441 - val_acc: 0.9860\n", + "Epoch 3/5\n", + "938/938 [==============================] - 29s 31ms/step - loss: 0.0674 - acc: 0.9792 - val_loss: 0.0414 - val_acc: 0.9868\n", + "Epoch 4/5\n", + "938/938 [==============================] - 30s 32ms/step - loss: 0.0564 - acc: 0.9822 - val_loss: 0.0326 - val_acc: 0.9886\n", + "Epoch 5/5\n", + "938/938 [==============================] - 29s 31ms/step - loss: 0.0511 - acc: 0.9843 - val_loss: 0.0298 - val_acc: 0.9899\n", + "Training with dropout = 0.8\n", + "Epoch 1/5\n", + "938/938 [==============================] - 31s 32ms/step - loss: 0.3832 - acc: 0.8766 - val_loss: 0.0849 - val_acc: 0.9732\n", + "Epoch 2/5\n", + "938/938 [==============================] - 29s 31ms/step - loss: 0.1563 - acc: 0.9521 - val_loss: 0.0686 - val_acc: 0.9797\n", + "Epoch 3/5\n", + "938/938 [==============================] - 32s 34ms/step - loss: 0.1253 - acc: 0.9616 - val_loss: 0.0490 - val_acc: 0.9854\n", + "Epoch 4/5\n", + "938/938 [==============================] - 33s 35ms/step - loss: 0.1105 - acc: 0.9658 - val_loss: 0.0395 - val_acc: 0.9872\n", + "Epoch 5/5\n", + "938/938 [==============================] - 34s 36ms/step - loss: 0.1022 - acc: 0.9680 - val_loss: 0.0363 - val_acc: 0.9878\n" + ] + } + ], + "source": [ + "def train(d):\n", + " print(f\"Training with dropout = {d}\")\n", + " model = keras.Sequential([\n", + " keras.layers.Conv2D(32, kernel_size=(3, 3), activation=\"relu\", input_shape=(28,28,1)),\n", + " keras.layers.MaxPooling2D(pool_size=(2, 2)),\n", + " keras.layers.Conv2D(64, kernel_size=(3, 3), activation=\"relu\"),\n", + " keras.layers.MaxPooling2D(pool_size=(2, 2)),\n", + " keras.layers.Flatten(),\n", + " keras.layers.Dropout(d),\n", + " keras.layers.Dense(10, activation=\"softmax\")\n", + " ])\n", + " model.compile(loss='sparse_categorical_crossentropy',optimizer='adam',metrics=['acc'])\n", + " hist = model.fit(x_train,y_train,validation_data=(x_test,y_test),epochs=5,batch_size=64)\n", + " return hist\n", + "\n", + "res = { d : train(d) for d in [0,0.2,0.5,0.8] }" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Now, let's plot validation accuracy graphs for different dropout values to see how fast the training goes:" + ] + }, + { + "cell_type": "code", + "execution_count": 16, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "" + ] + }, + "execution_count": 16, + "metadata": {}, + "output_type": "execute_result" + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYcAAAD4CAYAAAAHHSreAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAABMqUlEQVR4nO3dd3gU1frA8e+b3gtJgDQIXQIIYkCk2BDFckWxoaBIEVBQ7GK/tqteu1cUUURQKerVn+WqgNio0kVDr6mQRnrd3fP7Y4awCYFsIMlukvN5nn2yOzNn5p1JMu+eM2fOiFIKTdM0TbPn5uwANE3TNNejk4OmaZp2HJ0cNE3TtOPo5KBpmqYdRycHTdM07Tgezg6gPoSHh6u4uDhnh6FpmtakbNy4MUspFVHTvGaRHOLi4tiwYYOzw9A0TWtSROTgiebpZiVN0zTtODo5aJqmacfRyUHTNE07TrO45lCTiooKUlJSKC0tdXYop83Hx4eYmBg8PT2dHYqmaS2EQ8lBRIYDbwLuwAdKqRerzQ8FPgQ6AaXAeKXU3+a86cDtgADvK6XeMKe3AhYDccAB4Aal1BFz3iPABMAK3K2UWlLXHUtJSSEwMJC4uDhEpK7FXYZSiuzsbFJSUujQoYOzw9E0rYWotVlJRNyBmcBlQDxwk4jEV1vsUWCLUupM4FaMRIKI9MRIDP2B3sCVItLFLDMDWK6U6gIsNz9jrnsU0AMYDrxjxlAnpaWlhIWFNenEACAihIWFNYsakKZpTYcj1xz6A3uUUvuUUuXAImBEtWXiMU7wKKV2AHEi0gboDqxVShUrpSzAb8A1ZpkRwDzz/Tzgarvpi5RSZUqp/cAeM4Y6a+qJ4ajmsh+apjUdjiSHaCDZ7nOKOc3en8BIABHpD7QHYoC/gfNEJExE/IDLgVizTBulVDqA+bN1HbaHiEwSkQ0isiEzM9OB3dA0TWseiiuK2Xh4I/MT5/PjgR8bZBuOXHOo6Wtr9YdAvAi8KSJbgL+AzYBFKbVdRF4ClgGFGEnEUg/bQyk1G5gNkJCQ4LIPpfjxxx+ZPn06VquViRMnMmPGDGeHpGlaE1JcUczOIzvZlr2NxKxEErMT2Z+3H2WeFi/rcBnD44bX+3YdSQ4pHPu2D0aNIM1+AaVUPjAOQIw2kP3mC6XUHGCOOe9f5voADotIpFIqXUQigQxHt9dUWK1Wpk6dyrJly4iJiaFfv35cddVVxMdXv2SjaZoGpZZSdh7ZSWJWopEMshPZl7cPm7IBEO4bTo+wHgyPG06P8B7Eh8UT7hveILE4khzWA11EpAOQinGx+Gb7BUQkBCg2r0lMBH43EwYi0loplSEi7TCans41i30DjMWodYwFvrabvkBEXgOigC7AulPeQydat24dnTt3pmPHjgCMGjWKr7/+WicHTdMot5az68iuytrAtuxt7Mndg1VZAWjl04oeYT0Y2m4oPcJ60CO8B639Wtey1vpTa3JQSllEZBqwBKMr64dKqUQRmWLOn4Vx4Xm+iFiBbRjdUI/6r4iEARXA1KPdVTGSwmciMgFIAq4315coIp+Z67GYZayns5NPf5vItrT801nFceKjgnjqHz1OukxqaiqxsccqQTExMfzxxx/1Goemaa6vwlrB7tzdJGYnVtYKdufuxmIzWtlDvEPoEdaD82LOq0wEbfzaOLUzikP3OSilvge+rzZtlt37NRjf8GsqO+QE07OBoSeY9zzwvCOxubKans+tex5pWvNWYatgX+6+ykSQmJ3IriO7qLBVABDoFUiPsB6MjR9b2TQU5R/lcueGZnuHtL3avuE3lJiYGJKTj3W8SklJISoqyimxaJpW/yw2C/vz9lepEew8spMyaxkAAZ4BxIfFM6b7GOLD4+kR1oOYgJjTSgRWm2J/VhHb0vPZnp5PXJgfN/ZrV1+7VKlFJAdn6devH7t372b//v1ER0ezaNEiFixY4OywNE07BVablYP5B41EYCaDHTk7KLUaN6j6efjRPaw7N3a7sbJpKDYwFjc59SHsisos7DhUwLb0fLalGclgx6F8SiuMC9QebsL1CbHc2K9edrEKnRwakIeHB2+//TaXXnopVquV8ePH06OHc2oxmqY5zqZsHMw/WNljKDErke052ymxlADg6+FL91bdua7rdcSHxdMjvAdxQXGnnAiUUmQUlLEtLb9KItifXcTR1ukgHw/io4K4uX974qOC6B4ZSOfWAXh71HkACYfo5NDALr/8ci6//HJnh6Fp2gkopUguSD6WCMyeQ0UVRQB4u3tzRqszuKbzNUYiCOtBh+AOuLud2knZYrWxL6vouESQXVReuUxsK1/iI4MY0Sea+Kgg4qOCiAr2adTrEjo5aJrWYiilSC1MrUwAR38WlBcA4OXmRbdW3biy45X0CDMuFncK6YSH26mdKgtKK4xmobT8ymSw83AB5RajWcjL3Y1ubQMZ2r018ZFBxEcFc0ZkIEE+zh+BWScHTdOaJaUUh4oOVUkEidmJ5JXlAeDh5kHX0K7GDWXmNYJOIZ3wdKv7iVkpRVpeKdvtagPb0vNJyimuXCbUz5MeUcGMPddoFoqPDKZjhD+e7q75WB2dHDRNc76SI7DvV+NlKQOfYPAOMn6e7GU27SilyCjOqNIstC17GzmlOQB4iAddQrtwcbuLK68RdAnpgpe7V51DLbfY2JNRyPb0qokgr6SicpkO4f70ig7mhoSYykTQJsjb5bqrnoxODpqmNT6bFdK2wJ6fjFfqBlA28A4G32AozYPSfGoYVg2ALHc3Er28SPQLJNHHh20ebmS5Gcu6A53cAznPJ5weEd3pEdSBriGd8PYLt0ssITgy7mhecUVll9GjiWB3RgEVVmNb3h5unBEZxOW9Is0kEEi3tkEEeDf9U2vT3wNN05qGgkOw92cjGez92agtIBDdF857EDoNheizwd08LdlsUF5Adt5BtmX8SWLOdhJz97CtKIWMCuMagRvQ0c2PgXgTb4Ue5RV0Ky7EtzQVyrYbCedkvALAJxjlE0y5RwB5Nj+yrT6kl3mTVORJaqkX+fiTr/wQn2D6RbTmqoS2xEVH0Tk2krjWIXi4aLPQ6dLJQdO0hmEph+S1sGe58Tr8lzE9oA10vQw6D4WOF4J/WGWRvLI8Eg8lVmkeSi9KB0AQ4oLj6B97fuXF4jNanYGfp1/N27fZoLzQrIUce1UU55KZmUFOdib5uVmUFuRgzc/Fz1pEkGQQJMUkSAkXUISbp11ysQGHzdef5jRPf7Mm4kATmH2t5WizmUfdm7Uai04ODay2Ibs//fRTXnrpJQACAgJ499136d27tzNC1bTTl7P/WM1g/+/GydnNE9oNgIv/CZ0vhjY9wWx7t9qs/JWxhdVpq1mVuoq/sv6qHIq6fVB7+rTuw+iw0cSHxdO9VXcCvAIcj8XNjRyrD9uzytmWVsH2dAvb0q3syQCLLRjogp+XO2e0DSS+m3FdID4qiNZtAvH1cgelakgu+dU+51b9XJgBWbuPfa5tWDgP3zokFbvEcjQZeXifym/JITo5NCBHhuzu0KEDv/32G6Ghofzwww9MmjRJD86nNR3lRXBgpXntYDnk7DWmh7SHM280kkGHIeAdWFnkcNFhIxmkrWJN2hryy/NxEzd6hvdkSu8pJLRJ4IywMwjyCnI4DJtNkZRTXOW+gW3p+aTnHXu8btsgH7pHHu02aiSC9q38cHM7wUViESNu70AIjqn7sVHKOD6leVBWPanUkFhK86A4yziGRz/bann8jYcP9LwOrp5Z9/hqoZNDA3JkyO6BAwdWvh8wYAApKSnHrUfTXIZSkLHdrB0sh4OrwVoOnn4QNwTOmWwkhFYdK2sH5dZyNqWvZVXqKlalrWL3kd0AtPZtzUXtLmJQ9CDOjTyXYO9gh0IorbCys9qQEtvT8ykqN76lu7sJnSMCGNAxjO6RgcRHBtM9MpCwgIb7ll0jEfAOMF7HP8yydkpBRUkNSaVaYmndvb4jB1pKcvhhBhz6q37X2bYXXPbiSRep65Ddc+bM4bLLLqu3EDWtXhztZrrnJ9jzMxSYz95qHW8kg05Dod254OlTWSQpP4mVqStZlbaK9YfWU2IpwdPNk75t+nLf2fcxKHoQXUK61Nq1M6vw+CEl9mYWYjM7MQV4exAfGcT1CbGViaBLmwB8PBtmSIlGJQJefsYrKLLRN98ykoOT1GXI7l9++YU5c+awcuXKhg5L007uRN1MfYKNC8idhxoJIfjYt+HiimLWJf9qJITUVaQUGjXgdoHtuLrz1QyKGkS/tv1OfPEY4//l112ZrNufU5kIMgrKKudHh/jSPTKIy3pFEm8mgphQ3xM3C2mnpWUkh1q+4TcUR4fs3rp1KxMnTuSHH34gLCzsuPma1uDq2M1UKcWunJ2sSlvFqtRVbMrYhMVmwdfDl3PansPYHmMZFDWI2KDYk2/XXNcvOzN4bdku/k7Nx9Nd6Nw6kCFdIsx7B4xB5kL8XLdnT3PUMpKDkzgyZHdSUhIjR47k448/pmvXrk6KVGtxTqGbaW5pLmuT1rIydSWr01aTWZIJQNfQrtwSfwuDowbTp3Ufh+86Vkqxak82ry7byeakXGJb+fLK9b35R+/IBhtpVHOcTg4N6ERDds+aZTxEb8qUKTzzzDNkZ2dz5513VpbZsGGDM8PWmqt66mYa5BXEwKiBDIoexMCogaf0XON1+3N4delO/tifQ1SwDy+M7MV1Z8e47DhDLZHU1C7e1CQkJKjqJ9Tt27fTvXvDXMV3hua2P1ojOFk3084X17mb6eCowQyKHkSPsB6nPFz15qQjvLZsFyt2ZxER6M20Czszqn+srik4iYhsVEol1DRP1xw0rbmw72a65ydIWtPo3UxP5O/UPF5ftovlOzJo5e/FY5d3Z8yA9sbNZtpJKaWwFRVhyczEmpWFJSsLS6b5MysLn+7daXXLmHrfrkPJQUSGA29ijGn1gVLqxWrzQ4EPgU5AKTBeKfW3Oe9eYCLGCFp/AeOUUqUishjoZq4iBMhVSvURkThgO7DTnLdWKTXllPdQ05ozJ3YzdcSuwwW8vmwXP/x9iCAfDx68tBtjB8Y1i4HpTpetrOzYyb7KCT8TS1YW1qOfs7NRpaXHr8DDA4+wMNwD63DXeB3U+hsSEXdgJjAMSAHWi8g3Sqltdos9CmxRSl0jImeYyw8VkWjgbiBeKVUiIp8Bo4CPlFI32m3jVSDPbn17lVJ9TnPfNK35sVkhbbN5Ifn0u5mO6DSCwdGDa+1mWlf7Mgt5c/luvvkzDX8vD+4e2oUJgzsQ7Ov8h9g0JGWxYMnJwZqdfdwJ31rtG7+toKDGdbi3aoVHWBgeEeH4tu+LR3gEHuHheESEGz/Dw3EPD8c9OBhxa7hrNI6k7/7AHqXUPgARWQSMAOyTQzzwAoBSaoeIxIlIG7tt+IpIBeAHpNmvXIyvJzcAF53Ojmhas1VwyEgGe5fXSzfTW3vcyqCoQbQLalfvoSbnFPPW8t18uTkVL3c3ppzfiUlDOhLq33S7oSqlsOXlHfuGn5Vd48nekpWFNScHariO6xYQUHli9+7WDf9Bg6qc8N3Dw40k0CoU8XSNBOpIcogGku0+pwDnVFvmT2AksFJE+gPtgRil1EYReQVIAkqApUqppdXKDgEOK6V2203rICKbgXzgcaXUCof3SNOaugboZjooahBntT7rlB5u44j0vBLe/nkPi9cn4+Ym3DYwjinndyIisJGHrKgDW3Fx1Sad7KyaT/hZWaiKiuPKi5eXcWKPCMczJgbf3r0rT/juZiLwiIjAIywMN19fJ+zh6XEkOdTU8Fg9Nb4IvCkiWzCuK2wGLOa1iBFAByAX+FxExiilPrErexOw0O5zOtBOKZUtImcD/yciPZRS+VWCEpkETAJo167+vwFpWqM62s10z3I4sMKp3UzrIqOglHd/3cunfyShlOKm/u2YemFn2gb71F64Aajyciw5OeYJPrPy5G5827dr4snMwlZcfPwK3NxwD2tV2ZTj3bmz+Y0/7Ni3e/PbvltgYJN6sltdOZIcUgD72xxjqNY0ZJ64x0FlM9F+83UpsF8plWnO+xIYCHxifvbAqHGcbbeuMqDMfL9RRPYCXYEqfVWVUrOB2WB0ZXVob52gtiG7f/31V0aMGEGHDh0AGDlyJE8++aQzQtUaU5Vupj9Bzj5j+imOZnpH7ztOu5tpXeQUlfPeb3uZt+YAFVbFdX1juGtoZ2JC6++6xVHKZsOam1t5wq/SYye7ahOPNTe3xnW4BQdXNuv49uxV9Rt+mF3zTmgo4q57UIFjyWE90EVEOgCpGBeUb7ZfQERCgGKlVDlGz6TflVL5IpIEDBARP4xmpaFUPclfDOxQSqXYrSsCyFFKWUWkI9AF2HeqO+hMjgzZDTBkyBC+++47J0WpNYpau5lOcbibaYRvRL12M62LvJIKPlixjw9X7qe4wsrVfaKZPrQLceH+p7VeZbVSmphI0erVlCclG008dr11sB7/XATx9a084XvFdcCvX79jzTlHv+GHGd/43bya7jUPZ6k1OSilLCIyDViC0ZX1Q6VUoohMMefPAroD80XEinGheoI57w8R+QLYBFgwmptm261+FFWblADOA54REQtgBaYopXJOYx+dxpEhu7VmrDTv2HhFp9HN1MPNg7Nbn13v3UzrorDMwtyV+3l/xT7ySy1c0SuSey7uQpc2gbUXPoGKjAyKVq2maMUKilavNr71i+DRpo3ZWycC7/jux0721XrsuPmfXkLSTs6hzsZKqe+B76tNm2X3fg3GN/yayj4FPHWCebfVMO2/wH8dictRL617iR05O+pzlZzR6gwe7v/wSZdxdMjuNWvW0Lt3b6KionjllVfo0aNHvcaqNRKlIHsP7FoCu340agc2yyl1M40NjG2wbqZ1UVJuZf6aA8z6bS9HiisYFt+Gey/uSnyU4w/iOUqVl1O8eQtFK1dQuGIlZTuM/0n38HACLrgA/8GD8R80EI/Q0PreDe0U6DtRGpAjQ3b37duXgwcPEhAQwPfff8/VV1/N7t27jyunuShLORxcBbuXGgnh6LWD1vEw8C7ocinE9HOJbqZ1UVphZeG6JGb+speswjLO7xrBfcO60js2pE7rKU9OpmjlSgpXrKR47VrjIrCHB359+xJx/30EDB6Md7duDdpfXzs1LSI51PYNv6E4MmR3UNCxb2CXX345d955J1lZWYSHhzdanFodFWaYyWAJ7P0FygvA3Rs6nAcD7oSul0LIsZO7s7uZ1kW5xcbnG5N5++c9pOeVcm7HMGaN6UtCXCuHytuKiylat46ilasoWrGC8oMHAfCMiSFoxFUEDBmCX/9zcA/QTUKurkUkB2dxZMjuQ4cO0aZNG0SEdevWYbPZ9DMdXI1SkP6nkQx2L4HUjcb0wCjodS10HU5xTAKp5bmkFaaRdmgVaXs+J7UwleSCZHbm7KzSzfRoV9OG7mZaFxarjS83p/LW8t2kHCnh7PahvHp9bwZ2PvmXFKUUZbt3G8lg5QqK129AVVQgPj74ndOf0DFjCBgyGM/27Zt1t8/mSCeHBuTIkN1ffPEF7777Lh4eHvj6+rJo0SL9T+QKyouMMYt2/Qi7l1FUeIhUD0/S2p5B6tnXkh4YQZoqJ7UojbTN/yJ3bW6V4t7u3kT6RxIdEM0dve9gYPRAeob1bJRupnVhtSm+25rGGz/tZn9WEb2ig3nu6p6c3zXihH+H1rw8itasoXDlSopWrMRy+DAA3l06VyYD37PPxs3bdW+A02qnh+xuIprb/riawvJCUtPWk7ZnCWlp60jNO0C6u5Dq6UWalzd5VO1K6ePuQ1RAFJEBkUT7RxMVEEV0gPEzKiCKMJ8wl07yNptiSeIhXv9pF7sOF3JG20DuG9aVYfFtjov7aDfTwhUrKFq5ipI//wSbDbegIPzPPZeAIYPxHzwYz7ZtnbQ32qnSQ3ZrLV5BeQFphWmkFqZW/kwvTCPtyG5Siw6Rr6oOj+AbHESUbwRRIZ04MzCm8qR/NBG08mnl0if/E1FK8fOODF5duott6fl0ivDn7ZvP4vKekVWexXyibqY+vXoRPmUy/oOH4HtmL8RDn0KaK/2b1ZqF/PL8Kif/ygRQlE5qYSoF5VVHwPTFjeiKCqIqyjnTaiM6MIaoyH5EdxpGVHR/Qr1Dm+TJ/0SUUqzck8WrS3exJTmX9mF+vHZDb0b0icbdTVDl5RTpbqaaHZ0cNJenlKo8+VcmgKKj3/7TSStMo6Ci2snfw5fogGiiA6LoE9Ce6KIcojL2En14O1EVFYT4tEK6Xmr0LOp4IfjUvd9+U/HHvmxeXbqLdQdyiA7x5aVrezGybwwqLZX8xYtq7mZ6330EDNHdTFsynRw0pzt68rdv8qlMBEVGAiisKKxSxs/Dj+jAaKL9ozm7zdmVzT5RAVFEe4cRnLYV2b0Uti6BvCSjUNsz4Zzp0HU4RJ0FzfyktynpCK8t3cXKPVm0DvTmueGduMKaRtmSj0l63K6baXS00c108GD8zhmgu5lqgE4OWiNQSpFXlkdqUdUmH/umn6KKoipl/D39zW/+0fRv27+y58/RC79BXkFVm33y04yupqs/NHoZVRQb4xZ1vADOux+6XAJBVe8xaa7+Ts3jtWW7+Hn7YXpbsnnHL5PuuxMpW7SRQ9W6mfoPHoRXXFyzakLT6odODtppU0qRW5Z73Ek/rehYIiixlFQpE+AZQHRANLGBsQyIHFDl5B8VEHX8yb86m82432DXj0ZSOLTVmB7SDvqMNmoHcYOrjFvU3O08VMA732wme8VKzs3ezV05u/HJzQZAmd1M/QcPwi8hQXcz1Wqlk0MDq23I7ry8PMaMGUNSUhIWi4UHHniAcePGOSnak1NKsTptNbuO7KrS9JNWlHbcyT/QK5DogGjaBbZjQOSAY00+ZgII8jqFNv7SfGMgu11LYM8yKMoEcYPYAXDx08b1g4gzKkc1bQmU1crelRtYtfA7ArZuYNKRg7gphQQGEjBwoO5mqp0ynRwakCNDds+cOZP4+Hi+/fZbMjMz6datG6NHj8bLxYYYVkrx+qbXmfv3XACCvIKIDogmLjiOgdEDifI/dvKPDIg8tZN/TbL3mrWDH+HganMguxDoMswYt6jzUPBzbGiH5uJoN9PM5b9SuHo1PsUFJCDktetE0LW3E3bhBbqbqXba9F9PA3JkyG4RoaCgAKUUhYWFtGrVCg8X+6dWSvHyhpf5eNvH3NjtRqb3nU6g16kP1XxSlnJjNNOjI5vm7DWmR3SHc6cZtYOY/pUD2bUEJxrN9Ih3IJvbdsN/8GAuHzeCHjG6dqDVnxbxH3boX/+ibHv9Dtnt3f0M2j766EmXcWTI7mnTpnHVVVcRFRVFQUEBixcvxs2FetEopXhh3Qss3LGQ0d1H83C/h+v/4mVhptFMtOtH47kHlQPZDYEBdxgXk0Pb1+82XVyNo5m6e3C4fTeW9LyCjW26MeDic5g6tCttglrOdRWt8bSI5OAsjgzZvWTJEvr06cPPP//M3r17GTZsGEOGDKkyWquz2JSN59Y+x+e7Pmds/FiujZvCxHkbEBFiQn3Nl1/l+2BfT8cSh1LGBeRd5jDXqRsBBYGR0HOkcTG54/ng1XK6VJ5wNNPoaLwvv4JfgjrxZk4QBW7e3JAQw9yLuhAd0vQeWq81HS0iOdT2Db+hODJk99y5c5kxYwYiQufOnenQoQM7duygf//+jR1uFTZl4+k1T/Pl7i+Z2Gsi54aOYeS7q7EpiAz2Ye2+bArLLFXKBHh71Jg0okP8iPG3EXJ4tXHvwa6l5lPRBKL7woWPGs1Fbc9sMReTHRnNVCWcw9yDVuauPkDJEStX9zUeydk+rOUkTc15WkRycBZHhuxu164dy5cvZ8iQIRw+fJidO3dWXqNwFqvNypOrn+Sbvd8wpfcUomwjGPPBOmJCffnwtn7EhfsbN66VWEg+UkzKkRJSKn8a7//Yl0NweToXum3mIrfNdHXbhkgFRfiy3S+B5Ha3UdTuIlq1iTGSSJAfoUBzTg2Vo5maA9hVH830aDfTIuXG3FUHeH/xPgpKLVx5ZiT3XNyVzq0DnLwHWkuik0MDcmTI7ieeeILbbruNXr16oZTipZdecuqDfiw2C4+tfIzv93/P1D5TKc28iHt/+pMBHVsxa8zZhPgZvahEhGA/T4L9gukZbT7g3mqBlPWwa7nxMJyMbQAUBbRnd+gNbPI5h9WWrhzMtZCaVEz+rgwgo3Lbfl7uVWod0SFVayCt/L2a3M1apTt3UvDTT1VHMw0MxP9oN9NBg/CMjASguNzC7DUHmfXbXnKLK7gkvg33DutK90jnNzFqLY8esruJaIz9qbBVMOP3GSw9uJRpfe5mx45+fLU5lWv7xvDCyF54edRwobw4x7z34EfY8xOUHAE3D2g/0Lh20OVSCO9c4/bySipIraHWkZprvM8rqTZSqqf7saaqKk1Xxs8wF0seRxYu5NCzz4FS+PTsad5zcPxopqUVVj79I4l3f91DVmE5F3aL4L5h3egVE+zE6LWWQA/ZrdWqwlrBg78/yPKk5dx55j0sXxPPugOpPHBJV6Ze2PnYSVcpyNx57M7k5D9AWcEvHLpeBl0vgU4XgU/tJ7ZgX0+CfT1P+LD6/NKjyeNYAkk9UkJKbjGbk3PJLa6aPHw83YgJ9TNrHFVrHTGhfoQHNE7yUEqR+drrZL//PgEXXEDkv57Ho9Xx92KUW2ws3pDMzJ/3cCi/lEGdw3hvWDfObq9HPtWcz6HkICLDgTcBd+ADpdSL1eaHAh8CnYBSYLxS6m9z3r3AREABfwHjlFKlIvJP4HYg01zNo0qp780yjwATACtwt1JqyenspHZy5dZy7v/1fn5N+ZWJ8fey+KcOpOXl8tZNZ3FVb7sL6Dn7YMEoyNppfG7bC4bcZ9QOovtCPT/lLMjHk6BIzxM2qxSUVhi1jJyqNY6UIyVsTcnlSLXk4e3hVq3GUfV9RID3aScPVV5O+hNPkPf1N4TceCNtn3j8uJvRLFYbX25K5a2fjUdy9osL5fUb+3BuJ/14WM111JocRMQdmAkMA1KA9SLyjVJqm91ijwJblFLXiMgZ5vJDRSQauBuIV0qViMhnwCjgI7Pc60qpV6ptL95cpgcQBfwkIl2VUlUfxeUApZRLNTOcqoZs+iuzlnHvL/eyInUFozvdy9wfYnATCwtvP4ez29t92y3MgI9HQmkuXPm6kRCCoxssLkcE+nhyRltPzmhbc/IoLLNUNlsdSxxGDeTv1DxyisqrLO/t4UZ0SPUmK+N9bKgv4QHeVR6IU521sJDUu++maPUaIu6ZTtjkyVX+/qw2xbd/pvHmcuORnL1jgnn+ml6c1yW8Wfydas2LIzWH/sAepdQ+ABFZBIwA7JNDPPACgFJqh4jEiUgbu234ikgF4Aek1bK9EcAipVQZsF9E9pgxrHFwnwDw8fEhOzubsDDXflxjbZRSZGdn4+NT/zc6lVhKmP7zdNamr+UfUXcz54e2tGvlxdzb+tMuzO/YgqX58Mm1UHgYxn4LMTU2UbqcAG8PurUNpFvbmu/mLiqzmEmj2K75yvi8NO0Q2dWSh5f70ZrH8bWOyIpCSu+/m7K9e4n8178IGXlNZTmbTfFj4iFeX7aL3RmFdI8M4oNbExjavXWT/tvUmjdHkkM0kGz3OQU4p9oyfwIjgZUi0h9oD8QopTaKyCtAElACLFVKLbUrN01EbgU2APcrpY6Y21tbbXvHfUUVkUnAJDC6g1YXExNDSkoKmZmZx81ranx8fIiJianXdRZXFHPXz3ex/tB6BgXfyYLlUQzs1Ip3R59NsJ/nsQUtZbB4DBxOhJsXN5nE4Ah/bw+6tgmka5uak0dxuVnzqFbrSDlSwrJth8kqNJJHbMFhnl39PkEVJbw/dAoFOVHE/HercWOgnxcL/0hiW3o+nVsH8M7ovgzv0fakNRBNcwWOJIea/oqrt3O8CLwpIlswritsBizmtYgRQAcgF/hcRMYopT4B3gWeNdf1LPAqMN7B7aGUmg3MBqO3UvX5np6edOjQwYHda3mKKoqYunwqmzM2c4b7JH5YG8sNCTE8d3W1Hkk2G3w1Gfb/BlfPMga7a0H8vDzo0iaQLidIHiXlVpJ/X03FjKexenmw6o7ncQ+IpPBICT9tzyCrsAyAuDA/3rixD//oHYW7TgpaE+FIckgBYu0+x1CtaUgplQ+MAxCjnrzffF0K7FdKZZrzvgQGAp8opQ4fLS8i7wPfObo97dQVlhdyx093sDXrL9qUjmfdvg48NLwbd5zfqWoTh1Lw4wxI/AqGPQN9bnJe0C6q4pfl2B58EJ/oaGLff58zY6pWcEsrrBzOLyU6xBcPd9cZL0vTHOHIX+x6oIuIdBARL4yLxd/YLyAiIeY8MHom/W4mjCRggIj4mUljKLDdLBNpt4prgL/N998Ao0TEW0Q6AF2Adae2e5q9/PJ8Ji+bzF9Zf+F7ZCwHk7oy8+a+3HlB5+Pbvle+BuveM0ZCHXi3cwJ2YTnz55N6zz349OhB+wWf4hVz/MV5H0932of568SgNUm11hyUUhYRmQYswejK+qFSKlFEppjzZwHdgfkiYsW4UD3BnPeHiHwBbAIsGM1Ns81V/1tE+mA0GR0AJptlEs1eTdvMMlNPpaeSVlVeWR6Tl01mR85OrIduwVLek4WTEujbroY+9Zvmw/JnoNcNMOzZFjPekSOUzUbGK6+S8+GHBA67mKiXX8atAToLaJqzNds7pLVjjpQeYdKySew+sofi5DG0801g7m39iG3ld/zCO76HxaOh44Vw0yLwcK2HDjmTrbyc9BmPkP/994SOHk2bRx9B3Ov33g5Na0z6DukWLLskm9uX3s6+3AMUJN3KuZEDmTm6L8G+nscvnLQWvhgHkX3ghvk6Mdix5ueTMu0uiteto/UD99NqwgTdDVVr1nRyaMaySrIYv2QCB/NSKDw4lht6XsgzI3riWVMbeMZ2WHAjBEXD6M/BW48AelRFejrJkyZTduAAUS//m+B//MPZIWlag9PJoZnKKM7gth/Gk1JwiKKksTx0/hVMOq9jzd9281KMm9w8fOCWr8DfeaPCuprSnbtInjQJW1ER7Wa/h/+55zo7JE1rFDo5NEOHig5x6/fjSC/MpCJ1PG9fcw2X9YqseeHiHGNYjLICGPdDi3sc58kUrf2DlLvuws3Xl/affoJPt27ODknTGo1ODs1MWmEao7+7jaySI3hmTuHjsdfRJzak5oXLi2HBDXDkANzyJbTt2ZihurS8//2P9BmP4Nm+He1mz8az2hP8NK2508mhGUkuSOamb28jt7SA8MK7+OT264gJraFHEoC1Aj6/DVI2GBef4wY3aqyuSilFztyPyPj3v/FLSCBm5tu4B+vnKmgtj04OzcSB3IPc+O1YispL6MYDzJ10DUE+NfRIAuPu52+nw+4lcMVrEH9V4wbropTVyuGXXuLI/I8JvGw4US++iJu3t7PD0jSn0MmhGdiRvZfR391GqaWCC4Kf4M1rLj/5XbnLn4Ytn8IFj0C/CY0XqAuzlZWR9uBDFCxdSquxY2n98EOIm76zWWu5dHJo4tanbuf2pROpsNm4pf0LPDz0gpP3v1/zDqx8HRLGw/kPN16gLsyam0vy1GmUbNxI6xkPE3bbbc4OSdOcTieHJmzZni3c//sd2JTwcO83uDWh+kjq1fz1BSx5BLr/Ay5/RQ+LAVSkppI0aTIVSUlEv/4aQZdd5uyQNM0l6OTQRC3YsoYXNt0DypPXhrzLJV17nbzA3p/hqynQfhCM/KDeH+nZFJVu307ypMnYysqInfMB/v37OzskTXMZOjk0Qa/+tpy5ex/BQ3x4/9I59IvucvICqZtg8S0Q0Q1GLQBPPVBc4apVpN49HbfAQOI+/QTvLrUcQ01rYXRyaEJsNsUD337L0uxn8XYLYMGVc+kWHnfyQtl74dPrwa8VjP4CfEMaI1SXlvf116Q99jjeHTsS+/5sPNu0qb2QprUwOjk0ESXlVsYv+oy/LK/i7xnMFyPmExt0/DMEqig4DB9fAygY8xUEneAu6RZCKUX2+x+Q+dpr+A0YQMx/3sI9sOanvGlaS6eTQxOQUVDKmE8Wkub9NqE+4Xw+Yh5t/duevFBpnjFeUlEWjP0Wwjs3TrAuSlmtHHruOXIXLiLoyiuJ+tfziJcedVbTTkQnBxe341A+YxcspCj0Pdr4RbLoHx8R4Rdx8kIVpbBoNGRuh5sXQ8zZjROsi7KVlJD6wIMULl9O2MQJRNx3n76HQdNqoZODC/t1ZwbTvlqMtJlLbGAMH1/xIeG+tYyYarPCV5PgwAoY+T50vrhxgnVRliNHSJlyByVbt9Lm8cdpNWa0s0PStCZBJwcX9fHagzzz05f4xMynY3Accy+bQyufVicvpBT88BBs+xoueR7OvKFxgnVR5cnJJN8+iYr0dKLffIOgSy5xdkia1mTo5OBirDbFC99vZ+6W/+Efs4AurToz55L3CfEJqb3w7y/D+g9g4N0wcFqDx+rKSv5OJHnyZJTFQru5H+LXt6+zQ9K0JkUnBxdSXG5h+qIt/JK8HP/YhcSHncF7w94j2NuBUUE3fgS/PA+9b4KLn27wWF1Z4YoVpEy/B4+QEGI/no93x47ODknTmhx9Vc5FHM4v5Yb31vBryk/4xyykV0QP3r/kfccSw/bv4Lt7ofMwuOo/0IIvtub+90uSp9yBV1x72i9aqBODpp0ih84iIjJcRHaKyB4RmVHD/FAR+UpEtorIOhHpaTfvXhFJFJG/RWShiPiY018WkR1mma9EJMScHiciJSKyxXzNqqd9dVnb0vK5euYq9peswi96Ib1bn8l7F79HoJcDffAPrIIvxkNUX7hhHrifYJjuZk4pRebMmaQ/9hj+AwbQfv7HeLZu7eywNK3JqjU5iIg7MBO4DIgHbhKR+GqLPQpsUUqdCdwKvGmWjQbuBhKUUj0Bd2CUWWYZ0NMsswt4xG59e5VSfczXlFPeuybglx0ZXD9rNeU+63Fvs5C+bfoy6+JZBHgF1F74cCIsvAlC2sHNn4GXf8MH7IKUxcKhJ58i6z9vEzxiBLGz3sU9oGUeC02rL47UHPoDe5RS+5RS5cAiYES1ZeKB5QBKqR1AnIgcHZPAA/AVEQ/AD0gzl1uqlLKYy6wFYk5rT5qg+WsOMGHeesIj/6Si1QL6R/bjnYvfwc/zBE9vs5ebZNzk5uVnPOLTP6zhA3ZBtuJiUqZOI/fzzwmbMpnIF19APFtm7UnT6pMjySEaSLb7nGJOs/cnMBJARPoD7YEYpVQq8AqQBKQDeUqppTVsYzzwg93nDiKyWUR+E5EhNQUlIpNEZIOIbMjMzHRgN1yH1aZ4+ttEnvw6kZ5nbCPH7xMGRg3k7YvextfDt/YVFGXDxyOhohjGfGnUHFogS3Y2B8feRuGKFbT95z9pfc89J3+WhaZpDnMkOdT036aqfX4RCBWRLcBdwGbAIiKhGLWMDkAU4C8iY6qsXOQxwAJ8ak5KB9oppc4C7gMWiEjQcQEoNVsplaCUSoiIqOWOYRdSVGZh0vwNzF11gPPO3sV+5nNezHm8edGb+Hg4MFpqeREsuB7ykuGmRdCmegtfy1B+8CAHbrqZst27iXn7P4SOutHZIWlas+JIV9YUINbucwxm09BRSql8YByAGF/d9puvS4H9SqlMc96XwEDgE/PzWOBKYKhSSpnrKgPKzPcbRWQv0BXYcGq76DoO5ZUyYd56tqfnM+K8vfyc+SEXxl7IK+e/gpe7A+P8WCvgs1shbTPc+Am0H9jwQbugkq1bSZ5sXIpq/9FcfPv0cW5AmtYMOVJzWA90EZEOIuKFcUH5G/sFRCTEnAcwEfjdTBhJwAAR8TOTxlBgu1lmOPAwcJVSqthuXRHmRXBEpCPQBdh3OjvpChLT8rh65ioOZBVx8yX7+TnzfYa1H8arF7zqWGKw2eDrabDnJ7jydTjjioYP2gUV/PILB28di1tAAHELF+jEoGkNpNbkYF40ngYswTixf6aUShSRKSJytCdRdyBRRHZg9Gqabpb9A/gC2AT8ZW5vtlnmbSAQWFaty+p5wFYR+dMsO0UplXP6u+o8y7cf5vpZa3ATuPnS3Xyd9B7D44bz0nkv4enm4MXTn56CrYvgwsfh7NsaNF5XdWTxZ6RMnYZ3587ELVyAV1ycs0PStGZLzNacJi0hIUFt2OCarU5zV+3n2e+20SMqmPPP2cy87bO5ouMVPDfoOTzcHLxBffV/YOnj0O92uPzlFvfsZ6UUmW+9Rfa7s/A//zxiXnsNN3/dVVXTTpeIbFRKJdQ0Tw+f0UAsVhvPfreNeWsOMqx7a7r3WM3cxA8Y0WkETw98GndHn+H852IjMcRfDZe91PISQ0UF6U8+Rd5XXxF83bVE/vOfiIf+s9W0hqb/yxpAYZmFuxZs4pedmUwcHIdf26XMTfyQa7tcy5PnPombODi8xe6f4Os7IW4IjJwNjiaUZsJaWETqPfdQtHIl4dOmET71Tt1VVdMaiU4O9Sw9r4TxH21g1+ECnh3Rg0MenzM3cT43druRR8951PHEkLLR6JnUujuM+hQ8vBs2cBdjycwkefIUSnfuJPK5Zwm57jpnh6RpLYpODvXor5Q8JsxbT3G5lTljE1iTO4cF2xYwuvtoHu73sOPferN2G/cy+IfD6P+CjwOD7zUjZfv2k3z77Vhycoh9ZyYB55/v7JA0rcVpucN31rOliYe44b01eLq78fmUAazIeY8FOxYwNn5s3RJDfrpx9zMCt3wFgW1qLdKcFG/azMGbbsJWWkr7+fN1YtA0J9HJ4TQppfhgxT4mf7KRrm0C+O+dA1i8/3U+2/UZE3pO4P6E+x1PDCW5xnhJJTkw5gsI69Sgsbua/GXLSBo3DveQEOIWLcS3V8/aC2ma1iB0s9JpsFht/PPbRD5Zm8TwHm155fpevLDhab7Z+w2Tz5zM1D5THU8MFaWw6GbI2gWjP4Oosxo2eBeT8+mnHH7ueXzO7EXsu+/i0aqWR6JqmtagdHI4RQWlFUxbsJnfdmUy+byO3H9JZ55Y/Tjf7/+eqX2mMqV3HUYat1nhvxPg4Cq4dg50uqjhAncxymYj8/XXyX7/AwIuuojoV1/BzdeBwQc1TWtQOjmcgtTcEiZ8tJ7dGYW8MLIX1yVE8siKR1hyYAnT+05nYq+Jjq9MKfjffbDjOxj+EvRqOb1yVHk5aY89Tv633xIy6kbaPv64vodB01yE/k+so60puUyYt4HScivzxvXnnI7BPPTbQ/yU9BMPJDzA2B5j67bCX180nv88+F4Y0Kyfa1SFtaCAlLvvpnjNWiLuuYewyZP0PQya5kJ0cqiDH/8+xD2LNxPm782nd55DXLg39/12H78m/8qM/jMY3X103Va4fg789iL0GQNDn2qQmF1RxeHDJE+aTNnevUS++AIhV1/t7JA0TatGJwcHKKV4f8U+XvhhB71jQnj/1gSC/OCeX+5hReoKHj/ncW48o47PE9j2Nfzvfug6HP7xZosZFqNszx6Sbp+ELS+P2FmzCBg8yNkhaZpWA50calFhtfHk14ksXJfEFb0iefWG3iAV3P3zdNakreGpc5/iuq51vE6wfwX8dyLE9IPr5oJ7y/g1FK9fT/LUaYi3F+0/+Rif+Jb5oCJNawpaxlnpFOWXVjD1002s2J3FnRd04oFLulFqLeHu5Xez7tA6nhn0DFd3vrpuKz30l9FlNbQD3LzYeAZ0C5D/44+kPfgQnrGxxM6ejVdM9SfNaprmSnRyOIHknGImzFvPvswi/n3tmdzQL5biimLuXH4nmzM28/zg5/lHp3/UbaVHDhg3uXkHwi1fgl/L6MufM28eh198Cd+zziL2nZm4h4Q4OyRN02qhk0MNtiTnMnHeesosNuaN78+gzuEUlhdy5/I72Zq5lReHvMhlHS6r20qLsoxhMSxlMP5rCI5pmOBdiLLZyPj3y+R89BGBw4YR9fK/cfNx4DnZmqY5nU4O1Xz/Vzr3Lt5C6yBvFk0aQOfWgeSX53PHT3ewLWsbL5//MsPaD6vbSssK4dPrID8Nbv3aGGm1mbOVl5M+Ywb53/9A6JgxtHlkBuLesoYc17SmTCcHk1KKWb/t46Ufd9C3XQizb00gPMCbvLI8Ji+bzM4jO3n1gle5qF0d7162lMNnt0D6VmPo7XbnNMwOuBBrXh4p0+6ieP16Wj/4AK3Gj9f3MGhaE6OTA0aPpMe/+pvFG5K58sxIXrm+Nz6e7uSW5nL7stvZm7uXNy54g/Nj6zhCqM1mPKxn788wYiZ0q2NTVBNUkZ5O8qRJlB04SNTLLxP8jyudHZKmaaegxSeHvJIK7vx0I6v2ZDPtws7cN6wrbm5CTmkOE5dOJCk/if9c9B8GRdexP75SxuM9//ochj4JZ41pmB1wIaU7d5E8aRK2oiLavT8b/wEDnB2SpmmnyKEhu0VkuIjsFJE9IjKjhvmhIvKViGwVkXUi0tNu3r0ikigif4vIQhHxMae3EpFlIrLb/BlqV+YRc1s7ReTS+tjRmiTnFHPtu6tZtz+Hl687kwcu7Yabm5BVksWEJRNIzk/m7aFv1z0xAKx+C9bOhHOmwOD76j94F1O0di0HRxt3iLf/9BOdGDStias1OYiIOzATuAyIB24Skep3Lz0KbFFKnQncCrxplo0G7gYSlFI9AXdglFlmBrBcKdUFWG5+xlz3KKAHMBx4x4yh3llsCovVxvzx53B9QiwAGcUZjF8yntTCVN65+B0GRJ7CSW7LAlj2JPQYCZe+0Ozvfs777n8k3T4Jz8i2xC1aiE+3bs4OSdO00+RIzaE/sEcptU8pVQ4sAkZUWyYe4wSPUmoHECciRx9h5gH4iogH4AekmdNHAPPM9/OAq+2mL1JKlSml9gN7zBjqXYdwf36673zO7RQGwKGiQ4xfMp7DRYeZdfEs+rXtV/eV7loCX0+DDufDNbPArfk+T0kpRfacOaQ98AB+ffrQ/tNP8YyMdHZYmqbVA0fOXNFAst3nFHOavT+BkQAi0h9oD8QopVKBV4AkIB3IU0otNcu0UUqlA5g/W9dhe4jIJBHZICIbMjMzHdiNmnm4G4cgrTCNcT+OI7skm/eGvUffNn3rvrLk9fDZWGjbE278BDy8TzkuV6esVg4//y8yXn6FwMuGEzvnA9yDgpwdlqZp9cSR5FBTm4iq9vlFIFREtgB3AZsBi3kdYQTQAYgC/EWktiuzjmwPpdRspVSCUiohIiKillWeXEpBCuN+HEdeeR6zh82mT+s+dV9J5k5YcD0EtoXRX4BP8z1R2kpLSb3nXo588gmtbruN6Fdfxc3Ly9lhaZpWjxzprZQCxNp9juFY0xAASql8YByAGB3a95uvS4H9SqlMc96XwEDgE+CwiEQqpdJFJBLIcHR79SkpP4kJSydQYinhg0s+ID7sFAaDy0s17n528zSGxQhoXXuZJsqam0vynVMp2byZNo/MoNXYOj6/QtO0JsGRmsN6oIuIdBARL4yLxd/YLyAiIeY8gInA72bCSAIGiIifmTSGAtvN5b4Bjp5ZxgJf200fJSLeItIB6AKsO7XdO7kDeQcY9+M4yixlzLlkzqklhpIjxnhJpXkw5gto1bH+A3URFYczOHDzaEr/+ovo11/TiUHTmrFaaw5KKYuITAOWYPQ2+lAplSgiU8z5s4DuwHwRsQLbgAnmvD9E5AtgE2DBaG6aba76ReAzEZmAkUSuN8skishn5noswFSllLW+dtiep7snrf1a88ygZ+gS2qXuK6gogQWjIGev0ZQU2bv+g3QRFRkZJI0diyUjg9g5H+Dfv0H6CGia5iJEqeOa85uchIQEtWHDhlMqq5Q6taEdrBZjWIydP8B1H0LPkae0/abAkpnJwbG3UXHoEO3en43f2Wc7OyRN0+qBiGxUSiXUNK/F3yF9SolBKfjfvbDze7js5eadGLKzOThuHBXp6bSb/Z5ODJrWQjTfTvgN6ZfnYdN8OO9BOGeSs6NpMJacHJJuu42K1DRi35uFX79TuO9D07QmqcXXHOrsj9nw+8vQ91a48DFnR9NgLEeOkHTbOMqTU4idNUtfY9C0FkYnh7r4+0v44SHodjlc8XqzHRajMjEcPEjsrHfxH9D8hxnXNK0qnRwcte83+GoytBtgXIB2b56HzpqbS9L4CZTv30/MO+/gf+65zg5J0zQnaJ5nuPqW/icsGg2tOsFNC8HT19kRNQhrXp6RGPbsIeadmQQMPoXRaDVNaxb0Bena5OyDT64Dn2AY81/wDa29TBNkzc8nacJEynbvJubt/xAwZIizQ9I0zYl0zeFkCjOMYTFsFXDbdxB83Ph/zYK1oICkibdTunMnMW+9ScD5dXzinaZpzY5ODidSmm8Mi1FwCMZ+CxHN8xkF1sJCkifeTun27cS8+QaBF17o7JA0TXMBOjnUxFIGi8fA4US4aRHENs/+/dbCIpJvn0RJYiIxb7xO4EUXOTskTdNchE4O1dls8NUU2P8bXD0Lul7i7IgahK2oiOTJkynZupXo114j8OKLnR2SpmkuRF+QtqcU/DgDEr+Ei5+GPjc5O6IGYSsuJnnyFEq2bCH61VcIurR5JkBN006drjnYW/karHsPBkyFQdOdHU2DsBUXkzzlDoo3bSL6lZcJGj7c2SFpmuaCdM3hqE0fw/JnoNf1cMlzzfLuZ1tJCcl33Enxhg1E/fvfBF1+ubND0jTNRemaAxjDbn87HTpdBCPeAbfmlzNtpaWkTJ1K8bp1RL30IsFXXuHskDRNc2HN7yxYV0l/wOe3QeSZcMPH4NH8noVsKysjZeo0itasJfKFfxF81VXODknTNBfXspNDxg5YcAMERcPNn4N3gLMjqne2sjJSpt1F0erVRD73HCFXX+3skDRNawJadnLwCYKos+CWLyEgwtnR1DtbeTkpd99N0YoVRD77DCHXNt+HEmmaVr9a9jWHoCi49f+cHUWDsJWXk3r3dIp++522Tz9NyHXXOTskTdOakJZdc2imVHk5qffcS+Gvv9L2n08ReuMNzg5J07QmRieHZkZVVJB6//0U/vwzbZ54nNBRo5wdkqZpTZBDyUFEhovIThHZIyIzapgfKiJfichWEVknIj3N6d1EZIvdK19E7jHnLbabfkBEtpjT40SkxG7erPrb3ebNSAwPULDsJ9o89hitRo92dkiapjVRtV5zEBF3YCYwDEgB1ovIN0qpbXaLPQpsUUpdIyJnmMsPVUrtBPrYrScV+ApAKXWj3TZeBfLs1rdXKdXnNParxVEWC6kPPkTB0qW0eWQGrW4Z4+yQNE1rwhypOfQH9iil9imlyoFFwIhqy8QDywGUUjuAOBFpU22ZoRgn/YP2E0VEgBuAhacQv4aRGNIeepiCH3+k9UMP0WrsWGeHpGlaE+dIcogGku0+p5jT7P0JjAQQkf5AeyCm2jKjqDkBDAEOK6V2203rICKbReQ3EanxkWQiMklENojIhszMTAd2o3lSVitpMx4h//vvaf3gA4SNH+fskDRNawYcSQ41DTKkqn1+EQg1rxvcBWwGLJUrEPECrgI+r2FdN1E1aaQD7ZRSZwH3AQtEJOi4AJSarZRKUEolREQ0v3sUHKGsVtIeeYT8774j4r77CJswwdkhaZrWTDhyn0MKEGv3OQZIs19AKZUPjIPKZqL95uuoy4BNSqnD9uVExAOjxnG23brKgDLz/UYR2Qt0BTY4tkstg7JaSX/scfK/+ZaIe6YTPul2Z4ekaVoz4kjNYT3QRUQ6mDWAUcA39guISIg5D2Ai8LuZMI6qXjs46mJgh1IqxW5dEebFa0SkI9AF2OfoDrUEymYj/Yknyfu//yP8rmmET5ni7JA0TWtmaq05KKUsIjINWAK4Ax8qpRJFZIo5fxbQHZgvIlZgG1DZviEifhg9nSbXsPqarkOcBzwjIhbACkxRSuXUec+aKWWzceipp8j78kvC77yTiKlTnR2SpmnNkChV/fJB05OQkKA2bGj+rU7KZuPQ08+Qu3gxYVMmEzF9OtIMnzuhaVrjEJGNSqmEmubpO6SbCKUUh5591kgMkybpxKBpWoPSyaEJUEpx+NnnyF24iLCJE4i49x6dGDRNa1A6Obg4pRSH//UCRxYsoNW4cUTcf79ODJqmNTidHFyYUoqMF1/iyMcf02rsrbR+6EGdGDRNaxQ6ObgopRQZL79Czrx5hN5yC61nzNCJQdO0RqOTgwtSSpH56qvkfPghoTffTJtHH9GJQdO0RqWTg4tRSpH5+htkfzCHkJtG0eaJx3Vi0DSt0enk4EKUUmS+9RbZs2cTcsMNtH3iCZ0YNE1zCp0cXEjW2zPJfncWIddfR9t/PoW46V+PpmnOoc8+LiLznXfImjmT4JEjafv00zoxaJrmVPoM5AKyZr1H1lv/IXjECCKffUYnBk3TnE6fhZwsa/b7ZL7xBkFX/YPIfz2PuLs7OyRN0zSdHJwpe84cMl97jaArryTqhRd0YtA0zWXo5OAk2XM/IuPlVwi6/HKiXtSJQdM016KTgxPkzJtHxksvETh8OFH/fgnxcOSBfJqmaY1HJ4dGlvPxJxx+4UUCL7mE6Jf/rRODpmkuSSeHRpSzYAGHn3+egIuHEv3qK4inp7ND0jRNq5FODo3kyKJFHH7mWQIuuoiY117TiUHTNJemk0MjOLL4Mw7982kCLriAmDdeR7y8nB2SpmnaSenk0MByv/iCQ089hf/55xH91ps6MWia1iQ4lBxEZLiI7BSRPSIyo4b5oSLylYhsFZF1ItLTnN5NRLbYvfJF5B5z3j9FJNVu3uV263vE3NZOEbm0nva10eV++RXpTzyJ/5AhxLz1Fm46MWia1kTU2lVGRNyBmcAwIAVYLyLfKKW22S32KLBFKXWNiJxhLj9UKbUT6GO3nlTgK7tyryulXqm2vXhgFNADiAJ+EpGuSinrKe6jU+T+3/+R/thj+J97LjFv/wc3b29nh6RpmuYwR2oO/YE9Sql9SqlyYBEwotoy8cByAKXUDiBORNpUW2YosFcpdbCW7Y0AFimlypRS+4E9ZgxNRt6335L+yKP4DTiHmHdm6sSgaVqT40hyiAaS7T6nmNPs/QmMBBCR/kB7IKbaMqOAhdWmTTOboj4UkdA6bM9l5X33P9IenoFf//7EvvMObj4+zg5J0zStzhxJDjU9bUZV+/wiECoiW4C7gM2ApXIFIl7AVcDndmXeBTphNDulA6/WYXuIyCQR2SAiGzIzMx3YjYaX//33pD30EH4JCcS++w5uvr7ODknTNO2UOHJ7bgoQa/c5BkizX0AplQ+MAxDj0WX7zddRlwGblFKH7cpUvheR94HvHN2eWX42MBsgISHhuOTR2PJ//JHUBx/Ct+9ZxM56Fzc/P2eHpGmadsocqTmsB7qISAezBjAK+MZ+AREJMecBTAR+NxPGUTdRrUlJRCLtPl4D/G2+/wYYJSLeItIB6AKsc3SHnCF/6VJS738A3969affeezoxaJrW5NVac1BKWURkGrAEcAc+VEolisgUc/4soDswX0SswDZgwtHyIuKH0dNpcrVV/1tE+mA0GR04Ot9c92fmeizAVFfuqVTw00+k3nc/vr16ETt7Nm7+/s4OSdM07bSJUk5vkTltCQkJasOGDY2+3YKffyFl+nR84+OJnfMB7gEBjR6DpmnaqRKRjUqphJrm6TukT1HBL0Zi8OnendgP3teJQdO0ZkUnh1NQ+NtvpN49HZ9u3Wj3wfu4BwY6OyRN07R6pZNDHRWuWEnKXXfj3aUL7eZ8gHtQkLND0jRNq3c6OdRB4apVpEydilenTrT7cA7uwcHODknTNK1B6OTgoKI1a0i5cypeHToYiSEkxNkhaZqmNRidHBxQtPYPku+4E6/27Wn30Vw8QkNrL6RpmtaE6eRQi6J160ieMgWv2BidGDRNazF0cjiJ4g0bSJ48Bc/oaNrNnYtHq1bODknTNK1R6ORwAsWbNpE0aTKekZG0/2guHuHhzg5J0zSt0ejkUIPizZtJnng7nq1bG01JERHODknTNK1R6eRQTcmff5I88XbcI8JpN+8jPFu3dnZImqZpjU4nBzslW7eSNGEi7mFhtJ83D8821R9mp2ma1jLo5GAq+etvIzGEhtJ+3kd4tm3r7JA0TdOcRicHoCQxkaQJE3APCjISQ2Rk7YU0TdOasRafHEq3bydp/ATcAvxpN28enlFRzg5J0zTN6Vp0cijduYukceNx8/Oj/fz5eMVEOzskTdM0l9Cik4N7aAg+8fG0n/cRXjExzg5H0zTNZdT6mNDmzLN1a9p9OMfZYWiaprmcFl1z0DRN02qmk4OmaZp2HJ0cNE3TtOM4lBxEZLiI7BSRPSIyo4b5oSLylYhsFZF1ItLTnN5NRLbYvfJF5B5z3ssissMs85WIhJjT40SkxK7MrPrbXU3TNM0RtSYHEXEHZgKXAfHATSISX22xR4EtSqkzgVuBNwGUUjuVUn2UUn2As4Fi4CuzzDKgp1lmF/CI3fr2Hi2nlJpyynunaZqmnRJHag79gT1KqX1KqXJgETCi2jLxwHIApdQOIE5Eqg9MNBTjpH/QXG6pUspizlsL6L6kmqZpLsKR5BANJNt9TjGn2fsTGAkgIv2B9hx/sh8FLDzBNsYDP9h97iAim0XkNxEZUlMBEZkkIhtEZENmZqYDu6FpmqY5ypHkIDVMU9U+vwiEisgW4C5gM3C0VoCIeAFXAZ8ft3KRx8xlPzUnpQPtlFJnAfcBC0Qk6LgAlJqtlEpQSiVE6OctaJqm1StHboJLAWLtPscAafYLKKXygXEAIiLAfvN11GXAJqXUYftyIjIWuBIYqpRS5rrKgDLz/UYR2Qt0BTacKMCNGzdmichBB/blRMKBrNMo31B0XHWj46obHVfdNMe42p9ohiPJYT3QRUQ6AKkYzUM32y9g9jQqNq9JTAR+NxPGUTdRrUlJRIYDDwPnK6WK7aZHADlKKauIdAS6APtOFqBS6rSqDiKyQSmVcDrraAg6rrrRcdWNjqtuWlpctSYHpZRFRKYBSwB34EOlVKKITDHnzwK6A/NFxApsAybYBe4HDAMmV1v124A3sMyobLDW7Jl0HvCMiFgAKzBFKZVzerupaZqm1YVDYysppb4Hvq82bZbd+zUY3/BrKlsMhNUwvfMJlv8v8F9H4tI0TdMahr5D2jDb2QGcgI6rbnRcdaPjqpsWFZeY14E1TdM0rZKuOWiapmnH0clB0zRNO06LSQ4ODB4oIvKWOX+riPR1kbguEJE8u4EIn2ykuD4UkQwR+fsE8511vGqLq9GPl4jEisgvIrJdRBJFZHoNyzjreDkSmzOOmY85SOefZlxP17BMox8zB+Ny1v+kuxgjR3xXw7z6P1ZKqWb/wuiCuxfoCHhhDPcRX22ZyzGG8BBgAPCHi8R1AfCdE47ZeUBf4O8TzG/04+VgXI1+vIBIoK/5PhBjIEmn/33VITZnHDMBAsz3nsAfwABnHzMH43LW/+R9wIKatt0Qx6ql1BwcGTxwBDBfGdYCISIS6QJxOYVS6nfgZPeXOON4ORJXo1NKpSulNpnvC4DtHD/+mLOOlyOxNTrzOBSaHz3NV/XeMY1+zByMq9GJSAxwBfDBCRap92PVUpKDI4MHOrKMM+ICONes5v4gIj0aOCZHOeN4Ocppx0tE4oCzML5x2nP68TpJbOCEY2Y2k2wBMoBlSimXOGYOxAWNf7zeAB4CbCeYX+/HqqUkB0cGD3RkmfrmyDY3Ae2VUr2B/wD/18AxOcoZx8sRTjteIhKAcQPnParq8DHg5ONVS2xOOWZKKasynvUSA/QX8yFhdpxyzByIq1GPl4hcCWQopTaebLEapp3WsWopyaHWwQMdXKbR41JK5R+t5irjTnVPEQlv4Lgc4YzjVStnHS8R8cQ4+X6qlPqyhkWcdrxqi83Zf2NKqVzgV2B4tVlO/Rs7UVxOOF6DgKtE5ABG0/NFIvJJtWXq/Vi1lORQOXigGMOHjwK+qbbMN8Ct5lX/AUCeUird2XGJSFsRY/ApMZ6V4QZkN3BcjnDG8aqVM46Xub05wHal1GsnWMwpx8uR2Jx0zCLk2KOBfYGLgR3VFmv0Y+ZIXI19vJRSjyilYpRScRjniJ+VUmOqLVbvx8qhsZWaOuXY4IHfY1zx34PxONNxLhLXdcAdYgxEWAKMUmb3hIYkIgsxemWEi0gK8BTGxTmnHS8H43LG8RoE3AL8ZbZVg/Ho3HZ2cTnleDkYmzOOWSQwT4zHELsBnymlvnP2/6SDcTnlf7K6hj5WevgMTdM07TgtpVlJ0zRNqwOdHDRN07Tj6OSgaZqmHUcnB03TNO04OjlomqZpx9HJQdM0TTuOTg6apmnacf4fg/rFEqgwEDoAAAAASUVORK5CYII=", + "text/plain": [ + "
" + ] + }, + "metadata": { + "needs_background": "light" + }, + "output_type": "display_data" + } + ], + "source": [ + "for d,h in res.items():\n", + " plt.plot(h.history['val_acc'],label=str(d))\n", + "plt.legend()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "From this graph, you would probably be able to see the following:\n", + "* Dropout values in the 0.2-0.5 range, you will see the fastest training the best overall results\n", + "* Without dropout ($d=0$), you are likely to see less stable and slower training process\n", + "* High dropout (0.8) makes things worse" + ] + } + ], + "metadata": { + "interpreter": { + "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5" + }, + "kernelspec": { + "display_name": "Python 3.9.5 64-bit ('base': conda)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.8.12" + }, + "orig_nbformat": 4 + }, + "nbformat": 4, + "nbformat_minor": 2 +} diff --git a/4-ComputerVision/08-TransferLearning/TrainingTricks.md b/4-ComputerVision/08-TransferLearning/TrainingTricks.md new file mode 100644 index 00000000..1afe030d --- /dev/null +++ b/4-ComputerVision/08-TransferLearning/TrainingTricks.md @@ -0,0 +1,108 @@ +# Deep Learning Training Tricks + +As neural networks become deeper, the process of their training becomes more and more challenging. One major problem is so-called [vanishing gradients](https://en.wikipedia.org/wiki/Vanishing_gradient_problem) or [exploding gradients](https://deepai.org/machine-learning-glossary-and-terms/exploding-gradient-problem#:~:text=Exploding%20gradients%20are%20a%20problem,updates%20are%20small%20and%20controlled.). [This post](https://towardsdatascience.com/the-vanishing-exploding-gradient-problem-in-deep-neural-networks-191358470c11) gives a good introduction into those problems. + +To make training deep networks more efficient, there are a few techniques that can be used. + +## Keeping values in reasonable interval + +To make numerical computations more stable, we want to make sure that all values within our neural network are within reasonable scale, typically [-1..1] or [0..1]. It is not a very strict requirement, but the nature of floating point computations is such that values of different magnitudes cannot be accurately manipulated together. For example, if we add 10-10 and 1010, we are likely to get 1010, because smaller value would be "converted" to the same order as the larger one, and thus mantissa would be lost. + +Most activation functions have non-linearities around [-1..1], and thus it makes sense to scale all input data to [-1..1] or [0..1] interval. + +## Initial Weight Initialization + +Ideally, we want the values to be in the same range after passing through network layers. Thus it is important to initialize weights in such a way as to preserve the distribution of values. + +Normal distribution **N(0,1)** is not a good idea, because if we have *n* inputs, the standard deviation of output would be *n*, and values are likely to jump out of [0..1] interval. + +The following initializations are often used: + + * Uniform distribution -- `uniform` + * **N(0,1/n)** -- `gaussian`) + * **N(0,1/√n_in)** guarantees that for inputs with zero mean and standard deviation of 1 the same mean/standard deviation would remain + * **N(0,√2/(n_in+n_out)** -- so-called **Xavier initialization** (`glorot`), it helps to keep the signals in range during both forward and backward propagation + +## Batch Normalization + +Even with proper weight initialization, weights can get arbitrary big or small during the training, and they will bring signals out of proper range. We can bring signals back by using one of **normalization** techniques. While there are several of them (Weight normalization, Layer Normalization), the most often used is Batch Normalization. + +The idea of **batch normalization** is to take into account all values across the minibatch, and perform normalization (i.e. subtract mean and divide by standard deviation) based on those values. It is implemented as a network layer that does this normalization after applying the weights, but before activation function. As a result, we are likely to see higher final accuracy and faster training. + +Here is the [original paper](https://arxiv.org/pdf/1502.03167.pdf) on batch normalization, the [explanation on Wikipedia](https://en.wikipedia.org/wiki/Batch_normalization), and [a good introductory blog post](https://towardsdatascience.com/batch-normalization-in-3-levels-of-understanding-14c2da90a338) (and the one [in Russian](https://habrahabr.ru/post/309302/)). + +## Dropout + +**Dropout** is an interesting technique that removes a certain percentage of random neurons during training. It is also implemented as a layer with one parameter (percentage of neurons to remove, typically 10%-50%), and during training it zeroes random elements of the input vector, before passing it to the next layer. + +While this may sound like a strange idea, you can see the effect of dropout on training MNIST digit classifier in [`Dropout.ipynb`](Dropout.ipynb) notebook. It speeds up training and allows us to achieve higher accuracy in less training epochs. + +This effect can be explained in several ways: + * It can be considered to be a random shocking factor to the model, which takes optimiation out of local minimum + * It can be considered as *implicit model averaging*, because we can say that during dropout we are training slightly different model + +> *Some people say that when a drunk person tries to learn something, he will remember this better next morning, comparing to a sober person, because a brain with some malfunctioning neurons tries to adapt better to gasp the meaning. We never tested ourselves if this is true of not* + +## Preventing overfitting + +One of the very important aspect of deep learning is too be able to prevent [overfitting](../3-NeuralNetworks/05-Frameworks/Overfitting.md). While it might be tempting to use very powerful neural network model, we should always balance the number of model parameters with the number of training samples. + +> Make sure you understand the concept of [overfitting](../3-NeuralNetworks/05-Frameworks/Overfitting.md) we have introduced earlier! + +There are several ways to prevent overfitting: + + * Early stopping -- continuously monitor error on validation set and stopping training when validation error starts to increase. + * Explicit Weight Decay / Regularization -- adding an extra penalty to the loss function for high absolute values of weights, which prevents the model of getting very unstable results + * Model Averaging -- training several models and then averaging the result. This helps to minimize the variance. + * Dropout (Implicit Model Averaging) + +## Optimizers / Training Algorithms + +Another important aspect of training is to chose good training algorithm. While classical **gradient descent** is a reasonable choice, it can sometimes be too slow, or result in other problems. + +In deep learning, we use **Stochastic Gradient Descent** (SGD), which is a gradient descent applied to minibatches, randomly selected from the training set. Weights are adjusted using this formula: + +wt+1 = wt - η∇ℒ + +### Momentum + +In **momentum SGD**, we are keeping a portion of a gradient from previous steps. It is similar to when we are moving somewhere with inertia, and we receive a punch in a different direction, our trajectory does not change immediately, but keeps some part of the original movement. Here we introduce another vector v to + +vt+1 = γ vt - η∇ℒ + +wt+1 = wt+vt+1 + +Here parameter γ indicated the extent to which we take inertia into account: γ=0 corresponds to classical SGD; γ=1 is a pure motion equation. + +### Adam, Adagrad, etc. + +Since in each layer we multiply signals by some matrix Wi, depending on ||Wi||, the gradient can either diminish and be close to 0, or rise indefinitely. It is the essence of Exploding/Vanishing Gradients problem. + +One of the solutions to this problem is to use only direction of the gradient in the equation, and ignore the absolute value, i.e. + +wt+1 = wt - η(∇ℒ/||∇ℒ||) + +where ||∇ℒ|| = √∑(∇ℒ)2 + +This algorithm is called **Adagrad**. Another algorithms that use the same idea: **RMSProp**, **Adam** + +> **Adam** is considered to be a very efficient algorithm for many applications, so if you are not sure which one to use - use Adam. + +### Gradient clipping + +Gradient clipping is an extension the idea above. When the ||∇ℒ|| ≤ θ, we consider the original gradient in the weight optimization, and when ||∇ℒ|| > &theta - we divide the gradient by it's norm. Here θ is a parameter, in most cases we can take θ=1 or θ=10. + +### Learning rate decay + +Training success often depends on the learning rate parameter η. It is logical to assume that larger values of η result in faster training, which is something we typically want in the beginning of the training, and then smaller value of η allow us to fine-tune the network. Thus, in most of the cases we want to decrease η in the process of the training. + +This can be done by multiplying η by some number (eg. 0.98) after each epoch of the training, or by using more complicated **learning rate schedule**. + + +## Different Network Architectures + +Selecting right network architecture for your problem can be tricky. Normally, we would take an architecture that has proven to work for our specific task (or similar one). Here is a [good overview](https://www.topbots.com/a-brief-history-of-neural-network-architectures/) or neural network architectures for computer vision. + +> It is important to select an architecture that will be powerful enough for the number of training samples that we have. Selecting too powerful model can result in [overfitting](../3-NeuralNetworks/05-Frameworks/Overfitting.md) + +Another good way would be to use and architecture that will automatically adjust to the required complexity. To some extent, **ResNet** architecture and **Inception** are self-adjusting. [More on computer vision architectures](../07-ConvNets/CNN_Architectures.md) diff --git a/README.md b/README.md index 3cf76a24..4706c59f 100644 --- a/README.md +++ b/README.md @@ -42,8 +42,8 @@ For a gentle introduction to *AI in the Cloud* topic you may consider taking [Ge Notebook 4 Multi-Layered Perceptron and Creating our own FrameworkTextNotebook 5 - Intro to Frameworks (PyTorch/Tensorflow) - Text + Intro to Frameworks (PyTorch/Tensorflow)
Overfitting + Text
Text PyTorch Keras/Tensorflow IVComputer Vision @@ -51,8 +51,8 @@ For a gentle introduction to *AI in the Cloud* topic you may consider taking [Ge MS Learn PAT 6Intro to Computer Vision. OpenCVTextNotebook -7Convolutional Neural Networks
CNN Architectures
Training TricksText
Text
TextPyTorchTensorflow -8Pre-trained Networks and Transfer LearningTextPyTorchTensorflow +7Convolutional Neural Networks
CNN Architectures
Training TricksText
TextPyTorchTensorflow +8Pre-trained Networks and Transfer LearningText
TextPyTorchTensorflow
Dropout sample 9Autoencoders and VAEsTextPyTorchTensorflow 10 Generative Adversarial NetworksTextPyTorchTensorflow 11Object DetectionTextPyTorchTensorflow