Add convnet architectures and transfer learning

4-ComputerVision/07-ConvNets/CNN_Architectures.md

@@ -0,0 +1,37 @@
+# Most Known CNN Architectures
+
+### VGG-16
+
+VGG-16 is a network that achieved 92.7% accuracy in ImageNet top-5 classification in 2014. It has the following layer structure:
+
+![ImageNet Layers](images/vgg-16-arch1.jpg)
+
+As you can see, VGG follows traditional pyramid architecture, which is a sequence of convolution-pooling layers.
+
+![ImageNet Pyramid](images/vgg-16-arch.jpg)
+
+### ResNet
+
+ResNet is a family of models proposed by Microsoft Research in 2015. The main idea of ResNet is to use **residual blocks**:
+
+<img src="images/resnet-block.png" width="300"/>
+
+The reason for using identity pass-through is to have our layer predict **the difference** between the result of a previous layer and the output of the residual block - hence the name *residual*. Those blocks are much easier to train, and one can construct networks with several hundreds of those blocks (most common variants are ResNet-52, ResNet-101 and ResNet-152).
+
+You can also think of this network as being able to adjust its complexity to the dataset. Initially, when you are starting to train the network, weights values are small, and most of the signal goes through passthrough identity layers. As training progresses and weights become larger, the significance of network parameters grow, and the networks adjusts to accommodate required expressive power to correctly classify training images.
+
+### Google Inception
+
+Google Inception architecture takes this idea one step further, and builds each network layer as a combination of several different paths:
+
+<img src="images/inception.png" width="400"/>
+
+Here, we need to emphasize the role of 1x1 convolutions, because at first they do not make sense. Why would we need to run through the image with 1x1 filter? However, you need to remember that convolution filter also works with several depth channels (originally - RGB colors, in subsequent layers - channels for different filters), and 1x1 convolution is used to mix those input channels together using different trainable weights. It can be also viewed as downsampling (pooling) over channel dimension.
+
+Here is [a good blog post](https://medium.com/analytics-vidhya/talented-mr-1x1-comprehensive-look-at-1x1-convolution-in-deep-learning-f6b355825578) on the subject, and [original paper](https://arxiv.org/pdf/1312.4400.pdf).
+
+### MobileNet
+
+MobileNet is a family of models with reduced size, suitable for mobile devices. Use them if you are short in resources, and can sacrifice a little bit of accuracy. The main idea behind them is so-called **depthwise separable convolution**, which allows representing convolution filters by a composition of spatial convolutions and 1x1 convolution over depth channels. This significantly reduces the number of parameters, making the network smaller in size, and also easier to train with less data.
+
+Here is [a good blog post on MobileNet](https://medium.com/analytics-vidhya/image-classification-with-mobilenet-cc6fbb2cd470).

4-ComputerVision/07-ConvNets/ConfNetsTF.ipynb

@@ -206,7 +206,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 4,
    "metadata": {},
    "outputs": [
     {
@@ -215,13 +215,14 @@
      "text": [
       "Model: \"sequential\"\n",
       "_________________________________________________________________\n",
-      "Layer (type)                 Output Shape              Param #   \n",
+      " Layer (type)                Output Shape              Param #   \n",
       "=================================================================\n",
-      "conv2d (Conv2D)              (None, 24, 24, 9)         234       \n",
-      "_________________________________________________________________\n",
-      "flatten (Flatten)            (None, 5184)              0         \n",
-      "_________________________________________________________________\n",
-      "dense (Dense)                (None, 10)                51850     \n",
+      " conv2d (Conv2D)             (None, 24, 24, 9)         234       \n",
+      "                                                                 \n",
+      " flatten (Flatten)           (None, 5184)              0         \n",
+      "                                                                 \n",
+      " dense (Dense)               (None, 10)                51850     \n",
+      "                                                                 \n",
       "=================================================================\n",
       "Total params: 52,084\n",
       "Trainable params: 52,084\n",
@@ -255,7 +256,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 5,
    "metadata": {},
    "outputs": [
     {
@@ -263,15 +264,15 @@
      "output_type": "stream",
      "text": [
       "Epoch 1/5\n",
-      "1875/1875 [==============================] - 6s 3ms/step - loss: 0.0315 - acc: 0.9903 - val_loss: 0.0358 - val_acc: 0.9894\n",
+      "1875/1875 [==============================] - 15s 7ms/step - loss: 0.2099 - acc: 0.9410 - val_loss: 0.0879 - val_acc: 0.9735\n",
       "Epoch 2/5\n",
-      "1875/1875 [==============================] - 6s 3ms/step - loss: 0.0282 - acc: 0.9909 - val_loss: 0.0343 - val_acc: 0.9891\n",
+      "1875/1875 [==============================] - 13s 7ms/step - loss: 0.0858 - acc: 0.9753 - val_loss: 0.0682 - val_acc: 0.9791\n",
       "Epoch 3/5\n",
-      "1875/1875 [==============================] - 6s 3ms/step - loss: 0.0267 - acc: 0.9920 - val_loss: 0.0379 - val_acc: 0.9891\n",
+      "1875/1875 [==============================] - 13s 7ms/step - loss: 0.0665 - acc: 0.9808 - val_loss: 0.0553 - val_acc: 0.9829\n",
       "Epoch 4/5\n",
-      "1875/1875 [==============================] - 6s 3ms/step - loss: 0.0253 - acc: 0.9924 - val_loss: 0.0309 - val_acc: 0.9918\n",
+      "1875/1875 [==============================] - 15s 8ms/step - loss: 0.0582 - acc: 0.9835 - val_loss: 0.0513 - val_acc: 0.9835\n",
       "Epoch 5/5\n",
-      "1875/1875 [==============================] - 6s 3ms/step - loss: 0.0239 - acc: 0.9930 - val_loss: 0.0341 - val_acc: 0.9896\n"
+      "1875/1875 [==============================] - 14s 8ms/step - loss: 0.0527 - acc: 0.9847 - val_loss: 0.0503 - val_acc: 0.9833\n"
      ]
     }
    ],

4-ComputerVision/07-ConvNets/README.md

@@ -24,9 +24,26 @@ The way CNNs work is based on the following important ideas:
 
 ![Hierarchical Feature Extraction](images/FeatureExtractionCNN.png)
 
+
 ## Continue in Notebook
 
 Let's continue exploring how convolutional neural networks work, and how we can achieve trainable filters, in corresponding notebooks:
 
 * [Convolutional Neural Networks - PyTorch](ConvNetsPyTorch.ipynb)
 * [Convolutional Neural Networks - Tensorflow](ConvNetsTF.ipynb)
+
+## Pyramid Architecture
+
+Most of CNNs used for image processing follow so-called pyramid architecture. First convolutional layer applied to the original images typically has relatively low number of filters (8-16), which correspond to different pixel combinations, such as horizontal/vertical lines of strokes. At the next level, we reduce the spatial dimension of the network, and increase the number of filters, which corresponds to more possible combinations of simple features. With each layer, as we move towards the final classifier, spatial dimensions of the image decrease, and the number of filters grow.
+
+As an example, let's look at the architecture of VGG-16, a network that achieved 92.7% accuracy in ImageNet top-5 classification in 2014:
+
+![ImageNet Layers](images/vgg-16-arch1.jpg)
+
+![ImageNet Pyramid](images/vgg-16-arch.jpg)
+
+[**Often Used CNN Architectures**](CNN_Architectures.md)
+
+## CNNs for Other Tasks
+
+While CNNs are most often used for Computer Vision tasks, they are generally good for extracting fix-sized patterns. For example, if we are dealing with sounds, we may also want to use CNNs to look for some specific patterns in audio signal - in which case filters would be 1-dimensional (and this CNN would be called 1D-CNN). Also, sometimes 3D-CNN is used to extract features in multi-dimensional space, such as certain events occurring on video - CNN can capture certain patterns of feature changing over time. 

4-ComputerVision/08-TransferLearning/README.md

@@ -0,0 +1,26 @@
+# Pre-trained Networks and Transfer Learning
+
+Training CNNs can take a lot of time, and a lot of data is required for that task. However, much of the time is spent to learn the best low-level filters that a network is using to extract patterns from images. A natural question arises - can we use a neural network trained on one dataset and adapt it to classifying different images without full training process?
+
+This approach is called **transfer learning**, because we transfer some knowledge from one neural network model to another. In transfer learning, we typically start with a pre-trained model, which has been trained on some large image dataset, such as **ImageNet**. Those models can already do a good job extracting different features from generic images, and in many cases just building a classifier on top of those extracted features can yield a good result.
+
+## Pre-Trained Models as Feature Extractors
+
+Convolutional networks that we have talked about in previous section contained a number of layers, each of which is supposed to extract some features from the image, starting from low-level pixel combinations (such as horizontal/vertical line or stroke), up to higher level combinations of features, corresponding to things like an eye of a flame. If we train CNN on sufficiently large dataset of generic and diverse images, the network should learn to extract those common features.
+
+Both Keras and PyTorch contain functions to easily load pre-trained neural network weights for some common architectures, most of which were trained on ImageNet images. The most often used ones are described in [CNN Architectures](../07-ConvNets/CNN_Architectures.md) page. In particular, you may want to consider using one of the following:
+
+* **VGG-16/VGG-19** are relatively simple models, but they give good accuracy. Often using VGG as a first attempt is a good choice to see how transfer learning is working.
+* **ResNet** is a family of models proposed by Microsoft Research in 2015. They have more layers, and thus take more resources.
+* **MobileNet** is a family of models with reduced size, suitable for mobile devices. Use them if you are short in resources, and can sacrifice a little bit of accuracy.
+
+## Cats vs. Dogs Dataset
+
+In this example, we will use a dataset of [Cats and Dogs](https://www.microsoft.com/en-us/download/details.aspx?id=54765&WT.mc_id=academic-33554-dmitryso), which is very close to a real-life image classification scenario. 
+
+## Continue in Notebook
+
+Let's see transfer learning in action in corresponding notebooks:
+
+* [Transfer Learning - PyTorch](TransferLearningPyTorch.ipynb)
+* [Transfer Learning - Tensorflow](TransferLearningTF.ipynb)

README.md

@@ -51,8 +51,8 @@ For a gentle introduction to *AI in the Cloud* topic you may consider taking [Ge
   <td><a href="https://docs.microsoft.com/learn/modules/intro-computer-vision-tensorflow/?WT.mc_id=academic-33554-dmitryso">MS Learn</a></td>
   <td>PAT</td></tr>
 <tr><td>6</td><td>Intro to Computer Vision. OpenCV</td><td>Text<td colspan="2">Notebook</td><td></td></tr>
-<tr><td>7</td><td>Convolutional Neural Networks</td><td><a href="4-ComputerVision/07-ConvNets/README.md">Text</a></td><td><a href="4-ComputerVision/07-ConvNets/ConvNetsPyTorch.ipynb">PyTorch</a></td><td><a href="4-ComputerVision/07-ConvNets/ConvNetsTF.ipynb">Tensorflow</a></td><td></td></tr>
-<tr><td>8</td><td>Pre-trained Networks and Transfer Learning</td><td>Text</td><td>PyTorch</td><td>Tensorflow</td><td></td></tr>
+<tr><td>7</td><td>Convolutional Neural Networks</td><td><a href="4-ComputerVision/07-ConvNets/README.md">Text</a><br/><a href="4-ComputerVision/07-ConvNets/CNN_Architectures.md">Architectures</a></td><td><a href="4-ComputerVision/07-ConvNets/ConvNetsPyTorch.ipynb">PyTorch</a></td><td><a href="4-ComputerVision/07-ConvNets/ConvNetsTF.ipynb">Tensorflow</a></td><td></td></tr>
+<tr><td>8</td><td>Pre-trained Networks and Transfer Learning</td><td><a href="4-ComputerVision/08-TransferLearning/README.md">Text</a></td><td><a href="4-ComputerVision/08-TransferLearning/TransferLearningPyTorch.ipynb">PyTorch</a></td><td><a href="4-ComputerVision/08-TransferLearning/TransferLearningTF.ipynb">Tensorflow</a></td><td></td></tr>
 <tr><td>9</td><td>Autoencoders and VAEs</td><td>Text</td><td>PyTorch</td><td>Tensorflow</td><td></td></tr>
 <tr><td>10</td><td> Generative Adversarial Networks</td><td>Text</td><td>PyTorch</td><td>Tensorflow</td><td></td></tr>
 <tr><td>11</td><td>Object Detection</td><td>Text</td><td>PyTorch</td><td>Tensorflow</td><td></td></tr>