Add Intro, Frameworks, ConvNets NBs

1-Intro/README.md

@@ -57,12 +57,11 @@ Alternatively, we can try to model the simplest elements inside our brain – a
 A part of Artificial Intelligence that is based on computer learning to solve the problem based on some data is called **Machine Learning**. We will not consider classical machine learning in this course - we refer you to a separate [Machine Learning for Beginners](http://aka.ms/ml-for-beginners) Curriculum. | ![ML for Beginners](images/ml-for-beginners.png)
 -----|-----
 
-
 ## A Brief History of AI
 
 Artificial Intelligence was started as a field in the middle of XX century. Initially symbolic reasoning was a prevalent approach, and it led to a number of important successes, such as expert systems – computer programs that were able to act as an expert in some limited problem domain. However, it soon became obvious that such approach does not scale well. Extracting the knowledge from an expert, representing it in a computer, and keeping that knowledgebase accurate turns out to be a very complex task, and too expensive to be practical in many cases. This led to so-called [AI Winter](https://en.wikipedia.org/wiki/AI_winter) in the 1970s.
 
-![Brief History of AI](images/history-of-ai.png)
+<img alt="Brief History of AI" src="images/history-of-ai.png" width="70%"/>
 
 As time passed, computing resources became cheaper, and more data has become available, the neural network approaches started demonstrating great performance in competing with human beings in many areas, such as computer vision, or speech understanding. In the last decade, the term Artificial Intelligence is mostly used as a synonym for Neural Networks, because most of the AI successes that we hear about are based on them.
 
@@ -76,5 +75,27 @@ Similarly, we can see how the approach towards creating “talking programs” (
 
 * Early program of this kind, [Eliza](https://en.wikipedia.org/wiki/ELIZA), was based on very simple grammatical rule and re-formulation of the input sentence into a question.
 * Modern assistants, such as Cortana, Siri or Google Assistant, are all hybrid systems, that use Neural networks to convert speech into text and to recognize our intent, and then employ some reasoning or explicit algorithms to perform required actions
-* In the future, we may expect complete neural-based model to handle dialogue by itself, recent GPT family of neural networks show great success in this.
+* In the future, we may expect complete neural-based model to handle dialogue by itself. Recent GPT and [Turing-NLG](https://turing.microsoft.com/) family of neural networks show great success in this.
+
+<img src="images/turing-test-evol.png" width="70%"/>
+
+## Recent AI Research
+
+Recent huge growth in neural network research started around 2010, when large public datasets started to become available. A huge collection of images called [ImageNet](https://en.wikipedia.org/wiki/ImageNet), which contains around 14 million annotated images, gave birth to [ImageNet Large Scale Visual Recognition Challenge](https://image-net.org/challenges/LSVRC/).
+
+![ILSVRC Accuracy](images/ilsvrc.gif)
+
+In 2012, [Convolutional Neural Networks](../4-ComputerVision/07-ConvNets/README.md) were first used in image classification, which lead to significant drop in classification errors (from almost 30% to 16.4%). In 2015, ResNet architecture from Microsoft Research [achieved human-level accuracy](https://doi.org/10.1109/ICCV.2015.123).
+
+Since then, Neural Networks demonstrated very successful behaviour in many tasks:
+
+------|-------
+Year | Human Parity in 
+-----|--------
+2015 | [Image Classification](https://doi.org/10.1109/ICCV.2015.123)
+2016 | [Conversational Speech Recognition](https://arxiv.org/abs/1610.05256)
+2018 | [Automatic Machine Translation](https://arxiv.org/abs/1803.05567) (Chinese-to-English)
+2020 | [Image Captioning](https://arxiv.org/abs/2009.13682)
+
+Last years witnessed huge successes with large language models, such as BERT and GPT-3. This happens mainly due to the fact that there is a lot of general text data available, which allows us to train models that capture the structure and meaning of texts, pre-train them on general text collections, and then specialize those models for more specific tasks. We will learn more about [Natural Language Processing](../5-NLP/README.md) later in this course.
 

3-NeuralNetworks/03-Perceptron/Perceptron.ipynb

@@ -1048,11 +1048,6 @@
     "This notebook is a part of [AI for Beginners Curricula](http://github.com/microsoft/ai-for-beginners), and has been prepared by [Dmitry Soshnikov](http://soshnikov.com). It is inspired by Neural Network Workshop at Microsoft Research Cambridge. Some code and illustrative materials are taken from presentations by [Katja Hoffmann](https://www.microsoft.com/en-us/research/people/kahofman/), [Matthew Johnson](https://www.microsoft.com/en-us/research/people/matjoh/) and [Ryoto Tomioka](https://www.microsoft.com/en-us/research/people/ryoto/), and from [NeuroWorkshop](http://github.com/shwars/NeuroWorkshop) repository."
    ],
    "metadata": {}
-  },
-  {
-   "cell_type": "markdown",
-   "source": [],
-   "metadata": {}
   }
  ],
  "metadata": {

3-NeuralNetworks/04-OwnFramework/README.md

@@ -14,12 +14,12 @@ We will also develop our own modular framework in Python that will allows us to
 
 Let's start with formalizing the Machine Learning problem. Suppose we have a training dataset **X** with labels **Y**, and we need to build a model *f* that will make most accurate predictions. The quality of predictions is measured by **Loss function** ℒ. The following loss functions are often used:
 
-* For regression problem, when we need to predict a number, we can use **absolute error** &sum;<sub>i</sub>|f(x<sup>(i)</sup>)-y<sup>(i)</sup>|, or **squared error** &sum;<sub>i</sub>(f(x<sup>(i)</sup>)-y<sup>(i)</sub>)<sup>2</sup>
+* For regression problem, when we need to predict a number, we can use **absolute error** &sum;<sub>i</sub>|f(x<sup>(i)</sup>)-y<sup>(i)</sup>|, or **squared error** &sum;<sub>i</sub>(f(x<sup>(i)</sup>)-y<sup>(i)</sup>)<sup>2</sup>
 * For classification, we use **0-1 loss** (which is essentially the same as **accuracy** of the model), or **logistic loss**.
 
 For one-level perceptron, function *f* was defined as a linear function *f(x)=wx+b* (here *w* is the weight matrix, *x* is the vector if input features, and *b* is bias vector). For different neural network architectures, this function can take more complex form. 
 
-> In the case of classification, it is often desirable to get probabilities of corresponding classes as network output. To convert arbitrary numbers to probabilities (eg. to normalize the output), we often use **softmax** function &sigma;, for the function *f* becomes *f(x)=&sigma;(wx+b)*
+> In the case of classification, it is often desirable to get probabilities of corresponding classes as network output. To convert arbitrary numbers to probabilities (eg. to normalize the output), we often use **softmax** function &sigma;, and the function *f* becomes *f(x)=&sigma;(wx+b)*
 
 In the definition of *f* above, *w* and *b* are called **parameters** &theta;=⟨*w,b*⟩. Given the dataset ⟨**X**,**Y**⟩, we can compute an overall error on the whole dataset as a function of parameters &theta;.
 
@@ -43,7 +43,7 @@ One-layer network, as we have seen above, is capable of classifying linearly sep
 * z<sub>2</sub>=w<sub>2</sub>&alpha;(z<sub>1</sub>)+b<sub>2</sub>
 * f = &sigma;(z<sub>2</sub>)
 
-Here, &alpha; is a **non-linear activation function**, &sigma; is a softmax function, and &theta;=<*w<sub>1</sub>,b<sub>1</sub>,w<sub>2</sub>,b<sub>2</sub>*> are parameters.
+Here, &alpha; is a **non-linear activation function**, &sigma; is a softmax function, and parameters &theta;=<*w<sub>1</sub>,b<sub>1</sub>,w<sub>2</sub>,b<sub>2</sub>*>.
 
 The gradient descent algorithm would remain the same, but it would be more difficult to calculate gradients. Given the
  chain differentiation rule, we can calculate derivatives as:
@@ -51,9 +51,11 @@ The gradient descent algorithm would remain the same, but it would be more diffi
 * &part;&lagran;/&part;w<sub>2</sub> = (&part;&lagran;/&part;&sigma;)(&part;&sigma;/&part;z<sub>2</sub>)(&part;z<sub>2</sub>/&part;w<sub>2</sub>)
 * &part;&lagran;/&part;w<sub>1</sub> = (&part;&lagran;/&part;&sigma;)(&part;&sigma;/&part;z<sub>2</sub>)(&part;z<sub>2</sub>/&part;&alpha;)(&part;&alpha;/&part;z<sub>1</sub>)(&part;z<sub>1</sub>/&part;w<sub>1</sub>)
 
-Note that the beginning of all those expressions are the same, and thus we can effectively calculate derivatives starting from the loss function and going "backwards" through the computational graph. Thus the method of training multi-layered perceptron is called **back propagation**.
+Note that the left-most part of all those expressions is the same, and thus we can effectively calculate derivatives starting from the loss function and going "backwards" through the computational graph. Thus the method of training multi-layered perceptron is called **back propagation**.
 
-> We will cover back prop in much more detail in our notebook example.  
+<img src="images/ComputeGraphGrad.PNG" width="400px" align="right"/>
+
+We will cover back prop in much more detail in our notebook example.  
 ## [Proceed to Notebook](OwnFramework.ipynb)
 
 In the accompanying notebook, we will implement our own framework for building and training multi-layered perceptrons. You will be able to see in detail how modern neural networks operate. Proceed to [OwnFramework](OwnFramework.ipynb) notebook.

3-NeuralNetworks/05-Frameworks/README.md

@@ -0,0 +1,37 @@
+# Neural Network Frameworks
+
+As we have learnt already, to be able to train neural networks efficiently we need to do two things:
+
+* To operate on tensors, eg. to multiply, add, and compute some functions such as sigmoid or softmax
+* To compute gradients of all expressions, in order to perform gradient descent optimization
+
+While `numpy` library can do the first part, we need some mechanism to compute gradients. In [our framework](../04-OwnFramework/OwnFramework.ipynb) that we have developed in the previous section we had to manually program all derivative functions inside the `backward` method, which does back propagation. Ideally, a framework should give us the opportunity to compute gradients of *any  expression* that we can define.
+
+Another important thing is to be able to perform computations on GPU, or any other specialized compute units, such as [TPU](https://en.wikipedia.org/wiki/Tensor_Processing_Unit). Deep neural network training requires *a lot* of computations, and to be able to parallelize those computations on GPUs is very important.
+
+Currently, there are two most popular neural frameworks: [Tensorflow](http://tensorflow.org), and [PyTorch](https://pytorch.org/). Both provide low-level API to operate with tensors on both CPU and GPU. On top of the low-level API, there is also higher-level API, called [Keras](https://keras.io/) and [PyTorch Lightning](https://pytorchlightning.ai/) correspondingly.
+
+Low-Level API | [TensorFlow](http://tensorflow.org) | [PyTorch](https://pytorch.org/)
+--------------|-------------------------------------|--------------------------------
+High-level API| [Keras](https://keras.io/) | [PyTorch Lightning](https://pytorchlightning.ai/)
+
+**Low-level APIs** in both frameworks allow you to build so-called **computational graph**. This graph defines how to compute the output (usually the loss function) with given input parameters, and can be pushed for computation on GPU, if it is available. There are functions to differentiate this computational graph and compute gradients, which can then be used for optimizing model parameters.
+
+**High-level APIs** pretty much consider neural network as a **sequence of layers**, and make constructing most of the neural networks much easier. Training the model usually requires preparing the data and then calling `fit` function to do the job.
+
+High-level API allows you to construct typical neural networks very fast, without worrying about lots of details. At the same time, low-level API offer much more control over training process, and thus they are used a lot in research, when you are dealing with new neural network architectures. 
+
+It is also important to understand that you can use both APIs together, eg. you can develop your own network layer architecture using low-level API, and then use it inside the larger network constructed and trained with high-level API. Or you can define a network using high-level API as a sequence of layers, and then use your own low-level training loop to perform optimization. Both APIs use the same basic underlying concepts, and they are designed to work well together.
+
+## Learning
+
+In this course, we offer most of the content both for PyTorch and Tensorflow. You can chose your preferred framework and only go through the corresponding notebooks. If you are not sure which framework to chose - read some discussions on the internet regarding **PyTorch vs. Tensorflow**. You can also have a look at both frameworks to get better understanding.
+
+Where possible, we will use High-Level APIs for simplicity. However, we believe it is important to understand how neural networks work from the ground up, thus in the beginning we start by working with low-level API and tensors. However, if you want to get going fast and do not want to spend a lot of time on details, you can skip those, and go straight into high-level API notebooks.
+
+## Continue into Notebooks
+
+Low-Level API | [TensorFlow+Keras Notebook](IntroKerasTF.ipynb) | [PyTorch](IntroPyTorch.ipynb)
+--------------|-------------------------------------|--------------------------------
+High-level API| [Keras](IntroKeras.ipynb) | *PyTorch Lightning*
+

4-ComputerVision/07-ConvNets/README.md

@@ -0,0 +1,32 @@
+# Convolutional Neural Networks
+
+We have seen before that neural networks are quite good at dealing with images, and even one-layer perceptron is able to recognize handwritten digits from MNIST dataset with reasonable accuracy. However, MNIST dataset is very special, and all digits are centered inside the image, which makes the task simpler.
+
+In real life, we want to be able to recognize objects on the picture regardless of their exact location in the image. Computer vision is different from generic classification, because when we are trying to find a certain object in the picture, we are scanning the image looking for some specific **patterns** and their combinations. For example, when looking for a cat, we first may look for horizontal lines, which can form whiskers, and then certain combination of whiskers can tell us that it is actually a picture of a cat. Relative position and presence of certain patterns is important, and not their exact position on the image. 
+
+To extract patterns, we will use the notion of **convolutional filters**. As you know, an image is represented by a 2D-matrix, or 3D-tensor with color depth. Applying a filter means that we take relatively small **filter kernel** matrix, and for each pixel in the original image we compute the weighted average with neighboring points. We can view this like a small window sliding over the whole image, and averaging out all pixels according to the weights in the filter kernel matrix.
+
+![Vertical Edge Filter](images/filter-vert.png) | ![Horizontal Edge Filter](images/filter-horiz.png)
+----|----
+
+For example, if we apply 3x3 vertical edge and horizontal edge filters to the MNIST digits, we can get highlights (e.g. high values) where there are vertical and horizontal edges in our original image. Thus those two filters can be used to "look for" edges. Similarly, we can design different filters to look for other low-level patterns:
+
+<img src="images/lmfilters.jpg" width="500" align="center"/>
+
+However, while we can design the filters to extract some patterns manually, we can also design the network in such a way that it will learn the patterns automatically. It is one of the main ideas behind the CNN.
+
+## Main ideas behind CNN
+
+The way CNNs work is based on the following important ideas:
+* Convolutional filters can extract patterns
+* We can design the network in such a way that filters are trained automatically
+* We can use the same approach to find patterns in high-level features, not only in the original image. Thus CNN feature extraction work on a hierarchy of features, starting from low-level pixel combinations, up to higher level combination of picture parts.
+
+![Hierarchical Feature Extraction](images/FeatureExtractionCNN.png)
+
+## Continue in Notebook
+
+Let's continue exploring how convolutional neural networks work, and how we can achieve trainable filters, in corresponding notebooks:
+
+* [Convolutional Neural Networks - PyTorch](ConvNetsPyTorch.ipynb)
+* [Convolutional Neural Networks - Tensorflow](ConvNetsTF.ipynb)

4-ComputerVision/07-ConvNets/pytorchcv.py

@@ -0,0 +1,169 @@
+  
+# Script file to hide implementation details for PyTorch computer vision module
+
+import builtins
+import torch
+import torch.nn as nn
+from torch.utils import data
+import torchvision
+from torchvision.transforms import ToTensor
+import matplotlib.pyplot as plt
+import numpy as np
+from PIL import Image
+import glob
+import os
+import zipfile 
+
+default_device = 'cuda' if torch.cuda.is_available() else 'cpu'
+
+def load_mnist(batch_size=64):
+    builtins.data_train = torchvision.datasets.MNIST('./data',
+        download=True,train=True,transform=ToTensor())
+    builtins.data_test = torchvision.datasets.MNIST('./data',
+        download=True,train=False,transform=ToTensor())
+    builtins.train_loader = torch.utils.data.DataLoader(data_train,batch_size=batch_size)
+    builtins.test_loader = torch.utils.data.DataLoader(data_test,batch_size=batch_size)
+    
+def train_epoch(net,dataloader,lr=0.01,optimizer=None,loss_fn = nn.NLLLoss()):
+    optimizer = optimizer or torch.optim.Adam(net.parameters(),lr=lr)
+    net.train()
+    total_loss,acc,count = 0,0,0
+    for features,labels in dataloader:
+        optimizer.zero_grad()
+        lbls = labels.to(default_device)
+        out = net(features.to(default_device))
+        loss = loss_fn(out,lbls) #cross_entropy(out,labels)
+        loss.backward()
+        optimizer.step()
+        total_loss+=loss
+        _,predicted = torch.max(out,1)
+        acc+=(predicted==lbls).sum()
+        count+=len(labels)
+    return total_loss.item()/count, acc.item()/count
+
+def validate(net, dataloader,loss_fn=nn.NLLLoss()):
+    net.eval()
+    count,acc,loss = 0,0,0
+    with torch.no_grad():
+        for features,labels in dataloader:
+            lbls = labels.to(default_device)
+            out = net(features.to(default_device))
+            loss += loss_fn(out,lbls) 
+            pred = torch.max(out,1)[1]
+            acc += (pred==lbls).sum()
+            count += len(labels)
+    return loss.item()/count, acc.item()/count
+
+def train(net,train_loader,test_loader,optimizer=None,lr=0.01,epochs=10,loss_fn=nn.NLLLoss()):
+    optimizer = optimizer or torch.optim.Adam(net.parameters(),lr=lr)
+    res = { 'train_loss' : [], 'train_acc': [], 'val_loss': [], 'val_acc': []}
+    for ep in range(epochs):
+        tl,ta = train_epoch(net,train_loader,optimizer=optimizer,lr=lr,loss_fn=loss_fn)
+        vl,va = validate(net,test_loader,loss_fn=loss_fn)
+        print(f"Epoch {ep:2}, Train acc={ta:.3f}, Val acc={va:.3f}, Train loss={tl:.3f}, Val loss={vl:.3f}")
+        res['train_loss'].append(tl)
+        res['train_acc'].append(ta)
+        res['val_loss'].append(vl)
+        res['val_acc'].append(va)
+    return res
+
+def train_long(net,train_loader,test_loader,epochs=5,lr=0.01,optimizer=None,loss_fn = nn.NLLLoss(),print_freq=10):
+    optimizer = optimizer or torch.optim.Adam(net.parameters(),lr=lr)
+    for epoch in range(epochs):
+        net.train()
+        total_loss,acc,count = 0,0,0
+        for i, (features,labels) in enumerate(train_loader):
+            lbls = labels.to(default_device)
+            optimizer.zero_grad()
+            out = net(features.to(default_device))
+            loss = loss_fn(out,lbls)
+            loss.backward()
+            optimizer.step()
+            total_loss+=loss
+            _,predicted = torch.max(out,1)
+            acc+=(predicted==lbls).sum()
+            count+=len(labels)
+            if i%print_freq==0:
+                print("Epoch {}, minibatch {}: train acc = {}, train loss = {}".format(epoch,i,acc.item()/count,total_loss.item()/count))
+        vl,va = validate(net,test_loader,loss_fn)
+        print("Epoch {} done, validation acc = {}, validation loss = {}".format(epoch,va,vl))
+
+
+def plot_results(hist):
+    plt.figure(figsize=(15,5))
+    plt.subplot(121)
+    plt.plot(hist['train_acc'], label='Training acc')
+    plt.plot(hist['val_acc'], label='Validation acc')
+    plt.legend()
+    plt.subplot(122)
+    plt.plot(hist['train_loss'], label='Training loss')
+    plt.plot(hist['val_loss'], label='Validation loss')
+    plt.legend()
+
+def plot_convolution(t,title=''):
+    with torch.no_grad():
+        c = nn.Conv2d(kernel_size=(3,3),out_channels=1,in_channels=1)
+        c.weight.copy_(t)
+        fig, ax = plt.subplots(2,6,figsize=(8,3))
+        fig.suptitle(title,fontsize=16)
+        for i in range(5):
+            im = data_train[i][0]
+            ax[0][i].imshow(im[0])
+            ax[1][i].imshow(c(im.unsqueeze(0))[0][0])
+            ax[0][i].axis('off')
+            ax[1][i].axis('off')
+        ax[0,5].imshow(t)
+        ax[0,5].axis('off')
+        ax[1,5].axis('off')
+        #plt.tight_layout()
+        plt.show()
+        
+def display_dataset(dataset, n=10,classes=None):
+    fig,ax = plt.subplots(1,n,figsize=(15,3))
+    mn = min([dataset[i][0].min() for i in range(n)])
+    mx = max([dataset[i][0].max() for i in range(n)])
+    for i in range(n):
+        ax[i].imshow(np.transpose((dataset[i][0]-mn)/(mx-mn),(1,2,0)))
+        ax[i].axis('off')
+        if classes:
+            ax[i].set_title(classes[dataset[i][1]])
+
+
+def check_image(fn):
+    try:
+        im = Image.open(fn)
+        im.verify()
+        return True
+    except:
+        return False
+    
+def check_image_dir(path):
+    for fn in glob.glob(path):
+        if not check_image(fn):
+            print("Corrupt image: {}".format(fn))
+            os.remove(fn)
+
+
+def common_transform():
+    std_normalize = torchvision.transforms.Normalize(mean=[0.485, 0.456, 0.406],
+                          std=[0.229, 0.224, 0.225])
+    trans = torchvision.transforms.Compose([
+            torchvision.transforms.Resize(256),
+            torchvision.transforms.CenterCrop(224),
+            torchvision.transforms.ToTensor(), 
+            std_normalize])
+    return trans
+
+def load_cats_dogs_dataset():
+    if not os.path.exists('data/PetImages'):
+        with zipfile.ZipFile('data/kagglecatsanddogs_3367a.zip', 'r') as zip_ref:
+            zip_ref.extractall('data')
+
+    check_image_dir('data/PetImages/Cat/*.jpg')
+    check_image_dir('data/PetImages/Dog/*.jpg')
+
+    dataset = torchvision.datasets.ImageFolder('data/PetImages',transform=common_transform())
+    trainset, testset = torch.utils.data.random_split(dataset,[20000,len(dataset)-20000])
+    trainloader = torch.utils.data.DataLoader(trainset,batch_size=32)
+    testloader = torch.utils.data.DataLoader(trainset,batch_size=32)
+    return dataset, trainloader, testloader

4-ComputerVision/07-ConvNets/tfcv.py

@@ -0,0 +1,86 @@
+# Tensorflow Computer Vision Helper
+
+import tensorflow as tf
+from tensorflow import keras
+import numpy as np
+import matplotlib.pyplot as plt
+from PIL import Image
+import glob
+import os
+
+def plot_convolution(data,t,title=''):
+    fig, ax = plt.subplots(2,len(data)+1,figsize=(8,3))
+    fig.suptitle(title,fontsize=16)
+    tt = np.expand_dims(np.expand_dims(t,2),2)
+    for i,im in enumerate(data):
+        ax[0][i].imshow(im)
+        ximg = np.expand_dims(np.expand_dims(im,2),0)
+        cim = tf.nn.conv2d(ximg,tt,1,'SAME')
+        ax[1][i].imshow(cim[0][:,:,0])
+        ax[0][i].axis('off')
+        ax[1][i].axis('off')
+    ax[0,-1].imshow(t)
+    ax[0,-1].axis('off')
+    ax[1,-1].axis('off')
+    #plt.tight_layout()
+    plt.show()
+
+def plot_results(hist):
+    fig,ax = plt.subplots(1,2,figsize=(15,3))
+    ax[0].set_title('Accuracy')
+    ax[1].set_title('Loss')
+    for x in ['acc','val_acc']:
+        ax[0].plot(hist.history[x])
+    for x in ['loss','val_loss']:
+        ax[1].plot(hist.history[x])
+    plt.show()
+
+def display_dataset(dataset, labels=None, n=10, classes=None):
+    fig,ax = plt.subplots(1,n,figsize=(15,3))
+    for i in range(n):
+        ax[i].imshow(dataset[i])
+        ax[i].axis('off')
+        if classes is not None and labels is not None:
+            ax[i].set_title(classes[labels[i][0]])
+
+def check_image(fn):
+    try:
+        im = Image.open(fn)
+        im.verify()
+        return im.format=='JPEG'
+    except:
+        return False
+    
+def check_image_dir(path):
+    for fn in glob.glob(path):
+        if not check_image(fn):
+            print("Corrupt image or wrong format: {}".format(fn))
+            os.remove(fn)
+
+def load_cats_dogs_dataset(batch_size=64):
+    if not os.path.exists('data/PetImages'):
+        print("Extracting the dataset")
+        with zipfile.ZipFile('data/kagglecatsanddogs_3367a.zip', 'r') as zip_ref:
+            zip_ref.extractall('data')
+    print("Checking dataset")
+    check_image_dir('data/PetImages/Cat/*.jpg')
+    check_image_dir('data/PetImages/Dog/*.jpg')
+    data_dir = 'data/PetImages'
+    print("Loading dataset")
+    ds_train = keras.preprocessing.image_dataset_from_directory(
+        data_dir,
+        validation_split = 0.2,
+        subset = 'training',
+        seed = 13,
+        image_size = (224,224),
+        batch_size = batch_size
+    )
+    ds_test = keras.preprocessing.image_dataset_from_directory(
+        data_dir,
+        validation_split = 0.2,
+        subset = 'validation',
+        seed = 13,
+        image_size = (224,224),
+        batch_size = batch_size
+    )
+    return ds_train,ds_test

4-ComputerVision/08-TransferLearning/pytorchcv.py

@@ -0,0 +1,169 @@
+  
+# Script file to hide implementation details for PyTorch computer vision module
+
+import builtins
+import torch
+import torch.nn as nn
+from torch.utils import data
+import torchvision
+from torchvision.transforms import ToTensor
+import matplotlib.pyplot as plt
+import numpy as np
+from PIL import Image
+import glob
+import os
+import zipfile 
+
+default_device = 'cuda' if torch.cuda.is_available() else 'cpu'
+
+def load_mnist(batch_size=64):
+    builtins.data_train = torchvision.datasets.MNIST('./data',
+        download=True,train=True,transform=ToTensor())
+    builtins.data_test = torchvision.datasets.MNIST('./data',
+        download=True,train=False,transform=ToTensor())
+    builtins.train_loader = torch.utils.data.DataLoader(data_train,batch_size=batch_size)
+    builtins.test_loader = torch.utils.data.DataLoader(data_test,batch_size=batch_size)
+    
+def train_epoch(net,dataloader,lr=0.01,optimizer=None,loss_fn = nn.NLLLoss()):
+    optimizer = optimizer or torch.optim.Adam(net.parameters(),lr=lr)
+    net.train()
+    total_loss,acc,count = 0,0,0
+    for features,labels in dataloader:
+        optimizer.zero_grad()
+        lbls = labels.to(default_device)
+        out = net(features.to(default_device))
+        loss = loss_fn(out,lbls) #cross_entropy(out,labels)
+        loss.backward()
+        optimizer.step()
+        total_loss+=loss
+        _,predicted = torch.max(out,1)
+        acc+=(predicted==lbls).sum()
+        count+=len(labels)
+    return total_loss.item()/count, acc.item()/count
+
+def validate(net, dataloader,loss_fn=nn.NLLLoss()):
+    net.eval()
+    count,acc,loss = 0,0,0
+    with torch.no_grad():
+        for features,labels in dataloader:
+            lbls = labels.to(default_device)
+            out = net(features.to(default_device))
+            loss += loss_fn(out,lbls) 
+            pred = torch.max(out,1)[1]
+            acc += (pred==lbls).sum()
+            count += len(labels)
+    return loss.item()/count, acc.item()/count
+
+def train(net,train_loader,test_loader,optimizer=None,lr=0.01,epochs=10,loss_fn=nn.NLLLoss()):
+    optimizer = optimizer or torch.optim.Adam(net.parameters(),lr=lr)
+    res = { 'train_loss' : [], 'train_acc': [], 'val_loss': [], 'val_acc': []}
+    for ep in range(epochs):
+        tl,ta = train_epoch(net,train_loader,optimizer=optimizer,lr=lr,loss_fn=loss_fn)
+        vl,va = validate(net,test_loader,loss_fn=loss_fn)
+        print(f"Epoch {ep:2}, Train acc={ta:.3f}, Val acc={va:.3f}, Train loss={tl:.3f}, Val loss={vl:.3f}")
+        res['train_loss'].append(tl)
+        res['train_acc'].append(ta)
+        res['val_loss'].append(vl)
+        res['val_acc'].append(va)
+    return res
+
+def train_long(net,train_loader,test_loader,epochs=5,lr=0.01,optimizer=None,loss_fn = nn.NLLLoss(),print_freq=10):
+    optimizer = optimizer or torch.optim.Adam(net.parameters(),lr=lr)
+    for epoch in range(epochs):
+        net.train()
+        total_loss,acc,count = 0,0,0
+        for i, (features,labels) in enumerate(train_loader):
+            lbls = labels.to(default_device)
+            optimizer.zero_grad()
+            out = net(features.to(default_device))
+            loss = loss_fn(out,lbls)
+            loss.backward()
+            optimizer.step()
+            total_loss+=loss
+            _,predicted = torch.max(out,1)
+            acc+=(predicted==lbls).sum()
+            count+=len(labels)
+            if i%print_freq==0:
+                print("Epoch {}, minibatch {}: train acc = {}, train loss = {}".format(epoch,i,acc.item()/count,total_loss.item()/count))
+        vl,va = validate(net,test_loader,loss_fn)
+        print("Epoch {} done, validation acc = {}, validation loss = {}".format(epoch,va,vl))
+
+
+def plot_results(hist):
+    plt.figure(figsize=(15,5))
+    plt.subplot(121)
+    plt.plot(hist['train_acc'], label='Training acc')
+    plt.plot(hist['val_acc'], label='Validation acc')
+    plt.legend()
+    plt.subplot(122)
+    plt.plot(hist['train_loss'], label='Training loss')
+    plt.plot(hist['val_loss'], label='Validation loss')
+    plt.legend()
+
+def plot_convolution(t,title=''):
+    with torch.no_grad():
+        c = nn.Conv2d(kernel_size=(3,3),out_channels=1,in_channels=1)
+        c.weight.copy_(t)
+        fig, ax = plt.subplots(2,6,figsize=(8,3))
+        fig.suptitle(title,fontsize=16)
+        for i in range(5):
+            im = data_train[i][0]
+            ax[0][i].imshow(im[0])
+            ax[1][i].imshow(c(im.unsqueeze(0))[0][0])
+            ax[0][i].axis('off')
+            ax[1][i].axis('off')
+        ax[0,5].imshow(t)
+        ax[0,5].axis('off')
+        ax[1,5].axis('off')
+        #plt.tight_layout()
+        plt.show()
+        
+def display_dataset(dataset, n=10,classes=None):
+    fig,ax = plt.subplots(1,n,figsize=(15,3))
+    mn = min([dataset[i][0].min() for i in range(n)])
+    mx = max([dataset[i][0].max() for i in range(n)])
+    for i in range(n):
+        ax[i].imshow(np.transpose((dataset[i][0]-mn)/(mx-mn),(1,2,0)))
+        ax[i].axis('off')
+        if classes:
+            ax[i].set_title(classes[dataset[i][1]])
+
+
+def check_image(fn):
+    try:
+        im = Image.open(fn)
+        im.verify()
+        return True
+    except:
+        return False
+    
+def check_image_dir(path):
+    for fn in glob.glob(path):
+        if not check_image(fn):
+            print("Corrupt image: {}".format(fn))
+            os.remove(fn)
+
+
+def common_transform():
+    std_normalize = torchvision.transforms.Normalize(mean=[0.485, 0.456, 0.406],
+                          std=[0.229, 0.224, 0.225])
+    trans = torchvision.transforms.Compose([
+            torchvision.transforms.Resize(256),
+            torchvision.transforms.CenterCrop(224),
+            torchvision.transforms.ToTensor(), 
+            std_normalize])
+    return trans
+
+def load_cats_dogs_dataset():
+    if not os.path.exists('data/PetImages'):
+        with zipfile.ZipFile('data/kagglecatsanddogs_3367a.zip', 'r') as zip_ref:
+            zip_ref.extractall('data')
+
+    check_image_dir('data/PetImages/Cat/*.jpg')
+    check_image_dir('data/PetImages/Dog/*.jpg')
+
+    dataset = torchvision.datasets.ImageFolder('data/PetImages',transform=common_transform())
+    trainset, testset = torch.utils.data.random_split(dataset,[20000,len(dataset)-20000])
+    trainloader = torch.utils.data.DataLoader(trainset,batch_size=32)
+    testloader = torch.utils.data.DataLoader(trainset,batch_size=32)
+    return dataset, trainloader, testloader

4-ComputerVision/08-TransferLearning/tfcv.py

@@ -0,0 +1,86 @@
+# Tensorflow Computer Vision Helper
+
+import tensorflow as tf
+from tensorflow import keras
+import numpy as np
+import matplotlib.pyplot as plt
+from PIL import Image
+import glob
+import os
+
+def plot_convolution(data,t,title=''):
+    fig, ax = plt.subplots(2,len(data)+1,figsize=(8,3))
+    fig.suptitle(title,fontsize=16)
+    tt = np.expand_dims(np.expand_dims(t,2),2)
+    for i,im in enumerate(data):
+        ax[0][i].imshow(im)
+        ximg = np.expand_dims(np.expand_dims(im,2),0)
+        cim = tf.nn.conv2d(ximg,tt,1,'SAME')
+        ax[1][i].imshow(cim[0][:,:,0])
+        ax[0][i].axis('off')
+        ax[1][i].axis('off')
+    ax[0,-1].imshow(t)
+    ax[0,-1].axis('off')
+    ax[1,-1].axis('off')
+    #plt.tight_layout()
+    plt.show()
+
+def plot_results(hist):
+    fig,ax = plt.subplots(1,2,figsize=(15,3))
+    ax[0].set_title('Accuracy')
+    ax[1].set_title('Loss')
+    for x in ['acc','val_acc']:
+        ax[0].plot(hist.history[x])
+    for x in ['loss','val_loss']:
+        ax[1].plot(hist.history[x])
+    plt.show()
+
+def display_dataset(dataset, labels=None, n=10, classes=None):
+    fig,ax = plt.subplots(1,n,figsize=(15,3))
+    for i in range(n):
+        ax[i].imshow(dataset[i])
+        ax[i].axis('off')
+        if classes is not None and labels is not None:
+            ax[i].set_title(classes[labels[i][0]])
+
+def check_image(fn):
+    try:
+        im = Image.open(fn)
+        im.verify()
+        return im.format=='JPEG'
+    except:
+        return False
+    
+def check_image_dir(path):
+    for fn in glob.glob(path):
+        if not check_image(fn):
+            print("Corrupt image or wrong format: {}".format(fn))
+            os.remove(fn)
+
+def load_cats_dogs_dataset(batch_size=64):
+    if not os.path.exists('data/PetImages'):
+        print("Extracting the dataset")
+        with zipfile.ZipFile('data/kagglecatsanddogs_3367a.zip', 'r') as zip_ref:
+            zip_ref.extractall('data')
+    print("Checking dataset")
+    check_image_dir('data/PetImages/Cat/*.jpg')
+    check_image_dir('data/PetImages/Dog/*.jpg')
+    data_dir = 'data/PetImages'
+    print("Loading dataset")
+    ds_train = keras.preprocessing.image_dataset_from_directory(
+        data_dir,
+        validation_split = 0.2,
+        subset = 'training',
+        seed = 13,
+        image_size = (224,224),
+        batch_size = batch_size
+    )
+    ds_test = keras.preprocessing.image_dataset_from_directory(
+        data_dir,
+        validation_split = 0.2,
+        subset = 'validation',
+        seed = 13,
+        image_size = (224,224),
+        batch_size = batch_size
+    )
+    return ds_train,ds_test

README.md

@@ -51,7 +51,7 @@ For a gentle introduction to *AI in the Cloud* topic you may consider taking [Ge
   <td><a href="https://docs.microsoft.com/learn/modules/intro-computer-vision-tensorflow/?WT.mc_id=academic-33554-dmitryso">MS Learn</a></td>
   <td>PAT</td></tr>
 <tr><td>6</td><td>Intro to Computer Vision. OpenCV</td><td>Text<td colspan="2">Notebook</td><td></td></tr>
-<tr><td>7</td><td>Convolutional Neural Networks</td><td>Text</td><td>PyTorch</td><td>Tensorflow</td><td></td></tr>
+<tr><td>7</td><td>Convolutional Neural Networks</td><td><a href="4-ComputerVision/07-ConvNets/README.md">Text</a></td><td><a href="4-ComputerVision/07-ConvNets/ConvNetsPyTorch.ipynb">PyTorch</a></td><td><a href="4-ComputerVision/07-ConvNets/ConvNetsTF.ipynb">Tensorflow</a></td><td></td></tr>
 <tr><td>8</td><td>Pre-trained Networks and Transfer Learning</td><td>Text</td><td>PyTorch</td><td>Tensorflow</td><td></td></tr>
 <tr><td>9</td><td>Autoencoders and VAEs</td><td>Text</td><td>PyTorch</td><td>Tensorflow</td><td></td></tr>
 <tr><td>10</td><td> Generative Adversarial Networks</td><td>Text</td><td>PyTorch</td><td>Tensorflow</td><td></td></tr>