Add perceptron section

3-NeuralNetworks/03-Perceptron/README.md

@@ -0,0 +1,65 @@
+# Introduction to Neural Networks. Perceptron.
+
+One of the first attempts to implement something similar to a modern neural network was done by Frank Rosenblatt from Cornell Aeronautical Laboratory in 1957. It was hardware implementation called "Mark-1", designed to recognize primitive geometric figures, such as triangles, squares and circles.
+
+<img src='images/Rosenblatt-wikipedia.jpg' alt='Frank Rosenblatt'/> | <img src='images/Mark_I_perceptron_wikipedia.jpg' alt='The Mark 1 Perceptron' />
+-----|-----
+
+An input image was represented by 20x20 photocell array, so the neural network had 400 inputs and one binary output. Simple network contained one neuron, also called **threshold logic unit**. Neural network weights were potentiometers that required manual adjustment during the training phase.
+
+> New York Times wrote about perceptron at that time:
+> *the embryo of an electronic computer that [the Navy] expects will be able to walk, talk, see, write, reproduce itself and be conscious of its existence.*
+
+## Perceptron Model
+
+Suppose we have N features in our model, in which case the input vector would be a vector of size N. Perceptron is a **binary classification** model, i.e. it can distinguish between two classes of input data. We will assume that for each input vector x the output of our perceptron would be either +1 or -1, depending on the class. The output will be computed using the formula
+
+y(x) = f(w<sup>T</sup>x)
+
+where f is a step activation function
+
+<img src="http://www.sciweavers.org/tex2img.php?eq=f%28x%29%20%3D%20%5Cbegin%7Bcases%7D%0A%20%20%20%20%20%20%20%20%20%2B1%20%26%20x%20%5Cgeq%200%20%5C%5C%0A%20%20%20%20%20%20%20%20%20-1%20%26%20x%20%3C%200%0A%20%20%20%20%20%20%20%5Cend%7Bcases%7D%20%5C%5C%0A&bc=White&fc=Black&im=jpg&fs=12&ff=arev&edit=0" align="center" border="0" alt="f(x) = \begin{cases} +1 & x \geq 0 \\ -1 & x < 0 \end{cases} \\" width="154" height="50" />
+
+## Training the Perceptron
+
+To train a perceptron we need to find weights vector w that classifies most of the values correctly, i.e. results in the smallest **error**. This error is defined by **perceptron criterion** on the following manner:
+
+E(w) = -&sum;w<sup>T</sup>x<sub>i</sub>t<sub>i</sub>
+
+where
+* the sum is taken on those training data points i that result in the wrong classification
+* x<sub>i</sub> is the input data, and t<sub>i</sub> is either -1 or +1 for negative and positive examples accordingly. 
+
+This criteria is considered as a function of weights w, and we need to minimize it. Often, a method called **gradient descent** is used, in which we start with some initial weights w<sup>(0)</sup>, and then at each step update the weights according to the formula
+
+w<sup>(t+1)</sup> = w<sup>(t)</sup> - &eta;&nabla;E(w)
+
+Here &eta; is so-called **learning rate**, and &nabla;E(w) denotes the **gradient** of E. After we calculate the gradient, we end up with
+
+w<sup>(t+1)</sup> = w<sup>(t)</sup> + &sum;&eta;x<sub>i</sub>t<sub>i</sub>
+
+The algorithm in Python looks like this:
+
+```python
+def train(positive_examples, negative_examples, num_iterations = 100, eta = 1):
+
+    weights = [0,0,0] # Initialize weights (almost randomly :)
+        
+    for i in range(num_iterations):
+        pos = random.choice(positive_examples)
+        neg = random.choice(negative_examples)
+
+        z = np.dot(pos, weights) # compute perceptron output
+        if z < 0: # positive example classified as negative
+            weights = weights + eta*weights.shape
+
+        z  = np.dot(neg, weights)
+        if z >= 0: # negative example classified as positive
+            weights = weights - eta*weights.shape
+
+    return weights
+```
+
+## Proceed in Notebook
+
+To see how we can use perceptron to solve some toy as well as real-life problems, and to continue learning - go to [Perceptron](Perceptron.ipynb) notebook.

3-NeuralNetworks/README.md

@@ -0,0 +1,44 @@
+# Introduction to Neural Networks
+
+As we have discussed in the introduction, one of the ways to achieve intelligence is to train a **computer model** or an **artificial brain**. Since the middle of 20th century, researchers tried different mathematical models, until in recent years this direction proved to by hugely successful. Such mathematical models of the brain are called **Neural Networks** (sometimes *Artificial Neural Networks*, ANNs, in order to indicate that we are talking about models, not real networks of neurons).
+
+## Machine Learning
+
+Neural Networks are a part of larger discipline called **Machine Learning**, whose goal is to use data to train computer models that are able to solve problems. Machine Learning constitutes a large part of Artificial Intelligence, however, we do not cover classical ML in this curricula.
+
+> Visit our separate **[Machine Learning for Beginners](http://github.com/microsoft/ml-for-beginners)** curriculum to learn more about Machine Learning.
+
+In Machine Learning, we assume that we have some dataset of examples **X**, and corresponding output values **Y**. Examples are often N-dimensional vectors that consist of **features**, and outputs are called **labels**.
+
+We will consider two most common machine learning problems:
+* **Classification**, where we need to classify an input object into two or more classes. 
+* **Regression**, where we need to predict a numerical number for each of the input samples.
+
+> When representing inputs and outputs as tensors, input dataset is a matrix of size M×N, where M is number of samples, N is the number of features. Output labels Y is the vector of size M.
+
+In this curricula, we will only focus on neural network models.
+
+## A Model of a Neuron
+
+From biology we know that our brain consists of neural cells, each of them having multiple "inputs" (axons), and an output (dendrite). Axons and dendrites can conduct electrical signals, and connections between axons and dendrites can exhibit different degrees of conductivity (controlled by neuromediators).
+
+![Model of a Neuron](images/synapse-wikipedia.JPG) | ![Model of a Neuron](images/artneuron.png)
+----|----
+Real Neuron | Artificial Neuron
+
+Thus, simples mathematical model of a neuron contains several inputs X<sub>1</sub>, ..., X<sub>N</sub> and an output Y, and a series of weights W<sub>1</sub>, ..., W<sub>N</sub>. An output is calculated as
+
+<img src="http://www.sciweavers.org/tex2img.php?eq=Y%20%3D%20f%5Cleft%28%5Csum_%7Bi%3D1%7D%5EN%20X_iW_i%5Cright%29&bc=White&fc=Black&im=jpg&fs=12&ff=arev&edit=0" align="center" border="0" alt="Y = f\left(\sum_{i=1}^N X_iW_i\right)" width="131" height="53" />
+
+where f is some non-linear **activation function**.
+
+> Early models of neuron were described in classical paper [A logical calculus of the ideas immanent in nervous activity](http://www.springerlink.com/content/61446605110620kg/fulltext.pdf) by Warren McCullock and Walter Pitts in 1943. Donald Hebb in his book "[The Organization of Behavior: A Neuropsychological Theory](https://books.google.com/books?id=VNetYrB8EBoC)" proposed the way those networks can be trained.
+
+
+## In this Section
+
+In this section we will learn about:
+* [Perceptron](03-Perceptron/README.md), one of the earliest neural network models for two-class classification
+* [Modern multi-layered networks](04-OwnFramework/README.md) and [how to build our own framework](04-OwnFramework/OwnFramework.ipynb)
+* [Neural Network Frameworks](05-Frameworks/README.md), such as [PyTorch](05-Frameworks/IntroPyTorch.ipynb) and [Keras/Tensorflow](05-Frameworks/IntroKerasTF.ipynb)
+

README.md

@@ -29,27 +29,30 @@ For a gentle introduction to *AI in the Cloud* topic you may consider taking [Ge
 # Content
 
 <table>
-<tr><th>No</th><th>Lesson</th><th>Intro</th><th>PyTorch</th><th>Tensorflow</th><th>Lab</th></tr>
+<tr><th>No</th><th>Lesson</th><th>Intro</th><th>PyTorch</th><th>Keras/Tensorflow</th><th>Lab</th></tr>
 
 <tr><td>I</td><td colspan="4"><b>Introduction to AI</b></td><td>PAT</td></tr>
 <tr><td>1</td><td>Introduction and History of AI</td><td>Text</td><td></td><td></td><td></td></tr>
 
 <tr><td>II</td><td colspan="4"><b>Symbolic AI</b></td><td>PAT</td></tr>
 <tr><td>2 </td><td>Knowledge Representation and Expert Systems</td><td>Text</td><td></td><td></td><td></td></tr>
-<tr><td>III</td><td colspan="4"><b>Introduction to Neural Networks</b></td><td>PAT</td></tr>
+<tr><td>III</td><td colspan="4"><b><a href="3-NeuralNetworks/README.md">Introduction to Neural Networks</a></b></td><td>PAT</td></tr>
 <tr><td>3</td><td>Perceptron</td>
    <td><a href="3-NeuralNetworks/03-Perceptron/README.md">Text</a>
-   <td colspan="2"><a href="3-NeuralNetworks/03-Perceptron/Perceptron.ipynb">Perceptron</a><br/>
-   <a href="3-NeuralNetworks/03-Perceptron/OwnFramework.ipynb">Own Framework</a></td><td></td></tr>
-<tr><td>4 </td><td>Intro to Frameworks (PyTorch/Tensorflow)</td><td>Text</td><td>PyTorch</td><td>Tensorflow</td><td></td></tr>
-<tr><td>5 </td><td>Multi-Layered Perceptron</td><td>Text</td><td>PyTorch</td><td>Tensorflow</td><td></td></tr> 
+   <td colspan="2"><a href="3-NeuralNetworks/03-Perceptron/Perceptron.ipynb">Notebook</a></td><td></td></tr>
+<tr><td>4 </td><td>Multi-Layered Perceptron and Creating our own Framework</td><td>Text</td><td colspan="2"><a href="3-NeuralNetworks/04-OwnFramework/OwnFramework.ipynb">Notebook</a><td></td></tr> 
+<tr><td>5</td>
+   <td>Intro to Frameworks (PyTorch/Tensorflow)</td>
+   <td><a href="3-NeuralNetworks/05-Frameworks/README.md">Text</a></td>
+   <td><a href="3-NeuralNetworks/05-Frameworks/IntroPyTorch.ipynb">PyTorch</td>
+   <td><a href="3-NeuralNetworks/05-Frameworks/IntroKerasTF.md">Keras/Tensorflow</td><td></td></tr>
 <tr><td>IV</td><td colspan="2"><b>Computer Vision</b></td>
   <td><a href="https://docs.microsoft.com/learn/modules/intro-computer-vision-pytorch/?WT.mc_id=academic-33554-dmitryso">MS Learn</a></td>
   <td><a href="https://docs.microsoft.com/learn/modules/intro-computer-vision-tensorflow/?WT.mc_id=academic-33554-dmitryso">MS Learn</a></td>
   <td>PAT</td></tr>
 <tr><td>6</td><td>Intro to Computer Vision. OpenCV</td><td>Text<td colspan="2">Notebook</td><td></td></tr>
 <tr><td>7</td><td>Convolutional Neural Networks</td><td>Text</td><td>PyTorch</td><td>Tensorflow</td><td></td></tr>
-<tr><td>8</td><td>Pre-trained Networks and Transfer Learning</td><td>Text</td><td>PyTorch</td><td>Tensorflow</td><td</td></tr>
+<tr><td>8</td><td>Pre-trained Networks and Transfer Learning</td><td>Text</td><td>PyTorch</td><td>Tensorflow</td><td></td></tr>
 <tr><td>9</td><td>Autoencoders and VAEs</td><td>Text</td><td>PyTorch</td><td>Tensorflow</td><td></td></tr>
 <tr><td>10</td><td> Generative Adversarial Networks</td><td>Text</td><td>PyTorch</td><td>Tensorflow</td><td></td></tr>
 <tr><td>11</td><td>Object Detection</td><td>Text</td><td>PyTorch</td><td>Tensorflow</td><td></td></tr>
@@ -69,7 +72,7 @@ For a gentle introduction to *AI in the Cloud* topic you may consider taking [Ge
 <tr><td>VI</td><td colspan="4"><b>Other AI Techniques</b></td><td>PAT</td></tr>
 <tr><td>21</td><td>Genetic Algorithms</td><td>Text<td colspan="2">Notebook</td><td></td></tr>
 <tr><td>22</td><td>Deep Reinforcement Learning</td><td>Text</td><td>PyTorch</td><td>Tensorflow</td><td></td></tr>
-<tr><td>23</td><td>Multi-Agent Systems</td><td>Text</td><td></td><td></td><td</td></tr>
+<tr><td>23</td><td>Multi-Agent Systems</td><td>Text</td><td></td><td></td><td></td></tr>
 <tr><td>VII</td><td colspan="4"><b>AI Ethics</b></td><td>PAT</td></tr>
 <tr><td>24</td><td>AI Ethics and Responsible AI</td><td>Text</td><td></td><td></td><td></td></tr>