AI-For-Beginners/translations/en/lessons/3-NeuralNetworks/03-Perceptron

Introduction to Neural Networks: Perceptron

Pre-lecture quiz

One of the earliest attempts to create something resembling a modern neural network was made by Frank Rosenblatt from Cornell Aeronautical Laboratory in 1957. It was a hardware implementation called "Mark-1," designed to recognize basic geometric shapes like triangles, squares, and circles.

Images from Wikipedia

The input image was represented by a 20x20 array of photocells, meaning the neural network had 400 inputs and one binary output. This simple network consisted of a single neuron, also known as a threshold logic unit. The weights of the neural network acted like potentiometers, which had to be manually adjusted during the training process.

✅ A potentiometer is a device that allows the user to adjust the resistance in a circuit.

At the time, The New York Times described the perceptron as: the embryo of an electronic computer that [the Navy] expects will be able to walk, talk, see, write, reproduce itself and be conscious of its existence.

Perceptron Model

Imagine we have N features in our model, which means the input vector would be of size N. A perceptron is a binary classification model, meaning it can differentiate between two classes of input data. For each input vector x, we assume the perceptron's output will be either +1 or -1, depending on the class. The output is calculated using the formula:

y(x) = f(w^Tx)

where f is a step activation function.

Training the Perceptron

To train a perceptron, we need to find a weight vector w that correctly classifies most of the data points, minimizing the error. This error E is defined by the perceptron criterion as follows:

E(w) = -∑w^Tx_it_i

where:

• The sum is taken over the training data points i that are misclassified.
• x_i represents the input data, and t_i is either -1 or +1 for negative and positive examples, respectively.

This criterion is treated as a function of the weights w, and our goal is to minimize it. A common method used for this is gradient descent, where we start with some initial weights w⁽⁰⁾ and update them at each step using the formula:

w^(t+1) = w^(t) - η∇E(w)

Here, η is the learning rate, and ∇E(w) represents the gradient of E. After calculating the gradient, the update formula becomes:

w^(t+1) = w^(t) + ∑ηx_it_i

The algorithm in Python looks like this:

def train(positive_examples, negative_examples, num_iterations = 100, eta = 1):

    weights = [0,0,0] # Initialize weights (almost randomly :)
        
    for i in range(num_iterations):
        pos = random.choice(positive_examples)
        neg = random.choice(negative_examples)

        z = np.dot(pos, weights) # compute perceptron output
        if z < 0: # positive example classified as negative
            weights = weights + eta*weights.shape

        z  = np.dot(neg, weights)
        if z >= 0: # negative example classified as positive
            weights = weights - eta*weights.shape

    return weights

Conclusion

In this lesson, you learned about the perceptron, a binary classification model, and how to train it using a weight vector.

🚀 Challenge

If you'd like to try building your own perceptron, check out this lab on Microsoft Learn, which uses the Azure ML designer.

Post-lecture quiz

Review & Self Study

To explore how perceptrons can be used to solve both toy problems and real-world challenges, and to continue learning, check out the Perceptron notebook.

Here's an interesting article about perceptrons.

Assignment

In this lesson, we implemented a perceptron for a binary classification task and used it to distinguish between two handwritten digits. In this lab, your task is to solve the problem of digit classification entirely, i.e., determine which digit corresponds to a given image.

• Instructions
• Notebook

---