AI-For-Beginners/translations/en/lessons/3-NeuralNetworks/04-OwnFramework

Introduction to Neural Networks. Multi-Layered Perceptron

In the previous section, you learned about the simplest neural network model—the single-layer perceptron, which is a linear model for two-class classification.

In this section, we will expand this model into a more versatile framework, enabling us to:

• perform multi-class classification in addition to two-class classification,
• solve regression problems alongside classification tasks,
• distinguish between classes that are not linearly separable.

We will also develop our own modular framework in Python, which will allow us to construct various neural network architectures.

Pre-lecture quiz

Formalization of Machine Learning

Let’s begin by formalizing the Machine Learning problem. Suppose we have a training dataset X with labels Y, and we need to build a model f that makes the most accurate predictions. The quality of predictions is measured by the Loss function ℒ. Commonly used loss functions include:

• For regression problems, where we predict a numerical value, we can use absolute error ∑_i|f(x⁽ⁱ⁾)-y⁽ⁱ⁾| or squared error ∑_i(f(x⁽ⁱ⁾)-y⁽ⁱ⁾)².
• For classification tasks, we use 0-1 loss (essentially equivalent to the model’s accuracy) or logistic loss.

For a single-layer perceptron, the function f was defined as a linear function f(x)=wx+b (where w is the weight matrix, x is the vector of input features, and b is the bias vector). For more complex neural network architectures, this function can take on more intricate forms.

In classification tasks, it is often desirable for the network output to represent probabilities of the corresponding classes. To convert arbitrary numbers into probabilities (e.g., to normalize the output), we often use the softmax function σ, making the function f become f(x)=σ(wx+b).

In the definition of f above, w and b are referred to as parameters θ=⟨w,b⟩. Given the dataset ⟨X,Y⟩, we can compute the overall error across the entire dataset as a function of the parameters θ.

✅ The goal of neural network training is to minimize the error by adjusting the parameters θ.

Gradient Descent Optimization

A well-known method for function optimization is gradient descent. The idea is to compute the derivative (or gradient in multi-dimensional cases) of the loss function with respect to the parameters and adjust the parameters in a way that reduces the error. This can be formalized as follows:

• Initialize the parameters with random values w⁽⁰⁾, b⁽⁰⁾.
• Repeat the following steps multiple times:
  • w⁽ⁱ⁺¹⁾ = w⁽ⁱ⁾-η∂ℒ/∂w
  • b⁽ⁱ⁺¹⁾ = b⁽ⁱ⁾-η∂ℒ/∂b

During training, optimization steps are ideally calculated using the entire dataset (since the loss is computed as a sum over all training samples). However, in practice, we use small portions of the dataset called minibatches and calculate gradients based on these subsets. Since the subsets are chosen randomly each time, this method is referred to as stochastic gradient descent (SGD).

Multi-Layered Perceptrons and Backpropagation

A single-layer network, as we’ve seen, can classify linearly separable classes. To create a more powerful model, we can stack multiple layers in the network. Mathematically, this means the function f will have a more complex form and will be computed in multiple steps:

• z₁=w₁x+b₁
• z₂=w₂α(z₁)+b₂
• f = σ(z₂)

Here, α is a non-linear activation function, σ is the softmax function, and the parameters are θ=<w₁,b₁,w₂,b₂>.

The gradient descent algorithm remains the same, but calculating gradients becomes more complex. Using the chain differentiation rule, we can compute derivatives as follows:

• ∂ℒ/∂w₂ = (∂ℒ/∂σ)(∂σ/∂z₂)(∂z₂/∂w₂)
• ∂ℒ/∂w₁ = (∂ℒ/∂σ)(∂σ/∂z₂)(∂z₂/∂α)(∂α/∂z₁)(∂z₁/∂w₁)

✅ The chain differentiation rule is used to compute derivatives of the loss function with respect to the parameters.

Notice that the leftmost part of all these expressions is the same, allowing us to efficiently compute derivatives starting from the loss function and working "backwards" through the computational graph. This method of training a multi-layered perceptron is called backpropagation, or 'backprop'.

TODO: image citation

✅ We will explore backpropagation in much greater detail in our notebook example.

Conclusion

In this lesson, we built our own neural network library and used it for a simple two-dimensional classification task.

🚀 Challenge

In the accompanying notebook, you will implement your own framework for building and training multi-layered perceptrons. This will give you a detailed understanding of how modern neural networks function.

Proceed to the OwnFramework notebook and work through it.

Post-lecture quiz

Review & Self Study

Backpropagation is a widely used algorithm in AI and ML. It’s worth studying in more detail.

Assignment

In this lab, you will use the framework you built in this lesson to solve the MNIST handwritten digit classification task.

• Instructions
• Notebook

---