diff --git a/lessons/3-NeuralNetworks/04-OwnFramework/OwnFramework.ipynb b/lessons/3-NeuralNetworks/04-OwnFramework/OwnFramework.ipynb index 35b62448..19076da0 100644 --- a/lessons/3-NeuralNetworks/04-OwnFramework/OwnFramework.ipynb +++ b/lessons/3-NeuralNetworks/04-OwnFramework/OwnFramework.ipynb @@ -706,20 +706,22 @@ } }, "source": [ - "To compute $\\partial\\mathcal{L}/\\partial W$ we can use the **chaining rule** for computing derivatives of a composite function, as you can see in the formulae above. It corresponds to the following idea:\n", + "To compute $\\partial\\mathcal{L}/\\partial W$, we use the **chain rule** for derivatives of a composite function, as shown above. Backpropagation applies this rule from the output toward the parameters:\n", "\n", - "* Suppose under given input we have obtanes loss $\\Delta\\mathcal{L}$\n", - "* To minimize it, we would have to adjust softmax output $p$ by value $\\Delta p = (\\partial\\mathcal{L}/\\partial p)\\Delta\\mathcal{L}$ \n", - "* This corresponds to the changes to node $z$ by $\\Delta z = (\\partial\\mathcal{p}/\\partial z)\\Delta p$\n", - "* To minimize this error, we need to adjust parameters accordingly: $\\Delta W = (\\partial\\mathcal{z}/\\partial W)\\Delta z$ (and the same for $b$)\n", + "* For a given input, the forward pass produces a loss $\\mathcal{L}$.\n", + "* At the softmax output $p$, the upstream gradient is $\\partial\\mathcal{L}/\\partial p$.\n", + "* Propagating through the operation that computes $p$ from $z$ gives $\\partial\\mathcal{L}/\\partial z = (\\partial\\mathcal{L}/\\partial p)(\\partial p/\\partial z)$.\n", + "* Propagating one step further gives parameter gradients such as $\\partial\\mathcal{L}/\\partial W = (\\partial\\mathcal{L}/\\partial z)(\\partial z/\\partial W)$ (and similarly for $b$).\n", + "\n", + "An optimizer can then use these gradients to update the parameters in a direction that reduces the loss.\n", "\n", "\n", "\n", - "This process starts distributing the loss error from the output of the network back to its parameters. Thus the process is called **back propagation**.\n", + "This process propagates gradients from the network's output back to its parameters, which is why it is called **backpropagation**.\n", "\n", "One pass of the network training consists of two parts:\n", "* **Forward pass**, when we calculate the value of loss function for a given input minibatch\n", - "* **Backward pass**, when we try to minimize this error by distributing it back to the model parameters through the computational graph." + "* **Backward pass**, when we compute the gradients of the loss with respect to the model parameters by propagating derivatives backward through the computational graph." ] }, {