docs: correct backpropagation gradient explanation
This commit is contained in:
parent
33e781bf7b
commit
fe61851beb
|
|
@ -706,20 +706,22 @@
|
|||
}
|
||||
},
|
||||
"source": [
|
||||
"To compute $\\partial\\mathcal{L}/\\partial W$ we can use the **chaining rule** for computing derivatives of a composite function, as you can see in the formulae above. It corresponds to the following idea:\n",
|
||||
"To compute $\\partial\\mathcal{L}/\\partial W$, we use the **chain rule** for derivatives of a composite function, as shown above. Backpropagation applies this rule from the output toward the parameters:\n",
|
||||
"\n",
|
||||
"* Suppose under given input we have obtanes loss $\\Delta\\mathcal{L}$\n",
|
||||
"* To minimize it, we would have to adjust softmax output $p$ by value $\\Delta p = (\\partial\\mathcal{L}/\\partial p)\\Delta\\mathcal{L}$ \n",
|
||||
"* This corresponds to the changes to node $z$ by $\\Delta z = (\\partial\\mathcal{p}/\\partial z)\\Delta p$\n",
|
||||
"* To minimize this error, we need to adjust parameters accordingly: $\\Delta W = (\\partial\\mathcal{z}/\\partial W)\\Delta z$ (and the same for $b$)\n",
|
||||
"* For a given input, the forward pass produces a loss $\\mathcal{L}$.\n",
|
||||
"* At the softmax output $p$, the upstream gradient is $\\partial\\mathcal{L}/\\partial p$.\n",
|
||||
"* Propagating through the operation that computes $p$ from $z$ gives $\\partial\\mathcal{L}/\\partial z = (\\partial\\mathcal{L}/\\partial p)(\\partial p/\\partial z)$.\n",
|
||||
"* Propagating one step further gives parameter gradients such as $\\partial\\mathcal{L}/\\partial W = (\\partial\\mathcal{L}/\\partial z)(\\partial z/\\partial W)$ (and similarly for $b$).\n",
|
||||
"\n",
|
||||
"An optimizer can then use these gradients to update the parameters in a direction that reduces the loss.\n",
|
||||
"\n",
|
||||
"<img src=\"images/ComputeGraphGrad.PNG\" width=\"400px\" align=\"right\"/>\n",
|
||||
"\n",
|
||||
"This process starts distributing the loss error from the output of the network back to its parameters. Thus the process is called **back propagation**.\n",
|
||||
"This process propagates gradients from the network's output back to its parameters, which is why it is called **backpropagation**.\n",
|
||||
"\n",
|
||||
"One pass of the network training consists of two parts:\n",
|
||||
"* **Forward pass**, when we calculate the value of loss function for a given input minibatch\n",
|
||||
"* **Backward pass**, when we try to minimize this error by distributing it back to the model parameters through the computational graph."
|
||||
"* **Backward pass**, when we compute the gradients of the loss with respect to the model parameters by propagating derivatives backward through the computational graph."
|
||||
]
|
||||
},
|
||||
{
|
||||
|
|
|
|||
Loading…
Reference in New Issue