Backprop

Backpropagation is the method neural networks use to work out how much each weight contributed to the error, so the optimizer knows which way to nudge it. A network is a long chain of simple operations, and the chain rule from calculus says that the effect of an early weight on the final error is the product of the effects at each step in between. Backprop runs the network forward to get a prediction and an error, then walks backward from the error, computing each layer’s gradient from the one after it and reusing work along the way. That reuse is the point: it gets every gradient in about the cost of one extra pass, which is what makes training huge networks possible.