Backpropagation

How the model assigns blame for a mistake

Gradient descent needs the gradient: the slope of the loss with respect to every weight. A network can have hundreds of billions of them. Computing each one separately — nudge a weight, rerun the whole network, see how the loss changed, repeat a hundred billion times — would take longer than the age of the universe for a single training step. BackpropagationbackpropagationThe algorithm that computes the loss gradient for every parameter efficiently by applying the chain rule backward through the network, reusing intermediate results from the forward pass.See in glossary → gets all of them in one backward pass, for about the cost of running the network once more. It’s the algorithm that makes deep learning practical.

The chain rule is blame assignment

A network is a chain of simple operations: multiply, add, bend, multiply, add, and so on until the loss. Each operation is easy to differentiate on its own. The chain rulechain ruleThe calculus rule for differentiating composed functions. Backpropagation is just the chain rule applied layer by layer, from the loss back to the inputs.See in glossary → from calculus says how to combine those easy local slopes into the slope you actually want, and the intuitive reading of it is blame assignment.

The loss tells us how the final output was scored. Backprop passes a sensitivity signal backward through the network: a number saying how strongly a small change here would affect the loss. At each step, that signal is combined with the local slopes. By the time it has flowed to the first layer, every weight has an answer to one question: if this weight moved a tiny amount, how would the loss move? That answer is its gradient. Calling it “blame” is a useful memory aid, but the gradient does not prove what caused the model’s mistake.

Watch the blame flow

This tiny network runs a forward pass (values flowing left to right until they reach the loss) and then a backward pass, where the loss’s sensitivity signal flows right to left and each weight lands on its gradient. Step through it, or press play. Watch the numbers on the edges appear as the signal reaches them: teal if increasing that weight would raise the loss, rose if it would lower it.

Trace any connection used once and check the rule from the callout: the number on it really is the downstream sensitivity at its right-hand node times the value on its left. Combining that small multiplication across the whole network is backpropagation: the chain rule applied node by node, from the loss back to the inputs.

Forward caches, backward consumes

Two facts about this dance shape how large models are engineered, and they’re worth carrying forward:

• The backward pass needs the forward pass’s leftovers. To split blame at a node, you need the values that flowed through it during the forward pass. So those intermediate activationsactivationsThe values produced by a layer after applying its activation function. During training, intermediate activations are often kept for the backward pass.See in glossary → must be kept in memory from forward until backward. In a deep network over long inputs, storing them can dominate memory use, which is why large-scale training leans on tricks to recompute rather than store them.
• The gradients are efficient to compute, but they still take memory. One backward sweep yields all the parameter gradients, but those gradients, the saved activations, and the optimizer’s extra state all have to live somewhere. “Cheap flops, costly memory” explains an enormous amount about how training is actually done.

We now have every piece: a model, a loss, the gradient via backprop, and the update rule. The next section snaps them together into the loop used to train gradient-based neural networks.