When gradients vanish

The problem that stalled deep learning, and the fixes

The training loop ended on a warning: in a very deep network, the blame flowing back from the loss can fade to almost nothing before it reaches the early layers. That fading isn’t a rare glitch: it’s a mathematical consequence of how backpropagationbackpropagationThe algorithm that computes the loss gradient for every parameter efficiently by applying the chain rule backward through the network, reusing intermediate results from the forward pass.See in glossary → works, and for years it made deep networks nearly impossible to train. This section is about why it happens and the handful of fixes that finally broke the logjam.

A product of slopes

Recall the shape of the backward passbackward passThe second half of a training step: backpropagation walks from the loss back through the network, computing each parameter's gradient.See in glossary →: to get the gradient at an early layer, backprop repeatedly applies the local slope of every layer the signal passes through on the way back. A layer has many inputs and outputs, so its slopes are collected in a table called a JacobianJacobianA table containing all the local slopes from a function's many inputs to its many outputs. Backprop uses these slope tables to carry gradients through a layer.See in glossary →. The chain rulechain ruleThe calculus rule for differentiating composed functions. Backpropagation is just the chain rule applied layer by layer, from the loss back to the inputs.See in glossary → combines these tables across layers. Thinking about one slope per layer is a useful first picture, but a real layer can shrink some directions while stretching others.

Repeated multiplication is treacherous. In a one-number toy case, factors near $0.7$ shrink rapidly ($0.7^{20} \approx 0.0008$) while factors near $1.5$ grow to about $3300$. In a real network, the combined effect of many Jacobians can likewise shrink most directions or enlarge some of them. Gradients reaching early layers may then become too tiny to learn from (the vanishing gradient problem) or so large that updates destabilize into wild swings or NaN, the exploding gradient problem.

Saturating activationsactivationsThe values produced by a layer after applying its activation function. During training, intermediate activations are often kept for the backward pass.See in glossary → are the classic culprit. A sigmoid’s slope is at most 0.25 and is near zero whenever its input is large in magnitude. If many units spend time in that flat regime, stacking sigmoid layers makes the backward product prone to collapse.

Slide the depth up with saturating activations and watch the early bars sink toward zero: that network’s first layers are frozen. This is why, through the 2000s, simply making a network deeper often made it worse: the depth that should have helped was unreachable.

The fixes that unblocked deep learning

Four ideas, stacked together, turned “untrainable” into “routine”:

• Activations that don’t flatten on positive inputs. On its positive side, ReLU has slope exactly 1, so it avoids the tiny slope a sigmoid gets when the sigmoid flattens out. ReLU’s negative side has slope 0, and the weights still matter, so ReLU alone does not guarantee healthy gradients, but it removes one major source of repeated shrinking.
• Careful initializationinitializationThe scheme for setting parameters before training starts. Good initialization keeps activations and gradients at sane scales through a deep network so training can get going.See in glossary →. As covered earlier, scaling the starting weights so each layer preserves the size of its signals keeps those per-layer factors near 1 from step one, instead of starting the product already decaying.
• Normalization layers. LayerNormLayerNormLayer Normalization — rescales each token's activation vector to zero mean and unit variance (then applies learned scale/shift), stabilizing training. RMSNorm is the cheaper modern variant.See in glossary → and batch normalizationbatch normalizationA layer that re-centers and re-scales activations using batch statistics to keep them well-conditioned mid-network, easing and speeding training of deep models.See in glossary → re-center and re-scale activations mid-network, preventing them from drifting into the flat, saturated regions where slopes die.
• Residual connections. A residualresidual connectionoutput = x + f(x). Lets gradients flow through deep stacks and means each block adds a refinement rather than rewriting.See in glossary → (or skip) connection computes $x + f(x)$ instead of just $f(x)$, adding a direct path around each block. Backprop through that path includes a multiply-by-1 route, so the gradient has an easier way to reach earlier layers. Turn residuals on in the widget and the flow stays much healthier even with saturating activations.

For the exploding case, there’s also gradient clippinggradient clippingCapping the overall size (norm) of the gradient before the update, to stop occasional huge gradients from destabilizing training.See in glossary →: if the gradient’s magnitude exceeds a threshold, scale it back down before the update. It’s a blunt instrument, but it reliably stops a single bad step from detonating training.

With gradients flowing much more reliably through deep stacks, depth becomes something we can spend. The next question is how to shape that depth: different kinds of data have different structure, and matching the architecture to the data is what the next section is about.