Section 09

Tuning the descent

Making the descent faster and smarter

The last section left us with a complaint: one global learning ratelearning rateThe size of each parameter step. Too high and training can diverge; too low and it crawls.See in glossary → can’t suit a lopsided bowl, where the loss is steep in one direction and shallow in another. A step small enough to stay safe on the steep axis crawls on the shallow one; a step brisk enough for the shallow axis overshoots and zig-zags on the steep one. Real loss surfaces are lopsided in millions of directions at once. This section is about two ideas that help, and the optimizeroptimizerThe rule that turns gradients into parameter updates. Plain gradient descent is the simplest; Adam-family optimizers add per-parameter adaptive step sizes.See in glossary → that combines them and became a default choice for many neural networks.

Momentum: give the ball some weight

Plain SGDSGDStochastic Gradient Descent — gradient descent using a noisy gradient estimated from one mini-batch at a time rather than the whole dataset.See in glossary → has no memory. Each step looks only at the current gradient, so on a steep valley it bounces off one wall, then the other, wasting most of its motion going sideways. The consistent signal (“we should be heading down the length of the valley”) is drowned out by the loud back-and-forth.

MomentummomentumAn optimizer trick that accumulates a running average of past gradients, letting updates build up speed in consistent directions and damp out oscillations.See in glossary → gives the marble mass. Instead of using only the newest gradient, we keep a weighted memory (a velocity) where recent gradients count most and older ones gradually fade:

v←β v+∇L,θ←θ−η vv \leftarrow \beta \, v + \nabla L, \qquad \theta \leftarrow \theta - \eta \, v

Along the shallow axis the gradient always points the same way, so the velocity builds and the ball picks up speed. Across the steep axis the gradient flips sign every step, so those contributions largely cancel. The oscillation damps; the useful direction compounds. With β\beta around 0.90.9, a gradient’s influence fades substantially over roughly ten steps. Some versions rescale this memory slightly; that scale can be folded into the learning rate.

Adaptive rates: a different step size per parameter

Momentum smooths direction but still uses one η\eta for every parameter. The second idea attacks that directly: give each parameter its own effective step size, scaled by how large its gradients have recently been. A parameter whose gradient is consistently big (a steep direction) gets its step shrunk; one whose gradient is tiny (a shallow direction) gets its step enlarged. This is the RMSProp idea (divide each parameter’s step by the root-mean-square of its recent gradients) and it’s precisely the per-direction rescaling the lopsided bowl was begging for.

Adam = momentum + adaptive rates

AdamAdamAdaptive Moment Estimation — an optimizer that tracks running averages of the gradient (first moment) and its square (second moment) to give each parameter its own adaptive step size.See in glossary → is the obvious marriage of the two: keep a running average of the gradient (momentum, the first moment) and a running average of its square (the adaptive scale, the second moment), then step along the first divided by the square root of the second. Momentum picks a smooth direction; the adaptive term sets a sensible size for each parameter along it. You often need less tuning than with plain SGD, which is a large part of why Adam became so popular.

Three optimizers, one lopsided bowl
Same start, same learning rate, same surface. Plain SGD zig-zags across the steep valley; momentum and Adam smooth the ride and reach the floor sooner in this toy bowl.
plain SGD
step 0loss 6.528
SGD + momentum
step 0loss 6.528
Adam
step 0loss 6.528
At a moderate η, plain SGD (rose) bounces off the steep walls and inches down the shallow axis. Momentum (amber) builds speed along the consistent shallow direction while its oscillations cancel. Adam (teal) additionally shrinks the step on the steep axis and enlarges it on the shallow one, so in this bowl it drives almost straight to the bottom. Crank η up and watch which one blows up first.

Run all three from the same corner. Plain SGD (rose) rattles down the valley walls; momentum (amber) surges along the floor once its sideways bounces cancel; Adam (teal) rescales each axis and drives almost straight to the minimum in this toy bowl. Push η\eta up and you’ll see SGD blow up long before Adam flinches.

We now have optimizers that steer intelligently down a lopsided landscape. But every one of them, momentum and Adam included, has quietly assumed something we can’t actually afford: that we can compute the true gradient (the one averaged over the entire dataset) at every single step. The next section is about the compromise that makes real training possible: estimating that gradient from a small random sample of the data.