Section 08

Gradient descent

The loss surface, step size, and getting stuck

We know which way is downhill. Gradient descentgradient descentThe core training algorithm: repeatedly nudge each parameter a small step in the direction that lowers the loss, as told by the gradient.See in glossary → is the act of actually walking there: take the gradient, step against it, and repeat. It’s almost embarrassingly simple (one line of arithmetic looped a few hundred thousand times) and yet nearly everything that makes training hard shows up in how that loop behaves.

The loss as a landscape

Picture the loss as a surface. Each knob is a direction you can walk, and the height at any spot is the loss for those knob settings. Training a model is a marble rolling around this loss landscapeloss landscapeThe (extremely high-dimensional) surface of loss as a function of the parameters. Training is a walk downhill on this surface toward a low-loss region.See in glossary →, trying to find a low valley. With two knobs it’s a literal hilly terrain; with a billion it’s a landscape in a billion dimensions, but the intuitions from the 2-D picture carry over remarkably well.

At every step we compute the gradient (the steepest-uphill direction) and move the opposite way by a step whose size is set by the learning ratelearning rateThe size of each parameter step. Too high and training can diverge; too low and it crawls.See in glossary → η\eta:

θ←θ−η ∇L\theta \leftarrow \theta - \eta \, \nabla L

Here θ\theta (“theta”) just means “all the parameters at once.” Same rule as the single knob from last section, applied to every knob simultaneously.

The step size is everything

Roll the marble down this bowl. The bowl is deliberately lopsided (steep in one direction, shallow in the other) because that’s what real loss surfaces are like. Set the learning rate and step or run:

Gradient descent on a loss surface
The bowl is steeper top-to-bottom than side-to-side. Set the learning rate, then step the ball downhill along the negative gradient.
step 0loss 6.528
The gradient points uphill, so we step the opposite way: θ ← θ − η·∇L. One learning rate has to serve both directions at once — too large for the steep axis means zig-zagging or blowing up, too small for the shallow axis means crawling. Optimizers like Adam help by giving every parameter its own effective step size.

Play with η\eta and the central tension of all training appears at once:

  • Too small: progress is correct but glacial. You’ll converge eventually, after far more compute than you can afford.
  • Too large: the marble overshoots the valley floor, bounces up the far wall, and zig-zags or, past a threshold, flies off entirely. The loss can shoot upward instead of falling. Training has diverged, and that run usually has to be restarted from an earlier safe point.
  • Just right: brisk, stable descent to the bottom.

And notice the deeper problem the lopsided bowl reveals: no single learning rate is ideal for both directions. A step small enough to be safe on the steep axis is painfully slow on the shallow one. A real network has billions of axes with wildly different steepness, all sharing one η\eta. Fixing this is the entire job of the smarter step rules the next chapter builds, but this is the itch they scratch.

One honest simplification: we don’t use all the data each step

The true loss averages over the entire training set. Computing its exact gradient would mean running the whole dataset through the model for a single step, impossibly expensive when the dataset is huge. So instead we estimate the gradient from a small random mini-batchmini-batchThe chunk of training examples processed together in one step. Gradients are averaged over the mini-batch, trading off gradient noise against memory and compute.See in glossary → each step. With a properly random sample, the estimate is noisy but cheap and, on average, points the right way. This is stochastic gradient descentSGDStochastic Gradient Descent — gradient descent using a noisy gradient estimated from one mini-batch at a time rather than the whole dataset.See in glossary → (“stochastic” just meaning “with randomness”) and the noise can even help, jiggling the marble out of shallow traps.

Plain gradient descent works, but the lopsided bowl exposed its central weakness: one global step size can’t suit every direction at once, and real loss surfaces are lopsided in millions of directions. The next section is about the fixes (building up speed in the consistent directions, and giving each parameter its own step size) that make the descent far faster and steadier.