Section 14

The training loop

Forward → loss → backward → update → repeat

Everything so far has been a component. Here they click together into a single cycle — the training loop — that is repeated until the model is done. It’s four steps, and once you’ve seen them you’ve seen the basic pattern behind training neural networks, from a two-parameter line to a frontier language model. Real systems add engineering around it, but the core loop is this.

The four phases

Each steptraining stepOne iteration of the loop: forward pass on a batch, backward pass to get gradients, optimizer update. A large model is trained for hundreds of thousands of steps.See in glossary → of training does exactly this:

  1. Forward pass. Take a mini-batchmini-batchThe chunk of training examples processed together in one step. Gradients are averaged over the mini-batch, trading off gradient noise against memory and compute.See in glossary → of examples and run them through the model to get predictions.
  2. Loss. Compare predictions to the true answers, producing one number: how wrong we were on this batch.
  3. Backward pass. BackpropagatebackpropagationThe algorithm that computes the loss gradient for every parameter efficiently by applying the chain rule backward through the network, reusing intermediate results from the forward pass.See in glossary → to get the gradient: every parameter’s share of the blame.
  4. Update. Nudge every parameter a small step against its gradient: θ←θ−η ∇L\theta \leftarrow \theta - \eta\,\nabla L.

Then throw away the batch, grab a fresh one, and repeat. That’s the entire algorithm. Frontier models run this loop hundreds of thousands to millions of times.

Step through it

The widget fits a line to a handful of points, but it runs the real loop, one phase at a time so you can watch the hand-off. Only the update phase moves the line. The other three just work out which way and how far. The curve on the right records the loss after every completed step; watch it march downward as the line settles onto the data.

One training loop, four phases
Step through it, or let it run. Only the update phase changes the line; the other three just work out which way and how far to nudge it.
1. Forward
run each x through ŷ = w·x + b
2. Loss
measure how wrong: mean squared error
3. Backward
backprop → gradient for w and b
4. Update
nudge w, b downhill by the gradient
x →training step →loss
step 0w 0.20 b 0.00loss 29.979
Watch the loss curve step down and the line settle onto the points. Large training runs repeat this same core cycle many times, with billions of parameters instead of two — but the four phases, and the fact that only the update phase changes the parameters, stay recognizable.

The two things to take away from watching it: the loss falls in steps, not smoothly (each step is one update), and only one of the four phases actually changes the model. The rest are the bookkeeping that makes a good change possible.

Why this is the whole game

It’s worth pausing on how little there is here. A model is a function with knobs. A loss scores it. Backprop finds which way each knob should turn. The loop turns them, a little, a great many times. Scale each piece up (more knobs, more data, more steps) and the same basic loop can train models that handle language, images, or houses. Nothing in the loop knows what the input means; it only knows predict, score, assign blame, nudge.

This loop trains our small networks without complaint. But we glossed over step zero: before the very first forward pass, every weight needs a starting value. That sounds trivial and turns out to be anything but. Set them carelessly and the loop never gets off the ground, no matter how good the rest of it is. The next section is about where the numbers start.