The training loop

Forward → loss → backward → update → repeat

The training loop combines prediction, scoring, and parameter updates into a repeated cycle. Its four phases are shared by models ranging from a two-parameter line to a large language model.

The four phases

Each steptraining stepOne iteration of the loop: forward pass on a batch, backward pass to get gradients, optimizer update. A large model is trained for hundreds of thousands of steps.See in glossary → of training does exactly this:

1. Forward pass. Take a mini-batchmini-batchThe chunk of training examples processed together in one step. Gradients are averaged over the mini-batch, trading off gradient noise against memory and compute.See in glossary → of examples and run them through the model to get predictions.
2. Loss. Compare predictions to the true answers, producing one number: how wrong we were on this batch.
3. Backward pass. BackpropagatebackpropagationThe algorithm that computes the loss gradient for every parameter efficiently by applying the chain rule backward through the network, reusing intermediate results from the forward pass.See in glossary → to get the gradient: every parameter’s share of the blame.
4. Update. Nudge every parameter a small step against its gradient: $\theta \leftarrow \theta - \eta\,\nabla L$.

Then throw away the batch, grab a fresh one, and repeat. That’s the entire algorithm. Frontier models run this loop hundreds of thousands to millions of times.

Step through it

The widget fits a line to a handful of points, but it runs the real loop, one phase at a time so you can watch the hand-off. Only the update phase moves the line. The other three just work out which way and how far. The curve on the right records the loss after every completed step; watch it march downward as the line settles onto the data.

The two things to take away from watching it: the loss falls in steps, not smoothly (each step is one update), and only one of the four phases actually changes the model. The rest are the bookkeeping that makes a good change possible.

The same loop at different scales

A model is a function with knobs. A loss scores it. Backprop finds which way each knob should turn. The loop turns them, a little, a great many times. Scale each piece up (more knobs, more data, more steps) and the same basic loop can train models that handle language, images, or houses. Nothing in the loop knows what the input means; it only knows predict, score, assign blame, nudge.

Before the first forward pass, every weight needs a starting value. Choosing those values is called initializationinitializationThe scheme for setting parameters before training starts. Good initialization keeps activations and gradients at sane scales through a deep network so training can get going.See in glossary →. Poor choices can prevent the layers from learning even when the rest of the training loop is working correctly.