Section 07

Which way is downhill?

Slopes and just enough calculus

We have a model with knobs and a loss that scores them. To learn automatically, the machine needs to answer one question, over and over: if I nudge this knob a tiny bit, does the loss go up or down? That question has a name in mathematics — the derivative — and it’s the only calculus this explainer requires. No integrals, no limits to compute by hand. Just slopes.

The slope is the answer

Fix everything except one knob, w, and plot the loss as w varies. You get a curve. At any point on that curve, the derivativederivativeThe slope of a function at a point: how fast the output changes as you nudge the input, and in which direction. In training, the derivative of the loss with respect to a parameter tells you which way to turn that knob.See in glossary → is simply its slope there: how steeply the loss rises or falls as you increase w, and in which direction.

  • Slope positive: increasing w increases the loss. So to lower the loss, decrease w. Move left.
  • Slope negative: increasing w decreases the loss. Move right.
  • Slope zero: the curve is flat right here. You might be at a bottom, a top, or a flat spot. With many knobs, you could also be at a saddle pointsaddle pointA flat-looking point that is downhill in some directions but uphill in others — like the center of a horse saddle. A zero gradient can mark a saddle rather than a minimum.See in glossary →: downhill in some directions but uphill in others. The slope alone cannot tell you which.

That’s it. The sign of the slope tells you which way is downhill; the size of the slope tells you how steep the ground is. You never need the model to “understand” anything. It just needs to feel the slope under each knob.

Drag the point

Move the dot along the loss curve. The straight line touching it is the tangent, and its steepness is the slope at that spot. Watch the sign flip as you cross the bottom of the valley, and watch the arrow point the way the loss falls.

The slope tells you which way is downhill
Drag the dot along the loss curve. The straight line is the tangent — its steepness is the slope (the derivative) at that point. Its sign is all you need to know which way to move.
parameter w →loss
w = 0.60loss = 3.017slope dL/dw = -1.432
slope is negative → uphill is left, so downhill is RIGHT
Notice the rule: to go downhill you move against the slope. Positive slope → step left, negative slope → step right, and the size of the slope tells you how steep it is right there. That single sentence — w ← w − (learning rate)·slope — is gradient descent, and it's the engine of the next section.

The rule the widget keeps stating is the one to memorize:

w←w−η⋅dLdww \leftarrow w - \eta \cdot \frac{dL}{dw}

Read it plainly: new w = old w, minus a small step in the direction of the slope. The minus is the important part. It’s what makes you go against the slope, i.e. downhill. The little η\eta (the Greek letter “eta”) controls how big a step to take; it’s called the learning ratelearning rateThe size of each parameter step. Too high and training can diverge; too low and it crawls.See in glossary →, and the next section shows how much it matters.

From one knob to many

A real model has millions of knobs, not one. The idea generalizes without any new concepts: for each knob, ask “what’s the slope of the loss along this knob, holding the others fixed?” That per-knob slope is called a partial derivativegradientThe vector of partial derivatives of the loss with respect to every parameter — it points in the direction of steepest loss increase, so we step the opposite way to reduce the loss.See in glossary →, and the full list of them (one number per parameter) is the gradient. It’s just “the slope” pointing in every knob’s direction at once.

Knowing which way is downhill isn’t the same as knowing how to walk there without falling off a cliff or getting stuck. That walk (following the gradient, step after step) is the whole subject of the next chapter, and it has a surprising number of ways to go wrong.