Section 05

Measuring wrongness

How we measure a wrong answer

The loss is the number the whole machine is trying to make small, so it’s worth getting right. A useful loss has to reward predictions we actually consider good and give the training method a downhill direction to follow at almost every point. Some useful losses have sharp corners rather than perfectly smooth curves; training software can still handle those. How we build the loss depends on what kind of thing we’re predicting.

Predicting a number: squared error

When the answer is a quantity (a price, a temperature, a length) we already have our loss from the last section, the mean squared errormean squared errorA common loss for predicting numbers: average the squared gap between each prediction and its true value. Squaring makes all errors positive and gives especially large misses extra weight. L = (1/n) Σ (ŷ − y)².See in glossary →:

L=1n∑i=1n(y^i−yi)2L = \frac{1}{n}\sum_{i=1}^{n} (\hat{y}_i - y_i)^2

Each prediction’s gap from the truth is squared and averaged. Zero means every prediction is spot on. This is a common loss for regressionregressionA prediction task whose answer is a number (a price, a temperature), as opposed to a category. Usually scored with squared error.See in glossary →: it is simple and smooth, and it emphasizes large misses. That last property can be undesirable when a few examples are extremely unusual, so other losses sometimes count each miss in more direct proportion to its size.

Predicting a category: the surprise view

Most interesting problems aren’t “what number?” but “which one?” Is this email spam or not? Which item in a huge list comes next? This is classificationclassificationA prediction task whose answer is one of a fixed set of categories (spam / not spam, or which of 100,000 tokens comes next). The model outputs a probability per option and is scored with cross-entropy.See in glossary →, where cross-entropy is usually a better training objective than treating class labels as ordinary numeric targets. A classifier outputs probabilities: for a yes/no problem it might give the probability of “yes,” and for a many-way problem it gives a probability for each option. (Exactly how a model turns its raw scores into probabilities is the next chapter’s job; here we only need that it does.)

How wrong is a probability? The trick is to score it by surprise. If the true answer was “spam” and the model said 90%, it was barely surprised: small loss. If it had confidently said 1%, reality just blindsided it: huge loss. The mathematical measure of surprise is the negative logarithm of the probability the model assigned to the correct answer:

L=−log⁡p(correct answer)L = -\log p(\text{correct answer})

This is cross-entropy losscross-entropy lossA classification loss that penalizes the model according to the negative log-probability it assigned to the correct answer.See in glossary →. It is the standard loss for modern classifiers, and it will show up again when we get to language models. Say the right answer with probability 1 and the loss is −log⁡1=0-\log 1 = 0: no surprise, perfect. Say it with probability near 0 and −log⁡p-\log p shoots toward infinity: total shock, enormous penalty.

Feel it

Drag the probability the model assigns to the correct class and watch the loss respond. The relationship is not a straight line: that curve, gentle near “confident and right” and cliff-like near “confident and wrong,” is the whole personality of cross-entropy.

Cross-entropy loss, one answer at a time
Prompt: "The cat sat on the ___". Choose what the model predicts and how confident it is, and watch the loss it pays for the truth.
The correct answer is mat.
Select which answer the model predicts
✓ marks the correct answer.
Model's predicted distribution
✓mat
88.0%
sofa
8.4%
floor
2.6%
table
0.8%
roof
0.2%
Cross-entropy loss
0.128
= −ln(p of "mat")
Probability on truth
88.0%
p("mat")
The loss only looks at the probability the model gave the true answer "mat". When the model bets confidently on the wrong answer, it leaves "mat" almost no probability, so −ln(p) shoots toward infinity. Being confidently wrong costs far more than being unsure — that asymmetry is what teaches the model not to bluff.

One number, whatever the task

Regression or classification, the shape is the same: predictions in, one loss number out, low when the model is good. That single number is what makes automatic learning possible, because now “improve the model” has a precise meaning: reduce this number. But we glossed over something: cross-entropy needs a probability for each class, and a plain weighted sum of inputs can land anywhere from minus to plus infinity. How does a raw score become a probability, and how does that carve the world into categories? The next section opens up the classifier itself, before we turn to the machinery of actually minimizing the loss.