Section 19

Data quality & the three splits

Garbage in, and holding data back

The previous section ended by pointing at the data. Let’s take it seriously. A model is a machine for extracting patterns from examples, and it has no way to reach past those examples to the truth. If the examples are wrong, skewed, or secretly contaminated, the model faithfully learns the flaw. This is the oldest rule in computing, and it never stopped being true: garbage in, garbage out. Before we can judge a model fairly, we need data worth judging it on, and a disciplined way to split it.

Failure mode 1: label noise

The lossloss functionA single number measuring how wrong the model's predictions are on a batch of data. Training works by adjusting the model to make this number smaller.See in glossary → compares each prediction to a label: the “right answer” attached to that example. But labels are made by people, scraped from messy sources, or inferred by imperfect rules, so some of them are simply wrong. When the answer key itself contains errors, the model is graded against those errors and dutifully learns them.

The widget below shows a clean two-class problem: the dashed line is the true boundary, and at zero noise a simple model fits it almost perfectly. Now flip a fraction of the training labels and refit.

Label noise misleads what a model learns
Two clean classes split by the dashed true boundary. Drag the slider to flip a fraction of the training labels (amber rings), refit the model, and watch the honest test accuracy fall.
— true · — fitted
train acc (vs given labels) 100%test acc (clean, held out) 100%
At 0% the fitted line sits right on the true boundary and test accuracy is near perfect. As you inject noise, the flipped points pull the boundary off course — the model dutifully chases wrong answers it has no way to know are wrong. Training still looks respectable because it is graded against those same corrupted labels, while the clean test set shows the damage: garbage in, garbage out.

Watch what happens: the corrupted points drag the fitted boundary off the true line, and the honest test accuracy falls. The model can’t tell a mislabeled point from a real one: a wrong answer looks exactly like a pattern to be learned. Enough noise can cap how well any model agrees with a noisy answer key, and it can badly mislead training even when the final test set is clean.

Failure mode 2: biased, unrepresentative data

Even with perfect labels, data can mislead by what it leaves out. A model trained mostly on daytime photos may struggle at night; one trained on one hospital’s patients may fail at another. Models can be confidently wrong on examples very different from their training data, and an ordinary confidence score may not warn you. Special tools try to detect unfamiliar inputs, but they are imperfect. Your training set is an implicit claim that the future will look like the past; when a group, condition, or scenario is missing, the model may inherit that blind spot.

Failure mode 3: leakage and contamination

The subtlest failure is when information about the answers sneaks into training. If a test example (or a near-duplicate of it) appears in the training data, the model may benefit from that exposure and post a score that no longer estimates performance on unseen examples. This is contaminationdata contaminationWhen test or benchmark data leaks into the training corpus, inflating scores. Careful pipelines try to detect and remove contamination before training.See in glossary →. It invalidates the clean interpretation of the benchmark, whether the model copied the example verbatim or absorbed it in a less direct way.

The three-way split

You already know the train/test splittrain/test splitHolding back part of your data from training so you can measure the model on examples it never saw. Good results there are evidence that its patterns carry over to similarly collected new data.See in glossary →: learn on one pile, judge on another the model never saw. In practice you need a third pile, because building a model isn’t one decision: it’s hundreds. Which architecture? How much regularizationregularizationA change to training that tries to improve performance on new data, often by limiting how freely the model can fit the training examples. Weight penalties, dropout, and early stopping are common forms.See in glossary →? Which hyperparametershyperparameterA training setting you choose rather than learn — learning rate, batch size, number of layers, etc. Tuning these well is much of the craft of pre-training.See in glossary →? You answer those by trying options and keeping the best.

So split the data three ways:

  • Training set: used to fit the parametersparametersThe numbers (weights) inside a model that get adjusted during training. A “7B model” has 7 billion of them.See in glossary → (the knobs gradient descent turns).
  • Validation set: looked at repeatedly to compare models and tune hyperparameters; you pick the version that scores best here.
  • Test set: saved until the end, for the most honest estimate of real-world performance.

Why not just tune against the test set directly? Because the moment you start choosing models by their test score, you’re optimizing against it. After enough tries you’ll find a setup that got lucky on that particular test set, not one that’s genuinely better. You’d be overfitting to the test set with your own hands. The validation set absorbs that wear and tear; the test set stays pristine so its verdict still means something.

We now have clean data and a disciplined split. But suppose the model is trained and we hold up a fresh, honest slice of data. How exactly do we score success? The loss got us here, but as the next section shows, the loss is almost never the number you actually care about.