Section 03

Turning the world into numbers

Features, encoding, and why inputs get scaled

A model does arithmetic and nothing else. It multiplies numbers by its knobs, adds them up, and reads out a result. So before any learning can happen, whatever you want to predict from (a house, an email, a customer) has to be turned into a list of numbers. Getting that translation right matters more than beginners expect: a model can only ever be as good as the numbers you feed it.

Features: the numeric inputs

Each number describing an example is a featurefeatureAn individual input the model reads — a house's size, an email's word counts, a pixel. Each feature gets its own weight saying how much it matters.See in glossary →. A house might be described by the features size in square feet, age in years, number of bedrooms. Stack them into a list ([1800, 12, 3]) and that list is what the model actually sees. It never meets the house; it meets the vector.

Genuinely numeric features slot straight in: size, age, temperature, count. They already are numbers, and their magnitudes carry meaning a model can use.

Categories don’t have a number

Plenty of features aren’t numeric, though. A house’s neighborhood, a shirt’s color, a user’s country. These are categorical features, and they resist a naive translation. You might be tempted to number the neighborhoods 1, 2, 3, but that quietly lies to the model: it implies neighborhood 3 is “three times” neighborhood 1, or that 2 sits between them. There’s no such order.

The standard fix is one-hot encodingone-hot vectorA vector that is 1 at a single index and 0 everywhere else. Useful for representing categories without inventing a fake numeric order.See in glossary →: give each category its own column, set to 1 for the matching category and 0 everywhere else. Three neighborhoods become three columns; “Riverside” is [1, 0, 0], “Hillcrest” is [0, 1, 0]. Now no false ordering sneaks in: each category is just its own on/off switch. Plain yes/no features (has a garage?) are the easy case of the same idea: a single column of 0 or 1.

Why raw features need scaling

Now the subtle one. Suppose two of your features are size (roughly 500–4,000) and bedrooms (roughly 1–6). Both are perfectly valid numbers, but they live on wildly different scales: one is in the thousands, the other in single digits. For models trained by taking many small downhill steps, that mismatch can make learning much harder.

For a linear model with squared error, training has to find the bottom of a bowl-shaped error surface, and the shape of that bowl depends on the features. Feature scales can stretch it: a knob attached to a huge-valued feature has an outsized effect on the error, making its direction steep; a knob on a small-valued feature makes a shallow one. Mix the two and the bowl may become a long, thin canyon (steep across, shallow along) so one step size makes awkward progress, zig-zagging down the canyon. Other problems have more complicated surfaces, but mismatched scales can still make the downhill route harder to follow. (We’ll make this picture precise in a few chapters; here, just watch the shape.)

A common fix is standardizationstandardizationPutting numeric features on comparable scales by subtracting each feature's training-set average and dividing by its typical spread. This often makes slope-following training easier.See in glossary →: rescale each feature to a comparable range, typically by subtracting its training-set average and dividing by its typical spread, so each feature ends up centered at 0 with a similar scale. This preserves the information while often making the downhill route less stretched and easier to follow. It does not separate features that tend to move together or guarantee a perfectly round bowl, but it usually gives training a much friendlier starting shape. Toggle it and watch:

Why scaling the inputs matters
The two axes are the model’s two knobs. On raw features the loss bowl is stretched, so one learning rate can’t suit both directions. Normalize and the bowl rounds out.
step 0loss 7.050
lopsidedness (condition #) = 23.3×steps to converge = 25· stretched canyon — descent zig-zags and crawls
Both runs use the identical learning rate and starting point. On raw features one axis is far steeper than the other, so a step safe for the steep axis barely moves the shallow one — the marble slaloms down a canyon for dozens of steps. Normalizing rescales the axes to comparable curvature, and the same descent reaches the bottom in a handful of steps.

Once the inputs are clean numbers (categoricals encoded, scales tamed) the model can finally do its one trick: combine those numbers into a prediction. The next section builds the simplest machine that does exactly that, a weighted sum, and shows it’s already enough to learn something real.