Section 04

The simplest model

Weights, bias, and adjustable parameters

The one-dial machine could only stretch its input. Real problems need a model that can combine several inputs and shift its baseline. The simplest such model (the workhorse that neural networks are ultimately built from) is the linear model. Its weighted sums are also the basic calculations inside much larger networks.

A line, then a plane

Suppose we’re predicting a house’s price. With one input, size, the model is a straight line:

price=w⋅size+b\text{price} = w \cdot \text{size} + b

The weightweightOne adjustable number that scales an input inside a model — how strongly that input pushes the prediction up or down. The weights (plus biases) are the parameters training adjusts.See in glossary → ww is the slope: how many dollars each extra square foot adds. The biasbiasThe constant term added to a weighted sum — the model's baseline output when the inputs contribute nothing. It lets a line shift up and down rather than being pinned to the origin. (Unrelated to "bias" in the fairness sense.)See in glossary → bb is the intercept: the baseline price when everything else is zero. Two dials instead of one, and now the line can both tilt and slide up and down.

Add a second input (the house’s age) and the model becomes:

price=w1⋅size+w2⋅age+b\text{price} = w_1 \cdot \text{size} + w_2 \cdot \text{age} + b

Each input gets its own weight, saying how much that input pushes the prediction up or down; the bias sets the model’s baseline when all the features are zero. It is an intercept, not a lower bound: weighted inputs can push a prediction below it. This is regressionregressionA prediction task whose answer is a number (a price, a temperature), as opposed to a category. Usually scored with squared error.See in glossary →: predicting a number from a weighted combination of featuresfeatureAn individual input the model reads — a house's size, an email's word counts, a pixel. Each feature gets its own weight saying how much it matters.See in glossary → (the inputs). The weights and bias together are the model’s parametersparametersThe numbers (weights) inside a model that get adjusted during training. A “7B model” has 7 billion of them.See in glossary →, the formal name for those dials.

Fit it by hand

Two inputs make the model impossible to draw as a single tilted line, so the widget below shows the fit a different way. The main plot is predicted price vs. actual price: every house is a dot, and a dot lands on the diagonal when the prediction is exactly right. A model that’s fit well pulls all the dots onto that line.

Start with only the bias b switched on (a flat guess that ignores the house entirely) then switch on w₁·size, then w₂·age, tuning each knob to pull the dots onto the diagonal and shrink the loss. When you’ve had enough, “Auto-fit” solves for the best settings so you can see how close you got.

Fit house prices by hand
14 homes, each with a size and an age, and a sale price. Switch terms on and drag their knobs to make the model's predictions match reality. (size in thousands of sq ft, age in decades, price in $1000s.)
predicted vs actual price — dots on the line = perfect
actual →↑ predicted
price vs size — model curve at mid-range age
size →↑ price
180
0
0
0.0
0.0
0.0
price ≈ 180
loss L = (1/n) Σ (ŷ − y)² = 35102.9R² = —· poor fit
Auto-fit solves for the best coefficients of whatever terms are on — try beating it by hand.
Start with just b — a flat guess, blind to the house. Add w₁·size and the line tilts; add w₂·age and the leftover spread from age collapses. The coefficients are the knobs; the loss is the score. Learning, in one sentence, is just turning the knobs to shrink the score — and the next sections are about doing it automatically.

A few things worth noticing as you play:

  • With only b, the model is blind: it predicts the same price for a mansion and a shack, and the loss is huge.
  • Adding size tilts the fit and captures most of the signal. Adding age mops up the spread that size alone couldn’t explain.
  • You can keep adding fancier terms (size², size·age) and the loss keeps dropping. Fitting the training dots ever more tightly can lead to overfittingoverfittingWhen a model memorizes training-set quirks instead of learning general patterns, so it does well on training data but poorly on new data. Rarely the main worry in single-epoch LLM pre-training, but it shapes data choices.See in glossary →: learning details of these examples that do not help predict new ones.

The equation and the score, side by side

The widget shows two things live: the current equation (only the terms you switched on) and the loss, here the mean squared errormean squared errorA common loss for predicting numbers: average the squared gap between each prediction and its true value. Squaring makes all errors positive and gives especially large misses extra weight. L = (1/n) Σ (ŷ − y)².See in glossary →, written L=1n∑(y^−y)2L = \frac{1}{n}\sum(\hat{y}-y)^2. That formula just says: for each house, take the gap between the prediction y^\hat{y} and the truth yy, square it so under- and over-shooting both count as error, and average over all nn houses. One number for total wrongness.

From a linear model to a network

A linear model is just a weighted sum of inputs plus a bias, and “fitting” it means turning those knobs until the loss is small. A neural networkneural networkA function built by stacking many simple operations — mostly matrix multiplies with nonlinearities between them — whose behavior is shaped by tuning billions of internal numbers (its parameters) from data.See in glossary → combines many such weighted sums with functions that bend their outputs. Its parameters are still adjusted to reduce a loss, just as you did by hand. Before we automate the knob-turning, we need to pin down the score itself, because “how wrong are we?” is answered differently for numbers than for categories.