Section 11

Neural networks

Layers, activations, and why nonlinearity matters

A linear model draws straight lines. That’s a hard ceiling: no matter how you set the weights, w·x + b can only ever be a line (or, with more inputs, a flat plane). Most real patterns aren’t straight: prices bend, images curve, language loops back on itself. A neural networkneural networkA function built by stacking many simple operations — mostly matrix multiplies with nonlinearities between them — whose behavior is shaped by tuning billions of internal numbers (its parameters) from data.See in glossary → breaks the ceiling with one deceptively small change, and this section is about what that change is and why it’s enough.

Stacking lines isn’t enough, until you bend them

The obvious idea is to stack linear models: feed the output of one into the input of the next. It doesn’t work, and the reason is important. A line of a line is still a line. Stack a hundred linear layers and, algebraically, they collapse into a single linear layer. You’ve gained nothing but wasted compute.

The fix is to insert a simple nonlinear step between the linear layers: after each weighted sum, pass the result through a nonlinear function before handing it on. That function is called an activationactivationsThe values produced by a layer after applying its activation function. During training, intermediate activations are often kept for the backward pass.See in glossary → function, and its nonlinearity is what stops the layers from collapsing. Some activations are smoothly curved; ReLU is piecewise linear, with a sharp bend at zero. Either kind lets each layer reshape its input so stacking genuinely adds expressive power.

Common activation functions
Three nonlinearities you'll see inside the MLP block. They all do the same job — bend the otherwise-linear network — and differ mostly in the shape of the bend.
-4-2024024xf(x)
ReLUOriginal Transformer
A literal hinge at zero. Below zero: dead. Above zero: passes through unchanged. Simple, cheap, but has a "dying ReLU" problem where neurons can get stuck outputting 0 forever.
GELUGPT-2 / GPT-3 / BERT
A smooth, slightly curved relative of ReLU. Lets a small amount of negative signal through near zero, which empirically trains better. The default for most pre-Llama models.
SiLU (Swish)Llama, Mistral, Qwen (inside SwiGLU)
Even smoother than GELU. Has a small dip below zero before flattening, which seems to help gradient flow. Used as the gating function inside SwiGLU.

These are common choices. They share a family resemblance (many pass positive values and suppress negative ones) but their details do matter for optimization and numerical stability. For the big idea here, though, the crucial point is simpler: without a nonlinear bend between linear layers, the stack collapses back into one linear layer.

Anatomy of a network

The vocabulary is simple:

  • A layer is a batch of linear units (each a weighted sum + bias) followed by an activation.
  • The layers between input and output are hidden layershidden layerA layer of a neural network between the input and the output. "Hidden" because its values are internal scratch space you don't directly observe. Stacking hidden layers (with nonlinearities) is what gives networks their power.See in glossary →: “hidden” only because you don’t directly observe their values; they’re the network’s internal scratch space.
  • Depth is how many layers; width is how many units per layer. More of either means more parameters and more capacity.

Every one of those weights and biases is a knob, and — this is the payoff of the last three sections — the entire network is still just one big function with a loss, so gradient descent trains all of it at once, exactly as it trained the line.

Watch nonlinearity matter

Here are two classes of dots arranged in a checkerboard: no straight line can separate them. Train with the hidden layer off and the model is really just the linear classifier from before: it can only place one straight boundary, so this dataset keeps it near coin-flip accuracy. Switch the hidden layer on and train again: the same loop, same data, but now the boundary can bend, wrap the classes, and reach high accuracy.

A network learns a boundary
Two classes of dots, arranged so no straight line can split them. Train with the hidden layer off (a straight boundary) and on (a bendable one), and watch the accuracy.
epoch 0loss 0.669acc 59%
The linear model tops out around chance — one straight cut can't carve this checkerboard. Add a hidden layer and its 8 units each learn a simple half-plane; the output layer combines them into a curved boundary that wraps the classes. Same dots, same training loop — the only new ingredient is the nonlinearity between the layers.

Each hidden unit learns one simple straight cut; the output layer combines those cuts into a nonlinear boundary. With ReLU that boundary is assembled from straight pieces rather than being smoothly curved. Eight units are plenty to fence off this checkerboard. Nothing about the training changed, only the presence of the nonlinearity between two linear layers.

We can now build arbitrarily expressive models and we know, in principle, how to train them: step against the gradient. But we’ve leaned on one word without justifying it: deep. Why stack many layers instead of one very wide one? The next section shows what depth actually buys (each layer building features out of the layer below) before we tackle how to compute the gradient through all of them.