Why depth wins

Features built on features

We ended the last section with a loose thread: a single hidden layerhidden layerA layer of a neural network between the input and the output. "Hidden" because its values are internal scratch space you don't directly observe. Stacking hidden layers (with nonlinearities) is what gives networks their power.See in glossary →, given enough units, can already approximate almost any function. So why do real networks stack many layers instead of one enormously wide one? The answer is the single most important idea about why deep learning works, and it has a name: networks build a hierarchy of features, each layer standing on the shoulders of the one below.

Wide reinvents; deep reuses

Picture the task of recognizing objects in images. In a shallow-but-wide network, every hidden detector works directly from the raw pixels. It can still learn and reuse useful patterns through the output layer, but it has only one stage in which to turn pixels into “cat” or “car.”

A deep network works differently. Its first layerlayerOne stage of a neural network: a group of units that transform the values from the previous stage before passing them on.See in glossary → learns to spot the simplest things: little oriented edges, patches of color, tiny gradients. The second layer doesn’t look at pixels at all. It looks at the first layer’s edge detectors and learns to combine them into corners, curves, and textures. The third combines those into parts: an eye, a wheel, a wheel-arch. The next combines parts into whole objects. Each layer invents features by composing the features beneath it.

The potential win is reusing building blocks. Once an early layer responds to useful edge patterns, later layers can combine those responses instead of working from raw pixels again. When the problem itself is built from simpler patterns joined into larger ones, a deep network can sometimes represent it with far fewer parametersparametersThe numbers (weights) inside a model that get adjusted during training. A “7B model” has 7 billion of them.See in glossary → than a shallow alternative. That is not a guarantee for every task: depth is economical when its layered structure matches the problem.

The network invents its own features

Back in the early chapters, when we fit a line to house prices, we chose the features by hand: square footage, number of bedrooms, distance to downtown. The model only got to weight features a human had already decided were relevant. That approach caps the model at your imagination, and for images or language, nobody can hand-write the thousands of features that matter.

This is the quiet revolution in the hierarchy above: nobody hard-coded the exact edge, corner, and shape detectors. People designed the network, its training goal, and how its examples were prepared, but the detector weights emerged from training. Gradient descent, pushing to lower the loss, often discovers that useful early patterns make the whole job easier later and wires the layers accordingly. Learning useful internal features instead of specifying every featurefeatureAn individual input the model reads — a house's size, an email's word counts, a pixel. Each feature gets its own weight saying how much it matters.See in glossary → by hand is called representation learningrepresentation learningWhen a network discovers useful features from raw data on its own — building higher-level features out of lower-level ones — instead of relying on hand-designed inputs.See in glossary →, and it was a major reason deep networks displaced many hand-engineered systems in vision and language.

We now know why to stack layers: each one composes richer features from the layer below, so depth expresses complex functions cheaply and lets the network invent its own representations. But all of that only works if we can actually train every layer, and that means computing the gradientgradientThe vector of partial derivatives of the loss with respect to every parameter — it points in the direction of steepest loss increase, so we step the opposite way to reduce the loss.See in glossary → for every weight in the stack, including the ones buried deepest. How that’s done, efficiently, is the subject of the next section.