Section 17

Shapes of networks

Matching the architecture to the data

The network we’ve built so far (the plain fully-connected MLPMLPMulti-Layer Perceptron — a stack of dense (matrix-multiply + nonlinearity) layers applied per-token. The transformer’s feed-forward block.See in glossary →) has a blind spot: it has no built-in idea that nearby input numbers are related. Hand it an image and each pixel position gets its own separate weights; the fact that neighboring pixels form edges, or that a cat is still a cat when it slides ten pixels to the right, is not built into the wiring. It can learn those facts, but only by burning huge numbers of parameters and examples to rediscover structure that was staring it in the face. The fix is to change the shape of the network so the structure is built in.

Architecture is a set of built-in assumptions

Every architecture is really a bet about how the data is organized, a prior baked into the wiring. Get the bet right and the network needs far fewer parameters and far less data, because it isn’t wasting capacity learning things you already knew.

Convolutional networks (CNNs)convolutional network (CNN)A network that slides small shared filters over grid-like data such as images. Reusing each filter at every position saves parameters and makes shifted patterns easier to recognize.See in glossary → make the bet for images. Instead of wiring every unit to every pixel, a CNN uses a small filter (a few weights) that it slides across the whole image, computing one output per position. Two assumptions are wired in: locality (a pixel is best understood from its neighbors, not the far corner of the image) and shared pattern detection (a pattern should be recognized by the same weights wherever it appears). That second idea, called weight sharingweight sharingReusing the same weights at many positions (as a CNN filter does), so a pattern learned once is recognized everywhere and far fewer parameters are needed.See in glossary →, is what makes CNNs so parameter-efficient. In the simplest case, convolution has translation equivariancetranslation equivarianceA shift-in, shift-out property: when the input image moves, the grid of detected features moves with it. This differs from invariance, where the final answer would not change at all.See in glossary →: slide the input, and the feature map slides with it. Pooling and aggregation can make the final answer less sensitive to shifts, but ordinary CNNs are not automatically unchanged by them: image edges, added borders, larger step sizes, and skipped positions can all make a small shift change the answer.

A filter and its receptive field
A convolutional unit looks at only a small patch, slides the same weights across the whole image, and emits one number per position. Drag the filter, or switch to a fully-connected unit that touches every pixel.
weights 9 · reused at every positionoutput -0.089
The convolutional unit only sees its 3×3 patch — its receptive field — and produces a single number. The same nine weights slide across the whole image, so a pattern learned in one corner is recognized everywhere. That is locality plus weight sharing: far fewer parameters, and a built-in assumption that matches how images work.

Slide the filter around: it only ever sees a small patch (its receptive fieldreceptive fieldThe region of the input that a given unit actually looks at. Small near the input in a CNN, growing with depth.See in glossary →) yet the same nine weights are reused at every location. Flip to the fully-connected view and the contrast is stark: one unit, wired to all hundred pixels, with a hundred separate weights and no idea that neighbors belong together. Same data, wildly different efficiency.

Other shapes for other structure

Images aren’t the only kind of structured data, and each kind has an architecture shaped to fit it:

  • Recurrent networks (RNNs)recurrent network (RNN)An architecture that processes a sequence one step at a time, carrying a memory (hidden state) forward. A pre-transformer approach to sequences.See in glossary → are built for sequences: text, audio, time series. They process one step at a time, carrying a memory (a hidden state) forward that is intended to summarize the relevant history. That memory is limited and may forget information from far back. The built-in assumption is order: what came before conditions what comes next.
  • Attention and transformerstransformerA neural-network architecture introduced in "Attention Is All You Need" (2017), built from stacked self-attention and feed-forward layers.See in glossary → take a different bet on sequences: instead of passing information hand-to-hand down a chain, they let each position look directly at other allowed positions and decide what’s relevant. In many language models, “allowed” means earlier positions only, because the model is predicting what comes next. That direct access is why transformers came to dominate language, where one piece of text can depend on another one far away.

The unifying idea is worth stating plainly: an architecture is a set of assumptions about the data’s structure, and the closer those assumptions match reality, the fewer parameters and less data the model needs to learn well. CNNs assume locality and shared patterns, RNNs assume order, transformers assume positions may need direct access to other relevant positions. The plain MLP builds in fewer of these shortcuts, which is why it is often less efficient on structured data.

But whatever shape we choose, none of it counts unless the model performs on data it has never seen. Fitting the training set is easy; the real test is generalization, and that is where we turn next.