Section 21

A short history

From the perceptron to deep learning, and why now

Everything you’ve built in this explainer has a history, and it’s a bumpy one, full of hype, disillusion, and long quiet stretches where almost no one believed. Knowing that story does something useful: it shows that the core ideas are old, that the recent explosion wasn’t a single breakthrough, and that timing mattered as much as insight. Here’s the road from a single artificial neuron to the systems now writing code and prose.

From the perceptron to the transformer
Click a milestone to read what happened. Notice the long gap between the idea (backprop, 1986) and its payoff (2012) — the missing ingredients were data and compute.
1960197019801990200020102020Perceptron1958XOR critique1969Backprop1986AlexNet2012Transformer2017LLMs go public2022
1958Perceptron
Rosenblatt builds a single trainable linear unit and the press predicts machines that will walk, talk, and reproduce. Real, but only a straight-line classifier.

The perceptron and the first winter

In 1958 Frank Rosenblatt built the perceptronperceptronRosenblatt's 1958 single trainable linear unit — the earliest artificial neuron. Its inability to solve XOR helped trigger the first AI winter.See in glossary →: a single trainable unit that takes a weighted sum of inputs and fires if it crosses a threshold, exactly the linear neuronneuron (unit)A single computing unit in a network: a weighted sum of its inputs plus a bias, passed through an activation. Loosely inspired by biological neurons.See in glossary → from the start of this explainer. It could learn its weightsweightOne adjustable number that scales an input inside a model — how strongly that input pushes the prediction up or down. The weights (plus biases) are the parameters training adjusts.See in glossary → from examples, and the excitement was enormous; newspapers imagined machines that would soon walk and talk.

Then, in 1969, Minsky and Papert published a careful proof of the perceptron’s limits: a single unit can only draw a straight boundary, so it cannot even learn the humble XOR function, the same wall you saw a linear model hit against a checkerboard. The critique was mathematically correct and devastatingly timed. It contributed to a loss of confidence, and the field eventually entered its first AI winterAI winterA period of collapsed funding and interest in AI after expectations went unmet — first triggered in part by the perceptron's exposed limits.See in glossary →, a stretch of deep skepticism and thin budgets.

Backprop, and a long quiet

The answer was already implicit: stack the units into layers, and a single straight cut becomes a flexible curved boundary. The missing piece was a practical way to train those inner layers. In 1986 Rumelhart, Hinton, and Williams popularized backpropagationbackpropagationThe algorithm that computes the loss gradient for every parameter efficiently by applying the chain rule backward through the network, reusing intermediate results from the forward pass.See in glossary → (the chain-rule bookkeeping you’ve now seen) which hands every weight in a deep network its gradientgradientThe vector of partial derivatives of the loss with respect to every parameter — it points in the direction of steepest loss increase, so we step the opposite way to reduce the loss.See in glossary → in one backward pass. Multi-layer networks were suddenly trainable.

And yet, for roughly two decades, they mostly underwhelmed. The theory was sound, but networks were small, data was scarce, and computers were slow. Other methods often worked better in practice, and neural networks drifted to the margins of the field.

The deep-learning era ignites

The dam broke in 2012. A deep convolutional network called AlexNet won the ImageNet image-recognition contest by a stunning margin, and it did so by combining three things that had all quietly matured: a big labeled dataset, GPUs fast enough to train a deep network in days instead of years, and architectural tricks that made depth actually trainable. Suddenly neural networks weren’t a curiosity. They were the state of the art, and the whole field pivoted.

Five years later, in 2017, the transformertransformerA neural-network architecture introduced in "Attention Is All You Need" (2017), built from stacked self-attention and feed-forward layers.See in glossary → arrived. Its attentionself-attentionAttention where the queries, keys, and values all come from the same sequence, so each position can gather information from other allowed positions in that sequence.See in glossary → mechanism let a network mix information across a sequence directly, and it scaled with data and compute extremely well. Point that architecture at enormous text datasets and you get the large language models that touched off the current wave.

Why now?

If backprop is from the 1980s, why did the revolution wait until the 2010s? Because a neural network needs three things at once, and for decades they never all showed up together:

  • Data: the internet produced text and images at a scale no one could have hand-labeled before.
  • Compute: GPUs, built for graphics, turned out to be near-perfect for the dense arithmetic of training, cutting run times by orders of magnitude.
  • Algorithms: better architectures, activation functions, and optimizers that let very deep networks actually converge.

One idea from this era deserves a chapter of its own, because it’s the bridge from everything you’ve learned to how a language model reads. Before a network can predict what comes next in text, it has to turn text into numbers it can reason about geometrically, and that’s the subject of the next chapter.