Embeddings

Meaning as geometry

Back near the start we turned categories into numbers with one-hot encoding: each item becomes a vector that’s all zeros except a single 1 in its own slot. It works, but it hides a limitation. In one-hot space every item is exactly the same distance from every other: “cat” is no nearer to “dog” than to “Tuesday.” Any similarity has to be learned later from the data. Embeddings fix this, and in doing so they become the bridge from this explainer to a language model.

From sparse slots to dense vectors

An embeddingembeddingA dense vector representation of a token (typically d=2k–8k floats). Similar tokens get nearby vectors.See in glossary → replaces each item’s lonely one-hot slot with a short, dense vector of real numbers, say 4, or 300, or 12,000 of them. Crucially, those numbers are not chosen by hand: they’re parametersparametersThe numbers (weights) inside a model that get adjusted during training. A “7B model” has 7 billion of them.See in glossary →, learned by the very same gradient descentgradient descentThe core training algorithm: repeatedly nudge each parameter a small step in the direction that lowers the loss, as told by the gradient.See in glossary → that trains everything else. You stack all the vectors into an embedding matrixembedding matrixA table with one row per vocabulary entry. Looking up a token = indexing into this matrix.See in glossary → (one row per item) and treat it as just another slab of knobs the training loop is free to turn.

As the model trains on a task, it often discovers that giving similar items similar vectors lowers the loss. Words that behave alike tend to drift together; words that don’t tend to drift apart. Meaning is no longer just an arbitrary label: some of it is represented as a position in space.

A learned space can encode similarity and relationships

The map below is a tiny two-dimensional embedding. Click any word to see its nearest neighbors, and toggle the analogy arrows.

Two things jump out. First, distance can encode a useful kind of similarity: the animals huddle together, the numbers form their own island, and “cat” is close to “dog” and far from “seven.” That’s a property one-hot encoding couldn’t express on its own. Second, directions can sometimes carry relationships: the arrow from king to queen runs nearly parallel to the arrow from man to woman. In this toy map, both trace a similar relationship through the space. The result depends on what the model learned and on the rule we use to measure closeness; it is a useful pattern, not a guarantee about meaning.

Real embeddings live in hundreds or thousands of dimensions, not two, so a single word can be close to other words for many reasons at once: similar meaning, topic, grammatical role, and more. Squeezing that space into a 2-D drawing can distort which points look close. The core idea survives: positions and directions in the learned space can carry useful information, but what “close” means depends on the model, layer, task, and drawing method.

The bridge to language models

Here’s why this is the last idea you need. A language model’s first move, when it reads text, is to look up each tokentokenThe atomic unit of text the model sees. Roughly a word-fragment — “tokenization” is a piece of text → list of token IDs.See in glossary → in exactly such an embedding matrix, turning a discrete symbol into a dense vector the network can compute with. Those input vectors start as random parameters and are shaped, over trillions of tokens, by the same gradient descent you’ve watched all along, until the geometry of the space quietly encodes an enormous amount about how language works.

With tokens turned into vectors, and a network trained to predict the next one, we’ve assembled every piece of a language model. The final section snaps them together.