The objective

Next-token prediction and cross-entropy loss

Everything a language model learns in pre-training is squeezed through one narrow objective: predict the next token. A useful scoring rule must measure the error in a prediction and give training a way to reduce it.

The model is a next-token probability machine

A language modellanguage modelA model that assigns probabilities to sequences of tokens — in practice, one that predicts the probability distribution of the next token given the preceding ones.See in glossary → takes a sequence of tokenstokenThe atomic unit of text the model sees. Roughly a word-fragment — “tokenization” is a piece of text → list of token IDs.See in glossary → and outputs a probability distributionprobability distributionAn assignment of a non-negative probability to every possible outcome that sums to one. A language model produces one over the vocabulary at each step — its bet on the next token.See in glossary →, a probability for each possible next token. Concretely, given a context $x_1, x_2, \ldots, x_{t-1}$, it produces

a number for every entry in the vocabularyvocabularyThe fixed set of tokens a model knows about. Modern LLMs have ~32k–200k entries.See in glossary →, the model’s fixed list of tokens (often 100k–256k entries), all non-negative and summing to 1. The subscript $\theta$ (the Greek letter theta) is the model’s parameters, the thing training adjusts. Internally the network emits one raw score (a logitlogitsThe raw, pre-softmax scores the model produces — one per vocabulary token, per position. Bigger logit = the model finds that token more likely; the actual value can be any real number, positive or negative. Applying softmax across the vocabulary turns logits into a probability distribution that sums to 1. Sampling then picks one token from that distribution.See in glossary →) per vocabulary entry, and softmaxsoftmaxFunction that turns any vector into a probability distribution (positive, sums to 1) by exponentiating and normalizing.See in glossary → turns those logits into probabilities by raising the constant $e$ (approximately 2.718) to the power of each score, then dividing by the sum of the results. Raising a number to a power is called exponentiationexponentiationRaising a number to a power, such as squaring it. For a positive whole-number power, this means multiplying that many copies of the number.See in glossary →.

Because each token is predicted only from the tokens before it, this is a causal language modelcausal language modelA model that predicts each token using only earlier tokens (never future ones). "Causal" because information flows strictly left to right. The GPT family are causal LMs (Language Models).See in glossary →: information flows strictly left to right. The probability of a whole document factorizes into a product of next-token probabilities:

This is the autoregressiveautoregressiveGenerating one token at a time, where each new token is conditioned on every token that came before it.See in glossary → factorization. It is exact (just the chain rule of probability) and it is what makes next-token predictionnext-token predictionThe pre-training objective for GPT-style models: given the tokens so far, predict a probability distribution over the next token. Also called causal or autoregressive language modeling.See in glossary → a complete model of text rather than a heuristic.

Scoring a prediction: cross-entropy

We need a loss functionloss functionA single number measuring how wrong the model's predictions are on a batch of data. Training works by adjusting the model to make this number smaller.See in glossary → that is small when the model put high probability on the token that actually came next, and large when it did not. The natural choice is the cross-entropy losscross-entropy lossA classification loss that penalizes the model according to the negative log-probability it assigned to the correct answer.See in glossary →, equivalently the negative log-likelihoodnegative log-likelihoodAnother name for the cross-entropy LM (Language Model) loss: −log of the probability the model gave to the correct token. Big when the model was confidently wrong, small when it was confidently right.See in glossary → of the correct token:

Reading it term by term:

• $\mathcal{L}$, the loss: the single number we want to make as small as possible. Lower means the model predicted the real text better.
• $x_t$: the actual token at position $t$ (the ground truth sitting in the training document), out of the whole sequence $x_1, x_2, \ldots, x_T$.
• $x_{<t}$: shorthand for all the tokens before position $t$, i.e. $x_1, \ldots, x_{t-1}$. This is the context the model is allowed to look at.
• $p_\theta(x_t \mid x_{<t})$: the probability the model assigned to the true token $x_t$, given that context. The bar $\mid$ reads “given.” This is a single number between 0 and 1, the one value we care about out of the model’s full distribution over the vocabulary.
• $\theta$: the model’s parameters (its billions of weights). The subscript is a reminder that this probability depends on the current model; training changes $\theta$ to make the true tokens more probable.
• $\log$: the natural logarithmlogarithmThe exponent needed to obtain a number from a chosen base. The natural logarithm uses the base e, approximately 2.718; logarithms turn products into sums.See in glossary →, the power to which $e$ must be raised to obtain its input. Applied to a probability in $(0,1]$ it gives a number in $(-\infty, 0]$: $\log 1 = 0$ (perfect), and it dives toward $-\infty$ as the probability approaches $0$.
• $\sum_{t=1}^{T}$: sum over every position $t$ from $1$ to $T$, so every token in the sequence contributes one term.
• $T$: the number of tokens in the sequence (its length).
• $\tfrac{1}{T}$: divide by $T$ to turn that sum into an average per token, so the loss doesn’t simply grow with sequence length and is comparable across sequences.
• the leading $-$ (minus sign): flips the sign. Since each $\log p$ is negative, negating makes $\mathcal{L}$ a positive quantity where smaller is better, a penalty rather than a reward.

Put together: for each position we look up the probability the model assigned to the true next token, take its logarithm, negate it, and average over all positions. Maximizing the likelihood of the data is the same as minimizing this loss.

Why the logarithm? Because it makes the penalty grow without bound as the model’s probability for the truth approaches zero. A model that says “1% chance” for the token that actually appears pays $-\log(0.01) \approx 4.6$; one that says “0.001%” pays $\approx 11.5$. Cross-entropycross-entropy lossA classification loss that penalizes the model according to the negative log-probability it assigned to the correct answer.See in glossary → punishes confident mistakes strongly: assigning a tiny probability to the correct token costs much more than allowing for uncertainty.

Perplexity and bits-per-token: the same loss in friendlier units

The raw loss is in nats (natural-log units), which is hard to feel. Two alternative ways of expressing it are easier to interpret.

• PerplexityperplexityThe exponential of the cross-entropy loss — roughly "how many equally-likely tokens is the model choosing between?" Lower is better; a perplexity of 1 means perfect prediction.See in glossary → is $e^{\mathcal{L}}$. You can read it as “the model is as confused as if it were choosing uniformly among this many tokens.” A perplexity of 1 is perfect. Compare perplexity values only when the evaluation data and tokenization are the same; different vocabularies change the numeric scale. On a controlled comparison, lower perplexity means better next-token prediction.
• Bits-per-tokenbits-per-tokenCross-entropy loss measured in bits (log base 2) instead of nats. A compression-flavored view: a better language model encodes the next token in fewer bits.See in glossary → is $\mathcal{L}/\ln 2$, the loss in base-2. This connects pre-training to compression: a better language model assigns shorter codes to real text. Training to minimize bits-per-token is, quite literally, learning to compress the corpus, and good compression demands understanding.

Teacher forcing: a trillion examples at once

There is one more efficiency trick baked into the objective. During training we don’t generate text and then grade it; we feed the model the true sequence and ask it, at every position simultaneously, to predict the next token. This is teacher forcingteacher forcingDuring training, feeding the model the true previous tokens (not its own guesses) at every position, so all next-token predictions in a sequence can be learned in parallel.See in glossary →. Thanks to the causalcausal language modelA model that predicts each token using only earlier tokens (never future ones). "Causal" because information flows strictly left to right. The GPT family are causal LMs (Language Models).See in glossary → mask (each position can only see earlier ones), a single forward passforward passRunning inputs through the network to produce outputs (logits) and the loss, caching intermediate activations that backpropagation will need.See in glossary →, a calculation from model inputs to outputs, over a length-$T$ sequence produces $T$ next-token predictions and therefore $T$ loss terms, all in parallel.

This is why a document is worth its length in training signal, and why the context lengthcontext lengthThe maximum number of tokens the model can attend to at once (also called the context window or sequence length). Pre-training picks a context length; later stages often extend it.See in glossary → we train at (say 4k or 8k tokens) directly sets how much supervision each sequence yields. Context length also determines which relationships within a document the model has a chance to learn.

Minimizing this loss requires adjusting billions of parameters. Gradient descentgradient descentThe core training algorithm: repeatedly nudge each parameter a small step in the direction that lowers the loss, as told by the gradient.See in glossary → uses the loss’s sensitivity to each parameter to choose an update; backpropagationbackpropagationThe algorithm that computes the loss gradient for every parameter efficiently by applying the chain rule backward through the network, reusing intermediate results from the forward pass.See in glossary → calculates those sensitivities efficiently.