Sampling

Logits → the next token

Source: The Curious Case of Neural Text Degeneration — Holtzman et al., 2020

We have logits (one vector of 128,256 numbers) sitting at the last position of the model’s output. We need to turn this into a single chosen token to append to the sequence and feed back in. That choice is called samplingsamplingChoosing the next token from logits — greedy (argmax), temperature scaling, top-k, top-p, etc.See in glossary →, and the choice of how to sample affects how varied, predictable, or repetitive its output is.

Greedy decoding: just pick the max

The simplest possible strategy: take the argmax. Whichever token has the highest logit is the chosen token.

next_token = logits.argmax()

Greedy decoding is fully deterministic: same prompt always yields the same completion. It’s also great when there is one obviously correct answer (factual lookup, code completion of a syntactically constrained snippet). It’s bad when there are many reasonable continuations, because it always picks the model’s most confident guess, which can lead to repetitive, bland, or stuck-in-a-loop text.

Temperature scaling: sharpen or flatten the distribution

Before applying softmax, divide logits by a number $T$ called the temperaturetemperatureDivides logits before softmax. <1 sharpens (more deterministic), >1 flattens (more random). 0 = greedy.See in glossary →:

• $T = 1$: leaves the distribution unchanged.
• $T < 1$: sharpens (the top tokens get even more probability mass, the long tail gets squashed). $T \to 0$ recovers greedy decoding.
• $T > 1$: flattens (the model becomes more random and willing to pick unusual tokens).

Then we sample from this distribution: roll a weighted die.

probs = softmax(logits / T)
next_token = multinomial(probs)  # sample one

Temperature alone is the simplest random strategy. The problem: even at $T = 1$, the long tail of the vocabulary still has some non-zero probability. Every token has some chance of being chosen, including weird and clearly wrong ones. To clip the tail we add top-k or top-p.

Top-k: only consider the top k tokens

Keep only the $k$ highest-logit tokens, zero out the rest, re-normalize, sample.

top_k_logits, top_k_indices = topk(logits, k)
probs = softmax(top_k_logits / T)
next_token = top_k_indices[multinomial(probs)]

Common values: $k = 40$ or $k = 50$. Cheap, easy, and works well when the model is fairly confident. But $k$ is a fixed shape, while the right “cutoff” actually varies by context. Sometimes only 3 tokens are reasonable; sometimes 500 are. A fixed $k$ either truncates good candidates or admits bad ones.

Top-p (nucleus sampling): use the smallest set whose mass ≥ p

Top-p picks the cutoff dynamically. Sort tokens by probability, accumulate until you’ve covered fraction $p$ of the mass, and sample only from that “nucleus.”

sorted_probs, sorted_idx = sort(softmax(logits / T), descending=True)
cumulative = cumsum(sorted_probs)
keep = cumulative - sorted_probs < p  # include the token that crosses p
probs = renormalize(sorted_probs * keep)
next_token = sorted_idx[multinomial(probs)]

Common values: $p = 0.9$ or $p = 0.95$. This adapts: if the model is very confident, the nucleus is tiny; if many tokens are plausible, the nucleus widens. Top-p is a widely used sampling strategy for open-ended generation.

Combining them

In practice, production samplers often apply temperature and then top-k and/or top-p. The exact order and defaults are library-specific. You can also stack in:

• Repetition penalty / frequency penalty / presence penalty: alter the logits of tokens that have already appeared, discouraging loops. Frequency and presence penalties are commonly additive; repetition penalties use different rules in different implementations.
• Min-p: keep only tokens whose probability is at least $p_{\min}$ times the top probability, a newer alternative to top-p.
• Logit bias: directly add a value to specific token IDs (e.g. to forbid a token, set its logit to $-\infty$).
• Guided / constrained decoding: at every step, mask out any token that would violate a grammar (JSON schema, regex, function call format). vLLM supports this through configurable structured-output backends.

The whole inference loop

Now we can write the entire generation loop:

tokens = tokenize(prompt)
while True:
    logits = model.forward(tokens)
    next_token = sample(logits[-1])           # logits at the last position only
    if next_token == END_OF_TEXT:
        break
    tokens.append(next_token)
print(detokenize(tokens))

This captures the logic of generation, but it repeats far more work than necessary, because every iteration of that while loop runs the entire model — billions of parameters of work — to produce one token, and re-does all the work for every previous token along the way.

Reusing earlier calculations makes the loop much faster.

An inference engine separates prefillprefillThe first forward pass that processes the entire prompt at once. Compute-bound, parallel over prompt tokens.See in glossary →, processing the initial prompt, from decodedecodeThe autoregressive phase: one forward pass per generated token. Memory-bandwidth-bound — the GPU mostly waits on weights.See in glossary →, generating the response one token at a time. The KV cache retains results from earlier positions so decode can reuse them. That saves repeated calculation, but each active request now needs memory for its cache. Serving many users means managing both the computation and that growing memory demand.