Section 17

Chunked prefill

Stop blocking decoders with one big prompt

Sources: Chunked Prefill — vLLM documentation; DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving — Zhong et al., 2024

Even with batching and efficient cache management, a server can face a single huge prompt blocking everything else. Chunked prefillchunked prefillSplitting a long prompt into multiple smaller prefills so decoding requests aren’t blocked behind one giant compute step.See in glossary → is the fix.

The problem

Suppose 32 decode requests are running, each happily generating one token per step at, say, 50 ms per step. Now a new request arrives with a 64,000-token prompt. Its prefill needs to process all 64k tokens through the model. The attention-score computation scales quadratically with sequence length because every token attends to every earlier token, while the projections and MLP scale linearly. Depending on the model and hardware, such a prefill can take hundreds of milliseconds or more.

If you naively schedule the prefill as a single forward pass, every one of the 32 decoders is frozen for those 500 ms. Their inter-token latency spikes from 50 ms to 550 ms. Users start watching the text catch up in a sudden burst. SLOs blow up.

Worse, you can’t just lower the priority of the prefill: it has to happen before the new request can generate even its first token, and there is no way to “decode without prefilling first.”

The fix: cut the prefill into bites

Chunked prefill splits a long prompt across multiple forward passes. Instead of one 64k-token prefill, you do, say, 8 chunks of 8k tokens, interleaved with decode steps from the other 32 requests. Each step’s forward pass is:

[ 8k prefill tokens from request A ] + [ 1 decode token each from requests 1..32 ]

All 8,000 + 32 = 8,032 tokens flow through one packed batch. The attention kernel handles the variable-length structure. Per-step latency rises, but no one is frozen behind the entire prefill; everyone can make progress at each scheduled step.

After 8 such steps, request A’s prefill is done; it then joins the decoders, generating one token per step like everyone else.

Mixing prefill and decode kernels

The implementation challenge is that prefill tokens and decode tokens, in the same batch, have very different shapes for attention:

  • A prefill token at position pp attends to positions 0..pp within its own request (causal-masked).
  • A decode token attends to all of its request’s cached positions.

Serving engines pass metadata such as sequence lengths, positions, and KV-block tables so that their attention kernels can handle these different structures in a packed batch. The exact kernel layout is implementation- and version-dependent.

Balancing throughput and latency

Three settings control how chunked prefill divides GPU time between new and existing requests:

  1. Chunk size: smaller = better decoder latency, lower per-chunk compute efficiency.
  2. Max number of decode requests per step: bigger = more throughput, more per-step latency.
  3. Admission policy: do we accept a new long prompt now, or queue it until the decoder population is smaller?

A serving system can tune these settings against observed SLOs, but the choice involves policy as well as measurement. Operators typically benchmark representative workload mixes and configure vLLM’s scheduling limits accordingly.

Separating prefill and decode across GPUs

For very large deployments, the asymmetry between prefill (compute-bound, big matmuls) and decode (memory-bound, tiny matmuls) is so stark that some teams run them on different GPUs entirely:

  • Prefill GPUs: provisioned for prompt-processing throughput and the prompt/KV footprint they must hold. They receive prompts, do prefill, and emit KV cache.
  • Decode GPUs: provisioned for sustained decode throughput and KV-cache capacity. They receive KV cache and do decode.

The KV cache has to be transferred from the prefill GPU to the decode GPU (over NVLink or a network path) when prefill finishes. That transfer adds latency and consumes bandwidth, so whether disaggregation improves cost or throughput depends on the workload and deployment. vLLM supports disaggregated prefill/decode through KV-transfer interfaces.

Batching and scheduling determine which requests share each step. Speculative decodingspeculative decodingA small draft model proposes K tokens; the big target model verifies them all in one pass. Net effect: more tokens per target-model step.See in glossary → changes how much a step can produce: a fast draft proposes several tokens, and the main model verifies them together.