Prefill and decode are two different workloads on one GPU

TL;DR: Prefill can be compute-bound at large prompt or batch sizes. Decode emits one token per sequence per step and is often memory-bandwidth-bound at low batch sizes, because each pass reads the entire model to produce a single token. They compete on the same device, they scale on different signals, and measuring them together hides both.

The symptom this explains

A serving deployment is tuned for throughput and the throughput numbers are good. Users complain anyway, and they complain about two different things. Some say it takes ages before anything appears. Others say the text appears and then crawls, or stalls mid-sentence and resumes.

Those are not two reports of one problem. They are reports about two different workloads that happen to share a GPU, and a single average latency figure hides both of them. Separating the two is most of what this page is for.

Phase one: prefill

The model processes every token of the prompt in a single pass, in parallel, which makes this a large matrix multiplication that uses the tensor cores the way the hardware was designed to be used. Cost grows with prompt length. A 30,000-token prompt is not a slightly bigger version of a 300-token one; tokenwise work grows roughly a hundredfold while full-attention work has a quadratic component. Wall time also depends on kernels, caching and batch shape.

Prefill produces the KV cache entries for the prompt, plus the first output token. The user experiences it as the wait before anything appears.

Phase two: decode

Now the model generates one token at a time, and each step runs a full forward pass that reads all of the model's weights out of memory in order to produce that single token.

That is the whole reason decode behaves the way it does. At small batches, weight and KV reads can dominate the arithmetic. Larger batches amortize weight reads and can move the bottleneck toward compute; longer contexts also increase attention work and cache traffic.

Why batching helps decode so much more

If the weights must be read once per step regardless, then reading them on behalf of one sequence or of sixty costs nearly the same, and batching amortises that read across the whole batch.

The arithmetic makes the size of that effect concrete. Take an illustrative 7B model in FP16, so about 14 GB of weights, on an accelerator with roughly 2 TB/s of memory bandwidth. Reading every weight once takes at least 14 / 2,000 = 0.007 seconds, or 7ms, and that is a floor set by bandwidth alone before any arithmetic is performed. One decode step reads the weights once and produces one token per sequence in the batch.

So at a batch size of 1, those 7ms buy a single token: about 1 / 0.007 ≈ 143 tokens per second, with the device's compute units almost idle because there is very little arithmetic to do per byte read. At a batch size of 32, the same 7ms and the same single pass over the weights buy 32 tokens, which is roughly 32 / 0.007 ≈ 4,571 tokens per second.

Look at what did not change. Each sequence still receives one token every 7ms, so inter-token latency for an individual user is roughly the same at batch 32 as at batch 1, while aggregate throughput went up about thirty-twofold. That is the asymmetry, and it is why decode configurations look almost free to improve until they suddenly are not.

Three things bound that picture, and they are the reason the gain is not unlimited. Each sequence's KV cache has to be read every step as well, and that traffic grows with batch size and with context length, so the real step time rises as the batch grows rather than staying at 7ms. Beyond some batch size the arithmetic per pass catches up with the bandwidth and the phase becomes compute-bound, at which point more batch means more time. And a larger batch consumes more cache memory, which is the ceiling the KV cache describes. Per-sequence latency therefore rises modestly at first and then worsens, so the batch size has to be measured at the target throughput rather than derived from the floor above.

A prefill workload already saturating compute may gain less from larger batches, and adding work to a compute-bound phase adds time.

This asymmetry is why continuous batching exists: sequences join and leave the running batch as they arrive and finish, instead of waiting for a fixed batch to assemble and complete together.

The interference

Both phases compete for one device, and a scheduler running one batch at a time has to choose between them. Admit the long prefill and every decoding sequence pauses while it runs, so readers watching text appear see it stop. Defer it and that user waits.

Neither choice is good, which is why chunked prefill exists: split a long prefill into pieces and interleave them with decode steps, so the worst-case interruption becomes the chunk rather than the whole prompt.

What follows for measurement

Two metrics, not one:

• Time to first token covers queue time plus prefill. This is the wait, and it scales with input length.
• Inter-token latency covers the decode loop. This is the flow, and it degrades under batch pressure.

Total response time includes first-token delay plus subsequent token gaps. Use it when the product has a completion deadline, and stratify by output length so a workload change is not mistaken for an infrastructure regression.

What follows for cost

Input tokens are processed in parallel and are cheap per token. Output tokens each cost a full pass over the weights and are expensive per token. Providers price them differently for this reason, but the bill depends on actual input/output volume, rates and cache discounts. Compare the measured cost before choosing which side to optimize.

What follows for architecture

Because the phases have different bottlenecks, they can be scaled and even hosted separately: a prefill pool chosen for compute, a decode pool chosen for memory bandwidth and cache size, with the KV cache handed between them. Disaggregation of this kind pays off at large scale and adds a distributed-systems problem to a serving stack, which is the trade to weigh rather than a default to adopt.

Before that, the cheap version of the same idea is prefix caching. When many requests share a system prompt or a document, the prefill for the shared part is computed once and reused, which removes the expensive phase entirely for that portion.

Self-check

A chat product streams its responses. Requests average a 2,000-token prompt and a 400-token reply. Measured on the current configuration, prefill runs at about 10,000 tokens per second and each decode step takes 30ms. Work out the time to first token, the time to the last token, and which of the two a larger decode batch would damage. Then decide whether a configuration that doubles throughput while pushing inter-token latency from 30ms to 70ms is an improvement. Work it through before reading on.

Time to first token is the prefill, plus any queue wait: 2,000 / 10,000 = 0.2 seconds, so about 200ms before anything appears, assuming the request was not queued. The reply then takes 400 × 0.030 = 12 seconds of decode, so the last token arrives about 12.2 seconds after the request. Notice the proportions, because they decide where optimisation effort belongs: prefill is under 2 percent of the total here, and the decode loop is essentially the whole response time.

The batch change damages inter-token latency, which is the flow rather than the wait. At 70ms per token the same reply takes 400 × 0.070 = 28 seconds instead of 12, so the user watches text arrive for nearly half a minute.

Whether that is an improvement depends entirely on whether anyone is reading it as it arrives, and for a streaming chat product the answer is no. Comfortable reading is somewhere around 150 to 250 words per minute, and at 70ms per token the stream delivers roughly 60 / 0.070 ≈ 857 tokens per minute, which is on the order of 600 words per minute and still ahead of a reader. So the stream is not actually too slow to read; what breaks is the total, because a 28-second answer feels stalled even while tokens keep coming. The judgement is a product one, and the reason to have both numbers is that they disagree: throughput doubled and the experience got worse.

The same trade reads differently if the output is consumed whole rather than streamed, such as a batch summarisation job, where nobody is waiting on the first token and doubling throughput halves the hardware bill. Which is the general rule worth keeping: decode batch size is not a performance setting to maximise, it is where you choose between throughput and an individual user's flow, and you cannot make that choice from an average latency figure that mixes the two.

Next: the KV cache sets the memory ceiling on batch size, and tokens are the unit of everything prices the same token counts.

Sources: vLLM chunked prefill tradeoffs.