The KV cache is what decides how many users you can serve
Model weights, runtime memory and the KV cache compete for GPU memory. Estimate the cache budget and memory per sequence, then check whether that concurrency also meets your latency target.
TL;DR: Reserve memory for weights, activations, runtime workspaces and allocator overhead before sizing the KV cache. For a full-attention model, cache bytes grow with stored tokens. Cache budget divided by bytes per sequence gives a memory ceiling; configured admission limits and latency can impose a lower one. A workload whose prompts get longer serves fewer users on the same hardware, with no code change and no deploy.
The symptom this explains
An inference service that comfortably served 30 concurrent users last month now starts queueing at 15. The model is the same, the hardware is the same, the deploy history is empty, and GPU utilisation reads 95 percent as it always did.
Nothing regressed. The users' prompts got longer, because a prompt template grew or a retrieval step started returning more documents, and the memory that decides how many sequences fit at once is consumed per token. Capacity fell without anything in the change log moving, which is the failure mode this page exists to make predictable.
Two very different consumers of the same memory
Two major consumers of GPU memory are weights and the KV cache; activations and runtime buffers also need space. Weights are fixed: a 7B model in FP16 is about 14 GB and that number never moves. The KV cache is variable, and it holds, for every token of every active sequence, the key and value tensors that attention computed for it.
Weights answer "does this model fit". The cache answers "how many people can use it at once". The cache changes with the active workload, so a traffic-mix change can cause memory pressure even when the model weights are unchanged.
Why the cache exists at all
Generating token 500 requires attending over the previous 499. Caching avoids recomputing previous keys and values. Each new query still attends over stored tokens: for full attention, that part of decode work grows with context length. The cost per generated token is not constant.
The cost of that speedup is memory that grows linearly with every token, for every sequence in flight, for a full-attention cache. Sliding-window attention, eviction and offloading change the retained-memory behavior.
The arithmetic
Per token, per sequence, the cache holds a key and a value tensor for every layer:
bytes per token = 2 (K and V) × layers × kv_heads × head_dim × bytes_per_element
For an illustrative model with 32 layers, 32 KV heads, head dimension 128 and FP16 cache elements, this is 524,288 bytes, or 0.5 MiB per token. At 4,096 tokens, one sequence needs 2 GiB. A measured 60 GiB cache pool therefore fits at most 30 such sequences before allocation rounding and safety margin. This is not a universal 7B-model estimate: using eight KV heads instead of 32 cuts this cache calculation by four.
The pool size must come from the serving configuration after other memory reservations. Validate the calculation against the actual model and engine. Then examine batching and queue admission to decide how waiting requests enter that finite pool without breaking the latency contract.
Read the two rows together. The top row is why "does the model fit" is the easy question: weights are a fixed 14 GiB and either they fit or they do not. The bottom row is the question that decides your capacity, and every quantity in it moves with what your users send.
The consequences that surprise people
Memory bounds concurrency alongside configured admission and latency. A high sequence limit does not guarantee useful throughput: an engine may queue or preempt work under KV pressure. Check its actual admission behavior rather than assuming a high setting necessarily causes a process failure.
Longer prompts reduce capacity with no deploy. If a change doubles stored tokens per sequence, the simple full-attention estimate doubles cache bytes and halves the memory ceiling. Nothing in your change log explains the incident.
Utilisation lies here. The GPU can show high utilisation while the true constraint is cache memory. Compute is busy; admission is blocked. This is why the metric to watch is cache utilisation together with the count of requests waiting.
Eviction is a product decision. When the cache fills, an engine either queues new requests or preempts a running sequence and recomputes it later. The first adds latency at the front, the second wastes work already done. Which is right depends on whether your users are waiting on a stream.
What the design levers actually do
- Paged attention stores the cache in fixed-size blocks rather than one contiguous reservation per sequence, so memory is allocated as a sequence grows instead of at its theoretical maximum. This reduces waste from over-reservation and fragmentation, though partially filled blocks still consume memory.
- Grouped-query attention shares key and value heads across query heads, cutting cache size by a large factor. It is a model architecture choice, so it is decided before you deploy.
- Quantising the cache can reduce element storage, for example from 16 to eight bits. Measure quality and latency, and account for scale metadata and engine support.
- Prefix caching shares the cached prefix of a common system prompt or document across requests, which can avoid duplicated cache blocks and repeated prefill work for a matching resident prefix. It does not remove decode work over that prefix.
- Shorter contexts, which is unglamorous and frequently the largest available win.
The one number worth carrying
Cache-pool bytes divided by cache bytes per sequence is a useful memory ceiling for the stated architecture. Test the resulting concurrency against latency, throughput and the prompt/output-length distribution before treating it as serving capacity.
Self-check
Start from the illustrative model above: 32 layers, 32 KV heads, head dimension 128, FP16 cache, a 60 GiB cache pool, and sequences of 4,096 tokens. Work out three things. What happens to the ceiling if the model instead uses 8 KV heads? What happens if prompts grow so sequences average 8,192 tokens? And if you are serving 30 concurrent sequences and the engine starts preempting, what does the fix depend on? Work all three before reading on.
The KV head change is the largest lever available and it is not yours to pull. Cache bytes per token scale directly with kv_heads, so going from 32 heads to 8 divides the cache by four: bytes per token fall from 0.5 MiB to 0.5 / 4 = 0.125 MiB, a 4,096-token sequence needs 4,096 × 0.125 = 512 MiB or 0.5 GiB instead of 2 GiB, and the same 60 GiB pool has a theoretical ceiling of 60 / 0.5 = 120 sequences instead of 30. Four times the concurrency on identical hardware, decided by an architecture choice made before you deployed, which is why grouped-query attention is worth checking for when you select a model rather than treating serving capacity as a tuning problem.
The prompt length change runs the other way and needs no decision at all. Doubling stored tokens per sequence doubles cache bytes per sequence, so 4 GiB each and 60 / 4 = 15 sequences. Your capacity halved and nothing in your change log records it, which is the opening symptom of this page.
The preemption question is the one that matters operationally, and the honest answer is that the fix depends on what your users are doing. When the pool fills, an engine either queues arrivals at the front or preempts a running sequence and recomputes it later. If users are watching a stream, preempting a sequence mid-generation is the worse outcome, because it stalls a response already in progress and throws away decode work that was already paid for; queueing at the front is more honest, since a request that has not started can wait or be rejected. If the work is batch, preemption is usually fine and queueing is what hurts. So the lever is the engine's admission and preemption policy rather than any memory figure, and choosing it is a product decision about who waits.
One limit applies to all three answers. Every number here is a memory ceiling, not a throughput figure: 120 sequences fitting in memory does not establish that 120 concurrent requests meet a first-token or inter-token latency target, because they still contend for compute. The memory calculation tells you the hard cap, and only measurement under a representative prompt-length distribution tells you the useful one.
Next: tokens are the unit of everything shows the same token counts driving cost and latency, and why GPUs break schedulers covers why the scheduler above this cannot see any of it.
References
An OOM can also involve temporary tensors or allocation layout. GPU memory fragmentation separates live demand, cached blocks and memory outside framework accounting.