TL;DR: First distinguish CUDA device-memory exhaustion from a host-memory OOM kill. Longer contexts can exhaust the engine's KV-cache blocks at the same request rate; vLLM normally preempts and recomputes requests to recover those blocks, increasing latency. Check that path before tuning admission limits or adding replicas.
How to approach it
Identify which memory domain failed from the previous container logs and termination reason. Then ask what changed in prompt lengths, concurrent sequences, engine configuration and memory usage. The title gives an OOM symptom, not proof that KV cache caused it.
A strong answer
Inference engines like vLLM reserve GPU memory for the key-value caches of every sequence in flight: each token's attention state lives there until generation completes. Throughput comes from batching many sequences into that cache simultaneously. The failure mode is arithmetic: a prompt burst with 30k-token contexts consumes an order of magnitude more cache per request than the 2k-token traffic the pod was sized against. When available KV-cache blocks run out, vLLM can preempt requests and recompute their state later. Repeated preemption increases queueing and latency; it does not itself mean the process was killed. A CUDA out-of-memory exception instead points to a device allocation that could not fit, potentially involving activations, graph capture or another GPU process. Kubernetes OOMKilled concerns host RAM or its container limit, which is a separate investigation. Check the previous logs, termination reason, host memory and engine preemption counters together. vLLM explains its cache-preemption behavior.
Why "yesterday was fine" is expected rather than mysterious: request-rate alone does not determine memory pressure, request-shape does. A thousand short chats and fifty long document summaries can cost the same cache. Any autoscaler watching CPU or even SM utilisation is blind here, because inference is memory-bound and utilisation graphs stay deceptively busy-looking right up to allocation failure.
The fixes, in order of leverage:
Admission control. Set max_model_len and max_num_seqs from tested request shapes. Bound the gateway queue and reject excess demand with a clean 429; do not assume the engine has a bounded queue by default. Tune the KV-memory budget only after checking the other GPU allocations that must fit.
Scale on the signal that matters. Waiting-queue depth or time-to-first-token percentiles are the honest capacity signals; both are exposed by inference servers as metrics. Autoscale when waiters exceed a threshold sustained over minutes.
Graceful shutdown. On scale-in or deploy, stop admitting, finish generating in-flight sequences within the termination grace period, then exit. SIGKILL mid-generation abandons paying users mid-answer and multiplies support load.
Shape the traffic upstream. Prompt-length limits at the API edge, per-tier concurrency caps, and long-context workloads routed to dedicated pools sized for them.
| Signal | Tells you | Use it for |
|---|---|---|
| Queue depth / waiting requests | Demand exceeding current capacity | Scale trigger |
| TTFT / TPOT percentiles | User-visible health | SLOs and alerts |
| KV-cache utilisation | How close to the cliff | Capacity dashboards |
| GPU SM utilisation | Almost nothing here | Ignore for scaling |
What interviewers probe next
"Why not just raise the memory limit?" Raising the Kubernetes memory limit can help a host-RAM OOM if the node has room, but adds no GPU VRAM. For device-memory pressure, inspect the engine memory budget and request shapes before selecting a larger GPU or lowering concurrency.
"How would you have caught this before production?" Load-test with production-shaped request distributions, including tail context lengths, not synthetic uniform prompts. The gap between test traffic and real traffic is precisely where this incident lives.
"Does batching make latency worse?" More sequences per batch raise throughput but time-to-first-token climbs for everyone in the batch. State the trade-off and where your SLO pins it.
Common mistakes
Treating it as a memory leak because memory climbed until death. Cache growth under admitted load is by design; only unbounded growth beyond policy is a leak.
Scaling on CPU. Inference workers are not web servers; CPU tells you nothing about the constraint.
Restarting pods as remediation. It drops every in-flight generation and reproduces within minutes under the same request mix.