TL;DR: Web services also need workload-aware signals, readiness and draining. LLM inference adds token-shape and GPU-memory constraints: memory pressure scales with token volume and sequence shapes, GPUs cost too much for reactive scaling, and draining a replica means finishing live generations. Treat queue depth as a reactive signal, combine it with demand forecasts, keep a warm buffer, and treat drain time as a first-class constraint.
How to approach it
Contrast the two worlds property by property rather than listing tools: load unit, signal, reaction time, scale-in semantics, and unit economics. Each property fails for its own reason, and naming the reason is what distinguishes this from a list of things that are different about GPUs.
A strong answer
The load unit is wrong. Web request costs also vary: a cached homepage and an expensive search can consume very different resources. An inference request's cost scales with tokens processed: input context and output length vary by orders of magnitude across requests. Two hours of identical request rates can differ tenfold in actual resource consumption, so request-count-based targets (CPU, RPS per pod) measure something adjacent to, but not, capacity.
The real capacity unit is KV-cache memory, which fills with concurrent sequence state. That makes concurrency-with-shape the quantity to bound, and it makes utilisation signals misleading: a GPU at 90 percent SM utilisation may be healthy or one batch from failure.
Reaction economics invert. A lightweight web pod may boot quickly enough for reactive scaling, while other web workloads also require warm capacity. A GPU node takes minutes to allocate and initialise (scheduling, image pull of multi-gigabyte weights unless cached locally, engine warm-up) and costs on the order of dollars per hour. By the time reactive scaling fires, users have waited in queue through the entire ramp. Practical designs therefore hold a warm buffer sized from forecast peaks, use provisioned/reserved concurrency equivalents where the platform offers them, and treat predictive scaling as normal rather than exotic.
Worked example: sizing the warm buffer from the cold start
"Hold a warm buffer" is only actionable once it has a size, and the size comes from the ramp rate and the cold start rather than from a percentage. Illustrative figures.
A service runs at 100 requests per minute overnight and peaks at 400, climbing over about ten minutes, so demand rises at roughly (400 − 100) / 10 = 30 requests per minute, per minute. One replica serves 50 requests per minute, so the fleet moves from 100 / 50 = 2 replicas to 400 / 50 = 8 across that ramp.
A GPU replica takes about 5 minutes to become useful: scheduling, pulling multi-gigabyte weights unless they are cached on the node, and engine warm-up. So any replica you ask for now serves traffic that arrived 5 minutes ago. In those 5 minutes demand grows by 30 × 5 = 150 requests per minute, which is 150 / 50 = 3 replicas' worth of work with nowhere to go.
That is the buffer: at least 3 warm replicas beyond current demand, for this ramp. Not a percentage, and not a number anyone can reuse, because it is the product of your ramp rate and your cold start. Halve the cold start and the buffer halves; double the ramp and it doubles.
Then price it, because this is the trade the design is actually making. Three idle replicas at an illustrative $3 per GPU-hour is 3 × 3 = $9 an hour, or 9 × 24 = $216 a day, to hold capacity that serves nothing most of the time. The alternative is not free either: without the buffer, every morning ramp puts users in a queue for the length of a cold start, so the choice is $216 a day against five minutes of degraded service on a predictable schedule. Stated that way it is a business decision rather than a tuning question, which is the point.
What the arithmetic does not settle is whether the ramp is predictable. It assumes a shape learned from history, and it is exactly wrong for an unforecast spike, which is the case where queue depth as a reactive signal and a shed-or-queue policy have to carry the service instead.
Scale-in kills work in progress. Web services also need graceful drain. Retrying non-idempotent requests can duplicate effects, and even safe retries add latency. Removing an inference replica abandons generations mid-stream: users see truncated answers, and if they are paying per token, money is involved. Drain must mean "stop admitting, complete in-flight sequences", which can take minutes for long generations, so termination grace periods and scale-in cooldowns are correctness requirements, not tuning knobs.
| Property | Web service | LLM inference |
|---|---|---|
| Load unit | Workload-specific demand | Tokens plus sequence shape |
| Scaling signal | CPU/RPS | Queue depth, TTFT p99 |
| Cold start | Seconds | Minutes (weights, warm-up) |
| Scale-out posture | Reactive works | Proactive with warm buffer |
| Scale-in | Drain; retry only when safe | Drain live generations; handle timeout |
What interviewers probe next
"Where do batching and autoscaling interact?" Higher concurrency raises throughput until TTFT SLOs break; autoscaling on latency percentiles naturally holds batches inside the quality boundary while adding replicas when demand persists.
"Could you just overprovision and forget it?" At single-model scale sometimes yes, and honesty about that beats complexity worship. Multi-model serving changes the answer: weight memory dominates, so models get loaded/unloaded and the scheduling problem returns.
"What would you monitor first?" Queue depth and TTFT/TPOT percentiles against SLO, then cache utilisation for headroom planning. GPU temperature and errors matter operationally but say nothing about demand.
Common mistakes
Copying an HPA config from the API tier onto inference pods. The metrics are wrong, the timings are wrong, and scale-in behaviour actively harms users.
Treating model loading as free. Cold replicas serve nothing for minutes after scheduling; warm-buffer math must include it.
Ignoring request shape in capacity reviews. Peak context length, not peak RPS, decides when the fleet falls over.