Everything is a queue, and the queue is where the latency lives

TL;DR: In an M/M/1 model with unchanged mean service time, 80 percent utilisation gives four times the mean queue wait of 50 percent. Real services need a model of their arrivals, worker count and service times before that curve can justify a headroom target.

The symptom this explains

A service's p99 latency doubles. The team adds replicas, which is the obvious response to a slow service under load, and the p99 does not move. They add more, and it gets slightly worse.

Nothing about that is mysterious once you know which queue the requests were waiting in. They were waiting for a database connection, and the pool has a fixed size, so every new replica added competitors for the same slots without adding a single one. The remedy was correct for a different queue than the one that was full.

Finding the full queue is therefore the first step rather than a detail, and doing it requires knowing how many queues a single request passes through.

The chain of queues

Follow one request from the wire and count the places it can wait. It arrives at the NIC, where packets sit in a ring buffer until the kernel drains them. Once the connection is established it waits in the kernel's accept backlog if every worker is already busy. Inside the application it may sit in a work queue before a thread picks it up. When its handler needs the database it waits for a connection pool slot, which is a real queue whenever the pool is smaller than the number of threads that want it. The query itself can then wait behind a lock holder in the database, and the write it performs can wait in the disk's IO scheduler before it reaches the device.

Seven places, and a request that is slow is slow in one or two of them rather than in all of them. That matters because the total is additive for sequential work: latency is the sum of the waits plus the time actually spent executing and in transit. Execution can dominate, and often does, so a rising p99 is not by itself evidence that any queue is full.

What distinguishes the cases is a trace of one slow request with its waits broken out. Time spent waiting for a worker looks different from time spent inside a slow computation, which looks different again from time spent blocked behind a lock holder or a downstream call. Without that breakdown the choice of remedy is a guess, which is how the opening scenario happened.

Why any of them forms a queue at all

A queue forms because work arrives faster than it is served, and the reason that happens in a system with spare capacity on average is variability. If requests arrived perfectly evenly and each took exactly the same time to serve, no queue would form anywhere until arrivals reached capacity exactly. Neither of those is true of real traffic, and the simplest model that admits it is M/M/1.

M/M/1 assumes Poisson arrivals, independent exponentially distributed service times, one server, and an unbounded first-come-first-served queue in steady state. Write utilisation as ρ = λ/μ, the arrival rate over the service rate; the queue is stable only while ρ < 1. Mean waiting time before service is then Wq = ρ / (1 − ρ) × E[S], where E[S] = 1/μ is mean service time. Note what that expression returns: a mean wait, not a p99. MIT's queueing chapter derives it.

Substituting a few values shows why the expression is worth remembering even though the assumptions rarely hold exactly. At 50 percent utilisation ρ/(1−ρ) is 1, so a request waits about as long as it takes to serve. At 80 percent it is 4. At 90 percent, 9. At 95 percent, 19. The wait is not proportional to load; it accelerates, and the acceleration is what makes a gradual traffic increase feel like a sudden failure.

Those figures belong to the model rather than to your service. A pool of many workers, service times that barely vary, or a bounded queue all produce a different and usually kinder curve. Use M/M/1 for the shape and your own measurements under representative load, bursts included, to find where on it you are sitting; saturation and utilisation targets turns that into a defensible headroom figure.

Why bursts matter more than averages

The arithmetic above assumes a stationary Poisson arrival process, which already includes random arrival gaps; it does not model arbitrary correlated bursts. Real traffic arrives in bursts, and a queue that is fine on average forms during every burst and drains afterwards.

So a service averaging 40 percent utilisation can have a p99 dominated by queueing, because for a few hundred milliseconds many times a minute it is momentarily above capacity. Short overloads can disappear in fifteen-second averages. Inspect queue wait distributions and finer-resolution traffic alongside traces before attributing the tail to bursts.

Buffers absorb bursts, not sustained overload

A larger bounded buffer can absorb a finite burst if the service catches up before callers’ deadlines. Under sustained overload, it mostly increases waiting before rejection or timeout.

A request sitting in a 10,000-deep queue for eight seconds, then being served, is worse for the user than a request rejected immediately, because the client has usually given up and retried, so you are now doing the work twice for an answer nobody wants. This is bufferbloat in the network and it is the same mistake in an application.

For sustained overload, bound the queue, reject or shed when it is full, and apply backpressure so the producer slows down rather than the buffer growing. Choose queue capacity against the waiting-time budget and discard expired work before executing it.

What each response actually fixes

• More capacity lowers utilisation, which moves you back down the curve. It fixes queueing caused by genuine load.
• Faster service does the same thing from the other direction, and is usually cheaper.
• Shedding load protects the requests you keep by refusing the ones you cannot serve. It fixes nothing and prevents collapse.
• A bigger bounded buffer can absorb a short burst, but adds no service capacity and can make sustained overload fail more slowly.

Diagnosing which queue is full comes first, because these are not interchangeable. Adding replicas to a service whose requests are queueing for database connections adds more competitors for the same pool, and makes it worse.

Ask where the request is waiting before asking why the code is slow

The two questions lead to different work, and the order matters because only one of them is cheap to answer. "Where is it waiting" is answered by instrumentation you can add once and read repeatedly: queue depth and queue wait at each of the places in the chain, recorded alongside execution time and resource saturation. "Why is the code slow" is a profiling exercise on a specific code path, and it is wasted if the request was never in that code path for long.

The instrumentation also protects you from the inverse mistake. High latency on its own does not establish that any queue is full, so a team that reaches for capacity whenever the p99 rises will sometimes add replicas to a service that was waiting on a lock, a cold cache or a slow dependency. Measuring the waits is what makes the difference visible before the money is spent.

Self-check

A service serves 500 requests per second with a mean service time of 1.6ms, and its work queue is bounded at 10,000. A burst arrives at 900 per second and lasts four seconds. Work out whether the queue absorbs it, how long the backlog takes to drain, and what the last request in that backlog experiences. Then say whether raising the bound to 100,000 would help. Try it before reading on.

Service capacity is 1 / 0.0016 = 625 requests per second, so during the burst arrivals exceed capacity by 900 − 625 = 275 per second. Over four seconds the backlog grows to about 275 × 4 = 1,100 requests, comfortably inside the 10,000 bound, so nothing is rejected. Once arrivals fall back to 500 per second the queue drains at 625 − 500 = 125 per second, so 1,100 requests take about 1,100 / 125 = 8.8 seconds to clear. The request at the back of that backlog waits most of those 8.8 seconds before it is served, which is thousands of times its 1.6ms service time.

So the queue did its job and the user experience is still bad, which is the distinction worth keeping. The bound was never the constraint here, and raising it to 100,000 changes nothing about this burst; it only changes how deep the backlog can grow before requests are rejected instead of delayed. Under a sustained arrival rate above 625 per second, the larger bound is actively worse, because it lets more requests wait past the point where their callers have given up and retried. What actually helps is a deadline: discard queued work whose caller has already timed out, so the service stops spending capacity on answers nobody will read.

Next: saturation and utilisation targets turns this into a headroom number, and the p99 question traces a tail back to the queue producing it.