TL;DR: Latency multiplies concurrency (Little's law), exhausted workers stall every caller, and retries pile new load onto shrinking capacity until healthy services die of crowding. Prevent it with timeouts sized from SLOs, budgeted retries with jitter, and load shedding by priority, rehearsed before the incident.
How to approach it
Do the arithmetic out loud first; derive how the slowdown consumes the available concurrency before naming controls. Then separate what saves tonight from what stops a recurrence.
A strong answer
Start with one service and one slow dependency. If a frontend receives 100 requests per second and each call took 20ms, about 2 calls are in flight at any moment. When the dependency slows to 2 seconds, in-flight count becomes roughly 200. Nothing failed yet; the work is just slower. But those 200 requests occupy 200 worker threads or connections, and the pool was sized for 50, so everything else queues behind them, including requests to healthy dependencies on the same threads. The service is now globally slow while its dependency is merely locally slow.
Clients see timeouts arrive late, after spending their own patience, and retry. Retries add load exactly when capacity has collapsed. The multiplier compounds through tiers: if each request fans out to three dependencies and each client retries twice on timeout, a saturated leaf can face several times its normal traffic, generated by the people trying to recover. Load balancers then make it worse in a second way, and this loop is worth following because it runs without any retries at all. Ten instances each carry a tenth of the traffic. One saturates and misses health checks, so it is removed and its share redistributes: the nine survivors now carry 1 / 9 ≈ 11.1 percent each instead of 10, a 11 percent step increase applied to instances that were already at the edge. The next one fails, the remaining eight take 1 / 8 = 12.5 percent, then 1 / 7 ≈ 14.3, and each removal raises the load on everything still standing by a larger increment than the last. Health checking is removing capacity in response to a capacity shortage, which is why an overload can empty a pool in under a minute while every individual decision looks correct.
What saves the night is shedding, not healing. Drop low-priority traffic at the ingress with 429 and Retry-After so queues drain, disable non-critical callers of the sick dependency behind a flag, and add capacity for that dependency if it is yours. Root-causing the slowdown matters but happens after the bleed stops.
What stops the next one is a set of budgets enforced everywhere:
| Mechanism | What it bounds | Where it lives |
|---|---|---|
| Timeouts from SLOs | How long any caller waits | Every hop, never default infinity |
| Retry budget | Extra load retries may add, e.g. retries under 10 percent of live traffic | Client library or mesh |
| Backoff with jitter | Retry waves firing in synchronisation | Same |
| Circuit breaker | Calls into a failing dependency, fail fast instead of fail late | Per-dependency client |
| Bulkheads | Blast radius of pool exhaustion | Separate pools per dependency |
| Shedding | Who gets served during scarcity | Ingress, by priority class |
Retry transient failures only when the operation is safe to repeat or protected by an idempotency contract. Bound attempts and the total deadline. A percentage retry budget is one design; gRPC retry throttling instead maintains a per-server token balance, subtracting on qualifying failures and replenishing on successes. Retries stop below half of maxTokens until successful calls restore the balance.
Count attempts at the leaf
In a hypothetical chain A → B → C, A allows three attempts to B and B allows three attempts to C for each received attempt. One original request can generate up to nine calls to C. That bound assumes every eligible attempt is made; deadlines or cancellation may stop it earlier. Add a third retrying tier and the worst-case multiplication grows again.
Choose one retry owner where practical, and measure attempts per original request. For a write, a timeout leaves the outcome uncertain: the dependency may have committed before its response was lost. An idempotency key must be enforced by the operation's storage or service contract, not merely attached as a header that nobody checks.
During recovery, reduce admitted work until queues drain, then restore traffic gradually. A 429 with Retry-After helps only when callers honor it and do not synchronize their return. Verify retry rate, useful completion rate and queue age together; a lower error rate caused by admitting almost no useful work is a different outcome from restored service.
What interviewers probe next
"Where do circuit breakers go wrong?" They flap open and closed in lockstep without half-open hysteresis, or one shared breaker across a service that owns many dependencies converts a local failure into a global refusal to serve.
"Why jitter?" Without it, every client that timed out at the same instant retries at the same instant, and you have built a metronome for the overload you were escaping.
"Your shed logic needs priorities. Who assigns them?" Product owners, in code review, per endpoint. Priorities invented by the platform team during the incident are worthless because nobody trusts them.
Common mistakes
Treating thread-pool exhaustion as the root cause and sizing pools bigger. Bigger pools hold more work in flight longer, which deepens the queue at the dependency and delays your own timeouts past usefulness.
Setting timeouts generously "to let slow calls finish". A timeout exists to bound your contribution to the cascade; generous timeouts are how a two-second slowdown becomes a ten-minute outage.
Adding bulkheads after the fact but leaving one shared connection pool to the database. A shared database pool can still allow one slow path to block unrelated requests.
References