TL;DR: The mean and p99 describe different parts of the latency distribution. A large gap warrants checking which requests are slow and where they spend time, so you need per-request evidence rather than aggregates: traces with exemplars, grouped by the dimension that separates the slow ones. The usual causes are queueing, garbage collection, cold caches, an unlucky shard, and retries.
How to approach it
State immediately that averages cannot answer this, then say what evidence would. The round rewards someone who reaches for traces rather than more dashboards.
A strong answer
An average over a bimodal distribution describes neither mode. If 99 percent of requests take 150ms and 1 percent take 3s, the mean is around 180ms and looks healthy. So the first move is to stop looking at aggregates and find actual slow requests. Exemplars are the mechanism: they can attach a trace ID to a histogram observation. Follow an exemplar from a slow bucket, then confirm the trace was sampled and is still retained; a p99 estimate is not itself a bucket.
Then group the slow requests and look for what they share. The candidates, in the order I would check them:
Queueing. The most common cause and the one people miss, because the service is not saturated on average. If arrivals are bursty, a queue forms during the burst and the requests behind it wait. Utilisation of seventy percent on average can still mean momentary saturation many times a minute. The evidence is that slow requests cluster in time rather than by user or endpoint.
Garbage collection or runtime pauses. A stop-the-world pause of a few hundred milliseconds hits whatever requests are in flight. The tell is that slow requests are uncorrelated with anything about the request itself, and that they cluster on one instance at a time.
Cache misses. A warm cache serves in single-digit milliseconds and a miss goes to the origin. At a 99 percent hit rate, p99 lies near the hit/miss boundary and depends on the observed distribution and quantile estimator; slower percentiles are likely to include misses. Compare slow-request latency to your known cold-path cost. A matching duration supports the hypothesis; confirm the cache-miss attribute on those requests.
An unlucky backend. One slow shard, one degraded node, one replica with a failing disk. Group by instance and by shard key. A service-wide tail can therefore originate on one host.
Retries and timeouts. A request that fails and retries takes the timeout plus the retry. If your client timeout is 1s with two retries, 3s is not a mystery, it is arithmetic. Check whether the slow requests are actually successful second or third attempts.
The framing worth offering at the end: tail latency is a structural property, not a tuning problem. Fan-out makes it worse mechanically, because a request that waits on twenty backends is as slow as the slowest of twenty samples from the tail. That is why hedged requests and per-try timeouts exist.
Check the population before explaining the tail
For an illustrative minute containing 10,000 requests, 9,900 at 150 ms and 100 at 3,000 ms have a mean of 178.5 ms. The exact reported p99 depends on the estimator near that boundary. A histogram with a wide top bucket can also give a coarse quantile estimate, so inspect bucket counts and request volume before interpreting a precise-looking graph.
Do not average the p99 values of individual replicas to obtain the service p99. Aggregate compatible histogram bucket counts, or query a telemetry system that can merge the underlying distributions. Then compare the same route and time interval at the client and server; different start and end points can explain an apparent disagreement.
Use a retained slow trace to formulate a test. If it spends two seconds waiting for a connection and only 40 ms executing a query, investigate connection-pool wait and outstanding holders. Increasing the database query timeout would lengthen the symptom without addressing that wait.
What interviewers probe next
"Which percentile should you alert on?" Something users feel, so p99 or p99.9 for user-facing paths, and always alongside a rate so you know how many people that is.
"How do you find the slow requests if you sample traces at one percent?" Tail-based sampling, which decides after the trace completes and keeps the slow ones. Random head sampling gives slow traces the same inclusion probability as other traces, but can miss rare failures in a small sample. Tail sampling needs enough buffer capacity and complete trace routing to retain the desired cases.
"Averages hid this. What else do they hide?" Bimodal anything: two datacentres, cache hit and miss, retried and first-attempt.
Common mistakes
Proposing to scale out. If the cause is GC or one bad host, scaling alone does not explain the fault, although reducing per-instance load can help some GC or queueing cases.
Looking only at server-side latency. The client sees queueing and connection setup that server metrics never record.
Treating p99 as a tuning target without asking how many requests that is. One percent of a million requests is ten thousand requests, potentially including repeated requests from the same users.
References