TL;DR: Aggregate numerator and denominator separately with sum(rate(...)) and divide once, never average pre-computed ratios; know that rate and increase extrapolate toward window edges and inspect sparse-series boundary behavior; materialize anything anyone stares at daily as a recording rule and reach for subqueries only when a rule cannot express the rollup.
How to approach it
Pin down scope and source before typing: which routes count, which window the objective covers, whether the counter label is trustworthy across deploys. Then build bottom-up, numerator first, and sanity-check against a past incident's known numbers.
A strong answer
The base ratio, done right:
sum(rate(http_requests_total{service="checkout",status=~"5.."}[5m]))
/
sum(rate(http_requests_total{service="checkout"}[5m]))
Sum rates across instances, then divide once. The classic corruption is avg(rate(bad)/rate(good)): averaging per-pod ratios gives a quiet broken pod the same weight as a busy healthy one, and one idle-but-crashed replica drags the number with almost no traffic behind it. Aggregate counts first, divide once, slice later.
Window choice has a floor. rate() needs at least two samples inside the window; practical guidance is four times your scrape interval, so 30-second scrapes want windows of two minutes or more. Shorter windows amplify jitter and make every deploy blip look like an outage.
rate and increase share extrapolation. increase(x[1h]) is equivalent to rate(x[1h]) * 3600 for the same samples. Multiplying rate by seconds does not remove approximation. Prometheus adjusts resets and extrapolates toward range boundaries with limits based on observed sample spacing; a 20-minute-old counter does not automatically report triple its observed delta over one hour. Fractional increases are possible. Use durable event records when exact transaction counts are required.
Percentiles need deliberate grouping. Cluster-wide p99:
histogram_quantile(0.99,
sum by (le)(rate(http_request_duration_seconds_bucket{service="checkout"}[5m])))
Aggregate only over le and you cannot break the result down afterward; add the dimension you will slice by (sum by (le, route)) or the per-route panel becomes impossible. Sum compatible buckets to represent the combined population. A uniform scaling of every bucket preserves the interpolated quantile, so averaging buckets is not invariably numerically wrong; missing series and differing bucket layouts make that shortcut unsafe.
Subqueries buy flexibility and spend CPU. Mean error ratio over the objective window:
avg_over_time(
(
sum(rate(http_requests_total{service="checkout",status=~"5.."}[5m]))
/
sum(rate(http_requests_total{service="checkout"}[5m]))
)[28d:5m]
)
Valid and useful for display, but note what it computes: the mean of five-minute ratios, weighting quiet night hours equally with peak traffic, not the cumulative bad-over-total ratio. Each evaluation step recomputes the inner expression, so [28d:30s] across hundreds of series melts the server. The production answer is recording rules: compute job:http_errors:ratio_rate5m continuously, let dashboards and alerts read the same number, and keep subqueries for ad hoc exploration. Drift between the alert's query and the dashboard's query is a genuine audit finding; naming conventions (level:metric:operations) exist to prevent it.
Zero traffic is a correctness trap. No matching series returns an empty vector; zero total traffic can produce 0/0 = NaN. or vector(0) neither fixes NaN nor distinguishes absent telemetry from zero errors, and can falsely display health. Initialize expected error counters, show no traffic separately, and alert on missing/stale collection using up, absent_over_time or expected-target inventory. Configure dashboard no-data behavior explicitly.
What interviewers probe next
"Your p99 disagrees with the tracing system's p99. Why?" Different measurement planes: histogram buckets quantize and drop client-side timing detail while span latencies are exact but sampled; compare populations, sampling and bucket resolution before defining a justified tolerance.
"How do you test a PromQL change safely?" Run old and new expressions over a past incident window side by side, diff outputs, promote via recording-rule pull request; never edit the live dashboard query first.
"Pod restarted mid-hour, counter reset. Does the ratio break?" No, rate detects resets and compensates; hand-rolled deltas do not, which is why manual counter arithmetic fails quietly after every rollout.
Common mistakes
Averaging ratios instead of aggregating counts, then defending the pretty dashboard that lies during partial outages.
Twenty-eight-day subqueries at fine steps evaluated on demand, discovered later as the reason Prometheus CPU doubled.
Treating an empty panel as good news; blank means unknown, and unknown must alert.
References