TL;DR: Review every paging rule for urgency, actionability and the need for a human now. Move nonurgent signals to an owned ticket or dashboard. Use symptom-based burn-rate alerts with paired observation windows, and retain predictive alerts when timely action can prevent serious harm.
How to approach it
State the rule for what may page before describing any mechanism, because the mechanism is downstream of the rule. Then give the multi-window burn rate concretely, with numbers.
A strong answer
The failure is structural. Alerts accumulate: every incident produces a new one, none are ever removed, and most fire on causes (CPU above 80 percent, disk above 70, pod restarted) rather than on whether users are affected. Cause alerts fire constantly because systems are always slightly unhealthy somewhere, and once a page is usually noise, on-call learns to defer all of them, including the real one. That behaviour is rational, which is why exhorting people to take alerts seriously never works.
The rule I would apply, and enforce in review: a page requires all three of urgent, actionable, and needing a human now. If automatic recovery will arrive before unacceptable user harm, a page may be unnecessary; a long self-recovering outage can still need intervention. If there is no action, it is a dashboard. If it can wait until morning, it is a ticket. Anything failing the test gets downgraded, not deleted quietly but moved to a channel or a queue where it still exists.
Then change what is measured. Alert on symptoms, meaning the SLI a user would recognise: successful requests, latency under a threshold. One symptom alert covers dozens of causes, including the ones you have not thought of, and it does not fire when a cause is present but users are fine, which is most of the time.
Burn rate is the mechanism. Error budget is the allowance for a window; burn rate is how fast you are spending it. Rate 1 means you exactly exhaust it by the window's end. For a 30-day budget window, each row below gives a long observation window and, beside it, the shorter window that confirms the problem is still active. Both have to hold, so a spike that has already recovered does not page:
| Burn rate | Window | Budget spent | Action |
|---|---|---|---|
| 14.4x | 1 hour and 5 minutes | 2% | page |
| 6x | 6 hours and 30 minutes | 5% | page |
| 3x | 1 day and 2 hours | 10% | ticket |
| 1x | 3 days and 6 hours | 10% | ticket |
The fast pair detects rapid budget consumption; the slower pair detects sustained degradation. A short confirmation window reduces continued paging after recovery, but a severe brief incident can still cross both thresholds.
Around it: include a runbook link where available, but judge actionability by what the responder can do; a link alone proves nothing. Related alerts group into one notification rather than one per pod. And review the page volume monthly with the same seriousness as an incident, because the trend is the health metric for the whole system. A reasonable target is fewer than two pages per on-call shift, and if you are far above that the answer is not more resilience training.
Turn the threshold into an actual condition
For a 99.9 percent success objective, the allowed bad fraction is 0.001. A measured failure fraction of 0.0144 is therefore a burn rate of 14.4. Calculate failures divided by eligible requests over each window, aggregating counts before division. Averaging per-pod percentages would give a quiet pod as much influence as a busy one.
For the fast page, require the one-hour burn and five-minute burn to exceed the threshold together. The six-hour and thirty-minute pair forms a separate page condition; either qualifying pair can notify. The shorter window helps clear the condition after recovery. It does not guarantee that a short, severe outage will never page: such an outage can breach both windows and warrant action.
Test recorded incidents and quiet periods against the rule before enabling notifications. Include zero traffic, missing series, a telemetry outage and a deployment that changes labels. Missing observations are not zero errors. Use a separate telemetry-health rule and evaluate synthetic journeys separately so artificial successes do not dilute the real user failure ratio.
What interviewers probe next
"What would you still alert on that is not a symptom?" Predictive signals can justify a page when action is needed before the next working period: imminent disk exhaustion or a certificate expiry that cannot wait. Earlier warnings can be tickets. These failures may be recoverable afterward, but preventing the user harm is preferable.
"How do you handle a service with too little traffic for an SLO?" Low volume makes ratios volatile. Consider longer windows and explicit event-count context, accepting the detection tradeoff. Synthetic probes can provide a separate availability signal; do not mix their successes into the real-user denominator.
"Who owns the alert?" The team that owns the service. A central team writing alerts for services it does not operate is how the noise accumulated originally.
Common mistakes
Tuning thresholds instead of changing what is measured. It reduces volume for a quarter and it comes back.
Paging on cause metrics, which is the root of the problem.
Deleting alerts with no replacement and no owner, which trades noise for blindness.
References