Alert on symptoms, not causes
Choose symptom-based alerts from user outcomes. Compare CPU, latency and queue-age signals, calculate error-budget burn, and decide when a cause still deserves a page.
TL;DR: Alert on symptoms by measuring failed user outcomes: requests failing, responses missing a latency objective or data arriving too late. Keep internal metrics for diagnosis. An internal condition can still deserve a page when it predicts harm that someone must act now to prevent.
Symptoms and causes: examples
A symptom describes the service outcome. A cause describes a condition that might explain it. The distinction depends on whose service you are measuring: a database error is internal to a checkout journey, but can be a symptom for clients of a database service.
| Observation | What it tells you | Possible response |
|---|---|---|
| CPU is 90%, request latency is healthy | Little compute headroom; no measured request harm yet | Investigate capacity and saturation trends |
| Eligible checkout requests fail above the objective | Customers cannot complete the measured operation | Evaluate burn rate and mitigation urgency |
| Queue depth rises briefly, then drains | Arrival and processing rates differed temporarily | Inspect rate and age before paging |
| Oldest unprocessed order approaches its deadline | The customer outcome is at risk | Act if recovery will arrive too late |
| A disk is forecast to fill before the next staffed shift | Imminent failure may be preventable | Page when someone has an effective intervention |
Why cause alerts multiply
CPU thresholds, restarts and queue-depth limits often enter monitoring after an incident. Each can be useful evidence, but a threshold copied from one failure may also trigger during normal load. Review what action the alert requests and whether that action was needed the last time it fired.
Repeated notifications without useful action make it harder to distinguish urgent work. Consolidate related symptoms and preserve diagnostic metrics in dashboards or runbooks. Removing the page does not require discarding the measurement.
What a symptom alert covers
Measure the service level indicator: successful requests over eligible requests, latency inside the objective, or freshness of the data a consumer reads. A failure in code, networking or storage can all affect the same indicator. This gives the detector coverage beyond a list of predicted causes, within the user journey it actually measures.
Check that coverage explicitly. A healthy HTTP response does not prove an asynchronous order completed. A service-wide ratio can also hide a failing customer segment. Decide which journeys and cohorts need their own indicators before adjusting thresholds.
The three conditions a page has to meet
Before something is allowed to wake a person, it has to satisfy all three of these.
It has to be urgent: waiting for automatic recovery would permit unacceptable harm. It has to be actionable: a responder can do something useful now. And that action must be needed before normal working hours. A self-recovering outage can still satisfy all three conditions if recovery is too slow.
Anything failing one of the three goes elsewhere. Not deleted, because that is how the knowledge is lost and how the next incident review recreates the same alert, but moved to a ticket queue or a channel where it still exists and nobody is woken by it.
Burn rate is what gives a symptom alert its urgency
An error-rate threshold without a duration does not express how quickly a service spends its allowance. Burn rate compares the observed bad fraction with the fraction the objective allows; observation windows determine how much sustained evidence is required. Neither automatically tells you the remaining budget. The error budget concept works through the arithmetic.
The thresholds people quote are derived rather than conventional, and the derivation is worth seeing once. Pick how much of the budget you are willing to lose before paging, and over what window, then convert. For a 30-day objective the period is 30 × 24 = 720 hours. The following conversion assumes a time-based indicator or steady eligible-request volume; variable traffic needs request-weighted budget accounting:
burn rate = budget fraction / (window / 720 hours)
Losing 2 percent of the budget within one hour is 0.02 / (1/720) = 14.4. Losing 5 percent within six hours is 0.05 / (6/720) = 6. Losing 10 percent across three days is 0.10 / (72/720) = 1. Those are example response thresholds. Pair a longer window with a shorter confirmation window so a recovered spike can stop paging. In a multiwindow rule, combine the pair with AND; a short-window breach alone does not meet that rule.
Where a cause alert still earns a page
Cause alerts are diagnostics, and they are valuable as diagnostics: linked from the runbook, present on the dashboard, posted into the incident channel. They can earn a page when the condition predicts material harm far enough ahead that a person can prevent it, even before a symptom appears.
The distinction is prediction, not internality. A certificate expiring in seven days, where renewal takes an hour of working time, is an owned ticket with a deadline; there is no reason for it to be a page tonight. A disk forecast to fill in four hours may well be a page, because four hours is inside the response window and the consequence is a hard stop. GPU memory pressure earns a page when it is tied to an imminent admission decision that a person has to make.
Document the predicted harm and the available response window. A threshold without that explanation is difficult to review or act on.
Include coverage and response time in the decision
Consider an order service whose API returns quickly while a downstream queue stops processing. An HTTP latency alert stays green. A freshness indicator based on the oldest unprocessed order exposes the missed user outcome; an age approaching the fulfilment deadline may require action before customers complain. Write the indicator from the completed journey, including asynchronous stages that matter.
For each page, document the first action, expected harm if nobody responds, and the time available. A queue that normally recovers in twenty minutes may still require intervention when customers have a ten-minute delivery commitment. Automatic recovery is relevant only when it arrives soon enough. Conversely, a certificate expiring next week normally permits an owned daytime task.
Exercise the notification path with a controlled test: measurement, rule evaluation, routing, receiver and acknowledgement. A correct symptom rule cannot notify anyone if its telemetry disappears or its receiver is misconfigured. Give missing-data and delivery failures explicit owners. After a real incident, compare detection and mitigation times with the available response window before adjusting thresholds.
Self-check
A service has a 99.9 percent success objective over 30 days and receives a steady 200,000 eligible requests per hour. In the last hour, 1,400 requests failed. Work out the observed bad fraction, the burn rate, roughly what share of the month's budget that hour consumed, and whether this should page under a 14.4 threshold. Try it before reading on.
The allowed bad fraction is 1 − 0.999 = 0.001. Observed is 1400 / 200000 = 0.007, or 0.7 percent. Burn rate is 0.007 / 0.001 = 7. At steady volume, that hour consumed about 7 / 720 = 0.97 percent of the month's allowance.
The one-hour value is below 14.4. It does not meet a rule requiring both the one-hour and five-minute rates above 14.4. To evaluate a slower rule requiring six-hour and thirty-minute rates above 6, you need those two measurements; one hour of data is insufficient.
| One-hour burn | Five-minute burn | Six-hour burn | Thirty-minute burn | Result under these example rules |
|---|---|---|---|---|
| 20 | 2 | 4 | 3 | Neither pair qualifies; the recent spike recovered |
| 20 | 18 | 4 | 12 | Fast pair qualifies |
| 7 | 7 | 7 | 7 | Slow pair qualifies |
These rules do not decide every incident's urgency. Remaining budget, affected users and imminent harm can justify action independently. A burn of 7 does not by itself establish either an emergency or a daytime task.
Next: choose alert windows works through window pairs against a traffic profile, and the alert fatigue question rebuilds a noisy rotation from the symptom up.
Sources: Google's SLO alerting chapter for the multiwindow method and its low-traffic limitations.