Alertmanager routing, grouping and inhibition: delivering useful pages

TL;DR: Prometheus decides when an alert fires; Alertmanager decides how notifications are grouped and delivered. Match routes and suppression rules to real ownership and failure scope, then test both expected notifications and alerts that must remain visible.

Firing and paging are different states

A firing alert can exist without a notification reaching a person because it is grouped, delayed, silenced, inhibited or routed incorrectly. Diagnose the path in that order rather than changing the expression whenever a phone stays silent.

Alertmanager's overview describes deduplication, grouping, routing, silences and inhibition. Grouping combines related alerts into notifications. Inhibition suppresses notifications for target alerts when matching source alerts are active. A silence suppresses matching notifications for a defined interval.

None of those operations repairs the underlying service. Keep the alert state available for investigation even when paging is suppressed.

Group by the incident boundary

Grouping every alert by alertname alone can combine unrelated failures across production and staging or across clusters with different owners. Grouping by every label can generate one notification per Pod and preserve all the noise.

Choose labels such as environment, cluster and service according to the actual response boundary. The right key lets one responder understand one operational problem. Labels must be present and stable; a misspelled team label can send an alert to the default receiver rather than the intended on-call route.

The configuration reference defines route inheritance, matchers and notification timing. group_wait, group_interval and repeat_interval serve different purposes. Do not treat increasing all of them as a generic alert-fatigue fix.

Worked inhibition rule: a cluster outage

Suppose a production cluster has a confirmed connectivity outage and many workload-level alerts fire. The cluster-level page can carry the immediate response action, while selected dependent alerts are inhibited for that same cluster.

Require the source and target to match on environment and cluster, and verify those labels exist in both alert families. A source from cluster A must not suppress workload failures in cluster B. Missing equality labels deserve explicit tests because absent label values can undermine the intended separation.

Now introduce an independent application outage in cluster B during the test. It must still notify. This negative case proves more than a screenshot showing that the noisy cluster-A notifications disappeared.

Keep inhibition selective. A security alert or data-integrity failure may need independent action even during a cluster outage. Sharing infrastructure does not automatically make every alert redundant.

A complete routing and inhibition example

This lab configuration sends notifications to a local HTTP receiver on port 18080. Run a receiver that records POST bodies and returns 2xx before testing delivery. These URLs are placeholders for that local fixture, not a production paging integration. Save the configuration as alertmanager.yml.

route:
  receiver: default-lab
  group_by: [environment, cluster, alertname]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - receiver: payments-lab
      matchers:
        - 'team="payments"'
      continue: false
receivers:
  - name: default-lab
    webhook_configs:
      - url: http://127.0.0.1:18080/default
  - name: payments-lab
    webhook_configs:
      - url: http://127.0.0.1:18080/payments
inhibit_rules:
  - source_matchers:
      - 'alertname="ClusterUnavailable"'
      - 'environment=~".+"'
      - 'cluster=~".+"'
    target_matchers:
      - 'alertname="WorkloadUnavailable"'
      - 'environment=~".+"'
      - 'cluster=~".+"'
    equal: [environment, cluster]

An alert with team="payments" reaches payments-lab; other alerts use the default receiver. With multiple child routes, the first matching sibling stops the search unless its continue is true. The root must accept all alerts.

group_wait delays the first notification for a new group. group_interval controls checks for subsequent group changes. repeat_interval controls reminders for a group with no new changes and is checked on the group interval; use a multiple of that interval. These values affect delivery after an alert fires, separately from Prometheus's for duration.

The inhibition rule deliberately requires nonempty scope labels on both sides. Alertmanager treats missing and empty equality labels alike, so equal alone is insufficient when labels are absent. This rule suppresses only WorkloadUnavailable in the source's environment and cluster. Other alert names remain eligible to notify.

Validate syntax and the two receiver paths with amtool from the Alertmanager release you run:

amtool check-config alertmanager.yml
amtool config routes test --config.file=alertmanager.yml --verify.receivers=payments-lab team=payments alertname=WorkloadUnavailable environment=prod cluster=A
amtool config routes test --config.file=alertmanager.yml --verify.receivers=default-lab team=search alertname=WorkloadUnavailable environment=prod cluster=B

Route tests do not exercise inhibition or HTTP delivery. Send a source alert for production cluster A together with target alerts for A and B to an isolated Alertmanager. A's target should be inhibited; B's target must reach the fixture. Repeat with missing scope labels and a different alert name: neither should be suppressed by this rule. Resolve the source and check that a still-firing target can notify again.

Test delivery as an operational dependency

Validate configuration syntax, route selection and actual receiver delivery in a controlled setup. A correct route pointing to an expired integration credential still fails the user-facing purpose. Monitor notification failures and maintain a recovery path when the primary paging integration is unavailable.

Include a known test alert with representative labels and a documented expected receiver. Protect the test from accidentally paging the whole organization. A periodic end-to-end check can verify the path, but its frequency and destination should fit the team's process.

For a silence, record an owner, reason and expiry. Maintenance suppression should align with the affected service and interval, then expire automatically. A broad indefinite silence converts alert noise into undetected incidents.

Keep the response contract visible

Every page should link to a useful next action and identify the affected service. Symptom-based alerting determines which conditions deserve interruption; alert lifecycle management retires or repairs rules that no longer produce useful action.

If an alert is firing but no notification arrives, is the expression wrong? Not necessarily. Inspect route matching, grouping delays, suppression and receiver delivery before changing the detector.

Can inhibition safely match only severity? Usually that is too broad. Model the actual dependency and require the labels that identify its scope. Verify an independent incident still reaches its owner.