Alerting

What makes a good alert

An alert interrupts a person. The IT service management view, with thresholds and alert fatigue, is explained in Monitoring and Event Management; this page is about building alerts with Prometheus and Alertmanager that deserve the interruption. Google's Site Reliability Engineering (SRE) book is blunt about the cost:

"Paging a human is a quite expensive use of an employee's time. … When pages occur too frequently, employees second-guess, skim, or even ignore incoming alerts."

The same chapter adds that "you should never trigger an alert simply because 'something seems a bit weird.'" A good alert passes four tests: it is urgent (waiting until morning would hurt users), actionable (a human can do something), real (it rarely fires without a problem) and owned (it reaches the team that can fix it, with a runbook).

Symptom-based alerting means alerting on what users experience, such as failed requests, slow responses or readings that do not arrive, rather than on the internal causes that might produce it, such as high CPU, a busy disk or one restarted container. There are three reasons. A symptom covers every cause, including the ones nobody predicted. A cause often does no harm: a CPU at 90% that still answers quickly is a well-used server, not an incident. And a symptom tells the person who is woken up why it matters. Causes still belong on dashboards, where the person investigating the symptom will look for them.

Not every alert needs to wake someone. Kestrel uses two severity labels: severity="page" for problems that need a human now, at any hour, and severity="ticket" for problems that need a human this working day. How pages and tickets are handled by the people on call is the subject of On-Call. Alerting on how fast the error budget burns is the most refined form of symptom-based alerting and has its own page.

Alerting rules in Prometheus

An alerting rule is a PromQL expression that Prometheus evaluates at a fixed interval, together with a name, an optional for duration, labels and annotations; every series the expression returns becomes an active alert. Rules live in rule files, grouped by name, and are loaded through rule_files in prometheus.yml. The expressions often use recording rules such as job:http_requests_errors:ratio_rate5m, explained in PromQL.

Kestrel's rule file rules/alerts.yml:

groups:
  - name: kestrel-alerts
    rules:
      - alert: TargetDown
        expr: up == 0
        for: 2m
        labels:
          severity: ticket
        annotations:
          summary: "{{ $labels.instance }} ({{ $labels.job }}) cannot be scraped"
          description: "Prometheus has not been able to scrape {{ $labels.instance }} for more than 2 minutes."

      - alert: MeterApiHighErrorRate
        expr: job:http_requests_errors:ratio_rate5m{job="meter-api"} > 0.05
        for: 2m
        labels:
          severity: page
          team: metering
        annotations:
          summary: "meter-api returns {{ $value | humanizePercentage }} errors"
          description: "More than 5% of meter-api requests failed with a 5xx status over the last 5 minutes."
          runbook_url: "https://wiki.kestrel.example/runbooks/meter-api-errors"

      - alert: MeterApiSlow
        expr: job:http_request_duration_seconds:p95_5m{job="meter-api"} > 0.25
        for: 2m
        labels:
          severity: ticket
          team: metering
        annotations:
          summary: "meter-api p95 latency is {{ $value | humanizeDuration }}"

      - alert: PortalSlow
        expr: job:http_request_duration_seconds:p95_5m{job="portal"} > 0.5
        for: 5m
        labels:
          severity: ticket
          team: portal
        annotations:
          summary: "Customer portal p95 latency is {{ $value | humanizeDuration }}"

      - alert: MeterReadingsQueueFilling
        expr: |
          predict_linear(meter_readings_queue_depth[15m], 3600)
            > on (instance) meter_readings_queue_limit
        for: 5m
        labels:
          severity: ticket
          team: metering
        annotations:
          summary: "Readings queue on {{ $labels.instance }} will be full within an hour"
          description: "Current depth {{ with printf `meter_readings_queue_depth{instance=%q}` $labels.instance | query }}{{ . | first | value | humanize }}{{ end }} readings."

      - alert: PostgresConnectionsHigh
        expr: pg_connections_in_use / pg_connections_max > 0.8
        for: 5m
        labels:
          severity: ticket
          team: platform
        annotations:
          summary: "{{ $labels.instance }} uses {{ $value | humanizePercentage }} of max_connections"

      - alert: BillingExportStale
        expr: time() - billing_export_last_success_timestamp_seconds > 26 * 3600
        for: 10m
        labels:
          severity: ticket
          team: billing
        annotations:
          summary: "No successful billing export for {{ $value | humanizeDuration }}"
          description: "The nightly export normally finishes shortly after 02:30. Invoices for yesterday are not on their way."

The parts of a rule:

• alert: the name, which becomes the alertname label.
• expr: the condition. The rule fires for every series the expression returns, so up == 0 creates one alert per unreachable target.
• for: how long the condition must hold before the alert fires (next section).
• labels: added to the alert; Alertmanager routes on them. Keep them few and stable.
• annotations: text for humans, with templates: {{ $labels.instance }} inserts a label, {{ $value }} the value of the expression, and functions such as humanizePercentage and humanizeDuration make it readable. Put the link to the runbook in an annotation.

Two rules show patterns worth copying. BillingExportStale alerts on the absence of success: Joost Bakker's nightly export pushes the time of its last successful run, and the alert fires when that is more than 26 hours ago. That catches a crash, a hang, a job that was never started and a billing server that is down, with one rule, as long as the timestamp itself is still there; pair it with absent() (see PromQL) for the case where the series disappears. MeterReadingsQueueFilling alerts on a prediction, "full within an hour", instead of on a fixed threshold. It also contains a mistake, which the inhibition section uncovers.

promtool check rules rules/alerts.yml

Checking rules/alerts.yml
  SUCCESS: 7 rules found

Catching mistakes before they load

promtool check rules belongs in the pipeline that deploys rule files: a file Prometheus cannot load is rejected completely, and Prometheus keeps running the old rules. A file with three mistakes:

groups:
  - name: kestrel-broken
    rules:
      - record: job:http requests:rate5m
        expr: sum by (job) (rate(http_requests_total[5m]))

      - alert: MeterApiHighErrorRate
        expr: sum(rate(http_requests_total{job="meter-api",code=~"5.."}[5m]) / sum(rate(http_requests_total{job="meter-api"}[5m])) > 0.05
        for: 2m

      - alert: PortalSlow
        expr: job:http_request_duration_seconds:p95_5m{job="portal"} > 0.5
        for: 5 minutes
        annotations:
          summary: "Portal p95 is {{ $value | humanizeDuration }"

promtool check rules broken-rules.yml

Checking broken-rules.yml
  FAILED:
broken-rules.yml: unknown unit " minutes" in duration "5 minutes"
broken-rules.yml: unknown unit " minutes" in duration "5 minutes"

The duration error stops the check before anything else is looked at (and is printed twice). After changing 5 minutes to 5m, the next run finds the missing parenthesis and the broken template:

Checking broken-rules.yml
  FAILED:
broken-rules.yml: 8:15: group "kestrel-broken", rule 2, "MeterApiHighErrorRate": could not parse expression: 1:124: parse error: unclosed left parenthesis
broken-rules.yml: group "kestrel-broken", rule 3, "PortalSlow": annotation "summary": template: __alert_PortalSlow:1: unexpected "}" in operand

The space in job:http requests:rate5m is not reported: since Prometheus 3.0 any UTF-8 string is a valid metric name. It would still be a bad name, because every query would have to quote it. Fix the errors one run at a time, and keep the exit code in mind: in a test run, a duplicated recording rule produced "FAILED" and a lint message, but promtool still exited with 0 unless --lint-fatal was given (then it exited with 3). A pipeline that only checks the exit code needs that flag.

Pending, firing and resolved

A pending alert is an alert whose condition is true but has not yet been true for the whole for duration; Prometheus does not send it anywhere. The documentation describes for as checking "that the alert continues to be active during each evaluation" for that long. Only when the condition has held at every evaluation for the full duration does the alert become firing and go to Alertmanager. If the condition is false at one evaluation while pending, the alert disappears and the clock starts again. Rules without for fire at the first evaluation.

Prometheus keeps active alerts as series named ALERTS, with an alertstate label. In a test run, the target web03:8080 had nothing listening, so TargetDown (with for: 2m) went through both states after Prometheus started at 16:58:02 UTC:

promtool query range --start=2026-10-06T16:58:00Z --end=2026-10-06T17:01:00Z --step=30s http://localhost:9090 'ALERTS{alertname="TargetDown"}'

ALERTS{alertname="TargetDown", alertstate="firing", env="production", instance="web03:8080", job="portal", severity="ticket", team="portal"} =>
1 @[1791306030]
1 @[1791306060]
ALERTS{alertname="TargetDown", alertstate="pending", env="production", instance="web03:8080", job="portal", severity="ticket", team="portal"} =>
1 @[1791305940]
1 @[1791305970]
1 @[1791306000]

Pending at the steps from 16:59:00 (1791305940) to 17:00:00 and firing from 17:00:30 on: the condition became true at 16:58:25, so the alert fired two minutes later, at 17:00:25. Prometheus also stores the moment the condition became true in ALERTS_FOR_STATE, so a restart in the middle of a for period does not start the clock from zero.

Choosing for and keep_firing_for

for trades speed for certainty. Without it, one bad scrape or a few slow seconds during a deploy pages someone. With a very long for, a real outage runs for half an hour before anyone is told. A few minutes is typical for pages; tickets can wait longer. The opposite problem is an alert that flaps: a value hovering around the threshold resolves and fires again, sending a "resolved" message in the middle of an incident. keep_firing_for (since Prometheus 2.42) keeps an alert firing for a set time after the condition was last met. A simulation of one rule evaluated every 15 seconds, with a single bad evaluation at 17:09:30 and a dip below the threshold at 17:12:30:

EVAL = 15  # seconds between rule evaluations

# Is the error ratio above 5%? One value per evaluation, starting at 17:09:00.
condition = [False, False, True, False, False, False,           # a single bad evaluation
             True, True, True, True, True, True, True, True,
             False,                                             # one dip below the threshold
             True, True, True, True, False, False, False, False, False]


def simulate(hold, keep_firing):
    """Prometheus alert states for `for: hold` and `keep_firing_for: keep_firing` (seconds)."""
    state, active_at, false_since, log = "inactive", None, None, []
    for i, met in enumerate(condition):
        t = i * EVAL
        if met:
            false_since = None
            if state == "inactive":
                state, active_at = "pending", t
            if state == "pending" and t - active_at >= hold:
                state = "firing"
        elif state == "firing" and keep_firing:
            false_since = t if false_since is None else false_since
            if t - false_since >= keep_firing:
                state = "inactive"
        else:
            state = "inactive"
        log.append(state)
    return log


def show(log):
    changes = [(i, s) for i, s in enumerate(log) if i == 0 or s != log[i - 1]]
    return ", ".join(f"17:{9 + i * EVAL // 60:02d}:{i * EVAL % 60:02d} {s}" for i, s in changes)


print("for: 1m                    ", show(simulate(60, 0)))
print("for: 1m, keep_firing_for: 1m", show(simulate(60, 60)))

for: 1m                     17:09:00 inactive, 17:09:30 pending, 17:09:45 inactive, 17:10:30 pending, 17:11:30 firing, 17:12:30 inactive, 17:12:45 pending, 17:13:45 inactive
for: 1m, keep_firing_for: 1m 17:09:00 inactive, 17:09:30 pending, 17:09:45 inactive, 17:10:30 pending, 17:11:30 firing, 17:14:45 inactive

The single bad evaluation never fires, thanks to for. Without keep_firing_for, the dip at 17:12:30 resolves the alert, it goes pending again, and resolves for good before it can fire a second time: the on-call engineer gets a "resolved" message while the problem is still there. With keep_firing_for: 1m the alert fires once, from 17:11:30 to 17:14:45.

Testing alert rules

promtool check rules proves a file loads; it does not prove the rule fires when it should. promtool test rules does: it feeds invented series into the rules and compares the alerts at given times with what you expect. Kestrel's test file (tests/alerts_test.yml):

rule_files:
  - ../rules/recording.yml
  - ../rules/alerts.yml

evaluation_interval: 15s

tests:
  - interval: 15s
    input_series:
      # 100 requests per 15 s; from minute 10 on, 10 of them fail.
      - series: 'http_requests_total{job="meter-api",instance="meter-api-01:8080",method="POST",route="/v1/readings",code="202"}'
        values: '0+100x80'
      - series: 'http_requests_total{job="meter-api",instance="meter-api-01:8080",method="POST",route="/v1/readings",code="503"}'
        values: '0x40 10+10x40'
      - series: 'up{job="portal",instance="web03:8080"}'
        values: '1x20 0x60'

    alert_rule_test:
      - eval_time: 9m
        alertname: MeterApiHighErrorRate
        exp_alerts: []

      - eval_time: 15m
        alertname: MeterApiHighErrorRate
        exp_alerts:
          - exp_labels:
              job: meter-api
              severity: page
              team: metering
            exp_annotations:
              summary: "meter-api returns 9.091% errors"
              description: "More than 5% of meter-api requests failed with a 5xx status over the last 5 minutes."
              runbook_url: "https://wiki.kestrel.example/runbooks/meter-api-errors"

      - eval_time: 8m
        alertname: TargetDown
        exp_alerts:
          - exp_labels:
              job: portal
              instance: web03:8080
              severity: ticket
            exp_annotations:
              summary: "web03:8080 (portal) cannot be scraped"
              description: "Prometheus has not been able to scrape web03:8080 for more than 2 minutes."

  - interval: 1m
    input_series:
      # Test time starts at 0 (the Unix epoch), so a value of 0 means
      # "the last export succeeded when the test started".
      - series: 'billing_export_last_success_timestamp_seconds{job="billing",instance="billing01"}'
        values: '0x1800'
    alert_rule_test:
      - eval_time: 25h
        alertname: BillingExportStale
        exp_alerts: []
      - eval_time: 27h
        alertname: BillingExportStale
        exp_alerts:
          - exp_labels:
              job: billing
              instance: billing01
              severity: ticket
              team: billing
            exp_annotations:
              summary: "No successful billing export for 1d 3h 0m 0s"
              description: "The nightly export normally finishes shortly after 02:30. Invoices for yesterday are not on their way."

Series values use a short notation: '0+100x80' starts at 0 and adds 100 at each of 80 steps; '1x20 0x60' is 1 at the start and for 20 more steps (up to 5m), then 0 for the next 61 samples. The test also checks the rendered annotations, so a broken template or a wrong percentage is caught too. The first version evaluated TargetDown at 7m:

promtool test rules tests/alerts_test.yml

  FAILED:
    alertname: TargetDown, time: 7m, 
        exp:[
            0:
              Labels:{alertname="TargetDown", instance="web03:8080", job="portal", severity="ticket"}
              Annotations:{description="Prometheus has not been able to scrape web03:8080 for more than 2 minutes.", summary="web03:8080 (portal) cannot be scraped"}
            ], 
        got:[]

The test was wrong, not the rule: up is 0 from 5m15s on (step 21 of 15 seconds), so with for: 2m the alert is still pending at 7m and fires from 7m15s. With eval_time: 8m:

  SUCCESS

Writing the test forces you to think through exactly when an alert should fire, and that is often where the real mistakes surface. Run these tests in the same pipeline as promtool check rules.

Alertmanager

Prometheus' own documentation says its alerting rules are "good at figuring what is broken right now, but they are not a fully-fledged notification solution". That is the job of Alertmanager: the Prometheus component that receives firing alerts from one or more Prometheus servers, deduplicates and groups them, applies inhibition rules and silences, and sends notifications to receivers such as a pager service, email, chat or a webhook according to a routing tree. As of October 2026 the current version is 0.34.1.

Kestrel's alertmanager.yml, maintained by Tomasz Nowak in the NOC (in this test version every receiver is a webhook on localhost:5001; in production they would be the paging service, the ticket system and a mailbox):

# Alertmanager configuration for Kestrel Energy
route:
  receiver: ops-tickets
  group_by: [alertname, job]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - matchers:
        - severity="page"
      receiver: oncall-pager
      group_wait: 10s
      repeat_interval: 1h
    - matchers:
        - team="billing"
      receiver: billing-team
      group_by: [alertname]
      repeat_interval: 12h

inhibit_rules:
  # While a page is open for a job, hold back ticket-level alerts for the same job.
  - source_matchers:
      - severity="page"
    target_matchers:
      - severity="ticket"
    equal: [job]

receivers:
  - name: ops-tickets
    webhook_configs:
      - url: http://localhost:5001/tickets
        send_resolved: true
  - name: oncall-pager
    webhook_configs:
      - url: http://localhost:5001/page
        send_resolved: true
  - name: billing-team
    webhook_configs:
      - url: http://localhost:5001/billing

amtool check-config alertmanager.yml

Checking 'alertmanager.yml'  SUCCESS
Found:
 - global config
 - route
 - 1 inhibit rules
 - 3 receivers
 - 0 templates

Two typical mistakes, a route pointing at a receiver billing that is not defined, and a misspelled option (group_by_labels), each fail the check:

Checking 'alertmanager.yml'  FAILED: undefined receiver "billing" used in route

amtool: error: failed to validate 1 file(s)

Checking 'alertmanager.yml'  FAILED: yaml: unmarshal errors:
  line 6: field group_by_labels not found in type config.plain

amtool: error: failed to validate 1 file(s)

For high availability, run two or three Alertmanagers as a cluster (they share silences and notification state) and list all of them in every Prometheus server's configuration. The documentation is explicit that Prometheus should not reach them through a load balancer: each Prometheus sends every alert to every Alertmanager, and the cluster makes sure only one notification goes out.

Routing and grouping

A routing tree is the nested set of routes in Alertmanager's configuration that decides which receiver gets an alert and with which timing. Every alert enters at the top-level route, which must match all alerts, and goes down to the first child route whose matchers fit; it stops there unless the route sets continue: true. An alert that matches no child stays with the top-level route and its receiver.

amtool draws the tree and, more usefully, tells you where a given set of labels would end up, without sending anything:

amtool config routes show --config.file=alertmanager.yml
amtool config routes test --config.file=alertmanager.yml severity=page job=meter-api alertname=MeterApiHighErrorRate
amtool config routes test --config.file=alertmanager.yml severity=ticket team=billing alertname=BillingExportStale

Routing tree:
.
└── default-route  receiver: ops-tickets
    ├── {severity="page"}  receiver: oncall-pager
    └── {team="billing"}  receiver: billing-team
oncall-pager
billing-team

With --tree you see the path, which shows that a page from the billing team goes to the pager, not to the billing mailbox, because the page route comes first:

amtool config routes test --config.file=alertmanager.yml --tree severity=page team=billing

Matching routes:
.
└── default-route
    └── {severity="page"}  receiver: oncall-pager


oncall-pager

With --verify.receivers the command becomes a test that fails (exit code 1) when the result is not what you expect, which makes it suitable for a pipeline:

amtool config routes test --config.file=alertmanager.yml --verify.receivers=oncall-pager severity=ticket team=metering

ops-tickets
WARNING: Expected receivers did not match resolved receivers.

Grouping and timing

Alert grouping is Alertmanager's bundling of alerts that share the values of the route's group_by labels into one notification, so a failure that triggers many alerts at once produces one message per group instead of one per alert. The documentation's example is a network partition: hundreds of alerts, and "one only wants to get a single page". Three timers control a group:

Kestrel shortens group_wait to 10 seconds for pages: for a page, arriving twenty seconds sooner matters more than bundling a few more alerts into the first message. During the test incident, the first page went out exactly 10 seconds after the fast-burn alert fired at 17:12:33 UTC. This is what the webhook receiver got (Alertmanager's webhook format, version 4):

{
  "receiver": "oncall-pager",
  "status": "firing",
  "alerts": [
    {
      "status": "firing",
      "labels": {
        "alertname": "MeterApiErrorBudgetFastBurn",
        "job": "meter-api",
        "severity": "page",
        "team": "metering"
      },
      "annotations": {
        "description": "At this rate 2% of the monthly budget is gone within an hour. Error ratio over 1h: 2.987%.",
        "summary": "meter-api is burning its 30-day error budget at more than 14.4x"
      },
      "startsAt": "2026-10-06T17:12:33.182Z",
      "endsAt": "0001-01-01T00:00:00Z",
      "generatorURL": "http://monitor01:9090/graph?g0.expr=job%3Aslo_errors_per_request%3Aratio_rate1h%7Bjob%3D%22meter-api%22%7D+%3E+%2814.4+%2A+0.001%29+and+job%3Aslo_errors_per_request%3Aratio_rate5m%7Bjob%3D%22meter-api%22%7D+%3E+%2814.4+%2A+0.001%29&g0.tab=1",
      "fingerprint": "7893f423d547d8a6"
    }
  ],
  "notification_reason": "first notification",
  "groupLabels": {
    "alertname": "MeterApiErrorBudgetFastBurn",
    "job": "meter-api"
  },
  "commonLabels": {
    "alertname": "MeterApiErrorBudgetFastBurn",
    "job": "meter-api",
    "severity": "page",
    "team": "metering"
  },
  "commonAnnotations": {
    "description": "At this rate 2% of the monthly budget is gone within an hour. Error ratio over 1h: 2.987%.",
    "summary": "meter-api is burning its 30-day error budget at more than 14.4x"
  },
  "routeLabels": {},
  "externalURL": "http://monitor01:9093",
  "version": "4",
  "groupKey": "{}/{severity=\"page\"}:{alertname=\"MeterApiErrorBudgetFastBurn\", job=\"meter-api\"}",
  "truncatedAlerts": 0
}

groupLabels are the group_by values, commonLabels what all alerts in the group share, and endsAt of year 1 means "still firing". The generatorURL links back to the expression in Prometheus. Paging services and chat integrations receive the same information in their own formats. The main notifications during the incident (times in UTC; a selection from what the receiver logged):

The meter-api problem started at 17:09:07 and was over at 17:27:25. Two pages for one problem is acceptable (one is the budget burn, one the error ratio; they are grouped separately because their alertname differs). The resolved messages came at the next group_interval tick after the conditions cleared, not the moment they cleared.

Inhibition

Inhibition is Alertmanager's suppression of notifications for some alerts (the targets) while certain other alerts (the sources) are firing, when both have the same values for the labels listed in equal. The classic use: when a whole site is down, do not also send fifty alerts about services on that site. Kestrel's rule holds back ticket-level alerts for a job while a page for the same job is open, because the person handling the page is already looking at that service. At 17:15:25 UTC:

amtool alert query --alertmanager.url=http://localhost:9093 --active --silenced --inhibited

Alertname                    Starts At                Summary                                                          State       
TargetDown                   2026-10-06 17:05:25 UTC  web03:8080 (portal) cannot be scraped                            active      
MeterApiErrorBudgetFastBurn  2026-10-06 17:12:33 UTC  meter-api is burning its 30-day error budget at more than 14.4x  active      
MeterApiSlow                 2026-10-06 17:13:25 UTC  meter-api p95 latency is 409.2ms                                 suppressed  
MeterApiHighErrorRate        2026-10-06 17:14:25 UTC  meter-api returns 8.157% errors                                  active      
PortalSlow                   2026-10-06 17:14:55 UTC  Customer portal p95 latency is 2.147s                            suppressed  

"suppressed" covers both inhibited and silenced alerts. Asking for only the inhibited ones shows that MeterApiSlow is held back by the inhibit rule, while PortalSlow is suppressed for another reason (a silence on alertname="PortalSlow", job="portal", set while a fix for the slow invoice PDFs was deployed; silences are explained below):

amtool alert --alertmanager.url=http://localhost:9093 query --inhibited

Alertname     Starts At                Summary                           State       
MeterApiSlow  2026-10-06 17:13:25 UTC  meter-api p95 latency is 409.2ms  suppressed  

When inhibition silently does not apply

Later in the incident, MeterReadingsQueueFilling reached the ticket queue although it is a meter-api ticket and the meter-api pages were still open. The reason was in the rule expression. With on (instance), a comparison keeps only the matching label in its result:

promtool query instant http://localhost:9090 'predict_linear(meter_readings_queue_depth[15m], 3600) > on (instance) meter_readings_queue_limit'

{instance="meter-api-01:8080"} => 421307.14245082915 @[1791307638.367]
…

The same comparison without on (instance):

promtool query instant http://localhost:9090 'predict_linear(meter_readings_queue_depth[15m], 3600) > meter_readings_queue_limit'

{env="production", instance="meter-api-01:8080", job="meter-api", team="metering"} => 421312.8901219524 @[1791307638.431]
{env="production", instance="meter-api-02:8080", job="meter-api", team="metering"} => 419527.0574925534 @[1791307638.431]

(The first output is trimmed to one line; the second instance looks the same.) The first form drops job, env and team. The inhibit rule compares job: the source has job="meter-api", the target has no job at all, so they are not equal and nothing is inhibited. Here both sides already have identical labels, so the plain comparison is the fix:

predict_linear(meter_readings_queue_depth[15m], 3600) > meter_readings_queue_limit

The general lesson: routing and inhibition depend on labels, and aggregations (sum by, on, ignoring) can remove them without any error. Test alert labels with promtool test rules and check the labels in real notifications.

The opposite trap is documented in Alertmanager's own example configuration: a missing label and an empty label count as equal. If none of the labels in equal exist on either alert, they are "equal", and the inhibition applies. An inhibit rule with equal: [cluster] in a setup without a cluster label mutes every matching target everywhere.

Both effects, reproduced: the same alerts, the same routes and the same inhibit rule as in Kestrel's configuration, plus the PortalSlow silence.

# Some of the alerts firing at about 17:24 UTC (labels only)
alerts = [
    {"alertname": "TargetDown", "job": "portal", "instance": "web03:8080", "severity": "ticket", "team": "portal"},
    {"alertname": "MeterApiErrorBudgetFastBurn", "job": "meter-api", "severity": "page", "team": "metering"},
    {"alertname": "MeterApiHighErrorRate", "job": "meter-api", "severity": "page", "team": "metering"},
    {"alertname": "MeterApiSlow", "job": "meter-api", "severity": "ticket", "team": "metering"},
    {"alertname": "MeterApiErrorBudgetSlowBurn", "job": "meter-api", "severity": "ticket", "team": "metering"},
    {"alertname": "PortalSlow", "job": "portal", "severity": "ticket", "team": "portal"},
    {"alertname": "MeterReadingsQueueFilling", "instance": "meter-api-01:8080", "severity": "ticket", "team": "metering"},
    {"alertname": "MeterReadingsQueueFilling", "instance": "meter-api-02:8080", "severity": "ticket", "team": "metering"},
]

# Same structure as alertmanager.yml: first matching child route wins.
ROOT = {"receiver": "ops-tickets", "group_by": ["alertname", "job"]}
ROUTES = [
    {"match": {"severity": "page"}, "receiver": "oncall-pager"},
    {"match": {"team": "billing"}, "receiver": "billing-team", "group_by": ["alertname"]},
]
INHIBIT = {"source": {"severity": "page"}, "target": {"severity": "ticket"}, "equal": ["job"]}
SILENCES = [{"alertname": "PortalSlow", "job": "portal"}]


def matches(alert, matchers):
    return all(alert.get(name, "") == value for name, value in matchers.items())


def inhibited(alert):
    if not matches(alert, INHIBIT["target"]):
        return False
    return any(
        source is not alert and matches(source, INHIBIT["source"])
        and all(source.get(name, "") == alert.get(name, "") for name in INHIBIT["equal"])
        for source in alerts
    )


groups = {}
for alert in alerts:
    if any(matches(alert, s) for s in SILENCES):
        print(f"silenced   {alert['alertname']}")
        continue
    if inhibited(alert):
        print(f"inhibited  {alert['alertname']}")
        continue
    route = next((r for r in ROUTES if matches(alert, r["match"])), ROOT)
    group_by = route.get("group_by", ROOT["group_by"])
    key = (route["receiver"], tuple(f'{name}="{alert[name]}"' for name in group_by if name in alert))
    groups.setdefault(key, []).append(alert)

for (receiver, labels), members in groups.items():
    print(f"notify {receiver:<13} {{{', '.join(labels)}}}  {len(members)} alert(s)")

inhibited  MeterApiSlow
inhibited  MeterApiErrorBudgetSlowBurn
silenced   PortalSlow
notify ops-tickets   {alertname="TargetDown", job="portal"}  1 alert(s)
notify oncall-pager  {alertname="MeterApiErrorBudgetFastBurn", job="meter-api"}  1 alert(s)
notify oncall-pager  {alertname="MeterApiHighErrorRate", job="meter-api"}  1 alert(s)
notify ops-tickets   {alertname="MeterReadingsQueueFilling"}  2 alert(s)

The two meter-api tickets that carry job are held back; the queue alert without job slips through, and its two instances arrive as one grouped ticket. As an experiment, remove "job": "meter-api" from the two page alerts: the missing-equals-empty rule then makes them inhibit the queue alert, while MeterApiSlow and the slow-burn ticket are no longer held back.

Silences

A silence is a temporary mute in Alertmanager: a set of label matchers, a start and end time, an author and a comment. Matching alerts are still evaluated and visible, but no notifications are sent until the silence expires. Silences are for known, temporary situations: a planned change, or a problem that is already being handled and does not need to page again every hour.

Before web03 was reinstalled under change CHG-2297, its TargetDown alert was silenced for 30 minutes (the test setup user was amara). Without a comment, amtool refuses; this first attempt was also far too broad, as it would have muted every meter-api ticket:

amtool silence add --alertmanager.url=http://localhost:9093 --author=amara --duration=30m 'alertname=~"Meter.*"' severity=ticket

amtool: error: comment required by config

amtool silence add --alertmanager.url=http://localhost:9093 --author=amara --duration=30m --comment='web03 is being reinstalled (CHG-2297)' alertname=TargetDown instance=web03:8080
amtool silence query --alertmanager.url=http://localhost:9093 instance=web03:8080

44fa6b3d-15e9-4e0a-9bea-57e96731d0e7
ID                                    Matchers                                      Ends At                  Created By  Comment                                
44fa6b3d-15e9-4e0a-9bea-57e96731d0e7  alertname="TargetDown" instance="web03:8080"  2026-10-06 17:57:39 UTC  amara       web03 is being reinstalled (CHG-2297)  

silence add prints the new silence's ID. When the work is done earlier than planned, expire the silence instead of letting it run out, so a new problem with web03 is not hidden. -q prints only IDs, which makes this a one-liner:

amtool silence expire --alertmanager.url=http://localhost:9093 $(amtool silence query --alertmanager.url=http://localhost:9093 -q instance=web03:8080)

silence expire prints nothing on success. A query afterward shows only the header: no active silences.

amtool silence query --alertmanager.url=http://localhost:9093

ID  Matchers  Ends At  Created By  Comment  

For recurring quiet periods, such as a weekly maintenance window or tickets that should only arrive during office hours, use time intervals in the configuration instead of a silence someone must remember to create. A fragment (time zone names follow the IANA database):

time_intervals:
  - name: office-hours
    time_intervals:
      - weekdays: ["monday:friday"]
        times:
          - start_time: "08:00"
            end_time: "18:00"
        location: Europe/Amsterdam

route:
  receiver: ops-tickets
  routes:
    - matchers:
        - team="billing"
      receiver: billing-team
      active_time_intervals:
        - office-hours

active_time_intervals sends notifications only inside the interval; mute_time_intervals does the opposite. Alerts are still evaluated and shown during muted periods.

Fighting alert fatigue

Alert sets decay. Services change, thresholds that made sense a year ago fire every night, and people learn to swipe pages away. The cure is a regular review, monthly or after every on-call week, with three questions per alert: did it fire, did someone have to act, and was it urgent? A simple review of a month of meter-api pages in the portal rotation:

from collections import defaultdict

# September's meter-api pages in the portal rotation: (alert, hour of day, did someone have to act?)
pages = [
    ("MeterApiHighErrorRate", 3, True), ("MeterApiHighErrorRate", 14, True),
    ("MeterApiErrorBudgetFastBurn", 3, True), ("MeterApiErrorBudgetFastBurn", 10, True),
    ("DiskAlmostFull", 2, False), ("DiskAlmostFull", 4, False), ("DiskAlmostFull", 23, False),
    ("DiskAlmostFull", 1, True), ("DiskAlmostFull", 5, False), ("DiskAlmostFull", 3, False),
    ("CpuHigh", 2, False), ("CpuHigh", 9, False), ("CpuHigh", 13, False),
    ("CpuHigh", 0, False), ("CpuHigh", 22, False),
]

stats = defaultdict(lambda: {"pages": 0, "night": 0, "acted": 0})
for alert, hour, acted in pages:
    s = stats[alert]
    s["pages"] += 1
    s["night"] += hour < 7 or hour >= 22
    s["acted"] += acted

print(f"{'Alert':<28}{'Pages':>6}{'At night':>9}{'Acted':>7}  Verdict")
for alert, s in sorted(stats.items(), key=lambda item: -item[1]["pages"]):
    share = s["acted"] / s["pages"]
    verdict = "keep" if share >= 0.5 else "fix, demote to a ticket, or delete"
    print(f"{alert:<28}{s['pages']:>6}{s['night']:>9}{share:>7.0%}  {verdict}")
night = sum(s["night"] for s in stats.values())
print(f"{len(pages)} pages, {night} of them between 22:00 and 07:00")

Alert                        Pages At night  Acted  Verdict
DiskAlmostFull                   6        6    17%  fix, demote to a ticket, or delete
CpuHigh                          5        3     0%  fix, demote to a ticket, or delete
MeterApiHighErrorRate            2        1   100%  keep
MeterApiErrorBudgetFastBurn      2        1   100%  keep
15 pages, 11 of them between 22:00 and 07:00

Eleven of the fifteen pages came at night, and ten were for alerts nobody needed to act on. The two symptom alerts earned every page. DiskAlmostFull probably needs a prediction ("full within four hours", with predict_linear) instead of a fixed percentage, and CpuHigh is a cause that belongs on a dashboard. Turning them into tickets or dashboard panels cuts the pages from fifteen to four without hiding a single user-facing problem. That matters for people: every night page costs sleep, and the on-call page covers rest and compensation after night pages.

Some other habits that keep alerting healthy:

• Every page has a runbook link in its annotations, and every alert has an owning team in its labels.
• Alert on the alerting. If Prometheus or Alertmanager is down, nothing pages. A common pattern is a "watchdog" alert that always fires and is sent to an external service, which raises the alarm when it stops arriving. Alertmanager's own metric alertmanager_notifications_failed_total shows failed deliveries.
• Test the whole path after every change to receivers, with a test alert added by hand (amtool alert add alertname=TestNotification …). In a test run, with the receiver switched off, Alertmanager logged one warning at the first failed attempt and gave up five minutes later:

time=2026-10-06T19:38:50.525+02:00 level=WARN source=retry_stage.go:165 msg="Notify attempt failed, will retry later" component=dispatcher receiver=ops-tickets integration=webhook[0] aggrGroup="{}:{alertname=\"TestNotification\", job=\"alertmanager-test\"}" attempts=1 err="Post \"<redacted>\": dial tcp 127.0.0.1:5001: connect: connection refused"
time=2026-10-06T19:43:50.530+02:00 level=ERROR source=dispatch.go:589 msg="Notify for alerts failed" component=dispatcher aggrGroup="{}:{alertname=\"TestNotification\", job=\"alertmanager-test\"}" num_alerts=1 err="ops-tickets/webhook[0]: notify retry canceled after 14 attempts: Post \"<redacted>\": dial tcp 127.0.0.1:5001: connect: connection refused"

A pager integration with an expired key fails the same way, and silently from the on-call engineer's point of view: the alert fires, nobody is told. Alert on failed notifications, and send that alert through a different channel than the one that failed.