Error Budgets and Burn-Rate Alerts — Observability & SRE

TL;DR

The error budget is 100% minus the SLO: at 99.9% over 30 days, 0.1% of requests (or 43 min 12 s) may fail.
An error budget only works with a policy agreed in advance that says what happens when it runs out, such as a release freeze except for fixes.
Burn rate = error ratio ÷ (1 − SLO). Burn rate 1 uses the budget in exactly the SLO window; burn rate 14.4 uses 2% of a 30-day budget in one hour.
Multiwindow, multi-burn-rate alerts page when both a long and a short window burn too fast (14.4× over 1 h and 5 min, 6× over 6 h and 30 min) and send slow burns as tickets.
The thresholds 14.4, 6 and 1 assume a 30-day window. For other windows, recompute them.

The error budget

An error budget is the amount of failure an SLO allows: 100% minus the target, over the SLO's window. For Kestrel's customer portal, with an SLO of 99.9% of requests without a server error over a rolling 30 days:

The budget is 0.1% of requests. With 18 million portal requests in 30 days, that is 18,000 failed requests.
Expressed as time, on a 30-day basis (43,200 minutes), it is 43 min 12 s of complete outage.

The ITSM page shows the idea and a minute-based example, and the cloud page has a first burn-rate calculation. This page goes further: the policy that makes the budget matter, and alerts that page at the right moment.

A budget is meant to be spent. Releases, migrations, experiments and planned maintenance all carry some risk; spending budget on them is the point. A team that ends every month with 95% of its budget left is probably being too careful, or its SLO is too loose.

A burn-down of the error budget. The dashed lines are constant burn rates; a real month is a slow background burn with steep drops during incidents.

The error budget policy

A budget without consequences is just a number on a dashboard. An error budget policy is a short document, agreed in advance by the people who build the service, the people who run it and the people who own it, that says what happens when the budget is spent. The SRE Workbook's example policy contains two rules most teams start from:

"If the service has exceeded its error budget for the preceding four-week window, we will halt all changes and releases other than P0 issues or security fixes until the service is back within its SLO."

"If a single incident consumes more than 20% of error budget over four weeks, then the team must conduct a postmortem. The postmortem must contain at least one P0 action item to address the root cause."

Kestrel's portal policy is owned by Sanne de Vries as service owner and approved by Yara Haddad, the release manager, who enforces the freeze through the normal change schedule:

Error budget policy: customer portal              v1.0, 2026-09-01
Owner: Sanne de Vries (service owner)
Approved: Yara Haddad (release manager), Ravi Kumar (tech lead)
SLO: 99.9% of portal requests without a 5xx, rolling 30 days

1. Budget left: releases follow the normal change process.
2. Budget spent (SLI below the SLO over the last 30 days):
   - only security fixes and fixes for P1/P2 incidents are released,
     as emergency changes, until the SLI is back above the SLO;
   - the team's next sprint goal is reliability work.
3. One incident uses more than 20% of the budget: a postmortem with
   at least one high-priority action item, tracked as a problem.
4. Budget spent by an outage of another team's service or of a
   supplier: no freeze for this team; the incident is reviewed
   together with the team or supplier that caused it.
5. Disagreement about applying this policy: the service owner
   decides, and the decision is written down.
Reviewed every quarter, together with the SLO.

Three things make such a policy work:

It is written before it is needed. Arguing about a freeze while the budget is gone turns into a negotiation about one release; agreeing on it in a quiet month makes it a rule.
It is not a punishment. A freeze buys time for reliability work; it is not a verdict on the developers. If a policy is used to blame people, they will start hiding incidents from the SLI.
It covers the exceptions. Rule 4 is important: when a supplier's outage spends the budget, freezing the portal team's releases does nothing for reliability.

Burn rate

The SRE Workbook defines burn rate as "how fast, relative to the SLO, the service consumes the error budget". It is the current error ratio divided by the error ratio the SLO allows:

burn rate          = error ratio / (1 - SLO)
budget consumed    = burn rate x window / SLO period
time to exhaustion = SLO period / burn rate

With a 99.9% SLO, an error ratio of 0.1% is burn rate 1: the budget lasts exactly the 30 days. An error ratio of 1% is burn rate 10: the budget is gone in 3 days. The script below prints the Workbook's reference table, and what a 40-minute incident with 5% errors costs:

SLO = 0.999
BUDGET = 1 - SLO            # 0.1% of requests may fail
PERIOD_H = 30 * 24          # 30-day window = 720 hours


def how_long(hours):
    if hours >= 48:
        return f"{hours / 24:.1f} days"
    if hours >= 2:
        return f"{hours:.1f} hours"
    return f"{hours * 60:.0f} minutes"


print(f"{'Burn rate':>9}  {'Error ratio':>11}  {'Budget gone in':>14}")
for burn in (1, 2, 6, 10, 14.4, 1000):
    print(f"{burn:>9}  {burn * BUDGET:>11.2%}  {how_long(PERIOD_H / burn):>14}")

# A 40-minute incident with 5% errors
burn = 0.05 / BUDGET
used = burn * (40 / 60) / PERIOD_H
print(f"\n40 min at burn rate {burn:.0f} uses {used:.1%} of the 30-day budget")

Burn rate  Error ratio  Budget gone in
        1        0.10%       30.0 days
        2        0.20%       15.0 days
        6        0.60%        5.0 days
       10        1.00%        3.0 days
     14.4        1.44%        2.1 days
     1000      100.00%      43 minutes

40 min at burn rate 50 uses 4.6% of the 30-day budget

That last line is a real Kestrel incident. On Tuesday 22 September 2026 a portal release (CHG-0703) went out at 14:05, and from 14:10 one in twenty requests to /api/usage failed with a 500. Kwame Asante, on call that week, was paged at 14:27, rolled the release back at 14:50 following its back-out plan, and the errors stopped. INC-21986 used 4.6% of the budget, under the policy's 20% trigger, but the on-call rollback made a postmortem mandatory anyway; Sanne kept it short and focused on why the release had passed all its tests.

Judging an alert

Before building alerts on the budget, decide what a good alert is. The Workbook uses four criteria. The first two come in a pair: alert precision is the share of alerts that correspond to a significant event (100% if every page was worth waking up for), and recall is the share of significant events that produced an alert (100% if nothing serious was missed). The words come from statistics; the AI topic uses precision and recall in the same sense for classifiers. The other two are about time. Detection time is how long a problem runs before the alert fires, and reset time is how long the alert keeps firing after the problem is gone.

Simple alerts fail on at least one of the four. The Workbook walks through the obvious attempts:

Alert	What goes wrong
Error ratio over 10 minutes ≥ 0.1%	Poor precision: it fires on blips that spend only 0.02% of the budget, up to 144 times a day, while the SLO is met.
Error ratio over 36 hours > 0.1%	Better precision, but terrible reset time: after a short total outage the alert keeps firing for 36 hours.
Error ratio over 1 minute, with `for: 1h`	Poor detection and recall: a total outage alerts after one hour, the same as a tiny one, and an error ratio that dips below the line now and then never alerts at all.

The fix is to alert on the burn rate, with a threshold that corresponds to a meaningful part of the budget. Paging when 2% of a 30-day budget is spent within one hour means a threshold of 0.02 × 720 h ÷ 1 h = 14.4. Detection time then scales with the severity of the problem:

BUDGET = 0.001       # 99.9% SLO
WINDOW_MIN = 60      # long window of the fast-burn alert
THRESHOLD = 14.4     # burn rate that pages

for error_ratio in (1.0, 0.10, 0.05, 0.02, 0.0144, 0.01):
    burn = round(error_ratio / BUDGET, 6)    # avoid 14.399999...
    if burn < THRESHOLD:
        print(f"{error_ratio:>7.2%} errors, burn rate {burn:g}: never pages")
        continue
    minutes = BUDGET / error_ratio * WINDOW_MIN * THRESHOLD
    print(f"{error_ratio:>7.2%} errors, burn rate {burn:g}: pages after {minutes:.1f} min")

100.00% errors, burn rate 1000: pages after 0.9 min
 10.00% errors, burn rate 100: pages after 8.6 min
  5.00% errors, burn rate 50: pages after 17.3 min
  2.00% errors, burn rate 20: pages after 43.2 min
  1.44% errors, burn rate 14.4: pages after 60.0 min
  1.00% errors, burn rate 10: never pages

A total outage pages within a minute; INC-21986's 5% errors paged after 17 minutes, which is why Kwame's phone went off at 14:27. A 1% error ratio never pages through this alert. It would still use the whole budget in three days, so it needs a second, slower condition.

Multiwindow, multi-burn-rate alerts

The Workbook's recommended design combines two ideas. Multiple burn rates: a fast condition pages for sudden, severe problems, and slower conditions catch problems that would drain the budget over days. Multiple windows: each condition must be true over a long window and over a short window, about one twelfth of the long one. The result is the multiwindow, multi-burn-rate alert. For a 99.9% SLO over 30 days, the Workbook's starting point is:

Severity	Long window	Short window	Burn rate	Budget consumed
Page	1 hour	5 minutes	14.4	2%
Page	6 hours	30 minutes	6	5%
Ticket	3 days	6 hours	1	10%

The first row is a fast burn: a high burn rate that would empty the budget within days (at 14.4, in about two), and needs a person now. The last row is a slow burn: a low but persistent burn rate that would empty the budget within the window if nothing changes, which is a job for the next working day, not for 03:00. The Workbook's example rule file also contains a ticket condition at burn rate 3 over 24 hours and 2 hours.

Why the short window matters

The long window gives precision: a few bad seconds do not move a one-hour average much. But it also remembers: after the problem is fixed, the one-hour error ratio stays high for up to an hour. The short window solves that. It drops below the threshold minutes after the errors stop, and because both must be true, the alert clears. This simulation replays INC-21986 minute by minute, with 600 portal requests a minute:

BUDGET = 0.001
LIMIT = 14.4 * BUDGET                  # 1.44% errors
START = 13 * 60 + 10                   # the series starts at 13:10

# Portal minutes from 13:10 to 16:10: 600 requests per minute,
# 5% of them fail from 14:10 until the rollback at 14:50.
requests = [600] * 180
errors = [30 if 60 <= m < 100 else 0 for m in range(180)]


def ratio(minute, window):
    """Error ratio over the last `window` minutes, ending at `minute`."""
    first = max(0, minute - window + 1)
    return sum(errors[first:minute + 1]) / sum(requests[first:minute + 1])


def clock(minute):
    hours, mins = divmod(START + minute, 60)
    return f"{hours:02d}:{mins:02d}"


def firing(label, condition):
    minutes = [m for m in range(60, 180) if condition(m)]
    print(f"{label:<18} fires {clock(minutes[0])}, clears {clock(minutes[-1] + 1)}")


firing("1h window only", lambda m: ratio(m, 60) > LIMIT)
firing("1h and 5m windows", lambda m: ratio(m, 60) > LIMIT and ratio(m, 5) > LIMIT)

1h window only     fires 14:27, clears 15:32
1h and 5m windows  fires 14:27, clears 14:53

Both fire at the same moment, but the single-window alert keeps firing for 42 minutes after the rollback. During that time the responder cannot tell from the alert whether the fix worked, and a second problem would be hidden behind the first. With the short window, the reset time is three minutes.

The same incident seen by two alerts. The long window alone detects well but resets slowly; adding a short window makes the alert clear minutes after the errors stop.

The portal's rules

In Prometheus, each condition compares recorded error ratios (see recording the SLI per window) with the burn-rate threshold times the budget. The Workbook prints its example as shorthand, with severity at the top level; a real rule file puts it under labels. This is the portal's file, with both page conditions in one alert and the slow burn as a ticket:

rules/slo-portal-alerts.yml

groups:
  - name: portal-slo-alerts
    rules:
      - alert: PortalErrorBudgetBurn
        expr: |
          (
            job:slo_errors_per_request:ratio_rate1h{job="portal"} > (14.4 * 0.001)
            and
            job:slo_errors_per_request:ratio_rate5m{job="portal"} > (14.4 * 0.001)
          )
          or
          (
            job:slo_errors_per_request:ratio_rate6h{job="portal"} > (6 * 0.001)
            and
            job:slo_errors_per_request:ratio_rate30m{job="portal"} > (6 * 0.001)
          )
        labels:
          severity: page
        annotations:
          summary: "Portal is burning its 30-day error budget fast"
          runbook_url: "https://wiki.kestrel.example/runbooks/portal-error-budget"

      - alert: PortalErrorBudgetSlowBurn
        expr: |
          job:slo_errors_per_request:ratio_rate3d{job="portal"} > (1 * 0.001)
          and
          job:slo_errors_per_request:ratio_rate6h{job="portal"} > (1 * 0.001)
        labels:
          severity: ticket
        annotations:
          summary: "Portal will use its whole error budget within 30 days at this rate"

The expressions need recording rules for the portal's 5m, 30m, 1h, 6h and 3d error ratios, built like the meter-api rules on the SLI page. There is no for: clause: the windows already demand a sustained burn, and the Workbook advises against adding a duration. Three conditions that are true at once (a big burn also crosses the slower thresholds) should not cause three notifications. Combining the pages into one alert, and an Alertmanager inhibit rule that holds back tickets while a page for the same job is open, take care of that (see alerting).

Other SLO windows

The numbers 14.4, 6 and 1 are not magic. They are the budget share multiplied by the SLO period and divided by the alert window, for a 30-day period. A team with a 28-day or 7-day SLO that copies them gets alerts that fire at the wrong moment:

def threshold(budget_share, window_h, period_days):
    """Burn rate that spends budget_share of the budget within window_h."""
    return budget_share * period_days * 24 / window_h


conditions = [(0.02, 1), (0.05, 6), (0.10, 72)]   # (budget spent, window)
for period in (30, 28, 7):
    row = "  ".join(f"{share:.0%}/{w}h: {threshold(share, w, period):<5.3g}"
                    for share, w in conditions)
    print(f"{period:>2}-day SLO window  {row}".rstrip())

30-day SLO window  2%/1h: 14.4   5%/6h: 6      10%/72h: 1
28-day SLO window  2%/1h: 13.4   5%/6h: 5.6    10%/72h: 0.933
 7-day SLO window  2%/1h: 3.36   5%/6h: 1.4    10%/72h: 0.233

With a 7-day window the whole budget is smaller, so the same "2% in an hour" is reached at a burn rate of 3.36. Copying 14.4 would only page when about 8.6% of the weekly budget was gone in an hour.

Burn-rate alerts in a test run

The meter-api rules were tried in a test setup: a real Prometheus 3.15 and Alertmanager 0.34 against small fake exporters with invented Kestrel metrics, replaying the meter-api incident of 6 October that is described on the incident response page (meter-api failed about 10% of readings from 17:09 to 17:27 UTC, 19:09 to 19:27 in Amsterdam). These are the alerting rules, the second group of the file whose recording rules are on the SLI page:

rules/slo-meter-api.yml (second part)

groups:
  - name: slo-meter-api-burn-rate
    rules:
      - alert: MeterApiErrorBudgetFastBurn
        expr: |
          job:slo_errors_per_request:ratio_rate1h{job="meter-api"} > (14.4 * 0.001)
          and
          job:slo_errors_per_request:ratio_rate5m{job="meter-api"} > (14.4 * 0.001)
        for: 2m
        labels:
          severity: page
          team: metering
        annotations:
          summary: "meter-api is burning its 30-day error budget at more than 14.4x"
          description: "At this rate 2% of the monthly budget is gone within an hour. Error ratio over 1h: {{ $value | humanizePercentage }}."

      - alert: MeterApiErrorBudgetSlowBurn
        expr: |
          job:slo_errors_per_request:ratio_rate6h{job="meter-api"} > (6 * 0.001)
          and
          job:slo_errors_per_request:ratio_rate30m{job="meter-api"} > (6 * 0.001)
        for: 15m
        labels:
          severity: ticket
          team: metering
        annotations:
          summary: "meter-api is burning its 30-day error budget at more than 6x"
          description: "At this rate 5% of the monthly budget is gone within 6 hours. Error ratio over 6h: {{ $value | humanizePercentage }}."

Two differences from the Workbook table are deliberate choices of the metering team, to be reviewed after three months: the 6× condition is a ticket rather than a page, and both alerts have a short for: that delays them by 2 and 15 minutes. The portal file above follows the Workbook more closely.

Burn rate is simply the recorded error ratio divided by 0.001:

promtool query instant http://localhost:9090 'job:slo_errors_per_request:ratio_rate5m / 0.001'
promtool query instant http://localhost:9090 'job:slo_errors_per_request:ratio_rate1h / 0.001'

{job="meter-api"} => 63.70931249787731 @[1791306787.893]
{job="meter-api"} => 32.36099735003126 @[1791306787.96]

Four minutes into the incident the 5-minute window burns at 63.7×, the 1-hour window at 32.4×. Both are above 14.4, so the fast-burn alert is on its way. All four windows at once, labeled with label_replace, two and a half minutes later:

label_replace({__name__=~"job:slo_errors_per_request:ratio_rate.*"}, "window", "$1", "__name__", ".*_rate(.+)") / 0.001

{job="meter-api", window="5m"} => 81.78246894964485 @[1791306945.873]
{job="meter-api", window="30m"} => 42.66863068151602 @[1791306945.873]
{job="meter-api", window="1h"} => 42.66863068151602 @[1791306945.873]
{job="meter-api", window="6h"} => 42.66863068151602 @[1791306945.873]

Alertmanager sent the page ten seconds later (the page route's group_wait). The start of the notification as the on-call webhook received it:

{
  "receiver": "oncall-pager",
  "status": "firing",
  "alerts": [
    {
      "status": "firing",
      "labels": {
        "alertname": "MeterApiErrorBudgetFastBurn",
        "job": "meter-api",
        "severity": "page",
        "team": "metering"
      },
      "annotations": {
        "description": "At this rate 2% of the monthly budget is gone within an hour. Error ratio over 1h: 2.987%.",
        "summary": "meter-api is burning its 30-day error budget at more than 14.4x"
      },
      "startsAt": "2026-10-06T17:12:33.182Z",
…

The page resolved at 17:31:33 UTC, four minutes after the errors stopped. The slow-burn alert was already pending two minutes into the incident and fired at 17:24:48, after its 15 minutes of for:, but its ticket was held back by an inhibit rule while the page for the same job was open, and only arrived at 17:35:18, after the incident was over: the long windows go on burning for a while. That is expected behavior, and a good reason to say in the ticket's runbook "check whether this is the tail of an incident that has already been handled".

Proving the alert with promtool

A test setup with 40 minutes of history cannot show a real one-hour window. promtool test rules can: it evaluates the rules against synthetic series, minute by minute, in a fraction of a second. This test feeds 600 requests a minute, adds 30 failures a minute from minute 30 (an error ratio of about 4.8%), and states when the fast-burn alert should and should not be firing:

tests/slo_test.yml

rule_files:
  - ../rules/slo-meter-api.yml

evaluation_interval: 1m

tests:
  # meter-api serves 600 requests per minute. After 30 minutes, 30 of them
  # per minute fail with a 503: an error ratio of about 4.8%, a burn rate of about 48.
  - interval: 1m
    input_series:
      - series: 'http_requests_total{job="meter-api",instance="meter-api-01:8080",method="POST",route="/v1/readings",code="202"}'
        values: '0+600x90'
      - series: 'http_requests_total{job="meter-api",instance="meter-api-01:8080",method="POST",route="/v1/readings",code="503"}'
        values: '0x30 30+30x60'

    alert_rule_test:
      # Five minutes into the incident the 1h window is still below 14.4x.
      - eval_time: 35m
        alertname: MeterApiErrorBudgetFastBurn
        exp_alerts: []

      - eval_time: 50m
        alertname: MeterApiErrorBudgetFastBurn
        exp_alerts:
          - exp_labels:
              job: meter-api
              severity: page
              team: metering
            exp_annotations:
              summary: "meter-api is burning its 30-day error budget at more than 14.4x"
              description: "At this rate 2% of the monthly budget is gone within an hour. Error ratio over 1h: 1.961%."

The first version of the test expected "2%" in the description. promtool's failure shows what the rule really produced:

promtool test rules tests/slo_test.yml

  FAILED:
    alertname: MeterApiErrorBudgetFastBurn, time: 50m,
        exp:[
            0:
              Labels:{alertname="MeterApiErrorBudgetFastBurn", job="meter-api", severity="page", team="metering"}
              Annotations:{description="At this rate 2% of the monthly budget is gone within an hour. Error ratio over 1h: 2%.", summary="meter-api is burning its 30-day error budget at more than 14.4x"}
            ],
        got:[
            0:
              Labels:{alertname="MeterApiErrorBudgetFastBurn", job="meter-api", severity="page", team="metering"}
              Annotations:{description="At this rate 2% of the monthly budget is gone within an hour. Error ratio over 1h: 1.961%.", summary="meter-api is burning its 30-day error budget at more than 14.4x"}
            ]

With the description corrected to 1.961%, as in the file above:

  SUCCESS

The test proves the behavior that matters: five minutes into a 4.8% error burst, the one-hour ratio is still below 1.44%, so no page; twenty minutes in, the alert fires. The 1.961% is 600 errors out of 30,600 requests in the hour before minute 50. Keep such tests next to the rules and run them in the pipeline, so a change to a threshold cannot silently break the page.

Low-traffic services

Burn-rate alerts assume enough traffic for a ratio to mean something. The Workbook's example: "if a system receives 10 requests per hour, then a single failed request results in an hourly error rate of 10%. For a 99.9% SLO, this request constitutes a 1,000x burn rate and would page immediately". At night Kestrel's portal gets about 20 requests a minute. Over a full hour that is 1,200 requests, enough to need 18 failures before the one-hour window crosses 1.44%. But the five-minute window holds only 100 requests, so two failures put it at 2%, and a quiet hour after 02:00 can be much quieter than that.

The Workbook's options, from least to most invasive:

Generate synthetic traffic: a probe that logs in and loads the usage page every minute, so there is always a baseline (and so you notice outages nobody else is awake for). The Workbook warns about the downside: if a problem hits real users but not the probe, the probe's successful requests hide it.
Combine small services that share infrastructure into one SLO and one alert.
Change the service so a single failure hurts less, for example with a retry for idempotent reads.
Lower the SLO or lengthen the window, if the service really is used this little.

Many teams also add a minimum count to the condition (page only if at least, say, ten requests failed in the window). That is a common practice rather than a Workbook rule, and it trades some recall for precision at night.