On-Call

What being on call means

Services run around the clock; people do not. An on-call rotation makes people responsible, in turns, for responding to urgent alerts outside (and often during) working hours. The skills page on doing on-call duty well covers the personal side: preparing for the week, what to do when the phone goes off, and the legal rules. This page is about designing the system around those people: who gets paged, when, for what, and how you know whether it is sustainable.

The core unit is the page: a notification that demands a person's attention right now, delivered so that it interrupts (an app notification that overrides do-not-disturb, a text message, a phone call) and repeated or escalated until someone responds. The name comes from pagers, the small radio receivers ("beepers") used before mobile phones. It has nothing to do with memory paging or with the less pager on Linux.

Everything in this page follows from one fact: a page has a cost. It breaks concentration during the day, ruins an evening, and at 03:00 it takes sleep that a person cannot get back. The SRE book reports that a well-written playbook gives roughly a three times faster recovery than "winging it", but no playbook makes a needless page free.

Primary and secondary

Most rotations have two layers. The primary on-call is the person who receives pages first and is expected to respond within the agreed time. The secondary on-call is the backup: paged automatically when the primary does not respond, and available to help when the primary asks. A secondary covers the ordinary failures of a single person: a dead phone battery, a shower, a train through a tunnel, or an incident that needs two pairs of hands.

How fast "respond" must be depends on the service. The SRE book gives typical values: "5 minutes for user-facing or otherwise highly time-critical services, and 30 minutes for less time-sensitive systems". A response time is a promise of the rotation, so it shapes people's lives: five minutes means the laptop goes along to the supermarket. Pick the slowest time the service can tolerate.

Escalation policies and acknowledgment

Paging tools (PagerDuty is the best-known example; there are many others, including on-call features in monitoring suites) hold the schedules and decide who is notified, in which order. An escalation policy is that rule: which person or schedule is paged first, and after how many minutes without a response the page moves to the next level. It is automatic and mechanical, unlike an escalation path, which describes when a responder decides to bring in someone more senior or more specialized.

What stops the escalation is an acknowledgment: the responder confirms in the paging tool that they have seen the page and are working on it. Acknowledging does not mean the problem is solved; it means "someone has this". Kestrel reuses the escalation from its ITSM setup: primary first, the secondary after 10 minutes without acknowledgment, the duty manager after 20 minutes.

In a paging tool, the policy and the schedule behind it look roughly like this (the layout differs per product):

Escalation policy: Portal
  Level 1   Schedule "Portal primary"     notify immediately
  Level 2   Schedule "Portal secondary"   after 10 min without acknowledgment
  Level 3   Duty manager                  after 20 min without acknowledgment

Schedule: Portal primary   (weekly, handoff Monday 09:00 Europe/Amsterdam)
  05-11 Oct   Priya Shah
  12-18 Oct   Sofia Lindqvist
  19-25 Oct   Ravi Kumar

Alertmanager itself has no schedules, escalation or acknowledgment. It decides whether something is a page, groups and deduplicates, and hands it to a receiver; a pagerduty_configs or webhook_configs receiver then passes it on to the paging tool. Keep the integration key out of the configuration file itself:

receivers:
  - name: oncall-pager
    pagerduty_configs:
      - routing_key_file: /etc/alertmanager/secrets/pagerduty-portal.key
        send_resolved: true

Page or ticket?

The most important design decision is what may page at all. A useful rule, also on the skills page: every page should be urgent, need a human, and be something the person can act on. Anything that fails one of those tests becomes a ticket alert: an alert that creates a ticket or a message in a team queue, to be handled during working hours, without interrupting anyone. The SRE Workbook draws the line with the error budget: if a problem will use up the budget "within hours or a few days", page; otherwise a ticket for "the next working day" is enough (see fast and slow burns).

Kestrel's Alertmanager routes on the severity label that the alerting rules set. Its routing tree and two tests, from the test setup described on the alerting page:

amtool config routes show --config.file=alertmanager.yml
amtool config routes test --config.file=alertmanager.yml severity=page job=meter-api alertname=MeterApiHighErrorRate
amtool config routes test --config.file=alertmanager.yml severity=ticket team=billing alertname=BillingExportStale

Routing tree:
.
└── default-route  receiver: ops-tickets
    ├── {severity="page"}  receiver: oncall-pager
    └── {team="billing"}  receiver: billing-team
oncall-pager
billing-team

A page only at night: the billing export

The nightly billing export starts at 02:30 and has an SLO of finishing by 06:00 (see Kestrel's SLOs). Today a failure becomes a ticket for Joost Bakker, who sees it at 08:00, two hours after the deadline. If the run fails at 02:35, a rerun would still finish in time, but only if someone starts it. So a failed export is urgent between 02:00 and 06:00 and not at all urgent at 10:00. Alertmanager can express that with a time interval:

alertmanager.yml (proposed change)

route:
  receiver: ops-tickets
  group_by: [alertname, job]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - matchers:
        - severity="page"
      receiver: oncall-pager
      group_wait: 10s
      repeat_interval: 1h
    # A failed billing export pages the billing on-call, but only at night.
    # continue: true lets the alert also reach the billing-team route below.
    - matchers:
        - alertname="BillingExportFailed"
      receiver: billing-pager
      active_time_intervals: [billing-night]
      continue: true
    - matchers:
        - team="billing"
      receiver: billing-team
      group_by: [alertname]
      repeat_interval: 12h

time_intervals:
  - name: billing-night
    time_intervals:
      - times:
          - start_time: "02:00"
            end_time: "06:00"
        location: Europe/Amsterdam

inhibit_rules:
  - source_matchers:
      - severity="page"
    target_matchers:
      - severity="ticket"
    equal: [job]

receivers:
  - name: ops-tickets
    webhook_configs:
      - url: https://hooks.example.com/kestrel/ops-tickets
  - name: oncall-pager
    webhook_configs:
      - url: https://hooks.example.com/kestrel/oncall-pager
  - name: billing-pager
    webhook_configs:
      - url: https://hooks.example.com/kestrel/billing-pager
  - name: billing-team
    webhook_configs:
      - url: https://hooks.example.com/kestrel/billing-team

Between 02:00 and 06:00 Amsterdam time the billing on-call is paged and the billing team also gets its ticket. Outside that window the billing-pager route is inactive, and thanks to continue: true the alert still reaches the ticket route. Without continue, an inactive route still ends the matching: the alert would be muted at 10:00 and nobody would hear about it at all.

Runbooks, dashboards and access

A page at 03:00 should answer three questions on the phone's lock screen: what is broken for users, how bad is it, and where do I start. That means every paging alert carries:

• A summary that names the user-visible symptom ("meter-api returns 8.2% errors"), not the internal metric.
• A link to a runbook, usually in a runbook_url annotation. Kestrel's MeterApiHighErrorRate alert points to https://wiki.kestrel.example/runbooks/meter-api-errors.
• A link to a dashboard that shows the four golden signals of the service (see dashboards).

Runbooks go stale. The Workbook's advice is that on-call engineers update the playbook with fresh information whenever the corresponding page fires, which makes the page review a natural moment to fix them. And access must work before it is needed: the skills page has the checklist for the start of a shift (test page, VPN, multi-factor authentication, admin accounts).

Rotations and handoffs

Most teams rotate weekly, with the handoff on a weekday morning, so the outgoing and incoming person are both at work and can talk. Weekend handoffs and handoffs at midnight save nobody anything.

How many people a rotation needs follows from a cap on how much of someone's life it may take. The SRE book uses at most 25% of an engineer's time on call, and works out that a rotation with a primary and a secondary on week-long shifts at one site needs at least eight engineers. Kestrel's portal team has five developers available for the rotation (Ravi, Sofia, Kwame, Priya and Bram):

CAP = 0.25          # at most a quarter of someone's time on call

# Week-long shifts; every week needs one primary and one secondary.
for people in (5, 6, 8):
    weeks_on_call = 2                       # out of every `people` weeks
    share = weeks_on_call / people
    note = "ok" if share <= CAP else "too much"
    print(f"{people} people: on call {weeks_on_call} weeks in {people} "
          f"= {share:.0%} of the time ({note})")

needed = 2 / CAP
print(f"Minimum for primary + secondary within the cap: {needed:.0f} people")

5 people: on call 2 weeks in 5 = 40% of the time (too much)
6 people: on call 2 weeks in 6 = 33% of the time (too much)
8 people: on call 2 weeks in 8 = 25% of the time (ok)
Minimum for primary + secondary within the cap: 8 people

Two weeks in five is a lot, and for Kestrel, a Dutch company, it also sits close to the legal limit on on-call days (see below). In the United States there is no legal limit on on-call days; in the United Kingdom there is none either, but the 48-hour average week and 11 hours of daily rest still apply. Kestrel's options are the usual ones: add people from a neighboring team that knows the service, make the secondary a shared role across two teams, use follow-the-sun if there were colleagues in other time zones (there are not), or reduce what needs a night response at all. For now Sanne de Vries has agreed that the NOC acts as secondary at night, with a runbook limited to fail-over and rollback.

A handoff is a conversation plus a written handover note: open incidents, anything fragile, changes planned for the coming week, and the pages of the past week with what was done about them. The Workbook's guidelines for a new rotation put it simply: at the start of each shift, the on-call engineer reads the handoff from the previous shift, and minimizes user impact first, then makes sure the issue is fully addressed.

Learning to be on call

Nobody should get their first page alone. Two practices help.

In shadow on-call a new person receives the same pages as the real on-call engineer for a rotation or two, follows along, asks questions and practices on real problems, while responsibility stays with the experienced person. A common next step is the reverse: the new person responds and the experienced one watches. Emeka Obi, who joined Kestrel's NOC this year, shadows two of Tomasz Nowak's on-call weeks before he gets his own.

A Wheel of Misfortune is a role-play exercise from Google SRE (the SRE book describes it as disaster role playing): one person acts as game master and replays a past incident, the "victim" acts as on-call engineer and says what they would check and do, and the game master answers with what the dashboards and logs showed at the time. It is cheap, safe and surprisingly realistic. The SRE book also warns against the opposite problem, operational underload: engineers who are on call too rarely lose touch with production, so every member should be on call at least once or twice a quarter.

Rest and compensation

On-call is work, even when the phone stays silent: people plan their evenings and weekends around it. The skills page summarizes the legal on-call rules per region and has a roster checker.

Practices that keep a rotation humane:

• Rest after a night call-out. Someone who worked from 02:30 to 04:00 starts later the next day, without having to ask. Tired people make the production changes that cause the next incident.
• Swaps are normal. Life happens; swapping a shift should be one message, not a negotiation. Check the swap against the legal limits.
• Compensation that is clear, agreed in advance and paid for being available as well as for call-outs.
• The right to be unreachable when not on call (see right to disconnect). A rotation that quietly expects everyone to answer anyway is not a rotation.
• Load is a team problem. If pages keep someone up several nights a week, the alerts or the service need fixing, not the person's resilience (see burnout).

Measuring the load

What is not measured gets worse. The SRE book sets a clear upper bound. Dealing with one incident properly (root-cause analysis, remediation, and follow-up such as the postmortem) takes about six hours on average, so a 12-hour shift has room for at most two incidents. The pages should be spread out, "with a likely median value of 0" per day: an alert that pages every day is a problem in itself.

Alerting reviews a month of pages per alert; this review looks at one person's week, shift by shift. Kwame's week as portal primary, 21 to 27 September 2026, as exported from the paging tool and annotated by him in the handoff:

from collections import Counter

# Pages in Kestrel's portal rotation, 21-27 September 2026: (day, time, alert, acted on?)
# Shifts: day 08:00-20:00, night 20:00-08:00 (after midnight = previous night)
pages = [
    ("Mon", "09:41", "PortalErrorBudgetBurn", True),
    ("Mon", "23:12", "Web01DiskWarning", False),
    ("Tue", "02:34", "BillingExportSlow", False),
    ("Tue", "14:27", "PortalErrorBudgetBurn", True),
    ("Wed", "02:36", "BillingExportSlow", False),
    ("Thu", "11:10", "PortalLoginFailures", True),
    ("Thu", "21:40", "PortalErrorBudgetBurn", True),
    ("Fri", "02:31", "BillingExportSlow", False),
    ("Sun", "02:33", "BillingExportSlow", False),
]
DAYS = ["Mon", "Tue", "Wed", "Thu", "Fri", "Sat", "Sun"]


def shift(day, time):
    hour = int(time[:2])
    if 8 <= hour < 20:
        return f"{day} day"
    if hour < 8:
        day = DAYS[DAYS.index(day) - 1]
    return f"{day} night"


per_shift = Counter(shift(day, time) for day, time, _, _ in pages)
acted = sum(1 for page in pages if page[3])
nights = sum(n for name, n in per_shift.items() if name.endswith("night"))

print(f"Pages: {len(pages)}, acted on: {acted} ({acted / len(pages):.0%})")
print(f"Night pages: {nights}")
print(f"Shifts without a page: {14 - len(per_shift)} of 14")
print(f"Most pages in one shift: {max(per_shift.values())}")
noise = Counter(alert for *_, alert, ok in pages if not ok)
for alert, count in noise.most_common():
    print(f"Not acted on: {alert} x{count}")

Pages: 9, acted on: 4 (44%)
Night pages: 6
Shifts without a page: 7 of 14
Most pages in one shift: 2
Not acted on: BillingExportSlow x4
Not acted on: Web01DiskWarning x1

No shift had more than two pages, so the week stayed within the book's bound. The numbers still tell a clear story: more than half of the pages were not acted on, and four of the six night pages came from one alert. BillingExportSlow fires every night the export runs a few minutes longer than usual, which is normal when many customers have new contracts. It was never actionable for the portal on-call; the export belongs to Joost's team. At the weekly page review the team turns it into a ticket for billing, and adds the night-only BillingExportFailed page shown above for the case that really matters. The disk warning becomes a ticket too.

Useful numbers to track every week, per rotation:

• Pages per shift, and the share at night.
• The share of pages that were acted on (the precision of the paging alerts).
• Time to acknowledge, and how often a page escalated.
• The noisiest alerts, with an owner to fix or remove each.