Incident Response in Practice

This page is about the mechanics of a technical incident in the style of site reliability engineering (SRE): the first minutes after a page, who does what, and how the incident ends. It builds on three existing pages and does not repeat them: Incident Management (the ITIL practice: logging, priority, major incidents), Working Under Pressure (roles compared, checklists, decision logs, stop rules) and Status Updates, Emails and Reports (writing the updates themselves). Security incidents follow their own process, described on Incident Response.

Declaring an incident

Most outages start small: one alert, one odd graph, one customer call. The on-call engineer looks into it, and the question is when "looking into it" becomes "an incident". Declaring an incident is the explicit, announced decision that a problem will be handled with the incident process: it gets an incident record and a severity, someone takes command, a channel is opened and the stakeholders are told. Before the declaration, one person is troubleshooting; after it, an organization is responding.

Google's SRE book gives three simple tests. If any of them is true, declare:

• Do you need to involve a second team in fixing the problem?
• Is the outage visible to customers?
• Is the issue unsolved even after an hour's concentrated analysis?

The same book adds that it is better to declare early, find a simple fix and close the incident than to start the framework hours into a growing problem. The SRE Workbook puts it in four words: "Declare incidents early and often." A declaration that turns out to be unnecessary costs ten minutes and an apologetic message. A declaration that comes an hour late costs an hour in which nobody told the service desk, nobody checked the change schedule and nobody else was allowed to help.

Declaring is a short, fixed routine, the same every time:

1. Open the incident record (at Kestrel Energy an INC ticket) and set a severity.
2. Open the incident channel (#inc-22047) and the incident document, and post the link in the ticket.
3. Say who is in command. If nobody else is there yet, that is you: "the person who declares the incident typically steps into the IC role", in the Workbook's words.
4. Page the people the severity calls for, through the paging tool described on On-Call.

Severity levels

Incident severity is a level, usually SEV-1 (worst) to SEV-4 or SEV-5, that classifies an incident by its impact on the service and its users, and that decides who is paged, how often updates go out and whether a postmortem is mandatory. Do not confuse it with the syslog severity level of a single log message.

There is no standard scale. PagerDuty's public incident response guide uses SEV-1 to SEV-5, with "lower numbered severities being more urgent" and anything above SEV-3 counted as a major incident; Atlassian describes three levels. What matters is that your levels are written down with examples, so two engineers classify the same outage the same way. Kestrel Energy uses four:

Severity is the engineering view of impact. ITIL's priority is impact combined with urgency, used to order the queue of all incidents. Many organizations map them one to one (SEV-1 is a P1 major incident); that is a local convention, not a standard. At Kestrel, SEV-1 and SEV-2 incidents follow the major incident procedure: a SEV-1 is a P1, a SEV-2 usually a P2. Written as code, Kestrel's rules look like this:

def severity(core_broken, share_affected, data_at_risk, redundancy_lost):
    """Kestrel's scale: SEV-1 is the worst. When in doubt, pick the higher one."""
    if data_at_risk or (core_broken and share_affected >= 0.5):
        return "SEV-1"
    if core_broken and share_affected >= 0.05:
        return "SEV-2"
    if share_affected > 0 or redundancy_lost:
        return "SEV-3"
    return "SEV-4"

situations = [
    # what happened,                        core broken, share, data, redundancy
    ("Portal login fails for everyone",       True,  1.00, False, False),
    ("10% of meter readings rejected",        True,  0.10, False, False),
    ("Invoice PDF slow for some customers",   False, 0.02, False, False),
    ("web02 down, web01 carries all traffic", False, 0.00, False, True),
    ("Billing export wrote wrong amounts",    False, 0.01, True,  False),
    ("Typo on the outage map page",           False, 0.00, False, False),
]
for text, *facts in situations:
    print(f"{severity(*facts)}  {text}")

SEV-1  Portal login fails for everyone
SEV-2  10% of meter readings rejected
SEV-3  Invoice PDF slow for some customers
SEV-3  web02 down, web01 carries all traffic
SEV-1  Billing export wrote wrong amounts
SEV-4  Typo on the outage map page

Note the billing export: only 1% of customers are affected, but wrong amounts on invoices put data and money at risk, so it is a SEV-1. Severity can change during an incident. Raise it as soon as the impact grows; lower it only when the impact has really shrunk, and say so in the channel.

Roles: command, operations, communication

Most SRE incident processes, Google's included, are adapted from the Incident Command System (ICS): the standard structure that emergency services such as fire departments use to coordinate a response, with one commander, clear roles and a span of control that grows or shrinks with the incident. IT borrowed the principle, not the paperwork: a small number of named roles, one person in charge, and a structure that works the same for a 20-minute outage and a two-day crisis.

The incident commander (IC) coordinates and decides; the comparison of Google's and PagerDuty's role names, and the scribe, are explained on Working Under Pressure. The SRE Workbook names three main roles, and the other two deserve their own definitions:

• The operations lead (OL, or ops lead) leads the people who investigate and change the system during an incident. They decide which commands are run, by whom, and report findings to the incident commander. Only the operations group changes production.
• The communications lead (CL) is the public face of the response: they send the regular updates to stakeholders, the service desk and customers, keep the status page current and answer the questions that would otherwise interrupt the engineers.

"It is not intended that every role be filled by a different person", as PagerDuty's guide says. At 03:00 the on-call engineer may be IC, OL and CL at once for the first ten minutes. The point is that the roles exist, so that the first thing the IC does when help arrives is hand them out. By default, the IC holds every role that has not been delegated.

The three Cs

The SRE Workbook sums up what the roles are for with the three Cs of incident management: coordinate the response effort, communicate between responders, within the organization and to the outside world, and maintain control over the response. When an incident response goes wrong, the Workbook notes, the cause is usually in one of these three areas: people working on the same thing without knowing it (coordination), managers learning about the outage from customers (communication), or someone "freelancing" a change that makes it worse (control).

At Kestrel, the incident commander for major incidents is Sanne de Vries, the incident manager, or the duty manager on the evening rota. Ingrid Berg is the communications lead for major incidents; the on-call engineer who first responded usually becomes the operations lead, and specialists such as Hamid Rahimi (database) join the operations group when asked.

Mitigate first, fix later

In incident response, mitigation is any action that reduces or stops the impact on users before the cause is fully understood or repaired: rolling back a release, failing over, draining a broken node, switching a feature off, adding capacity or shedding low-priority load. A mitigation can be crude and temporary; the real fix comes later, often through a problem record (see problem management).

The SRE Workbook's order of work for an active incident is: assess the impact, mitigate the impact, perform a root-cause analysis, and after the incident fix the cause and write a postmortem. Its argument is simple: "customers do not care whether or not you fully understand what caused an outage. What they want is to stop receiving errors." The Workbook calls the blunt, always-available options generic mitigations, and recommends building them before you need them:

Rolling back a change that "cannot be the cause" is often the fastest way to prove that it was, or to rule it out. Decide in advance when you stop diagnosing and mitigate instead: that is a stop rule.

A worked incident: INC-22047

Tuesday 6 October 2026, a normal evening. At 19:03 the deployment pipeline releases meter-api 2.15.0 (CHG-0731, a standard change). Priya Shah, a developer on the portal team, is primary on call for the portal and meter-api this week. What follows is the record her colleagues reconstructed afterward; times are local (CEST).

• 19:09 meter-api starts rejecting about 10% of smart-meter readings with HTTP 503 and answers 3.5 times slower. The readings queue grows; the portal, which reads usage from the same database, slows down.
• 19:12 Priya's phone goes off: the fast-burn alert on meter-api's error budget. A second page for the error rate follows at 19:14.
• 19:15 Priya acknowledges, opens the meter-api dashboard and checks what is paging.

Alertmanager can tell her which page-level alerts are active right now. This is the test setup's recording of that moment:

amtool alert query --alertmanager.url=http://localhost:9093 severity=page

Alertname                    Starts At                Summary                                                          State
MeterApiErrorBudgetFastBurn  2026-10-06 17:12:33 UTC  meter-api is burning its 30-day error budget at more than 14.4x  active
MeterApiHighErrorRate        2026-10-06 17:14:25 UTC  meter-api returns 8.157% errors                                  active  

amtool prints UTC: 17:12 UTC is 19:12 in Amsterdam in summer. Two pages, both meter-api, both about errors. Her first questions are the ones every responder asks: what changed, where does it fail, and how much room is left? A few PromQL queries answer them:

# What changed? Which meter-api versions are running right now?
count by (version) (kestrel_build_info{job="meter-api"})

# Where does it fail? Share of 5xx responses per route over 5 minutes
sum by (route) (rate(http_requests_total{job="meter-api", code=~"5.."}[5m]))
  / sum by (route) (rate(http_requests_total{job="meter-api"}[5m]))

# How much room is left? Minutes until db01 reaches max_connections
(pg_connections_max - pg_connections_in_use) / deriv(pg_connections_in_use[10m]) / 60

Readings are failing, which customers will see as missing usage in the portal; the database connections are climbing since 19:03, and at 19:16 the last query says db01 reaches its limit in about ten minutes (the recording is on Capacity Planning and Performance); the only change is the release. That meets two of the three declaration tests, so she declares:

• 19:17 Priya declares INC-22047, SEV-2, opens #inc-22047 and pages the incident commander on duty. She posts: "meter-api rejects ~10% of readings since 19:09, db01 connections rising since the 2.15.0 deploy at 19:03. I'm IC until Sanne joins."
• 19:21 Sanne de Vries joins from home and takes command after a one-minute handover. Priya becomes operations lead; Ingrid Berg is paged as communications lead. Sanne asks Hamid Rahimi to join for db01.
• 19:22 Decision (Sanne): roll back CHG-0731 to 2.14.3 now; diagnose afterward. Hamid confirms that the connections are still climbing and db01 will refuse new ones within minutes.
• 19:25 Ingrid posts the first status page update.
• 19:27 Rollback complete. Errors stop within a minute; db01 connections drop back to their normal level; the queue starts to drain.
• 19:32 and 19:34 Both pages resolve on their own.
• 20:05 After half an hour without errors and with the queue back to its usual size, Sanne resolves the incident.

The alert times come from Alertmanager in UTC, the rest from the chat in local time. The scribe's first job is to put everything in one time zone; the second is to measure where the time went:

from datetime import datetime
from zoneinfo import ZoneInfo

local = ZoneInfo("Europe/Amsterdam")
utc = ZoneInfo("UTC")

# Alert times come from Alertmanager (UTC); the rest from the channel (local).
events = [
    ("impact",    "2026-10-06 17:09", utc,   "meter-api starts failing 10% of readings"),
    ("page",      "2026-10-06 17:12", utc,   "page: error budget fast burn"),
    ("ack",       "2026-10-06 19:15", local, "Priya acknowledges"),
    ("declared",  "2026-10-06 19:17", local, "INC-22047 declared, SEV-2"),
    ("mitigated", "2026-10-06 17:27", utc,   "rollback of CHG-0731 complete"),
    ("resolved",  "2026-10-06 20:05", local, "Sanne resolves after 30 min watch"),
]
t = {}
for name, stamp, zone, text in events:
    t[name] = datetime.fromisoformat(stamp).replace(tzinfo=zone).astimezone(local)
    print(f"{t[name]:%H:%M %Z}  {text}")

def minutes(a, b):
    return int((t[b] - t[a]).total_seconds() // 60)

print()
print(f"time to detect:      {minutes('impact', 'page'):3} min")
print(f"time to acknowledge: {minutes('page', 'ack'):3} min")
print(f"time to declare:     {minutes('page', 'declared'):3} min")
print(f"time to mitigate:    {minutes('impact', 'mitigated'):3} min")
print(f"incident open:       {minutes('declared', 'resolved'):3} min")

19:09 CEST  meter-api starts failing 10% of readings
19:12 CEST  page: error budget fast burn
19:15 CEST  Priya acknowledges
19:17 CEST  INC-22047 declared, SEV-2
19:27 CEST  rollback of CHG-0731 complete
20:05 CEST  Sanne resolves after 30 min watch

time to detect:        3 min
time to acknowledge:   3 min
time to declare:       5 min
time to mitigate:     18 min
incident open:        48 min

Three things went right and are worth copying. The page came from a symptom (the error budget burning), not from a CPU graph. Priya declared within five minutes of the page instead of debugging alone. And Sanne chose the generic mitigation, a rollback, while the cause was still unknown; with db01 minutes away from its connection limit, waiting for a diagnosis was a bad bet. Why the release leaked connections is a question for the postmortem.

Communication and status pages

The communications lead serves several audiences: the people in the incident channel, the service desk and contact center, management, and customers. How to write each update is covered on Status Updates, Emails and Reports; the cadence rule is the same everywhere: give the time of the next update, and keep it even when nothing has changed.

A status page is a public or internal web page that shows the current state of a service and its components, with time-stamped incident updates that people can read or subscribe to, so they do not have to call or email to find out. For customers it replaces dozens of identical calls to the contact center; for other teams it answers "is it us or them?". Cloud providers publish their own (see cloud monitoring services).

Many status page products use the same sequence of states: investigating, identified, monitoring, resolved. Ingrid's updates for INC-22047 (illustrative):

Delayed usage data in My Kestrel

Resolved · 20:05
Usage data is up to date again. No meter readings were lost.

Monitoring · 19:40
A fix is in place. Usage from this evening is being added now and
should be complete within 30 minutes. Next update by 20:10.

Investigating · 19:25
Since about 19:10, usage shown in My Kestrel may be up to an hour
behind and pages may load slowly. Your meter readings are kept and
will appear later. Your bills are not affected. Next update by 19:55.

Some habits that make a status page trustworthy:

• Impact, not cause. "Usage may be up to an hour behind" helps a customer; "connection pool exhaustion after a release" does not, and may be wrong.
• Post early. The first update can say little more than "we know, we are on it". Silence makes people assume nothing is happening.
• Host it elsewhere. A status page on the same servers, DNS or cloud account as the service goes down with it.
• A person decides what it says. Monitoring can suggest a state, but an automatic "all systems operational" next to a flood of complaints destroys trust faster than no page at all (compare the watermelon effect).

When the clock is legal

INC-22047 involved no personal data and no attack, so no regulator had to be told. That is not always so.

The practical consequence for responders: the incident process needs one step that asks "could this be a security incident or a data breach?" and, if the answer is "maybe", pulls in the security team and starts the clock. The details are on Incident Response in the Security topic.

Closing out

Incidents rarely end with a bang. Errors fade, graphs return to normal, people drift away from the channel. An explicit ending matters for the same reason an explicit start does: if nobody says it is over, half the organization assumes it is still going on. Before the incident commander resolves it:

• Confirm recovery with data. The service level indicator is back to normal and has stayed there for an agreed time (Kestrel uses 30 minutes); queues have drained.
• Undo the temporary measures, or own them. Silences, disabled alerts, scaled-up servers and feature flags switched off during the incident get an owner and an end date, or are removed.
• Send the final update on every channel that got the earlier ones, and set the status page to resolved.
• Complete the record. Timeline, decisions, the incident document and the ticket; the scribe's notes are the raw material for the review.
• Decide on the postmortem. Name an owner and a date. INC-22047 needed a rollback by the on-call engineer, which is one of Kestrel's triggers (see Postmortems).
• Look after the people. Thank them by name. Make sure whoever was paged at night can start late or take the morning off, as the rules on working hours and plain decency require. A long incident needs a proper handover instead of one exhausted person seeing it through.

Resolved is not the same as closed. In ITIL terms the incident is resolved when the service is restored; the underlying cause (here, a connection leak in the readings writer) becomes a problem record, so the incident can be closed while the fix follows its own path. The mean time to restore counts to restoration, not to the end of the postmortem.

Practicing before it counts

Incident response is a skill, and skills fade without practice. The SRE Workbook recommends training responders in a simple framework (who may delegate, mitigation first, the three roles) and then drilling it: invent an outage, or replay one from an old postmortem, and let the team respond with the real tools. You can also treat a minor problem as a major one on purpose, to practice the process with low stakes. The role-play format for on-call training is described on On-Call.

Practice also exposes what is missing before a real incident does: the runbook link that points nowhere, the rollback that needs a password nobody on call has, the status page login that expired. Each of those is cheaper to find on a quiet Thursday afternoon than at 19:15 with two pages open.