Postmortems

The basics are on Blameless Postmortems and Learning from Mistakes: what a postmortem is for, blameless and just culture, building the incident timeline, listing contributing factors and writing action items. In ITIL terms the meeting is a post-incident review. This page adds the site reliability engineering (SRE) practice around them: when a postmortem is required, what goes in it, the thinking traps that spoil the analysis, and how the lessons get out of the document and into the system. The running example is INC-22047 from Incident Response in Practice: meter-api 2.15.0 leaked database connections and was rolled back after 18 minutes.

When a postmortem is required

A postmortem trigger is a condition, agreed before any incident happens, that makes a postmortem mandatory, such as user-visible impact above a set threshold, any data loss or an on-call engineer having to roll something back. Deciding the triggers in advance takes the judgment out of a moment when people are tired and the team that caused the outage would rather move on. Google's SRE book lists these common triggers:

• User-visible downtime or degradation beyond a certain threshold
• Data loss of any kind
• On-call engineer intervention (release rollback, rerouting of traffic, etc.)
• A resolution time above some threshold
• A monitoring failure (which usually implies manual incident discovery)

Two more are worth adding. Any stakeholder may ask for a postmortem, the book says, and an error budget policy often has its own: the example policy in Google's SRE Workbook requires a postmortem when a single incident consumes more than 20% of the error budget over four weeks. Kestrel's meter-api has a service level objective (SLO) of 99.9% of requests without a server error over a rolling 30 days. Did INC-22047 need a postmortem?

SLO = 0.999
WINDOW_MIN = 30 * 24 * 60          # rolling 30 days = 43,200 minutes
budget_min = (1 - SLO) * WINDOW_MIN

incident = {
    "error_ratio": 0.08,            # about 8% of meter-api requests failed
    "minutes": 18,                  # 19:09 to 19:27
    "user_visible": True,
    "data_lost": False,
    "oncall_intervened": True,      # rollback of CHG-0731
    "minutes_to_resolve": 56,       # 19:09 to 20:05
    "found_by_monitoring": True,
}
spent = incident["error_ratio"] * incident["minutes"] / budget_min
print(f"Budget: {budget_min:.1f} min of full outage per 30 days")
print(f"This incident used {spent:.1%} of it")

triggers = {
    "user-visible degradation": incident["user_visible"],
    "data loss": incident["data_lost"],
    "on-call intervention (rollback, failover)": incident["oncall_intervened"],
    "resolution took over 60 minutes": incident["minutes_to_resolve"] > 60,
    "found by a person, not by monitoring": not incident["found_by_monitoring"],
    "more than 20% of the error budget": spent > 0.20,
}
hits = [name for name, hit in triggers.items() if hit]
print("Postmortem required:", "yes" if hits else "no")
for name in hits:
    print(" -", name)

Budget: 43.2 min of full outage per 30 days
This incident used 3.3% of it
Postmortem required: yes
 - user-visible degradation
 - on-call intervention (rollback, failover)

The incident used only 3.3% of the 30-day budget, far below the policy's 20%, but the rollback alone makes a postmortem mandatory. That is deliberate: an incident that was handled well and cost little is the cheapest one to learn from. Small incidents and near misses deserve a short write-up; the full template is for the ones that meet a trigger.

The postmortem template

A postmortem template is the fixed outline every postmortem in an organization follows: the same headings in the same order, with metadata such as severity, duration and services involved. A template makes the writer cover everything, lets readers find the impact or the action items without reading the whole story, and makes postmortems comparable, so patterns across incidents show up. Google publishes an example postmortem in its SRE book, PagerDuty publishes its template in its public incident response guide, and most incident tools include one. Kestrel's template, filled in for INC-22047:

"Where we got lucky" is the section most templates forget and the one that finds the next incident: luck is a control nobody maintains. Keep the document short and link the raw material (chat transcripts, graphs, logs) instead of pasting it in. Google's SRE Workbook describes a good postmortem that was "written and circulated less than a week after the incident was closed", and explains why speed matters: the people affected are waiting for an explanation, and "the longer you wait, the more they will fill the gap with the products of their imagination."

Why there is rarely one root cause

"What was the root cause?" is the first question many managers ask, and ITIL's root cause analysis uses the term too. The SRE book defines a root cause carefully: "A defect in a software or human system that, if repaired, instills confidence that this event won't happen again in the same way. A given incident might have multiple root causes." The second sentence is the important one. Production failures in complex systems usually need several conditions at once, each harmless on its own. Remove any one of them and the incident is smaller or does not happen.

For INC-22047, the review found these contributing factors (the rule from contributing factors):

• The trigger. meter-api 2.15.0 did not return a database connection to the pool when a write timed out, so every slow write leaked one connection.
• Testing. The release pipeline ran a five-minute load test in acceptance. At that traffic the leak added a handful of connections, far below any limit; only a longer test (see soak tests) would have shown the growth.
• Blast radius. Releases went to both meter-api instances at once, so all traffic got the new version (compare a canary release).
• Shared limits. meter-api, the portal and the billing export share one connection limit on db01, so a leak in one service slowed another.
• Detection of the cause. The database connection alert is a ticket at 80% for five minutes; it fired at 19:24, after both pages, and nothing showed which client held the connections. That gap was already known: the "connections per client" dashboard panel on db01, an action item from the postmortem of INC-31902 on 1 October 2026 (see Blameless Postmortems and Learning from Mistakes), was still open when INC-22047 hit the same shared db01 limit five days later.

Ask "how" instead of "who": not "who approved this release?" but "how could a connection leak pass every test?" The answer to that question is a list of things the organization can change. A single-cause story ("a bug in 2.15.0") is true and useless: bugs will always be written, and the postmortem's job is to make the next one smaller. The leak itself becomes a problem record, PRB-0712, which Amara Okafor, Kestrel's problem manager, follows up.

Hindsight, outcomes and counterfactuals

Everyone who reads a postmortem knows how the story ends. That knowledge changes how the decisions in it look, and three thinking traps follow from it. Naming them makes them easier to catch in a draft.

Hindsight bias is the tendency, once you know the outcome, to see events as more predictable than they were at the time. After the review, the rising connection graph looks like an obvious warning; at 19:05, it was one line among dozens, inside its normal range, on a dashboard nobody was watching.

Outcome bias is judging a decision by its result instead of by the information and options available when it was made. Deploying in the evening looks careless because this deploy failed; the hundred evening deploys that went fine were judged sensible. A good decision can have a bad outcome, and a reckless one can turn out fine.

A counterfactual is a statement about what did not happen: "should have", "could have", "failed to", "if only". It describes an imagined, better version of events instead of the real ones, and quietly turns a description into an accusation. "Priya should have rolled back at once" tells you nothing about why the rollback took the time it did. "At 19:15 Priya saw two error alerts and a deploy twelve minutes earlier; she declared at 19:17 so that someone could decide on a rollback while she checked the database" does.

A reviewer can catch many of these phrases mechanically. This check is crude, and it will never replace a careful reader, but it is a good habit before a draft goes out:

import re

# Phrases that usually signal blame, hindsight or a counterfactual
WARNINGS = {
    r"\bshould have\b": "counterfactual: describe what happened instead",
    r"\bfailed to\b": "counterfactual: what made it hard to do?",
    r"\bforgot\b": "why was it easy to forget?",
    r"\bcareless\w*": "judges a person, not the system",
    r"\bhuman error\b": "where the analysis should start, not end",
    r"\bobvious\w*": "hindsight: was it obvious at the time?",
}

draft = """\
19:15 Priya should have rolled back immediately.
The pipeline failed to run the load test, and the reviewer forgot to check
the pool setting. Root cause: human error.
19:22 Sanne decided to roll back; the deploy at 19:03 was the only change.
"""

for number, line in enumerate(draft.splitlines(), start=1):
    for pattern, advice in WARNINGS.items():
        for match in re.finditer(pattern, line, flags=re.IGNORECASE):
            print(f"line {number}: '{match.group()}' - {advice}")

line 1: 'should have' - counterfactual: describe what happened instead
line 2: 'failed to' - counterfactual: what made it hard to do?
line 2: 'forgot' - why was it easy to forget?
line 3: 'human error' - where the analysis should start, not end

Line 4 passes: it says what was decided, by whom, and on which evidence. Language matters beyond the document too. Google's Workbook gives the example of a reviewer asking a manager "why aren't you making sure that everyone finishes the training?", which puts the manager on the defensive, and suggests asking instead whether the training should be required before joining the on-call rotation, adding that "escalation is not a sin".

Action items that get done

In the words of Ben Treynor Sloss, quoted in the SRE Workbook: "To our users, a postmortem without subsequent action is indistinguishable from no postmortem." The Workbook lists what good action items have in common:

• Ownership: one owner and a tracking number each. Several owners means none.
• Priority: not all equal, so it is clear which comes first.
• Measurability: a verifiable end state ("alert when any client holds more than 50 connections"), not "improve monitoring".
• Prevention as well as mitigation: items that stop a recurrence, not only items that make it shorter. Items that try to change people ("be more careful") instead of systems rarely work.

Most organizations track the items where the team already works: a problem record, a change, a backlog item. At Kestrel the open items come back every month in the service report to Customer Operations, and every quarter in the service review. A small script over the tracker export catches the two usual failures, items that are not real actions and items that quietly expire:

from datetime import date

today = date(2026, 11, 20)          # the monthly service report
actions = [
    # (kind, what, owner, due, ticket, done)
    ("prevent",  "Fix connection leak in readings writer", "Priya Shah",
     date(2026, 10, 30), "PRB-0712", True),
    ("prevent",  "Two-hour soak test before each release", "Ravi Kumar",
     date(2026, 11, 13), "CHG-0748", False),
    ("detect",   "Alert on db01 connections per client", "Hamid Rahimi",
     date(2026, 11, 27), "REQ-6120", False),
    ("mitigate", "Deploy to meter-api-01, watch, then -02", "Yara Haddad",
     date(2026, 10, 23), "CHG-0739", True),
    ("prevent",  "Be more careful with releases", None, None, None, False),
]

for kind, what, owner, due, ticket, done in actions:
    missing = [name for name, value in
               (("owner", owner), ("due date", due), ("ticket", ticket))
               if value is None]
    if missing:
        status = "NOT AN ACTION: no " + ", no ".join(missing)
    elif done:
        status = "done"
    elif due < today:
        status = f"OVERDUE by {(today - due).days} days"
    else:
        status = f"open, due in {(due - today).days} days"
    print(f"{kind:8}  {what:40}  {status}")

prevent   Fix connection leak in readings writer    done
prevent   Two-hour soak test before each release    OVERDUE by 7 days
detect    Alert on db01 connections per client      open, due in 7 days
mitigate  Deploy to meter-api-01, watch, then -02   done
prevent   Be more careful with releases             NOT AN ACTION: no owner, no due date, no ticket

The overdue soak test is the interesting line. Overdue items are normal; silently overdue items are not. In the review, Ravi explains that the acceptance environment has too little capacity for a two-hour test, and the item gets a new date and a dependency instead of disappearing.

Reviewing and sharing

"An unreviewed postmortem might as well never have existed," the SRE book says. A review meeting closes the open discussions in the document and agrees the action items. The book suggests that senior engineers check a draft with questions like these:

• Was key incident data collected for posterity?
• Are the impact assessments complete?
• Was the root cause sufficiently deep?
• Is the action plan appropriate and are resulting bug fixes at appropriate priority?
• Did we share the outcome with relevant stakeholders?

Invite everyone who took part, including the service desk and the communications lead: the people closest to the work see factors that a team writing alone misses. Then share the result as widely as you can. Google's practices include a postmortem of the month in a newsletter, postmortem reading clubs where old incidents are discussed with newcomers, and a weekly report of all outages. The Workbook's example of a bad postmortem was shared only with the team that wrote it; the good one was shared with all engineers.

Old postmortems are also training material: on-call training often replays one as a role-play (see On-Call). And they are evidence. Three postmortems that name the same contributing factor are a problem waiting to be opened.

Learning from incidents

Learning from incidents is the broader practice of treating incidents as a way to understand how work really gets done: how people noticed trouble, adapted and kept the system running, not only what broke. Its community grew out of resilience engineering and safety science, and it pushes postmortems in three directions:

• Study success as well as failure. Why did Priya declare within five minutes when many engineers debug alone for an hour? Whatever made that easy (clear criteria, a team that thanks people for declaring) is worth protecting.
• Value the story over the score. Counting incidents and mean times says little about why they happen. Interviews with the people involved, done soon and without judgment, often teach more than the metrics.
• Look across incidents. Tag postmortems with the same metadata (service, trigger, contributing-factor types) so that themes show up: "five of the last eight incidents involved a shared database limit" is a capacity and architecture finding no single postmortem would make.

None of this replaces action items. It changes which questions you ask before you write them, and it keeps postmortems from becoming a form that people fill in to close a ticket.