Blameless Postmortems and Learning from Mistakes

What a postmortem is for

A postmortem is a written analysis made after an incident, describing what happened, its impact, how it was handled, which factors contributed to it and which actions will prevent it or reduce its effect next time. The name comes from medicine; many teams prefer "incident review" or "post-incident review". In ITIL terms it is the report of a post-incident review, explained on the incident management page. This page is about writing one well, and about the attitude that makes it work. A fixed outline for the document itself is shown under the postmortem template.

The purpose is learning. An outage has already cost time, money and trust; the postmortem is how the organization gets something back for that price. If nothing changes afterward, the same incident will happen again, often to someone else.

Blameless, and why it works

Google's site reliability engineering (SRE) book puts it this way: "For a postmortem to be truly blameless, it must focus on identifying the contributing causes of the incident without indicting any individual or team for bad or inappropriate behavior." And: "A blamelessly written postmortem assumes that everyone involved in an incident had good intentions and did the right thing with the information they had."

The reason is practical, not just kind. If people expect punishment, they stop telling the whole story: they leave out the shortcut they took, the warning they ignored, the command they were not sure about. Those are exactly the details that show how to prevent the next incident. The same book notes that blameless culture "originated in the healthcare and avionics industries where mistakes can be fatal", and sums up the logic: "You can't 'fix' people, but you can fix systems and processes to better support people making the right choices."

Blameless does not mean anything goes. The idea of a just culture, developed in aviation and healthcare safety and associated with the safety researcher Sidney Dekker, separates honest mistakes, which are treated as information about the system, from reckless or deliberately harmful behavior, which still has consequences. In a just culture people are held accountable for telling what happened and for helping to improve the system, not punished for errors any reasonable person could have made in that situation. A widely read 2012 blog post by John Allspaw, then at Etsy, brought these ideas into IT operations.

When to write one

Teams agree their own triggers in advance, so that nobody has to decide after the fact whether an incident "deserves" a review. Typical triggers:

• every major incident, and incidents above a set impact (for example a P1 or P2, or customer impact longer than 30 minutes);
• any loss or exposure of data, including a possible personal data breach;
• a failed change that needed a back-out, or an emergency change;
• an incident someone involved asks to review, because they think there is something to learn.

Site reliability teams write these down as postmortem triggers, including error-budget thresholds.

Do not forget the near misses. A near miss is an event that could have caused an incident but did not, through luck or a last-minute catch: the backup that turned out to be empty when you happened to check, the firewall rule that a reviewer caught minutes before the change. Near misses show the same weaknesses as real incidents, without the damage, so they are cheap lessons. A short write-up of a near miss is often enough.

Building the timeline

The incident timeline is the time-stamped sequence of events in an incident, from the first trigger through detection, response and recovery, including what people knew and decided at each moment. It is the backbone of the postmortem: before you can ask why something happened, everyone has to agree on what happened, and in which order.

Build it from evidence, not memory: monitoring graphs, alerts, log files, the ticket, the chat channel and the decision log kept during the incident (see working under pressure). Use one time zone, and record what people believed at the time, not only what turned out to be true. "21:58 the on-call engineer raises the pool size, believing the limit is too low" explains a decision that looks odd in hindsight.

A timeline with labeled events also gives you the durations that show where the time went. This is the login incident from 1 October, INC-31902:

from datetime import datetime

timeline = """\
2026-10-01 21:35 START    report job starts holding DB connections
2026-10-01 21:40 IMPACT   portal login errors begin
2026-10-01 21:47 DETECT   alert: login error rate above 5%
2026-10-01 21:52 RESPOND  INC-31902 opened, P2, on-call engineer on it
2026-10-01 22:31 MITIGATE report job paused
2026-10-01 22:40 RECOVER  error rate back to normal
"""

events = {}
for line in timeline.splitlines():
    day, time, kind, text = line.split(maxsplit=3)
    events[kind] = datetime.fromisoformat(f"{day} {time}")


def minutes(start, end):
    return int((events[end] - events[start]).total_seconds() // 60)


print(f"Time to detect:   {minutes('IMPACT', 'DETECT'):>3} min")
print(f"Time to respond:  {minutes('DETECT', 'RESPOND'):>3} min")
print(f"Time to mitigate: {minutes('RESPOND', 'MITIGATE'):>3} min")
print(f"Customer impact:  {minutes('IMPACT', 'RECOVER'):>3} min")

Time to detect:     7 min
Time to respond:    5 min
Time to mitigate:  39 min
Customer impact:   60 min

Detection and response were fast; most of the hour went into diagnosis. That points the review toward questions such as "why did it take 39 minutes to stop the report job?" rather than toward the alerting.

Contributing factors instead of a culprit

A contributing factor is a condition or event that helped an incident happen, made it worse or delayed recovery, without being enough to cause it on its own. Complex systems rarely fail for one reason. Looking for "the" root cause tempts people to stop at the first convenient answer, often a person; listing contributing factors keeps the investigation open.

For INC-31902, the factors might be:

• A new scheduled report ran a slow query against the same database as the login service.
• The report job was added as a standard change, so nobody checked its load on the database.
• The login service and the report shared one connection pool limit.
• No dashboard showed which job held the connections, so diagnosis relied on Hamid's experience.
• The first fix (raising the pool size) treated the symptom and was tried before the cause was clear.

The shared limit did not stay a one-off: five days later, on 6 October, INC-22047 hit the same shared connection limit on db01, while the dashboard item below was still open (see Postmortems).

Techniques such as the five whys and Ishikawa diagrams, explained under root cause analysis, help to dig deeper. Use them with one caution: a single chain of "whys" follows one path and can end at a person. Ask "how" questions instead of "who" questions: not "who added the job?" but "how could a job with this load reach production without anyone noticing?" Hindsight and "should have" sentences are the traps to watch for; see hindsight, outcomes and counterfactuals.

The wording of the postmortem matters as much as the analysis:

Joost added a badly written report job without testing it,
which caused the outage. Joost must test his jobs in future.

The report job was registered as a standard change; the
standard change procedure does not include a load check
against shared databases, and acceptance has 5% of production
data, so the query looked fast there.

The second version explains why a reasonable engineer did what he did, and it points at two things the organization can change. The first version points at one person and changes nothing, except making Joost less likely to be open next time.

Action items that actually happen

An action item is a specific improvement agreed in a postmortem, with one owner, a due date and a ticket, so that it can be tracked until it is done. A good set of action items addresses different contributing factors and usually mixes three kinds: prevent (stop it happening), detect (notice it sooner) and mitigate (make it smaller or shorter).

- Be more careful with report jobs
- Improve monitoring
- Look into connection pools

PREVENT   Add a load check against shared databases to the
          standard change procedure for scheduled jobs.
          Owner: Yara Haddad · due 26 Oct · CHG-7760
PREVENT   Separate connection pool for reporting queries.
          Owner: Hamid Rahimi · due 10 Nov · PRB-0709
DETECT    Dashboard panel "connections per client" on the
          customer database (db01). Owner: Hamid · due 16 Oct
          · REQ-4410
MITIGATE  Runbook step: pause non-critical scheduled jobs when
          login errors exceed 5%. Owner: Ravi Kumar · due 16 Oct
          · KB-0455

Each specific item could be checked off by someone other than its owner. Track them where the team already tracks work (problem records, changes, the continual improvement register; see continual improvement), and review open items regularly. A postmortem whose action items quietly expire is worse than none: it teaches the team that reviews are theater. How SRE teams prioritize items and catch overdue ones is on action items that get done.

Owning your own mistakes

Sooner or later you will cause an incident yourself: the wrong server, a typo in a firewall rule, a deleted file. Everyone who has worked in IT for a while has a story like this. What matters most is the next ten minutes.

1. Say it immediately, in the incident channel or to your lead. The people fixing it need to know what changed, and every minute you stay silent, they look in the wrong places.
2. Stick to facts: what you did, when, on which system, and what you have done since.
3. Do not try to fix it quietly with more untested changes. Offer to help, and follow the incident lead's direction.
4. Afterward, take part in the review openly. Your account of why it made sense at the time is the most valuable part of the postmortem.

Hmm, not sure, might be related to something I did earlier?
Looking into it.

@Sanne INC-31944: I think I caused this. At 14:02 I applied
the new firewall rule (CHG-7748) to fw-02 instead of fw-03.
VPN logins through fw-02 fail since then. I have not changed
anything else. The previous rule set is saved; restoring it
takes about 2 minutes. Shall I, or do you want someone else
to do it?

The clear message probably saves half an hour of diagnosis. In a healthy team, it earns respect rather than blame. If your team reacts to such messages with punishment, that is a problem with the team's culture, and worth raising with your manager, because it will make every future incident longer.

Learning as a team

Postmortems help most when they are read beyond the team that wrote them. Some habits that spread the learning:

• Hold the review meeting within a few days, while memories are fresh, with someone facilitating who was not deeply involved.
• Share the document widely, with personal data and security details removed where needed. Other teams often have the same weakness.
• Read other teams' postmortems, and published ones from other companies. For a starter they are an excellent way to learn how real systems fail.
• Look for patterns across reviews: the same contributing factor in three incidents is a problem record waiting to be opened (see problem management).
• Celebrate good catches: near misses reported and back-outs done on time are signs of a working safety culture, not of failure.