Working Under Pressure

What pressure does to your thinking

It is 18:10, the customer portal is down during a storm, the contact center queue is growing and three managers have joined the call. This is when most people make their worst technical decisions. Not because they lack knowledge, but because stress changes how the brain works: attention narrows, memory for routine steps gets worse, and the urge to do something beats the urge to understand.

Human-factors researchers in aviation call the core of this problem attentional tunneling. In everyday IT language it is tunnel vision: focusing on one hypothesis, one screen or one task for longer than is useful, and as a result missing other signals, failing to consider other explanations or forgetting other tasks. In an incident it looks like this:

• The first theory ("it's the database") becomes the only theory, and every new symptom is read as proof of it.
• Everyone stares at the same dashboard while a second, unrelated alarm goes unnoticed.
• The person fixing the problem forgets to tell anyone what they are doing.
• "One more try" turns into an hour, because stopping feels like admitting defeat.

You cannot switch stress off, but you can build the work so that it matters less. That is what the rest of this page is about: roles, checklists, discipline about changes and rules decided in advance, when everyone was calm.

Clear roles in a major incident

When everyone does everything, nobody keeps the overview. Structured incident response therefore separates coordinating from fixing. Google's site reliability engineering (SRE) book bases its model on the Incident Command System used by emergency services, and PagerDuty's public incident response guide uses similar roles.

The incident commander (IC) is the person who coordinates the response to a major incident: they keep the overall picture, set priorities, assign tasks, make or obtain decisions, and do not do hands-on fixing themselves. Google's SRE book says the commander "holds the high-level state about the incident" and holds every role they have not delegated; PagerDuty describes the IC as the "single source of truth of what is currently happening and what is going to happen". In ITIL-based organizations this role is often called the incident manager. At Kestrel Energy, Sanne de Vries, the incident manager, plays it.

The two best-known role models use different names for similar ideas:

The scribe is the person in an incident who records what happens as it happens: findings, actions, decisions and their times, in the incident channel or document. The scribe frees the commander and the engineers from writing while they think, and their record becomes the starting point of the postmortem.

One rule from the Google SRE book is worth remembering word for word: "The operations team should be the only group modifying the system during an incident." Changes made by well-meaning people outside that group, who happen to have access, are a classic way to make an outage longer and impossible to understand afterward.

The operations lead and communications lead roles, and how an incident is declared and closed, are on Incident Response in Practice.

In the storm outage on the incident management page, Sanne coordinates, Ravi Kumar, Hamid Rahimi and Tomasz Nowak work on the systems, and Ingrid Berg handles major-incident communication. If you are new, you will most often be in the operations group or asked to keep the record. Both are real jobs: a clear timeline is one of the most useful things anyone can produce during an incident.

Checklists

A checklist is a short written list of steps or checks to confirm at a defined moment, so that important items are not skipped when people are busy, tired or interrupted. A checklist does not teach anyone how to do the work; it assumes experts and protects them from lapses of memory and attention.

The best-known evidence comes from surgery. After the World Health Organization's surgical safety checklist was introduced in hospitals in several countries, a study published in the New England Journal of Medicine in 2009 found that the death rate fell from 1.5% to 0.8% and inpatient complications from 11.0% to 7.0%. Atul Gawande, one of its authors, wrote The Checklist Manifesto about why such simple lists help skilled professionals.

In IT, the moments that benefit most are the ones where steps are easy to forget under stress: declaring a major incident, starting a risky change, handing over, closing an incident. A good checklist is short (one screen), uses verbs, and is used at the moment, not read beforehand.

□ Incident ticket opened, priority set, major incident declared
□ Incident commander named and announced in #major-incident
□ Bridge (call) opened; link posted in the ticket
□ Scribe named; timeline started in the incident document
□ Impact stated: which service, which users, since when
□ "What changed?" asked: change schedule and deployments, last 24 h
□ First status update sent (status page, contact center, management)
□ Next update time set and announced
□ Operations people named; everyone else read-only

One change at a time

When the pressure is high, it is tempting to try three fixes at once: restart the service, raise a limit and clear a cache. If the problem goes away, you do not know which change fixed it, so you cannot write it down, cannot tell whether the other two changes did harm, and cannot prevent it next time. If the problem gets worse, you do not know which change to undo.

So the discipline is: one change, announced before you make it, then observe long enough to see the effect, then decide the next step. It feels slow. It is usually faster. A saying often attributed to military special forces puts it this way: "slow is smooth, and smooth is fast". Its origin is not documented, but the idea holds up in every major incident review.

Each of those changes and choices goes into a decision log: a time-stamped record of the decisions taken during an incident, who took them and why, kept in the incident channel or document. It prevents people from re-arguing a settled question, lets someone joining late catch up, and gives the postmortem an honest account of what was known at each moment.

21:52 DECISION (Sanne): no rollback yet – no deployment in the last
      48 h, so a rollback has nothing to undo.
21:58 CHANGE (Ravi): raise login-service DB pool 50 → 80.
      Announced in channel before applying. Watch 10 min.
22:09 RESULT: errors down from 30% to 12%, still climbing slowly.
      Pool raise treats the symptom.
22:11 DECISION (Sanne): Hamid checks for slow queries before any
      further change. Ravi makes no changes in the meantime.
22:24 FINDING (Hamid): one report query holds connections for 40 s;
      started 21:35 by a scheduled job (new since Tuesday).
22:26 DECISION (Sanne): pause the report job; Joost informed.
22:31 CHANGE (Hamid): report job paused.
22:40 RESULT: errors 0.3%, normal. Monitoring for 30 min.

Notice that "do nothing for ten minutes" is also a decision, and that the log records results, not only actions.

Knowing when to stop

A stop rule is a condition agreed before work starts at which you stop the current approach and switch to a fallback, such as rolling back, escalating or calling in more people. Examples: "if the smoke test fails or the error rate is above 1% fifteen minutes after the change, we roll back", or "if we have no confirmed cause after 30 minutes, we fail over to the second data center and continue diagnosis there".

Stop rules matter because under pressure, people overestimate how close they are to a fix. Each "one more try" seems cheap; together they can cost hours. Deciding the limit in advance, while calm, takes the decision away from the moment when judgment is worst. It is the same idea as a timebox, applied to risk.

Typical moments for a stop rule:

• Changes: the back-out criteria in the change record (see working safely on production systems).
• Troubleshooting: a time limit per priority, after which you escalate (see escalating at the right moment).
• Major incidents: a point at which the commander switches from "find the cause" to "restore service any way we can", such as failing over or switching a feature off (see mitigate first, fix later).
• Yourself: a point at which you hand over because you have been working too long to think clearly.

Answering "when will it be fixed?"

Sooner or later someone asks for an estimated time of arrival (ETA) for the fix. If you do not know the cause, you do not know the ETA, and a guessed time causes more damage than an honest "we don't know yet": people plan around it, and when it passes, trust is gone.

Should be fixed in about half an hour, we're nearly there.

We don't have a fix time yet. We know the login service runs out of
database connections; we're testing whether a scheduled report job
causes it. If it does, recovery takes about 10 minutes after we find
it. If it doesn't, we fail over at 22:45. Next update at 22:30,
earlier if anything changes.

The clear answer separates what is known from what is not, gives a range only where it is honest, mentions the fallback and promises the next update at a fixed time. It also gives the manager something useful to pass on. More on writing these messages in status updates, emails and reports.

If you are the engineer and a manager asks you directly, it is fine to say: "Please ask Sanne, she has the overview; I need to stay on this." Shielding the people who fix things from repeated questions is part of the commander's job.

Looking after yourself and each other

Long incidents are physically tiring, and tired people make more mistakes. Practical habits help more than willpower:

• Drink, eat and stand up. In a long incident, the commander or scribe can schedule short breaks for each person, one at a time.
• Rotate. After a few hours, swap people out with a proper handover. Fresh eyes often see what tired ones miss.
• Say what you are about to do before you do it. It gives others a moment to say "wait", and it slows you down just enough to catch your own mistake.
• Keep the tone calm. Panic and blame spread quickly on a call. Short, factual sentences help everyone.
• Say when you are lost. "I don't understand what I'm seeing" is useful information for the commander, not a confession.

Feeling shaky or drained after a bad incident is normal. If incidents keep coming, or the stress does not fade, talk to your manager; see health and sustainable work. Afterward, the incident deserves a review that looks at how the system and the response can improve, not at who is to blame: see blameless postmortems and learning from mistakes.