Incident Management
Restoring normal service as quickly as possible: logging and categorizing incidents, priority from impact and urgency, escalation, major incidents, communication and the review afterward.
What an incident is
An incident is an unplanned interruption of a service, or an unplanned reduction in its quality. At Kestrel Energy, the customer portal returning errors is an incident. So is the portal loading in twelve seconds instead of two, even though it technically still works: quality dropped. A third kind is easy to miss: a failure that has not affected anyone yet. When one of the two database servers behind the billing system fails and the other takes over, no customer notices, but the service has lost its safety net. That is an incident too, and fixing it before the second server fails is exactly the point.
Incidents reach the service desk in many ways: a phone call from a planner whose laptop will not connect, a chat message, a customer complaint forwarded by the contact center, or, ideally, an automatic alert from monitoring before any user notices. The step from a monitoring event to an incident is covered on monitoring and event management.
Restore first, explain later
The purpose of incident management is to minimize the harm that incidents cause by getting normal service back as quickly as possible. Not to find out why it broke. If a reboot brings the portal back in five minutes, the reboot is the right incident response, even if nobody yet understands what went wrong. The investigation into causes is problem management, which usually runs alongside or afterward and often involves other people.
Without a defined way of handling incidents, the same things go wrong in every IT department: calls get lost, two engineers work on the same outage without knowing it, a manager chases the wrong fault because it shouted loudest, and users hear nothing for hours. Incident management exists to make the response predictable.
The incident lifecycle
Tools and organizations name the steps differently, but the shape is nearly always the same:
- Detect and log. Every incident gets a ticket, however small, with who reported it, when, and what they saw. An unlogged incident cannot be tracked, measured or linked to a pattern later.
- Categorize. Which service and which component: "Customer portal > login", "Workplace > laptop > Wi-Fi". Categories route the ticket to the right team and make reports meaningful. Keep the list short; nobody chooses correctly from two hundred categories.
- Prioritize. Decide how important it is, from impact and urgency (next section).
- Diagnose. The service desk tries first, using knowledge articles and known errors. Many incidents end here.
- Escalate. If the first team cannot restore service, the ticket moves to a team with more specialized knowledge (functional escalation) or management is involved (hierarchical escalation). Both are explained on the service desk page.
- Resolve and recover. Apply the fix or a workaround, then check that the service really works again.
- Close. Confirm with the user that the problem is gone, record what was done, and close the ticket. Many tools close automatically after a few days without a reply from the user.
Every ITSM tool supports this flow: ServiceNow, Jira Service Management, TOPdesk and Freshservice are common examples. The tool matters less than whether everyone fills in the ticket in a way the next person can understand.
Impact, urgency and priority
When ten tickets arrive at once, the service desk has to decide which to work on first. Most organizations derive the order from two separate judgments:
- Impact is how much harm the incident causes to the business: how many users or customers are affected, which services, and whether money, safety, legal obligations or reputation are at stake.
- Urgency is how quickly the harm becomes worse or a deadline is missed. A broken monthly report is not urgent on the 3rd of the month; on the last working day it is.
- Priority is the relative importance of the incident, derived from impact and urgency. It decides the order of work and which response and resolution targets apply.
Separating the two makes the decision less personal. A director whose printer fails has a strong feeling of urgency; the matrix asks how many people are affected and how fast it gets worse. Many organizations use a three-by-three matrix with five priorities:
| Impact \ urgency | High | Medium | Low |
|---|---|---|---|
| High | P1 critical | P2 high | P3 medium |
| Medium | P2 high | P3 medium | P4 low |
| Low | P3 medium | P4 low | P5 planning |
Kestrel Energy's targets, for example, are a response within 15 minutes and restoration within 4 hours for P1, down to a response within 3 working days for P5. Such targets are agreed per organization, usually in a service level agreement; ITIL does not prescribe numbers. Engineering teams often also give an incident a severity (SEV-1 to SEV-4) that describes only its impact on users; Incident Response in Practice shows how Kestrel's severities relate to these priorities.
The matrix as a function
The pattern in the table is simple: give high, medium and low the values 1, 2 and 3, add impact and urgency, and subtract one. ITSM tools implement exactly this kind of lookup, so the service desk only chooses impact and urgency and the tool computes the priority:
LEVELS = {"high": 1, "medium": 2, "low": 3}
def priority(impact, urgency):
"""Return "P1" (most important) to "P5" from impact and urgency."""
return f"P{LEVELS[impact] + LEVELS[urgency] - 1}"
print("Impact / urgency high medium low")
for impact in LEVELS:
cells = [priority(impact, urgency) for urgency in LEVELS]
print(f"{impact:<16} {cells[0]:<4} {cells[1]:<6} {cells[2]}")
print()
tickets = [
("INC-4711", "Customer portal down for all customers", "high", "high"),
("INC-4712", "VPN fails for the field engineers", "medium", "high"),
("INC-4713", "Monthly finance report runs late", "medium", "low"),
("INC-4714", "One NOC wall screen flickers", "low", "low"),
]
for number, summary, impact, urgency in tickets:
print(f"{number} {priority(impact, urgency)} {summary}")Impact / urgency high medium low high P1 P2 P3 medium P2 P3 P4 low P3 P4 P5 INC-4711 P1 Customer portal down for all customers INC-4712 P2 VPN fails for the field engineers INC-4713 P4 Monthly finance report runs late INC-4714 P5 One NOC wall screen flickers
Some organizations use four priorities, or a four-by-four matrix. The exact grid matters less than applying it the same way every time, and allowing a manager to override it, with a reason, in the rare case it gives an obviously wrong answer.
Major incidents
A major incident is an incident with such a large impact on the business that the normal procedure is too slow, so it is handled by a separate procedure with more people, shorter timelines and dedicated communication. Each organization defines its own criteria in advance, for example "a customer-facing service is unavailable for more than 50% of users", "safety is at risk" or "a regulator must be informed".
Roles and the war room
One evening during a winter storm, thousands of Kestrel customers open the portal to report power outages and check the outage map. At 18:05 the portal starts returning errors. This is a major incident, and the procedure sets these roles in motion:
- Incident manager (Sanne): owns the incident from start to finish. She coordinates, decides and keeps the timeline; she does not fix servers herself.
- Technical leads (Ravi for the portal, Hamid for the database, Tomasz from the network operations center, NOC): diagnose and restore.
- Communications lead (Ingrid): writes updates for the status page, the contact center and management, so engineers are not interrupted with questions.
- Business representative (Daniel, head of customer service): can make business decisions, such as switching off a feature or redirecting customers to the phone line.
Site reliability engineering calls the same jobs incident commander, operations lead and communications lead; see Incident Response in Practice.
Instead of passing the ticket from team to team, the people who might be needed join one call or chat channel at the same time, often called a war room or bridge. This way of working is called swarming: everyone who can help works on the incident together from the start, instead of waiting for it to be escalated tier by tier. Experts who turn out not to be needed leave again. Swarming is also used for complex everyday tickets, as an alternative to strict escalation tiers.
Communication
During a major incident, silence is read as "nobody is working on it". Good practice is a fixed cadence: a first message within minutes, then an update every 30 minutes even when there is nothing new ("still investigating, next update at 19:00"). Audiences differ:
- Customers see a short, plain status page message on a separate host (status.example.com), so it stays up when the portal is down.
- The contact center gets a script: what to tell callers and what workaround to offer (report outages by phone).
- Management gets impact, expected recovery time and any decisions needed.
- Suppliers, such as the cloud provider, are contacted through their own support channel with the right severity, as agreed in the contract.
The ticket and timeline
The ticket afterward tells the story in a form others can read and review:
Number: INC-4711 Type: Incident (major) Service: Customer portal (portal.example.com) Category: Customer portal > login Impact/urgency High / High Priority: P1 Reported by: Monitoring alert + 214 contact center calls Incident mgr: Sanne de Vries Assigned: Portal team (Ravi), Database team (Hamid), NOC (Tomasz) Linked: PRB-0198 (problem), CHG-8840 (emergency change) Status: Closed Timeline (local time) 18:05 Alert: login error rate 38% on portal web tier 18:09 Service desk logs INC-4711, P1; incident manager paged 18:14 Major incident declared; bridge opened 18:16 Status page: "Customer portal unavailable, we are working on it" 18:31 Login service out of database connections; storm traffic 6x normal 18:40 Business decision: outage map moved to static page (CHG-8840) 18:52 Database connection limit raised, login pods restarted 19:03 Error rate 0.4%; status page: "recovering, monitoring" 19:35 Stable for 30 min; customers confirm logins via contact center 19:50 Status page: "resolved"; incident resolved Next day Problem PRB-0198 opened for root cause; review scheduled
Incident, service request, problem or security incident?
The service desk receives all of these through the same channels, but each is handled differently and measured separately:
| Record | What it is | Goal | Kestrel example |
|---|---|---|---|
| Incident | Something broke or got worse, unplanned | Restore service quickly | "The billing app freezes when I open a contract." |
| Service request | A normal request for something the service already offers | Fulfill it as agreed | "I need access to the outage map editor." |
| Problem | The cause, or possible cause, of incidents | Find and remove the cause | "Why does the billing app freeze every Monday?" |
| Security incident | An incident that threatens confidentiality, integrity or availability of information | Contain, eradicate, recover, preserve evidence | "I entered my password on a fake login page." |
Service requests have their own page, service request management. A security incident is still an incident and usually gets an incident ticket, but it is handled with the security incident response process: the service desk's job is to recognize it, avoid destroying evidence (do not wipe the laptop), and hand it over quickly to the security team.
Measuring incident management
The best-known measure is mean time to restore (MTTR): the average time from the start of an incident, or the moment it was logged, until normal service is back. The definition of start and end must be fixed in advance, or numbers from different teams cannot be compared. The program below computes it from a month of Kestrel tickets:
from datetime import datetime
from statistics import mean, median
# (ticket, priority, opened, service restored)
tickets = [
("INC-5001", "P3", "2026-03-02 08:10", "2026-03-02 11:40"),
("INC-5002", "P2", "2026-03-04 13:00", "2026-03-04 14:15"),
("INC-5003", "P4", "2026-03-09 09:30", "2026-03-10 09:30"),
("INC-5004", "P1", "2026-03-14 09:40", "2026-03-14 13:25"),
("INC-5005", "P3", "2026-03-20 10:00", "2026-03-20 12:30"),
("INC-5006", "P2", "2026-03-27 07:45", "2026-03-27 08:30"),
]
def hours_to_restore(opened, restored):
fmt = "%Y-%m-%d %H:%M"
delta = datetime.strptime(restored, fmt) - datetime.strptime(opened, fmt)
return delta.total_seconds() / 3600
durations = {}
for number, prio, opened, restored in tickets:
hours = hours_to_restore(opened, restored)
durations.setdefault(prio, []).append(hours)
print(f"{number} {prio} {hours:5.2f} h")
all_hours = [h for hours in durations.values() for h in hours]
print(f"\nMTTR, all incidents: {mean(all_hours):.2f} h")
print(f"Median: {median(all_hours):.2f} h")
for prio in sorted(durations):
print(f"MTTR {prio}: {mean(durations[prio]):5.2f} h (n={len(durations[prio])})")INC-5001 P3 3.50 h INC-5002 P2 1.25 h INC-5003 P4 24.00 h INC-5004 P1 3.75 h INC-5005 P3 2.50 h INC-5006 P2 0.75 h MTTR, all incidents: 5.96 h Median: 3.00 h MTTR P1: 3.75 h (n=1) MTTR P2: 1.00 h (n=2) MTTR P3: 3.00 h (n=2) MTTR P4: 24.00 h (n=1)
Two lessons hide in this output. One slow, low-priority ticket pulls the overall average to almost six hours while the median is three, so report per priority and look at the median too. And this example counts calendar hours; agreements often count only business hours for lower priorities, which is worked out on service level management.
Splitting the time into detection, acknowledgment and mitigation shows where it went; see the worked incident INC-22047.
Other common measures:
- Number of incidents per service and per category, as a trend.
- Percentage resolved within target per priority, and the share resolved at first contact by the service desk.
- Reopened incidents: a high number suggests tickets are closed before the fault is really gone.
- User satisfaction with the handling, from a short survey after closure.
Post-incident review and blameless culture
A post-incident review is a structured meeting and written report after a significant incident, in which the people involved reconstruct what happened, what went well, what went badly and what should change. Other names are postmortem, retrospective or after-action review. For the storm outage, Sanne invites the technical leads, Ingrid and Daniel within a few days, while memories are fresh. Typical questions:
- How did we find out, and could we have found out earlier?
- What slowed down diagnosis or recovery?
- Did communication reach the people who needed it?
- What actions will prevent this, or make it smaller next time, and who owns each one?
The review is not the root cause analysis itself; that is opened as a problem record (PRB-0198 in the ticket above). It produces improvement actions, some of which become changes.
Reviews only work in a blameless culture: people can describe their actions and mistakes openly without fear of punishment, because the review looks for the conditions that made a mistake possible and easy, not for a person to blame. If the engineer who raised the wrong connection limit last month expects to be punished, the honest account of how it happened disappears, and so does the chance to fix the review step that should have caught it. Blameless does not mean nobody is accountable: people are accountable for learning and for carrying out the agreed actions.
When a review is mandatory, what goes into the written postmortem and how its action items are tracked is covered on Postmortems.
Common mistakes
Two smaller ones: logging service requests as incidents to get a faster response, which distorts both sets of numbers, and restarting things without writing down what was done, which removes the evidence problem management needs later.