Problem Management
Finding and removing the causes of incidents: problems, errors and known errors, workarounds, root cause analysis techniques and the known error database.
Why problem management exists
For a month, Kestrel Energy's nightly billing job fails about once a week. Each time, the operations team reruns it in the morning, invoices go out a few hours late, and the incident is closed as resolved. Every individual incident was handled well. Yet the organization keeps paying for the same failure, and nobody owns the question why does this keep happening?
That question is the job of problem management. Its purpose is to reduce how often incidents happen and how much harm they do, by finding their causes, managing workarounds while the cause is still there, and getting the causes removed. Incident management works against the clock; problem management can take days or weeks, because understanding a cause properly takes time.
Problem, error, known error and workaround
Four terms are used precisely in service management, and mixing them up causes real confusion in tickets and reports.
- A problem is the cause, or a possible cause, of one or more incidents. "Why does the billing job fail?" is a problem, even before anyone knows the answer. A problem can be raised after a single serious incident, after a series of similar ones, or before any incident has happened at all.
- An error is a flaw in a component of a service, such as a bug in code, a wrong setting or a weak design, that causes or can cause incidents. The error is the actual defect; the problem is the record and investigation around it.
- A known error is a problem that has been analyzed but not yet resolved. The team understands it well enough to describe its cause or its symptoms and, usually, how to work around it, but the permanent fix has not been made.
- A workaround is a way to reduce or remove the impact of an incident or problem while no full resolution is available, for example rerunning the billing job by hand, or restarting a service every night before it runs out of memory. A workaround keeps the business going; it does not remove the cause.
The three phases
ITIL describes problem management in three phases. They overlap in practice, but each answers a different question.
Problem identification
Which problems do we have? Problems are identified by:
- Trend analysis of incident records: the same symptom on the same component, again and again.
- Major incidents: almost every major incident gets a problem record, even if service was restored.
- Information from others: a supplier announces a defect in a firmware version you use, developers find a design weakness, testers report a bug that was released anyway.
- Monitoring: a disk that fills up a bit faster each week has not caused an incident yet, but will.
The program below does the simplest form of trend analysis: it groups a month of incidents by configuration item (CI, the component the incident was logged against) and symptom, and flags any group with three or more tickets:
from collections import defaultdict
# One month of incidents: (ticket, configuration item, symptom)
incidents = [
("INC-6101", "batch-01", "billing job failed"),
("INC-6102", "laptop-0412", "no Wi-Fi"),
("INC-6107", "portal-web", "slow login"),
("INC-6112", "batch-01", "billing job failed"),
("INC-6118", "printer-2F", "paper jam"),
("INC-6125", "portal-web", "slow login"),
("INC-6131", "batch-01", "billing job failed"),
("INC-6140", "batch-01", "billing job failed"),
("INC-6141", "laptop-0377", "no Wi-Fi"),
]
groups = defaultdict(list)
for ticket, ci, symptom in incidents:
groups[(ci, symptom)].append(ticket)
THRESHOLD = 3 # this many similar incidents in a month: raise a problem
ranked = sorted(groups.items(), key=lambda item: (-len(item[1]), item[0]))
for (ci, symptom), tickets in ranked:
flag = "candidate problem" if len(tickets) >= THRESHOLD else ""
print(f"{len(tickets)} {ci:<12} {symptom:<19} {flag}".rstrip())4 batch-01 billing job failed candidate problem 2 portal-web slow login 1 laptop-0377 no Wi-Fi 1 laptop-0412 no Wi-Fi 1 printer-2F paper jam
The billing job stands out immediately. Notice what the grouping misses: two Wi-Fi complaints on different laptops look unrelated, but if both laptops sit near the same access point, the real CI is the access point. That is why good categorization and an accurate record of which CIs depend on which matter so much to problem management.
Problem control
What causes it, and how do we live with it meanwhile? Problem control is the analysis itself: collecting evidence, reproducing the fault, applying root cause analysis techniques (below), and documenting workarounds as soon as they are found, so the service desk can use them on the next incident. Problems are prioritized like incidents, by their impact and how often they occur, because no team can investigate everything.
Error control
What do we do about the known errors? Error control manages known errors over time: reassessing them, improving workarounds, and deciding on a permanent solution. That decision is a business decision as well as a technical one. Fixing the cause may require a change to code, configuration or infrastructure, which then goes through change enablement. Sometimes the fix costs more than living with the workaround, and the known error is accepted and stays documented, for example a quirk in a system that will be replaced next year.
Root cause analysis
Root cause analysis (RCA) is the structured search for the underlying cause of a problem: the factor that, if removed, prevents it from happening again, rather than the first fault that is visible. Several techniques exist; most teams use two or three of them. In complex systems an incident usually has several contributing factors rather than one root cause; Postmortems works through an example.
Five whys
The five whys technique asks "why?" repeatedly, each time about the previous answer, until it reaches a cause that can be acted on. Five is a rule of thumb, not a rule. For the billing job, the problem manager Amara works through it with the operations and database teams:
- Why did the billing job fail? It was stopped by its 4-hour timeout.
- Why did it take longer than 4 hours? The query that reads smart meter readings took over 3 hours instead of 40 minutes, on nights with many readings.
- Why was the query so slow? The readings table no longer has its index on customer and reading date, so the database scans the whole table.
- Why is the index missing? A database migration in last month's smart meter release recreated the table without it.
- Why was that not caught before the release? The migration review does not compare indexes, and the test database holds about 1% of the production data, so the slow query was not noticed in testing.
Stopping at "why 1" gives the useless fix "raise the timeout". Stopping at "why 3" gives "recreate the index", which solves this case but lets the next migration do the same. The fifth answer leads to fixes that prevent a whole class of failures: an automatic index comparison in the deployment pipeline and performance tests on realistic data volumes.
Ishikawa (fishbone) diagram
A single chain of whys can lead past other contributing causes. An Ishikawa diagram, also called a fishbone or cause-and-effect diagram, puts the effect at the head of a "fish" and groups possible causes along bones by category. It is mainly a brainstorming tool: it makes a group consider people, process, technology, data, environment and suppliers before agreeing on the most likely causes.
The diagram does not find the cause by itself. Each candidate cause still has to be checked against evidence: logs, timings, change records. Here, the job log showed no disk errors and the meter data arrived on time, which ruled out both causes on the suppliers bone.
Timelines and fault trees
Two other techniques are common. A timeline lists every relevant event in order (deployments, alerts, configuration changes, user reports) and often reveals the cause by what happened just before the first symptom. The change record of last month's smart meter release would have appeared right there. A fault tree works top-down from the failure, splitting it into the combinations of conditions that could cause it, connected by AND and OR. It suits failures with several contributing conditions, such as an outage that needs both a failed power feed and a failed generator.
The known error database
A known error database (KEDB) is the collection of known error records, each describing a known error's symptoms, cause if known, and workaround, so that anyone who meets the symptom can find it. In most organizations it is not a separate database but a record type or a filtered view in the ITSM tool, linked to the knowledge base.
Its value is at the service desk: when the nightly invoice run
bill-nightly (a separate job from the 02:30 billing
export) fails again at 06:00, the operator searches the symptom,
finds the record, applies the workaround and links the incident,
without waking anyone up.
Number: KE-0087 Linked problem: PRB-0207
Title: Nightly billing job stops on 4-hour timeout
Service/CI: Billing / batch-01, db01
Status: Known error, fix scheduled (CHG-8851, Sat 02:00)
Symptoms: Job "bill-nightly" ends with TIMEOUT after 04:00:00;
invoices not sent by 07:00; more likely after
month-end and storm nights (high reading volume)
Cause: Index on readings(customer_id, read_at) missing since
release 7.4 migration
Workaround: 1. Check that no other batch job is running on batch-01
2. Start "bill-nightly --resume" from the ops console
3. Expect 50-70 minutes; inform billing team lead
4. Link the incident to KE-0087
Owner: Amara Okafor (problem management)
Incidents: INC-6101, INC-6112, INC-6131, INC-6140
Reactive and proactive problem management
Reactive problem management starts from incidents that have already happened: a major incident or a recurring pattern. Most problem work in most organizations is reactive.
Proactive problem management looks for problems before they cause incidents: analyzing trends in monitoring data, reviewing supplier advisories about defects in products you use, checking known errors in software before a rollout, or learning from incidents at other organizations. When Kestrel's network team reads that a router firmware version has a memory leak that crashes the router after about 200 days, and finds three routers at 170 days of uptime, they raise a problem before anyone loses connectivity.
How incidents and problems relate
The two practices work on the same failures with different goals, and often with different people:
| Incident management | Problem management | |
|---|---|---|
| Goal | Restore service quickly | Find and remove causes; reduce future incidents |
| Pace | Minutes to hours, against targets | Days to weeks, thoroughness over speed |
| Typical people | Service desk, on-call engineers, incident manager | Problem manager, specialists, developers, suppliers |
| Result | A working service (perhaps with a workaround) | Known errors, workarounds, changes that remove causes |
One problem usually has many incidents linked to it; an incident is normally linked to one problem. The link works both ways: incident records give problem management its evidence, and known error records give the service desk its workarounds. When the fix is deployed, linked incidents that are still open can be resolved together. For the incident side, see incident management.
Common mistakes
- Problems without owners. A problem record that nobody owns stays open for a year. Every problem needs a named person and a review date.
- Workarounds kept in heads. If only Joost knows how to rerun the job, the workaround fails the night he is on holiday. Write it in the known error record.
- Closing the problem when the workaround works. A working workaround does not resolve the problem: it stays open, usually as a known error, until the cause is removed or the risk is formally accepted.
- Confusing problem management with blame. Root cause analysis finds conditions, not culprits; see blameless reviews.