Incident Response in OT

What is different in OT

OT incident response is the organized handling of a cybersecurity incident, or a suspected one, that affects industrial control systems: the controllers, networks and screens that run a physical process. It follows the same lifecycle as incident response in IT — prepare, detect and analyze, contain, eradicate, recover, learn — but three things change the priorities.

• Safety comes first. A compromised office laptop leaks data; a compromised controller can open a valve, overheat a transformer or disable a protection function. The first question is always: is anyone or anything physically at risk right now? Protection relays and safety instrumented systems (see Safety and Reliability) are designed to act on their own, but responders must check that they still can.
• The process keeps running. A power grid, a water plant or a chemical reactor cannot simply be switched off while the team investigates. Shutting down has its own risks and costs: customers without power or water, a batch that spoils, equipment stressed by an unplanned stop. So the response often happens while the process runs.
• Three groups must work as one. Operators know the process, control engineers know the controllers and their logic, and IT security knows the attacker's techniques. None of them can handle an OT incident alone.

Not every OT incident is an attack. A failing network switch, a misconfigured controller or a software bug can look exactly like one at first. The response process is the same until the evidence says otherwise, and it links to normal operational incident management — a suspected cyber incident in OT is usually also a major incident in the IT service sense.

Who is involved

An OT incident response team is broader than an IT computer security incident response team (CSIRT). Typical roles:

Agree in advance who has the final say over the process. In most organizations that is the shift lead or operations manager, not the security team: security advises, operations decides, and both write down why.

The OT response flow

The flow below adapts the incident response lifecycle to OT. The two steps that do not appear in a typical IT plan are highlighted: assessing the safety impact before anything else, and deciding whether the process keeps running or moves to a safe state.

1. Detect. An alert from OT network monitoring, an operator who sees a breaker or valve move without a command, an engineer who finds a changed program, a warning from a CSIRT (such as NCSC-NL, CISA or the UK NCSC, or the sector's own) or a vendor.
2. Assess the safety impact. What can this affect physically? Are protection and safety systems still working? Can operators still see and control the process? The answers set the urgency of everything else.
3. Contain. Stop the spread and cut the attacker's access, in the way that disturbs the process least (see the next section).
4. Keep operating, or move to a safe state. Operations decides whether the process can keep running under closer watch, must switch to manual operation, or must be brought to a safe state.
5. Eradicate. Remove the attacker's foothold: rebuild compromised workstations and servers, reset credentials, close the entry point. Controllers are restored, not "cleaned".
6. Recover. Restore from known-good copies, verify, test with operations, and return to automatic control step by step.
7. Lessons learned. What allowed it, what slowed the response, and what changes in architecture, monitoring, playbooks and training.

Deciding to isolate or disconnect

In IT, containment often means pulling an infected machine off the network. In OT the same reflex can do harm: disconnecting a controller from its supervisory system can leave operators blind, and stopping a process can be more dangerous than letting it run. The choices form a ladder, from light to heavy:

Isolating the IT/OT boundary is often the first step because it is usually reversible and keeps the process itself running. The 2021 ransomware attack on Colonial Pipeline shows the trade-off at its sharpest: the ransomware hit IT systems, and the company shut down the pipeline as a precaution, without a reported compromise of its OT. Fuel supply was disrupted for days. A precautionary stop can be the right decision, but it should be a deliberate one, taken with a clear view of what each option costs (more in Threats and Notable Incidents).

Make these decisions possible in advance. Joint guidance from several national cybersecurity agencies, among them the UK National Cyber Security Centre and NCSC-NL (January 2026), lists having an isolation plan as one of its principles for secure OT connectivity: know which links can be cut, by whom, and what keeps working afterward. An isolation switch that nobody dares to use because its effects are unknown does not help.

Manual operation and safe state

Two terms come up in every OT incident plan.

Manual operation means running the process with people instead of (or with less) automation: operators on site use local control panels, switches and hand valves, read local gauges, and coordinate by phone or radio. In a substation that can mean operating breakers from the local control panel instead of from the control center. Manual operation keeps the process going when the control system cannot be trusted, but it needs enough trained staff who know the local procedures, and it only works for as long as those people can keep it up.

The United States Cybersecurity and Infrastructure Security Agency (CISA) and its partners list "practice manual operations" among their five primary mitigations for OT (May 2025). The reason is simple: if a site has not run manually for years, nobody knows whether it still can.

A safe state is the condition a process is brought into so that it cannot cause harm when control is lost or deliberately stopped. It is defined per process during design: a pump stopped, a valve closed, a breaker open, a burner off. A safe state is not always "everything off" — a reactor may need cooling to keep running, and in a power grid, switching off a line can overload the others. That is why the safe state comes from the process and safety engineers, not from the incident team.

OT playbooks

A playbook gives responders a tested sequence of steps for a known type of incident. OT playbooks add process checks, name the operations and engineering roles explicitly, and list what not to do. Common OT playbooks cover ransomware on the IT side, unexpected remote access, unexplained process behavior, loss of view (operators cannot see the process) and loss of control.

Trigger   Monitoring shows a program download to PLC-PUMP-02 outside a
          change window, or an engineer finds logic that differs from
          the approved version.

1  Operations: check the process. Is PLC-PUMP-02 behaving normally?
   Are alarms, interlocks and the safety system healthy?
   If not: follow the process procedure (manual operation or safe
   state) before anything else.
2  Shift lead: open an incident, call the OT security on-call and the
   control engineer for this area. Note the time of awareness.
3  OT security: check the change calendar and the remote access logs.
   Was there an approved change? Who was connected, from where?
4  Control engineer: export the running program (read-only, with the
   vendor tool) and compare it with the known-good baseline. Do NOT
   download or "fix" anything yet.
5  If unexplained: contain. Disable remote and vendor access to the
   zone, keep the engineering workstation powered on but disconnected,
   and set the PLC mode key to RUN if the site uses one.
6  Preserve evidence: network capture from the monitoring sensor,
   engineering workstation and HMI logs, historian data for the period.
7  Decide with operations: restore the known-good program now, keep
   running under watch, or move to manual operation.
8  Escalate: crisis team informed; legal/compliance assesses the
   reporting duty (24-hour early warning clock).
9  Restore and verify (see Recovery), then close with lessons learned.

Do not   reboot the PLC, scan the network, or wipe the engineering
         workstation before evidence is preserved.

Step 5 refers to the physical mode key that many controllers have: in the RUN position, the controller refuses program changes over the network. CISA has recommended using it as a mitigation after attacks on internet-exposed controllers in 2026. Each playbook needs an owner, a date of last review, and a test in an exercise.

Evidence without disturbing the process

Digital forensics in IT often starts by capturing a computer's memory and imaging its disk. Controllers do not work that way: they have little or no logging, their memory cannot be read without vendor tools, and many cannot be stopped for analysis. OT evidence therefore comes mostly from around the controllers:

• Network captures and logs from passive OT monitoring (see Asset Inventory and Monitoring): which device talked to which, with which commands, when. Monitoring that was in place before the incident is worth far more than anything set up afterward.
• Windows-based systems — engineering workstations, human-machine interface (HMI) servers, historians, jump hosts — can be investigated with normal forensic methods, once operations agrees that taking them out of service is safe.
• Process data from the historian: setpoints, valve positions and alarms before and during the event, which can show what the attacker changed in the physical world.
• Controller programs and configurations, exported read-only and compared with known-good copies, as in the example below.

Firmware and memory analysis of controllers is specialist work, normally done with the vendor. Record what was collected, by whom and when, as in any chain of custody.

Comparing with a known-good baseline

A quick way to see whether controller programs were changed is to compute a cryptographic hash of each exported program or configuration file and compare it with the hash recorded after the last approved change (see hash functions). Equal hashes mean identical files; a different hash means something changed, but not what or why — that is for the engineer to find out. In this sketch, short byte strings stand in for the exported files:

import hashlib

def fingerprint(data: bytes) -> str:
    """SHA-256 of a program or configuration export, as hex."""
    return hashlib.sha256(data).hexdigest()

# Known-good exports, stored offline after the last approved change.
# (Here the "files" are short byte strings so the example is self-contained.)
known_good = {
    "PLC-PUMP-01": b"LD I0.0\nAND I0.1\nST Q0.0\n",
    "PLC-PUMP-02": b"LD I0.0\nAND I0.1\nST Q0.0\n",
    "RTU-BAY-07": b"[iec104]\ncommon_address=7\nt1=15\n",
    "PLC-VALVE-03": b"LD I1.2\nANDN I1.3\nST Q1.0\n",
}
baseline = {asset: fingerprint(data) for asset, data in known_good.items()}

# Exports collected during the incident (read-only, by the control engineer).
collected = {
    "PLC-PUMP-01": b"LD I0.0\nAND I0.1\nST Q0.0\n",
    "PLC-PUMP-02": b"LD I0.0\nOR I0.4\nST Q0.0\n",
    "RTU-BAY-07": b"[iec104]\ncommon_address=7\nt1=15\n",
    "HMI-04": b"screens=12\n",
}

for asset in sorted(baseline.keys() | collected.keys()):
    if asset not in collected:
        status = "NOT COLLECTED - follow up"
    elif asset not in baseline:
        status = "NO BASELINE - unknown asset or missing record"
    elif fingerprint(collected[asset]) == baseline[asset]:
        status = "matches known-good"
    else:
        status = "DIFFERS - compare logic, escalate"
    shown = fingerprint(collected[asset])[:12] if asset in collected else "-" * 12
    print(f"{asset:<13} {shown}  {status}")

HMI-04        6d30746b2f06  NO BASELINE - unknown asset or missing record
PLC-PUMP-01   4e92d8b05837  matches known-good
PLC-PUMP-02   3c707661bf1a  DIFFERS - compare logic, escalate
PLC-VALVE-03  ------------  NOT COLLECTED - follow up
RTU-BAY-07    6833cb4bee0b  matches known-good

All three non-matching outcomes need follow-up. A missing baseline is a gap in preparation, an uncollected asset is a gap in the investigation, and a difference may be an attack — or an undocumented change by a colleague. In practice the comparison works only if exports are reproducible: some engineering tools add timestamps to every export, so teams compare the logic itself with the vendor's compare function and use hashes for the stored files.

Recovery

Recovery in OT means bringing back a process that does exactly what it was designed to do, and proving it. The building blocks:

• Known-good programs and configurations for every controller, protection relay, switch and firewall: versioned, recorded with a hash, and updated after every approved change through change enablement.
• Backups of the servers and workstations — HMI, historian, engineering workstations, domain controllers if OT has its own — kept offline or immutable, following the 3-2-1 rule, and with restores that are tested.
• Installation media and licenses for the exact software and firmware versions in use; vendor portals may not be reachable or may no longer offer old versions.
• Spare hardware for controllers and network equipment with long delivery times.

Restore in a fixed order: first the network and the security controls that keep the attacker out, then engineering workstations (rebuilt, not cleaned), then servers, then controllers. After each controller is restored, compare it with the baseline again, test the functions together with operations, and only then hand control back to automation, one area at a time. For the wider recovery planning — priorities, recovery time objectives, alternate sites — see Business Continuity.

Exercises

Plans that have never been practiced fail in predictable places: phone numbers that are out of date, decisions nobody feels authorized to take, manual procedures that need tools no longer on site. Exercises find those gaps first.

• A tabletop exercise walks a scenario through in a meeting room. For OT, include the shift lead and a control engineer, and let the scenario force the hard choices: isolate or not, stop or not, who reports.
• Manual operation drills on site, ideally during planned maintenance, test whether the process really can be run by hand and for how long.
• Restore tests on a test bench or spare controller prove that the stored programs actually load and work.
• Joint exercises with outside parties. For an electricity company that means the connected grid operators (the transmission and distribution system operators, see The Power Grid), the CSIRT (NCSC-NL, CISA or the UK NCSC, or the sector's own) and the supervisor; national crisis exercises are one way to bring these together. The EU Network Code on Cybersecurity for electricity includes exercises among its measures, and US pipeline operators must exercise their incident response plan under TSA's directives (see OT Regulation).

Reporting duties

Reporting runs alongside the technical work, and its first clocks start when the organization becomes aware of the incident or determines that it is reportable. The deadlines differ per region and sector; as of October 2026:

Who reports to whom, and the details of each law, are on OT Regulation. The general principles — log when you became aware, let legal or compliance decide, give them facts fast — are in Incident Response.