Monitoring and Event Management

Why monitor services at all

Without monitoring, IT learns about failures from its users. At Kestrel Energy that means the first sign of a broken payment page is a queue of angry customers at the contact center, and the first sign of a full disk on the billing database is a failed monthly billing run. Monitoring and event management exists to turn that around: to notice changes in services and their components, decide which ones matter, and respond — ideally before anyone outside IT notices.

The practice does three things: it monitors (collects data about the state of services, servers, network devices, applications and cloud resources), it manages events (decides what each change means and what to do with it), and it connects the result to other practices, mainly incident management, problem management and capacity planning.

Monitoring and observability

Monitoring answers questions you thought of in advance: is the router up, is the disk below 85%, does the login page respond within two seconds? Observability is the property of a system that lets you investigate questions you did not think of, from its metrics, logs and traces. The Cloud topic explains both in depth. The Observability & SRE topic goes further, from metrics, logs and traces to a working Prometheus setup. Event management works on top of either: whatever the data source, someone has to decide which changes deserve attention.

What an event is

An event is any change of state that has significance for the management of a service or one of its components: a backup that finished, a disk that crossed a threshold, an interface that went down, a user who logged in from a new country. Events come from many places — a monitoring agent on a server, a Simple Network Management Protocol (SNMP) trap from a switch, a syslog message from a router, a cloud provider's health notification, an application writing an error.

An event is not the same as an incident. Most events are normal: the nightly backup finishing is an event, and nobody should be called about it. An incident is an unplanned interruption or reduction in the quality of a service; some events reveal one, most do not.

Informational, warning and exception events

Events are commonly sorted into three types, each with its own handling:

• An informational event confirms that something happened as expected. It needs no action, but it is logged — for audits, reports and later investigation.
• A warning event says a component is approaching a limit or behaving unusually. Nothing is broken yet, but someone should look before it is.
• An exception event says something is wrong: a component has failed or performance is outside its acceptable range. It usually needs action now, often as an incident.

Security events follow the same idea but are collected and analyzed in a security information and event management (SIEM) system and handled with security incident response; see Security Monitoring and the SOC.

Thresholds

A threshold is the value of a monitored measurement at which the monitoring system creates an event, such as "disk usage above 85%" for a warning and "above 95%" for an exception. Choosing thresholds well is most of the work in monitoring.

• A static threshold is a fixed line. It is simple and predictable, and right for hard limits: a disk can never be more than 100% full.
• A dynamic threshold (or baseline-based threshold) learns what is normal for this time of day and week, and fires when the value deviates. Portal logins at 09:00 on Monday look nothing like logins at 03:00 on Sunday; a single fixed line would be either too loud or too quiet.
• A duration keeps short spikes from firing: CPU above 90% for 10 minutes, not for one sample.
• Hysteresis uses a different value to clear an event than to raise it: raise at 90%, clear only below 80%. Without it, a value hovering around 90% produces an endless stream of "problem — resolved — problem" events, called flapping. Zabbix, for example, calls this a recovery expression.

A threshold as a Prometheus alerting rule, with a duration and an owner:

groups:
  - name: kestrel-billing
    rules:
      - alert: DiskAlmostFull
        expr: |
          100 * (1 - node_filesystem_avail_bytes{mountpoint="/var"}
                   / node_filesystem_size_bytes{mountpoint="/var"}) > 85
        for: 10m
        labels:
          severity: warning
          team: database
        annotations:
          summary: "/var on {{ $labels.instance }} is more than 85% full"
          runbook_url: "https://wiki.example.com/runbooks/disk-almost-full"

Kestrel's Alertmanager later routes on two values of this label, page and ticket, so a warning like this one becomes a ticket, never a page; see Alerting.

Filtering, deduplication and correlation

When a carrier's fiber is cut, the edge router reports its uplink down. Seconds later, every switch, server and application behind it reports "unreachable", each one every minute, from every monitoring probe. Two hundred alerts for one cause is not information; it is noise that hides the cause. Event management reduces it in three steps:

1. Filtering: informational events go to the log, not to a person.
2. Deduplication: the same event from the same device within a short window is dropped (or counted), not repeated.
3. Event correlation: related events are linked to one likely cause, using knowledge of how components depend on each other (often from the configuration management database) and how close together in time they happened.

This program applies all three to a small list of events from Kestrel's data center. A device that becomes unreachable shortly after a device it depends on has failed is treated as a symptom, not as a separate alert:

DEDUP_WINDOW = 300      # seconds: identical events within 5 minutes are one
CORRELATE_WINDOW = 120  # seconds: symptoms within 2 minutes of a cause

# Which device each device depends on (from the CMDB)
UPSTREAM = {
    "sw-dc1-a": "rtr-dc1-edge",
    "sw-dc1-b": "rtr-dc1-edge",
    "app-portal-01": "sw-dc1-a",
    "app-portal-02": "sw-dc1-b",
}

events = [  # time, device, type, message
    ("10:02:05", "rtr-dc1-edge", "exception", "Gi0/0/1 down (peer 198.51.100.1)"),
    ("10:02:07", "sw-dc1-a", "exception", "Host unreachable"),
    ("10:02:08", "sw-dc1-a", "exception", "Host unreachable"),
    ("10:02:09", "sw-dc1-b", "exception", "Host unreachable"),
    ("10:02:11", "app-portal-01", "exception", "Host unreachable"),
    ("10:02:41", "app-portal-01", "exception", "Host unreachable"),
    ("10:02:30", "app-portal-02", "exception", "Host unreachable"),
    ("10:03:00", "backup-01", "informational", "Backup job finished"),
    ("10:04:10", "db01", "warning", "Disk /var 85% full"),
]


def seconds(clock):
    hours, minutes, secs = map(int, clock.split(":"))
    return hours * 3600 + minutes * 60 + secs


# 1. Filter: informational events are logged, not alerted
logged = [e for e in events if e[2] == "informational"]
actionable = sorted((e for e in events if e[2] != "informational"),
                    key=lambda e: seconds(e[0]))

# 2. Deduplicate: same device and message within the window
unique, last_seen = [], {}
for event in actionable:
    key = (event[1], event[3])
    if key in last_seen and seconds(event[0]) - last_seen[key] <= DEDUP_WINDOW:
        continue
    last_seen[key] = seconds(event[0])
    unique.append(event)

# 3. Correlate: if a device upstream failed shortly before,
#    this event is a symptom of that failure, not a new alert
alerts = []       # [cause event, list of symptom devices]
alert_for = {}    # device -> the alert it belongs to
for event in unique:
    device, parent = event[1], UPSTREAM.get(event[1])
    while parent and parent not in alert_for:
        parent = UPSTREAM.get(parent)
    if parent and seconds(event[0]) - seconds(alert_for[parent][0][0]) <= CORRELATE_WINDOW:
        alert_for[parent][1].append(device)
        alert_for[device] = alert_for[parent]
    else:
        alert_for[device] = [event, []]
        alerts.append(alert_for[device])

print(f"{len(events)} events in, {len(actionable) - len(unique)} duplicates dropped,"
      f" {len(logged)} logged only, {len(alerts)} alerts")
for (clock, device, kind, message), symptoms in alerts:
    print(f"ALERT {clock} {kind:<9} {device}: {message}")
    if symptoms:
        print(f"      + {len(symptoms)} related: {', '.join(symptoms)}")

9 events in, 2 duplicates dropped, 1 logged only, 2 alerts
ALERT 10:02:05 exception rtr-dc1-edge: Gi0/0/1 down (peer 198.51.100.1)
      + 4 related: sw-dc1-a, sw-dc1-b, app-portal-01, app-portal-02
ALERT 10:04:10 warning   db01: Disk /var 85% full

Real tools do the same with more rules: dependency trees in Zabbix and Nagios suppress "unreachable" for hosts behind a failed parent, and Prometheus Alertmanager groups alerts with the same labels into one notification. Alerting shows grouping, inhibition and silences step by step. Correlation is also where AI-based tools help most (see AIOps below).

From event to alert to incident

After filtering and correlation, what is left becomes an alert: a notification to a person or a system that something needs attention. An exception on a service that matters creates an incident ticket automatically, with the event details, the affected configuration item and a link to the runbook already filled in. The network engineer who is paged opens a ticket that is already there, instead of typing one in the middle of the night.

INC-20577   Priority: P2 (auto)        Source: Zabbix
Opened:     Thu 01 Oct 2026 10:02:14    Assigned: Network team (on-call: Tomasz)
CI:         rtr-dc1-edge (203.0.113.1)  Service: Customer Portal
Summary:    Uplink Gi0/0/1 to carrier down (peer 198.51.100.1)
Correlated: 4 related events (sw-dc1-a, sw-dc1-b, app-portal-01, app-portal-02)
Runbook:    https://wiki.example.com/runbooks/edge-uplink-down
Note:       Traffic failed over to secondary uplink; portal reachable,
            no redundancy until uplink restored.

When the monitoring system sees the interface come back up, it can add a note to the ticket — but it should not close it on its own if a person still has to confirm the cause and check that the carrier's repair holds. What happens to the ticket after it is created is the subject of incident management.

Alert fatigue

Alert fatigue is what happens when people receive so many alerts, most of which need no action, that they stop reacting to them — and then miss the one that matters. It is the most common way monitoring fails: not because nothing was detected, but because the detection drowned in noise. At Kestrel, a network operations center (NOC) that receives 600 alerts per shift will acknowledge them in bulk, and the disk warning that predicted next week's billing outage goes with them.

What helps:

• Actionable alerts only. If nobody needs to do anything, it is a log entry or a dashboard graph, not an alert. Ask of every alert: what would the person who receives it do?
• Ownership. Every alert routes to one team, with a runbook that says what it means and what to check first. An alert that goes to everyone is read by no one.
• Review noisy alerts. Each week, list the alerts that fired most often and those that were closed without action. Tune the threshold, add a duration, or delete them.
• Alert on symptoms users feel (portal errors, slow logins) rather than every possible cause; see thresholds and alert fatigue in the Cloud topic for alerting on service level objectives, and fighting alert fatigue for a review of a month of real pages.

AIOps

AIOps (AI for IT operations) applies machine learning to monitoring data: learning dynamic thresholds, spotting anomalies, correlating events across tools that do not know about each other, and suggesting a probable cause. It helps most where the volume is too high for fixed rules. It does not remove the need for clean data — correlation is only as good as the dependency information behind it — and its suggestions need a human check before anyone acts on them. See AI in Service Management.

Tools

Monitoring tools overlap a lot; most organizations run more than one. Some common names:

Whatever the tool, the value is in the integration: events should create tickets in the ITSM tool (ServiceNow, Jira Service Management, TOPdesk and others), link to configuration items, and page the right on-call person.

A day in the NOC

A NOC is the team, often working in shifts, that watches the network and infrastructure and responds to events. Kestrel's NOC works from a checklist that turns event management into a routine:

07:00  Handover from night shift: open incidents, ongoing changes,
       anything flapping. Read the shift log, ask questions.
07:15  Morning checks: backup report (all jobs finished?), certificate
       expiry list (< 30 days), disk and capacity warnings.
08:00  Check today's change schedule: which alerts to expect during
       maintenance, which to silence (and until when).
       Then: watch the dashboards, triage new alerts within 5 minutes,
       link correlated events to one ticket, escalate per runbook.
14:00  Review: alerts closed without action today - candidates for
       tuning. Add them to the weekly noisy-alert list.
15:00  Handover to evening shift. Write the shift log.

Two habits matter most. Silences have an end time: alerts muted for a maintenance window must come back afterward, or the next real failure goes unseen. And the shift log is written for a stranger: whoever takes over at 15:00 should not need to call you to understand what is going on.