What Is Observability?

Monitoring and observability

The Cloud topic introduces both words in Monitoring, Logging and Observability, and the ITSM topic shows where monitoring fits in Monitoring and Event Management. This topic goes deeper: how the data is produced, stored and queried, and how teams turn it into reliability targets, alerts and calm incident response.

Monitoring is collecting, processing and displaying measurements of a system and checking them against conditions you chose in advance: is the portal answering, are fewer than 1% of requests failing, is the disk of db01 less than 85% full? Google's Site Reliability Engineering (SRE) book defines it in the same spirit, as collecting and aggregating "real-time quantitative data about a system". Monitoring answers questions you already knew to ask.

Observability is a property of the system, not a product: how well you can work out what is going on inside it from the outside, including for a problem nobody predicted. At Kestrel Energy, a question like "why are usage graphs slow only for customers whose meters report through the second meter-api instance, since the deploy at 14:10?" is an observability question. You can answer it only if the services emit data with enough detail (instance, version, route), and if you can slice and connect that data.

The two are not rivals. Monitoring tells you that something is wrong; observability helps you find out why. You build monitoring on top of an observable system.

Telemetry: the raw material

Telemetry is the umbrella word for all data that software and infrastructure emit about their own behavior so it can be collected elsewhere: metrics, logs, traces, and newer kinds such as profiles. The word comes from measuring at a distance, and that is still the point: the data leaves the machine while it is produced, so it survives when a container is replaced or a server crashes.

Telemetry is either pulled (a monitoring server asks each service for its current numbers, as Prometheus does; see Prometheus) or pushed (the service or an agent sends data to a collector, as most log shippers and OpenTelemetry exporters do; see OpenTelemetry). Either way it costs something: CPU to produce it, network to move it and storage to keep it. Choosing what to collect is part of the job.

The three signals

Most telemetry falls into three kinds, often called the three signals or the three pillars. Each answers a different question, at a different cost:

The short definitions of monitoring metric, log and trace are in the Cloud topic; the pages linked in the table cover each signal in depth.

Why "three pillars" is a simplification

Collecting all three signals does not make a system observable. Three separate stores that cannot be connected leave you guessing which log line belongs to which spike. What matters is correlation: the same service and instance names everywhere, clocks kept in step with Network Time Protocol (NTP), and a trace ID in each log line (see correlation IDs).

Other kinds of telemetry fill gaps the three leave:

• Profiles show which functions use the CPU or memory. OpenTelemetry calls profiles an emerging fourth signal; as of October 2026 its profiles support is still in Alpha, so expect changes.
• Change events such as deployments, configuration changes and failovers. Many incidents start right after a change, so drawing deploys on the same graph as the error rate saves time (see events in ITSM).
• Synthetic checks: results of probes that act like a user (next section).

Black-box and white-box monitoring

The SRE book asks two questions of every monitoring system: "what's broken, and why?" The first is about symptoms, the second about causes. Two ways of looking at a system match them.

Black-box monitoring tests the externally visible behavior of a service the way a user experiences it, without knowing anything about its internals. Kestrel runs a probe from outside the data center that loads https://portal.example.com/login every minute, checks for HTTP status 200 and for the word "Log in" on the page, and records how long it took. Prometheus' blackbox exporter and hosted uptime checkers do this kind of work.

White-box monitoring is based on data the system exposes about its own internals: request counters and latency histograms from the application, queue depths, database connection counts, logs and traces. It needs instrumentation, but it sees problems that are building up before users notice, and failures that retries hide.

Use both. Page a person on symptoms, because a symptom means users are affected now; the black-box view is ideal for that. Use the white-box data to diagnose, and to raise a ticket for problems that are coming but not yet hurting anyone (a disk that will be full in three days). Note that one team's symptom is another team's cause: slow queries are a symptom for Hamid Rahimi, the DBA, and a cause for Ravi Kumar's portal team.

The four golden signals

The SRE book gives a shortlist for any user-facing system: "If you can only measure four metrics of your user-facing system, focus on these four." These are the four golden signals:

• Latency: how long it takes to serve a request. Keep successful and failed requests apart: a database error that returns in 2 ms makes the average look better while users suffer, and "a slow error is even worse than a fast error".
• Traffic: how much demand reaches the system, in a unit that fits it: HTTP requests per second for the portal, readings per second for meter-api.
• Errors: the rate of requests that fail, either explicitly (HTTP 500), implicitly (a 200 with wrong or empty content) or by policy (Kestrel counts a portal page slower than 3 s as failed).
• Saturation: how full the service is (below).

Here are the four signals of meter-api during a simulated incident on Kestrel's test monitoring server, in which the readings consumer slowed down at 17:09 UTC. The commands use promtool query, Prometheus' command-line client; the queries themselves are explained in PromQL. Traffic, the three busiest routes in requests per second, just before the incident:

$ promtool query instant http://localhost:9090 'topk(3, sum by (job, route) (rate(http_requests_total[5m])))'

{job="meter-api", route="/v1/readings"} => 76.02134080733286 @[1791306532.795]
{job="meter-api", route="/v1/meters/{meter_id}/readings"} => 11.999999999999998 @[1791306532.795]
{job="portal", route="/login"} => 9.99649753036437 @[1791306532.795]

Errors, as the share of requests with a 5xx status over the last 5 minutes, two and a half minutes into the incident (a recording rule stores the result under the name job:http_requests_errors:ratio_rate5m):

$ promtool query instant http://localhost:9090 'job:http_requests_errors:ratio_rate5m'

job:http_requests_errors:ratio_rate5m{job="portal"} => 0 @[1791306694.152]
job:http_requests_errors:ratio_rate5m{job="meter-api"} => 0.03874094577733076 @[1791306694.152]

Latency, as the portal's 95th percentile in seconds, every two minutes from 17:06 to 17:16 UTC (the portal waits for meter-api, so it slows down too):

$ promtool query range --start=2026-10-06T17:06:00Z --end=2026-10-06T17:16:00Z --step=2m \
    http://localhost:9090 'histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket{job="portal"}[5m])))'

{} =>
0.3908953227691752 @[1791306360]
0.3897239333102146 @[1791306480]
0.8188247011952179 @[1791306600]
1.7207677165354307 @[1791306720]
2.1553592814371245 @[1791306840]
2.1419863861386133 @[1791306960]

The p95 climbs from 0.39 s to 2.16 s. It takes a few minutes to get there because each point summarizes the 5 minutes before it.

Saturation

Saturation is how "full" a service or resource is: how close it is to the limit of its most constrained part, and how much work is waiting because it cannot be served right away. Typical measures are queue length, threads waiting for a CPU, connections in use out of the maximum, and disk space left. Saturation also includes predictions such as "the disk of db01 will be full in four hours".

For meter-api, the readings queue is the saturation signal. During the incident it was filling at about 90 readings per second on each instance, because 150 readings per second arrived and only 60 were written to the database:

$ promtool query instant http://localhost:9090 'meter_readings_queue_depth'

meter_readings_queue_depth{env="production", instance="meter-api-01:8080", job="meter-api", team="metering"} => 33467 @[1791306931.747]
meter_readings_queue_depth{env="production", instance="meter-api-02:8080", job="meter-api", team="metering"} => 33778 @[1791306931.747]

Before the incident the depth stayed between 5 and 40. Many systems slow down well before they reach 100% of anything, which is why the SRE book notes that rising latency is often the first sign of saturation. The Capacity Planning and Performance page explains why.

RED: for request-driven services

The RED method (Tom Wilkie, 2015) asks three things of every service that handles requests: its Rate (requests per second), its Errors (requests that fail) and the Duration of requests (as a distribution, so percentiles). It is the four golden signals without saturation, aimed at the many small services of a modern application. Its strength is uniformity: if every service has the same three graphs in the same place, anyone on call can read any service's dashboard.

In PromQL, for meter-api (the counters and histogram are described in Metrics):

# Rate: requests per second, per route
sum by (route) (rate(http_requests_total{job="meter-api"}[5m]))

# Errors: share of requests that failed with a 5xx status
sum(rate(http_requests_total{job="meter-api", code=~"5.."}[5m]))
  / sum(rate(http_requests_total{job="meter-api"}[5m]))

# Duration: 95th percentile latency in seconds
histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket{job="meter-api"}[5m])))

A 404 or a 400 is usually not counted as an error of the service: the client asked for something that does not exist or sent a bad reading. Count 5xx responses, plus any request that broke a rule you set, such as "slower than 3 s".

USE: for resources

The USE method by Brendan Gregg is a checklist for resources rather than services: "For every resource, check utilization, saturation, and errors." A resource is anything with a limited capacity: CPUs, memory, disks, network interfaces, but also a database connection pool or a thread pool.

• Utilization: the share of time the resource was busy (or, for capacity such as memory, the share in use).
• Saturation: work the resource cannot handle yet and has to queue.
• Errors: the count of error events.

Some of these as PromQL queries, using node_exporter's metric names for the host and Kestrel's own names for the database:

# Utilization: share of CPU time not idle, per host
1 - avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m]))

# Saturation: 1-minute load average per CPU (above 1 means work is waiting)
node_load1 / on (instance) count by (instance) (node_cpu_seconds_total{mode="idle"})

# Utilization of the connection limit on db01
pg_connections_in_use / pg_connections_max

# Errors: network receive errors per second
rate(node_network_receive_errs_total[5m])

Utilization is an average over time, and averages hide bursts. meter-api's host api01 receives smart-meter readings in bursts at :00, :15, :30 and :45 past the hour. One minute of per-second samples shows what the average hides:

# api01, one minute, one sample per second:
#   busy  = fraction of that second the CPUs were busy (utilization)
#   queue = threads waiting for a CPU at that moment (saturation)
busy = []
queue = []
for second in range(60):
    in_burst = second % 15 < 3       # readings arrive at :00, :15, :30 and :45
    busy.append(1.0 if in_burst else 0.55)
    queue.append(6 - 2 * (second % 15) if in_burst else 0)

utilization = sum(busy) / len(busy)
saturated = sum(1 for waiting in queue if waiting > 0)

print(f"Average utilization: {utilization:.0%}")
print(f"Seconds at 100% busy: {busy.count(1.0)} of {len(busy)}")
print(f"Seconds with work waiting: {saturated} ({saturated / len(queue):.0%})")
print(f"Longest queue: {max(queue)} threads")

Average utilization: 64%
Seconds at 100% busy: 12 of 60
Seconds with work waiting: 12 (20%)
Longest queue: 6 threads

An average utilization of 64% looks comfortable, yet in one second out of five there was work waiting. That is why USE asks for saturation separately: it shows the queueing that utilization averages away. The fix here is not a bigger server but spreading the meters' send times, which the Reliability Patterns page covers under jitter.

Jobs that are not request-driven

RED and USE fit services and machines. They do not fit Joost Bakker's nightly billing export, which starts at 02:30, runs for about twelve minutes and then stops. It has no request rate, and its CPU is idle most of the day. For a batch job, ask three questions instead:

• Did it run, and did it succeed?
• How long did it take, compared with earlier runs?
• Did it do the expected amount of work (number of invoices)?

The job publishes the Unix time of its last successful run, so the monitoring system can raise an alert when that time is too old. More than 26 hours ago means last night's run did not succeed, with two hours of slack for a slow run:

time() - billing_export_last_success_timestamp_seconds > 26 * 3600

The slack also covers the two nights a year when the clocks change: in late March 02:30 Amsterdam time does not exist, and on 25 October 2026 it happens twice, so the time between two runs can be about 23 or 25 hours (how cron handles those hours is on Scheduling Tasks).

Alerting on "the last success is too old" catches most kinds of failure: the job crashed, cron was disabled, or the job hangs. An alert on "the job logged an error" catches only the first. One gap remains: if the server that exposes the timestamp is off, the series disappears and the expression returns nothing. So pair it with an alert on up == 0 or absent(), or push the timestamp to a Pushgateway (see Prometheus).

From observability to SRE

Telemetry is only useful when someone acts on it. The rest of this topic follows that path:

1. Produce good data: Metrics, Logs, Distributed Tracing and OpenTelemetry.
2. Store and query it: Prometheus, PromQL, Dashboards and Logs at Scale.
3. Decide what "good enough" means and act when it is not: SLIs, SLOs and SLAs, Error Budgets and Burn-Rate Alerts and Alerting.
4. Respond and learn as a team: What Is Site Reliability Engineering?, On-Call, Incident Response in Practice and Postmortems.

Site reliability engineering is the discipline that ties these together. The SRE book puts it plainly: teams "carefully avoid any situation that requires someone to 'stare at a screen to watch for problems'". The aim is not a wall of screens in the network operations center, but a small set of signals that page a person only when users are affected, plus data rich enough to find out why.