Monitoring, Logging and Observability

Monitoring versus observability

Monitoring means collecting data about a system and checking it against expectations you set in advance: is the server up, is the disk less than 90% full, do fewer than 1% of requests fail? It answers questions you already knew to ask.

Observability is a property of the system: how well you can work out what is happening inside it from the data it produces, including for problems nobody predicted. "Checkout is slow, but only for customers in one country who pay by card, since the deploy at 14:10" is an observability question. You can only answer it if your services emit rich enough data, and if you can slice that data by country, payment method and version.

The two are not competitors. Monitoring tells you that something is wrong; observability helps you find out why. Both are built from the same three kinds of data, often called the three signals (or pillars): metrics, logs and traces.

In the cloud this matters more than in a single server room. An application may run as dozens of small services on machines that appear and disappear with autoscaling. You cannot log in to "the server" to look around, because by the time you look, it may already be gone. The data has to be shipped somewhere central while it is produced.

This page is the overview; Observability & SRE covers metrics, logs, traces, OpenTelemetry, Prometheus, alerting and SRE practice in depth.

Metrics

A metric is a numeric measurement recorded at regular intervals: a time series such as "central processing unit (CPU) utilization of instance web-3, every 60 seconds" or "requests per second to the checkout application programming interface (API)". Each data point is small (a name, a timestamp, a number and a few labels such as region=eu-west), so metrics are cheap to store for months and fast to graph and compare.

Cloud providers collect basic metrics for their services automatically: CPU, network traffic and disk operations of virtual machines, request counts of load balancers, connections of managed databases. Memory and disk-space usage inside a virtual machine are usually not visible to the provider; you install an agent to report them. Your own application adds custom metrics such as orders per minute or items in a queue.

A metric summarizes; it does not tell stories. "Error rate 4%" says that something is wrong, not which request failed or why. For that you need logs and traces.

What to measure: golden signals, RED and USE

You can measure thousands of things. Three well-known shortlists help you choose:

Saturation means how "full" something is: a queue that keeps growing, a disk at 95%, a CPU run queue that is never empty. For latency, look at percentiles rather than the average. The 95th percentile (p95) is the value that 95% of requests stay under. An average of 80 ms can hide the fact that one request in twenty takes two seconds — and those are the customers who give up.

Logs

A log is a time-stamped record of an individual event, written by an application, operating system or service: a request that was handled, a login that failed, an exception with its stack trace. Logs carry the detail that metrics leave out.

Structured logs

A traditional log line is free text meant for a human: 2026-10-01 09:00:03 ERROR checkout failed after 2004 ms. Searching thousands of such lines means writing fragile text patterns. A structured log writes each event as key-value data, usually one JavaScript Object Notation (JSON) object per line:

{"time": "2026-10-01T09:00:03Z", "level": "ERROR", "service": "checkout", "path": "/api/checkout", "status": 503, "latency_ms": 2004, "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736"}

Now a log service can filter on status >= 500, group by path or compute the average latency_ms without guessing where the number is in the text. The trace_id field links this line to the trace of the same request (see Traces). Use Coordinated Universal Time (UTC) timestamps in ISO 8601 format, and never log passwords, tokens or full card numbers: logs are read by many people and kept for a long time.

Centralized logging, retention and cost

In the cloud, logs are shipped to a central log service as they are written: by the platform itself for managed services, and by an agent or the application for virtual machines and containers. The central store is searchable across all services, survives when a container is deleted, and can be protected from the people whose actions it records.

Logs are also a common surprise on the bill. You typically pay per gigabyte ingested and per gigabyte stored per month, and a chatty debug setting in production can produce gigabytes per hour. Decide per log type how long you really need it (the retention period) and set it explicitly: at the time of writing, a new Amazon CloudWatch Logs log group keeps its data forever unless you set a retention period. Move old logs you must keep for legal reasons to cheaper archive storage. See Cloud Costs and FinOps.

Analyzing structured logs in Python

Because each line is JSON, a few lines of Python can turn raw logs into metrics: requests, server-error rate and p95 latency per endpoint. One line in the input is plain text, as happens when a library writes to the same output; the script counts it instead of crashing.

import json
import math

LOG_LINES = """\
{"time": "2026-10-01T09:00:01Z", "level": "INFO", "path": "/api/orders", "status": 200, "latency_ms": 48}
{"time": "2026-10-01T09:00:02Z", "level": "INFO", "path": "/api/orders", "status": 200, "latency_ms": 52}
{"time": "2026-10-01T09:00:02Z", "level": "INFO", "path": "/health", "status": 200, "latency_ms": 3}
{"time": "2026-10-01T09:00:03Z", "level": "ERROR", "path": "/api/checkout", "status": 503, "latency_ms": 2004}
{"time": "2026-10-01T09:00:04Z", "level": "INFO", "path": "/api/orders", "status": 200, "latency_ms": 61}
{"time": "2026-10-01T09:00:05Z", "level": "INFO", "path": "/api/checkout", "status": 200, "latency_ms": 310}
{"time": "2026-10-01T09:00:05Z", "level": "WARN", "path": "/api/orders", "status": 404, "latency_ms": 12}
{"time": "2026-10-01T09:00:06Z", "level": "INFO", "path": "/api/checkout", "status": 200, "latency_ms": 280}
payment service unreachable, retrying
{"time": "2026-10-01T09:00:07Z", "level": "INFO", "path": "/api/orders", "status": 200, "latency_ms": 45}
{"time": "2026-10-01T09:00:07Z", "level": "INFO", "path": "/health", "status": 200, "latency_ms": 4}
{"time": "2026-10-01T09:00:08Z", "level": "ERROR", "path": "/api/checkout", "status": 500, "latency_ms": 95}
{"time": "2026-10-01T09:00:09Z", "level": "INFO", "path": "/api/orders", "status": 200, "latency_ms": 390}
{"time": "2026-10-01T09:00:09Z", "level": "INFO", "path": "/api/checkout", "status": 200, "latency_ms": 295}
{"time": "2026-10-01T09:00:10Z", "level": "INFO", "path": "/api/orders", "status": 200, "latency_ms": 50}
"""


def p95(values):
    """95th percentile, nearest-rank method."""
    ordered = sorted(values)
    rank = math.ceil(0.95 * len(ordered))
    return ordered[rank - 1]


stats = {}
skipped = 0
for line in LOG_LINES.splitlines():
    try:
        event = json.loads(line)
    except json.JSONDecodeError:
        skipped += 1  # not structured: count it, don't crash
        continue
    entry = stats.setdefault(event["path"], {"requests": 0, "errors": 0, "latencies": []})
    entry["requests"] += 1
    if event["status"] >= 500:  # server errors only; a 404 is the client's problem
        entry["errors"] += 1
    entry["latencies"].append(event["latency_ms"])

print(f"{'Endpoint':<15}{'Requests':>9}{'Error rate':>12}{'p95 (ms)':>10}")
for path in sorted(stats):
    entry = stats[path]
    rate = entry["errors"] / entry["requests"]
    print(f"{path:<15}{entry['requests']:>9}{rate:>12.1%}{p95(entry['latencies']):>10}")
print(f"Skipped {skipped} line(s) that were not valid JSON")

Endpoint        Requests  Error rate  p95 (ms)
/api/checkout          5       40.0%      2004
/api/orders            7        0.0%       390
/health                2        0.0%         4
Skipped 1 line(s) that were not valid JSON

The p95 uses the nearest-rank method: sort the values and take the one at position ⌈0.95 × n⌉. With only five requests that is simply the slowest one; with thousands of requests it ignores the worst 5% of outliers. Status 404 is not counted as an error here: a "not found" is usually the client's mistake, not an outage. Log services such as CloudWatch Logs Insights, Azure Monitor's Kusto Query Language (KQL) and Google Cloud's Log Analytics do the same kind of aggregation at scale.

Traces

One click on "Pay" may pass through a front end, a checkout service, a database, a payment service and a fraud check. If it takes two seconds, which part was slow? A trace records the path of a single request through all the services it touches, with the time spent in each. This is called distributed tracing.

A trace is made of spans. Each span is one unit of work — an incoming request, a database query, a call to another service — with a start time, a duration and a parent span. The first service creates a trace ID and passes it along in a request header (the World Wide Web Consortium (W3C) Trace Context standard defines the traceparent header), so every service adds its spans to the same trace.

Tracing every request can be expensive, so systems often use sampling: they keep, for example, 10% of normal traces but all traces with errors or unusual latency (see head and tail sampling).

Dashboards and alerts

A dashboard puts the most important graphs of a service on one screen, so anyone can see its health at a glance and spot when a change started. Good dashboards are few and focused: one per service showing its golden signals, plus drill-down views for investigations.

An alert is a notification that fires automatically when monitoring data meets a condition, such as "error rate above 2% for 5 minutes". It sends an email or chat message, opens a ticket, pages the on-call engineer (the person responsible for responding outside office hours), or triggers an automatic action such as scaling out.

Thresholds and alert fatigue

A static threshold fires when a value crosses a fixed line (CPU above 90%). Some services also offer dynamic thresholds or anomaly detection, which learn the normal daily pattern and fire when traffic deviates from it. Most alerts also need a duration: a CPU spike of ten seconds is normal; ten minutes at 100% may not be.

Alert fatigue is what happens when people receive so many alerts, most of them harmless, that they start ignoring all of them — including the one that matters. Some rules that help:

• Page a person only for problems that users feel now or will feel soon, and that need a human to fix.
• Alert on symptoms (errors, latency, a full disk in two hours) rather than on every possible cause.
• Every alert has an owner and a short runbook: what it means and what to check first.
• Review alerts regularly. An alert that fired twenty times without anyone doing anything is noise: tune it or delete it.

More on fighting alert fatigue.

Alerting on service level objectives

Mature teams alert on their service level objective (SLO), the reliability target they promise themselves, such as "99.9% of requests succeed over 30 days" (see High Availability, Scaling and Disaster Recovery). The 0.1% of requests that may fail is the error budget. The burn rate says how fast you are using it up: a burn rate of 1 uses exactly the whole budget in 30 days, a burn rate of 15 uses it in two days.

SLO = 0.999                 # 99.9% of requests must succeed
WINDOW_DAYS = 30            # measured over 30 days
error_budget = 1 - SLO      # 0.1% of requests may fail

# Last hour, from the metrics
requests = 120_000
failed = 1_800

error_rate = failed / requests
burn_rate = error_rate / error_budget
days_left = WINDOW_DAYS / burn_rate

print(f"Error budget: {error_budget:.1%} of requests")
print(f"Error rate last hour: {error_rate:.2%}")
print(f"Burn rate: {burn_rate:.1f}x")
print(f"At this rate the 30-day budget is gone in {days_left:.1f} days")
if burn_rate >= 14.4:
    print("Page the on-call engineer")
elif burn_rate >= 1:
    print("Open a ticket")
else:
    print("Within budget")

Error budget: 0.1% of requests
Error rate last hour: 1.50%
Burn rate: 15.0x
At this rate the 30-day budget is gone in 2.0 days
Page the on-call engineer

A burn rate of 14.4 over one hour means 2% of a 30-day budget is gone in that hour; Google's Site Reliability Workbook uses it as an example threshold for paging. A slow, steady burn gets a ticket instead of a wake-up call. The effect: people are woken up only when users are really being hurt. Why one hour, and why a second, shorter window: multiwindow, multi-burn-rate alerts.

Audit logs: who did what

Application logs describe what your software did. An audit log records the actions taken on the cloud account itself: who (which user, role or service) did what (created a virtual machine, changed a firewall rule, deleted a bucket), when, from which IP address, and whether it succeeded. Every action in the cloud is an API call, whether it came from the web console, a command-line tool or infrastructure-as-code, so the provider can record all of them.

Audit logs have real security value. After an incident they answer "which credentials were used, and what else did they touch?". Used live, they can raise an alert when someone disables logging, creates an access key for the root account or opens a port to the internet. They also show auditors that changes are controlled. Who may do what in the first place is the subject of Identity and Access Management.

A quick look at recent deletions of storage buckets with the AWS Command Line Interface (CLI):

$ aws cloudtrail lookup-events --max-results 1 \
    --lookup-attributes AttributeKey=EventName,AttributeValue=DeleteBucket

{
    "Events": [
        {
            "EventId": "6f1c2a0e-5b7d-4c1e-9a53-2d8e1f4b7c90",
            "EventName": "DeleteBucket",
            "ReadOnly": "false",
            "EventTime": "2026-09-30T14:22:07+00:00",
            "EventSource": "s3.amazonaws.com",
            "Username": "j.devries",
            "Resources": [
                {
                    "ResourceType": "AWS::S3::Bucket",
                    "ResourceName": "shop-test-exports"
                }
            ],
            "CloudTrailEvent": "{\"eventVersion\":\"1.11\", …}"
        }
    ]
}

Cloud monitoring services and OpenTelemetry

Every provider has a built-in family of services for the three signals. They work with the provider's own services without extra setup, which makes them the natural starting point.

Google Cloud groups its services as Google Cloud Observability (formerly Stackdriver). Many organizations also use third-party or open-source tools such as Prometheus and Grafana for metrics, especially when they run on several clouds or on Kubernetes.

Tailing a log group from the terminal, filtered on the JSON field level:

$ aws logs tail /shop/checkout --since 1h --filter-pattern '{ $.level = "ERROR" }'

2026-10-01T09:00:03.412000+00:00 checkout/web/7d3e91 {"time": "2026-10-01T09:00:03Z", "level": "ERROR", "path": "/api/checkout", "status": 503, "latency_ms": 2004}
2026-10-01T09:00:08.107000+00:00 checkout/web/7d3e91 {"time": "2026-10-01T09:00:08Z", "level": "ERROR", "path": "/api/checkout", "status": 500, "latency_ms": 95}

OpenTelemetry: the open standard

If your code sends traces straight to one vendor's product, moving to another product means changing every service. OpenTelemetry (often OTel) solves this: a vendor-neutral set of application programming interfaces (APIs), software development kits (SDKs) and a wire protocol (OpenTelemetry Protocol, OTLP) for producing metrics, logs and traces. It is a Cloud Native Computing Foundation (CNCF) project, formed from the earlier OpenTracing and OpenCensus projects, and AWS, Microsoft and Google all accept its data.

How the collector receives, processes and exports data: see the Collector.