Observability & SRE

How to use this topic

This topic goes deeper than the overview of monitoring and observability in the Cloud topic and the IT service management pages on monitoring and event management and incident management. The first ten pages are about observability: the three signals, OpenTelemetry, Prometheus and PromQL, dashboards, logs at scale and alerting. The last eight are about site reliability engineering: service level objectives, error budgets, on-call, incident response, postmortems, capacity planning and reliability patterns.

The Prometheus and Alertmanager output on these pages was recorded with real runs on a test machine, against small test programs that serve invented Kestrel Energy metrics. Output from tools that were not installed, such as Grafana, Loki or the OpenTelemetry Collector, is marked illustrative. Every query and rule file is checked with the Prometheus tools, and the calculations are short programs with their exact output.

Pages are tagged with the certifications they help with: PCA (Prometheus Certified Associate) on the observability, Prometheus and SLI pages, and OTCA (OpenTelemetry Certified Associate) on the pages about tracing and OpenTelemetry; both exams are from the Cloud Native Computing Foundation and the Linux Foundation. The SRE practice pages go beyond those exams and are untagged.

Learning path

1. 01What Is Observability?The difference between monitoring and observability, the three signals (metrics, logs and traces), and the frameworks that tell you what to watch: the four golden signals, RED and USE.BeginnerPCAOTCA
2. 02MetricsNumbers over time: counters, gauges, histograms and summaries, labels and cardinality, sampling intervals, and why averages hide the problems that percentiles show.BeginnerPCA
3. 03LogsWriting logs that help: structured logging, log levels, correlation IDs, what not to log, retention and privacy rules, and getting logs off the machine.BeginnerPCA
4. 04Distributed TracingFollowing one request through many services: traces and spans, context propagation with W3C Trace Context, sampling, and reading a trace to find the slow part.IntermediatePCAOTCA
5. 05OpenTelemetryThe vendor-neutral standard for telemetry: APIs and SDKs, automatic and manual instrumentation, the Collector and its pipelines, OTLP, and semantic conventions.IntermediateOTCA
6. 06PrometheusHow Prometheus collects metrics: the pull model, exporters and the exposition format, scrape configuration, service discovery, storage and retention.IntermediatePCA
7. 07PromQLAsking Prometheus questions: instant and range vectors, label matchers, rate and increase, aggregation with sum by, histogram_quantile, and recording rules.IntermediatePCA
8. 08DashboardsTurning telemetry into dashboards people use: Grafana basics, panels and variables, layouts built on golden signals, and the habits that make dashboards misleading.BeginnerPCA
9. 09Logs at ScaleStoring and searching logs from many machines: Loki and LogQL, Elasticsearch and OpenSearch, indexing trade-offs, sampling and the cost of keeping everything.Intermediate
10. 10AlertingAlerts that wake the right person for the right reason: alert rules, Alertmanager routing, grouping, inhibition and silences, symptom-based alerting and fighting alert fatigue.IntermediatePCA
11. 11What Is Site Reliability Engineering?Where SRE comes from and what it means in practice: reliability as a feature, toil and its limits, the relation between SRE, DevOps and IT operations, and the core practices.Beginner
12. 12SLIs, SLOs and SLAsMeasuring reliability from the user's point of view: choosing service level indicators, setting objectives, how they differ from agreements, and the number of nines.BeginnerPCA
13. 13Error Budgets and Burn-Rate AlertsTurning an SLO into a budget for failure: calculating the error budget, error budget policies, burn rate, and multi-window alerts that catch both fast and slow burns.Intermediate
14. 14On-CallBeing on call without burning out: rotations and handoffs, escalation policies, runbooks, paging versus tickets, compensation and rest, and measuring the load.Beginner
15. 15Incident Response in PracticeWhat happens when it breaks: declaring an incident, roles such as incident commander, communication and status pages, keeping a timeline, mitigating before fixing, and closing out.Intermediate
16. 16PostmortemsLearning from incidents: blameless postmortems, writing the timeline and contributing factors, why there is rarely one root cause, action items that get done, and sharing what you learned.Intermediate
17. 17Capacity Planning and PerformanceMaking sure there is enough before it runs out: utilization and saturation, headroom, forecasting growth, load testing, queues and Little's Law, and latency under load.Intermediate
18. 18Reliability PatternsDesign choices that keep services up when parts fail: timeouts, retries with backoff and jitter, circuit breakers, rate limiting, graceful degradation, redundancy and chaos engineering.Intermediate

Observability & SRE glossary

Short definitions of the terms used in this topic. The full glossary covers every topic.