What Is Site Reliability Engineering?

Where SRE comes from

Site reliability engineering (SRE) runs production systems with software engineering methods: reliability targets set from the user's point of view, measured as service level objectives, protected with error budgets and defended with automation (the ITSM page gives the short version). It started at Google in 2003, when Benjamin Treynor Sloss was asked to run a "Production Team" of seven engineers. He had always been a software engineer, so he organized operations the way a software engineer would. His own one-line definition, from the introduction of the book Site Reliability Engineering (2016):

"SRE is what happens when you ask a software engineer to design an operations team."

In a classic setup, developers build features and hand them over; an operations team keeps them running, mostly by hand. The two groups have opposite incentives. Developers are rewarded for shipping change, operations for stability, and change is the main cause of outages. The result is a slow, permanent tug of war: launch reviews, change freezes, and blame after every incident.

SRE replaces that tug of war with a few rules that both sides agree on in advance: a measured reliability target, a budget for failure that follows from it, and a cap on manual work. Google published three books about it, all free to read on sre.google: Site Reliability Engineering (2016), The Site Reliability Workbook (2018), with practical examples, and Building Secure and Reliable Systems (2020). This topic quotes the first two often.

Reliability is a feature, and 100% is wrong

Reliability is the most important feature of any service: if people cannot log in, nothing else matters. But it is still a feature, which means it has a cost and competes with other features for time and money. The SRE book calls this attitude embracing risk: deliberately choosing how unreliable a service may be, instead of pretending it can be perfect. Its chapter on the subject puts it bluntly:

"It turns out that past a certain point, however, increasing reliability is worse for a service (and its users) rather than better!"

Three reasons why:

• Cost grows steeply. Each extra nine is an order of magnitude less downtime: 99.9% allows 43 minutes per 30 days, 99.99% only about 4. Getting there means redundant sites, automatic failover and slower, more careful releases.
• Users cannot tell the difference. The book's example: "a user on a 99% reliable smartphone cannot tell the difference between 99.99% and 99.999% service reliability!" Their home Wi-Fi, mobile network and laptop fail far more often than your last nine.
• Change stops. The only way to be close to 100% is to change almost nothing. Security fixes and new features then wait, which is a risk of its own.

So SRE picks a target that is good enough for users, writes it down as a service level objective, and treats the gap between that target and 100% as an error budget to spend on releases, experiments and maintenance. The error budget page shows how that budget turns into release decisions and alerts.

Toil and the 50% cap

Toil is the SRE word for a specific kind of operational work (the ITSM page has a short version). The SRE book's chapter on eliminating it defines toil as work "tied to running a production service that tends to be manual, repetitive, automatable, tactical, devoid of enduring value, and that scales linearly as a service grows." The last property is the dangerous one: when the number of customers doubles, the toil doubles, and the team does not.

Not everything unpleasant is toil:

• Overhead such as team meetings, hiring, goal setting and HR paperwork is necessary work, but not tied to running the service. It is not toil.
• Grungy work with lasting value is not toil either. Cleaning up a messy alerting configuration once is boring, but it leaves the service better.
• Incident reviews and postmortems require judgment and produce lasting improvements, so they are not toil.

Google's SRE organization has an advertised goal of keeping toil below 50% of each engineer's time, because, in the book's words, "toil tends to expand if left unchecked and can quickly fill 100% of everyone's time." Quarterly surveys there put the average at about 33%, with large differences between people. The cap is a safety valve: when a team is over it for a long time, operational work is pushed back to the developers or staff is added, until the balance is restored.

Measuring toil

You cannot cap what you do not measure. Ravi Kumar, tech lead of Kestrel Energy's customer portal team, asks the five people on the team to tag a month of time entries. The result is a short toil inventory. Four of the five items grow with the number of smart meters; the re-run of the nightly billing export after a database lock on db01 does not. The script below shows what happens to the team's toil share if the number of meters grows by half or doubles, and what one piece of engineering work changes:

TEAM_SIZE = 5           # Ravi, Sofia, Kwame, Priya, Bram
HOURS_PER_WEEK = 40
CAP = 0.50              # keep toil below half of the team's time

# Hours per week, measured over the last month.
# True = the work grows with the number of smart meters.
toil = {
    "On-call interrupts and pages": (22, True),
    "Unlocking customer portal accounts": (12, True),
    "Copying usage CSVs for customer service": (9, True),
    "Restarting the meter-api importer": (6, True),
    "Re-running the billing export after a db01 lock": (5, False),
}


def report(label, items, growth):
    capacity = TEAM_SIZE * HOURS_PER_WEEK
    hours = sum(h * growth if scales else h for h, scales in items.values())
    share = hours / capacity
    status = "OVER THE CAP" if share > CAP else "ok"
    print(f"{label:<22} toil {hours:5.1f} h of {capacity} h = {share:.0%}  {status}")


for growth in (1.0, 1.5, 2.0):
    report(f"Meters x{growth:.1f}", toil, growth)

# Engineering work: a self-service unlock page removes one item for good.
del toil["Unlocking customer portal accounts"]
report("x2.0, self-service", toil, 2.0)

Meters x1.0            toil  54.0 h of 200 h = 27%  ok
Meters x1.5            toil  78.5 h of 200 h = 39%  ok
Meters x2.0            toil 103.0 h of 200 h = 52%  OVER THE CAP
x2.0, self-service     toil  79.0 h of 200 h = 40%  ok

At today's size the team is fine. The point of the exercise is the trend: if the meter roll-out doubles the fleet, toil takes over half the team's time, and the work that would remove it no longer gets done. A self-service unlock page is a few weeks of work once; the 12 hours a week (and growing) it removes are gone for good. The same reasoning picks the next candidate: the weekly importer restart is usually a symptom of a bug, and fixing the bug beats scripting the restart.

Engineering work

The other half of the cap is what the "engineering" in SRE refers to. Engineering work, in the SRE book's sense, is novel work that needs human judgment and leaves a permanent improvement: it lets the same team run a bigger service, or more services, without growing at the same pace. Examples:

• Writing a tool that replaces a manual procedure, or changing the software so the procedure is no longer needed.
• Defining service level objectives and the alerts that defend them (see SLIs, SLOs and SLAs).
• Designing for failure: timeouts, retries and graceful degradation (see reliability patterns).
• Capacity models and load tests (see capacity planning and performance).
• Improving monitoring so the next incident is found faster (see what is observability).

The book adds that engineering work lets an SRE organization grow "sublinearly with service size". That is the real test: if the team must grow as fast as the service, it is doing operations by hand, whatever its job titles say.

SRE, DevOps and ITIL

DevOps emerged in 2008 as a movement to bring development and operations together: shared responsibility, heavy automation, and engineering practices applied to operations. The SRE book describes the relation like this:

"One could view DevOps as a generalization of several core SRE principles to a wider range of organizations, management structures, and personnel. One could equivalently view SRE as a specific implementation of DevOps with some idiosyncratic extensions."

In short: DevOps says what culture to aim for, SRE gives concrete mechanisms to get there (SLOs, error budgets, a toil cap, blameless postmortems). The DevOps Research and Assessment DORA metrics measure delivery; SLOs measure what users get. Healthy teams watch both.

ITIL is not the opposite of either. The ITIL and DevOps section shows how the practices fit together, and ITIL (Version 5) includes SRE and observability vocabulary. In an organization that uses ITIL, SRE practices change how the work is done, not the records: an outage is still an incident, a recurring cause is still a problem, and a release is still a change. What SRE adds is the user-facing target, the budget and the discipline to remove manual work.

How organizations set up SRE

There is no single SRE org chart. Common models, often mixed:

• A central SRE team that runs the most critical services together with the developers who build them. Google's model; it needs enough engineers to staff its own on-call rotations.
• Embedded SREs: a reliability specialist sits in a product team for a while, introduces SLOs, alerting and automation, and moves on.
• A platform team that offers shared tools (monitoring, deployment pipelines, SLO dashboards) so product teams can run their own services well.
• No SRE team at all: product teams own their services completely and adopt SRE practices themselves.

The last model is often summed up as you build it, you run it: the team that develops a service is also on call for it and answers for its reliability. The phrase is usually credited to Amazon's CTO Werner Vogels, in a 2006 interview. It gives developers a direct reason to build services that do not wake them up at night. It only works if the team has the time, the access and the people to do so; a five-person team cannot carry a 24/7 rotation alone forever (see on-call).

Production readiness reviews

When operations responsibility moves from one team to another, or a new service goes live, many organizations run a production readiness review (PRR): a structured check that the service can be run safely before anyone carries a pager for it. Google's SRE teams use such a review before they take on a service; the questions are useful for any go-live:

• Are there SLOs, and do alerts fire on them rather than on CPU?
• Is there a dashboard that shows the four golden signals (see what is observability)?
• Does every alert link to a runbook, and has someone other than the author tried it?
• Can a release be rolled back in minutes, and has that been tested?
• Is capacity known, with headroom for the next peak (for Kestrel: the meter readings at every quarter hour, the billing run at 02:30)?
• Are backups restored regularly, and are dependencies and their owners known?

A PRR is not a gate to block launches. Its output is a short list of gaps with owners and dates, the same kind of list ITIL calls acceptance criteria for operations.

The core practices

The SRE book lists what an SRE team is responsible for: "availability, latency, performance, efficiency, change management, monitoring, emergency response, and capacity planning". The rest of this topic covers each practice on its own page:

Change management is on the list for a reason: the SRE book reports that roughly 70% of Google's outages were caused by changes to a live system. Progressive rollouts, fast detection and safe rollback are therefore reliability work, not paperwork.

Adopting SRE practices without an SRE team

Kestrel Energy has no SRE team, and does not plan one. Ravi's portal team adopts the practices inside the existing ITIL processes, in this order:

1. One SLO for the most important journey: 99.9% of portal requests succeed over a rolling 30 days, agreed with Sanne de Vries as service owner.
2. An error budget policy that says in advance what happens when the budget runs out, signed by Sanne and by Yara Haddad, the release manager.
3. Alerts on the SLO instead of on CPU and disk, so pages mean users are affected.
4. A toil inventory every quarter, like the one above, with one item engineered away each time.
5. Blameless postmortems for incidents that use a large part of the budget, with action items tracked as problem records.

None of this needs new job titles. It needs agreement between the people who build the service, the people who run it and the people who pay for it, and the discipline to follow the rules when they are inconvenient.