Reliability Patterns

Every service eventually calls something that is slow or down: a database, another team's API, a supplier. The patterns on this page decide whether that failure stays small or spreads. They are about behavior during partial failure; redundancy at the infrastructure level is covered on High Availability, Scaling and Disaster Recovery, and having enough capacity in the first place on Capacity Planning and Performance. The examples follow Kestrel Energy's customer portal, which calls meter-api for usage data, which in turn writes to the database server db01.

Timeouts

A timeout is the maximum time a caller waits for a remote operation, such as opening a connection or getting a response, before it gives up and treats the call as failed. Without one, a single hung dependency can hold every thread and connection of the caller until it stops serving anything. Many client libraries have no timeout, or an extremely long one, by default, so "no timeout" is usually an accident, not a decision.

Marc Brooker's article on timeouts and retries in the Amazon Builders' Library gives the rule: "set a timeout on any remote call … This includes both a connection timeout and a request timeout." It also gives a method for choosing the value: decide what share of healthy calls you are willing to cut off by mistake, say 0.1%, and look up that latency percentile (the 99.9th) of the service you call, adding some padding. With 10,000 recorded meter-api response times:

import random

rng = random.Random(42)
# 10,000 recorded meter-api response times in seconds: mostly fast, a slow tail
latencies = sorted(
    rng.lognormvariate(-2.6, 0.5) if rng.random() < 0.995 else rng.uniform(0.2, 0.75)
    for _ in range(10_000)
)

def percentile(p):
    return latencies[round(p / 100 * (len(latencies) - 1))]

for p in (50, 99, 99.9):
    print(f"p{p:<5} {percentile(p) * 1000:4.0f} ms")

# Which share of normal, healthy calls would each timeout cut off?
for timeout in (0.2, 0.3, 0.5, 0.8):
    too_slow = sum(t > timeout for t in latencies) / len(latencies)
    print(f"timeout {timeout * 1000:3.0f} ms: {too_slow:.2%} false timeouts")

p50      75 ms
p99     259 ms
p99.9   649 ms
timeout 200 ms: 2.80% false timeouts
timeout 300 ms: 0.56% false timeouts
timeout 500 ms: 0.17% false timeouts
timeout 800 ms: 0.00% false timeouts

Ravi Kumar, the portal's tech lead, sets the portal's request timeout for meter-api to 800 ms: p99.9 plus padding. A timeout of 200 ms would look snappy but fail almost 3% of perfectly healthy calls.

Timeouts must also fit together along the chain. If the browser gives up after 5 seconds, a portal that waits 30 seconds for meter-api keeps working on requests nobody is waiting for any more. Google's SRE book calls this the problem of missed deadlines and recommends deadline propagation: the first service sets an absolute deadline, and every call further down gets the time that is left, not a fresh, hard-coded value. The book also shows what an oversized timeout does under partial failure: if 5% of requests never complete and the timeout is 100 seconds, those requests tie up so many threads that in its example the frontend can serve only 19.6% of all requests, an 80.4% error rate caused by 5% of failures.

Retries

Many failures are brief: a dropped connection, a server restarting, one overloaded node behind a load balancer. Trying again a moment later often works, which is why almost every client library can retry. But retries add load at exactly the moment a system is struggling. Brooker's article puts it bluntly: "Retries are 'selfish'", and "when failures are caused by overload, retries that increase load can make matters significantly worse."

A retry storm is a surge of repeated requests from clients retrying failed calls, often multiplied across several layers, that keeps an overloaded or recovering service down long after the original cause has gone. The multiplication is the dangerous part. If every layer of a call chain tries three times, the attempts multiply:

# Each layer tries a call up to 3 times (1 try + 2 retries) before giving up.
# portal -> meter-api -> readings service -> db01: how many calls reach db01
# for ONE user request while db01 is down?
attempts_per_layer = 3

for layers in range(1, 6):
    print(f"{layers} layer(s) retrying: {attempts_per_layer ** layers:3} calls to db01")

print(f"Retry only at the edge:  {attempts_per_layer} calls to db01")

1 layer(s) retrying:   3 calls to db01
2 layer(s) retrying:   9 calls to db01
3 layer(s) retrying:  27 calls to db01
4 layer(s) retrying:  81 calls to db01
5 layer(s) retrying: 243 calls to db01
Retry only at the edge:  3 calls to db01

Brooker's article uses the same numbers: a five-deep stack with retries at every layer increases the load on the database 243 times, and "our best practice is to retry at a single point in the stack." The rules that keep retries safe:

• Retry only what may be repeated. Reading usage data is safe; creating an invoice twice is not. "APIs with side effects aren't safe to retry unless they provide idempotency," the article says. Kestrel's billing export sends an idempotency key per invoice batch, so a retried call can never bill a customer twice (see idempotence and at-least-once delivery).
• Retry only what may succeed. A timeout or a 503 can be retried; a 400 Bad Request or a 403 will fail the same way every time.
• Limit the attempts, per request and overall. Google's SRE book describes a per-request limit of three attempts and a per-client retry budget: retry only while retries are less than 10% of that client's requests. A server-wide budget ("only allow 60 retries per minute in a process", in the book's example) stops a retry storm in its tracks.
• Wait between attempts, longer each time, with some randomness. That is the next section.

Backoff and jitter

Exponential backoff doubles the wait after each failed attempt, up to a cap, so a struggling service gets more and more breathing room. On its own it has a flaw: clients that failed at the same moment also retry at the same moments, in synchronized waves. Jitter fixes that: it is a random amount added to or taken from each retry delay, or from the start time of scheduled work, so that clients that would otherwise act at the same moment spread out over time. (In networking, jitter means something else: the variation in packet delay.)

Brooker's 2015 post on the AWS Architecture Blog compared three variants. The commonly quoted formulas, with base the first delay and cap the maximum:

• Full jitter: a random delay between 0 and min(cap, base × 2attempt).
• Equal jitter: half of that exponential delay, plus a random amount up to the other half.
• Decorrelated jitter: a random delay between base and three times the previous delay, capped.

In his simulations full jitter used the least work, decorrelated jitter finished slightly sooner, and equal jitter was "the loser". Full jitter is the usual default. A simulation of 100 meter gateways that all lost their connection at the same moment and retry five times:

import random
from collections import Counter

BASE, CAP = 0.5, 30.0          # seconds: first backoff, longest backoff
CLIENTS, RETRIES = 100, 5      # 100 meter gateways that all failed at t = 0
SLOT = 0.1                     # count arrivals per 100 ms

def no_jitter(attempt, rng):
    return min(CAP, BASE * 2 ** attempt)

def full_jitter(attempt, rng):
    return rng.uniform(0, min(CAP, BASE * 2 ** attempt))

def arrivals(backoff):
    rng = random.Random(1)
    per_slot = Counter()
    last = 0.0
    for _ in range(CLIENTS):
        t = 0.0
        for attempt in range(RETRIES):
            t += backoff(attempt, rng)
            per_slot[int(t / SLOT)] += 1
        last = max(last, t)
    return per_slot, last

for name, backoff in (("no jitter", no_jitter), ("full jitter", full_jitter)):
    per_slot, last = arrivals(backoff)
    print(f"{name:11}: at most {max(per_slot.values()):3} retries in one 100 ms slot,"
          f" last retry after {last:4.1f} s")

no jitter  : at most 100 retries in one 100 ms slot, last retry after 15.5 s
full jitter: at most  30 retries in one 100 ms slot, last retry after 13.0 s

Without jitter, all 100 retries hit the server in the same 100 milliseconds, five times over: the service gets five spikes instead of a steady trickle. With full jitter the worst moment has 30, and the whole sequence even finishes sooner.

Jitter is just as useful for scheduled work. Brooker recommends it for "all timers, periodic jobs, and other delayed work". Kestrel's 400,000 smart meters report every 15 minutes; if they all report at :00, :15, :30 and :45, meter-api gets fifteen times the average load in the first minute (the arithmetic is on Capacity Planning and Performance). Giving each meter a fixed random offset within the quarter hour flattens the load. The article suggests a jitter that is the same every time for the same host, derived from its name for example, so that a problem caused by timing can be reproduced.

Circuit breakers

A circuit breaker in software is a wrapper around calls to a dependency that, after a number of failures, stops making the calls for a while and fails immediately instead, then lets a trial call through to test whether the dependency has recovered. The name comes from the electrical device, which cuts the current when a circuit is overloaded. Martin Fowler's description, which credits Michael Nygard's book Release It! with popularizing the pattern, has three states:

Here is a minimal circuit breaker around the portal's calls to meter-api, with a fake clock so the example is repeatable:

class CircuitBreaker:
    """Closed -> open after `threshold` failures in a row; after `cooldown`
    seconds one trial call is let through (half-open)."""

    def __init__(self, threshold=3, cooldown=30):
        self.threshold, self.cooldown = threshold, cooldown
        self.state, self.failures, self.opened_at = "closed", 0, None

    def call(self, now, backend_ok):
        if self.state == "open":
            if now - self.opened_at < self.cooldown:
                return "rejected (fast fail)"
            self.state = "half-open"            # cooldown over: try once
        trial = "trial call: " if self.state == "half-open" else ""
        if backend_ok:
            self.state, self.failures = "closed", 0
            return trial + "success"
        self.failures += 1
        if trial or self.failures >= self.threshold:
            self.state, self.opened_at = "open", now
        return trial + "failure"

breaker = CircuitBreaker()
# (time in seconds, is meter-api healthy at that moment?)
calls = [(0, True), (5, False), (6, False), (7, False), (8, False),
         (20, False), (40, False), (55, True), (75, True), (76, True)]
for now, healthy in calls:
    result = breaker.call(now, healthy)
    print(f"t={now:3}s  meter-api {'up  ' if healthy else 'down'}  "
          f"{result:21} -> {breaker.state}")

t=  0s  meter-api up    success               -> closed
t=  5s  meter-api down  failure               -> closed
t=  6s  meter-api down  failure               -> closed
t=  7s  meter-api down  failure               -> open
t=  8s  meter-api down  rejected (fast fail)  -> open
t= 20s  meter-api down  rejected (fast fail)  -> open
t= 40s  meter-api down  trial call: failure   -> open
t= 55s  meter-api up    rejected (fast fail)  -> open
t= 75s  meter-api up    trial call: success   -> closed
t= 76s  meter-api up    success               -> closed

Look at t=55: meter-api is healthy again, but the breaker still rejects the call, because its cooldown restarted when the trial at t=40 failed. That is the cost of the pattern. Brooker's article warns that circuit breakers "introduce modal behavior into systems that can be difficult to test, and can introduce significant addition time to recovery"; AWS limits retries locally with a token bucket instead (the same algorithm as in rate limiting), so that once failures have used up the tokens, retries continue only at a low fixed rate, without a separate open state. Use a circuit breaker where failing fast clearly beats waiting, test its open state on purpose, and, as Fowler advises, alert when it trips: an open breaker is an incident that the users may not have noticed yet.

Load shedding and graceful degradation

When more work arrives than a service can handle, something has to give. The worst option is to accept everything into an ever-longer queue: every request gets slow, many time out at the caller after the server has done the work anyway, and the service stays slow long after the burst has passed. Load shedding is deliberately rejecting part of the incoming work, quickly and cheaply, when a service approaches overload, so that the work it does accept is still served within its objectives. Google's SRE book gives the goal: keep the server from "serving with extremely high latency, or any of the other symptoms associated with overload, while still doing as much useful work as it can." (The power grid uses the same term for disconnecting consumers to save the rest of the system; the idea is the same.)

A simple and effective method is a limit on the queue: beyond it, answer immediately with 503 Service Unavailable and a Retry-After header (see status codes). meter-api writes 600 readings per second; a burst of 2,400 per second arrives for three seconds:

CAPACITY = 600            # readings meter-api can write per second
QUEUE_LIMIT = 1_200       # shed above this: about 2 seconds of work
arrivals = [450, 2_400, 2_400, 2_400, 450, 450, 450, 450]   # burst at :15

def run(limit):
    queue, rows = 0, []
    for n in arrivals:
        space = n if limit is None else max(0, limit - queue)
        accepted = min(n, space)
        queue += accepted
        queue -= min(queue, CAPACITY)
        rows.append((n - accepted, queue / CAPACITY))   # rejected, wait in s
    return rows

print("sec  arrive | no limit: wait | shedding: 503s  wait")
for sec, (n, (_, w1), (shed, w2)) in enumerate(
        zip(arrivals, run(None), run(QUEUE_LIMIT))):
    print(f"{sec:3}  {n:6,} | {w1:10.1f} s    | {shed:12,}  {w2:4.1f} s")

sec  arrive | no limit: wait | shedding: 503s  wait
  0     450 |        0.0 s    |            0   0.0 s
  1   2,400 |        3.0 s    |        1,200   1.0 s
  2   2,400 |        6.0 s    |        1,800   1.0 s
  3   2,400 |        9.0 s    |        1,800   1.0 s
  4     450 |        8.8 s    |            0   0.8 s
  5     450 |        8.5 s    |            0   0.5 s
  6     450 |        8.2 s    |            0   0.2 s
  7     450 |        8.0 s    |            0   0.0 s

Without a limit, the backlog grows to nine seconds and is still eight seconds long four seconds after the burst. Every reading that arrives from the third second of the burst on waits more than five seconds, so a meter gateway with a 5-second timeout gives up on it, and the server writes readings that will be sent again anyway. With shedding, 4,800 readings get a fast 503, retry later with jitter, and everything that was accepted is written within about two seconds. Shedding by priority is even better: reject the catch-up batch import before live readings, and live readings before a customer's login. Shedding on the server is the counterpart of rate limiting, which limits each client.

Graceful degradation

Graceful degradation is designing a service to keep offering a reduced but useful version of its function when a dependency fails or capacity runs short, instead of failing completely. Google's SRE Workbook gives a game as an example: "if the User Data store is unavailable, the game is still playable but uses generic imagery." At Kestrel, when meter-api does not answer within its 800 ms timeout, the portal still shows the customer's account, invoices and contract, with a notice: "Your usage data is delayed. Last update: 14:00." That is a far better experience than an error page, and it keeps the contact center quiet.

The SRE book adds two warnings. Degraded mode "shouldn't trigger very often", and because it is rarely used, it is the code path most likely to be broken when you need it: test it regularly. And measure it. A degraded response is not a full success, so count it in a quality service level indicator, or a permanently degraded portal looks perfect on the dashboard.

Bulkheads

A ship's hull is divided into watertight compartments, so that a leak floods one compartment instead of sinking the ship. A bulkhead in software is the same idea: dividing a shared resource, such as connections, threads, memory or servers, into separate pools for different clients or kinds of work, so that one of them exhausting its pool cannot starve the others.

The meter-api incident of 6 October 2026 shows why (see its postmortem). A connection leak in meter-api used up connections on db01, and because the portal and the billing export shared the same limit, the portal slowed down too. With bulkheads, each service gets its own share:

The price is efficiency: separate pools leave capacity idle that a shared pool could have used. Choose the boundaries where a failure in one part must not take down another. PostgreSQL can limit connections per role with ALTER ROLE … CONNECTION LIMIT; ask your database administrator before changing it on a live server.

Redundancy and failover

All the patterns above assume something else can take over the work. Redundancy provides it, and the infrastructure side is covered elsewhere: removing each single point of failure, load balancers with health checks, N+1 capacity, database failover and Kubernetes readiness probes. Three points connect them to this page:

• Redundancy needs headroom. Two servers at 70% are not redundant: when one fails, the other is at 140%. The calculation is on Capacity Planning and Performance.
• Health checks must check the right thing. A check that only proves the process answers keeps sending traffic to a node whose database connection is broken; a check that tests every dependency takes all nodes out at once when one shared dependency hiccups.
• Failover is a code path too. Like degraded mode, it is rarely used and therefore often broken. Test it, which brings us to the last pattern.

Chaos engineering

Chaos engineering is running controlled experiments that inject real-world failures into a system, such as stopping a server, adding latency or cutting a network link, to check that it keeps working as expected and to find weaknesses before they cause an incident. The Principles of Chaos Engineering define it as "the discipline of experimenting on a system in order to build confidence in the system's capability to withstand turbulent conditions in production", and list among the weaknesses it finds "retry storms from improperly tuned timeouts". Netflix's Chaos Monkey made the idea famous; the cloud providers now offer managed fault-injection services (see testing disaster recovery).

An experiment has four steps, according to the principles:

1. Define a steady state as a measurable output of the system that shows normal behavior, such as throughput, error rate or latency percentiles.
2. Form a hypothesis that the steady state will continue in both a control group and the experimental group.
3. Introduce variables that reflect real-world events: servers that crash, disks that fail, network connections that are cut.
4. Try to disprove the hypothesis by looking for a difference between the two groups.

Kestrel's first experiment, run by Ravi Kumar's team as a planned change on a Thursday morning in the acceptance environment:

Steady state  Synthetic logins at a busy morning's rate (45 req/s):
              99.9% succeed, p99 under 300 ms
Hypothesis    Stopping web02 leaves the steady state unchanged:
              the load balancer moves traffic to web01 within 30 s
Variable      Stop the portal service on web02 at 10:00
Abort if      Error ratio above 1% for 2 minutes, or on request of
              anyone watching; restart web02 (takes 1 minute)
Result        Hypothesis disproved. Traffic moved in 12 s, but p99 rose
              to 1.9 s and 0.4% of logins failed: one node handles
              about 40 req/s within the objective. Not N+1 on a busy
              morning, let alone at the 52 req/s winter peak.

The experiment found what the headroom calculation predicted, but with evidence nobody could argue with. The principles go further than Kestrel did: they recommend running experiments in production, because only real traffic shows real behavior, and automating them to run continuously. They also require the experimenter to "minimize blast radius", and a team new to the practice is right to start in acceptance, during office hours, with an abort switch and people watching (compare blast radius).