High Availability, Scaling and Disaster Recovery

Availability, durability and fault tolerance

These three words are often mixed up, but they answer different questions:

• Availability: can users use the service right now? Measured as the percentage of time it works.
• Durability: is the data still there and intact, even if it is briefly unreachable? Cloud object storage is designed for extremely high durability (see cloud storage). Data can be durable but unavailable, for example during an outage.
• Fault tolerance (explained on the RAID page): the ability to keep working without interruption when a component fails.

High availability (HA) is a design goal: a system built so that it stays available for a very high share of the time, by having no single component whose failure takes it down and by recovering from failures quickly and automatically. HA usually accepts a short disruption — a few seconds while a load balancer stops sending traffic to a failed server — where full fault tolerance would avoid even that, at a higher price.

Measuring availability: the nines

Availability is the time a service worked divided by the total time, usually written as a percentage with a number of nines: "three nines" is 99.9%. The numbers look close together, but the downtime they allow is not. This program works it out, using a 365-day year and a month of one twelfth of a year, rounded down to two units:

YEAR = 365 * 24 * 60 * 60  # seconds in a 365-day year
PERIODS = {"year": YEAR, "month": YEAR / 12, "week": 7 * 24 * 60 * 60}


def duration(seconds):
    """Format seconds as the two largest units, e.g. '8 h 45 min'."""
    seconds = round(seconds)
    days, rest = divmod(seconds, 86400)
    hours, rest = divmod(rest, 3600)
    minutes, secs = divmod(rest, 60)
    parts = [(days, "d"), (hours, "h"), (minutes, "min"), (secs, "s")]
    while parts[0][0] == 0 and len(parts) > 1:
        parts.pop(0)
    return " ".join(f"{value} {unit}" for value, unit in parts[:2])


print(f"{'Availability':<14}{'per year':>16}{'per month':>16}{'per week':>16}")
for percent in ("99", "99.9", "99.95", "99.99", "99.999"):
    down = 1 - float(percent) / 100
    row = "".join(f"{duration(down * s):>16}" for s in PERIODS.values())
    print(f"{percent + '%':<14}{row}")

Availability          per year       per month        per week
99%                   3 d 15 h      7 h 18 min      1 h 40 min
99.9%               8 h 45 min     43 min 48 s      10 min 5 s
99.95%              4 h 22 min     21 min 54 s       5 min 2 s
99.99%             52 min 34 s      4 min 23 s       1 min 0 s
99.999%             5 min 15 s            26 s             6 s

Every extra nine cuts the allowed downtime by a factor of ten — and usually multiplies the cost. Five nines (about five minutes a year) leaves no room for a human to even notice a problem, so it needs fully automatic failover. Ask what the business actually needs: an internal reporting tool and a payment service deserve different targets. SLO tooling usually counts a 30-day month (43 min 12 s at 99.9%) instead; see the nines, and which month.

SLA, SLO and SLI

• A service level agreement (SLA) is a formal commitment from a provider to a customer about a service's level — usually monthly availability — and what happens if it is missed: typically a service credit, a partial refund on the bill.
• A service level objective (SLO) is an internal target for a measured value, such as "99.95% of requests succeed over 30 days". It is set stricter than the SLA, so the team reacts before the contract is breached.
• A service level indicator (SLI) is the measurement itself: the actual percentage of successful requests, or of requests answered within 300 milliseconds.

Cloud providers publish SLAs per service, and the percentage often depends on how you deploy: a single virtual machine usually gets a lower SLA than several spread over availability zones. An SLA is a financial promise, not a guarantee that nothing fails, and the credit rarely covers what an outage costs your business. This is the short version; choosing SLIs and setting SLOs in practice is on SLIs, SLOs and SLAs.

Components in series and in parallel

A service depends on a chain of components. When all of them must work (in series), the total availability is the product of their availabilities, so it is always lower than that of the weakest link. When any one of several identical components is enough (in parallel), the group is only down when all of them are down at once: 1 minus the product of their unavailabilities.

from math import prod


def series(*parts):
    """All parts are needed: the chain is up only if every part is up."""
    return prod(parts)


def parallel(*parts):
    """Any one part is enough: down only if every part is down at once."""
    return 1 - prod(1 - a for a in parts)


load_balancer = 0.9999
web_server = 0.999
database = 0.9995

one_web = series(load_balancer, web_server, database)
two_web = series(load_balancer, parallel(web_server, web_server), database)

for label, a in (("One web server", one_web), ("Two web servers", two_web)):
    minutes = (1 - a) * 365 * 24 * 60
    print(f"{label:<16} {a:.4%}  about {minutes:,.0f} minutes down per year")

print(f"Two web servers in parallel alone: {parallel(web_server, web_server):.4%}")

One web server   99.8401%  about 841 minutes down per year
Two web servers  99.9399%  about 316 minutes down per year
Two web servers in parallel alone: 99.9999%

Two lessons. First, every component you add in series lowers the total, so a long chain of "highly available" services can still add up to something mediocre. Second, the parallel formula assumes the copies fail independently. Two web servers in the same rack, on the same power feed, running the same faulty release, can fail together; the math then overestimates the result. Independence is exactly what spreading copies over availability zones buys.

Designing for high availability

Single points of failure

A single point of failure (SPOF) is any component whose failure alone stops the whole service. The first step of any HA design is to list them: the one web server, the one database, the one network link, but also the one person who knows the password, the one DNS (Domain Name System) provider, or the one region. Then decide for each whether to make it redundant or to accept the risk.

Load balancers and health checks

Redundant servers only help if traffic avoids the broken ones. A load balancer spreads requests over a group of servers and stops sending to a server that fails its health check. A health check is a test that a load balancer or orchestrator runs repeatedly against each instance — usually an HTTP (Hypertext Transfer Protocol) request to a path such as /healthz that must answer within a time limit — to decide whether it can receive traffic. After a set number of failures the instance is taken out of rotation; after a set number of successes it is put back. Autoscaling groups and Kubernetes (see Kubernetes basics) use the same signal to replace unhealthy instances.

A good health check tests what users need: an endpoint that returns "OK" while the application cannot reach its database keeps a broken server in rotation. But be careful about checking shared dependencies too deeply: if the database has a short hiccup and every server then reports unhealthy, the load balancer removes all of them at once. Kubernetes splits this into separate checks: liveness, readiness and startup probes.

The managed load balancers are Elastic Load Balancing (with Application and Network Load Balancers) on Amazon Web Services (AWS), Azure Load Balancer and Azure Application Gateway on Azure, and Cloud Load Balancing on Google Cloud. They are themselves run redundantly by the provider.

Multi-AZ and multi-region

Spreading instances over several availability zones (AZs) in one region protects against the failure of a data center or a zone, with low latency between the copies, so data can be replicated synchronously. This is the standard HA design in the cloud, and many managed services offer it as a single setting.

Spreading over several regions protects against the loss of a whole region, a rare but real event, and can bring the service closer to users. It is much harder: the distance makes replication asynchronous (so recent writes can be lost in a failover), data residency rules may limit where data may go, and the cost roughly doubles. Multi-region is usually a disaster-recovery decision rather than everyday HA.

Stateless design and scaling

An application server is stateless when it keeps nothing between requests that another server would need: sessions live in a shared cache or database, uploaded files in object storage. Then any server can handle any request, a failed server loses nothing, and adding capacity is as simple as starting another copy. That is horizontal scaling, and with autoscaling it also makes the service resilient to load peaks, which cause as many outages as broken hardware (see headroom). State does not disappear; it moves to a few components (databases, queues, storage) that are built for redundancy, usually as managed services.

Disaster recovery

High availability handles the expected failures: a server, a disk, a zone. Disaster recovery (DR) is the set of plans, procedures and infrastructure to restore a service after an event that HA cannot absorb: the loss of a region, a ransomware attack, a destructive mistake that was faithfully replicated to every copy, or a provider account that is compromised. Notice the last two: redundancy copies corruption and deletion as quickly as good data, which is why DR always includes separate, protected backups.

Two numbers drive every DR decision. They are explained on the backup and recovery page: the recovery point objective (RPO, how much data, measured in time, you can afford to lose) and the recovery time objective (RTO, how long the service may be down). The smaller both are, the more the DR setup costs.

Disaster recovery strategies

The common strategies form a scale from cheap and slow to expensive and fast. The names below follow AWS's widely used terms; the ranges are typical orders of magnitude, not promises — the real numbers depend on your data volume, automation and practice.

In an active-active setup, two or more sites (regions or data centers) all serve live traffic at the same time, so losing one only means the others take its share. The opposite is active-passive: one site serves traffic and another waits to take over (pilot light and warm standby are forms of it). Active-active is the hardest to build, because if every site accepts writes, conflicting writes in different regions must be resolved.

Different parts of one system can use different strategies: the order database might be warm standby while the internal reporting system is backup and restore. Whatever the strategy, keep at least one backup copy that an attacker with your production credentials cannot delete: a separate account, immutable (write-once) storage, or both.

Testing recovery

A DR plan is only real once it has been carried out. Restores fail for ordinary reasons: a backup job that silently stopped months ago, a missing encryption key, a runbook that refers to a server that no longer exists, or an RTO that assumed things nobody had ever timed.

• Restore tests: regularly restore a backup into a separate environment and check the data.
• Game days: planned exercises in which the team simulates a disaster — "the primary region is gone" — and follows the runbook while the clock runs, then fixes what went wrong.
• Chaos engineering: deliberately injecting failures (stopping instances, adding network delay) in a controlled way, often in production, to find weaknesses before they find you. Netflix's Chaos Monkey made the idea famous; AWS Fault Injection Service and Azure Chaos Studio offer it as a service (see Chaos engineering).

Resilience is a shared responsibility

As with security (see the shared responsibility model), the provider and the customer each own a part of resilience. The provider is responsible for the resilience of the cloud: data centers, power, cooling, the network between zones, and the redundancy inside managed services. The customer is responsible for resilience in the cloud: choosing multi-AZ options, running more than one instance, configuring health checks and autoscaling, making backups, writing and testing the DR plan.

A single virtual machine in a single zone is exactly as available as that machine. The cloud makes high availability easy to build, but it does not build it for you.