Disaster Recovery and Site Resilience

When the whole site is gone

Everything else in a data center — dual power feeds, N+1 cooling, RAID, clusters — is built so that a single failed component does not stop the service (see availability tiers and redundancy). All of it stands in the same building. A fire, a flooded basement, a collapsed roof, a power outage that outlasts the generator fuel, or a police cordon around the block takes out every redundant component at once.

Disaster recovery (DR) is the answer to that: the plans, sites and technology to bring IT services back elsewhere. Its targets come from the business: the business impact analysis sets, per service, the recovery point objective (RPO, how much data in time may be lost) and the recovery time objective (RTO, how long the service may be down). This page is about how a data center team turns those numbers into technology.

DR sites

A DR site (recovery site) is a second location, far enough from the primary data center not to share its disasters, where IT services can be restarted when the primary is lost. The classic types differ in how much is ready, which is explained with a comparison table on the business continuity page:

• Hot site: hardware running, data replicated continuously; services can take over in minutes to hours.
• Warm site: hardware and network in place, but data must be restored or brought up to date first; hours to days.
• Cold site: space, power and cooling only; equipment has to be delivered and installed; days to weeks.
• Cloud site: VMs are replicated to a public cloud and started there only when needed, so you pay mostly for storage until the day you fail over.

Choosing the location is a balance. Too close, and both sites share the same flood plain, power grid substation or road access. Too far, and synchronous replication becomes impossible and staff cannot reach the site quickly. Many organizations use a colocation facility as their DR site. Kestrel Energy does it the other way round: it runs its billing platform in a rented colocation facility and keeps a hot DR environment for it in the small data center at its head office, plus backup copies in a public cloud.

Check what the sites really share: the same telecom carrier in the same duct, the same DNS provider, the same identity system, the same key administrator. An independent second site is surprisingly often not independent at all.

Replication

To recover at another site, the data must already be there. Replication keeps a copy at the DR site up to date, and it can happen at several layers:

• Storage array: the array copies changed blocks of a volume to an array at the other site. It works for every operating system, but both arrays usually have to come from the same vendor.
• Hypervisor: VM replication products copy changed blocks of virtual disks to another cluster, independent of the storage hardware.
• Application or database: the database ships its transaction log to a replica (see database replication). The application knows what a consistent state is, so this is the most reliable form for databases.
• File: file servers replicate changed files to a copy elsewhere, sometimes in both directions.

Synchronous replication

With synchronous replication, a write is acknowledged to the application only after it has been stored at both sites. If the primary site disappears, every acknowledged write is also at the DR site: an RPO of zero. The price is speed. Every write now waits for a round trip to the other site, and light in fiber needs about 5 µs per kilometer each way — roughly 1 ms of round trip per 100 km of fiber route, before any switch or array adds its own delay.

The delay matters most for workloads that write one thing after another and wait each time, such as a database flushing its transaction log. This program adds the fiber round trip to a fast local write and shows how many such serial writes per second remain:

FIBER_US_PER_KM = 5      # one-way delay in fiber, about 5 microseconds per km
LOCAL_WRITE_MS = 0.25    # a write acknowledged by the local flash array

print(f"{'Fiber route':>11}  {'Round trip':>10}  {'Write time':>10}  {'Serial writes/s':>15}")
for route_km in (5, 25, 80, 150, 400, 1000):
    round_trip_ms = 2 * route_km * FIBER_US_PER_KM / 1000
    write_ms = LOCAL_WRITE_MS + round_trip_ms
    writes_per_second = 1000 / write_ms
    print(f"{route_km:>8} km  {round_trip_ms:>7.2f} ms  {write_ms:>7.2f} ms  {writes_per_second:>15,.0f}")

Fiber route  Round trip  Write time  Serial writes/s
       5 km     0.05 ms     0.30 ms            3,333
      25 km     0.25 ms     0.50 ms            2,000
      80 km     0.80 ms     1.05 ms              952
     150 km     1.50 ms     1.75 ms              571
     400 km     4.00 ms     4.25 ms              235
    1000 km    10.00 ms    10.25 ms               98

Compared with 5 km, a 25 km route already costs 40% of the serial write rate, and at 400 km only about 7% is left. This is why synchronous replication is a metro technology, typically used up to around 100 km of fiber. Note that the fiber route is usually much longer than the distance on a map. Vendors therefore specify a maximum round-trip latency rather than a distance: VMware vSAN stretched clusters, for example, require less than 5 ms round trip between the two data sites (as of 2026).

Asynchronous replication

With asynchronous replication, a write is acknowledged as soon as the primary site has stored it, and the change is sent to the DR site afterward — continuously, or in cycles every few seconds or minutes. Distance no longer slows the application, so the DR site can be hundreds or thousands of kilometers away. The cost is an RPO above zero: whatever had not arrived yet when the primary was lost is gone.

How much that is depends not only on the interval but also on the bandwidth. This program models snapshot-based replication every five minutes over a link with 100 MB/s usable, during a month-end batch run that changes far more data than usual:

INTERVAL_S = 300          # a replication cycle starts every 5 minutes
LINK_MB_PER_S = 100       # usable bandwidth of the 1 Gbit/s link to the DR site

# Changed data per 5-minute cycle, in MB: quiet hours, then a month-end batch run.
changes = [4_000, 5_000, 4_500, 42_000, 45_000, 40_000, 6_000, 5_000, 4_000, 4_500]

backlog_s = 0             # transfer time still owed from earlier cycles
print("Cycle  Changed   Transfer  Worst-case data loss")
for cycle, changed_mb in enumerate(changes, start=1):
    transfer_s = changed_mb / LINK_MB_PER_S
    backlog_s = max(0, backlog_s - INTERVAL_S) + transfer_s
    worst_loss_s = INTERVAL_S + backlog_s
    print(f"{cycle:>5}  {changed_mb / 1000:>5.1f} GB  {transfer_s:>6.0f} s  "
          f"{worst_loss_s / 60:>6.1f} min")

Cycle  Changed   Transfer  Worst-case data loss
    1    4.0 GB      40 s     5.7 min
    2    5.0 GB      50 s     5.8 min
    3    4.5 GB      45 s     5.8 min
    4   42.0 GB     420 s    12.0 min
    5   45.0 GB     450 s    14.5 min
    6   40.0 GB     400 s    16.2 min
    7    6.0 GB      60 s    12.2 min
    8    5.0 GB      50 s     8.0 min
    9    4.0 GB      40 s     5.7 min
   10    4.5 GB      45 s     5.8 min

In quiet hours, the worst case is a little over the five-minute interval. During the batch run, each cycle needs longer than five minutes to transfer, a backlog builds up, and the real RPO climbs to 16 minutes — while the dashboard may still show "replication every 5 minutes". Size the link for the peaks, and monitor the actual lag, not the configured interval.

Other replication terms

• Constant (continuous) replication sends every change as it happens; background replication collects changes and sends them at intervals or when there is capacity.
• Application-consistent replication coordinates with the application so that the copy is a state the application can start from cleanly, not just a crash-consistent image.
• Mirroring keeps an exact, usually synchronous, copy of a volume elsewhere.
• Bidirectional replication allows changes at both ends. It needs file locking or conflict resolution, so that two users cannot change the same file at the two sites at the same time and overwrite each other's work.

Active-passive and active-active sites

In an active-passive design, one site runs production while the other receives replicated data and stands ready; it takes over only after a failover. It is simpler to build and reason about, the DR site can be smaller or run test workloads, and only one copy of each application accepts writes. The downside is that the passive site is rarely exercised, and its problems surface on the worst possible day.

In an active-active design, both sites serve production at the same time. Losing one means the other carries the full load, so each site must have capacity for everything — or the business must accept degraded service. Within a metro distance, active-active is often built as a stretched cluster: one virtualization cluster whose hosts and synchronously mirrored storage are spread over two sites. When one site fails, VM high availability restarts its VMs at the other site within minutes, with no loss of stored data (in-memory state is lost, as in any HA restart).

Some organizations combine the two: a stretched cluster between two nearby buildings for zero data loss, plus asynchronous replication to a third, distant site for a regional disaster.

Split-brain and the quorum witness

Two sites connected by a network link face an awkward question when that link fails: is the other site down, or just unreachable? If both sites conclude that the other one is dead, both start the same VMs or both accept writes to the same volume. Split-brain is that situation: two parts of a cluster that can no longer communicate each believe they are the surviving half and run the same workload independently. The two copies of the data then diverge, and merging them afterward ranges from painful to impossible. It is worse than an outage.

The standard defense is a majority vote. Each site has a vote, and a third vote sits at a third location: a quorum witness, a small component (a witness host, a file share or a cloud service, depending on the product) that holds no production data and only helps the cluster decide which side may continue. A side that can reach the witness has two of three votes and keeps running; a side that is isolated has one vote and stops its workloads on purpose.

Placement is everything. A witness at site A breaks the moment site A burns down, because site B is then alone with one vote out of three and stops as well. The witness belongs in a third location — a small colocation rack or a cloud VM — with network paths to each site that do not pass through the other. The same idea, an odd number of votes so that only one partition can have a majority, is used by database clusters, Ceph monitors and Kubernetes control planes.

Failover and failback

Failing over

A failover moves production to the DR site. Within a stretched cluster it can be automatic, but a failover between distant sites is normally a decision by people, because it is disruptive and hard to reverse, and because a site that only looks dead from the monitoring system may still be running. The DR plan says who may declare a disaster and start the failover. The technical steps, written in a runbook and ideally automated:

1. Stop or fence what remains of the primary, if it is still reachable, so it cannot keep writing (and cause split-brain).
2. Promote the replicas at the DR site to read-write: storage volumes, VM replicas, database replicas.
3. Start the services in dependency order: identity and DNS first, then databases, then application servers, then front ends.
4. Redirect users: change DNS records, move or announce IP networks at the DR site, or switch the load balancer. Networks that are stretched between sites keep their addresses; otherwise servers come up with new IP addresses at the DR site, and everything that refers to the old ones must follow.
5. Check that the services work, tell users and the service desk, and record the times for the evaluation.

Failing back

Failback is the planned return of production from the DR site to the primary site once it has been repaired. It is often harder than the failover itself: while the DR site ran production, all new data was written there, so replication must first run in the reverse direction until the primary is up to date. Only then is there a second, controlled switchover, in a maintenance window, with the same steps in the other direction.

Two things go wrong easily. If the primary storage was not destroyed, its old data must not be brought back online as if it were current; the DR site's copy is now the truth. And if the reverse replication was never configured or tested, the organization can be stuck at the DR site for weeks.

Testing the DR plan

A DR test is a planned exercise that checks whether IT services can actually be recovered at the DR site as the plan describes and within the agreed RTO and RPO. Tests range from talking through the plan to really moving production; the full range of exercise types, from walkthrough and tabletop exercise to full interruption, is described on the business continuity page. For a data center team, three levels matter most:

• Tabletop: the infrastructure, application and network teams walk through a scenario ("the main data center has lost power and the generator has failed") with the runbook in hand. It finds missing steps, unclear responsibilities and outdated contact details.
• Simulated (isolated) failover: replicas are started at the DR site in an isolated network bubble while production keeps running. Most replication products can do this without stopping replication. It proves that the VMs boot and the applications work, and it measures the technical recovery time.
• Live failover: production really moves to the DR site for a while, and then fails back. It is the only test that proves the network changes, the users' access and the failback, and it carries real risk, so it is planned as a change with a rollback plan.

Measure and keep: the time from declaring the disaster to each service being back, the actual data loss, every step that had to be improvised. Then update the runbooks and test again. DR plans decay quickly, because every new server, firewall rule or IP address change can quietly break them.