Backup and Recovery

What a backup must survive

The basics — what a backup is, full, incremental and differential backups, and the recovery point and recovery time objectives (RPO and RTO) — are explained in the Operating Systems topic. This page is about doing it for a whole data center: hundreds of virtual machines (VMs), databases and file shares, tens of terabytes per night, and an attacker who knows exactly where backups live.

A data center already has several mechanisms that look like protection, and each one covers a different failure:

The key words are separate and point in time. A backup is a copy on a different system, under different credentials, that still holds yesterday's and last month's data after today's data has been destroyed. Snapshots on the storage array are a useful first line for quick undo, but they share the array's fate.

How data center backups work

In a small office, a backup is often an agent on each computer copying files to a disk. A data center separates the roles:

• The backup server (management server) holds the jobs, schedules, retention policies and the catalog: the index of which file or VM is in which backup, on which disk or tape.
• Proxies (data movers) do the heavy lifting: they read data from the hypervisor or the storage array, deduplicate and compress it, and send it on. Adding proxies adds throughput.
• Repositories are where backups land: a server full of disks, a deduplicating backup appliance, object storage or a tape library.

Image-level backups of virtual machines

Most servers in a data center are VMs, and they are usually backed up as a whole, at the hypervisor level, without an agent inside. The backup product asks the hypervisor for a temporary VM snapshot, reads the virtual disks from it, and deletes the snapshot again. The hypervisor keeps track of which blocks changed since the last run — VMware calls this Changed Block Tracking (CBT), Hyper-V Resilient Change Tracking (RCT) — so a nightly incremental of a 500 GB VM reads only the few gigabytes that changed instead of the whole disk.

From one image backup you can restore the whole VM, a single virtual disk, or individual files and database objects. Physical servers and some databases still use agents, and Server+ also distinguishes file-level backups (selected files and folders) from system-state backups (on Windows: the registry, boot files, Active Directory on a domain controller and similar components needed to rebuild the operating system).

Crash-consistent and application-consistent

A VM snapshot taken without warning captures the disk as it would be after a sudden power cut: crash-consistent. File systems and databases are built to recover from that, but transactions in flight may be lost and the database must replay its log on restore. An application-consistent backup first asks the applications to flush their data and pause writes for a moment. On Windows that is the Volume Shadow Copy Service (VSS), which also solves the open file problem: files that are in use are copied from a frozen shadow copy instead of being skipped. On Linux, hypervisor guest tools can run freeze and thaw scripts, for example to put a database in backup mode.

Full, incremental and synthetic full

The classic schedule — a full backup on Sunday and incrementals or differentials on weekdays, described on the OS backup page — runs into a wall at data center scale. A weekly full of 80 TB reads every block of production storage once a week, competes with daytime load if it overruns, and fills the repository.

A synthetic full backup is a full backup that the backup system assembles from the previous full and the incrementals that followed, using data already in the repository instead of reading it from production again. Production only ever sends incrementals; the repository does the merging. Many products go one step further with forever incremental: one full at the start, then only incrementals, with the oldest incremental merged into the full each night as it ages out of retention.

Synthetic fulls have a catch. Every restore point now depends on blocks written weeks or months earlier, so one corrupt block in the repository can damage many restore points at once. Backup products counter this with regular health checks of the repository (re-reading and checksumming stored blocks) and, from time to time, a real "active" full.

An archive is different again: a copy kept for a long time, often years, for legal or historical reasons, usually on cheap media and rarely read. It is not meant for quick recovery.

The 3-2-1-1-0 rule

The 3-2-1 rule — three copies, two kinds of media, one off-site — is usually attributed to photographer Peter Krogh (2005). It was written for a world of disk failures, fires and theft. Ransomware added a new failure: an attacker with administrator rights who deletes or encrypts every copy that is reachable over the network. The 3-2-1-1-0 rule extends the original with two more digits: at least one copy is offline, air-gapped or immutable, and restores are verified with zero errors. The extension was popularized by the backup vendor Veeam; it is a rule of thumb, not a standard.

Immutable backups

An immutable backup is a backup that cannot be changed or deleted by anyone, administrators included, until a retention period set when it was written has passed. The storage itself enforces this, so stolen backup credentials are not enough to destroy it. This is often called WORM: write once, read many. Common forms:

• Object storage with object lock, such as the S3 Object Lock feature and compatible on-premises object stores (see versioning and immutability). In compliance mode, no user can shorten the retention.
• A hardened Linux repository: a backup server with immutable file flags, no SSH access for the backup software, and single-use credentials, so the backup server can write new files but cannot delete locked ones.
• WORM tape cartridges and deduplication appliances with a retention lock.

The S3 API sets a default retention on a bucket with a small JSON document like this one (Object Lock must be enabled on the bucket first, which requires versioning; it can be turned on at creation or later, but never turned off):

{
  "ObjectLockEnabled": "Enabled",
  "Rule": {
    "DefaultRetention": {
      "Mode": "COMPLIANCE",
      "Days": 30
    }
  }
}

Offline copies and air gaps

An air gap is a separation in which a copy of the data has no network path to the systems it protects: a tape on a shelf, a removable disk in a safe, or a vault system that is only connected for a short, controlled window. Nothing online can reach it, which makes it the strongest protection against an attacker or a runaway script. The price is slower access and manual handling. Vendors also use "logical air gap" for an isolated, separately administered system that is still on a network; that is valuable, but it is immutability plus isolation, not a true air gap.

Backup media: disk, object storage and tape

Server+ also lists print as a backup medium: for a few critical items, such as recovery keys, the emergency contact list and the restore runbook itself, paper in a safe is the copy that survives when every system is down.

LTO tape

LTO (Linear Tape-Open) is the open magnetic tape format, defined by a consortium of tape vendors, that dominates data center tape backup. Tape is far from obsolete for large archives and offline copies: a cartridge on a shelf needs no power and cannot be reached over a network. A tape library holds hundreds of cartridges and a robot arm that moves them between slots and drives.

As of 2026, the current generation is LTO-10, announced in August 2025: 30 TB native per cartridge (75 TB at the assumed 2.5:1 compression) and up to 400 MB/s native transfer rate. A 40 TB native LTO-10 cartridge followed, shipping from early 2026 and usable in the same drives. Two points matter when you plan a refresh:

• LTO-10 drives cannot read LTO-9 tapes. For many generations a drive could read the cartridges of one or two earlier generations; LTO-8 reduced that to one, and LTO-10 drops it. Keep at least one working LTO-9 drive for as long as you retain LTO-9 tapes, or migrate the data.
• The "compressed" capacity assumes data that compresses well. Backups that are already deduplicated, compressed or encrypted reach roughly the native figure.

Tape is a streaming device: if data arrives slower than the drive writes, the drive stops, rewinds a little and restarts ("shoe-shining"), which wears the tape and the drive. Backups to tape are therefore usually staged on disk first and then copied to tape at full speed.

Retention and media rotation

Backup retention is the policy that says how long each backup is kept before it expires and its space or media is reused. It is set per kind of data, from business needs (how late are problems discovered?) and legal ones (financial records, for example, often have a statutory retention period of several years). Retention is also a ransomware defense: attackers often sit in a network for weeks before they encrypt, so a restore point from before the intrusion must still exist.

Keeping every nightly backup for a year is expensive. The grandfather-father-son (GFS) scheme, introduced on the OS backup page, keeps dailies for a short time, weeklies longer and monthlies longest. This program applies a GFS policy to nine months of nightly backups and shows which restore points survive:

from datetime import date, timedelta

# One backup per night from 1 January to 30 September 2026.
LAST = date(2026, 9, 30)
backups = [date(2026, 1, 1) + timedelta(days=n) for n in range(273)]

KEEP_DAILY = 7      # sons: the last seven nights
KEEP_WEEKLY = 4     # fathers: the last four Sunday backups
KEEP_MONTHLY = 6    # grandfathers: the last backup of each of six months

kept = {}
for day in backups[-KEEP_DAILY:]:
    kept.setdefault(day, []).append("daily")

sundays = [d for d in backups if d.weekday() == 6]
for day in sundays[-KEEP_WEEKLY:]:
    kept.setdefault(day, []).append("weekly")

month_ends = [d for d in backups if (d + timedelta(days=1)).day == 1]
for day in month_ends[-KEEP_MONTHLY:]:
    kept.setdefault(day, []).append("monthly")

for day in sorted(kept):
    print(f"{day:%a %d %b %Y}  {' + '.join(kept[day])}")
print(f"{len(kept)} restore points kept, {len(backups) - len(kept)} expired")
print(f"Oldest restore point: {min(kept)} ({(LAST - min(kept)).days} days back)")

Thu 30 Apr 2026  monthly
Sun 31 May 2026  monthly
Tue 30 Jun 2026  monthly
Fri 31 Jul 2026  monthly
Mon 31 Aug 2026  monthly
Sun 06 Sep 2026  weekly
Sun 13 Sep 2026  weekly
Sun 20 Sep 2026  weekly
Thu 24 Sep 2026  daily
Fri 25 Sep 2026  daily
Sat 26 Sep 2026  daily
Sun 27 Sep 2026  daily + weekly
Mon 28 Sep 2026  daily
Tue 29 Sep 2026  daily
Wed 30 Sep 2026  daily + monthly
15 restore points kept, 258 expired
Oldest restore point: 2026-04-30 (153 days back)

Fifteen restore points cover five months, where keeping every nightly backup for the same period would need more than ten times as many. One backup can play several roles: Sunday 27 September is both a daily and a weekly, and it expires only when both roles have expired.

With tape, retention becomes physical media rotation: which cartridges go off-site on which day, when they come back, and when they may be overwritten. That requires a media inventory: every cartridge labeled with a barcode, its location and contents recorded in the backup catalog, and a periodic check that what the catalog says is in the vault is actually there.

Protecting the backup system

Modern ransomware operators look for the backup server early, because an organization that can restore does not pay. Deleting backups, disabling jobs and encrypting the repository are standard steps before the visible attack (see ransomware in depth). Treat the backup system as one of the most sensitive systems you run:

• Separate identity. The backup server should not be joined to the production Active Directory domain, and backup administrators use dedicated accounts with multi-factor authentication. A domain administrator in production should have no rights on the backups.
• Least privilege for the software. The accounts the backup software uses to read VMs and write to repositories get only those rights.
• At least one copy that cannot be deleted: immutable or offline, as above.
• Four-eyes deletion. Some products require a second administrator to approve deleting backups or shortening retention.
• Encryption. Backups that leave the building (tape, cloud) are encrypted, and the keys are stored separately and printed or escrowed, because a backup you cannot decrypt is no backup.
• Monitoring. Alert on failed jobs, on jobs that suddenly back up far more changed data than usual (mass encryption looks like a huge change rate), and on any change to retention settings.

Restoring and testing restores

Restore methods

• Overwrite (restore in place): the backup replaces the current data. Fast and simple, but the current state is gone — make sure that is what you want, and snapshot first if you can.
• Side by side: the restored copy is placed next to the original, for example as budget (restored).xlsx or a VM with a new name, so the user can compare and pick.
• Alternate location: the data is restored to a different path, server or isolated network. This is the normal method for restore tests and for investigating what changed.
• Instant recovery: many products can start a VM directly from the backup repository within minutes, and move it back to production storage with live migration while it runs. It buys time; performance is limited until the move completes.

Will the restore fit the RTO?

Restore time is mostly arithmetic: amount of data divided by the rate the slowest part of the chain can sustain. Kestrel Energy's infrastructure team checks whether an 18 TB file server can be restored within an RTO of 8 hours from each copy it keeps. Real restores rarely reach the nominal rate of a link or drive, so the program applies an efficiency factor:

DATA_TB = 18     # file server to restore, in decimal terabytes
RTO_HOURS = 8

# (source, nominal rate in MB/s, share of it a real restore achieves)
SOURCES = [
    ("Disk repository, 10 GbE", 1250, 0.70),
    ("LTO-10 tape drive", 400, 0.90),
    ("Two LTO-10 drives in parallel", 800, 0.90),
    ("Cloud object storage, 1 Gbit/s", 125, 0.80),
]

print(f"Restoring {DATA_TB} TB, RTO {RTO_HOURS} h")
for name, rate, efficiency in SOURCES:
    seconds = DATA_TB * 1_000_000 / (rate * efficiency)
    hours = seconds / 3600
    verdict = "meets RTO" if hours <= RTO_HOURS else "misses RTO"
    print(f"{name:<32}{rate * efficiency:>6.0f} MB/s {hours:>6.1f} h  {verdict}")

Restoring 18 TB, RTO 8 h
Disk repository, 10 GbE            875 MB/s    5.7 h  meets RTO
LTO-10 tape drive                  360 MB/s   13.9 h  misses RTO
Two LTO-10 drives in parallel      720 MB/s    6.9 h  meets RTO
Cloud object storage, 1 Gbit/s     100 MB/s   50.0 h  misses RTO

The off-site cloud copy is fine as insurance but far too slow as the plan for a quick recovery, and one tape drive is not enough either. In practice the numbers are often worse: restoring from a deduplicating repository reads scattered blocks and can be much slower than backing up, and restoring many servers at once shares the same links. Only a timed test shows the real figure.

Restore tests

A restore test is the deliberate, scheduled recovery of real backups into a separate environment, followed by a check that the restored systems start and their data is complete and usable, and a comparison of the time it took with the RTO. A green "job succeeded" status only means data was written; it does not prove that it can be read back, that the application starts, or that the encryption key is still known.

A useful test program at data center scale:

• Automated, frequent tests. Many backup products can start VMs from backup in an isolated network every night or week, check that they boot, that services answer and that a database mounts, and report the result. This is where the "0" of 3-2-1-1-0 comes from.
• Regular full tests of the critical systems, with application owners checking the data, and the time recorded.
• Tests from every kind of copy, including the tape and the immutable cloud copy, at least once a year — they are the ones you will need on the worst day.
• Media and equipment checks: can the library still read old cartridges, and is a compatible drive available?