Software-Defined Storage and Hyperconverged Infrastructure

From storage array to software

A classic storage array is a dedicated box: two controllers, shelves of drives and the vendor's software, sold together. It grows by adding drive shelves until the controllers are the bottleneck; then you buy a bigger array and migrate. That is scale-up.

Software-defined storage (SDS) separates the storage software from the hardware. The software runs on ordinary x86 servers with local drives, pools the drives of all servers into one storage system, and decides where every piece of data lives. Redundancy comes from the software placing copies on different servers, not from a RAID controller inside one box. Examples are Ceph (open source), VMware vSAN, Nutanix's distributed storage and Microsoft Storage Spaces Direct.

Scale-out

SDS is built to scale out: you add capacity and performance by adding another server (a node) to the cluster rather than by making one system bigger. Each new node brings drives, CPU, memory and network ports, so throughput grows with capacity. The software then moves a share of the existing data to the new node so that all nodes stay evenly filled. It is the storage version of horizontal scaling.

The price is complexity and network dependence. A write is not finished until it has reached several nodes over the network, so latency depends on the network as much as on the drives. A dedicated array hides all of that behind one support contract; with SDS, your team owns the design.

How SDS protects data

SDS assumes hardware fails. With annual drive failure rates of around 1% or more, a cluster of 1,000 drives can expect a failed drive every month or two, and servers go down for patching every month. The question is not whether a node is missing but how many can be missing at the same time without losing data.

Replication factor

The simplest protection is to keep whole copies. The replication factor is the number of copies the cluster keeps of every piece of data, each on a different node. With a replication factor of 3, a cluster can lose two nodes and still have one copy; usable capacity is a third of raw capacity. Reads are fast (any copy will do) and a rebuild only needs to copy data, with no calculation.

Ceph calls the replication factor the pool's size. It also has a min_size: the number of copies that must be available before Ceph accepts writes. The common setting is size 3, min_size 2. When only one copy is left, Ceph pauses I/O for the affected data rather than accept writes that exist in only one place.

Erasure coding

Erasure coding (EC) splits data into k data parts, calculates m extra coding parts from them, and stores all k + m parts on different nodes. Any k of the parts are enough to rebuild the data. A 4+2 scheme stores 6 parts for every 4 parts of data: 1.5 times the data, and two nodes may be lost. It is the same idea as parity in RAID 6, spread over servers instead of the drives in one box.

Erasure coding costs CPU and latency: a small write means recalculating coding parts, a read needs parts from several nodes, and a rebuild must read k parts to recreate one. That is why many clusters use replication for VM disks and databases (small random writes) and erasure coding for large, mostly sequential data such as backups, archives and object storage.

Failure domains

Copies only help if they do not fail together. A failure domain is a group of hardware that can fail as one: a drive, a server, a rack (one top-of-rack switch, one pair of power feeds), a room. The software must put the copies of one piece of data in different failure domains. The default is "different hosts", so three copies never end up on three drives in the same server. Larger clusters set the failure domain to the rack, so that losing a rack switch or a power feed takes away at most one copy.

This sets a minimum cluster size. Replication factor 3 at host level needs three hosts; erasure coding 4+2 needs six. To self-heal, that is to recreate the missing copies after a node fails, you need one more node than that, with enough free space on it.

Usable capacity in practice

Raw capacity is what the drives add up to. What you can actually fill is much less: divide by the protection overhead, keep enough free space to rebuild the data of one failed node on the others, and stay below the level where the software starts warning. Ceph, by default, warns when a drive's storage daemon reaches 85% full (nearfull) and stops writes at 95% (full). The example compares four schemes on a six-node cluster.

NODES = 6
DRIVES_PER_NODE = 10
DRIVE_TB = 7.68        # NVMe drive, decimal terabytes as on the label
NEARFULL = 0.85        # Ceph's default warning level for a full OSD

raw_tb = NODES * DRIVES_PER_NODE * DRIVE_TB
print(f"Raw: {NODES} nodes x {DRIVES_PER_NODE} drives x {DRIVE_TB} TB = {raw_tb:.1f} TB")
print()

# (name, data parts k, extra parts m); "replica n" means k=1, m=n-1
schemes = [("replica 2", 1, 1), ("replica 3", 1, 2), ("EC 4+2", 4, 2), ("EC 8+3", 8, 3)]

print("scheme     efficiency  node losses  usable    plan for  on 6 nodes")
for name, k, m in schemes:
    efficiency = k / (k + m)
    usable = raw_tb * efficiency
    # leave room to rebuild one failed node and stay below the warning level
    plan_for = usable * (NODES - 1) / NODES * NEARFULL
    if k + m + 1 <= NODES:
        verdict = "ok, can self-heal"
    elif k + m <= NODES:
        verdict = "fits, cannot self-heal"
    else:
        verdict = f"no, needs {k + m}+ nodes"
    print(f"{name:<10} {efficiency:>10.0%} {m:>12}  {usable:>5.1f} TB  {plan_for:>5.1f} TB  {verdict}")

Raw: 6 nodes x 10 drives x 7.68 TB = 460.8 TB

scheme     efficiency  node losses  usable    plan for  on 6 nodes
replica 2         50%            1  230.4 TB  163.2 TB  ok, can self-heal
replica 3         33%            2  153.6 TB  108.8 TB  ok, can self-heal
EC 4+2            67%            2  307.2 TB  217.6 TB  fits, cannot self-heal
EC 8+3            73%            3  335.1 TB  237.4 TB  no, needs 11+ nodes

Of 460.8 TB of drives, a safe plan with three copies is about 109 TB, less than a quarter. "Node losses" is how many failure domains can be lost at once without losing data; whether I/O continues meanwhile depends on min_size. Erasure coding 4+2 doubles the plannable capacity, but on exactly six nodes it has nowhere to rebuild a lost node, so the cluster stays degraded until the node returns. Capacity trends of this kind belong in your capacity planning.

Ceph

Ceph is an open-source software-defined storage system that offers block, file and object storage from one cluster of ordinary servers. It is used on its own, as the storage layer of OpenStack clouds and in Proxmox VE clusters. Its core is RADOS (Reliable Autonomic Distributed Object Store), which stores everything as objects spread over the cluster; three interfaces sit on top:

• RBD (RADOS Block Device): virtual disks for VMs and servers, the block storage of the cluster.
• CephFS: a shared POSIX file system, mounted by Linux clients much like an NFS share.
• RGW (RADOS Gateway): object storage with an S3-compatible API, see Storage Fundamentals.

The daemons

Placement without a lookup table

Ceph has no central table that says where each object lives; such a table would be a bottleneck and a single point of failure. Objects are grouped into placement groups (PGs), and an algorithm called CRUSH (Controlled Replication Under Scalable Hashing) calculates from the cluster map which OSDs hold each placement group, respecting the failure domains. Every client and every OSD can do the same calculation, so a client talks directly to the right OSD.

A good placement function also moves little data when the cluster changes. The toy example below is not CRUSH, but it shows the idea with rendezvous hashing: every node gets a pseudo-random score per object and the three highest scores hold the copies. It compares that with naive "hash modulo number of nodes" placement when a sixth node joins.

import hashlib

COPIES = 3
OBJECTS = [f"rbd_data.vm-{n:05d}" for n in range(12000)]


def score(node, obj):
    digest = hashlib.sha256(f"{node}/{obj}".encode()).digest()
    return int.from_bytes(digest[:8], "big")


def place_rendezvous(obj, nodes):
    # every node gets a pseudo-random score per object; the top 3 win
    return set(sorted(nodes, key=lambda node: score(node, obj))[-COPIES:])


def place_modulo(obj, nodes):
    first = score("", obj) % len(nodes)
    return {nodes[(first + i) % len(nodes)] for i in range(COPIES)}


before = [f"node{n}" for n in range(1, 6)]
after = before + ["node6"]

for name, place in [("modulo", place_modulo), ("rendezvous", place_rendezvous)]:
    moved = sum(len(place(o, after) - place(o, before)) for o in OBJECTS)
    total = len(OBJECTS) * COPIES
    print(f"{name:<11} copies moved after adding node6: {moved:>5} of {total} ({moved / total:.0%})")

load = {node: 0 for node in after}
for obj in OBJECTS:
    for node in place_rendezvous(obj, after):
        load[node] += 1
print("rendezvous copies per node:", ", ".join(f"{n}={c}" for n, c in load.items()))

modulo      copies moved after adding node6: 18026 of 36000 (50%)
rendezvous  copies moved after adding node6:  5865 of 36000 (16%)
rendezvous copies per node: node1=6021, node2=6049, node3=6059, node4=5963, node5=6043, node6=5865

With six nodes, the new node should end up with one sixth of the copies, so about 17% of the data must move. Rendezvous hashing moves almost exactly that and spreads the copies evenly; modulo placement moves half of everything, which on a real cluster means days of extra network and drive load. CRUSH adds drive weights and the failure-domain hierarchy on top of the same idea.

Reading cluster health

ceph -s (status) is the first command to run on a Ceph cluster. Below, one drive of a 30-drive cluster has failed:

  cluster:
    id:     4f1c2a9e-0d3b-4c5e-9a7f-2b8e6d1c0a53
    health: HEALTH_WARN
            1 osds down
            Degraded data redundancy: 41210/1236630 objects degraded (3.332%), 102 pgs degraded

  services:
    mon: 3 daemons, quorum hci1,hci2,hci3 (age 12d)
    mgr: hci1(active, since 12d), standbys: hci2
    osd: 30 osds: 29 up (since 4m), 30 in (since 12d)

  data:
    pools:   3 pools, 1057 pgs
    objects: 412.21k objects, 1.5 TiB
    usage:   4.6 TiB used, 205 TiB / 210 TiB avail
    pgs:     41210/1236630 objects degraded (3.332%)
             955 active+clean
             102 active+undersized+degraded

Read it from the top. HEALTH_WARN with all monitors in quorum means the cluster still serves data: the affected placement groups have two of their three copies (undersized, degraded), above min_size. The OSD is down but still in: Ceph waits (by default ten minutes) before marking it out and copying the missing data elsewhere, so that a quick reboot does not start a full rebuild. HEALTH_ERR needs immediate attention, and monitors out of quorum or PGs that are not active mean data is unavailable: that is an incident, not a warning.

Hyperconverged infrastructure

Hyperconverged infrastructure (HCI) runs virtual machines and software-defined storage on the same servers. Every node is a virtualization host with its own local drives; the storage software pools the drives of all nodes into one shared datastore that every host can use. One cluster of identical nodes replaces three separate systems: the hosts, the storage array and the storage area network between them (see DAS, NAS and SAN). The node hardware itself is covered in Servers: Rack, Blade and Hyperconverged.

HCI platforms

Licensing and platform choices are covered in Data Center Virtualization in Practice.

What HCI changes

• Simple to start and grow. Three or four nodes form a complete platform; growing means adding a node, without planning a new array or SAN.
• Compute and storage grow together. If you need only capacity, you still buy CPUs and memory (and per-core licenses) with it. Some platforms allow storage-only nodes or, as Nutanix now does, external arrays to break that link.
• A node is a storage node. Putting a host into maintenance mode or rebooting it for a patch takes away its copy of the data. The cluster must either move that data first or run with reduced redundancy during the window.
• The storage software uses host resources. CPU and memory reserved for the storage service are not available for VMs; size the nodes with that in mind.

The network is part of the storage

In an array, writes cross a short backplane; in SDS and HCI, every write crosses the network to two or more other nodes before it is confirmed. A slow or flaky network is therefore a slow or flaky storage system. Common practice:

• At least two 25 GbE (gigabit Ethernet) ports per node, and increasingly 100 GbE for all-NVMe nodes, connected to two separate switches so that one switch can fail or be updated. See Data Center Networking.
• A separate VLAN (or separate ports) for storage and replication traffic, apart from VM traffic and management.
• Jumbo frames (an MTU of 9,000 bytes) on storage networks where the platform recommends them, set consistently on every node and switch port. A mismatch often shows up as storage that works for small I/O and hangs on large I/O.
• Rebuild traffic in the plan: after a node failure, the cluster copies terabytes between the survivors while still serving production I/O.

Stretched clusters

Some platforms can stretch one cluster over two sites, with copies in both, so that a whole site can fail. Because writes wait for the other site, distance adds latency to every write. VMware documents a maximum of 5 ms round-trip time between the two data sites for a vSAN stretched cluster, plus a small witness at a third location that breaks the tie when the sites lose contact. Why the witness matters is explained in Disaster Recovery and Site Resilience.

Failures and rebuilds

What an operator sees when hardware fails in an SDS cluster, from least to most serious:

1. One drive fails. Only the copies on that drive are missing. The cluster rebuilds them from the other copies onto free space on many drives at once, so a rebuild often takes minutes to hours rather than the day or more a large RAID rebuild can take (see RAID failures and rebuilds). There is no hot spare; the free space of the cluster is the spare.
2. One node fails or reboots. All its drives disappear at once. Data stays available (with replication factor 3 and min_size 2). The cluster waits a set time in case the node returns (10 minutes by default in Ceph, 60 in vSAN), then rebuilds, which only works if the other nodes have room for the missing copies.
3. A second node fails during the rebuild. Some data now has a single copy and pauses writes; nothing is lost yet. This is why rebuild speed and headroom matter.
4. The cluster loses quorum. If a majority of the monitor or witness votes is gone, the cluster stops serving data rather than risk two halves writing different versions. Restore the network or the missing nodes; do not force a minority back online without the vendor's procedure.

Array, SDS or HCI?

Kestrel Energy, the fictional energy supplier used on other pages, is a typical case: its own small data center runs a four-node HCI cluster for the office and infrastructure VMs, while its smart-meter archive (hundreds of terabytes, written once and read rarely) sits on erasure-coded object storage where capacity per euro matters more than latency.