Storage in Kubernetes

Why storage is different in a cluster

On a single Docker host, a volume or bind mount is a directory on that host, and the container always comes back on the same machine. In a cluster that assumption breaks: a pod can be rescheduled to any node, and a new pod replaces an old one after every rollout. Data written inside the container, or into a directory on one node, stays behind on the old node — or is deleted with the pod.

Kubernetes therefore separates three questions, each answered by a different object:

• What does this pod need? "20 GiB, writable" — a PersistentVolumeClaim, written by the application team.
• Which kinds of storage exist? "fast SSD, kept after deletion" — a StorageClass, set up by the platform team.
• Which actual piece of storage is it? A specific disk or share — a PersistentVolume, usually created automatically.

The storage itself still lives in the usual places — block storage such as a cloud disk or an iSCSI LUN, or file storage such as NFS — and is connected through a CSI (Container Storage Interface) driver, the plug-in that each storage vendor or cloud provides.

Volumes in a pod

A pod lists its volumes in spec.volumes, and each container mounts the ones it needs. Some volume types live and die with the pod; others are a window onto storage that outlives it.

emptyDir

An emptyDir volume is an empty directory created when a pod starts on a node and deleted when the pod is removed from it. It survives a container restart inside the same pod, which makes it useful for caches and for sharing files between containers, but never for data you need to keep. sizeLimit caps its size; medium: Memory puts it in RAM (and counts against the pod's memory).

volumes:
  - name: render-cache
    emptyDir:
      sizeLimit: 500Mi

hostPath, and why to avoid it

A hostPath volume mounts a directory of the node into the pod. For application data it fails in two ways: when the pod moves to another node the data seems to "disappear" (it is still on the old node), and a pod that can mount arbitrary host paths can read or change the node itself. The Pod Security Standards forbid hostPath at both the Baseline and the Restricted level; Restricted allows only a fixed list of volume types (configMap, csi, downwardAPI, emptyDir, ephemeral, persistentVolumeClaim, projected, secret). See namespaces, RBAC and Kubernetes security.

PersistentVolumes and PersistentVolumeClaims

A PersistentVolume (PV) is a cluster-wide object that represents one piece of real storage — a cloud disk, an iSCSI LUN, an NFS export — with a capacity, access modes and a reclaim policy. Its lifecycle is independent of any pod. A PersistentVolumeClaim (PVC) is a namespaced request for storage: "20 GiB, ReadWriteOnce, of class kestrel-ssd-retain". Kubernetes binds each claim to exactly one PV, and a pod refers only to the claim, never to the PV.

PVs can also be created by hand, for storage that already exists. Tomasz Nowak registers Kestrel's archive share on the NAS as a PV:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-nfs-archive
spec:
  capacity:
    storage: 100Gi
  accessModes: [ReadWriteMany, ReadOnlyMany]
  persistentVolumeReclaimPolicy: Retain
  storageClassName: manual-nfs
  nfs:
    server: 192.0.2.40
    path: /exports/kestrel-archive

When claims are matched against such pre-made PVs, Kubernetes looks for an available PV with the same storage class, the requested access mode and at least the requested size, and prefers the smallest one that fits. A claim always gets the whole PV, even if it asked for less. A simplified model of that matching:

GI = 2**30
pvs = [  # name, size, access modes, storage class (made by an administrator)
    ("pv-nfs-archive", 100 * GI, {"ReadWriteMany", "ReadOnlyMany"}, "manual-nfs"),
    ("pv-ssd-01", 20 * GI, {"ReadWriteOnce"}, "manual-ssd"),
    ("pv-ssd-02", 50 * GI, {"ReadWriteOnce"}, "manual-ssd"),
]
claims = [  # name, requested size, access mode, storage class
    ("report-cache", 10 * GI, "ReadWriteOnce", "manual-ssd"),
    ("billing-exports", 30 * GI, "ReadWriteOnce", "manual-ssd"),
    ("shared-uploads", 10 * GI, "ReadWriteMany", "manual-ssd"),
    ("archive", 50 * GI, "ReadWriteMany", "manual-nfs"),
]

bound = set()
for claim, size, mode, sc in claims:
    fitting = [pv for pv in pvs
               if pv[0] not in bound and pv[3] == sc and mode in pv[2] and pv[1] >= size]
    if not fitting:
        print(f"{claim:16} Pending (no available PV matches)")
        continue
    pv = min(fitting, key=lambda p: p[1])  # the smallest PV that is big enough
    bound.add(pv[0])
    print(f"{claim:16} Bound to {pv[0]} ({pv[1] // GI}Gi for a {size // GI}Gi request)")

report-cache     Bound to pv-ssd-01 (20Gi for a 10Gi request)
billing-exports  Bound to pv-ssd-02 (50Gi for a 30Gi request)
shared-uploads   Pending (no available PV matches)
archive          Bound to pv-nfs-archive (100Gi for a 50Gi request)

A claim that finds nothing stays Pending, and so does every pod that uses it. The phases to know: a PV is Available, Bound, Released (its claim was deleted, the data is still there) or Failed; a PVC is Pending, Bound or, if its PV has disappeared, Lost.

StorageClasses and dynamic provisioning

Creating PVs by hand does not scale. A StorageClass describes a kind of storage the cluster offers: which CSI driver (the provisioner) creates it, with which driver-specific parameters, which reclaim policy, and whether volumes may grow. It is not the same as a storage class in object storage, which is a price and access tier for stored objects.

With dynamic provisioning, a new claim that names a StorageClass makes that class's driver create a new volume in the storage system and a matching PV, which is then bound to the claim. Nobody has to prepare disks in advance. The platform team at Kestrel offers two classes:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: kestrel-ssd-retain
provisioner: csi.example.com            # the CSI driver of your storage or cloud
parameters:
  type: ssd                             # driver-specific
reclaimPolicy: Retain                   # keep the volume when the claim is deleted
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer # create the volume where the pod lands
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: billing-exports
  namespace: kestrel-billing
spec:
  accessModes: [ReadWriteOnce]
  storageClassName: kestrel-ssd-retain
  resources:
    requests:
      storage: 20Gi

$ kubectl get storageclass
$ kubectl get pvc -n kestrel-billing
$ kubectl describe pvc billing-exports -n kestrel-billing

NAME                    PROVISIONER       RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
kestrel-ssd (default)   csi.example.com   Delete          WaitForFirstConsumer   true                   210d
kestrel-ssd-retain      csi.example.com   Retain          WaitForFirstConsumer   true                   210d

NAME              STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS         VOLUMEATTRIBUTESCLASS   AGE
billing-exports   Pending                                      kestrel-ssd-retain   <unset>                 12s

…
Events:
  Type     Reason                 Age                From                          Message
  ----     ------                 ----               ----                          -------
  Normal   WaitForFirstConsumer   8s (x2 over 12s)   persistentvolume-controller   waiting for first consumer to be created before binding

Pending is expected here. With WaitForFirstConsumer, the volume is only created once a pod uses the claim, so that the disk is made in the same zone as the node the scheduler picks; with Immediate it would be created at once, possibly in a zone where the pod can never run. As soon as the billing-export CronJob starts a pod that mounts the claim:

$ kubectl get pvc -n kestrel-billing
$ kubectl get pv

NAME              STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS         VOLUMEATTRIBUTESCLASS   AGE
billing-exports   Bound    pvc-5d1c8e2a-7b4f-4c19-9e62-0a8f3b7d2c41   20Gi       RWO            kestrel-ssd-retain   <unset>                 3m5s

NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                             STORAGECLASS         VOLUMEATTRIBUTESCLASS   REASON   AGE
pvc-5d1c8e2a-7b4f-4c19-9e62-0a8f3b7d2c41   20Gi       RWO            Retain           Bound    kestrel-billing/billing-exports   kestrel-ssd-retain   <unset>                          3m2s

Dynamically created PVs are named after the claim's unique ID. A claim without storageClassName gets the class marked (default); a cluster without a default class leaves such claims Pending until a default class is created (Kubernetes then assigns it retroactively). To get no class at all — for binding to a hand-made PV without a class — set storageClassName: "" explicitly.

Access modes

An access mode states how a volume may be mounted: read-write or read-only, and by one node, many nodes or one pod. A claim requests a mode, and only volumes whose driver supports it can satisfy the claim.

Reclaim policies, quotas and expansion

What happens to the volume when its claim is deleted is the PV's reclaim policy:

• Delete — the PV and the disk in the storage system are deleted. This is the default for dynamically provisioned volumes, inherited from the StorageClass.
• Retain — the PV moves to Released and the data stays. An administrator decides what happens next: copy the data, delete the volume, or make the PV available again after cleaning it.
• Recycle — deprecated; do not use it.

Storage requests are resource quantities with the same suffixes as memory: Gi is 2³⁰ bytes, G is 10⁹ bytes (see binary prefixes). A ResourceQuota can cap the total storage that claims in a namespace may request. Joost Bakker checks whether a new claim still fits the billing namespace's quota of 100Gi:

import re

UNITS = {"": 1, "k": 10**3, "M": 10**6, "G": 10**9, "T": 10**12,
         "Ki": 2**10, "Mi": 2**20, "Gi": 2**30, "Ti": 2**40}
GIB = 2**30


def to_bytes(quantity):
    m = re.fullmatch(r"(\d+(?:\.\d+)?)(Ki|Mi|Gi|Ti|k|M|G|T)?", quantity)
    if not m:
        raise ValueError(f"not a storage quantity: {quantity!r}")
    return int(float(m.group(1)) * UNITS[m.group(2) or ""])


quota = to_bytes("100Gi")  # ResourceQuota requests.storage for kestrel-billing
claims = {"billing-exports": "20Gi", "billing-archive": "50G", "report-cache": "500Mi"}

used = 0
for name, size in claims.items():
    used += to_bytes(size)
    print(f"{name:16} {size:>6} = {to_bytes(size) / GIB:6.2f} GiB")
print(f"{'in use':16} {'':6}   {used / GIB:6.2f} GiB of {quota / GIB:.0f} GiB")

for request in ("30Gi", "35Gi"):
    total = used + to_bytes(request)
    verdict = "accepted" if total <= quota else "rejected: exceeded quota"
    print(f"new claim {request}: {total / GIB:.2f} GiB -> {verdict}")

billing-exports    20Gi =  20.00 GiB
billing-archive     50G =  46.57 GiB
report-cache      500Mi =   0.49 GiB
in use                     67.05 GiB of 100 GiB
new claim 30Gi: 97.05 GiB -> accepted
new claim 35Gi: 102.05 GiB -> rejected: exceeded quota

Someone wrote 50G where 50Gi was meant; the claim is about 7% smaller than intended. If a volume turns out too small, a class with allowVolumeExpansion: true lets you raise spec.resources.requests.storage in the claim; the driver grows the disk and the file system. Volumes can grow but never shrink.

StatefulSets

A Deployment's pods are interchangeable: random names, and if they mount a claim, they all share the same one. A database replica is not interchangeable. A StatefulSet is a workload controller for pods that need a stable identity: each replica gets a fixed ordinal name (db-0, db-1), a stable DNS name through a headless Service, and its own PersistentVolumeClaim created from volumeClaimTemplates. Pods are created in order and replaced under the same name, reattached to the same volume.

Hamid Rahimi's PostgreSQL for the test environment:

apiVersion: v1
kind: Service
metadata:
  name: db
  namespace: kestrel-data
spec:
  clusterIP: None                     # headless: DNS names per pod
  selector:
    app.kubernetes.io/name: db
  ports:
    - name: postgres
      port: 5432
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: db
  namespace: kestrel-data
spec:
  serviceName: db
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: db
  template:
    metadata:
      labels:
        app.kubernetes.io/name: db
    spec:
      containers:
        - name: postgres
          image: postgres:17
          ports:
            - name: postgres
              containerPort: 5432
          env:
            - name: POSTGRES_PASSWORD_FILE
              value: /run/secrets/db/password
            - name: PGDATA                  # a subdirectory of the mount
              value: /var/lib/postgresql/data/pgdata
          volumeMounts:
            - name: data
              mountPath: /var/lib/postgresql/data
            - name: credentials
              mountPath: /run/secrets/db
              readOnly: true
      volumes:
        - name: credentials
          secret:
            secretName: db-credentials
  volumeClaimTemplates:               # one claim per replica: data-db-0, data-db-1 …
    - metadata:
        name: data
      spec:
        accessModes: [ReadWriteOncePod]
        storageClassName: kestrel-ssd-retain
        resources:
          requests:
            storage: 20Gi

$ kubectl get pods,pvc -n kestrel-data

NAME       READY   STATUS    RESTARTS   AGE
pod/db-0   1/1     Running   0          41d

NAME                              STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS         VOLUMEATTRIBUTESCLASS   AGE
persistentvolumeclaim/data-db-0   Bound    pvc-9a3e6f1b-2d58-4e7c-b104-6c2f8d9e0a17   20Gi       RWOP           kestrel-ssd-retain   <unset>                 41d

The claim is named after the template and the pod: data-db-0. The pod can be reached as db-0.db.kestrel-data.svc.cluster.local (see cluster DNS). By default, deleting the StatefulSet or scaling it down leaves the claims, and therefore the data, in place; persistentVolumeClaimRetentionPolicy can change that.

Backups: a volume is not a backup

A PersistentVolume keeps data alive across pod restarts. It does nothing against a dropped table, a corrupt file system, ransomware, a deleted namespace or a lost storage system. Volumes need backups like any other storage.

• Volume snapshots. With a CSI driver that supports them, a VolumeSnapshot object takes a snapshot of a claim, and a new claim can be created from it. Like a VM snapshot, it usually lives on the same storage system, so it is a fast restore point, not an off-site backup — and for a running database it is only crash-consistent.
• Logical backups. For databases, a logical backup such as pg_dump, run by a CronJob and written to object storage in another location, can be restored anywhere. Database operators add point-in-time recovery.
• Cluster backup tools. Tools such as Velero back up both the Kubernetes objects of a namespace and its volumes, which helps when a whole namespace or cluster must be rebuilt.

Whatever the method, a backup that has never been restored is a hope, not a backup. Schedule restore tests into a separate namespace and check that the application actually starts on the restored data.