Pods, Deployments and ReplicaSets

Pods in practice

A pod is one or more containers that are scheduled together onto one node, share one network address (they reach each other on localhost) and can share volumes. Most pods have one main container. A pod is never repaired: when it or its node dies, it is gone, and something else has to create a new one with a new name and address.

That "something else" is a controller (see controllers and desired state). A bare pod created with kubectl run or a kind: Pod manifest has no controller behind it, so after a node failure it simply disappears. Use bare pods only for one-off experiments.

A pod can hold two kinds of helper containers next to the main one.

Init containers

An init container runs to completion before the main containers start. Init containers run one after another; if one fails, the kubelet retries it and the pod stays in Init: status. Typical jobs: wait until a dependency answers, run a database migration, or fetch a file the application needs.

Sidecar containers

A sidecar container runs next to the main container for the pod's whole life and supports it: shipping logs, renewing certificates, or acting as a proxy. Since Kubernetes 1.33, native sidecars are stable: you declare them as init containers with restartPolicy: Always. They start before the main container, keep running, and are stopped after it — which fixes the old start-up and shutdown races of sidecars declared as ordinary containers.

The pod template of Kestrel's meter-api uses both kinds:

spec:
  initContainers:
    - name: wait-for-db                  # init container: runs once, must succeed
      image: postgres:17
      command: ["sh", "-c", "until pg_isready -h db.kestrel-data -p 5432; do sleep 2; done"]
    - name: log-forwarder                # native sidecar: restartPolicy Always
      image: registry.example.com/kestrel/log-forwarder:0.9.1
      restartPolicy: Always
      volumeMounts:
        - name: logs
          mountPath: /var/log/meter-api
          readOnly: true
  containers:
    - name: meter-api
      image: registry.example.com/kestrel/meter-api:2.3.0
      ports:
        - containerPort: 8000
      volumeMounts:
        - name: logs
          mountPath: /var/log/meter-api
  volumes:
    - name: logs
      emptyDir: {}

The shared emptyDir volume lives as long as the pod (see storage in Kubernetes). Containers that log to standard output need no log sidecar at all; kubectl logs already shows that output.

Labels and selectors

A label is a key-value pair attached to an object, such as app.kubernetes.io/name: portal or tier: frontend. Labels carry no meaning for Kubernetes itself; they exist so that you and the controllers can find objects. A label selector is a query over labels. It is how a ReplicaSet finds its pods, a Service its endpoints and kubectl get -l its results.

Selectors come in two flavors:

• Equality-based: tier=frontend, track!=canary.
• Set-based: tier in (frontend, backend), tier notin (batch), tier (key exists), !tier (key absent).

Requirements separated by commas must all match. A Python model of the matching rules, run against four pods in the Kestrel cluster:

import re

PODS = {
    "portal-7c9d8b6f4d-x2v9k": {"app.kubernetes.io/name": "portal", "tier": "frontend", "track": "stable"},
    "portal-canary-5b8f9d7c6-q4zrm": {"app.kubernetes.io/name": "portal", "tier": "frontend", "track": "canary"},
    "meter-api-6d9f7b8c5-l2kpw": {"app.kubernetes.io/name": "meter-api", "tier": "backend"},
    "billing-export-29854110-8xk2d": {"app.kubernetes.io/name": "billing-export"},
}


def requirements(selector):
    """Split on commas that are not inside parentheses."""
    return [part.strip() for part in re.split(r",(?![^(]*\))", selector)]


def matches(requirement, labels):
    if m := re.fullmatch(r"(\S+)\s+(in|notin)\s+\((.*)\)", requirement):
        key, op = m.group(1), m.group(2)
        values = {v.strip() for v in m.group(3).split(",")}
        if op == "in":
            return labels.get(key) in values
        return labels.get(key) not in values  # also true when the key is missing
    if m := re.fullmatch(r"(\S+?)\s*(==|=|!=)\s*(\S+)", requirement):
        key, op, value = m.groups()
        if op == "!=":
            return labels.get(key) != value  # also true when the key is missing
        return labels.get(key) == value
    if requirement.startswith("!"):
        return requirement[1:] not in labels  # key must be absent
    return requirement in labels  # key must exist, any value


def select(selector):
    reqs = requirements(selector)
    names = [n for n, labels in PODS.items() if all(matches(r, labels) for r in reqs)]
    print(selector)
    print("  ->", ", ".join(names) or "(no pods)")


select("app.kubernetes.io/name=portal")
select("app.kubernetes.io/name=portal,track!=canary")
select("tier in (frontend, backend)")
select("!tier")
select("app.kubernetes.io/name=portal,tier=backend")

app.kubernetes.io/name=portal
  -> portal-7c9d8b6f4d-x2v9k, portal-canary-5b8f9d7c6-q4zrm
app.kubernetes.io/name=portal,track!=canary
  -> portal-7c9d8b6f4d-x2v9k
tier in (frontend, backend)
  -> portal-7c9d8b6f4d-x2v9k, portal-canary-5b8f9d7c6-q4zrm, meter-api-6d9f7b8c5-l2kpw
!tier
  -> billing-export-29854110-8xk2d
app.kubernetes.io/name=portal,tier=backend
  -> (no pods)

Note the trap in != and notin: they also match objects that do not have the key at all. The same selectors work on the command line: kubectl get pods -l 'tier in (frontend, backend)'.

Kubernetes recommends a common set of labels with the prefix app.kubernetes.io/, which tools such as dashboards and Helm understand:

ReplicaSets

A ReplicaSet keeps a fixed number of identical pods running. Its controller counts the pods that match its selector; if there are too few it creates pods from its template, if there are too many it deletes some. Each pod it creates gets an ownerReferences entry pointing back to the ReplicaSet, so Kubernetes knows who owns what and deletes the pods when the ReplicaSet is deleted.

You do not write ReplicaSets yourself. A Deployment creates one per version of its pod template, and adds a label pod-template-hash to the template and the selector so that the ReplicaSets of two versions never claim each other's pods:

$ kubectl get pods -n kestrel-portal -l app.kubernetes.io/name=portal --show-labels

NAME                      READY   STATUS    RESTARTS   AGE   LABELS
portal-7c9d8b6f4d-2kq8n   1/1     Running   0          6d    app.kubernetes.io/name=portal,app.kubernetes.io/part-of=kestrel,app.kubernetes.io/version=1.4.2,pod-template-hash=7c9d8b6f4d

The names follow the chain: Deployment portal → ReplicaSet portal-7c9d8b6f4d (the template hash) → pod portal-7c9d8b6f4d-2kq8n (a random suffix).

Writing a Deployment

A Deployment adds versions on top of ReplicaSets: it owns them, decides which one should have how many pods, and moves pods from the old version to the new one. Its central part is the pod template (spec.template): the complete description of the pods to create — labels, containers, images, probes, volumes. Every change to the template is a new version.

You can let kubectl write a starting point. With --dry-run=client -o yaml it prints the manifest instead of sending it to a cluster:

$ kubectl create deployment meter-api --image=kestrel-meter-api:1.1 --replicas=2 --port=8000 --dry-run=client -o yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: meter-api
  name: meter-api
spec:
  replicas: 2
  selector:
    matchLabels:
      app: meter-api
  strategy: {}
  template:
    metadata:
      labels:
        app: meter-api
    spec:
      containers:
      - image: kestrel-meter-api:1.1
        name: kestrel-meter-api
        ports:
        - containerPort: 8000
        resources: {}
status: {}

The generator names the container after the image (kestrel-meter-api, from a local test image) and adds empty strategy, resources and status fields. Treat it as a skeleton: rename the container, remove status, and fill in what production needs. Ravi Kumar's Deployment for the customer portal:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: portal
  namespace: kestrel-portal
  labels:
    app.kubernetes.io/name: portal
    app.kubernetes.io/part-of: kestrel
spec:
  replicas: 3
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app.kubernetes.io/name: portal        # stable labels only
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1                           # at most 4 pods during an update
      maxUnavailable: 0                     # never fewer than 3 ready
  template:                                 # the pod template
    metadata:
      labels:
        app.kubernetes.io/name: portal
        app.kubernetes.io/part-of: kestrel
        app.kubernetes.io/version: "1.4.2"
    spec:
      containers:
        - name: portal
          image: registry.example.com/kestrel/portal:1.4.2
          ports:
            - containerPort: 8080
          readinessProbe:
            httpGet:
              path: /readyz
              port: 8080
          resources:
            requests:
              cpu: 250m
              memory: 256Mi
            limits:
              memory: 512Mi

• The selector uses only app.kubernetes.io/name; the template carries that label plus others.
• maxUnavailable: 0 with maxSurge: 1 means an update adds one new pod, waits until it is ready, then removes one old pod — capacity never drops.
• The readiness probe decides when a new pod counts as available, so the rollout waits for a pod that really answers. Probes, requests and limits are explained on health, resources and scaling.
• revisionHistoryLimit (default 10) is how many old ReplicaSets are kept for rollbacks.

Rolling updates

A rollout is the process in which a Deployment replaces the pods of one template version with those of the next. It starts when the pod template changes — usually a new image tag — and the Deployment then scales the new ReplicaSet up and the old one down, step by step. This is a rolling deployment; the other strategy, Recreate, stops all old pods first and accepts downtime.

maxSurge and maxUnavailable

Two settings bound a rollout. maxSurge is how many pods may exist above the desired count; maxUnavailable is how many may be missing from the ready count. Both accept a number or a percentage and both default to 25%. A percentage is turned into pods by rounding up for maxSurge and rounding down for maxUnavailable, so a small Deployment never ends up with both at zero:

import math


def absolute(value, replicas, round_up):
    """Turn 1 or "25%" into a number of pods."""
    if isinstance(value, int):
        return value
    exact = replicas * int(value.rstrip("%")) / 100
    return math.ceil(exact) if round_up else math.floor(exact)


print("Defaults (maxSurge 25%, maxUnavailable 25%):")
for replicas in (1, 2, 3, 4, 10):
    surge = absolute("25%", replicas, round_up=True)
    unavailable = absolute("25%", replicas, round_up=False)
    print(f"  {replicas:2} replicas: surge {surge}, unavailable {unavailable}"
          f" -> at most {replicas + surge} pods, at least {replicas - unavailable} ready")


def rollout(replicas, max_surge, max_unavailable):
    surge = absolute(max_surge, replicas, round_up=True)
    unavailable = absolute(max_unavailable, replicas, round_up=False)
    old, new, step = replicas, 0, 0
    print(f"\n{replicas} replicas, maxSurge {max_surge}, maxUnavailable {max_unavailable}:")
    while old > 0:
        step += 1
        remove = min(old, old + new - (replicas - unavailable))
        old -= remove
        start = min(replicas + surge - (old + new), replicas - new)
        new += start
        print(f"  step {step}: stop {remove} old, start {start} new"
              f" -> old {old}, new {new}")


rollout(3, 1, 0)
rollout(4, "25%", "25%")

Defaults (maxSurge 25%, maxUnavailable 25%):
   1 replicas: surge 1, unavailable 0 -> at most 2 pods, at least 1 ready
   2 replicas: surge 1, unavailable 0 -> at most 3 pods, at least 2 ready
   3 replicas: surge 1, unavailable 0 -> at most 4 pods, at least 3 ready
   4 replicas: surge 1, unavailable 1 -> at most 5 pods, at least 3 ready
  10 replicas: surge 3, unavailable 2 -> at most 13 pods, at least 8 ready

3 replicas, maxSurge 1, maxUnavailable 0:
  step 1: stop 0 old, start 1 new -> old 3, new 1
  step 2: stop 1 old, start 1 new -> old 2, new 2
  step 3: stop 1 old, start 1 new -> old 1, new 3
  step 4: stop 1 old, start 0 new -> old 0, new 3

4 replicas, maxSurge 25%, maxUnavailable 25%:
  step 1: stop 1 old, start 2 new -> old 3, new 2
  step 2: stop 2 old, start 2 new -> old 1, new 4
  step 3: stop 1 old, start 0 new -> old 0, new 4

The simulation is simplified: it assumes that the new pods of one step are ready before the next step begins. In a real cluster each step waits for readiness probes, and a new pod that never becomes ready stalls the rollout — which is exactly the protection you want, because the old pods keep serving.

Watching a rollout

Yara Haddad, the release manager, changes the image to 1.4.3 in Git; the pipeline applies it. To follow it:

$ kubectl rollout status deployment/portal -n kestrel-portal
$ kubectl get deployment,replicaset,pods -n kestrel-portal -l app.kubernetes.io/name=portal

Waiting for deployment "portal" rollout to finish: 1 out of 3 new replicas have been updated...
Waiting for deployment "portal" rollout to finish: 2 out of 3 new replicas have been updated...
Waiting for deployment "portal" rollout to finish: 1 old replicas are pending termination...
deployment "portal" successfully rolled out

NAME                     READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/portal   3/3     3            3           6d

NAME                                DESIRED   CURRENT   READY   AGE
replicaset.apps/portal-6b8d5f9c7w   3         3         3       2m10s
replicaset.apps/portal-7c9d8b6f4d   0         0         0       6d

NAME                          READY   STATUS    RESTARTS   AGE
pod/portal-6b8d5f9c7w-4lqtn   1/1     Running   0          2m10s
pod/portal-6b8d5f9c7w-h8zbm   1/1     Running   0          95s
pod/portal-6b8d5f9c7w-w9v2s   1/1     Running   0          58s

UP-TO-DATE counts pods from the newest template; AVAILABLE counts pods that have been ready for at least minReadySeconds (default 0). If a rollout makes no progress for progressDeadlineSeconds (default 600), the Deployment's condition turns to ProgressDeadlineExceeded and rollout status exits with an error, which lets a pipeline fail the release. Kubernetes does not roll back by itself.

Rollbacks and revision history

Every template version is a numbered revision, kept as an old ReplicaSet (up to revisionHistoryLimit). The annotation kubernetes.io/change-cause, if your pipeline sets it, appears in the history:

$ kubectl rollout history deployment/portal -n kestrel-portal
$ kubectl rollout undo deployment/portal -n kestrel-portal

deployment.apps/portal
REVISION  CHANGE-CAUSE
1         <none>
2         CHG-4127 portal 1.4.2
3         CHG-4133 portal 1.4.3

deployment.apps/portal rolled back

rollout undo copies the template of the previous revision back into the Deployment, which is itself a new rollout (revision 4) — so it is as gradual and as safe as any other update. --to-revision=2 picks a specific one.

Other workload kinds

Deployments suit stateless, long-running services. Kubernetes has other controllers for other shapes of work:

Joost Bakker's billing export runs every night at 02:30 Amsterdam time. Without timeZone, the schedule would follow the time zone of the control plane, usually UTC, and shift by an hour twice a year:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: billing-export
  namespace: kestrel-billing
spec:
  schedule: "30 2 * * *"
  timeZone: "Europe/Amsterdam"
  concurrencyPolicy: Forbid          # never two exports at once
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 3
  jobTemplate:
    spec:
      backoffLimit: 2                # retry a failed pod twice
      template:
        metadata:
          labels:
            app.kubernetes.io/name: billing-export
        spec:
          restartPolicy: Never
          containers:
            - name: billing-export
              image: registry.example.com/kestrel/billing-export:1.0.7
              args: ["--date", "yesterday"]

Each run creates a Job named after the scheduled time (billing-export-29854110, the number of minutes since 1970), which creates the pod. Pods of Jobs use restartPolicy: Never or OnFailure; the Always of Deployments would restart a finished export forever.