Health, Resources and Scaling

Three questions Kubernetes keeps asking

Once a Deployment runs its pods, Kubernetes needs answers to three questions that it cannot work out on its own:

• Is the application healthy? A process can be running while the application inside hangs, waits for a lock forever or has not finished starting. You answer this with probes.
• How much does it need? The scheduler has to know how much CPU (central processing unit) and memory a pod needs before it can choose a node, and the node has to know how far a container may go. You answer this with requests and limits.
• How many copies are needed right now? Monday morning, when customers check their energy bills, the Kestrel Energy portal needs more pods than at 03:00. The HorizontalPodAutoscaler answers this from measurements.

All three are part of the pod template in your manifests, so they are reviewed, versioned and rolled out like any other change. This page builds on the manifest walk-through in Kubernetes Basics; there is no live cluster behind this site, so all kubectl output on this page is illustrative, written from the Kubernetes documentation.

Probes: is the container healthy?

A probe is a check the kubelet runs against a container at a fixed interval. It is the Kubernetes form of a health check. There are four ways to probe:

• httpGet: an HTTP request to a path and port; any status code from 200 to 399 counts as success.
• tcpSocket: success if a TCP connection to the port can be opened.
• grpc: the standard gRPC health-checking protocol, for applications that speak gRPC.
• exec: runs a command inside the container; exit status 0 is success. Costs a process start every time, so use it sparingly.

What Kubernetes does with a failure depends on the kind of probe. That difference is the most important thing on this page.

Liveness probes

A liveness probe answers "should this container be restarted?". When it fails a number of times in a row, the kubelet kills the container and starts it again (following the pod's restart policy). It is meant for situations the application cannot recover from by itself, such as a deadlock: the process still exists, but it will never answer again.

A liveness probe should therefore check only the process itself: "can I still answer a trivial request?". The Kestrel meter-api exposes /healthz for this; it returns 200 without touching the database.

Readiness probes

A readiness probe answers "should this pod receive traffic now?". While it fails, the pod is taken out of the endpoints of every Service that selects it, so no new requests reach it. The container is not restarted; as soon as the probe passes again the pod is put back. Readiness is the right place to check dependencies: meter-api's /readyz also checks that it can reach PostgreSQL, because without the database it cannot do useful work.

Readiness also protects rolling updates: a Deployment counts a new pod as available only once it is ready, so a version that never becomes ready stops the rollout instead of replacing all healthy pods.

Startup probes

A startup probe answers "has the application finished starting?". While a container has a startup probe that has not yet succeeded, Kubernetes does not run its liveness and readiness probes. Once it succeeds it is switched off for the rest of the container's life; if it keeps failing beyond its threshold, the container is killed and restarted.

This is the clean solution for slow starters such as Java applications that warm up for a minute: without a startup probe, the liveness probe kills them before they ever finish starting, and the pod ends up in a restart loop. The startup window is failureThreshold × periodSeconds: with 30 × 2 seconds, the application gets up to 60 seconds.

Probe settings and timing

With the defaults, a liveness probe restarts a hung container after roughly 3 × 10 = 30 seconds of failures. The one-second timeout is short: an application that answers in 1.2 seconds under load "fails" its probe even though it works.

The three probes in a manifest

The meter-api Deployment, with all three probes and its resource settings (explained in the next section):

apiVersion: apps/v1
kind: Deployment
metadata:
  name: meter-api
  namespace: kestrel-portal
  labels:
    app.kubernetes.io/name: meter-api
    app.kubernetes.io/part-of: kestrel
spec:
  replicas: 3
  selector:
    matchLabels:
      app.kubernetes.io/name: meter-api
  template:
    metadata:
      labels:
        app.kubernetes.io/name: meter-api
        app.kubernetes.io/part-of: kestrel
    spec:
      containers:
        - name: meter-api
          image: registry.example.com/kestrel/meter-api:2.3.0
          ports:
            - containerPort: 8000
          startupProbe:            # up to 30 × 2 s = 60 s to start
            httpGet:
              path: /healthz
              port: 8000
            periodSeconds: 2
            failureThreshold: 30
          readinessProbe:          # checks the database too
            httpGet:
              path: /readyz
              port: 8000
            periodSeconds: 5
            timeoutSeconds: 2
          livenessProbe:           # the process only, never dependencies
            httpGet:
              path: /healthz
              port: 8000
            periodSeconds: 10
            timeoutSeconds: 2
            failureThreshold: 3
          resources:
            requests:
              cpu: 250m
              memory: 256Mi
            limits:
              memory: 512Mi

Requests and limits

Each container can declare, per resource, a request and a limit. They do different jobs, at different moments.

Requests: what the scheduler reserves

A resource request is the amount of CPU or memory the scheduler sets aside for a container when it picks a node. A node with 4 cores fits at most sixteen pods requesting 250m each, whether or not those pods are busy. Requests are a reservation, not a measurement: a pod may use less, or (for CPU) more when the node has spare capacity.

Requests are also what makes overcommitment safe to reason about, the same idea as resource overcommitment on hypervisors: the sum of the limits on a node may be far above its capacity, but the sum of the requests never is. A pod whose requests fit on no node stays Pending; see kubectl and Troubleshooting.

Limits: the ceiling

A resource limit is the maximum a container may use, enforced on the node through Linux cgroups (see What Are Containers?). The two resources behave very differently at the limit:

Memory cannot be "throttled" — memory in use cannot be taken back without the program's cooperation — so the only enforcement is a kill. That is why the memory limit deserves the most care. If you set a limit and no request, Kubernetes sets the request equal to the limit.

A common choice, shown in the manifest above, is a CPU request without a CPU limit, and a memory limit somewhat above the memory request. Without a CPU limit, a busy pod can use idle CPU on the node instead of being throttled while cores sit unused; the request still guarantees its share under contention. Other teams set CPU limits to make performance predictable. Neither is wrong; decide deliberately, and measure.

CPU and memory units

CPU is measured in cores: 1 means one physical core or one vCPU, 0.5 is half of one, and the m suffix means millicores (thousandths of a core), so 0.5 and 500m are the same. The smallest unit is 1m.

Memory is measured in bytes, with decimal suffixes (k, M, G: powers of 1000) and binary prefixes (Ki, Mi, Gi: powers of 1024). The same m suffix exists for memory too, where it means millibytes — a classic typo. This script converts the quantities the way Kubernetes reads them:

from decimal import Decimal

# Kubernetes quantity suffixes: binary (Ki, Mi, Gi), decimal (k, M, G) and milli (m)
SUFFIXES = {
    "Ki": 2**10, "Mi": 2**20, "Gi": 2**30,
    "k": 10**3, "M": 10**6, "G": 10**9,
    "m": Decimal("0.001"),
}


def parse_quantity(text):
    """Turn a quantity such as '250m', '1.5' or '512Mi' into a number."""
    for suffix in sorted(SUFFIXES, key=len, reverse=True):
        if text.endswith(suffix):
            return Decimal(text[: -len(suffix)]) * SUFFIXES[suffix]
    return Decimal(text)


print("CPU (1 = one core or vCPU)")
for cpu in ["100m", "250m", "0.5", "1", "1.5"]:
    print(f"  {cpu:>5} = {int(parse_quantity(cpu) * 1000):>5} millicores")

print("Memory")
for memory in ["256Mi", "256M", "1Gi", "1G", "400m"]:
    print(f"  {memory:>5} = {parse_quantity(memory):>13,} bytes")

mi, m = parse_quantity("512Mi"), parse_quantity("512M")
print(f"512Mi is {(mi - m) / m:.2%} more than 512M")

CPU (1 = one core or vCPU)
   100m =   100 millicores
   250m =   250 millicores
    0.5 =   500 millicores
      1 =  1000 millicores
    1.5 =  1500 millicores
Memory
  256Mi =   268,435,456 bytes
   256M =   256,000,000 bytes
    1Gi = 1,073,741,824 bytes
     1G = 1,000,000,000 bytes
   400m =         0.400 bytes
512Mi is 4.86% more than 512M

Decimal keeps the arithmetic exact, as Kubernetes does. Two lessons: memory: 400m asks for less than one byte (you meant 400Mi). The API server still accepts it, with only a warning that a fractional byte value is invalid — easy to miss in pipeline output. As a request it reserves nothing; as a limit no process fits in it. And M versus Mi is almost a 5% difference — enough to turn a carefully sized limit into an OOM kill. Use Mi and Gi consistently.

Quality of service classes

From the requests and limits of its containers, Kubernetes gives every pod a quality of service class (QoS class). When a node runs short of memory, the kubelet evicts pods to protect the node, and the class decides who goes first:

• Guaranteed: every container has a CPU and a memory limit, and its requests equal its limits. Evicted last.
• Burstable: at least one container has a request or a limit, but the pod does not meet the Guaranteed rule.
• BestEffort: no container has any request or limit. Evicted first.

def qos_class(containers):
    """Work out a pod's quality of service class from its containers."""
    if not any(c.get("requests") or c.get("limits") for c in containers):
        return "BestEffort"
    for c in containers:
        limits = c.get("limits", {})
        # A request that is left out defaults to the limit.
        requests = {**limits, **c.get("requests", {})}
        for resource in ("cpu", "memory"):
            if resource not in limits or requests[resource] != limits[resource]:
                return "Burstable"
    return "Guaranteed"


pods = {
    "meter-api": [{"requests": {"cpu": "250m", "memory": "256Mi"},
                   "limits": {"memory": "512Mi"}}],
    "billing-export": [{"requests": {"cpu": "500m", "memory": "1Gi"},
                        "limits": {"cpu": "500m", "memory": "1Gi"}}],
    "portal": [{"limits": {"cpu": "500m", "memory": "256Mi"}}],
    "debug-shell": [{}],
}
for name, containers in pods.items():
    print(f"{name:<15} {qos_class(containers)}")

meter-api       Burstable
billing-export  Guaranteed
portal          Guaranteed
debug-shell     BestEffort

The portal is Guaranteed although it sets only limits, because the missing requests default to the limits. (Real Kubernetes compares quantities, so 0.5 and 500m are equal; this sketch compares the strings.) kubectl describe pod shows the result in the line QoS Class:. Since Kubernetes 1.37, Memory QoS — using the cgroup v2 memory controls to protect requested memory more precisely — is in beta.

Horizontal pod autoscaling

The HorizontalPodAutoscaler (HPA) is a Kubernetes object, and a controller behind it, that changes the number of replicas of a Deployment or StatefulSet based on a measured metric. It is Kubernetes' horizontal scaling; for the general idea of autoscaling see the Cloud topic. The stable API is autoscaling/v2; older manifests with v2beta2 must be rewritten.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: portal
  namespace: kestrel-portal
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: portal
  minReplicas: 2
  maxReplicas: 10
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70    # percent of the CPU REQUEST
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300   # the default, written out

Utilization is measured as a percentage of the pods' requests. A pod without a CPU request has no utilization, and the HPA cannot act on it. The measurements come from the resource metrics API (metrics.k8s.io, which became stable in Kubernetes 1.37 after years in beta), usually served by the Metrics Server add-on; many managed clusters install it for you.

$ kubectl get hpa portal -n kestrel-portal

NAME     REFERENCE           TARGETS        MINPODS   MAXPODS   REPLICAS   AGE
portal   Deployment/portal   cpu: 91%/70%   2         10        4          12d

The formula

About every 15 seconds the controller computes:

desiredReplicas = ceil(currentReplicas × currentMetricValue ÷ desiredMetricValue)

If the ratio is within a tolerance of 1.0 (10% by default), it does nothing, so small fluctuations do not cause constant scaling. The result is then clamped between minReplicas and maxReplicas. Scaling down is deliberately slow: the HPA applies the highest recommendation of the last 300 seconds, so a short dip does not remove pods that are needed a minute later.

import math

MIN_REPLICAS, MAX_REPLICAS = 2, 10
TARGET = 70  # target average CPU utilization, percent of the request


def desired_replicas(current, utilization, tolerance=0.1):
    """The HorizontalPodAutoscaler formula, with tolerance and min/max."""
    ratio = utilization / TARGET
    if abs(ratio - 1.0) <= tolerance:
        return current  # close enough: do nothing
    wanted = math.ceil(current * ratio)
    return max(MIN_REPLICAS, min(MAX_REPLICAS, wanted))


for current, utilization in [(4, 90), (6, 75), (6, 160), (10, 20)]:
    print(f"{current:>2} pods at {utilization:>3}% -> "
          f"{desired_replicas(current, utilization)} pods")

# Scaling down waits: the HPA uses the HIGHEST recommendation
# of the last 300 seconds (the downscale stabilization window).
print()
recommendations = [(0, 10), (60, 7), (120, 4), (240, 3), (330, 3), (420, 3)]
for now, _ in recommendations:
    window = [r for t, r in recommendations if now - 300 < t <= now]
    print(f"t={now:>3}s  recommended {window[-1]:>2}  applied {max(window):>2}")

 4 pods at  90% -> 6 pods
 6 pods at  75% -> 6 pods
 6 pods at 160% -> 10 pods
10 pods at  20% -> 3 pods

t=  0s  recommended 10  applied 10
t= 60s  recommended  7  applied 10
t=120s  recommended  4  applied 10
t=240s  recommended  3  applied 10
t=330s  recommended  3  applied  7
t=420s  recommended  3  applied  3

Four pods at 90% against a 70% target need ceil(4 × 90 ÷ 70) = ceil(5.14) = 6 pods. Six pods at 75% are within the tolerance (75 ÷ 70 = 1.07), so nothing happens. A spike to 160% would ask for 14 but stops at the maximum of 10, and when load drops the replica count follows only after the window has passed.

Other ways to scale

• Vertical scaling changes the requests and limits of pods instead of their number — the vertical scaling counterpart. The Vertical Pod Autoscaler (VPA) is an add-on that recommends or applies new values based on usage; resizing a running pod in place is also possible in recent Kubernetes versions. Do not let a VPA and an HPA act on the same CPU metric: they fight each other.
• Event-driven scaling: KEDA (Kubernetes Event-driven Autoscaling) is a popular add-on that feeds queue lengths, database rows or other event sources to the HPA as external metrics. It is a natural fit for jobs like Kestrel's billing export.
• Node autoscaling: when new pods do not fit on any node, a cluster autoscaler (such as Cluster Autoscaler or Karpenter on the managed cloud services) adds nodes, and removes them when they are empty. The HPA adds pods; something else must add the machines they run on.

Right-sizing in practice

Good numbers come from measurement, not from guesses copied between manifests. A practical loop, which mirrors right-sizing in the cloud:

1. Start with a reasonable request based on a load test or the old VM's usage, and a memory limit with headroom.
2. Watch actual usage over at least a full business cycle (a week with a Monday morning in it), with kubectl top for a quick look and your monitoring system's metrics for history.
3. Set the request near typical usage and the memory limit above the observed peak; check for CPU throttling and OOM kills after each change.

$ kubectl top pods -n kestrel-portal -l app.kubernetes.io/name=meter-api

NAME                         CPU(cores)   MEMORY(bytes)
meter-api-6d9f7c8b5d-4kq2x   212m         301Mi
meter-api-6d9f7c8b5d-9zrtm   187m         296Mi
meter-api-6d9f7c8b5d-lp7wd   240m         318Mi

The meter-api pods use around 200m CPU against a 250m request, and about 300Mi of memory: above the 256Mi request but well below the 512Mi limit. Raising the memory request to 320Mi would make the scheduler's picture match reality.