kubectl and Troubleshooting

Working with kubectl and finding faults: get, describe, logs and events, and the common failures CrashLoopBackOff, ImagePullBackOff, Pending and OOMKilled, with a step-by-step approach.

Which cluster am I talking to?

kubectl sends every command to an API server, and which one it uses comes from a kubeconfig file: a YAML file with clusters (API server addresses and their CA certificates), users (credentials) and contexts that pair a cluster with a user and a default namespace. kubectl reads ~/.kube/config unless the KUBECONFIG environment variable or the --kubeconfig option points elsewhere. The file holds credentials, so protect it like a password. 

$ kubectl config get-contexts
$ kubectl config current-context
CURRENT   NAME                CLUSTER             AUTHINFO          NAMESPACE
          kestrel-test        kestrel-test        ravi@kestrel      kestrel-portal
*         kestrel-prod        kestrel-prod        ravi@kestrel      kestrel-portal
kestrel-prod

The asterisk marks the current context — here production. Switch with kubectl config use-context kestrel-test, or pass --context on a single command. Like every Kubernetes page in this topic, the cluster output on this page is illustrative, written from the Kubernetes documentation; the one exception is the real failure in the last section.

Keep kubectl within one minor version of the cluster: kubectl 1.37 is supported against API servers 1.36, 1.37 and 1.38.

The everyday commands

Command What it is for
kubectl get pods -n kestrel-portal The list: status, readiness, restarts, age
kubectl get pods -o wide Adds pod IP and node — is it always the same node?
kubectl get pods -l app.kubernetes.io/name=meter-api Only pods with that label
kubectl get pods -A All namespaces
kubectl get deploy meter-api -o yaml The full object as stored, including status
kubectl get pods -w Watch changes as they happen
kubectl explain deployment.spec.strategy Documentation of a field, from the cluster's own API 
$ kubectl get pods -n kestrel-portal
NAME                         READY   STATUS             RESTARTS        AGE
portal-7f9c6d5b8-2kq7x       1/1     Running            0               3d4h
portal-85c4b7d9f-x8w2p       0/1     ImagePullBackOff   0               6m12s
meter-api-6d9f7c8b5d-4kq2x   0/1     CrashLoopBackOff   6 (2m10s ago)   11m
meter-api-6d9f7c8b5d-9zrtm   1/1     Running            3 (41m ago)     3d4h
meter-api-7b6c9d4f8-r2xkd    0/1     Pending            0               4m

One screen, four different problems. The rest of this page goes through them; first the tools that explain them.

describe: the object and its story

kubectl describe prints a human-readable summary of one object — for a pod: its node, labels, each container's image, state, last state, exit code, restart count, probes, requests and limits, volumes — and at the bottom the recent events that concern it. When a pod misbehaves, it is almost always the first command to run after get.

Events

An event is a short record that a Kubernetes component writes about an object: the scheduler assigned a pod, the kubelet pulled an image, a probe failed, a ReplicaSet could not create a pod. Each has a type (Normal or Warning), a reason such as FailedScheduling or BackOff, and a message. Repeated events are counted instead of duplicated (x4 over 6m). 

$ kubectl events -n kestrel-portal --types=Warning
LAST SEEN            TYPE      REASON             OBJECT                           MESSAGE
4m                   Warning   FailedScheduling   Pod/meter-api-7b6c9d4f8-r2xkd    0/3 nodes are available: 3 Insufficient cpu. …
61s (x20 over 6m)    Warning   Failed             Pod/portal-85c4b7d9f-x8w2p       Error: ImagePullBackOff
50s (x31 over 11m)   Warning   BackOff            Pod/meter-api-6d9f7c8b5d-4kq2x   Back-off restarting failed container meter-api in pod meter-api-6d9f7c8b5d-4kq2x_kestrel-portal(…)

Events are kept only for a limited time — by default about an hour — so look early, and copy relevant ones into the incident ticket. Events about a Deployment or ReplicaSet (for example FailedCreate when Pod Security Admission rejects pods) are on those objects, not on any pod.

logs

kubectl logs shows what a container wrote to standard output and standard error. Useful options: -f (follow), --since=10m, --tail=50, -c <container> for pods with several containers, -l app.kubernetes.io/name=meter-api for all pods with a label, and above all --previous (-p): the logs of the previous run of the container — the one that crashed.

exec, port-forward and debug

  • kubectl exec -it <pod> -- sh runs a command in a running container, if the image has a shell.
  • kubectl port-forward svc/meter-api 18000:80 forwards a local port to a Service or pod through the API server — like SSH port forwarding. Close it when done, especially against production.
  • kubectl debug -it <pod> --image=busybox --target=meter-api adds an ephemeral debug container to a running pod: the way in for minimal images without a shell (see Debugging Containers for the Docker side).

For many objects at once, -o json combined with jq or a short script beats reading screens; an example follows below.

A method that works

The general structured troubleshooting method applies unchanged. For a workload in Kubernetes it usually becomes:

  1. Scope: context and namespace right? One pod, all pods of a Deployment, or a whole node? (-o wide shows the node.)
  2. Status: read the STATUS column and the READY and RESTARTS counts.
  3. Story: describe the pod; read State, Last State, exit code and the events.
  4. Logs: logs, and logs --previous after a restart.
  5. What changed: kubectl rollout history, the last merge in Git, the last change ticket. Most incidents follow a change.
You see First command Likely causes
CrashLoopBackOff logs --previous Application error at start, missing configuration, wrong command, failing liveness probe
ImagePullBackOff / ErrImagePull describe (events) Wrong name or tag, no pull secret for a private registry, rate limit, wrong architecture
Pending describe (FailedScheduling) Requests too large, node selectors or taints, unbound volume claim
Restarts, Last State OOMKilled describe (Last State) Memory limit too low, or a memory leak
Running but READY 0/1 describe (readiness probe) Readiness probe failing: dependency down, wrong path or port
CreateContainerConfigError describe (events) A referenced ConfigMap or Secret (or key) does not exist

Triage many pods with a script

With dozens of pods, a script over kubectl get pods -o json lists only what needs attention, with the first step for each: 

import json

# Trimmed output of: kubectl get pods -n kestrel-portal -o json
pods_json = """
{"items": [
 {"metadata": {"name": "portal-7f9c6d5b8-2kq7x"},
  "status": {"phase": "Running", "containerStatuses": [
   {"ready": true, "restartCount": 0, "state": {"running": {}}}]}},
 {"metadata": {"name": "meter-api-6d9f7c8b5d-4kq2x"},
  "status": {"phase": "Running", "containerStatuses": [
   {"ready": false, "restartCount": 6,
    "state": {"waiting": {"reason": "CrashLoopBackOff"}},
    "lastState": {"terminated": {"reason": "Error", "exitCode": 1}}}]}},
 {"metadata": {"name": "meter-api-6d9f7c8b5d-9zrtm"},
  "status": {"phase": "Running", "containerStatuses": [
   {"ready": true, "restartCount": 3, "state": {"running": {}},
    "lastState": {"terminated": {"reason": "OOMKilled", "exitCode": 137}}}]}},
 {"metadata": {"name": "portal-85c4b7d9f-x8w2p"},
  "status": {"phase": "Pending", "containerStatuses": [
   {"ready": false, "restartCount": 0,
    "state": {"waiting": {"reason": "ImagePullBackOff"}}}]}},
 {"metadata": {"name": "meter-api-7b6c9d4f8-r2xkd"},
  "status": {"phase": "Pending", "conditions": [
   {"type": "PodScheduled", "status": "False", "reason": "Unschedulable"}]}}
]}
"""

FIRST_STEP = {
    "CrashLoopBackOff": "kubectl logs --previous",
    "ImagePullBackOff": "kubectl describe pod (events)",
    "Unschedulable": "kubectl describe pod (FailedScheduling)",
    "OOMKilled": "check memory limit and usage",
}

for pod in json.loads(pods_json)["items"]:
    name, status = pod["metadata"]["name"], pod["status"]
    containers = status.get("containerStatuses", [])
    problem, restarts = "", sum(c["restartCount"] for c in containers)
    for c in containers:
        waiting = c["state"].get("waiting", {}).get("reason")
        last = c.get("lastState", {}).get("terminated", {}).get("reason")
        problem = waiting or ("OOMKilled" if last == "OOMKilled" else "")
    for condition in status.get("conditions", []):
        if condition["type"] == "PodScheduled" and condition["status"] == "False":
            problem = condition["reason"]
    if problem:
        print(f"{name:<28} {status['phase']:<8} {problem:<17} restarts={restarts}")
        print(f"{'':<28} next: {FIRST_STEP[problem]}")
meter-api-6d9f7c8b5d-4kq2x   Running  CrashLoopBackOff  restarts=6
                             next: kubectl logs --previous
meter-api-6d9f7c8b5d-9zrtm   Running  OOMKilled         restarts=3
                             next: check memory limit and usage
portal-85c4b7d9f-x8w2p       Pending  ImagePullBackOff  restarts=0
                             next: kubectl describe pod (events)
meter-api-7b6c9d4f8-r2xkd    Pending  Unschedulable     restarts=0
                             next: kubectl describe pod (FailedScheduling)

Note the phases. A pod in CrashLoopBackOff is still in phase Running, and a pod waiting for its image is Pending: the phase is a coarse summary (Pending, Running, Succeeded, Failed, Unknown), while words like CrashLoopBackOff are container states that kubectl shows in its STATUS column. A pod that restarted after an OOM kill looks perfectly healthy now — only its restart count and last state give it away.

CrashLoopBackOff

CrashLoopBackOff means a container keeps exiting, and the kubelet is waiting before it restarts it again. The waits grow: exponential backoff from 10 seconds, doubling, capped at 5 minutes, and reset once the container has run for 10 minutes without problems. 

def backoff_delays(restarts, first=10, cap=300):
    """CrashLoopBackOff waits: 10 s, doubling each time, at most 5 minutes."""
    delay = first
    for _ in range(restarts):
        yield delay
        delay = min(delay * 2, cap)


RUNTIME = 5  # this container crashes 5 seconds after every start
clock = RUNTIME
print("restart   wait   restarted at")
for number, delay in enumerate(backoff_delays(8), start=1):
    clock += delay
    print(f"{number:>7}  {delay:>3} s   {clock // 60:>2} min {clock % 60:>2} s")
    clock += RUNTIME
restart   wait   restarted at
      1   10 s    0 min 15 s
      2   20 s    0 min 40 s
      3   40 s    1 min 25 s
      4   80 s    2 min 50 s
      5  160 s    5 min 35 s
      6  300 s   10 min 40 s
      7  300 s   15 min 45 s
      8  300 s   20 min 50 s

So after the first few minutes a crashing pod restarts only every 5 minutes, and a fix can take up to 5 minutes to show — or delete the pod to get a fresh one immediately. The reason for the crash is in the logs of the run that died: 

$ kubectl logs meter-api-6d9f7c8b5d-4kq2x -n kestrel-portal --previous
2026-10-06T09:41:12Z INFO  meter-api 2.3.0 starting
2026-10-06T09:41:12Z ERROR configuration: environment variable DATABASE_HOST is not set

Here a ConfigMap key was renamed in the last change. Other usual suspects: a wrong command or arguments, a missing file or permission (non-root user, read-only file system), a dependency that is not reachable at start, or a liveness probe that kills a healthy but slow application (see Health, Resources and Scaling). If logs --previous is empty, the process died before writing anything: read Last State in describe. A reason StartError with a message that the executable was not found means the container's command or entrypoint is wrong.

ImagePullBackOff

ImagePullBackOff means the kubelet could not pull the container image and is waiting before it tries again (with the same growing back-off). The first failed attempt shows as ErrImagePull. The events say why: 

$ kubectl describe pod portal-85c4b7d9f-x8w2p -n kestrel-portal
…
Events:
  Type     Reason     Age                    From               Message
  ----     ------     ----                   ----               -------
  Normal   Scheduled  6m12s                  default-scheduler  Successfully assigned kestrel-portal/portal-85c4b7d9f-x8w2p to worker-2
  Normal   Pulling    4m40s (x4 over 6m11s)  kubelet            Pulling image "registry.example.com/kestrel/portal:1.4.3"
  Warning  Failed     4m40s (x4 over 6m10s)  kubelet            Failed to pull image "registry.example.com/kestrel/portal:1.4.3": rpc error: code = NotFound desc = failed to pull and unpack image "registry.example.com/kestrel/portal:1.4.3": failed to resolve reference "registry.example.com/kestrel/portal:1.4.3": registry.example.com/kestrel/portal:1.4.3: not found
  Warning  Failed     4m40s (x4 over 6m10s)  kubelet            Error: ErrImagePull
  Normal   BackOff    61s (x20 over 6m9s)    kubelet            Back-off pulling image "registry.example.com/kestrel/portal:1.4.3"
  Warning  Failed     61s (x20 over 6m9s)    kubelet            Error: ImagePullBackOff

not found: the tag 1.4.3 was never pushed — the pipeline failed after the manifest was updated. The message text points to the cause:

  • not found / manifest unknown: typo in name or tag, or the image was never pushed.
  • unauthorized / authentication required: a private registry without an imagePullSecrets entry (or with an expired credential).
  • 429 Too Many Requests: a registry rate limit, such as Docker Hub's limits for anonymous pulls (see Images, Layers and Registries).
  • no match for platform in manifest: the image has no variant for the node's architecture (an amd64-only image on arm64 nodes).
  • timeouts or name-resolution errors: the node cannot reach the registry — a firewall, proxy or DNS problem on the node, not in the pod.

Pending pods

A pending pod has been accepted by the API server but has not started running: usually because the scheduler cannot find a node for it, sometimes because its images are still being pulled. When the scheduler is the problem, describe shows a FailedScheduling event that counts the nodes and the reasons: 

$ kubectl describe pod meter-api-7b6c9d4f8-r2xkd -n kestrel-portal
…
    Requests:
      cpu:     4
      memory:  256Mi
…
Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  4m    default-scheduler  0/3 nodes are available: 3 Insufficient cpu. preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod.

The new version requests cpu: 4 — four whole cores — where 400m was meant; no node has four cores free. Remember that scheduling counts requests, not actual usage: a node that is 10% busy can still be "full". Other reasons in the same message:

  • Insufficient memory: the same, for memory requests.
  • didn't match Pod's node affinity/selector: a nodeSelector or affinity rule that no node satisfies (a label typo, or the right nodes are full).
  • had untolerated taint: nodes reserved for something else, such as control-plane nodes.
  • A pod waiting for a PersistentVolumeClaim that is not bound stays Pending too (see Storage in Kubernetes).

OOMKilled and not ready

A container that goes over its memory limit is killed by the kernel and restarted. The pod may look fine a minute later; describe keeps the evidence in Last State: 

    State:          Running
      Started:      Tue, 06 Oct 2026 08:58:03 +0000
    Last State:     Terminated
      Reason:       OOMKilled
      Exit Code:    137
      Started:      Tue, 06 Oct 2026 08:12:47 +0000
      Finished:     Tue, 06 Oct 2026 08:58:01 +0000
    Ready:          True
    Restart Count:  3
    Limits:
      memory:  512Mi
    Requests:
      cpu:     250m
      memory:  256Mi

Exit code 137 is 128 + 9: killed with SIGKILL. With reason OOMKilled, the memory limit did it. Compare usage over time with the limit: a steady climb before every kill points to a leak, a sudden peak under load to a limit that is simply too low. The fix is in the manifest (or the code), not in deleting the pod. The details of units and limits are on Health, Resources and Scaling.

A pod that is Running with READY 0/1 is the readiness probe at work: the container runs, but it gets no traffic. describe shows Readiness probe failed events with the HTTP status or error; check the dependency the probe tests, and whether the Service has any ready endpoints at all (kubectl get endpointslices -l kubernetes.io/service-name=meter-api).

A real case: a control plane that never starts

While this topic was written, the plan was to record real cluster output with kind (Kubernetes in Docker), which runs a test cluster inside Docker containers, on a Raspberry Pi whose kernel boots with cgroup_disable=memory. That plan failed — and the failure is a better lesson than the output would have been. Everything in this section is real; only resource names were shortened.

The symptom

$ kind create cluster --name kestrel --config kind-config.yaml --kubeconfig "$HOME/.kube/kind-kestrel"
Creating cluster "kestrel" ...
 • Ensuring node image (kindest/node:v1.37.0) 🖼️  ...
 ✓ Ensuring node image (kindest/node:v1.37.0) 🖼️
 • Preparing nodes 📦   ...
 ✓ Preparing nodes 📦 
 • Writing configuration 📜  ...
 ✓ Writing configuration 📜
 • Starting control-plane 🕹️  ...
 ✗ Starting control-plane 🕹️
Deleted nodes: ["kestrel-control-plane"]
ERROR: failed to create cluster: failed to init node with kubeadm: command "docker exec --privileged kestrel-control-plane kubeadm init --config=/kind/kubeadm.conf --skip-token-print --v=6" failed with error: exit status 1

Command Output: I1006 12:24:58.630620     191 initconfiguration.go:262] loading configuration from "/kind/kubeadm.conf"
W1006 12:24:58.632798     191 initconfiguration.go:363] [config] WARNING: Ignored configuration document with GroupVersionKind kubeadm.k8s.io/v1beta4, Kind=JoinConfiguration
…
error: error execution phase wait-control-plane: cannot obtain client without bootstrap: could not bootstrap the admin user in file admin.conf: unable to create ClusterRoleBinding: client rate limiter Wait returned an error: rate: Wait(n=1) would exceed context deadline
…

Read it from the bottom up. The last error is a timeout: would exceed context deadline while creating a ClusterRoleBinding. That is a symptom: kubeadm, the tool that sets up the control plane, gave up waiting for an API server that never answered. A timeout tells you where things stopped, not why. Retrying, or raising the timeout, would only fail more slowly.

One layer down

kind deletes the failed node unless you add --retain. With the node container kept, you can ask its container runtime directly what happened to the control-plane containers, using crictl (the command-line tool for any runtime behind the Container Runtime Interface): 

$ docker exec kestrel-control-plane crictl ps -a
CONTAINER           IMAGE               CREATED             STATE               NAME                      ATTEMPT             POD ID              POD                                             NAMESPACE
7bddc3b0c3611       384eabc4fe526       2 seconds ago       Exited              kube-scheduler            4                   20b6913de0e08       kube-scheduler-kestrel-control-plane            kube-system
a28a47a538a90       6003d52023b9d       39 seconds ago      Exited              kube-apiserver            3                   28d286c11927a       kube-apiserver-kestrel-control-plane            kube-system
2a18e277e560a       c2e426bbd1cd6       40 seconds ago      Exited              etcd                      3                   20c99b1db548e       etcd-kestrel-control-plane                      kube-system
2f57dc6b00bb0       b1ad0e33c9012       40 seconds ago      Exited              kube-controller-manager   3                   36f29ccfd80a1       kube-controller-manager-kestrel-control-plane   kube-system

All four control-plane components — etcd, the API server, the controller manager and the scheduler — are Exited, and the ATTEMPT column shows the kubelet restarting them over and over. The API server never stayed up long enough to answer kubeadm. Why did they exit? crictl inspect on one of them holds the status reason and message: 

$ docker exec kestrel-control-plane crictl inspect CONTAINER-ID
…
    "message": "failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error setting cgroup config for procHooks process: unified resource \"memory.low\" can't be set: controller \"memory\" not available",
    "reason": "StartError",
…

The root cause

The last clause is the real error: controller "memory" not available. The containers did not crash; runc could not even create them, because it was asked to set a memory setting (memory.low, which the kubelet uses to protect requested memory) in a cgroup without a memory controller. The kernel was booted with cgroup_disable=memory, so the memory controller simply does not exist on this machine. Kubernetes needs it: the kubelet enforces memory limits and requests through it.

Docker had been saying the same thing all along, to anyone who looked: 

$ docker info 2>&1 | grep WARNING
WARNING: No memory limit support
WARNING: No swap limit support

On such a machine, docker run --memory limits are discarded with a warning, and containers simply have no memory limit — easy to miss for months.

The fix, and the lessons

Two workarounds that switch off parts of the kubelet's cgroup handling were tried; both failed with new errors, because the kubelet's node-level resource management depends on the memory controller. The real fix is to enable the controller: add cgroup_enable=memory to the kernel command line (on a Raspberry Pi, the cmdline.txt file in the boot partition) and reboot. (The Pi firmware adds cgroup_disable=memory itself, so it is not in cmdline.txt; cgroup_enable=memory, appended later, overrides it.) On this machine the fix was not applied: it needs administrator rights and a reboot of a machine that runs other services, so it is a planned change with an owner and a maintenance window, not something to do in passing while writing documentation.

  • Follow the chain down. kind error → kubeadm timeout → exited control-plane containers → runtime start error → kernel setting. Every layer reported the failure of the layer below in its own words.
  • The first real error is the one that matters, and it is rarely the last line on the screen. Look for words like "failed to create", "not available", "permission denied" under the timeouts.
  • Check the prerequisites before the tool. A quick look at warnings in docker info would have predicted the failure before the first download.
  • Know when to stop. Tweaking settings until the error message changes is not a fix. When the cause is found and the fix carries risk, write it down and hand it to the change process — see working safely on production systems.