Debugging Containers

A method for broken containers

The general approach is the one in a structured troubleshooting method: gather facts first, form a theory, test it, change one thing at a time. For containers, the facts come in a useful order, from the cheapest and safest to the most intrusive:

1. Status: docker ps -a. Is it running, restarting, exited (with which code), or only created?
2. Logs: docker logs. What did the program say before it stopped?
3. Configuration and state: docker inspect. Which image, command, environment, mounts, networks and health status?
4. Inside: docker exec. Only when the container runs and the first three did not explain it.
5. Around it: docker events, docker stats, the host's logs and disk space.

The recorded sessions on this page come from a test machine (Docker Engine 28.2 on a 64-bit ARM Raspberry Pi).

Reading the status

docker ps shows only running containers; a crashed one is only visible with -a:

$ docker run -d --name crash -e DB_HOST=db.kestrel.internal kestrel-meter-api:1.1 python -c 'import os,sys; print("connecting to", os.environ["DB_HOST"], flush=True); sys.exit("FATAL: database unreachable")'
$ docker ps -a --filter name=crash --format '{{.Names}} {{.Status}}'

195262544ec9cedaad545e57cd622e9821f6154b47c59dad9d50fb0fd8786f8f
crash Exited (1) 2 seconds ago

The status words tell you where to look next:

The exit codes in short (they are explained with recorded examples on Running Containers, and the 128 + signal rule on exit status): 0 is success; 1 and other low numbers come from the program itself; 125 means docker run itself failed, 126 that the command could not be executed and 127 that it was not found, so in those three cases the program never ran; 137 is 128 + 9 (SIGKILL) and 143 is 128 + 15 (SIGTERM).

When a service desk ticket lists several containers, a few lines of Python turn the State part of docker inspect into a first verdict for each:

import json

# The State part of `docker inspect` for six containers (trimmed).
containers = json.loads("""[
  {"Name": "/db", "RestartCount": 14,
   "State": {"Status": "restarting", "ExitCode": 1, "OOMKilled": false}},
  {"Name": "/meter-api", "RestartCount": 0,
   "State": {"Status": "exited", "ExitCode": 137, "OOMKilled": true}},
  {"Name": "/billing-export", "RestartCount": 0,
   "State": {"Status": "exited", "ExitCode": 137, "OOMKilled": false}},
  {"Name": "/portal", "RestartCount": 0,
   "State": {"Status": "running", "ExitCode": 0, "OOMKilled": false,
             "Health": {"Status": "unhealthy", "FailingStreak": 3}}},
  {"Name": "/report", "RestartCount": 0,
   "State": {"Status": "exited", "ExitCode": 127, "OOMKilled": false}},
  {"Name": "/export-test", "RestartCount": 0,
   "State": {"Status": "exited", "ExitCode": 0, "OOMKilled": false}}
]""")


def verdict(c):
    s = c["State"]
    code = s["ExitCode"]
    if s["Status"] == "restarting" or c["RestartCount"] > 5:
        return f"restart loop ({c['RestartCount']} restarts): read docker logs"
    if s["OOMKilled"]:
        return "killed for exceeding its memory limit: raise --memory or fix the leak"
    if s.get("Health", {}).get("Status") == "unhealthy":
        return "runs, but fails its health check: read State.Health.Log"
    if s["Status"] == "running":
        return "running"
    if code == 0:
        return "finished normally"
    if code in (125, 126, 127):
        return f"never started (exit {code}): check the command and the image"
    if code == 137:
        return "got SIGKILL: docker kill, or it ignored SIGTERM on docker stop"
    if code == 143:
        return "stopped with SIGTERM"
    return f"the application failed (exit {code}): read docker logs"


for c in containers:
    print(f"{c['Name'][1:]:15} {verdict(c)}")

db              restart loop (14 restarts): read docker logs
meter-api       killed for exceeding its memory limit: raise --memory or fix the leak
billing-export  got SIGKILL: docker kill, or it ignored SIGTERM on docker stop
portal          runs, but fails its health check: read State.Health.Log
report          never started (exit 127): check the command and the image
export-test     finished normally

Logs

docker logs prints what a container's main process wrote to its standard output and standard error. Docker captures both streams and keeps them, also after the container has stopped, until the container is removed. For the crashed container above:

$ docker logs crash

connecting to db.kestrel.internal
FATAL: database unreachable

Two lines answer the ticket: the program tried db.kestrel.internal and could not reach it. The next question is a network or DNS one (see Container Networking), not a container one. The options you will use most:

With the default json-file logging driver, Docker stores each line as one JSON object in a file under /var/lib/docker/containers/ on the host. The default does not rotate those files, which is how a chatty container fills a disk; see Running Containers for the settings. Because each line is JSON, it is easy to analyze, for example errors per minute (see also log correlation):

import json
from collections import Counter

# Lines as the json-file log driver stores them on the host
raw = r"""{"log":"meter-api listening on :8000\n","stream":"stdout","time":"2026-10-06T12:20:41.120Z"}
{"log":"GET /readings 200\n","stream":"stdout","time":"2026-10-06T12:20:50.031Z"}
{"log":"ERROR database timeout after 5s\n","stream":"stderr","time":"2026-10-06T12:21:03.402Z"}
{"log":"ERROR database timeout after 5s\n","stream":"stderr","time":"2026-10-06T12:21:09.877Z"}
{"log":"GET /health 200\n","stream":"stdout","time":"2026-10-06T12:21:30.005Z"}
{"log":"ERROR database timeout after 5s\n","stream":"stderr","time":"2026-10-06T12:22:01.660Z"}
{"log":"ERROR connection pool exhausted\n","stream":"stderr","time":"2026-10-06T12:22:02.118Z"}"""

errors = Counter()
first = {}
for line in raw.splitlines():
    entry = json.loads(line)
    if entry["stream"] == "stderr" and entry["log"].startswith("ERROR"):
        minute = entry["time"][11:16]            # HH:MM, in UTC
        errors[minute] += 1
        first.setdefault(minute, entry["log"].strip())

for minute in sorted(errors):
    print(f"{minute} UTC  {errors[minute]} errors  first: {first[minute]}")

12:21 UTC  2 errors  first: ERROR database timeout after 5s
12:22 UTC  2 errors  first: ERROR database timeout after 5s

Inspecting configuration and state

docker inspect prints everything Docker knows about a container (or an image, volume or network) as JSON: the command and environment it was started with, its mounts, networks, limits, and its current state. The full output runs to hundreds of lines; --format picks out fields with a Go template:

$ docker inspect --format 'exit={{.State.ExitCode}} error={{.State.Error}} finished={{.State.FinishedAt}}' crash
$ docker inspect --format '{{range .Config.Env}}{{println .}}{{end}}' crash

exit=1 error= finished=2026-10-06T12:24:34.559286341Z
DB_HOST=db.kestrel.internal
PATH=/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
GPG_KEY=7169605F62C751356D054A26A821E680E5FA6305
PYTHON_VERSION=3.13.16
PYTHON_SHA256=f4b1bfb3c79b5bb11b8d228a12504163b4c0dab4d679828d8f5f26b6cb6ab35d

State.Error is empty, so Docker itself had no trouble; the program chose to exit. The environment shows the variable from the command line plus those the Python base image sets (GPG_KEY and PYTHON_SHA256 are public values used to verify the Python download, not secrets). It also shows why real passwords do not belong in environment variables: anyone who can run docker inspect reads them. Fields worth knowing:

docker inspect meter-api | jq '.[0].State' is an alternative when you prefer jq to Go templates.

Looking inside

docker exec starts an extra process in a running container, in the same namespaces as its main process: the same file system, network and process list. With -it it gives you an interactive shell (docker exec -it meter-api sh). It only works on a running container:

$ docker exec crash sh

Error response from daemon: container 195262544ec9cedaad545e57cd622e9821f6154b47c59dad9d50fb0fd8786f8f is not running

For a container that crashes at start, start a new container from the same image with a shell as its entry point. It has the same files and the same software, without the crashing program:

$ docker run --rm --name crash-shell --entrypoint sh kestrel-meter-api:1.1 -c 'ls -la /app; python --version'

total 16
drwxr-xr-x 1 root root 4096 Oct  6 12:20 .
drwxr-xr-x 1 root root 4096 Oct  6 12:24 ..
-rw-r--r-- 1 root root  966 Oct  6 12:20 app.py
-rw-r--r-- 1 root root   17 Oct  6 12:20 requirements.txt
Python 3.13.16

Add -it and leave out -c … for an interactive shell. Without a terminal, for example in a script or a CI job, -t fails:

$ docker exec -it meter-api sh

the input device is not a TTY

Several commands answer questions without a shell inside the container at all. docker top lists its processes (with host process IDs; STIME is the host's local time), docker diff lists files changed in its writable layer (A added, C changed, D deleted), and docker cp copies files out for a closer look:

$ docker top meter-api
$ docker exec meter-api sh -c 'echo debug > /tmp/trace.txt'; docker diff meter-api
$ docker cp meter-api:/app/app.py ./app-from-container.py && head -1 app-from-container.py

UID                 PID                 PPID                C                   STIME               TTY                 TIME                CMD
10001               2409570             2409547             0                   14:20               ?                   00:00:00            python app.py
C /tmp
A /tmp/trace.txt
"""Kestrel Energy meter-api: a tiny demo service (fictional company)."""

Exit 137 and OOMKilled

A container with a memory limit (--memory 256m) that tries to use more is stopped by the Linux kernel's out-of-memory killer, which sends SIGKILL (see the OOM killer). OOMKilled is the field in a container's state (State.OOMKilled in docker inspect) that Docker sets to true when that happened because of the container's memory limit. On a host with working memory limits, a meter-api with a memory leak looks like this:

$ docker run -d --name meter-api --memory 128m registry.example.com/kestrel/meter-api:2.3.0
$ docker ps -a --filter name=meter-api --format 'table {{.Names}}\t{{.Status}}'
$ docker inspect --format 'exit={{.State.ExitCode}} oom={{.State.OOMKilled}}' meter-api

3f9c1e7a52d04b8e96a1f0c3d7b25e4a8c6f1d09e2b7a3c5f8d4e6b1a9c0f2d7
NAMES       STATUS
meter-api   Exited (137) 12 seconds ago
exit=137 oom=true

Exit code 137 alone does not prove a memory problem. On the test machine, two containers ended with 137 without any memory limit: one was stopped with docker kill, the other was a process that ignored the polite SIGTERM of docker stop, so Docker sent SIGKILL after the 10-second grace period:

$ docker inspect --format '{{.Name}} exit={{.State.ExitCode}} oom={{.State.OOMKilled}}' kill pid1

/kill exit=137 oom=false
/pid1 exit=137 oom=false

Reading 137, then:

• oom=true: the container hit its memory limit. Check docker stats for its normal usage, then raise the limit or fix the leak. For Java, see JVM memory in containers.
• oom=false and it took about ten seconds to stop: the program ignores SIGTERM, often because it runs as process 1 through a shell (see Running Containers).
• oom=false without a stop or kill: look in the host's kernel log (journalctl -k | grep -i 'out of memory'). Without a container limit, the host itself can run out of memory and the kernel kills a process there.

A container does not always stop when memory runs out. If the kernel kills a child process, for example one Gunicorn worker of the meter-api, the main process starts a new worker and the container keeps running; only the application log mentions that a worker was killed. Repeated worker restarts in the application log are the clue.

When memory limits are silently ignored

A real-world gotcha: on the Raspberry Pi where this page's output was recorded, the kernel boots with the memory controller of cgroups switched off (cgroup_disable=memory, the Raspberry Pi OS default on that machine). Docker warns, and then starts the container anyway, without the limit:

$ docker info 2>&1 | grep WARNING
$ docker run --rm --pull never --name mem --memory 16m nginx:alpine sh -c 'cat /sys/fs/cgroup/memory.max 2>&1; ls /sys/fs/cgroup | head -5'

WARNING: No memory limit support
WARNING: No swap limit support
WARNING: Your kernel does not support memory limit capabilities or the cgroup is not mounted. Limitation discarded.
cat: can't open '/sys/fs/cgroup/memory.max': No such file or directory
cgroup.controllers
cgroup.events
cgroup.freeze
cgroup.kill
cgroup.max.depth

"Limitation discarded": the container runs without any memory limit, and docker stats on that host shows 0B / 0B in the memory column. Anyone who reads only the docker run command believes the container is limited to 16 MB. The warning appears once, at start, and is easy to miss in a script's output. The same missing controller stopped a local Kubernetes cluster from starting on that machine (see kubectl and Troubleshooting). On Raspberry Pi OS the usual fix is to add cgroup_enable=memory to the kernel command line in /boot/firmware/cmdline.txt and reboot; plan that as a change, because it restarts everything on the host.

Health checks

A running process is not necessarily a working service: the meter-api can be up but unable to answer requests. The HEALTHCHECK instruction in a Dockerfile defines a command that Docker runs inside the container at a regular interval to test it; exit code 0 means healthy and 1 unhealthy. Compose's healthcheck setting does the same per service (see Docker Compose).

HEALTHCHECK --interval=30s --timeout=3s --start-period=10s --retries=3 \
  CMD ["python", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8000/healthz', timeout=2)"]

Without options, the defaults are an interval of 30 seconds, a timeout of 30 seconds, three retries and no start period. The status starts as starting, becomes healthy after the first passing check, and unhealthy after retries failures in a row. Failures during the start period do not count, which gives a slow application time to start.

When a check fails, its own output is stored in the container's state. In the test run of Docker Compose with a health check on the wrong port:

$ docker inspect --format '{{.State.Health.Status}} streak={{.State.Health.FailingStreak}}' kestrel-unhealthy-meter-api-1
$ docker inspect --format '{{(index .State.Health.Log 0).ExitCode}} {{(index .State.Health.Log 0).Output}}' kestrel-unhealthy-meter-api-1

unhealthy streak=3
1 Traceback (most recent call last):
…
ConnectionRefusedError: [Errno 111] Connection refused
…
urllib.error.URLError: <urlopen error [Errno 111] Connection refused>

"Connection refused" while the service itself runs fine points at the check, not at the service: here it tried port 9999 instead of 8000. Plain Docker Engine only reports health; it does not restart an unhealthy container. Compose uses the status to hold back dependent services, and orchestrators replace unhealthy containers, Kubernetes with its own probes (see Health, Resources and Scaling). Keep the check cheap and local: it runs every interval, and a check that also tests the database turns a database hiccup into "every service unhealthy".

Debugging images without a shell

Minimal images, such as distroless images, contain the application and its libraries but no shell and no tools (see Building Good Images). That is good for security and awkward for debugging: docker exec meter-api sh then fails with an error that contains executable file not found in $PATH.

The answer is to bring the tools in a second container that shares the broken container's namespaces. It sees the same network (--network container:…) and the same processes (--pid container:…), but has its own file system with the tools:

$ docker run --rm -it \
    --network container:meter-api \
    --pid container:meter-api \
    busybox:1.37 sh

From that shell, wget -qO- http://127.0.0.1:8000/healthz tests the service from its own point of view and ps shows its processes; the application's files are reachable under /proc/1/root/ (with enough privileges). Use a debugging image your organization trusts; images with network tools such as tcpdump and dig are common. In Kubernetes, kubectl debug does the same with an ephemeral container (see kubectl and Troubleshooting).

Events and common start-up errors

docker events streams what the daemon does: containers created, started, dying, health status changes, executions. With --since and --until it shows the history of a time window, which reconstructs a restart loop or a colleague's actions:

$ docker events --since 1791289474 --until $(date +%s) --filter container=crash --filter container=crash-shell --filter container=meter-api --format '{{.Type}} {{.Action}} {{.Actor.Attributes.name}}' | head -20

container create crash
container start crash
container die crash
container create crash-shell
container attach crash-shell
container start crash-shell
container die crash-shell
container destroy crash-shell
container destroy crash
container top meter-api
container exec_create: sh -c echo debug > /tmp/trace.txt meter-api
container exec_start: sh -c echo debug > /tmp/trace.txt meter-api

(--since takes a Unix timestamp or a duration such as 30m.) The messages below cover most containers that refuse to start. Where the wording was recorded on the test machine, it is quoted exactly:

Write what you found in the ticket as you go: the status, the exit code, the relevant log lines and the inspect fields. It turns the next handover into a two-minute read (see reading errors and logs).