Docker — Production Errors, Mistakes and Recovery

Docker exit codes, OOM kills, stop signals, full disks, registry, network and daemon errors, decoded from the Docker docs. Plus the mistakes (prune, overwritten tags, baked-in secrets) and how to recover.

Overview

Docker failures in production fall into a small number of shapes. A container exits with a number you have to decode. A host fills its disk with images, build cache and logs nobody rotated. A pull fails because of a rate limit, a missing tag or the wrong CPU architecture. A stop takes ten seconds and ends in a kill. And a large share of incidents are not errors at all: someone pruned a volume, overwrote :latest, baked a token into a layer or ran a command against the wrong host.

The idea to keep: an exit code or an error string tells you which layer failed. 125 is the Docker daemon, 126 and 127 are the command you asked it to start, anything else is your program. 137 is a SIGKILL, and the container state says whether the kernel OOM killer sent it. Read the number first, then open the right layer.

What you will be able to do

Decode exit codes 0, 1, 125, 126, 127, 137, 139 and 143 and say which layer produced each.
Tell an OOM kill from a stop-timeout kill with docker inspect and docker events.
Fix the PID 1 signal problem that turns every docker stop into a 10-second wait and a 137.
Find what is filling the disk with docker system df and clean it without deleting data.
Diagnose pull failures: rate limits, authentication, missing manifests and platform mismatches.
Explain what a prune, docker rm -v or an overwritten tag destroys, what can be recovered, and which guardrails prevent it.

Symptom to first command

Container exited, unknown reason — docker inspect --format '{{.State.ExitCode}} {{.State.OOMKilled}}' web — Exit codes and signals
Exit 137 after docker stop — docker inspect --format '{{.Config.Entrypoint}} {{.Config.Cmd}}' web — Stop, PID 1 and signals
Writes fail with no space left on device — docker system df -v — Full disks
Pull fails: rate limit (429), authentication required, manifest unknown — docker pull IMAGE and read the registry error code — Registry, tags and platforms
exec format error at start — docker buildx imagetools inspect IMAGE — Registry, tags and platforms
port is already allocated — docker ps --filter publish=8080 — Networking
Containers cannot reach each other by name — docker network inspect bridge — Networking
Builds slow, every layer rebuilt — docker image history --no-trunc IMAGE and the Dockerfile order — Builds, cache and secrets
Cannot connect to the Docker daemon — docker context ls and env for DOCKER_HOST — Daemon, socket and permissions
Permission denied on a bind mount — docker exec web id and ls -ln on the host path — Daemon, socket and permissions
Data or an image vanished after a cleanup — docker volume ls, docker images --digests — Mistakes

Key concepts

Exit 125, 126, 127: Codes Docker itself returns: 125 an error in the Docker daemon, 126 the command could not be invoked, 127 the command was not found. Any other code belongs to the container process.
128 + n: Exit status of a process killed by signal n: 137 is SIGKILL (9), 139 is SIGSEGV (11), 143 is SIGTERM (15).
OOMKilled: Container state field that is true when a process in the container was killed for running out of memory since the container last started; paired with the oom event.
Stop timeout: Time docker stop waits after the stop signal (SIGTERM by default) before sending SIGKILL: 10 seconds for Linux containers, 30 for Windows; set with --stop-timeout or docker stop --timeout.
PID 1 and --init: The init process of a PID namespace only receives signals it has handlers for; --init runs a tiny init (backed by tini) that forwards signals and reaps processes.
Dangling image: An untagged : image left when a new build takes over its tag; docker image prune removes these by default.
Anonymous vs named volume: Anonymous volumes get a random name; docker rm -v, --rm and default volume prune remove them, while named volumes survive unless removed explicitly or pruned with --all.
Image digest: A SHA-256 identifier of image content; content tied to a digest cannot change, unlike a tag, which is a movable pointer.
Immutable tags: Registry setting (Docker Hub tag mutability, ECR IMMUTABLE) that refuses a push to an existing tag, so a release cannot be overwritten.
Build secret mount: RUN --mount=type=secret exposes a secret to one build step without storing it in a layer, unlike ARG and ENV, which persist.

When to use it

A container exits or restarts and you need to know whether Docker, the command, a signal or the application caused it.
Every docker stop takes the full timeout and ends in exit 137, or shutdowns lose in-flight work.
A Docker host runs out of disk and you need to reclaim space without deleting data.
Pulls fail in CI or production with rate limit, authentication, manifest or exec format errors.
Someone ran a prune, removed a container with -v or overwrote a tag, and you need to recover and add guardrails.
You are reviewing Dockerfiles and host setup for secrets in layers, root containers, socket mounts and --privileged.

Interview questions

A docker image tag my app:latest was overwritten in the registry and now a rollback to the previous version is impossible. How should the repo have been configured to prevent this, and can you still recover the old image?
Someone ran docker system prune -a --volumes on a production host. What was lost, what can you recover, and how do you prevent it?
A container exits with code 137 a few seconds after docker stop. What is happening?
A container keeps restarting with exit code 137 under load. How do you decide between a memory problem and something else?
A Docker host is out of disk. Walk me through reclaiming space safely.
CI started failing with HTTP 429 "You have reached your pull rate limit" when pulling base images. What do you do?
An image works on a developer laptop but fails in production with exec format error. Why?
Two containers on the same host cannot reach each other by name, but the IP works. What is wrong?
A security scan found an API key in a production image. The Dockerfile deletes the file in a later step. What do you do?
docker commands fail with Cannot connect to the Docker daemon. How do you troubleshoot?
A teammate fixed a production bug by running docker exec and editing a file inside the container. Why is that a problem, and what should happen instead?