Container Errors — OOMKilled, Exit Codes, Probes and Stale Images

Failures inside the container boundary: OOM kills, exit codes 1, 126, 127, 139, 143 and 255, StartError, readiness and liveness probe failures, and stale images, each with the mechanism, the commands and the EKS, AKS and GKE view.

Overview

At this layer the pod was scheduled and the image was pulled, so the problem is the process itself or the box it runs in. The runtime could not start the entrypoint, the process started and then exited, the kernel killed it, or it runs but Kubernetes will not send it traffic. One more failure hides here too: everything looks healthy, yet the old build is still the one serving.

Work outside-in, the same way every time. First read Reason and Exit Code in Last State, then read the previous container's log, and only then change something. The exit code already tells you which part of the system acted. Codes above 128 mean a signal killed the process, and the signal number is the code minus 128: 137 is SIGKILL (9), 143 is SIGTERM (15), 139 is SIGSEGV (11). Codes up to 128 came from the process, or from a shell inside the container.

The one idea to keep: find out who stopped the process. The kernel (OOMKilled), the kubelet (a failed liveness probe, or the end of the grace period), the runtime (StartError) and the application (exit 1) each leave a different trace. Each one needs a different fix, and raising a limit fixes only one of them.

Symptom to first command

Reason: OOMKilled, exit 137, restarts on a cycle — Memory limit below the real working set, or a runtime heap sized too close to it — kubectl get pod -n prod -o jsonpath='{.status.containerStatuses[*].lastState}'
Reason: Error, exit 1 or 255, CrashLoopBackOff — The application gave up: a dependency, config or migration failed; 255 can also be an entrypoint the runtime could not exec — kubectl logs -n prod --previous
Exit 126, or StartError with permission denied — Entrypoint not executable, wrong user, noexec mount — kubectl describe pod -n prod (read the Message line)
Exit 127, or StartError with no such file or directory — Wrong path, binary not copied into the final stage, no shell in the image, missing dynamic loader — docker run --rm --entrypoint sh -c 'ls -l /usr/local/bin/'
0/1 Running, zero restarts, 503 at the Ingress — Readiness probe failing, so the pod is left out of the EndpointSlice — kubectl get endpointslices -n prod -l kubernetes.io/service-name=orders
Rollout is done but the old behavior remains — Unchanged or mutable tag, cached image, or a GitOps revert — kubectl get pods -n prod -l app=orders -o jsonpath='{..imageID}'
Exit 143 or 137 with Reason: Error during deploys — Slow shutdown: the grace period ran out, or SIGTERM never reached the app — kubectl get deploy orders -n prod -o jsonpath='{.spec.template.spec.terminationGracePeriodSeconds}'
Exit 139 — Segmentation fault in native code or a libc mismatch — kubectl logs -n prod --previous --tail=50
Restarts under load, failed liveness probe events — CPU throttling makes a strict liveness probe time out, or liveness checks a dependency — kubectl events -n prod --for pod/

What you will be able to do

Read Last State and exit codes and say who stopped the process: the kernel, the kubelet, the runtime or the application.
Tell a container OOM kill apart from a node OOM kill and from a kubelet eviction, and size JVM, Node.js and Go runtimes to the container limit.
Explain why a StartError, a 126 and a 127 are different traces of the same missing or non-executable file.
Fix Running but not Ready, and CPU-throttled liveness restarts, without deleting the probes.
Prove which image digest is running, and make stale deploys impossible with digests, immutable tags and an admission policy.

Key concepts

Exit codes above 128: The process was killed by a signal; the signal number is the code minus 128. 137 is SIGKILL, 143 is SIGTERM, 139 is SIGSEGV.
StartError (exit 128): How containerd-based clusters report an entrypoint that runc could not start, such as a missing or non-executable file. Docker reports the same failures as 127 or 126.
cgroup v2 memory.max: The kernel limit a memory limit becomes. Anonymous memory, page cache and tmpfs are charged to it; the OOM killer runs only when reclaim cannot free enough.
Working set: Memory usage minus inactive file cache. The kubelet uses it for eviction, and it is the best metric to compare against the limit.
CFS quota throttling: A CPU limit becomes a CPU-time budget per 100 ms period; once it is used up, every thread waits for the next period, which adds latency even at low average CPU.
Readiness, liveness and startup probes: Readiness controls traffic through the EndpointSlice, liveness restarts the container, and startup holds both off until the app has started once.
Termination grace period: The time from the start of pod deletion, including preStop, until the kubelet sends SIGKILL; 30 seconds by default.
Native sidecar: An init container with restartPolicy Always (stable since 1.33) that starts before the app containers and runs for the life of the pod.
Image digest: The sha256 content address of an image. Pinning by digest means the spec changes whenever the content does, and a cached copy is always the right one.
In-place pod resize: Changing CPU and memory of a running pod through the resize subresource; beta and on by default in 1.33, GA in 1.35. It is not kept across a rollout.

When to use it

A container shows Last State Terminated with an exit code, and you need to know whether the kernel, the kubelet, the runtime or the application stopped it.
Pods restart on a cycle with OOMKilled, or memory climbs toward the limit and you need to size the JVM, Node.js or Go runtime to the container.
A new image fails to start with StartError, exit 126, 127 or 255, often after moving to a distroless, Alpine or Arm base.
Pods are Running but 0/1 Ready and the Ingress answers 503, or liveness restarts pile up at peak traffic.
A rollout reports success but the old build is still serving.
Rolling updates cause 502s or exits with 137 even though memory is fine.

Interview questions

A Java service in prod restarts every few hours with exit code 137. The team already doubled the memory limit once. Walk me through it.
After a deploy, the orders pods show 0/1 Running with zero restarts, and users get 503 from the Ingress. What happened, and how do you prove it?
A pipeline says the deploy succeeded, but the new feature is not live. How do you find out what is running, and how do you make this impossible?
A container fails with 'exec: /app/entrypoint.sh: permission denied'. One engineer says the code is 126, another says it is 128. Who is right?
Every rolling update produces a burst of 502s and a few pods end with exit 137. There is no memory pressure. Explain.
Pods restart under peak load, all at once, with 'failed liveness probe' events, but CPU averages 40 percent of the limit. What is going on?
How does diagnosing an OOM kill or a bad probe differ on EKS, AKS and GKE?