OOMKilled Pod Diagnosis

Diagnose OOMKilled container failures in Kubernetes using metrics, heap dumps, and memory profiling.

Overview

Overview

OOMKilled is a pod termination state: the Linux kernel's OOM killer ended a container that exceeded its memory limit. The pod exits with reason OOMKilled and exit code 137 (128 + SIGKILL), and the process gets no warning, no graceful shutdown signal and no chance to write error logs; it simply stops and Kubernetes restarts it.

Request vs limit

  • Used by — The scheduler, to find a node with enough free memory — The Linux kernel, via cgroup memory.limit_in_bytes
  • Meaning — A reservation — A hard ceiling; exceed it and the container is killed

> [!WARNING] A limit equal to the request (request: 512Mi, limit: 512Mi) means any spike above 512Mi is an OOM kill. Set the limit 25-50% above the request to absorb spikes.

Checks and what each tells you

  • kubectl describe pod pod-name — Last State: Terminated, Reason: OOMKilled, Exit Code: 137 confirms the OOM kill
  • kubectl top pod pod-name --containers — Current memory per container
  • container_memory_working_set_bytes (Prometheus) — The trend; the metric Kubernetes uses for eviction; memory that cannot be freed without application action, unlike reclaimable page cache
  • Heap dump in Eclipse MAT or VisualVM — The object type holding the most retained heap

> [!TIP] A container running at 90% of its limit is OOM-kill-prone.

The four causes

  • Memory leak — Objects allocated and never released — Rising slope or sawtooth; kills at the same absolute memory value
  • Bulk data processing — Whole dataset loaded into memory (entire CSV, complete JSON) instead of streamed — Sudden spikes proportional to input size
  • Connection pool growth — DB connections, gRPC channels or HTTP clients accumulate without a pool bound — Each connection holds buffers and state
  • JVM native memory — -Xmx is right, but heap + metaspace + code cache + thread stacks exceed the limit — Kill while the heap looks healthy

JVM fix: before and after

> [!IMPORTANT] Setting -Xmx to 75% of the limit is not enough on its own: add -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/heapdump.hprof to capture evidence.

Key concepts

OOMKilled
A Kubernetes pod termination state (exit code 137) caused by the Linux kernel OOM killer terminating a container that exceeded its cgroup memory limit.
container_memory_working_set_bytes
The Prometheus metric used by Kubernetes for memory accounting and eviction decisions — represents memory that cannot be freed without application action, excluding reclaimable page cache.
Memory Limit
The hard ceiling on container memory usage enforced by Linux cgroups — exceeding this limit triggers an immediate SIGKILL from the kernel OOM killer with no warning.
HeapDumpOnOutOfMemoryError
JVM flag (-XX:+HeapDumpOnOutOfMemoryError) that causes the JVM to write a heap dump to disk when the application runs out of heap memory, enabling post-mortem analysis.
Memory Leak
A pattern where allocated memory is not released after use, causing working set to grow monotonically until it hits the container limit, triggering repeated OOM kills at regular intervals.

When to use it

  • Diagnosing pods that restart repeatedly with OOMKilled status
  • Distinguishing between memory leaks, bulk processing spikes, and misconfigured limits
  • Setting memory requests and limits correctly to absorb spikes without frequent OOM kills
  • Configuring JVM heap settings to prevent native memory from exceeding container limits
  • Using heap dumps to identify the root cause of memory growth in JVM applications

Interview questions

  • How do you distinguish between a memory leak and a legitimate memory spike causing OOM kills?
  • What is the correct approach to setting memory requests and limits in Kubernetes to handle both baseline and spike usage?
  • How do you configure a JVM containerized application to avoid OOM kills from native memory exceeding the container limit?

Sign up free to read the full topic

More Troubleshooting topics

Start free on Cariara · Coding problems · System design studio