Memory Leak Diagnosis
Container OOMKill events, heap dumps, JVM tuning, Go pprof, Python tracemalloc, and RSS growth patterns.
Overview
Overview
A memory leak makes a process's RSS (Resident Set Size) grow steadily over time with no matching rise in load. Eventually the process passes its memory limit (the container limit or free OS memory) and the OOM killer ends it. In Kubernetes this shows up as OOMKilled with exit code 137.
Diagnosis flow
Confirm it is a leak first, then find which memory region grows, then find the allocation site.
> [!IMPORTANT] A high but stable RSS is not a leak. A true leak shows a steady, non-plateauing upward slope.
Leak patterns by type
- Heap leak — Objects that are never garbage collected — JS retained event listeners, Java collections that grow forever
- Connection leak — Database or HTTP connections — Connections held open
- File descriptor leak — Open files — Files opened but not closed
- Native memory leak — Memory outside the managed heap — JVM metaspace and off-heap buffers, Go CGO allocations
Tools and what each tells you
- RSS over time — Plot memory per pod — Leak (steady climb) or normal use (plateau)
- JVM heap dump — -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/heapdump.hprof — A snapshot of the heap at the moment of OOM
- Eclipse Memory Analyzer (MAT) — Open the .hprof — Dominator tree of the largest object trees; leak suspects report of objects accumulating across snapshots
- Go pprof — import _ "net/http/pprof" exposes /debug/pprof/ — Heap profile via go tool pprof http://service/debug/pprof/heap
- Go allocs vs heap — Choose the profile — allocs is all-time allocations; heap is live allocations
> [!TIP] Take two Go heap profiles minutes apart and compare them: the allocations that grow between the two are the leak.
Key concepts
- RSS growth over time
- Definitive sign of a memory leak. Plot container memory usage in Prometheus over 24+ hours. A true leak shows steady slope; a cache fills and stabilizes.
- Heap dump (JVM)
- Snapshot of the JVM heap. Enable with -XX:+HeapDumpOnOutOfMemoryError. Analyze with Eclipse MAT. Look for dominator tree and leak suspects report.
- Go pprof heap
- Go profiling endpoint: /debug/pprof/heap. Compare two snapshots: go tool pprof -base heap1.out heap2.out. Shows delta allocations — what grew between snapshots.
- OOMKilled (exit code 137)
- Kubernetes terminates a container that exceeds its memory limit. kubectl describe pod shows "OOMKilled" in Last State. dmesg on the node shows the OOM kill event with memory stats.
- finalization vs WeakReference (Java)
- Objects with finalizers (or held by strong references from caches/listeners) are not garbage collected even when seemingly unreachable. Common source of heap leaks in Java.
When to use it
- Diagnosing a container that is repeatedly OOMKilled with increasing memory over time
- Taking a heap dump from a JVM application at the moment of OOM
- Using Go pprof to find which goroutine or allocation is growing
- Distinguishing a true memory leak from a large but stable cache
Interview questions
- A Node.js service's memory usage grows 50 MB per hour and restarts every 8 hours. How do you find the leak?
Sign up free to read the full topic
More Troubleshooting topics
- HPA Not Scaling
- IAM Permission Denied Errors
- JVM Garbage Collection Pauses
- Kubernetes Service Not Reachable
- Lambda Cold Start Latency
- Log Volume Explosion and Cost Runaway
- Metrics Gaps in Dashboards
- Missing Traces in Distributed Tracing
- Multi-Region High Latency
- NAT Gateway Bandwidth and Throttling
- Network Bandwidth Saturation
- OOMKilled Pod Diagnosis
Start free on Cariara · Coding problems · System design studio