Node.js Memory Leak That Required Daily Restarts
A closures-based event listener accumulation caused heap growth of 40 MB/hour, requiring nightly automated restarts to prevent OOM crashes.
Overview
A Node.js API crashed every night around 3 AM for months because every request added a listener to a shared Redis client and never removed it.
What happened
The nightly restart took only 30 seconds and hit low traffic, so engineers accepted it as normal. Scaling out made the pattern impossible to ignore.
Why it broke
Each request registered a close listener on the shared Redis client to handle request teardown. The listener was never removed, and its closure kept the whole request object alive.
- Restart time — 30 seconds
- Instances before and after scaling — 2, then 8
- Snapshot interval that exposed the leak — 4 hours
- Listeners after 10,000 requests — 10,000
The fix
One line of cleanup, plus a guard that makes the next leak loud instead of silent.
- Listener lifetime — Forever — Removed when the response ends
- Leak visibility — A nightly restart nobody questioned — Warning when listeners pass the threshold
> [!NOTE] A crash that a health check "fixes" every night is still a crash. Scaling out turned a quiet 3 AM restart into daytime outages.
Key concepts
- Heap snapshot
- A point-in-time dump of the JavaScript heap showing all allocated objects, their sizes, and reference chains — used to identify what is growing over time.
- Event listener accumulation
- A common Node.js leak pattern where listeners are added to emitters without corresponding removeListener calls, causing references to accumulate.
- Closure reference leak
- A function (closure) that captures outer-scope variables keeps those variables alive in memory even after the outer scope should have been garbage collected.
- MaxListeners warning
- Node.js emits a warning when more than 10 listeners are added to a single event on an emitter, indicating a potential leak.
When to use it
- Discussing Node.js memory management and closure gotchas
- Explaining how to diagnose memory leaks with heap snapshots
- Justifying operational observability requirements (heap metrics, process memory)
Interview questions
- A Node.js service is growing in memory usage by 50 MB per hour. How do you diagnose and fix it?
Sign up free to read the full topic
More War Stories topics
- DynamoDB Hot Partition That Throttled a Flash Sale
- Failover to a Lagging Replica Lost 4 Minutes of Orders
- Internal DNS Cache Poisoning That Redirected API Traffic
- Malicious npm Package Exfiltrated Environment Variables at Build Time
- N+1 Query Pattern That Collapsed Under Load
- NAT Gateway Port Exhaustion Under Bursty Lambda Traffic
- Public S3 Bucket Exposed Customer PII for 8 Months
- Silent Data Corruption from an Off-By-One Migration
- Single-Region Dependency That Caused a Global Outage
- The Cascade That Took Down Three Services
- Third-Party API Outage That Was Mistaken for Own Code
- Thundering Herd After a Cache Cluster Restart
Start free on Cariara · Coding problems · System design studio