Thundering Herd After a Cache Cluster Restart
Restarting an ElastiCache cluster during low-traffic hours triggered a database overload at 3 AM when all cache misses hit simultaneously.
Overview
A routine Redis node-type change in the 3 AM window left the whole fleet with an empty cache, and the database went down under the misses for 22 minutes.
What happened
The plan was to rotate the ElastiCache Redis cluster to a new node type during the low-traffic window. The maintenance itself finished in 11 minutes. The trouble started when the application reconnected.
Why it broke
Every node in the application fleet saw a cold cache at the same moment. The read replicas, which normally handled 80% of traffic, were not warm either.
> [!WARNING] A low-traffic window did not help. With a 94% hit ratio, losing the cache multiplies database load no matter how quiet the hour is.
Impact
- Cache hit ratio — 94% — 0%
- RDS primary query rate — Normal — 47x within 90 s
- API p99 latency — 45ms — 8 s
- Downtime — Planned as zero — 22 minutes
The fix
Never send live traffic to a cold cache, and bound the misses when it happens anyway.
Key concepts
- Thundering herd
- Multiple processes or clients simultaneously contacting the same resource (typically a database) after a shared cache becomes unavailable, overwhelming the backend.
- Cache warming
- Pre-populating a cache before directing live traffic to it, either by replaying recent read traffic or by explicitly seeding frequently accessed keys.
- Request coalescing
- Deduplicating concurrent cache-miss requests for the same key so only one request reaches the database; all waiters receive the same result.
- Probabilistic early expiry
- Cache keys are refreshed slightly before their TTL expires using a probability function, avoiding simultaneous expiry of many popular keys.
When to use it
- Explaining cache warming strategies in system design interviews
- Justifying probabilistic cache expiry and request coalescing patterns
- Discussing how low-traffic windows can paradoxically expose certain failure modes
Interview questions
- How do you safely replace an ElastiCache cluster without a thundering herd?
Sign up free to read the full topic
More War Stories topics
- Node.js Memory Leak That Required Daily Restarts
- Public S3 Bucket Exposed Customer PII for 8 Months
- Silent Data Corruption from an Off-By-One Migration
- Single-Region Dependency That Caused a Global Outage
- The Cascade That Took Down Three Services
- Third-Party API Outage That Was Mistaken for Own Code
- TLS Certificate Expiry That Took Down an Internal API
Start free on Cariara · Coding problems · System design studio