Cache Stampede on a Flash-Sale Product Page

Overview

Every product page key expired at the same second a flash sale began, and 200 servers queried PostgreSQL for the same pages at once.

What happened

Product page TTLs were exactly 3600 seconds, and every product had been cached at the last flush, exactly one hour before the flash sale started at noon, so every key expired across the fleet at the peak.

How it spread

With 200 application servers all finding the cache empty, each fired its own query for each product page. Timeouts were treated as errors and retried, which doubled the query count again.

Impact

• Concurrent queries in the first 500ms — About 4,000
• Query queue depth — 600
• Cache-miss query response time — 12 seconds
• Time for popular pages to recover — 4 minutes
• Time for less popular pages — 18 minutes
• Elevated error rates — 11 minutes, at the peak of the sale

The fix

Spread expiry out, and make sure only one request per key ever refreshes it.

Key concepts

Cache stampede

Multiple concurrent cache misses for the same key that each independently query the database, multiplying backend load.

TTL jitter

Adding a random offset (e.g., plus or minus 10%) to cache TTLs so keys expire at different times and misses are spread out.

Cache lock

A pattern where the first cache miss acquires a distributed lock, queries the database, and populates the cache; subsequent misses wait or serve stale data rather than hitting the database independently.

Stale-while-revalidate

Serving the expired (stale) cached value to users while asynchronously refreshing the cache in the background, eliminating synchronous miss latency entirely.

When to use it

• Discussing cache expiry strategies and TTL design
• Explaining the value of cache locking and probabilistic expiry
• Interviewer asks about high-traffic event preparation

Interview questions

• How do you prevent a cache stampede for a high-traffic product page?

More War Stories topics

• Accidental DELETE Without a WHERE Clause in Production
• An IAM Misconfiguration That Let a Developer Become Admin
• AWS Credentials Committed to a Public GitHub Repository
• Backups That Were Never Actually Completing
• BGP Route Leak That Rerouted Traffic Through an Unexpected AS
• Blue-Green Deployment That Resulted in Split-Brain Sessions
• Connection Pool Exhaustion That Brought Down 12 Microservices
• DynamoDB Hot Partition That Throttled a Flash Sale
• Failover to a Lagging Replica Lost 4 Minutes of Orders
• Internal DNS Cache Poisoning That Redirected API Traffic
• Malicious npm Package Exfiltrated Environment Variables at Build Time
• N+1 Query Pattern That Collapsed Under Load