DynamoDB Hot Partition That Throttled a Flash Sale

Overview

A flash sale for one limited-edition sneaker sent every read to a single DynamoDB partition at more than twice its limit, and product pages failed for all users.

What happened

The catalog table used product_id as the partition key. The hero product's item was read by three different sources at once, and all of them landed on the same partition.

Why it broke

At peak, each page request made 3 DynamoDB reads. With no retry logic on the read path, a throttled read became a failed page.

Impact

• Peak sale page traffic — 2,200 requests per second
• DynamoDB reads per request — 3
• Reads on the hot partition — 6,600 per second
• Per-partition read limit — 3,000 RCU per second

The fix

• Hero product reads — All hit one partition — Absorbed before DynamoDB, or spread across keys
• A throttled read — Fails the whole page — Retried on the read path

> [!IMPORTANT] The 3,000 RCU limit is per partition, and every read of one partition key lands on the same partition.

Key concepts

Hot partition

A DynamoDB partition that receives a disproportionate share of traffic, hitting the 3,000 RCU/1,000 WCU per partition per second limit and causing throttling.

Partition key design

Choosing partition keys with high cardinality and even access distribution to spread load across all partitions.

Write sharding

Appending a random suffix (1-N) to a partition key to distribute writes across N partitions; reads must query all N shards and merge results.

DAX (DynamoDB Accelerator)

An in-memory cache for DynamoDB that handles microsecond reads and absorbs read spikes without hitting DynamoDB directly.

When to use it

• Designing DynamoDB schemas and partition key selection
• Explaining hot partition problems and mitigation strategies
• Discussing caching strategies in front of DynamoDB for high-read items

Interview questions

• How do you redesign a DynamoDB product catalog to handle flash-sale access patterns without hot partitions?

More War Stories topics

• AWS Credentials Committed to a Public GitHub Repository
• Backups That Were Never Actually Completing
• BGP Route Leak That Rerouted Traffic Through an Unexpected AS
• Blue-Green Deployment That Resulted in Split-Brain Sessions
• Cache Stampede on a Flash-Sale Product Page
• Connection Pool Exhaustion That Brought Down 12 Microservices
• Failover to a Lagging Replica Lost 4 Minutes of Orders
• Internal DNS Cache Poisoning That Redirected API Traffic
• Malicious npm Package Exfiltrated Environment Variables at Build Time
• N+1 Query Pattern That Collapsed Under Load
• NAT Gateway Port Exhaustion Under Bursty Lambda Traffic
• Node.js Memory Leak That Required Daily Restarts