Failover to a Lagging Replica Lost 4 Minutes of Orders

Overview

The database failover worked in 62 seconds, but the application kept reading from a different replica that was 4 minutes behind, and 847 fulfilled orders were fulfilled again.

What happened

The orders database ran on RDS Multi-AZ with a synchronous standby in another availability zone, configured correctly. The analytics team had also added an asynchronous read replica for reporting. A heavy reporting query that locked table statistics had left it 4 minutes behind.

Why it broke

The failover itself was fine. The application's read connection strings pointed to the analytics replica by hostname alias, not to the Multi-AZ standby, and it sent all non-critical reads there. After failover, that stale replica kept answering.

Impact

• Failover time of the Multi-AZ standby — 62 seconds
• Replica lag — 4 minutes
• Stale status pages — Orders shown "pending" after they had shipped
• Orders fulfilled twice — 847

The fix

Route each read by how much staleness it can tolerate, and never let a job that triggers actions read from a replica.

Key concepts

Replication lag

The delay between a write on the primary and its visibility on a read replica. Async replicas can accumulate minutes of lag under heavy load.

Read-after-write consistency

Guarantee that a read immediately following a write will see the write. Async replicas do not provide this without additional routing logic.

Replica promotion

Elevating a read replica to primary after a failover. RDS Multi-AZ promotion is automatic and targets the synchronous standby; other replicas are not promoted.

RDS Multi-AZ vs read replica

Multi-AZ standby uses synchronous replication and is promoted automatically on failure. Read replicas use asynchronous replication and are not promoted automatically.

When to use it

• Discussing replication lag monitoring and read replica placement
• Designing failover strategies that account for stale read replicas
• Explaining why replica lag is a consistency risk, not just a performance metric

Interview questions

• How do you design a read replica strategy that avoids serving stale data to business-critical reads?

More War Stories topics

• Backups That Were Never Actually Completing
• BGP Route Leak That Rerouted Traffic Through an Unexpected AS
• Blue-Green Deployment That Resulted in Split-Brain Sessions
• Cache Stampede on a Flash-Sale Product Page
• Connection Pool Exhaustion That Brought Down 12 Microservices
• DynamoDB Hot Partition That Throttled a Flash Sale
• Internal DNS Cache Poisoning That Redirected API Traffic
• Malicious npm Package Exfiltrated Environment Variables at Build Time
• N+1 Query Pattern That Collapsed Under Load
• NAT Gateway Port Exhaustion Under Bursty Lambda Traffic
• Node.js Memory Leak That Required Daily Restarts
• Public S3 Bucket Exposed Customer PII for 8 Months