Read Repair

Overview

In a leaderless database such as Cassandra, Riak or Amazon's original Dynamo, every key is stored on several replicas, and a write succeeds once enough of them acknowledge it. The replicas that missed the write, because they were down, overloaded or cut off, keep an older value. Over time the copies drift apart.

Read repair fixes that drift as a side effect of reading. The node coordinating a read asks several replicas for the key, compares what they return, answers the client with the newest version, and writes that version back to any replica that returned something older. The Dynamo paper describes it as repairing replicas that missed a recent update at an opportunistic time, which takes work off the background repair process.

Read repair is cheap, because it only works on keys someone is already reading, and that is also its limit: a key nobody reads is never repaired this way. It is one of three repair paths, alongside hinted handoff and background anti-entropy repair, and interviewers expect you to know what each one covers.

What you will be able to do

• Walk through a quorum read that detects and fixes a stale replica, including digest reads.
• Explain how the newest version is chosen, and why deletes need tombstones to win.
• Explain why Cassandra blocks on read repair, using the monotonic-read example.
• Place read repair next to hinted handoff and Merkle-tree anti-entropy, and say what each misses.

Key concepts

• Replica divergence
• Coordinator
• Digest reads
• Last write wins
• Blocking read repair
• Monotonic quorum reads
• Hinted handoff
• Anti-entropy with Merkle trees
• Tombstones and gc_grace_seconds

Interview questions

• Walk through how read repair works in a Cassandra quorum read.
• Why does Cassandra make read repair blocking?
• How do read repair, hinted handoff and anti-entropy repair fit together?
• Why did Cassandra 4.0 remove read_repair_chance?
• How can read repair bring deleted data back, and how do you prevent it?
• When does a replica pick the wrong winner during read repair?
• Does a Raft-based or leader-based database need read repair?

More System Design topics

• Proximity Service
• Quorum Consensus
• Raft Consensus
• RAG System (Retrieval-Augmented Generation)
• Rate Limiter
• Rate Limiting
• Read-Heavy vs Write-Heavy Systems
• Real-Time AI Translation System
• Real-Time Bidding Platform
• Real-Time Collaboration System
• Real-Time Event Stream Processing
• Real-Time Feature Database