Hinted Handoff

How leaderless databases keep accepting writes while a replica is briefly down: another node keeps the write with a hint naming its real owner and hands it over when the owner returns.

Overview

In a leaderless, Dynamo-style database, each key is stored on several replicas, and a write succeeds once enough of them acknowledge it. So what happens to the copy meant for a replica that is down for a few minutes, for a restart, a deploy or a network blip?

Hinted handoff is the answer most of these systems use. Another node accepts the write on the missing replica's behalf and stores it together with a hint: a note saying which node the write really belongs to. When the missing replica is reachable again, the node holding the hint hands the write over and deletes its copy. The idea was described in Amazon's 2007 Dynamo paper, and Apache Cassandra and Riak both implement it, each in its own way.

It matters because most outages are short. The Dynamo authors noted that node outages in their environment were usually transient, so they should not trigger rebalancing or a full repair. Hints bridge those short gaps cheaply, and leave the rare long gap to heavier repair tools.

What you will be able to do

  • Explain how a hint is created, stored and replayed.
  • Tell a sloppy quorum from a strict one, and show why a sloppy quorum can break R + W > N.
  • Describe how Cassandra, Riak and the Dynamo paper differ.
  • Name the cases hints do not cover, and what repairs them.
  • Operate hints: choose the window, throttle replay, and plan repairs after long outages.

Key concepts

  • Hint
  • Sloppy quorum
  • Strict quorum
  • Preference list
  • Hint window
  • Replay throttling
  • Consistency level ANY
  • Read repair and anti-entropy

Interview questions

  • What is hinted handoff and why do Dynamo-style databases use it?
  • Explain sloppy quorums. Why can a sloppy quorum break R + W > N?
  • How does Cassandra's hinted handoff differ from Dynamo's?
  • A replica was down for five hours. What do you do when it comes back?
  • How do hinted handoff, read repair and anti-entropy repair relate?
  • Why is consistency level ANY risky?
  • Does DynamoDB use hinted handoff and sloppy quorums?

Sign up free to read the full topic

More System Design topics

Start free on Cariara · Coding problems · System design studio