Split Brain

What happens when a cluster splits and two nodes both believe they are in charge, and the defenses that stop it: majority quorums, witnesses, fencing, leases and majority-acknowledged writes.

Overview

Split brain is what happens when a cluster that should have one leader ends up with two or more, each accepting writes on its own. It usually starts with a network partition: the nodes are fine, but they can no longer reach each other. Each side sees silence from the other, decides the other has failed, and carries on as if it were in charge.

It is dangerous because nothing looks wrong while it happens. Both leaders answer requests successfully. The damage appears later, when the network heals and two copies of the data turn out to disagree: the same order ID used twice, the same seat sold twice, one account balance with two different histories. Some of those writes must now be merged by hand or thrown away.

Every design with a single writer and automatic failover has to answer one question: when the old leader cannot be reached, how do we make sure it has stopped before a new one starts? This topic is about the answers.

What you will be able to do

  • Explain the sequence that leads to split brain, and why failure detection alone cannot prevent it.
  • Use majority quorums and witnesses so that only one side of a partition can elect a leader.
  • Use fencing (STONITH or fencing tokens) and leases so that a deposed leader cannot do damage.
  • Configure a primary to refuse writes it cannot replicate, and describe what real systems do.
  • Explain how to detect and recover from split brain, and why AP systems accept divergence by design.

Key concepts

  • Network partition
  • Two leaders
  • Majority quorum
  • Odd cluster sizes
  • Witness or quorum device
  • Fencing and STONITH
  • Fencing tokens
  • Leases and clock drift
  • Refusing unreplicated writes
  • Reconciliation

Interview questions

  • Walk through a split-brain scenario, step by step.
  • How do quorum-based systems prevent split brain?
  • What is fencing, and what is the difference between STONITH and fencing tokens?
  • How do you handle a two-node cluster?
  • How do leases prevent split brain, and where do they fail?
  • Split brain happened. How do you recover?
  • Is split brain always a bug? How does it relate to CAP?

Sign up free to read the full topic

More System Design topics

Start free on Cariara · Coding problems · System design studio