Fencing
How to stop a leader or lock holder that has been replaced, but does not know it yet, from writing: fencing tokens checked by the storage, epochs and terms, and physical fencing when the storage cannot help.
Overview
A zombie is a process that used to be the leader, or used to hold a lock, and still acts as if it does. It happens because a process cannot notice its own pauses. A garbage-collection pause, a virtual machine being suspended, or a slow disk can stop a process for seconds. Meanwhile its lease runs out, the system gives the lock to someone else, and then the old holder wakes up and carries on writing.
Fencing is how a system makes those late writes harmless. The usual tool is a fencing token: a number that goes up every time the lock or leadership is granted. The holder sends the token with each write, and the storage refuses any write whose token is older than the newest one it has already accepted. The zombie does not need to know it is a zombie; the storage knows for it.
The same idea appears under other names across distributed systems: terms in Raft, epochs in Kafka, generation numbers in Google's Chubby lock service. When the storage cannot check a number, clusters fall back on blunter tools, such as cutting the old node's power before promoting a new one.
What you will be able to do
- Explain why a distributed lock with a timeout is not enough on its own to protect data.
- Design a fencing check at the storage, including the SQL for it.
- Name where real systems get their tokens: ZooKeeper, etcd, Chubby, Raft and Kafka.
- Choose between fencing tokens, conditional writes, lock delays and power fencing.
- Judge whether a lock needs fencing at all, using the efficiency-versus-correctness test.
Key concepts
- Zombie leader
- Process pauses
- Fencing token
- Check at the resource
- Epochs and terms
- Efficiency vs correctness locks
- Conditional writes
- STONITH
Interview questions
- What is the zombie leader problem, and how do fencing tokens solve it?
- Why is a distributed lock with a TTL not safe on its own?
- Walk through the Redlock debate. What should a practitioner take from it?
- How does Raft use terms as fencing tokens?
- Where would you get fencing tokens in a real system?
- How do you fence writes to a store that does not understand tokens?
- What is STONITH, and when would you use it instead of fencing tokens?
Sign up free to read the full topic
More System Design topics
Start free on Cariara · Coding problems · System design studio