ZooKeeper Deep Dive

Apache ZooKeeper, the coordination service: znodes, the ensemble and its ZAB protocol, what its consistency guarantees do and do not promise, sessions, watches, and the leader election and lock recipes, with the fencing tokens they need.

Overview

Apache ZooKeeper is a small, replicated service that distributed programs use to coordinate: to agree on a leader, hold a lock, store shared configuration, or see which members of a group are alive. It was built at Yahoo! and described in a 2010 paper; Hadoop, HBase, Kafka and many other systems relied on it.

ZooKeeper gives you a tree of small data nodes, called znodes, that looks like a file system. The whole tree lives in memory on every server, and every change is logged to disk and agreed on by a majority of servers before it is applied. Three features make it useful for coordination: nodes that vanish when their owner's session ends, nodes that get a sequence number, and one-time notifications when a node changes.

ZooKeeper does not hand you a lock or a leader election directly. It gives you these building blocks, and the recipes built from them are client code. That is why interviewers like it: to use it well you have to reason about sessions, pauses, stale reads and races.

What you will be able to do

  • Explain znodes, sessions and watches, and what each type of znode is for.
  • Walk through how a write is agreed by the ensemble, and why reads can be stale.
  • Implement leader election and a lock with sequential ephemeral nodes, without a herd effect.
  • Explain why a lock from any lock service needs a fencing token.
  • Size an ensemble, and compare ZooKeeper with etcd and Consul.

Key concepts

  • Znodes
  • Ephemeral and sequential nodes
  • Ensemble and quorum
  • ZAB and the zxid
  • Consistency guarantees and sync
  • Sessions
  • Watches
  • Leader election and locks
  • Fencing tokens
  • Observers

Interview questions

  • How do you implement leader election with ZooKeeper?
  • How does a write work in ZooKeeper, and why can a read be stale?
  • What happens to ephemeral nodes when a client pauses or loses its network?
  • Why does a distributed lock need a fencing token?
  • What are the limits of ZooKeeper watches?
  • How many servers should a ZooKeeper ensemble have?
  • Use ZooKeeper for service discovery. What are the pitfalls?
  • Compare ZooKeeper with etcd.

Sign up free to read the full topic

More System Design topics

Start free on Cariara · Coding problems · System design studio