ZooKeeper, etcd, and Distributed Coordination

How ZooKeeper and etcd package consensus as a small, strongly consistent store, and how systems build leader election, locks, membership and configuration on top of it.

Overview

Why a coordination service exists

Many distributed systems need a few decisions that every server must see the same way: who the leader is, which server owns which partition, which servers are alive, what the current configuration is. Getting those right needs consensus, and consensus is hard to build. A coordination service does it once, well, and hands everyone else a small store with strong guarantees.

ZooKeeper and etcd are the two best-known examples. Each runs as a small cluster, usually three or five servers, that replicates every write through a consensus protocol: Zab in ZooKeeper, Raft in etcd. Applications talk to it as clients.

What they guarantee

Both are built for small amounts of data that is read often and changed rarely. ZooKeeper holds its whole tree in memory, and the etcd docs describe its sweet spot as a few gigabytes at most. The default limits make that clear: ZooKeeper checks that each node holds less than 1 MB, and etcd caps a request at 1.5 MiB by default and its total storage at 2 GiB by default, with 8 GiB suggested as the maximum.

Consensus protocol — Zab (atomic broadcast) — Raft
Data model — A tree of paths called znodes — A flat, sorted key space with prefix ranges
Writes — Linearizable: go through the leader, in one total order — Linearizable: go through the Raft leader
Reads by default — From the server the client is connected to; may be stale — Linearizable (confirmed with the cluster)
Fresher or cheaper reads — Call sync() before reading — Ask for a serializable read: faster, may be stale
Liveness of a client — Session; ephemeral znodes die with it — Lease with a TTL; keys attached to it die with it
Change notification — Watches (one-time; persistent watches since 3.6.0) — Watch streams from a chosen revision, in order
Ordering number — zxid (epoch plus counter) — Revision, a 64-bit counter for the whole cluster

Linearizable means every operation appears to take effect at one instant between its start and its end, so once a write is acknowledged, every later read sees it. ZooKeeper's default read is weaker: each client sees updates in order, but two clients may briefly see different versions. etcd makes the opposite default choice and lets you opt out.

The building blocks

Every coordination recipe uses the same few tools.

Ephemeral nodes and leases tie data to a client being alive. A ZooKeeper ephemeral znode is deleted when the session that created it ends. An etcd key attached to a lease is deleted when the lease expires, which happens if the client stops sending keepalives.
Sequential nodes and revisions give a total order. ZooKeeper can append a 10-digit increasing counter to a znode's name. Every etcd key records the revision at which it was created.
Atomic compare-and-set turns a race into a clear winner. ZooKeeper creates a node only if it does not exist; etcd transactions run if/then/else on key versions and revisions.
Watches push changes to clients, so nobody has to poll.

Recipe: leader election and locks

The two recipes are almost the same. Each contender creates an ephemeral, sequential node under one parent. The one with the lowest number is the leader, or holds the lock. Everyone else waits.

The detail interviewers like: each waiter watches only the node just ahead of it, not the lowest node. If every waiter watched the holder, one release would wake them all at once, the herd effect. The ZooKeeper recipes describe this exact fix. When a holder crashes, its session expires, its ephemeral node disappears, and only the next client in line wakes up.

etcd ships the same idea as ready-made Lock and Election APIs. Both take a lease; if the lease expires, the lock is released or leadership passes to the next campaigner.

Why a lock is not enough: fencing tokens

A lock from a coordination service guarantees that at most one client holds it according to the service. It cannot stop a client that has paused, for example during a long garbage-collection pause, from waking up and acting after its lease has expired and someone else has taken the lock.

The fix is a fencing token: a number that grows every time the lock is granted, sent with every write, and checked by the resource being protected. In ZooKeeper, the zxid or the znode version can serve as the token. The etcd docs describe the fencing token in etcd as the revision, the cluster-wide counter that grows with every change. A lease ID is only an identifier; nothing promises that it grows.

The check has to live in the storage system. That is the catch: the protected resource must understand tokens, which many do not.

Other uses

Service discovery and membership — Each server registers an ephemeral key; clients list and watch the parent
Configuration — One key holds the config; every server watches it and reloads on change
Partition assignment — A leader writes which server owns which partition; others watch
Cluster metadata store — Kubernetes keeps all its API server data in etcd

Kafka is a useful example of the trend. It used ZooKeeper for its metadata for years, then built KRaft, its own Raft-based quorum of controllers. Kafka 4.0 supports only KRaft mode; ZooKeeper mode has been removed.

What you will be able to do

Explain what a coordination service is for, and what data does not belong in it.
Compare ZooKeeper and etcd on data model, read consistency and liveness.
Build leader election and a lock from ephemeral sequential nodes, avoiding the herd effect.
Explain why a lock needs a fencing token, and which number to use as the token.

Key concepts

Coordination service: A small replicated store, built on consensus, that holds the few facts every server must agree on: leaders, locks, membership, configuration.
Znode: A node in ZooKeeper's tree of paths. It holds a small amount of data and may have children, unless it is ephemeral.
Ephemeral znode: A znode deleted automatically when the client session that created it ends. It turns a crash into a visible event.
Sequential znode: A znode whose name gets an increasing 10-digit counter appended, which gives contenders a total order.
Lease (etcd): A grant with a time to live. Keys attached to it are deleted when it expires, which happens if the client stops sending keepalives.
Watch: A subscription to changes. ZooKeeper watches fire once unless they are persistent; etcd watches stream every change from a chosen revision.
Fencing token: A number that increases with each lock grant. The protected resource rejects writes that carry an older token.
Herd effect: Many waiting clients woken by one change and all rushing the service at once. Watching only the next node avoids it.

When to use it

Designing leader election for a service where two leaders at once would corrupt data.
Guarding a shared resource with a lock that must survive client crashes, together with fencing tokens.
Tracking which servers are alive and which partitions each one owns.
Explaining how Kubernetes, or Kafka before KRaft, keeps its cluster metadata consistent.

Interview questions

How do you implement a distributed lock safely with etcd?
How does leader election work in ZooKeeper, and what happens when the leader dies?
ZooKeeper versus etcd: how do they differ, and which would you choose?
Why not just put a lock row in the main database instead?