etcd and the API Server — The Write Path, Raft, and Why 3 vs 5 Members Matters

How a Kubernetes write travels from kubectl through the API server into etcd and back out as watch events, how Raft quorum decides availability and write latency, and how to run, size, repair and restore etcd without losing the cluster.

Overview

Every Kubernetes object lives in etcd. The API server is the only component that should talk to it, and every create, update and delete becomes an etcd transaction that a majority of etcd members must write to disk before the API server answers. When etcd is slow, every write in the cluster is slow. When etcd loses quorum, the cluster cannot change state: the Kubernetes docs say that pods already scheduled may keep running, but no new pods can be scheduled.

Most etcd outages are not exotic. The etcd FAQ names the usual causes: a slow or shared disk that delays the write-ahead log fsync, CPU starvation, a congested network between members, a keyspace that was never compacted, and membership changes done in the wrong order. Restores done while API servers are still running, or without handling the revision going backwards, break the controllers that cache cluster state.

Symptom to first command

Every write is slow, reads are fine — Disk under the WAL — etcdctl endpoint status --cluster -w table and the etcd_disk_wal_fsync_duration_seconds p99 — Slow disk and leader elections
Log line failed to send out heartbeat on time — Leader disk, CPU or network — etcdctl endpoint status --cluster -w table (RAFT TERM rising) — Slow disk and leader elections
Log line apply request took too long — Disk or expensive requests — etcd_disk_backend_commit_duration_seconds p99 — Slow disk and leader elections
Writes fail with mvcc: database space exceeded — Space quota, NOSPACE alarm — etcdctl alarm list — Database space exceeded
An update returns 409 Conflict — Optimistic concurrency, not etcd health — kubectl get -o yaml and compare metadata.resourceVersion — The write path
Controllers relist, watches get 410 Gone — History compacted — API server logs, --etcd-compaction-interval — Watches, revisions and the watch cache
Member down, cluster still serving — Lost fault tolerance — etcdctl member list — Replacing members, backup and restore
No leader, API server /readyz fails on etcd — Quorum lost — kubectl get --raw='/readyz?verbose' — Replacing members, backup and restore

What you will be able to do

Walk a write from kubectl to the etcd WAL and back out as a watch event, naming each check that can reject it.
Explain quorum math, why even member counts add no fault tolerance, and why five members tolerate more failures but commit each write more slowly than three.
Read the etcd warnings (failed to send out heartbeat on time, apply request took too long, mvcc: database space exceeded) and fix them with the documented commands.
Replace a failed member in the safe order, add new members as learners, and back up and restore etcd for a Kubernetes cluster.
Name the mistakes that turn a small etcd problem into a lost cluster, and the guardrails that prevent them.

Key concepts

Quorum: A majority of members, (n/2)+1 rounded down. Every key-value update and membership change needs it; without it etcd accepts no writes.
Failure tolerance: How many members can fail while quorum survives: 1 for 3 members, 2 for 5, 3 for 7. Even sizes add none.
Raft commit: An entry is committed once the leader has replicated it on a majority of servers; each member persists it to its WAL with fsync first.
Revision: A keyspace-wide counter that increases by one for each atomic change. kube-apiserver uses the write revision as the object's resourceVersion.
resourceVersion: An opaque string clients pass back unmodified. A stale value on update returns 409 Conflict; an expired one on watch returns 410 Gone.
Compaction: Dropping history older than a revision. Frees space inside the database; old revisions then fail with required revision has been compacted.
Defragmentation: Rebuilding a member's database file to return freed space to the filesystem. Per member, not replicated, and it blocks that member while running.
NOSPACE alarm: Raised when a member's database exceeds --quota-backend-bytes (default 2 GiB). The cluster then accepts only reads and deletes until compact, defrag and alarm disarm.
Learner: A non-voting member (etcd v3.4+) that receives the log but does not count toward quorum until promoted with member promote.
Watch cache: kube-apiserver's in-memory copy of state fed by etcd watches; it serves Any and Not older than reads and, since Kubernetes v1.31 with etcd 3.4.31+ or 3.5.13+, consistent (most recent) reads.

When to use it

When you run self-managed Kubernetes control planes (kubeadm, bare metal, VMs) and own the etcd cluster, its disks and its backups.
When API writes are slow or time out and you need to tell an etcd disk or quorum problem from an API server or admission problem.
When deciding between 3 and 5 etcd members, or whether to spread members across zones or regions.
When writing the runbooks for member replacement, snapshot backup, restore, compaction and defragmentation.
When preparing for platform, SRE or Kubernetes administrator interviews, where the write path and quorum math are standard questions.

Interview questions

Explain what happens in etcd when the kubernetes API server writes an object, and why etcd cluster size (3 vs 5 nodes) matters for both availability and write latency.
A user gets 409 Conflict with "the object has been modified". Is etcd broken?
Why do etcd clusters use odd member counts, and why not run 7 or 9 members for safety?
etcd keeps changing leaders and API writes are slow. How do you troubleshoot it?
Writes fail with "mvcc: database space exceeded". Walk through the recovery.
How do you replace a failed etcd member in a 3-member cluster, and why does the order matter?
How do you back up and restore etcd for a Kubernetes cluster, and what breaks if you do it wrong?
Should you spread etcd members across availability zones or regions?
What is an etcd learner and when would you use one?
How does the API server avoid sending every read and watch to etcd, and what does 410 Gone mean?