High-Water Mark

The position in a replicated log up to which entries are safely replicated: how Kafka's high watermark and Raft's commit index are computed, why readers must not pass them, and what happens to entries beyond them on failover.

Overview

In a leader-based replicated log, the leader appends new entries first and the followers copy them a moment later. At any instant the replicas hold logs of different lengths. The high-water mark is the position in the log below which every entry is known to be replicated widely enough to survive losing the leader. Entries below it are committed. Entries above it exist on the leader and perhaps some followers, but could still disappear in a failover.

The mark matters because it is the line readers may not cross. If a consumer or a state machine saw an entry above the mark and the leader then crashed, the new leader might not have that entry, and the reader would have seen data that no longer exists.

Kafka calls this position the high watermark; Raft calls it the commit index. Unmesh Joshi's catalog of distributed-systems patterns names the general idea High-Water Mark. The rule for computing it differs between systems, and that difference is a favorite interview topic.

What you will be able to do

  • Read a partition's state: each replica's log end offset, the high watermark, and what consumers can see.
  • Compute Kafka's high watermark and Raft's commit index, and explain why they differ.
  • Explain what acks=all and min.insync.replicas guarantee, and what they do not.
  • Walk through a failover and say which entries are kept and which are cut off.
  • Name the other marks in a log: the last stable offset and the low-water mark.

Key concepts

  • Log end offset (LEO)
  • High watermark
  • Commit index
  • In-sync replicas (ISR)
  • acks and min.insync.replicas
  • Truncation on failover
  • Leader epochs
  • Last stable offset
  • Low-water mark

Interview questions

  • What is the high-water mark in Kafka, and why can't consumers read past it?
  • How is Raft's commit index different from Kafka's high watermark?
  • What happens to messages above the high watermark when the leader fails?
  • What do acks=all and min.insync.replicas guarantee together?
  • Why did Kafka introduce leader epochs for log truncation?
  • What is the low-water mark, and how does it relate to the high-water mark?
  • Where else does this pattern show up beyond Kafka and Raft?

Sign up free to read the full topic

More System Design topics

Start free on Cariara · Coding problems · System design studio