Distributed Message Queue

A Kafka-style log broker

Design a Kafka-style broker: partitioned, replicated logs on disk, producers and consumer groups, and the guarantees about loss, duplicates and order that survive failures.

Overview

A message queue lets one service hand data to another without both being up at the same time. In this problem you design the queue itself, the broker cluster, in the style of Apache Kafka: producers append messages to named topics, the cluster stores them durably for a set time, and any number of consumer groups read them at their own pace and can read them again.

The interviewer is not asking how to use Kafka. They want to see you build one: how a topic is split into partitions so it can grow past one machine, how each partition is stored as append-only files on disk, how copies on other brokers keep a write safe, how a group of consumers shares the work, and what the system promises about loss, duplicates and order when brokers and consumers crash. The Distributed Messaging topic explains these ideas as concepts (and lists the Kafka settings worth knowing); this page designs the machinery and walks the interview.

Questions to ask first

Queue (delete on read) or log (keep and replay)? — A log: messages stay for the retention period, and several groups read the same data
Volume and size? — 500,000 messages a second on average, 1,000,000 at peak, 1 KB each
How long do we keep data? — 7 days, then whole files are deleted; some topics are compacted instead
Ordering? — Per key: messages with the same key are read in the order they were written
Delivery guarantee? — At-least-once by default; exactly-once for pipelines that read and write the queue
Durability? — An acknowledged write survives the loss of any one broker
Latency? — Tens of milliseconds end to end is fine; throughput matters more
Delayed messages, dead-letter queues? — Yes, built on top of plain topics

What you will be able to do

Lay out how a partition is stored on disk and how a read finds any offset quickly.
Walk a message from the producer's batch to a committed, replicated record, and say when the producer is told it is safe.
Explain consumer groups, partition assignment, rebalancing and where committed offsets live.
State exactly which delivery guarantee each configuration gives, and how idempotent producers and transactions remove duplicates.
Add delayed delivery, retries and dead-letter topics without breaking the log model.

Key concepts

Append-only segments
Offsets
Partitioner and batching
acks and the ISR
High watermark
Leader epoch
Consumer groups and rebalancing
Committed offsets
Idempotent producer and transactions
KRaft controller

Functional requirements

Producers publish messages, with an optional key, to a named topic
Consumers read a topic from any offset, alone or as a group that shares the partitions
Messages with the same key are read in the order they were written
Messages are kept for a retention period, or compacted to the latest value per key
Several consumer groups can read the same messages independently, and re-read them
Delivery guarantee is configurable: at-most-once, at-least-once or exactly-once
Delayed delivery, retries and dead-letter topics for failed messages

Non-functional requirements

Throughput: 1,000,000 messages a second at peak, and more by adding brokers
Durability: no acknowledged write is lost when one broker fails
Availability: a partition keeps serving after a broker failure, within seconds (a target we chose)
Latency: under 100 ms from produce to consume at p99 when consumers keep up (a target we chose)
Scales out: topics, partitions, brokers and consumers can all be added while running

Interview questions

Walk me through what happens when a producer sends a message with acks=all.
How is a partition stored, and how do you read from an arbitrary offset fast?
Why do consumers pull instead of the broker pushing?
What is the high watermark and why does it exist?
How do consumer groups divide work, and what happens in a rebalance?
How do you get exactly-once delivery?
What does the controller do, and why did Kafka move from ZooKeeper to KRaft?
How would you support delayed messages and a dead-letter queue?
How does this system guarantee ordering, and how can it break?