Distributed Messaging

How a distributed log such as Kafka spreads messages across machines, keeps them safe through failures and delivers them in order: partitions, replication, consumer groups, delivery guarantees and compaction.

Overview

A distributed log is a messaging system that stores messages as an append-only sequence, spread over many machines and copied for safety. Apache Kafka is the best-known one; Apache Pulsar, Amazon Kinesis and Redpanda follow the same idea. Producers append records to the end. Consumers read from any position, and reading deletes nothing, so many services can read the same stream at their own pace, and any of them can go back and read it again.

That changes what messaging can do. A classic queue hands each message to one worker and forgets it once it is acknowledged. A log keeps records for a set time (or forever, for a compacted topic), which makes it a good home for event streams, change data capture, audit trails and stream processing, not just background jobs.

The design questions are about spreading and protecting that log. How records are split into partitions, and what order survives the split. How copies stay in sync, and when a write counts as safe. How a group of consumers shares the work, and what happens when one of them crashes halfway. Interviewers ask these in any design with an event pipeline, a news feed, notifications or analytics.

What you will be able to do

  • Explain partitions, keys and offsets, and choose a partition key and a partition count.
  • Configure a topic so that an acknowledged write survives a broker failure, and say what it costs.
  • Explain how consumer groups share partitions and what a rebalance does.
  • Say precisely what at-least-once and exactly-once mean in Kafka, and where exactly-once stops.
  • Choose between Kafka, RabbitMQ and Pulsar for a given workload.

Key concepts

  • Distributed log
  • Partitions and keys
  • Offsets
  • Leaders, followers and the ISR
  • acks and min.insync.replicas
  • Consumer groups
  • Exactly-once
  • Log compaction
  • KRaft
  • Kafka, RabbitMQ and Pulsar

Interview questions

  • How does Kafka keep an acknowledged write from being lost?
  • Kafka guarantees ordering. What exactly is ordered?
  • How do consumer groups work, and what happens when a consumer dies?
  • Explain at-most-once, at-least-once and exactly-once in Kafka.
  • Why is Kafka fast even though it writes everything to disk?
  • What is log compaction, and when would you use it?
  • How do you choose the number of partitions and the partition key?
  • When would you choose Kafka, RabbitMQ or Pulsar?

Sign up free to read the full topic

More System Design topics

Start free on Cariara · Coding problems · System design studio