Event Streams and Message Brokers

How events move from producers to consumers through a broker: queues that delete a message once handled, logs that keep every message, and what each means for ordering, scaling and delivery guarantees.

Overview

An event is a small, immutable record that something happened: a click, a payment, a sensor reading. An event stream is the never-ending sequence of such events. Producers write events; consumers read and act on them.

Producers and consumers could talk directly, but then a slow or crashed consumer loses events, and every producer must know every consumer. A message broker sits in between. Producers hand events to the broker, the broker stores them, and consumers read at their own pace. The broker absorbs bursts, survives consumer restarts, and lets you add consumers without touching producers.

Two kinds of broker: queues and logs

Brokers differ mainly in what happens after a consumer handles a message.

After a message is handled — Deleted once the consumer acknowledges it — Kept until the retention limit, whoever has read it
Where the read position lives — In the broker, per message (delivered, acknowledged) — In the consumer, as one offset per partition
Several consumers on one stream — They compete: each message goes to one of them — Each consumer group reads every message
Replay old messages — Not possible once acknowledged — Rewind the offset and read again
Ordering — Can change when messages are redelivered — Kept within each partition
Good for — Task queues where each job runs once — Event history, many independent readers, rebuilding state

RabbitMQ hands each message in a queue to the next consumer in turn and deletes it after a positive acknowledgment. If a consumer's connection closes before it acknowledges, the message goes back on the queue and is delivered again with a redelivered flag. Kafka instead appends every event to a log and keeps it for a set time; its docs note that, unlike classic messaging systems, events are not deleted after consumption.

The line has blurred. RabbitMQ now offers streams, an append-only log that consumers can read repeatedly. Kafka 4.2 made share groups production-ready: a queue-style mode where several consumers take records from the same partition and acknowledge them one by one. The two models are still the clearest way to reason about the choice.

How a log broker works: topics, partitions, offsets

A Kafka topic is split into partitions. Each partition is an append-only file, and each event in it gets an increasing number called the offset. A producer chooses the partition: if the event has a key, the partition comes from a hash of the key, so all events for one key land in the same partition. With no key, the producer sticks to one partition until it has sent about a batch of data, then moves to another.

Kafka guarantees that a consumer reads each partition in exactly the order the events were written. There is no order across partitions. So the key is a design decision: key by the entity whose events must stay in order (an order, an account, a device).

Consumer groups: scaling reads

Consumers that share a group ID split the partitions between them. Each partition is read by exactly one consumer in the group at a time, which keeps the per-partition order. The cost is a hard ceiling: a group can never have more busy consumers than the topic has partitions.

8 — 4 — Each consumer reads 2 partitions
8 — 8 — One partition each; the most parallel this group can be
8 — 10 — 8 busy, 2 idle standbys

Different groups do not affect each other. The billing service and the search indexer each have their own group, each read every event, and each keeps its own offsets. Adding a third reader is just a new group ID.

Offsets and delivery guarantees

A consumer's position in a partition is a single number, the offset of the next event to read. The consumer periodically commits that number back to Kafka so that it, or a replacement, can resume after a crash. When it commits, relative to when it finishes processing, sets the guarantee.

Before processing — The event is marked done but never processed — At-most-once: may lose events
After processing — The event is processed again by the next consumer — At-least-once: may duplicate events
In the same atomic write as the output — Output and offset both happen or neither does — Exactly-once effect

Kafka's default is at-least-once, so consumers should be idempotent: applying an event twice must have the same effect as once. Watch the defaults: enable.auto.commit is true and commits every 5 seconds in the background, which can commit offsets for events your code has not finished.

Retention, replay and compaction

Because the log keeps events, a consumer can rewind. That is the log broker's biggest advantage: fix a bug, reset the offset, and reprocess last week's events; or start a new service and let it build its state from history.

log.cleanup.policy — delete — Old log segments are removed by time or size
log.retention.hours — 168 (7 days) — How long events are kept under the delete policy
log.retention.bytes — -1 (no size limit) — Optional size cap per partition
log.cleanup.policy=compact — off — Keep at least the latest event for each key; older values for the key are removed
auto.offset.reset (consumer) — latest — Where a new group with no committed offset starts; set earliest to read from the start

A compacted topic works like a table's change history reduced to the latest row per key. A new consumer that reads it from the start sees at least the final value for every key, in order, which is enough to rebuild the current state. That idea, a stream that can be turned into a table and back, is the basis of change data capture and stream-table joins.

Why a log is fast

A log broker only ever appends to the end of a file and reads forward from an offset, so disk access is sequential. Kafka leans on the operating system's page cache instead of its own cache, and on Linux it uses the sendfile system call to send bytes from the page cache straight to the network socket without copying them through the application. When consumers keep up, reads are served from memory. Kafka's docs add that performance is effectively constant with respect to data size, so long retention is fine.

Exactly-once inside Kafka

Two features remove duplicates when data stays inside Kafka. The idempotent producer, on by default in current clients, tags each batch with a producer ID and sequence number, so the broker drops a batch that a retry sends twice. Transactions let a producer write to several partitions atomically, and a consumer offset can be written in the same transaction as the output. Readers must set isolation.level=read_committed (the default is read_uncommitted) to see only committed data. Once the output leaves Kafka, for example to a database or an email service, you need an idempotent write or to store the offset with the output.

What you will be able to do

Choose between a queue broker and a log broker for a given workload, and say why.
Pick a partition key that keeps the right events in order, and size partitions for the consumers you need.
Explain how commit timing gives at-most-once or at-least-once, and make a consumer idempotent.
Use retention, offset reset and compaction to replay history or rebuild state.

Key concepts

Message broker: A server that stores messages between producers and consumers, so they can run at different speeds and fail independently.
Log-based broker: A broker that appends messages to a partitioned log and keeps them for a retention period. Consumers track their own offsets and can re-read.
Partition: An ordered, append-only part of a topic. Order is guaranteed only within one partition. Events with the same key go to the same partition.
Offset: The position of an event in a partition. A consumer's progress is one committed offset per partition: the next event to read.
Consumer group: Consumers sharing a group ID. Each partition goes to one member at a time, so the partition count caps the group's parallelism. Different groups read independently.
At-least-once delivery: Commit the offset after processing. A crash in between causes the event to be processed again, so consumers must be idempotent.
Log compaction: A cleanup policy that keeps at least the latest event for each key, so the topic can rebuild current state without the full history.
Idempotent producer: A producer whose batches carry a producer ID and sequence number, so the broker discards duplicates caused by retries.

When to use it

Designing an event-driven system where several services react to the same events
Choosing between a task queue (each job once) and an event log (many readers, replay)
Picking partition keys and partition counts for ordering and consumer parallelism
Explaining at-most-once, at-least-once and exactly-once delivery in an interview

Interview questions

How does Kafka guarantee message ordering?
When would you choose RabbitMQ-style queues over a Kafka-style log?
A topic has 8 partitions and a consumer group has 10 consumers. What happens, and how would you get more throughput?
How do you make sure a consumer neither loses nor double-applies events?
Your team shipped a bug that wrote wrong values for three days. How do you use the log to fix it?