Segmented Log
Splitting an append-only log into a series of files, so old data can be deleted or compacted a file at a time and any record can be found in a few steps.
Overview
An append-only log is the simplest durable storage there is: new records go on the end, and nothing already written ever changes. Kafka topics, database write-ahead logs and consensus logs are all built on one. Kept as a single file, though, a log only grows. It fills the disk, it is slow to scan at startup, and deleting its oldest part means rewriting the whole file.
A segmented log splits the log into a series of files called segments. Only the newest one, the active segment, receives writes. When it reaches a size or age limit it is closed and a new one starts. Closed segments never change, so the system can delete an old one by removing a file, compact it by writing a smaller copy, or move it to cheaper storage, all without touching the write path.
Unmesh Joshi describes this as a named pattern in Patterns of Distributed Systems. Kafka's partition storage is the best-known example, and the same design appears in PostgreSQL's write-ahead log and etcd's.
What you will be able to do
- Describe how a partition is laid out on disk and when segments roll.
- Walk through finding any offset in a few steps with a sparse index.
- Explain delete and compact retention, and what each guarantees to readers.
- Choose a segment size and say what it costs either way.
Key concepts
- Append-only log
- Active and closed segments
- Rolling by size or time
- Base offset file names
- Sparse offset index
- Retention by deletion
- Log compaction
- Tombstones
- Tiered storage
Interview questions
- How does Kafka store a partition on disk?
- How does a broker find offset N quickly?
- What is the difference between deleting and compacting a log?
- What does Kafka guarantee about a compacted topic?
- Why is data sometimes kept longer than retention.ms?
- How does tiered storage change the design?
- Where else does the segmented log pattern appear?
Sign up free to read the full topic
More System Design topics
Start free on Cariara · Coding problems · System design studio