Write-Ahead Log (WAL)

The pattern behind durable state in databases, brokers and consensus systems: record each change in an append-only log before applying it, rebuild state by replaying the log, and keep the log small with segments and snapshots.

Overview

A write-ahead log (WAL) is an append-only file of changes. The rule is that a change is written to the log first, and only then applied to the real data structures, whether those are database pages, an in-memory table or a key-value map. If the process crashes at any point, the log holds every change that was acknowledged, so the state can be rebuilt by reading the log from a known starting point and applying it again.

The idea is old and everywhere. Relational databases use it for crash recovery. LSM-tree stores such as RocksDB write each change to a WAL before putting it in memory. Consensus systems such as etcd and ZooKeeper persist their replicated log the same way. Kafka turns it inside out: the log is the product.

This topic treats the WAL as a design pattern you can use in your own components: what a log record needs, how replay stays correct, how the log is kept from growing forever, and how the same log becomes a replication stream. The Databases page's topic on WAL and durability covers the storage-engine side in depth: fsync, group commit, torn pages, and the PostgreSQL and InnoDB settings.

What you will be able to do

  • Explain why writing to a log first makes a component crash-safe.
  • Design a log record format with ordering and corruption checks.
  • Make recovery correct even when replay runs more than once.
  • Keep a log bounded with segments, snapshots and a low-water mark.
  • Use the same log for replication, and explain what Aurora means by 'the log is the database'.

Key concepts

  • Log first, then apply
  • Log sequence number
  • Record framing and checksums
  • Idempotent replay
  • Segmented log
  • Snapshots and the low-water mark
  • Log shipping
  • The log is the database

Interview questions

  • How does a write-ahead log make a component crash-safe?
  • What fields would you put in a log record, and why?
  • How do you stop the log from growing forever?
  • Why must replay be idempotent, and how do you make it so?
  • How does a WAL relate to replication?
  • What does Aurora mean by 'the log is the database'?
  • Kafka does not fsync each message. How is it durable?

Sign up free to read the full topic

More System Design topics

Start free on Cariara · Coding problems · System design studio