Unbundling the Database

The idea of treating a whole data platform as one database taken apart: an ordered log of writes at the center, and each index, cache and warehouse built from it as derived data.

Overview

A database is several tools in one box

A single relational database does many jobs. It stores rows durably, keeps secondary indexes, answers full-text search, maintains materialized views, replicates to followers and runs triggers. Inside the box, one piece of machinery keeps all of those in step: every change goes into the database's log first, and everything else is derived from that log.

Once a system outgrows one box, teams add specialized tools: a search engine, a cache, a warehouse, a stream processor. Each one does its job better than the general database, but now the machinery that keeps everything in step is gone. Unbundling the database means rebuilding that machinery across the tools: an ordered log of changes in the middle, and each tool built from it.

Write-ahead log — A durable, ordered event log (for example Kafka)
Secondary index — A search engine or key-value store built from the log
Materialized view — A table kept up to date by a stream processor
Replication to followers — Consumers that each keep their own copy
Trigger — A consumer that reacts to each change
Query planner choosing an index — Your code choosing which store to read

Systems of record and derived data

Every piece of data in the platform is one of two kinds. The system of record is where a fact is first written and whose answer wins in a disagreement. Derived data is anything computed from it: an index, a cache, a denormalized table, a model's features. Derived data can always be thrown away and rebuilt by reprocessing the source.

Getting this split right is most of the design. If two systems both accept writes for the same fact, you are back to dual writes: each may be right on its own, and nothing decides between them.

Why a log, not direct calls

There are two ways to connect the tools. The application can call each store directly on every write, or it can write once and let each store follow an ordered log.

Consistency — Partial failure leaves stores different — Every consumer applies the same changes in the same order
Coupling — The writer must know every store — The writer knows only the log
Adding a new store — Change the writer; backfill by hand — New consumer reads from the start of the log (or a snapshot)
A slow store — Slows every write — Only that consumer falls behind
Freshness — Immediate, when it works — Asynchronous: consumers lag by some amount

This is the argument Martin Kleppmann made in his 2014 talk Turning the Database Inside-Out: make the log that databases hide into the shared, public interface, and build the read-optimized views from it with stream processors.

Federated reads versus unbundled writes

There are two different goals people mean by one interface over many stores. Federation is about reads: one query language that can read from several stores in place. Unbundling is about writes: keeping many stores in step with each other. They solve different problems, and a platform can use both.

Precompute on write, or compute on read

Every derived view moves work between the write path and the read path. A search index does work at write time (tokenizing, updating postings) so reads are cheap. A plain table scan does no work on write and all of it on read. Caches, indexes and materialized views are all points on that line, and the log lets you choose per view.

No index — None — Scan everything
Secondary index — Update the index entry — Look up and fetch
Materialized view — Recompute or update the result — Read one row
Cache — Fill or update an entry — Hit, or fall back to the source

Start inside one database

Most of these views exist inside PostgreSQL already. Before adding systems, check whether the built-in versions are enough. Here are an index, full-text search and a materialized view, all in one database.

ana — 2
ben — 2

Note the refresh: PostgreSQL recomputes the whole view when asked, rather than updating it with each write. That limit, along with read load and features such as search ranking at scale, is what eventually pushes teams to unbundle.

The costs

Unbundling trades one system's guarantees for many systems' flexibility.

Stale reads — Derived stores follow the log asynchronously — Show freshness; read your own writes from the system of record
More to operate — Each store and the log need monitoring, upgrades, on-call — Unbundle only the parts that need it
No cross-store transactions — Each store commits on its own — One system of record per fact; idempotent consumers
Harder debugging — A wrong value may come from any stage — Keep the log so any view can be rebuilt and compared

What you will be able to do

Map each feature of a single database to its unbundled counterpart.
Separate systems of record from derived data in a design, and say why only one place accepts writes for each fact.
Explain why an ordered log is safer than direct writes to many stores.
Decide when one database is enough and when to add specialized stores.
Name the costs of unbundling and how to contain them.

Key concepts

Unbundled database: A platform built from specialized stores that are kept in step by an ordered log of changes, playing the role that one database plays internally.
System of record: The one place a fact is first written. If copies disagree, its answer wins.
Derived data: Anything computed from the system of record, such as an index, cache or warehouse table. It can be deleted and rebuilt.
Ordered log: An append-only sequence of changes that every consumer reads in the same order, so every view converges on the same state.
Materialized view: A stored query result. Inside PostgreSQL it is refreshed on demand; in an unbundled system a stream processor keeps it current.
Federation: One query interface that reads from several stores in place. It unifies reads, not writes.
Write path and read path: The work done when data is written versus when it is read. Derived views move work from reads to writes.

When to use it

Designing a data platform where search, caching, analytics and serving all need the same facts.
Explaining why a company runs several data stores, and how they stay consistent.
Arguing for or against adding a new specialized store to a system.
Choosing where each fact is written and which views are rebuilt from it.

Interview questions

How would you design the data layer for a social app that needs profiles, a feed, search and analytics?
Why keep derived stores in step through a log instead of writing to each one from the application?
When is a single database the better choice?