Time Series Database

Storage Engine for Metrics

Design the storage engine of a metrics database: take a million samples a second without losing acknowledged writes, then turn them into compact, immutable blocks that are merged, downsampled and deleted on schedule.

Overview

This problem asks you to design the engine inside a metrics database, not the monitoring product around it. The data model is simple: a series is a metric name plus a set of labels, and it receives samples (a timestamp and a float) every few seconds. The Time-Series Database Design topic explains that model, how delta-of-delta and XOR encoding compress a sample to one or two bytes, and why label cardinality matters. Here we follow a sample through the engine, from the network to the day it is deleted.

The design below follows the layout that Prometheus's storage engine uses, because it is well documented by its maintainers and makes every step concrete: a write-ahead log, an in-memory head, two-hour immutable blocks, compaction into larger blocks, and retention by dropping whole blocks. Downsampling follows the long-term store Thanos, which builds lower resolutions from finished blocks.

Questions to ask first

How many series and how often? — 15 million active series, one sample every 15 seconds: 1 million samples a second
Can a sample arrive late? — Mostly in order; a bounded window of late samples is accepted
How long is raw data kept? — 15 days; downsampled data for a year
What do queries look like? — Label selectors over a time range, mostly the last hours; dashboards over weeks and months
Durability? — An acknowledged sample survives a process crash; machine loss is handled by replication above the engine
Updates and deletes? — No updates; rare deletes of whole series ranges (for example, removing leaked data)

What you will be able to do

Walk a sample through the WAL, the head and its chunks, and explain what a crash loses (nothing acknowledged).
Describe what an immutable block contains and how a label query finds its series.
Plan compaction levels and explain what compaction removes.
Size retention correctly, including the extra block kept at the boundary.
Design a downsampling job whose aggregates still give correct averages and rates.

Key concepts

Write-ahead log
Head block
Chunks of 120 samples
Memory-mapped chunks
Immutable blocks
Postings index
Compaction levels
Tombstones
Block-level retention
Downsampled aggregates

Functional requirements

Accept batches of samples for many series; acknowledge only once durable
Answer range queries by label selectors, from memory and from disk
Delete a series or time range on request (for example, data written by mistake)
Drop raw data older than the retention period, or when a size limit is reached
Build 5-minute and 1-hour downsampled data and query the right resolution
Accept late samples inside a configured window

Non-functional requirements

Sustain 1 million samples a second with room for 3× bursts (a target we chose)
No acknowledged sample lost on a process crash; restart replays the log
Queries over the last hour answer in well under a second (a target we chose)
Background work (compaction, downsampling, deletion) never blocks ingestion
Storage close to 1 to 2 bytes per sample on disk

Interview questions

Walk me through what happens to one sample.
How does the engine avoid losing data on a crash without writing to disk per sample?
Why are blocks immutable, and what does that buy?
How does a label query find its data quickly?
How do retention and deletes work?
What would you store in a downsampled bucket?
How do you scale past one machine?