Metrics Monitoring and Alerting

Datadog

Design a system that collects numeric measurements from thousands of machines, stores them as time series, draws dashboards from them and pages an engineer when something is wrong.

Overview

A metrics system collects numbers such as CPU use, request counts and latencies from every machine and service, every few seconds. Each stream of numbers with one name and one set of labels (for example http_requests_total{service="checkout", status="500"}) is a time series, and each reading is a sample: a timestamp and a value. Engineers look at the series on dashboards, and alert rules watch them around the clock.

The load is mostly writes and it is steady: each target reports on a fixed schedule, so ingestion runs at full rate even at night when no dashboard is open. Queries are fewer but read wide ranges, from the last five minutes to the last year. And the alerting path has an unusual requirement: it must keep working during the outages it exists to report.

How the storage engine compresses samples is covered in the Time-Series Database Design topic, and what to measure (golden signals, SLOs, burn-rate alerts) in the Monitoring topic. This page is the system around them.

Questions to ask first

How many sources, and how many series each? — 20,000 targets (hosts, containers, services), about 1,000 series each: 20 million active series
How often is each sampled? — Every 15 seconds
How long is data kept? — Raw for 15 days, rollups for 13 months
How fast must an alert fire? — Within about a minute of the problem starting (a target we chose)
One team or many? — Many teams, so per-team limits on series and query cost
Exact counts needed, as for billing? — No. A few lost samples are acceptable

What you will be able to do

Choose between pull and push collection and explain the failure each one hides.
Size ingestion and storage from series count and scrape interval, with retention tiers.
Shard series across ingesters and keep queries fast over long ranges.
Design alert evaluation (pending, firing, resolve) and notification routing without paging storms.
Stop cardinality explosions and keep the monitoring system alive during an outage.

Key concepts

Time series and labels
Pull vs push collection
Series sharding
Downsampling
Recording rules
Pending and firing
Grouping and inhibition
Cardinality limits
Meta-monitoring

Functional requirements

Collect samples from hosts, containers and application code
Store each series by metric name and labels
Query with filters, rates, sums and percentiles, grouped by label, over any time range
Define alert rules; route notifications by team and severity; silence alerts during maintenance
Keep long-term data at lower resolution

Non-functional requirements

Ingest about 1.3 million samples a second, with headroom for 2× growth
An alert fires within about a minute of its condition becoming true (a target we chose)
Dashboard queries over the last hour return in under 1 second at p99 (a target we chose)
The alerting path is more available than the systems it watches and fails in a separate failure domain
One team's bad label cannot take down ingestion for everyone

Interview questions

Pull or push: which would you use to collect metrics?
How do you store months of data affordably?
How do alerts avoid paging people for noise?
How do you make alerting itself highly available?
What is cardinality and how do you control it?
How do you shard the write path?
Could this system also count ad clicks for billing?
How do dashboards stay fast over a year of data?