AI Anomaly Detection System

Anomaly Alerts for Service Metrics

Design a system that watches millions of service metrics, learns what normal looks like for each one, and pages an engineer only when something is really wrong, with the related signals grouped into one incident.

Overview

A large company runs thousands of services, and each one emits metrics: request rate, error rate, latency, queue depth, CPU. With every host, endpoint and region as a separate label, that is millions of series (one sequence of numbers per metric and label combination). Engineers cannot set a hand-written threshold for each, and fixed thresholds fail anyway, because normal traffic at 3 p.m. and 3 a.m. are very different numbers.

An anomaly detection system learns what normal looks like for each series and flags values that fall outside it. The hard part comes after detection: when a database slows down, dozens of metrics across many services go wrong at once, and the on-call engineer needs one page that says where to look, not fifty.

Questions to ask first

How many series, how often? — 2 million active series, one point every 10 seconds
How fast must a real problem page someone? — Within 5 minutes of it starting
What counts as success? — Few pages, nearly all worth acting on; missing a real outage is worse than one extra page
Do we know the service dependencies? — Yes, from a service catalog or tracing
Do deploys and maintenance happen? — Yes: planned maintenance must not page; a deploy still pages, with the deploy attached
Can engineers give feedback? — Yes: mark an alert as useful or noise

What you will be able to do

Explain why fixed thresholds fail and build a baseline that follows daily and weekly cycles.
Score points with a robust z-score and say why median and MAD beat mean and standard deviation.
Do the false-alarm arithmetic that shows why per-point alerting cannot work.
Turn a burst of anomalies into one incident and one page, safely under concurrency.
Handle deploys, maintenance, new metrics and a detector that has itself gone blind.

Key concepts

Seasonal baseline
Median and MAD
Robust z-score
Sustain window
Keyed stream state
Alert grouping
Inhibition and silences
Dependency-aware correlation
Feedback loop
Cold start

Functional requirements

Accept metric points in batches from agents and exporters
Learn a baseline per series that follows its daily and weekly cycle
Score each series every minute and emit an anomaly when it stays outside its band
Group related anomalies into one incident, using time and service dependencies
Page once per incident; honor silences and maintenance windows, and attach recent deploys to the incident
Let engineers mark an alert useful or noise, and tune sensitivity per series
Keep incident history for postmortems

Non-functional requirements

Page within 5 minutes of a sustained anomaly starting (a target we chose)
At most a few pages per team per week that are not worth acting on (our target)
The detector is more available than what it watches, and alerts when its own input stops
Ingest keeps accepting points during a scorer failure; scoring catches up from the stream
No duplicate pages for one incident, even with retries and parallel workers

Interview questions

Why not just alert when a metric is 3 standard deviations from its mean?
How do you build and store the baseline?
How does scoring keep up with 33,000 checks a second?
How do you avoid paging 50 times for one outage?
How do you find the likely root cause?
What about deploys and maintenance?
A new service launches today. How is it monitored?
How do you know the detector is getting better?