Anomaly Detection in Ops

Algorithms for catching metric deviations without static thresholds — 3-sigma and ESD on stationary series, STL / Prophet / DeepAR for seasonal forecasting, isolation forest and autoencoders for multivariate. Covers when statistical methods are sufficient, when ML is genuinely needed, and the operational pitfalls.

Overview

The problem

Anomaly detection in operations separates metric values that represent genuine problems from the natural variance of a healthy system. A static threshold (alert when CPU exceeds 80 percent) solves a simpler problem. Most production metrics are not stationary: they have daily business-hours cycles, weekly peaks, growth trends and holiday distortions.

The algorithm hierarchy

Pick the simplest method that matches the shape of the series.

3-sigma — Rolling mean and standard deviation; alert beyond mean plus or minus three standard deviations — No trend or seasonality; fast and interpretable
ESD (Generalized Extreme Studentized Deviate) — Detects up to k outliers in a sample without specifying which ones in advance — Stationary samples with several outliers
STL (Seasonal-Trend decomposition using LOESS) — Splits the series into trend, seasonal and residual, then applies statistical detection only on the residual — A clear daily or weekly cycle; fixes the "alert every Monday morning" problem
Prophet (Meta, 2017) — Bayesian additive model: flexible piecewise trend, Fourier series for seasonality, explicit holiday regressors — Several seasonal periods at once plus events such as Black Friday or product launches

When ML goes beyond statistics

Isolation Forest — Random tree splits; anomalies are easy to isolate (few splits), normal points cluster (many splits); no labels needed; handles high dimensions better than distance-based methods such as LOF — Production workhorse for multivariate detection
LSTM autoencoder — Trained to reconstruct normal sequences; reconstruction error spikes on anomalous ones — Expensive to deploy and maintain; reserved for security UEBA, IoT predictive maintenance, fraud detection

Operational realities

The algorithms are not the hard part.

Sensitivity tuning: too tight fires on every deploy, too loose misses real anomalies.
Regime changes after deploys: the model's learned normal is now wrong.
Holidays and one-off events: the model has not seen them.
Low-volume series: not enough data for statistical detection.

> [!TIP] Most production anomaly detection is STL plus 3-sigma on the residual, with Prophet for series that have explicit holiday patterns. Deep learning is rarely needed in pure ops contexts.

Interview questions

Walk me through choosing an anomaly detection algorithm for a new ops use case.
How do you reduce alert fatigue from an anomaly detection system that fires too much?
Compare Datadog Watchdog, Dynatrace Davis anomaly detection, and a DIY Prophet-based detector.
How does anomaly detection interact with SLO-based alerting -- should you replace one with the other?
What are the production pitfalls of deploying anomaly detection on a Kubernetes or microservices environment at scale?