ML Monitoring and Alerting

ML monitoring tracks both infrastructure health and model performance over time, using drift detection and alerting strategies to trigger retraining or rollback before degradation impacts users.

Overview

Deploying a model is only the beginning. Without continuous monitoring, even a well-trained model degrades silently as the world changes around it. Monitoring has two dimensions: infrastructure (is the system up and fast?) and model performance (are predictions still good?).

Question — Is it available and fast? — Is it still predicting well?
Cadence — Real time, synchronous — Scheduled batch jobs, asynchronous
Signals — Latency, throughput, errors, resources — Drift in inputs, outputs and attributions
Escalation — Pages immediately — Tickets after drift is confirmed over several windows

Infrastructure monitoring

These signals come from classic site reliability engineering.

Latency — P50, P95, P99 for inference endpoints
Throughput — Requests per second; tokens per second for LLMs
Error rate — 4xx client vs 5xx server errors, per route
Resource utilization — CPU, GPU memory, VRAM saturation, host memory
Request queuing — Queue depth, wait time before a worker picks up the request

Prometheus and Grafana handle this layer. Kubernetes exporters expose pod-level GPU metrics (DCGM exporter for NVIDIA), and an Alertmanager rule can page on-call when P99 latency exceeds the SLO for 5 minutes.

Model performance monitoring

Model monitoring is harder because the ground-truth label is often delayed or never arrives, so teams rely on proxy signals.

Prediction distribution shift — The histogram of model outputs moves away from the training baseline
Input data drift — Feature distributions change (covariate shift); detected with PSI, KL divergence or Maximum Mean Discrepancy
Concept drift — The true input-output relationship changes; visible only when labels arrive
Feature importance drift — SHAP attributions change, so different features now drive predictions
Outlier rate — Share of inputs outside the training manifold

Azure ML DataCollector logs inference inputs and outputs to a storage account; scheduled monitoring jobs compute drift against a baseline and raise alerts in Azure Monitor.

Alerting strategies

Threshold — A metric crosses a fixed value, such as PSI > 0.2 on any feature — Simple; false positives on seasonal data
Anomaly detection — A metric leaves a learned range (statistical process control or ML); Azure Monitor dynamic thresholds — Adapts to patterns; less predictable
SLO-based — An error budget is burning too fast — Needs a measurable model SLO, such as accuracy above 85% on labeled production samples

Alert fatigue

Alerting on every threshold breach floods on-call engineers and teaches them to ignore alerts.

Tiering: P1 pages immediately, P2 opens a ticket, P3 goes to a dashboard only.
Correlation: group related signals into one incident instead of 20 alerts for one bad data pipeline.
Inhibition: suppress downstream alerts while the upstream root-cause alert is firing.
Prioritization: surface alerts on high-traffic or high-revenue segments first.

Retraining triggers

Monitoring closes the loop by starting retraining.

Scheduled — Simple, predictable — Retrains whether needed or not
Drift-triggered — Reacts to real change — Noisy signals can cause spurious retraining
Performance-triggered — Tied to actual quality — Needs labels
Human-in-the-loop — Judgment before spending compute — Slower

Tooling ecosystem

Evidently AI — Open-source drift reports and test suites; integrates with MLflow and Airflow
Arize AI — Commercial; embedding drift for unstructured data, LLM tracing
Fiddler AI — Explainability first; SHAP-based attribution drift
WhyLabs — Lightweight profiling with whylogs for high-throughput streaming inference
Azure ML Data Monitoring — Native with DataCollector, Log Analytics and Azure Monitor alerts
Prometheus + Grafana — Standard for infrastructure and custom business metrics

> [!TIP] In an interview, separate the two layers first: infrastructure alerts page in seconds, model drift alerts open tickets after confirmation.

Interview questions

Walk me through how you would design an end-to-end monitoring system for a fraud detection model serving 10 million transactions per day.
What is the difference between data drift, concept drift, and model drift, and how do you detect each in practice?
How does Azure ML's built-in data monitoring work, and what are its limitations compared to a custom solution?
How do you avoid alert fatigue in a production ML monitoring setup, and what is the role of alert tiering and inhibition?

More MLOps & LLMOps topics

Start free on Cariara · Coding problems · System design studio