Model Drift Detection

Overview

What drift is

A deployed model assumes the world it was trained on still exists. Drift is the umbrella term for distribution shifts that degrade its predictions. Three kinds of drift are causes; performance decay is the symptom.

The four drift types

• Covariate shift — P(X); P(Y given X) stays the same — Mobile payments dominate 2026 traffic, but training data was desktop-heavy 2024 — No, input data alone
• Concept drift — P(Y given X): same inputs, different correct output — Fraudsters adapt their patterns — Yes, which leads to the delayed-label problem
• Label drift (prior probability shift) — P(Y) — Fraud rate goes from 1% to 3%; probabilities calibrated for 1% now underestimate risk — No, visible in predicted scores
• Performance decay — Model quality against ground truth — The outcome of the other three — Yes

> [!NOTE] Covariate shift means the model's logic is still valid, but it has never seen these inputs. Concept drift means the logic itself is now wrong.

Detection methods

• PSI (Population Stability Index) — Tabular features — Under 0.1 stable, 0.1–0.25 moderate, over 0.25 significant
• KS test — Continuous distributions — Two-sample non-parametric comparison
• Jensen-Shannon divergence — Categorical or binned features — Symmetric, bounded version of KL divergence
• MMD (Maximum Mean Discrepancy) — High-dimensional embeddings — Kernel-based, no binning
• Domain classifier — Any feature type — Train a classifier to tell reference from current; high accuracy means drift

The delayed-label problem

Many systems get labels weeks or months after predicting: loan default at 60 days, churn at the billing period. Without recent labels you cannot compute current performance directly, so you watch earlier signals.

> [!WARNING] The delayed-label problem is the hardest practical challenge. Layer input drift, predicted-distribution monitoring, early proxies and a labeled reference panel.

The 2026 monitoring landscape

• Evidently AI (OSS) — Most widely adopted for tabular drift, with prebuilt dashboards and CI integration
• Arize AI — Leads on embedding and LLM observability
• Fiddler AI — Regulated finance, with explainability integration
• WhyLabs — whylogs, a lightweight profiling library for distributed pipeline stages
• Cloud-native — SageMaker Model Monitor, Vertex AI Model Monitoring, Azure ML Data Drift Monitor

Interview questions

• Explain the four types of drift and how you would detect each in production.
• What is Population Stability Index and how do you use it in a production monitoring pipeline?
• How do you handle the delayed-label problem for a model monitoring system?
• Compare Evidently, Arize, and Fiddler as monitoring platforms. When would you choose each?
• Design a complete model monitoring system for a fraud detection model with 60-day label delay.

More MLOps & LLMOps topics

• ML Pipeline Orchestration
• ML Security and Adversarial Robustness
• ML Testing Infrastructure
• MLOps CI/CD Automation
• MLOps Lifecycle
• Model Deployment Strategies
• Model Evaluation — Offline and Online
• Model Packaging and Formats
• Model Registry
• Model Serving
• Online and Continual Learning
• Real-World MLOps Case Studies