Model Evaluation — Offline and Online

Rigorous offline validation and controlled online experiments to safely promote ML models to production.

Overview

Two phases of evaluation

Offline evaluation happens before deployment on held-out data. Online evaluation measures behavior on live traffic. Offline metrics alone are not enough: a model that scores well on a static test set can still fail in production from distribution shift, latency limits or interactions with other components.

Holdout test set — Offline — None — Does it generalize to unseen labeled data?
K-fold / walk-forward CV — Offline — None — How stable is that estimate?
Fairness slices — Offline — None — Does any subgroup fail?
Shadow mode — Online — None — Crashes, latency, distribution mismatch on real traffic?
Champion / challenger — Online — Small share of traffic — Does the challenger meet success criteria?
A/B test — Online — Split of users — Does it move the business metric?

Offline evaluation

The holdout test set is a random or stratified sample of labeled examples set aside before any training. It is only valid without leakage, including leakage from feature pipelines that look at the whole dataset.

Classification — Precision, recall, F1, AUC-ROC
Regression — MAE, RMSE, R²

Cross-validation

K-fold cross-validation averages performance over K non-overlapping test folds for a lower-variance estimate. Time-series data needs walk-forward validation: training always ends before testing begins.

> [!WARNING] Random K-fold on time-series leaks the future into training (look-ahead bias). Keep temporal order.

Slices and thresholds

A model can be 92% accurate overall and 75% accurate for one demographic subgroup. Fairlearn, TFX Model Analysis and Google's What-If Tool compute metrics per slice.

> [!NOTE] Choosing the operating point on the ROC curve is a business decision, not a purely statistical one.

A/B testing

Two user segments see different model versions, and you measure a business metric (conversion, engagement, revenue) as well as ML metrics.

Minimum detectable effect (MDE) — Set by the business
Statistical power — 80%
Significance level — 5%

> [!TIP] Run long enough to cover weekly seasonality and let novelty effects fade.

Shadow mode

Shadow mode is the safest online technique and the standard first step for a new architecture. The candidate sees a copy of production traffic, but only the production model's answer reaches users.

Champion / challenger

Both models serve live traffic with a fixed split, for example 90% champion and 10% challenger. The challenger is promoted only if it meets pre-defined success criteria over a long enough window.

> [!IMPORTANT] Define rollback criteria (error rate spike, p99 latency regression, output distribution shift) before the experiment starts.

Delayed labels

Ground truth often arrives long after the prediction: a loan default label 30–90 days after origination, a click milliseconds after a recommendation.

> [!WARNING] Evaluating before the label window closes gives artificially optimistic metrics.

Interview questions

How would you design an offline evaluation framework for a production recommendation model that handles temporal leakage and per-segment fairness?
Design an A/B test to evaluate a new fraud detection model. Cover sample size, assignment, metric selection, and stopping rules.
Explain the shadow mode deployment pattern and describe the operational infrastructure needed to support it.

More MLOps & LLMOps topics

Start free on Cariara · Coding problems · System design studio