ML Testing Infrastructure

Building a complete ML test pyramid covering data validation, behavioral model tests, pipeline determinism, and serving contracts.

Overview

Why ML systems need specialized testing

Unit, integration and end-to-end tests apply to ML systems but are not enough on their own. A mature ML testing setup extends the standard test pyramid with layers for data and model behavior.

Three failure modes traditional tests miss

Silent data quality degradation: bad data trains a bad model without any error.
Unexpected behavior on edge inputs: inputs at the edge of the distribution.
Training non-determinism: the pipeline produces different results run to run.

Unit — Transformations, feature logic, pre/post-processing, forward-pass shapes — Every commit — pytest
Integration — Pipeline stages compose correctly — Pipeline changes — Small synthetic dataset
Data validation — Schema, nulls, ranges, cardinality, drift — Every new data batch — Great Expectations, Soda, TFDV
Training pipeline — Determinism, convergence, overfitting — Every training code commit — CPU in CI
Behavioral — Invariance, direction, minimum functionality — After training, before registration — pytest parametrize
Serving — Schema contract, latency SLO, throughput — Against the endpoint — Load and contract tests

The ML test pyramid

Unit tests form the base: deterministic code that is fast and cheap to test on every commit. A transformation mapping raw categorical values to embeddings is tested for correctness, null handling and boundary values; architecture tests check that a forward pass returns the expected output shape.

Integration tests run the full pipeline on a small synthetic dataset and check the joins between stages.

> [!NOTE] Integration tests catch interface mismatches between stages that unit tests cannot see.

Data validation tests

Data validation is a first-class test category. An expectation suite is generated from a reference batch of clean training data and run against every new batch.

Great Expectations — "Expectations" on schema, nulls, ranges and distributions as versioned suites
Soda — SQL-first; soda scan checks defined in YAML
TFX Data Validation (TFDV) — Built into TensorFlow Extended; distribution-based anomaly detection

Schema conformance — Column names and types
Null rate — 0, price > 0
Cardinality — No categories in serving that were absent in training
Distribution drift — PSI or KS statistic against training statistics

Model behavioral tests

Behavioral testing, inspired by the CheckList framework (Ribeiro et al. 2020), judges the model on meaningful test cases instead of aggregate metrics.

Invariance — Small input changes must not change the output — Changing a name on a resume must not change a hiring model's score
Directional — Raising a feature moves the output the expected way — A higher credit score raises loan approval probability
Minimum functionality — Obvious cases must be classified correctly — A hand-curated set of clearly positive and clearly negative examples

> [!IMPORTANT] A model that fails behavioral tests is not registered, regardless of its aggregate AUC.

Training pipeline tests

These tests check the training process itself and catch silent bugs: a wrong loss function, broken gradient flow from detached tensors, a mis-implemented learning rate schedule. They run quickly on CPU and belong in CI on every training code commit.

Determinism — Train twice with a fixed seed on the same data — Weights bit-for-bit identical
Convergence — 10 gradient steps on a tiny batch — Loss decreases monotonically
Overfitting — 100 epochs on 10 examples — Near-zero training loss (gradients correct, enough capacity)

Serving tests and hermetic environments

Serving tests check the contract between the model and its serving infrastructure.

Schema contract — Send a valid inference request — Response fields, types and value ranges match
Latency SLO — Send N concurrent requests — p99 below the SLO
Throughput — Sustain the target QPS — No error rate degradation

Hermetic testing isolates tests from outside dependencies, so results are the same whenever and wherever they run.

> [!TIP] Hermeticity is the fundamental property that makes CI reliable.

Interview questions

Design an ML test suite for a text classification model that catches data quality issues, behavioral regressions, and serving contract violations.
How would you implement and maintain a behavioral test suite for an ML model that evolves over time?
Describe a complete CI/CD pipeline for an ML project that integrates all testing layers and production deployment gates.

More MLOps & LLMOps topics

Start free on Cariara · Coding problems · System design studio