AI Data Quality and Monitoring System

Data Observability for Pipelines and ML Features

Design a platform that checks every pipeline output for freshness, volume, schema and distribution problems, stops bad data before models and dashboards read it, and uses lineage to find the cause and everything affected.

Overview

Most data pipeline failures are quiet. A job finishes 'successfully' but loads half the rows, an upstream team renames a column, a mobile release starts sending a price in cents instead of dollars. Nothing crashes. A dashboard shows a dip, or a model starts making confident wrong predictions, and someone notices days later.

A data quality platform (often called data observability) watches pipeline outputs the way monitoring watches services. After each load it checks freshness (did the data arrive on time), volume (about the usual number of rows), schema (same columns and types), rules (no null IDs, amounts are positive), and distributions (does each important column still look like it did). When a check fails, it can stop the bad data from being published, find which upstream table started the problem, and list every table, feature and model downstream.

The hard parts are not running checks; they are precision and reach. Too many false alerts and people mute the channel. Too few checks and the silent failures continue. And a failure is only useful if it comes with its cause and its blast radius.

Questions to ask first

How many tables and pipelines? — 10,000 tables, refreshed on average 4 times a day
Batch or streaming? — Mostly batch tables in a lakehouse or warehouse; streams are checked in windows
Who writes checks? — Table owners write rules; the platform adds freshness, volume, schema and profile checks automatically
What about ML? — Features used by production models are compared with the data the model was trained on
Can we block data? — Yes, for critical tables: publish only after checks pass
How fast? — Alert within 5 minutes of a load finishing (our target)

What you will be able to do

List the five kinds of checks and write freshness and volume checks that tolerate normal variation.
Detect distribution drift with the KS test, and explain why a p-value alone pages people for nothing on big tables.
Stop bad data from reaching consumers with write-audit-publish and quarantine.
Use a lineage graph to find the root cause of a failure and everything it affects, and group alerts into one incident.

Key concepts

Freshness and volume
Schema drift
Expectations
Column profiling
KS test
Effect size
Seasonal baselines
Write-audit-publish
Lineage graph
Alert grouping

Functional requirements

Run freshness, volume and schema checks automatically on every monitored table after each load
Let owners declare rules (not null, unique, in range, matches a pattern, custom SQL) with a severity
Profile columns (null rate, distinct count, quantiles, top values) and keep the history
Detect distribution drift for ML features against the training snapshot and against recent history
Block publication of a failing load for critical tables, and quarantine already-published bad partitions
Build lineage from pipeline runs; show downstream impact and upstream root-cause candidates for a failure
Group related failures into one incident, route it to the owning team, and learn from dismissed alerts

Non-functional requirements

Checks for a load finish, and alerts go out, within 5 minutes of the load completing (our target)
Checks run on the data platform's own compute close to the data; raw data is not copied out
Alert precision is tracked; each check that pages people must be worth reading
Metric history kept for at least 90 days to learn seasonality
If the quality platform is down, pipelines continue; only critical tables wait at the publish gate

Interview questions

Which checks would you run first on a new warehouse?
How do you detect distribution drift, and what goes wrong on big tables?
How do you avoid alert fatigue?
How do you stop bad data from reaching a model?
A feature table fails. How do you find the cause and the impact?
Where do checks run, and how does it scale?
How would you measure whether the platform works?