Data Quality, Contracts and Lineage
How to keep bad data from readers: what to check and where, which failures stop a load, how contracts and compatible schema changes protect consumers, and how lineage traces a failure.
Overview
A pipeline can succeed and still be wrong. The job exits with status 0, yet the table it wrote is missing half of yesterday's orders, or every customer_id is null because a source system renamed a column. Nothing crashed, so nobody notices until a dashboard looks odd or a model starts making strange predictions. Data quality work is about catching these silent failures before anyone reads the data.
The work rests on three ideas that support each other. Checks test each load for freshness, volume, schema and valid values, and the important ones stop a bad load from being published. Data contracts turn the unwritten expectations between the team that produces a table and the teams that read it into something written, versioned and tested, including how the schema may change. Lineage records which jobs read and write which tables (and which columns feed which columns), so when a check fails you can see what it breaks downstream and where the problem started upstream.
This topic covers those ideas for any analytics or data platform. For the ML side (profiling features at scale, distribution drift, anomaly detection on quality metrics, and the design of a whole monitoring service) see the AI Data Quality and Monitoring System problem. For how batch and streaming pipelines are built in the first place, see the Batch vs Stream Processing topic.
What you will be able to do
- Pick the checks a table needs and say where each one should run.
- Decide which checks block a load and which only warn, and explain the cost of each choice.
- Implement write-audit-publish so consumers never see a load that failed its checks.
- Write a data contract and evolve a schema without breaking its readers.
- Use table and column lineage, in the OpenLineage model, to find the impact and root cause of a bad load.
Key concepts
- Freshness and volume checks
- Null, uniqueness and referential checks
- Write-audit-publish
- Blocking vs warning checks
- Data contracts
- Backward and forward compatibility
- Table and column lineage
- OpenLineage runs, jobs, datasets and facets
- Data incident flow
Interview questions
- A daily table was wrong for three days before anyone noticed. How would you stop that happening again?
- Explain write-audit-publish. How would you implement it?
- What is a data contract, and what goes in one?
- What is the difference between backward and forward compatibility, and which side do you upgrade first?
- How would you rename a column that twenty teams read?
- What is lineage for? Table level or column level?
- Describe the OpenLineage model.
- A finance dashboard shows revenue down 40% this morning. Walk me through what you do.
Sign up free to read the full topic
More System Design topics
- Customer Onboarding and KYC
- Customer Support Chatbot System
- Data Labeling and Annotation System
- Data Lakes and Lakehouse Table Formats
- Data Partitioning
- Data Pipeline Orchestration and Backfills
- Data Warehousing and Dimensional Modeling
- Database Indexes
- Databases
- Dealing with Contention
- Delivery Semantics
- Deployment System
Start free on Cariara · Coding problems · System design studio