Data Pipeline Orchestration and Backfills
How batch data pipelines are scheduled and kept correct: tasks in a DAG, runs that each own a fixed time window, idempotent loads that overwrite one partition, retries and deadlines, backfills, and dependencies between pipelines.
Overview
A data pipeline moves data from where it is produced to where it is used: read raw orders, clean them, join them with customers, and write a daily revenue table. Each step is a task. An orchestrator such as Apache Airflow or Dagster decides when each task runs, in what order, what happens when one fails, and how to run old dates again.
Running a script on a timer works for one job. It breaks down at fifty. Jobs depend on each other, sources arrive late, a task fails at 3 a.m., and someone finds a bug that has been writing wrong numbers for a month. The orchestrator's job is to make all of that routine: retry what failed, wait for what is late, and rebuild any past day with one command.
The central idea is simple. Each run processes a fixed slice of time, and each task can be run again for that slice without changing the answer. Get those two things right and retries, backfills and recovery all become safe.
What you will be able to do
- Model a pipeline as a DAG and explain what starts each run.
- Explain data intervals and why a run must not read the latest data.
- Design an idempotent load that overwrites one partition through a staging table.
- Plan a backfill, estimate its cost, and keep it from flooding shared systems.
- Choose between time schedules, sensors and asset-driven triggers, and know when a stream processor is the better tool.
Key concepts
- DAG of tasks
- Schedules and data triggers
- Data interval
- Idempotent partition overwrite
- Retries and timeouts
- Backfill and catchup
- Sensors
- Asset-driven scheduling
- Task vs asset orientation
Interview questions
- Design the orchestration for a daily revenue pipeline that feeds a 7 a.m. dashboard.
- What is a data interval, and why does the run for Monday start on Tuesday?
- How do you make a load task idempotent?
- You found a bug that wrote wrong numbers for the last 90 days. Walk me through the backfill.
- Sensor or asset-driven scheduling: how do you link two pipelines?
- What is the difference between task-oriented and asset-oriented orchestration?
- Your pipeline runs every 15 minutes and the business wants data within 10 seconds. What do you do?
- How do you know a pipeline has silently stopped?
Sign up free to read the full topic
More System Design topics
- CRDTs: Conflict-Free Replicated Data Types
- Customer Onboarding and KYC
- Customer Support Chatbot System
- Data Labeling and Annotation System
- Data Lakes and Lakehouse Table Formats
- Data Partitioning
- Data Quality, Contracts and Lineage
- Data Warehousing and Dimensional Modeling
- Database Indexes
- Databases
- Dealing with Contention
- Delivery Semantics
Start free on Cariara · Coding problems · System design studio