ML Pipeline Orchestration

Overview

The orchestration problem

An ML pipeline is a directed acyclic graph of steps. Running them by hand works for one experiment but breaks at scale: steps need retries, intermediate artifacts need caching, runs need schedules, and teams need visibility into what ran, when and with what inputs.

• Kubeflow Pipelines — ML-specific, Kubernetes-native — @component and @pipeline in Python — Container per step, automatic lineage, step caching — Needs Kubernetes
• Apache Airflow — Task-centric — Python DAG files in dags/ — Backfill, sensors, huge operator library — No built-in notion of data assets
• Prefect — Task-centric, async-native — Decorated flows and tasks; DAG inferred from data flow — Declarative retries, timeouts and caching per task — —
• Dagster — Asset-centric — Software-Defined Assets — Freshness, partitions, asset catalog — —
• Metaflow — ML-specific — Python class with @step methods — Same code local and in the cloud; per-step resources — —

Kubeflow Pipelines

KFP runs each step in its own Docker container for full environment isolation. Components declare typed inputs and outputs, and KFP passes artifacts between them through MinIO or a cloud object store.

• Lineage: trace any model back through evaluation, training and features to raw data.
• Step caching: keyed on component code and input fingerprints; reruns skip cached steps.
• UI: visual DAG, run history and artifact viewer in the Kubeflow dashboard.

Apache Airflow

Airflow is the most widely deployed workflow orchestrator in the data industry. Pipelines are Python DAG files, and operators do the work.

• Operators: PythonOperator, BashOperator, KubernetesPodOperator, DockerOperator.
• XCom: tasks push small values for downstream tasks to pull.
• Sensors: block until an external condition holds, such as a file in S3 or a finished Spark job.
• Backfill: reprocesses historical intervals, a natural fit for time-partitioned ML pipelines.

> [!WARNING] Airflow treats tasks as the unit, not data assets. There is no built-in record of which dataset a task produces, so lineage is an add-on.

Prefect

Prefect is built on an async-native Python runtime. Flows and tasks are decorated functions, and Prefect infers the DAG from data dependencies.

• Deployment: Prefect Cloud is the managed control plane; agents run locally or in Kubernetes and poll for work.
• Policies: retries, timeouts and result caching are declared on each task.
• Failure modes: fail-fast surfaces errors immediately; deferred retry queues failed tasks for later without blocking the rest of the flow.

Dagster

Dagster is asset-centric. You declare the assets that should exist (a training dataset, a feature table, a model) and Dagster derives the job graph from their dependencies.

• Partial materialization: build a subset of assets on demand and check their freshness.
• Partitions: materialization policies at the partition level.
• Type checks: typed inputs and outputs catch schema mismatches in the framework, not at runtime.
• UI: asset catalog with lineage graphs, run history and per-asset status.

Metaflow

Netflix built Metaflow and open-sourced it in 2019. A pipeline is a Python class whose @step methods are DAG nodes.

• @resources: CPU, memory and GPU per step; Metaflow provisions the matching AWS Batch or Kubernetes job.
• @conda and @pypi: pin dependencies per step.
• Local mode: the same code runs locally and in the cloud without changes.
• Artifacts: versioned automatically and retrievable through the Client API.

Choosing an orchestrator

• Scheduled ETL with well-understood steps — Airflow, Prefect
• Data freshness and lineage as core concepts — Dagster
• GPU scheduling, artifact lineage, experiment tracking — Kubeflow Pipelines, Metaflow

> [!TIP] Many mature platforms use two layers: Airflow schedules the data pipelines, and KFP or Metaflow runs the training DAG.

Interview questions

• Compare task-centric and asset-centric orchestration models and explain when you would choose each for an ML platform.
• How would you design a fault-tolerant ML training pipeline that handles transient failures without rerunning expensive steps?
• Describe how you would implement end-to-end artifact lineage tracking across a multi-step ML pipeline.

More MLOps & LLMOps topics

• Hyperparameter Tuning and AutoML
• LLM Serving
• LLMOps
• ML Experiment Tracking
• ML Governance and Compliance
• ML Monitoring and Alerting
• ML Security and Adversarial Robustness
• ML Testing Infrastructure
• MLOps CI/CD Automation
• MLOps Lifecycle
• Model Deployment Strategies
• Model Drift Detection