MLOps CI/CD Automation

Overview

What MLOps CI/CD is

MLOps CI/CD applies continuous integration and delivery to the whole ML lifecycle. Unlike software CI/CD, it has three moving parts: code, data and models. A change to any one can break production, so automation validates all three before promoting an artifact.

Continuous integration for ML

CI covers everything before training at scale, and should finish in under 15 minutes.

• Code quality — flake8/ruff linting, mypy type checks, import sorting
• Unit tests — Feature engineering functions, custom loss functions, preprocessing
• Data validation — Schema checks (Great Expectations, Pandera), distribution drift tests, null-rate thresholds
• Smoke-test training — A run on a data sample catches shape mismatches and config errors early
• Model unit tests — Output shape, prediction range, serialization round-trip

> [!NOTE] Fail fast and cheap. CI uses a tiny data subset to prove the code path works; full training waits for CD.

Continuous delivery for ML

CD takes a merged change to a live endpoint in four stages.

GitHub Actions workflow structure

• ci.yml — Pull request — Linting, unit tests, data validation, smoke-train
• cd-staging.yml — Merge to main — Full training, metric comparison, staging deploy
• cd-production.yml — Manual dispatch or staging approval — Canary rollout and promotion

• Parallel by default: jobs in a workflow run at the same time; needs makes them sequential.
• Passing artifacts: model files and evaluation reports move between jobs with actions/upload-artifact and actions/download-artifact.

Triggering ML pipelines

ML pipelines have richer triggers than software builds.

• Data arrival: a data platform event calls the GitHub Actions REST API to fire workflow_dispatch.
• Manual dispatch: workflow_dispatch input fields for hyperparameter overrides, dataset versions or experiment names.
• Drift alert: monitoring detects production drift and calls the dispatch API for emergency retraining.

Secrets management

Pipelines touch cloud credentials, model registry tokens, evaluation API keys and Weights and Biases tokens.

• Store centrally: GitHub Actions Secrets or an external vault such as HashiCorp Vault or Azure Key Vault.
• Scope by environment: staging deployments cannot reach production cloud credentials.
• Rotate automatically: on a schedule, not by hand.
• Never log secrets: use ::add-mask:: to redact dynamic values.
• Reference model weights: by a content-addressed URI in a model registry, not stored in the workflow.

Infrastructure as code for ML

Terraform manages compute clusters, storage buckets, container registries and serving endpoints; Helm charts manage Kubernetes-based serving. Both are version-controlled and applied through the same CI/CD pipeline, which catches environment drift between dev, staging and production early.

Common pitfalls

• Slow pipelines — Full training in CI makes developers wait hours — Two tiers: smoke tests in CI, full training only in CD
• Flaky data tests — Exact row counts or absolute thresholds fail as data drifts — Relative thresholds and statistical tests
• Environment drift — Trains in the cloud pipeline, fails in the serving container — Pinned dependency files and image digests, not floating ranges
• Missing rollback — Deploy is automated, rollback is not — A tested rollback that restores the previous version with one press or an automated trigger

> [!IMPORTANT] Every CD pipeline needs a tested rollback path, not only a deployment path.

Interview questions

• Walk me through a complete GitHub Actions CI/CD pipeline for an ML project, from pull request to production deployment.
• How do you handle secrets and credentials securely across a multi-environment ML pipeline that touches cloud providers, model registries, and external APIs?
• Describe how you would design an automated evaluation gate that decides whether a new model should be promoted to production.
• What are the main pitfalls teams encounter when building ML CI/CD pipelines, and how do you address each one?

More MLOps & LLMOps topics

• ML Experiment Tracking
• ML Governance and Compliance
• ML Monitoring and Alerting
• ML Pipeline Orchestration
• ML Security and Adversarial Robustness
• ML Testing Infrastructure
• MLOps Lifecycle
• Model Deployment Strategies
• Model Drift Detection
• Model Evaluation — Offline and Online
• Model Packaging and Formats
• Model Registry