Model Deployment Strategies

Blue/green, canary, shadow, A/B, and feature flag strategies for safely promoting ML models to production.

Overview

Why deployment strategy matters for ML

Shipping a new model is riskier than shipping new code. Code is deterministic for the same inputs; a new model can return different outputs for the exact same request, in ways offline metrics do not predict. The deployment strategy controls the blast radius: how fast a bad model reaches users and how fast it can be pulled back.

Blue/green — All at once — Instant switch back — Large changes; no native traffic splitting; rollback speed first — Two full environments during the switch
Canary — 1% → 10% → 50% → 100% — Return traffic to 0% — Gradual, metric-gated exposure — Slower; needs gate design
Shadow — None — Nothing to roll back — Crash, latency and distribution checks — Does not prove better predictions
Feature flags — Chosen cohorts — Toggle in seconds — Targeting users; fast rollback — Selection logic lives in app code
Multi-armed bandit — Shifts toward the winner — Reallocation — Maximizing reward during the test — No definitive statistical answer

Blue/green deployment

Two identical production environments run side by side: blue has the current model, green has the new one. The load balancer switches traffic atomically, and switching back is the rollback.

> [!NOTE] Serving infrastructure cost doubles during the transition window, because both environments run at full size.

Canary deployment

A small share of traffic goes to the new model, raised gate by gate while metrics are checked. If any gate metric leaves its acceptance window, the rollout halts and the new model returns to 0%.

Key design decisions

Gate fractions: what share of traffic at each step.
Observation window: how long to watch each gate.
Gate metrics: business metrics (conversion, revenue per session) need larger samples and longer windows than infrastructure metrics (error rate, latency).
ML metrics: use KS tests to detect significant divergence in output or score distributions.

Shadow mode

The new model gets a copy of every request, but its response never reaches users. It is the lowest-risk evaluation because user impact is zero.

Responses without crashing — That predictions are better
Latency under real traffic patterns — Anything needing user reaction
Output distributions as expected — Use an A/B test or champion/challenger for that

Feature flags

Flags decouple deployment from release. The new model is deployed everywhere, and a flag service (LaunchDarkly, Unleash, GrowthBook) decides which users get it.

Multi-armed bandit

A bandit keeps shifting traffic toward the better variant on a reward metric (clicks, conversions), unlike an A/B test with a fixed split. Thompson Sampling and Upper Confidence Bound (UCB) are the common algorithms.

> [!TIP] Pick a bandit when the goal is maximizing reward during the experiment, as in recommendation and ad ranking; pick an A/B test when you need a statistically definitive answer.

Rollback triggers

Every strategy needs rollback triggers defined before the rollout starts: error rate > X%, latency p99 > Y ms, output distribution KS statistic > Z.

Automated — Monitoring alerts roll back with no human in the loop — Fastest; preferred for fast-moving systems
Manual — On-call engineer sees the alert, judges severity, runs the procedure — Adds minutes

Interview questions

Design a complete canary rollout procedure for a new ranking model at a large e-commerce platform.
Compare feature flags and canary deployments as model rollout strategies. When would you choose each?
How do you implement automated rollback for a production ML model and what signals drive it?

More MLOps & LLMOps topics

Start free on Cariara · Coding problems · System design studio