CI/CD & Progressive Delivery

The deployment-strategy spectrum, when to use each, and SLO-gated automated rollback.

Overview

Where incidents are born

Deployment is where most production incidents start: about 60-80% trace to recent deploys, across multiple industry surveys. How you deploy matters more than what code you ship.

The strategy spectrum, lowest risk to highest

1. Feature flags — Code deployed but inactive; roll out by cohort: internal, 1%, 10%, 100% — Flip the flag, instantly — LaunchDarkly, Unleash, Flagsmith, Statsig
2. Canary — New version on 1-5% of traffic, checked against SLOs over a 10-30 min bake, then 10%, 50%, 100% — Automatic on SLO degradation — Argo Rollouts, Flagger (Flux), Istio primitives
3. Blue/Green — Blue runs v1, Green runs v2; traffic flips at one moment — Flip back to Blue — Faster than canary, needs duplicate capacity during cutover
4. Rolling update — Instances replaced gradually (the Kubernetes default) — Slower; no instant revert — Cheaper than blue/green
5. Big-bang / recreate — Stop everything, deploy, start everything — Redeploy — Mandatory downtime; only for changes that can't coexist

Choosing a strategy

Stateless with good observability — Canary + feature flags: the 2026 standard
Stateful or shared schema — Feature flags + careful migration
Old monolith — Blue/green if you can afford the capacity; rolling otherwise
Database schema changes — Always feature-flagged in the application, using expand-and-contract

SLO-gated automated rollback

During the bake, the new version's SLI is compared with the old version's. If error rate or latency exceeds the old by a threshold (often 1.5×), it rolls back automatically. Argo Rollouts and Flagger implement this directly.

> [!IMPORTANT] This is the modern standard because it removes "an engineer notices something's wrong" from the critical path.

Expand-and-contract for DB migrations

1. Expand — Add the new column or table; old code unaffected
2. Migrate — Backfill the new column from the old
3. Dual-write — Old and new code write to both columns
4. Switch reads — New code reads the new column; old code still works
5. Switch writes — Only new code writes; the new column is the source of truth
6. Contract — Drop the old column

> [!TIP] Each step deploys independently, and rollback at any step is safe.

Deploy frequency vs reliability (Accelerate / DORA)

Deploy frequency — Multiple times per day — Monthly to weekly
MTTR — 1 day
Change-failure rate — 45%

> [!IMPORTANT] The non-obvious finding: frequent small deploys are more reliable than infrequent large ones. Smaller surface area, easier rollback, less cumulative risk.

Key concepts

Feature flags: Runtime toggles for new code paths. Decouple deploy from release. Instant rollback (flip flag).
Canary deployment: Small fraction of traffic to new version; bake period; SLO-gated promotion. Standard for stateless services.
Blue/Green: Two complete environments; instant cutover. Fast rollback; needs duplicate capacity.
SLO-gated rollback: Auto-rollback if canary's SLI exceeds baseline by threshold. Removes human-in-loop from rollback decision.
Expand-and-contract: DB migration pattern: add column → backfill → dual-write → switch reads → switch writes → drop. Each step independently rollbackable.
DORA metrics: Deploy frequency, lead time, MTTR, change-failure rate. Industry benchmark for delivery performance.

When to use it

Designing the deployment pipeline for a new service — start with canary + feature flags
Reviewing existing pipeline — does it have SLO-gated auto-rollback?
Database schema changes — expand-and-contract is required, not optional
Post-incident — was the deploy strategy a contributing factor? Often yes

Interview questions

Walk me through canary deployment with SLO-gated rollback.
How do you handle a database schema change safely?
What does the DORA research say about deploy frequency vs reliability?