Deployment Rollback Strategies

Overview

Overview

The most reliable systems treat rollback as a design goal, not an edge case: fast, tested and almost automatic. The core challenge is that code and data roll back differently. Code swaps instantly; schema changes are often hard or impossible to reverse safely.

Code rollback: two approaches

• Deploy a previous artifact — Point the deployment system at an older stored build — Lower; the artifact was already validated in production
• Git revert and deploy — A new commit undoes the change — Higher; a new commit can introduce its own issues. Correct for long-lived branches, not for speed

> [!TIP] Most CI/CD systems keep build artifacts for a configurable period, so rollback is just picking an older one.

Database rollback

• Add column, create table, add index — Generally yes
• Drop column or table once data exists — No

The industry answer is forward-only migrations: no rollback scripts for destructive changes, and every schema works with both the old and new code at once.

> [!WARNING] Never remove a column in the same deployment that removes the last code reference to it.

Release strategies and their rollback

• Feature flags — Turn off the flag; the migration can land weeks before the code, and code can ship dark — The flag becomes the release mechanism, separate from deployment
• Blue/green — A single DNS change or ALB weight shift back to blue — Two full production environments at once
• Canary — Set canary traffic weight to 0 — Knowing when to roll back

Canary rollback triggers

Define and automate thresholds before the deployment, not during an incident. Example thresholds:

• Error rate — More than 0.5% above stable
• p99 latency — Regression of more than 20%
• Business metric (revenue per request) — Drop of more than 2%

> [!IMPORTANT] Improvised rollback criteria during an incident lead to slow, argued decisions.

Key concepts

Expand/Contract Pattern

A database migration approach where schema changes are made in two phases: expand (add new structure alongside old) while both versions run, then contract (remove old structure) after the new version is stable and the old is no longer referenced.

Forward-Only Migration

A database migration philosophy where rollback scripts are not written for destructive changes — instead, migrations are designed so old and new code versions can both run against the migrated schema.

Blue/Green Deployment

Maintaining two identical production environments, with traffic switched between them. Rollback is instant — return traffic to the previous environment — but requires running two environments simultaneously.

Feature Flag

A runtime configuration that enables or disables a feature without deploying code. Separates deployment (code is in production) from release (users can access the feature), enabling instant rollback by disabling the flag.

Automatic Rollback Threshold

Pre-defined metric thresholds (error rate, latency, business metrics) that trigger an automated rollback without human intervention, typically targeting rollback within 5 minutes of anomaly detection.

When to use it

• Designing a deployment pipeline that can roll back within 5 minutes of detecting a regression
• Explaining the expand/contract database migration pattern in system design interviews
• Choosing between blue/green, canary, and rolling deployment strategies for different risk profiles

Interview questions

• You need to rename a column in a high-traffic PostgreSQL table from user_name to username. How do you do this safely with zero downtime and the ability to roll back?

More Troubleshooting topics

• Connection Pool Exhausted
• CoreDNS Failures in Kubernetes
• CPU Runaway Process Investigation
• CrashLoopBackOff
• Database Deadlock Diagnosis
• Database Migrations in Production
• Disk IOPS Saturation
• DNS Failures
• EC2 Instance Unresponsive
• ECS Task Failures and Restarts
• ELB 5xx Errors
• Flaky CI/CD Pipeline Failures