A Deployment That Could Not Be Rolled Back

Overview

A deploy that shipped an API response change and a column rename together broke old mobile clients, and the rename made rolling back impossible.

What happened

The bundle held two changes: a new response shape for /api/v1/orders and a migration renaming orders.status to orders.fulfillment_status. Both passed in staging together.

Why it broke

The code and the schema changed in the same deploy, so each one depended on the other. Neither direction of rollback was safe.

Impact

• Instances deployed — 8
• Time to page — Within 3 minutes
• Error rate on /api/v1/orders — 34%
• Users on the old mobile app — 12%
• Writes a schema rollback would have undone — 2 hours

The fix

The team shipped a new version that worked with both column names, using a database view as a compatibility shim.

• Column names the code works with — One (orders.status or orders.fulfillment_status) — Both, through the view
• Recovery path — Rollback, which was blocked — Forward-fix

> [!WARNING] Passing in staging proves the new code and new schema work together. It says nothing about old code against the new schema, which is exactly what a rollback runs.

Key concepts

Expand-contract migration

A two-phase approach: first add the new column/table (expand), migrate data and update code to write both, then remove the old column/table (contract) after all code is updated.

Backward-compatible migration

A database schema change that both the old and new versions of the application can use simultaneously, enabling safe rollback of application code.

Blue-green deployment

Running two identical environments (blue and green) and switching traffic between them — requires that both versions can use the same database schema.

Database view

A virtual table that presents a query result as a table — can be used as a compatibility shim to present old column names while the underlying table uses new names.

When to use it

• Explaining why database migrations and application deployments must be decoupled
• Discussing backward-compatible database schema changes
• Designing deployment strategies that preserve rollback ability

Interview questions

• What is the expand-contract pattern and how does it make database deployments safe to roll back?

More War Stories topics

• A Failed Migration That Left the Database in a Partial State
• A Feature Flag Change That Triggered a Production Outage
• Accidental DELETE Without a WHERE Clause in Production
• An IAM Misconfiguration That Let a Developer Become Admin
• AWS Credentials Committed to a Public GitHub Repository
• Backups That Were Never Actually Completing
• BGP Route Leak That Rerouted Traffic Through an Unexpected AS
• Blue-Green Deployment That Resulted in Split-Brain Sessions
• Cache Stampede on a Flash-Sale Product Page
• Connection Pool Exhaustion That Brought Down 12 Microservices
• DynamoDB Hot Partition That Throttled a Flash Sale
• Failover to a Lagging Replica Lost 4 Minutes of Orders