Deployment System

Progressive delivery at scale

Design the system that takes a built version of a service to production with no downtime, rolling it out in steps (rolling, canary, blue-green) and rolling it back on its own when metrics get worse.

Overview

A deployment system moves a new version of a service from 'built' to 'serving all traffic'. The build is done: an image sits in a registry. What is left is the risky part: replacing thousands of running instances of a live service without dropping requests, and finding out quickly, from real traffic, whether the new version is worse.

Most outages start with a change, so the whole design is about limiting how much damage one change can do and how long it lasts. A rollout exposes the new version in steps; at each step the system compares it with the current version and either widens it or undoes it. Rolling back must be fast, automatic and always possible, which puts rules on how database changes ship.

Questions to ask first

How many services and deploys? — 500 services, 2,000 deploys a day, most in working hours
Where do services run? — Containers on Kubernetes clusters in 3 regions
Which strategies? — Rolling by default, canary for important services, blue-green where needed
Can a rollout undo itself? — Yes, on failed health checks or worse metrics, without a person
Is downtime allowed? — No: no dropped requests during a rollout
Who can deploy? — Service owners; production needs a passing build and an approved change

What you will be able to do

Explain rolling, canary and blue-green rollouts and when each fits.
Drain instances so a rollout drops no requests.
Decide from metrics whether a canary is worse, including when there is too little data.
Make rollback safe by shipping database changes in expand and contract steps.
Limit blast radius with steps, regional waves and one rollout at a time.

Key concepts

Desired state and reconcile
Rolling update
Readiness and draining
Canary
Traffic splitting
Canary analysis
Blue-green
Expand and contract
Deployment waves
Last known good

Functional requirements

Deploy an image version of a service to one environment and several regions
Support rolling, canary and blue-green strategies, chosen per service
Pause between steps, analyze metrics, then promote or abort automatically
Roll back to the last known good version with one call
Allow only one rollout per service and environment at a time
Record every deployment, step and decision for audit

Non-functional requirements

No dropped requests during a rollout (a target we chose)
Rollback of a canary takes under 60 seconds; traffic moves first, instances later (a target we chose)
A bad canary is caught within about 10 minutes for a service with normal traffic
If the deployment system is down, running services are unaffected
Every change to production is traceable to an image, a commit and a person

Interview questions

Rolling, canary or blue-green: which would you use?
How does automated canary analysis decide?
How do you guarantee no dropped requests during a rollout?
How fast can you roll back, and what can stop you?
How do you roll out across regions?
How do you stop two deploys of the same service from colliding?
Your canary analysis passed, but users are complaining. What went wrong?