Game Days & Chaos Engineering

Practicing failure deliberately. Google's DiRT, Netflix Chaos Monkey, modern chaos engineering tools.

Overview

Practice failure on purpose

Practicing failure separates "we hope it works" from "we know it works." Three related practices do this at different scales.

Game Days — Scheduled full-team incident simulations: page on-call, run command structure, mitigate, postmortem — Amazon, around 2003; Google formalized it as DiRT
Chaos Engineering — Continuous, targeted failure injection in production or production-like environments; now covers network failures, latency, region outages, dependency failures — Netflix Chaos Monkey (2010): random instance termination during weekday business hours, so engineers build for "instances die at any time"
DiRT (Disaster Recovery Testing) — Google's annual company-wide week: power outages, network partitions, regional failures, leadership-loss scenarios — Documented in the SRE Book and Workbook; finds gaps monthly game days miss

Why practice deliberately

Untested failover doesn't work — The first DR-region failover always finds something misconfigured (IAM, secrets, DNS); better found in a controlled setting
Knowledge transfer — New on-call engineers experience incident response without customer impact
Process gaps surface — "Where's the contact list?" "Who can approve a rollback?" "How do I update the status page?"
Hidden dependencies — "We depend on service X but didn't know it," usually through legacy code calling it directly

Principles of Chaos Engineering (2017 manifesto)

1 — Build a hypothesis around steady-state behavior — "Under normal load, p99 latency is [!IMPORTANT] The hardest step is buy-in for injecting failure into production. Untested failure paths break worse, with worse customer impact: inject controlled failure now to avoid uncontrolled failure later.

> [!TIP] Successful programs start with staging-only chaos, graduate to prod with a limited blast radius (single instance, single AZ), then extend to larger experiments such as region drains.

Key concepts

Game Day: Scheduled team-wide incident simulation. Realistic failure scenario; respond as if real; postmortem.
Chaos Engineering: Continuous, automated failure injection in production. Netflix Chaos Monkey origin (2010).
DiRT: Google's annual Disaster Recovery Testing week. Company-wide simulated disasters.
Steady-state hypothesis: Define normal behavior before chaos experiment. "p99 < 500ms under normal load."
Minimize blast radius: Chaos experiments scope tight initially; expand only after success. Kill switch always available.
White team: Game day observers who watch the response without interfering. Take notes for the debrief.

When to use it

Pre-launch — game day for the new service before it carries real traffic
After major architectural changes — verify failure modes still work
Quarterly cadence — game days as a recurring practice
Team training — new on-call engineers run a game day before going on rotation

Interview questions

How would you run a game day for a new service?
How do you run chaos engineering safely in production?
What's the difference between game days and chaos engineering?