Chaos Engineering

Deliberately injecting failures into production-like systems to uncover weaknesses before they cause outages.

Overview

Break it on purpose

Chaos Engineering is the practice of deliberately introducing failures into a system to build confidence in its ability to withstand turbulent conditions. Netflix pioneered it with Chaos Monkey in 2011, and it has become a rigorous experimental discipline. You find weaknesses before incidents find them first.

Why it matters

Distributed systems fail in unexpected ways. You cannot reason about resilience from architecture diagrams alone; you must observe the system under stress.

> [!TIP] A 30-minute planned game day that kills a pod is far better than an unplanned 3 AM outage that kills revenue.

The scientific method

1 — Define the steady-state metric: requests per second, error rate, p99 latency
2 — Hypothesize that the steady state continues during the failure
3 — Inject the failure: kill a pod, add network latency, exhaust memory
4 — Observe whether the hypothesis holds
5 — Improve the system if it does not

> [!IMPORTANT] Always define the abort condition before starting.

Blast radius control

> [!WARNING] Never inject failures into a degraded system: chaos runs on healthy systems to find latent weaknesses, not to compound existing problems.

Failure primitives and tools

Process kills — Pod or VM termination
Network failures — Latency, packet loss, blackholes, DNS failures
Resource exhaustion — CPU saturation, memory pressure, disk fill
Clock skew — Nodes disagree on time
Dependency outages — External API unavailable

Tools that automate injection at different layers: Chaos Monkey, Litmus Chaos, Chaos Mesh, Gremlin and AWS Fault Injection Simulator.

GameDays

GameDays are structured team exercises. An SRE or chaos team injects a realistic failure scenario ("one AZ goes dark") while the on-call team responds as if it were a real incident.

Muscle memory — Responders have done it before
Runbook gaps surfaced — Fixed before a real event
Better alert quality — Missing or noisy alerts show up

Key concepts

Steady-state hypothesis: A measurable, quantified description of normal system behavior (e.g., "error rate < 0.1%, p99 < 200ms"). A chaos experiment validates that this hypothesis holds during the injected failure.
Blast radius: The scope of impact of a chaos experiment. Start small (one pod, one AZ) and expand only after confirming the system handles the smaller failure correctly.
Chaos Monkey: Netflix open-source tool that randomly terminates VM instances in production during business hours, forcing engineers to design for instance failure from day one.
Litmus Chaos: CNCF-incubating Kubernetes-native chaos framework. Provides ChaosExperiment and ChaosEngine CRDs. Runs experiments as Kubernetes Jobs with built-in observability.
GameDay: A structured, pre-announced failure injection exercise where a team deliberately breaks a system while on-call engineers respond, revealing runbook gaps and alert quality issues.
Fault injection: Programmatically introducing a specific failure (latency, packet loss, error code) into a request path. Service meshes (Istio, Linkerd) support HTTP fault injection natively via traffic management rules.

When to use it

Validating that circuit breakers, retries, and fallbacks work as designed under real failure
Preparing for a major product launch by stress-testing dependencies
Building confidence before migrating a stateful system to a new region
Improving on-call readiness by running GameDay exercises with realistic failure scenarios
Interview questions about how you ensure high availability beyond architecture diagrams

Interview questions

How would you design a chaos engineering program from scratch at a company with no existing practice?
What is the difference between chaos engineering and load testing?