Cascading Failure

The four canonical cascading-failure patterns from the SRE Book and how to prevent each.

Overview

What a cascading failure is

Cascading failure is the deepest material in SRE Book Chapter 22. One component fails or slows, and a series of compounding effects brings down the whole system.

Every SRE has seen one, and many architectures do not survive their first.

> [!IMPORTANT] Cascading failures are positive-feedback loops. Fixing the trigger is not enough; you must break the loop.

The four canonical patterns

1. Server overload — Capacity falls short, latency rises, threads and connections exhaust, health checks fail, the load balancer removes nodes, and the survivors get more load until they drop out too
2. Resource exhaustion — One resource (memory, CPU, file descriptors, connections) is constrained; GC pressure, thread starvation and pool exhaustion each amplify it until collapse
3. Service unavailability — A dependency dies, callers hang with no timeout or breaker, their threads exhaust, and the outage propagates up the stack
4. Retry amplification — Failed requests are retried by every client, turning a small failure into 2-5x load on a system already failing

The prevention stack

Each layer prevents specific failure modes.

Circuit breakers — Service-unavailability propagation
Bulkheads — One bad dependency exhausting all your threads
Timeouts — Hung calls accumulating
Bounded retries with jitter — Retry amplification
Load shedding — Collapse, by dropping low-priority requests first
Graceful degradation — Hard failures, by serving degraded results
Capacity planning and autoscaling — Under-provisioning for the failure-mode peak

The GC death spiral

In Java systems, insufficient CPU triggers more GC, which uses more CPU and leaves less for real work.

Prevention — Heap headroom, GC tuning, bounded queue depths

The retry storm and the thundering herd

SRE Book Chapter 22: "Retries can amplify load, particularly during cascading failures." When a service is degraded, automatic retries double or triple its load exactly when it can least handle it.

Bounded retries, max 2-3 attempts — Caps the multiplier
Exponential backoff — Each retry waits longer
Jitter — A random component prevents synchronized retries
Idempotent requests — Retries are safe

The thundering herd: 10,000 clients retry at the same instant after a brief outage. A 5-second outage becomes a retry storm at second 6 that takes another minute to subside.

Jittered exponential — wait = base * 2^attempt * (0.5 + random())
AWS Full Jitter — wait = random(0, base * 2^attempt)

Metastable failures

A 2021 paper by Bronson et al. introduced metastable failures: systems with two stable states. A small perturbation moves them from healthy to degraded, and they do not recover without intervention.

Healthy — Low — Low — Spare capacity
Degraded — High — High — Retries amplifying load

> [!TIP] Metastability is cascading failure formalized: the trigger can disappear while the degraded state persists.

Key concepts

Server overload spiral: Capacity insufficient → latency up → nodes drop out → remaining nodes overloaded → more drop out → collapse.
Resource exhaustion: One resource (memory, threads, connections) constrained; subsidiary effects amplify.
Service unavailability propagation: Dependency dies; caller's threads exhaust waiting; caller dies; propagates up the stack.
Retry amplification: Client retries multiply load on a failing service. Without jitter and backoff, 2-5× load multiplier.
GC death spiral: Insufficient CPU → more GC → more CPU consumed → less for work → more queue → more memory pressure → more GC.
Metastable failure: System has two stable states; small perturbation moves it from healthy to degraded; doesn't auto-recover.

When to use it

Postmortem analysis — was a cascading failure pattern present? Which one?
Architecture review — does the design have all the prevention layers?
Pre-launch — load test specifically for cascading-failure scenarios
Incident response — when responding to a cascade, identify which pattern is active

Interview questions

Walk me through a real cascading-failure scenario.
How do you recover from a live cascading failure?
What's the GC death spiral and how do you prevent it?