The Cascade That Took Down Three Services

A single slow dependency turned synchronous timeouts into a process-pool meltdown that propagated upstream.

Overview

A payment processor slowed down, and four unrelated services went down with it because every one of them waited too long for it.

What happened

At 2:14 AM on a Tuesday, the on-call engineer was paged: the checkout API was returning 503s. Within eight minutes, four services were down. Their only link was a shared downstream dependency on a payment processor that had started responding slowly.

How it spread

The slow processor was the trigger, not the root cause. Every service had a thread pool sized for peak traffic, and every thread was allowed to wait up to 30 seconds for the processor. When the processor slowed to 8-second responses, threads stopped coming back to the pool faster than requests arrived.

> [!WARNING] The retries were the amplifier. Callers that timed out retried at once, adding load to services that had no free threads left.

Impact

Incident length — 47 minutes
Users affected — 18,000
Services down — 4
Processor degraded for — 11 minutes
Extra time to drain retries and stabilize — 36 minutes

The fix

The processor recovered on its own after 11 minutes. The design had to change so that the next slow dependency stays contained in the one pool that calls it.

Key concepts

Cascading failure: A failure in one component that causes failures in dependent components, typically because callers exhaust their own capacity waiting for a slow downstream.
Thread pool exhaustion: When all available worker threads are blocked waiting for a response, no new requests can be processed, causing the service itself to appear down.
Retry storm: Clients that receive errors immediately retry, multiplying load on an already-degraded service and preventing recovery.
Circuit breaker: A pattern that detects consecutive failures to a dependency and "opens" the circuit — fast-failing calls to that dependency for a cooling-off period — to protect the caller.
Bulkhead pattern: Isolating different downstream calls into separate thread pools so that one slow dependency cannot exhaust the entire process.

When to use it

Explaining cascading failures in system design interviews
Justifying timeout, circuit breaker, and bulkhead decisions
Discussing why retry storms amplify rather than resolve outages
Answering "How do you design for partial failure?"

Interview questions

You are on-call. The checkout service is down, and three services that depend on it are also timing out. How do you diagnose and recover?
Why are long timeouts (like 30 seconds) dangerous in synchronous service-to-service calls?