Cascading Failure at 3am

Overview

The hardest incident class

Cascading failures are the hardest incident class because the symptoms mislead. Several services are degraded at once, and fixing the wrong one first makes things worse.

Identifying which failure pattern you are in determines the correct first action.

> [!IMPORTANT] The first service to alert is often not the origin. Cascades propagate upstream from the source.

The patterns

SRE Book Chapter 22 covers server overload, resource exhaustion and retry storms. Metastable failure is the later framing from Bronson et al. (2021).

• Server overload — Latency rises as load exceeds capacity — Shed load
• Resource exhaustion — Error rate climbs monotonically as one service dies — Restore or restart the exhausted resource
• Retry storm — Error rate oscillates as services recover and get hammered again — Fix retry backoff, shed load at the caller
• Metastable failure — Degraded state persists after the trigger is gone — Break the feedback loop

Interview questions

• At 3am, your monitoring fires alerts on 6 services simultaneously. All show high latency and elevated error rates. What do you do in the first 10 minutes?
• Five minutes in, you find that Service A (a payment processor) is receiving 10x normal traffic from Service B (checkout). Service B has no recent deploy. What is the most likely cause and what do you do?
• The cascade is resolved. What does the postmortem need to cover to prevent recurrence?

More SRE topics

• Blameless Postmortems
• Bulkheads
• Canary Analysis
• Capacity Planning
• Cardinality & Observability Cost
• Cascading Failure
• Choosing the Right SLI
• CI/CD & Progressive Delivery
• Circuit Breakers
• Configuration Management
• Cost vs Reliability
• Critical User Journeys