Single-Region Dependency That Caused a Global Outage

A globally deployed service collapsed because its auth token endpoint was pinned to a single AWS region.

Overview

A service running in four AWS regions failed in all four at once because every region checked tokens against one endpoint in us-east-1.

What happened

The service advertised 99.99% uptime in its SLA, and customers paid for multi-region resilience. One night us-east-1 had elevated API error rates for about 40 minutes. Compute in all four regions kept running, yet authenticated requests began failing everywhere.

Why it broke

Every region's auth middleware validated JWTs by calling a shared introspection endpoint in us-east-1. That endpoint sat behind an Application Load Balancer that depended on IAM calls, which were degraded during the event.

> [!IMPORTANT] A soft dependency called inline on every request turns into a hard one under failure. Here it cancelled all the geographic redundancy.

The fix

Each region has to be able to authenticate a request without leaving the region.

Online introspection call in us-east-1 per request — Local JWT validation with a cached public key in each region
No cache — Last-known-valid result cached for a short grace period
No circuit breaker — Breaker that stops waiting on a slow endpoint
Cross-region call on the inline path — Regional endpoint first, cross-region only as a fallback

Key concepts

Soft dependency: A dependency that is not required for the happy path but is called on every request — auth, logging, feature flags — that can silently become a hard dependency under failure.
Data plane vs control plane: Data plane handles actual traffic (serving requests); control plane handles management (auth, config, orchestration). Outages hit data planes hardest when control-plane calls are inline.
Token caching: Storing validated JWT results in a short-lived local cache so that auth introspection calls are not made per-request.
Regional isolation: Each region should be able to operate independently without calling cross-region endpoints in the request path.

When to use it

Illustrating why multi-region architecture requires dependency mapping, not just compute placement
Designing auth and control-plane services in interviews
Discussing the difference between data-plane and control-plane resilience

Interview questions

How would you redesign the auth system to eliminate the single-region dependency?
How do you audit an existing system for hidden single-region or single-AZ dependencies?