Circuit Breakers

The three-state machine, why fail-fast beats hanging, and how circuit breakers prevent cascading failures.

Overview

Why circuit breakers exist

The circuit breaker is a pattern from Michael Nygard's *Release It!* (2007). It became canonical through Netflix Hystrix and now ships in service meshes such as Istio and Linkerd.

The principle: when a downstream dependency is failing, stop calling it for a while. The failing dependency gets a chance to recover, and your service stops wasting threads and CPU on doomed calls.

> [!IMPORTANT] An open breaker protects both sides: your service keeps its threads, and the sick dependency gets room to recover.

The three states

CLOSED (normal) — Requests pass through; failure rate (errors plus timeouts) is tracked — Trips to OPEN when failures exceed the threshold, such as 50% over 1 minute
OPEN (failing) — Requests fail fast with an error or fallback; the dependency is not called — Moves to HALF-OPEN after a timeout, such as 30 s
HALF-OPEN (probing) — A few requests go through to test the dependency — Success returns to CLOSED; failure returns to OPEN

Why fail-fast beats hanging

Each request waits for the timeout, often 5-30 s — Failure rate over 50% trips the breaker to OPEN
Threads and connections stay allocated — Subsequent requests fast-fail in under 1 ms
Under load, resources run out within minutes — Threads and connections are not consumed waiting
Your service goes down too — Users get a cached or fallback response instead of a hang

Implementations

Netflix Hystrix — Java — The original; now in maintenance mode, and Netflix recommends resilience4j
resilience4j — Java — Modern, lighter, functional; integrates with Spring and Reactor
Polly — .NET — The .NET equivalent
Istio — Service mesh — outlierDetection on a DestinationRule sets failure threshold and ejection time
Linkerd — Service mesh — Failure-accrual circuit breaking configured with annotations

No code change — Less granular than per-call breakers in code
Per-service configuration — Only HTTP and gRPC, not in-process calls
Observable

Configuration parameters

Failure threshold — % of requests that must fail to trip — 50% over a 1-minute window
Minimum requests — Volume needed before tripping, so 1 of 1 failing does not trip — 20+ requests
Open timeout — Time in OPEN before HALF-OPEN — 30-60 s
Half-open requests — Probes allowed in HALF-OPEN — 1-5

Common mistakes

No fallback when OPEN — A 5xx beats a hang, but a cached response, default value or partial result is better
Per-instance breakers — Each instance trips on its own; usually fine, since all eventually trip
Wrong thresholds — Too low and the breaker oscillates; too high and cascades are not prevented
No observability — Emit state changes as metrics and alert on OPEN and HALF-OPEN transitions

The pattern in practice

Wrap every external call: database, cache, downstream HTTP or gRPC service, queue, third-party API.

Key concepts

Three-state machine: CLOSED (normal) → OPEN (failing fast-fail) → HALF-OPEN (probing) → CLOSED. Standard since Hystrix.
Fail-fast: When OPEN, return error/fallback in <1ms instead of hanging on timeout. Saves threads, prevents cascading failure.
resilience4j: Modern Java circuit-breaker library; replacement for Hystrix (which is in maintenance mode).
Istio outlierDetection: Service-mesh-level circuit breaker. No app code changes; configured per DestinationRule.
Failure threshold + min requests: Trip on 50% failure over 1m, min 20 requests. Both required to avoid false-positive trips.
Fallback: What to return when breaker is OPEN. Cached response, default value, partial result. Better than 5xx.

When to use it

Every external dependency call — DB, cache, HTTP/gRPC service, third-party API
Service mesh deployments — enable per-service circuit breaker policies
Reviewing a postmortem with cascading failure — was a circuit breaker missing?
Designing a new microservice — circuit breakers around every external call by default

Interview questions

Walk me through the circuit breaker state machine.
How does Istio's circuit breaker compare to Hystrix?
When does a circuit breaker hurt instead of help?