Bulkheads

Per-dependency thread pools, the ship-bulkhead analogy, and how Hystrix popularized the pattern.

Overview

The ship analogy

Bulkheads are an isolation pattern from Michael Nygard's *Release It!* (2007). A ship's hull is divided into watertight compartments, so a hole in one does not flood the whole ship.

In software, you isolate resource pools so a failure in one cannot exhaust the whole service's capacity.

> [!IMPORTANT] With bulkheads, one sick dependency degrades one feature instead of taking the whole service down.

Without and with bulkheads

Pools — One global pool of 100 threads — 30 for B, 30 for C, 30 for D, 10 reserve
B slows from 50ms to 5s — All 100 threads wait on B — Only the 30 B-pool threads block
Calls to C and D — Also fail — Succeed at full capacity
Service A — Fully unavailable — Partially degraded: only B features down

Thread vs semaphore isolation

Netflix Hystrix (2012) made the pattern famous with two implementations. Hystrix is now in maintenance mode, and resilience4j and service meshes implement the same patterns.

How — Separate thread pool per dependency — Same threads, counted concurrent permits per dependency
Pros — Full isolation; one slow dependency cannot block others — Lightweight, no thread overhead
Cons — 1-2 MB stack per thread; more context switching — Does not isolate CPU contention or blocking on other resources
Use for — Slow or blocking dependencies: DB, external APIs, anything that might hang — Fast in-memory or trusted-fast operations

The connection-pool bulkhead

A slow query now consumes only the user-request pool while admin and batch continue. The cost is memory: each pool carries idle connections and monitoring threads, so keep to a practical limit of 5-10 pools per service.

Bulkheads in a service mesh

Istio — maxConnections per destination plus maxRequestsPerConnection
Linkerd — Similar, via service profiles

Per-destination concurrency caps in both act as bulkheads.

The discipline

1. Allocate — A dedicated pool for each dependency that can plausibly fail or slow down
2. Size — Just enough for normal throughput plus a buffer
3. Monitor — Track saturation per pool; alert when a specific pool is exhausted

One global pool for everything — The canonical anti-pattern: one bad actor blocks everyone
Per-pool configuration drift — Inconsistent timeouts and limits are hard to reason about
Over-bulkheading — Too many tiny pools reduce total capacity; 5-10 is enough

Combined with circuit breakers

> [!TIP] Bulkhead plus circuit breaker is the standard reliability stack: the bad dependency cannot cascade, and its pool is freed quickly.

Key concepts

Bulkhead pattern: Per-dependency resource pool. One bad dependency can't exhaust the whole service. From Nygard's Release It!.
Thread isolation: Separate thread pool per dependency. Strongest isolation; thread overhead.
Semaphore isolation: Same threads, counted permits per dependency. Lightweight; less full isolation.
Connection-pool bulkheads: Separate DB pools per workload (user, admin, batch). Slow query in one pool doesn't starve others.
Hystrix legacy: Popularized bulkhead pattern in microservices. resilience4j is modern replacement.
Combined with circuit breaker: Bulkhead isolates; breaker stops calling. Together they prevent cascade and recover quickly.

When to use it

Designing a service with multiple downstream dependencies
Reviewing for cascading-failure resilience — does one bad dep take down the whole service?
DB connection pool design — separate pools per workload type
Service mesh policy — per-destination concurrency caps

Interview questions

How do bulkheads differ from circuit breakers?
Walk me through configuring a bulkhead for a database connection pool.
When are bulkheads NOT worth it?