Health Checks and Circuit Breaking
Active vs passive health checks, HTTP 200 probes, TCP checks, intervals and thresholds, and circuit breaker patterns.
Overview
Overview Health checks are the mechanism by which a load balancer determines whether a backend is capable of serving traffic. Without them, requests are sent to dead or degraded backends, resulting in timeouts and errors that the client experiences directly.
Active health checks are periodic requests the load balancer sends to each backend independently of real traffic. A typical HTTP active check sends a GET to /health every 10 seconds. If the backend returns 200 within a timeout window, it is considered healthy. After a configurable number of consecutive failures (the unhealthy threshold — typically 2 or 3), the backend is removed from the rotation. After a number of consecutive successes (the healthy threshold — typically 2), it is reinstated. TCP active checks simply test that the port accepts a connection, which is faster but verifies less.
Passive health checks (circuit breakers) infer health from real traffic. The load balancer or proxy watches response codes and latency on production requests. If a backend returns 5xx errors above a threshold, it is ejected. Envoy's outlier detection is the canonical implementation: if a host returns 5 consecutive 5xx responses, it is ejected for a base ejection time (e.g., 30 seconds), exponentially backing off on repeated ejections.
The circuit breaker pattern has three states: Closed (normal, requests flow through), Open (backend is unhealthy, requests are short-circuited with an error), and Half-Open (a probe request is allowed through to test recovery). Failing fast with a circuit-breaker error is better than letting requests queue up against a dead backend, exhausting thread pools and propagating the failure upstream.
Combining active and passive checks is best practice: active checks catch total failures quickly; passive outlier detection catches partial degradation on a subset of requests.
Key concepts
- Active Health Check
- A synthetic probe the load balancer sends to each backend on a fixed interval, independent of real traffic. Detects total outages quickly. Requires a dedicated health endpoint that tests dependencies.
- Passive Health Check (Outlier Detection)
- Infers health from production traffic. The proxy watches error rates and latency. If a backend exceeds thresholds, it is ejected from the pool. Catches partial degradation that active checks miss.
- Unhealthy / Healthy Threshold
- Unhealthy threshold: consecutive failures before a backend is removed (e.g., 3). Healthy threshold: consecutive successes before a backend is reinstated (e.g., 2). Prevents flapping.
- Circuit Breaker States
- Closed: normal operation. Open: backend ejected, requests fail fast. Half-Open: one probe request is allowed to test recovery. Transitions are governed by thresholds and timeouts.
- Liveness vs Readiness (Kubernetes)
- Liveness probe: is the container alive? Failing it causes a restart. Readiness probe: is the container ready to serve traffic? Failing it removes the pod from the Service endpoints.
When to use it
- Configuring health checks on an AWS ALB target group for an ECS or EC2 service
- Implementing circuit breaking in Envoy or Istio service mesh to prevent cascading failures
- Tuning health check intervals and thresholds to balance fast failure detection against false positives
- Designing a /health endpoint that accurately reflects application readiness
- Explaining the difference between liveness and readiness probes in Kubernetes
Interview questions
- How do you configure health checks on an AWS ALB target group, and what makes a good /health endpoint?
- Explain the circuit breaker pattern and how Envoy implements outlier detection.
Sign up free to read the full topic
More Networking topics
Start free on Cariara · Coding problems · System design studio