Heartbeats and Failure Detection
How distributed systems decide that a node has failed: periodic heartbeats, timeouts and their trade-off, phi accrual detectors, SWIM-style gossip, and leases with fencing so a wrongly condemned node does no harm.
Overview
A heartbeat is a small message a node sends at a fixed interval to say 'I am still here'. A failure detector listens for those messages and, after enough silence, decides the node has failed. That decision starts recovery: electing a new leader, moving partitions to other consumers, re-creating lost replicas, or taking a server out of a load balancer.
The idea is simple; the difficulty is that silence is ambiguous. A crashed node, a node paused for garbage collection, an overloaded node and a broken network link all look exactly the same from the outside. So every failure detector trades detection time (how long a real failure goes unnoticed) against false positives (healthy nodes declared dead). Good designs accept that the detector will sometimes be wrong and make wrong decisions safe.
Heartbeats sit inside most systems you will draw in an interview: Raft leaders, ZooKeeper sessions, Kafka consumer groups, Cassandra gossip, Kubernetes node status, load balancer health checks.
What you will be able to do
- Explain why timeouts can only guess, and choose an interval and a timeout with numbers.
- Describe how a phi accrual detector adapts to the network, and compute phi.
- Explain how gossip protocols such as SWIM detect failures in large clusters without quadratic traffic.
- Use leases and fencing tokens so a node wrongly declared dead cannot corrupt data.
- Design what happens after a failure is detected without causing a cascade.
Key concepts
- Heartbeat and timeout
- Push vs pull
- False positives vs detection time
- Phi accrual detector
- Gossip and SWIM
- Suspicion before declaring
- Leases
- Fencing tokens
- Randomized election timeouts
Interview questions
- How do you pick heartbeat interval and timeout values?
- How does a phi accrual failure detector work?
- How do you detect failures in a cluster of thousands of nodes?
- A leader is declared dead but was only paused. How do you prevent split brain?
- What is the difference between push heartbeats and pull health checks, and when do you use each?
- How can heartbeat-based failover cause a cascading failure, and how do you prevent it?
- Why does Raft randomize election timeouts?
Sign up free to read the full topic
More System Design topics
Start free on Cariara · Coding problems · System design studio