Load Balancing
How a load balancer spreads traffic, hides failures and stays up itself: layer 4 and layer 7, the algorithms, health checks, draining, affinity, TLS and long-lived connections.
Overview
A load balancer sits in front of a group of servers and decides which server handles each connection or request. Clients see one address. Behind it you can add servers, remove them, or lose one, and clients do not notice.
It does four jobs. It spreads work so that no server is swamped while others sit idle. It checks which servers are healthy and stops sending traffic to the ones that are not. It lets you take servers out one at a time, for a deploy or a scale-in. And it is a natural place for shared work, such as decrypting HTTPS.
Fundamentals introduced load balancers in a paragraph. This chapter goes deeper: what a balancer can see at each network layer, which algorithm to choose and why, how it finds out a server is sick, how to add and remove servers without dropping requests, why long-lived connections defeat simple balancing, and how to stop the load balancer itself from becoming the single point of failure.
What you will be able to do
- Choose between a layer 4 and a layer 7 load balancer, and say what each can and cannot see.
- Pick an algorithm for a workload, and explain why round robin is often not enough.
- Set up health checks, draining and slow start so that a deploy drops no requests.
- Explain why gRPC and WebSocket traffic ends up unbalanced, and how to fix it.
- Design a load balancing tier with no single point of failure.
Key concepts
- Layer 4 and layer 7
- Round robin and least connections
- Power of two choices
- Consistent hashing
- Health checks and outlier detection
- Draining and slow start
- Sticky sessions
- TLS termination
- Long-lived connections
- Load balancer high availability
Interview questions
- What is the difference between layer 4 and layer 7 load balancing, and when do you use each?
- Which load balancing algorithm would you choose, and why is round robin often not enough?
- What is the power of two random choices, and why use it instead of always picking the least loaded server?
- How do you deploy a new version behind a load balancer without dropping requests?
- Your gRPC service has 10 pods, but one pod handles most of the traffic. Why, and how do you fix it?
- How do you keep the load balancer from being a single point of failure?
- Health checks pass, but users get errors from one server. What is going on, and what would you change?
- Should you use sticky sessions?
- Where should TLS be terminated?
Sign up free to read the full topic
More System Design topics
Start free on Cariara · Coding problems · System design studio