Load Balancing Algorithms

Round-robin, weighted, least-connections, IP hash, random, power-of-two-choices, and consistent hashing with virtual nodes.

Overview

Overview Load balancing algorithms determine how a load balancer distributes incoming requests across a pool of backend servers. Choosing the wrong algorithm for your traffic pattern causes hot spots, wasted capacity, or session loss.

Round-robin is the simplest: requests cycle through servers in order. It works well when all servers are homogeneous and request cost is uniform. Weighted round-robin assigns a weight to each server proportional to its capacity, so a server with weight 3 receives three requests for every one sent to a server with weight 1.

Least-connections routes each new request to the server with the fewest active connections at that moment. This naturally adapts to variable-cost requests. Weighted least-connections combines both approaches.

IP hash computes a hash of the client IP address and always routes that client to the same server. This provides basic session stickiness without cookies but breaks if the client changes IP (NAT, mobile, VPN) or if a server is added/removed and the hash modulo changes.

Power-of-two choices (P2C) is used in Nginx Plus and Envoy: pick two servers at random, route to the one with fewer active requests. This achieves near-optimal load distribution with O(1) per-request overhead.

Consistent hashing with virtual nodes is used in distributed caches (Memcached, Redis Cluster). Each server is mapped to multiple points on a hash ring. A request key hashes to a position on the ring and is routed to the nearest server clockwise. Adding or removing a server only remaps a fraction of keys. Virtual nodes (vnodes) improve key distribution uniformity by giving each physical server multiple ring positions.

Key concepts

Round-Robin
Requests are distributed to backends in sequential order. Simple and fair for homogeneous workloads with uniform request cost. No server state is tracked.
Least Connections
New requests are sent to the backend with the fewest active connections. Adapts to heterogeneous request cost — slow requests accumulate connections and the server receives fewer new ones.
Consistent Hashing
Servers are mapped to positions on a hash ring. A key hashes to a ring position and is routed to the nearest server. Adding or removing a server only remaps 1/N of keys, minimizing cache misses on topology changes.
Virtual Nodes (Vnodes)
Each physical server is assigned multiple positions on the hash ring. This improves key distribution uniformity and means that when a server leaves the ring, its load is spread across all remaining servers.
Power-of-Two Choices (P2C)
Pick two candidate backends at random; route to the one with fewer active requests. Achieves near-optimal load distribution in O(1) without tracking global state. Used in Envoy and Nginx Plus.
IP Hash
The client source IP is hashed to determine the backend. Provides basic session affinity without cookies but is fragile under NAT and breaks when backends are added or removed.

When to use it

  • Selecting a load balancing algorithm for a new service based on traffic characteristics
  • Explaining why consistent hashing is used in distributed caches instead of modulo hashing
  • Diagnosing hot spots where one backend server is overloaded while others are idle
  • Choosing between ALB (round-robin / least-outstanding) and custom HAProxy algorithms

Interview questions

  • Explain consistent hashing and why it is preferred over modulo hashing for distributed caches.
  • When would you choose power-of-two choices over least-connections?
  • How does nginx implement load balancing, and how do you configure weighted least-connections?

Sign up free to read the full topic

More Networking topics

Start free on Cariara · Coding problems · System design studio