A load balancer’s job sounds simple: spread requests across a pool of servers so none is overloaded. Round robin does that if every request costs the same and every server is equally fast. Neither is true in production. A search request that scans a large index costs twenty times a health-check ping, and one server always ends up on a noisy host or in the middle of a garbage collection pause.
This guide covers what actually decides whether load is balanced: the algorithm’s response to uneven costs, how health checks decide who’s in the pool, and why long-lived connections quietly defeat balancing altogether.
Equal counts, unequal load
Picture two API servers behind round robin, each getting 500 requests per second. Server A happens to get a few more report exports, which take 2 seconds of CPU each; server B gets mostly lookups at 5 ms. Request counts are identical. Server A is at 95% CPU with a growing queue, and server B is at 30%. Round robin can’t see this, because it only counts.
Algorithms that look at the servers’ current state handle it better:
- Least connections / least outstanding requests. Send the next request to the server with the fewest requests in flight. A server stuck on slow requests has more outstanding work, so it gets fewer new ones.
- Power of two choices. Pick two servers at random and send the request to the less busy of the two. It needs much less coordination than tracking every server’s load exactly, which matters when many balancer instances each have only a partial view.
- Weighted variants. Give bigger instances a larger share, or ramp new instances up slowly (slow start) so a cold JVM doesn’t get a full share before its caches warm.
Simulating the algorithms
To compare them, a discrete-event simulation in Node sends 400,000 requests to 10 servers with 4 worker slots each. 90% of requests cost 10 ms and 10% cost 200 ms, and the arrival rate is 80% of total capacity. The policy is the only thing that changes:
if (policy === 'round robin') s = rr++ % SERVERS;
else if (policy === 'random') s = Math.floor(rand() * SERVERS);
else if (policy === 'least outstanding') { s = 0; for (let j = 1; j < SERVERS; j++) if (outstanding[j] < outstanding[s]) s = j; }
else { const a = Math.floor(rand() * SERVERS), b = Math.floor(rand() * SERVERS); s = outstanding[a] <= outstanding[b] ? a : b; } // power of two choicesThe latency measured includes time waiting in a server’s queue:
all servers healthy, 80% utilisation:
round robin p50 15 ms p99 350 ms p99.9 485 ms
random p50 28 ms p99 426 ms p99.9 605 ms
power of two choices p50 10 ms p99 212 ms p99.9 298 ms
least outstanding p50 10 ms p99 201 ms p99.9 229 ms
one of the 10 servers runs at half speed (noisy neighbour):
round robin p50 13 ms p99 176405 ms p99.9 194684 ms
random p50 24 ms p99 175888 ms p99.9 192466 ms
power of two choices p50 10 ms p99 251 ms p99.9 440 ms
least outstanding p50 10 ms p99 209 ms p99.9 407 msWith healthy servers, round robin’s p99 is 350 ms even though the slowest request only takes 200 ms to process. The extra time is queueing: round robin keeps handing work to a server that’s already busy with several 200 ms requests while another sits idle. Random is worse still. Least outstanding gets p99 down to 201 ms, barely above the cost of the expensive request itself, and power of two choices gets close with only two samples.
The second block is the case that matters in production. One server runs at half speed. Round robin and random still send it 10% of traffic, which is more than it can process, so its queue grows without limit and requests that land on it wait minutes. That’s what the 176,405 ms p99 means: about 10% of requests went to a server that was falling further behind for the whole run. The adaptive policies notice the growing outstanding count and route around it, keeping p99 near 250 ms.
That’s the core interview point. A server that slows down, rather than failing outright, is invisible to count-based balancing and to most health checks. Only an algorithm that reacts to each server’s backlog or latency protects you from it.
Health checks decide who’s in the pool
A balancer can only spread load across servers it believes are healthy. Two kinds of checks feed that belief:
- Active checks. The balancer calls
GET /healthzon each instance every few seconds and removes an instance after, say, 3 consecutive failures. It re-adds it after 2 successes. - Passive checks (outlier detection). The balancer watches real traffic and ejects an instance that returns too many
5xxresponses or times out. Envoy and most service meshes support this.
What the health endpoint checks matters more than how often it’s called. A shallow check (“the process is running and can answer HTTP”) catches crashes and hung processes. A deep check that also queries the database sounds more thorough, and it has a famous failure mode: when the shared database has a hiccup, every instance fails its check at once, the balancer removes the whole fleet, and a brief database blip becomes a total outage.
Two rules avoid that. Health checks should test what the instance itself controls (its process, its disk, its local config), not shared dependencies. And the balancer should fail open: if every instance in the pool fails its checks, keep sending traffic to all of them rather than to none. AWS Application and Network Load Balancers fail open when every target is unhealthy, and Envoy has a panic threshold (by default, when fewer than half the hosts look healthy it balances across all of them), because at that point the health check is more likely wrong than the whole fleet.
Kubernetes splits the idea further. A readiness probe failing removes a pod from Service endpoints without restarting it (useful during startup or when it’s temporarily overloaded). A liveness probe failing restarts the container. Pointing liveness at a deep dependency check means a database outage restarts every pod in a loop, which is the same mistake with extra steps.
Long-lived connections defeat balancing
HTTP/1.1 clients often open a connection per request or keep a small pool, so a layer 4 (TCP) balancer sees many connections and spreads them reasonably. HTTP/2 and gRPC multiplex thousands of requests over one long-lived connection. A layer 4 balancer picks a backend once, at connection time, and every request on that connection goes to the same server for hours.
The symptoms are distinctive. You scale from 4 to 8 pods and the new pods get almost no traffic, because existing clients keep their connections to the old ones. Or two big clients happen to connect to the same pod, which runs hot while others idle.
The fixes: balance per request at layer 7 (an HTTP/2-aware proxy like Envoy, or gRPC’s client-side balancing with a list of backends), or set a maximum connection age on the server (gRPC’s MaxConnectionAge) so clients reconnect periodically and redistribute. Sticky sessions cause a similar imbalance on purpose; prefer keeping session state in a shared store so any instance can serve any request.
Interview practice: the hot pod
“A gRPC service runs 6 pods behind a Kubernetes Service. Dashboards show 2 pods at 90% CPU and 4 at around 20%. Autoscaling added 3 more pods an hour ago, and they’re nearly idle. What’s going on, and what would you change?”
Answer
This pattern points at connection-level balancing of long-lived HTTP/2 connections. A Kubernetes Service with kube-proxy balances at layer 4: each client connection is assigned to a pod once, and gRPC sends every request over that one connection. A couple of high-volume clients landed on the same two pods, and the new pods only get connections that were opened after they started, which for long-lived clients is almost none. Adding pods can’t fix it, which is why autoscaling didn’t help.
I’d move to per-request balancing. Options are a service mesh or an Envoy proxy that balances each gRPC call, or gRPC client-side load balancing against a headless Service so clients see every pod’s address and use a round robin or least-request policy. As a quick mitigation, I’d set MaxConnectionAge on the server (for example 5 minutes with a grace period) so clients reconnect regularly and spread across all pods, including new ones. I’d prefer a least-request policy over plain round robin, since the service probably has uneven request costs, and confirm the fix by checking that requests per second and CPU even out across pods within a few minutes of a scale-out.