Load balancing

Like the other experiments, this is a discrete-event simulation: arrivals and completions are events on a simulated clock, processed in time order. Playback speed only changes how fast you watch; the numbers are the same at any speed.

The model

1. Requests arrive at random (a Poisson process) at the chosen rate. Each carries an amount of work: exactly the average, or random (exponential) when Uneven requests is on.
2. Each server handles one request at a time, first come first served, and queues the rest. Slow servers take the slowdown factor times longer for the same work.
3. The balancer picks a server the moment a request arrives, using up-to-date counts of each server's outstanding requests (waiting + in service).
4. A server with 50 outstanding requests rejects new ones, like an HTTP 503. Rejected requests are not retried and are not counted in latency.
5. Latency is the time from arrival until a server finishes the request: time waiting in the queue plus service time.

The four strategies

• Round robin sends requests to s1, s2, s3… in turn. It is simple and perfectly even, but blind to how busy each server is.
• Random picks any server. It needs no shared state, but short-term luck can pile requests onto one server.
• Least connections picks the server with the fewest outstanding requests. Ties are shared out using a rotating start, so s1 isn't always favoured.
• Power of two choices picks two different random servers and uses the less busy one. It only compares two servers per request, yet avoids most of random's bad luck.

How the comparison is fair

One seeded request stream (arrival times and work) feeds all four balancers at once, each with its own copy of the server pool. Random picks come from a hash of the seed and request number, so one strategy's choices never change another's. The offered load shown next to the settings is the arrival rate divided by what the pool can finish per second at full speed.

Under the hood

The model is plain TypeScript with no UI code, built on the same seeded PRNG and binary-heap event queue as the other experiments. Percentiles come from a small logarithmic histogram (2% buckets, about ±1% accuracy), so memory stays flat on long runs. Unit tests cover each strategy's choices, FIFO service, slow servers, queue limits, utilization, metric consistency, the histogram, and repeatability.

Tradeoffs it shows

• Even isn't the same as balanced: round robin gives every server the same number of requests, not the same amount of work.
• Tail latency is where strategies differ: the median often looks similar while p99 is several times worse.
• Load awareness routes around slow servers without anyone marking them as slow.
• Two choices go a long way: comparing just two servers gets most of the benefit of checking all of them, which matters when there are many servers or many balancers.
• No strategy fixes too little capacity: above 100% offered load, queues grow and requests are rejected whatever the balancer does.

What this model leaves out

• It is a simplified model for building intuition, not a production benchmark.
• The balancer always knows exact, current server load. Real balancers often see stale or partial information, and several balancers that each run least connections can all pick the same idle server at once.
• Servers process one request at a time; real servers handle many concurrently and slow down gradually as they fill up.
• No network time, health checks, retries, timeouts, sticky sessions or weighted servers.
• Production balancers add refinements this one doesn't: weights, slow-start for new servers, latency-aware picks (e.g. EWMA) and outlier ejection.