Software Engineer
Loading posts...
Feel free to contact me at kanishksachdev@gmail.com
A single server can only do so much. Point enough traffic at it and it falls over. The usual answer is to run several servers and put a load balancer in front, whose only job is to decide which server each incoming request goes to.
That decision sounds trivial, and the naive version (just go round and round) is one line of code. But the wrong choice quietly drops requests while half your fleet sits idle. Below is a working load balancer you can poke at. Each section builds up one idea, and the charts are generated from the same simulation you are watching, so you can see the difference rather than take my word for it.
Give every request a size of roughly one unit of work, and give this server a power of 2: it clears about two units, so about two requests, per second. That is its ceiling. Drag the traffic slider up to 3 per second and the arithmetic turns brutal: one request per second has nowhere to go. The queue (the tube above the server) fills, and once it is full, new requests are dropped on the floor. Watch the drop rate climb.
More servers means more headroom. Two power-2 servers can clear about four requests per second between them. The simplest way to share the load is Round Robin: deal requests out like playing cards, server 0, server 1, server 0, and so on. At three requests per second across two equal servers, each sees about 1.5, comfortably under its limit of 2. Nothing drops, and the per-server chart stays even.
Real fleets are rarely uniform. Maybe one box is new and one is old, or they run different instance sizes. Here server 0 has power 3 and server 1 has power 1. Round Robin does not know or care: it still deals a perfect 50/50. But server 1 can only clear about one request per second, and Round Robin keeps handing it 1.5. Its queue backs up, spills over, and starts bleeding red, while the strong server coasts.
Same total capacity as before (four units per second), same traffic, yet we are dropping requests. The problem is not the servers, it is that the balancer treats unequal servers equally.
The obvious fix is to give the stronger server more of the deck. Weighted Round Robin sends requests in proportion to each server's power. With powers of 3 and 1, server 0 takes three of every four requests and server 1 takes one. Now each server gets work matched to what it can actually clear. Nothing changed about the hardware or the traffic, only the routing, and the drops mostly disappear.
Weighted Round Robin assumes you know every server's capacity up front. Often you do not, and it gets worse: requests are not equal either. Some are cheap, some are monsters, and a server's real load drifts moment to moment. Least Connections skips the guessing entirely. It sends each new request to whichever server has the fewest in flight right now. A server bogged down by an expensive request stops receiving new work automatically, no weights required. Watch the latency distribution stay tight even though these three servers (power 3, 2, and 1) are wildly unequal.
Now it is a sandbox. Switch algorithms mid-stream and watch the charts react. Try Random, Round Robin's dumber cousin, which is cheap and surprisingly okay until it is not. Add and remove servers, and push the traffic past what the fleet can handle to see which strategies degrade gracefully and which fall off a cliff.
This is a teaching model, not a benchmark: it ignores network latency, connection setup, health checks, and slow starts. For the real, gorgeous version that inspired it (and goes deeper, into dynamic weighting and peak-EWMA), read samwho.dev/load-balancing.