Load Balancers: L4 vs L7

Horizontal scaling gives you several interchangeable servers, but it creates a new question: when a request arrives, which server gets it? Clients cannot be expected to know, and you do not want them to. The component that answers is the load balancer. It becomes the new front door of your logic tier: the single address clients talk to, which quietly fans their requests out to the servers behind it.

Picture an airport check-in hall. Instead of a separate queue at each counter, there is a single queue and a marshal at the front. Each time a counter frees up, the marshal sends the next passenger to it. Passengers never choose a counter and never need to know how many are open. If a counter's agent goes home sick, the marshal simply stops sending anyone there.

• -The passengers are requests, and the counters are your servers.
• -The marshal is the load balancer. Everyone talks to the marshal, and only the marshal knows about the counters.
• -Deciding who goes where is the balancing algorithm.
• -Noticing that a counter has closed is the health check: the balancer regularly asks each server whether it is fine, and stops using any that fail to answer.

The two pictures below show the problem and the fix:

• -Spread: no single server takes all the load, so each stays in its comfortable range.
• -Resilience: a server that fails its health check is taken out of rotation automatically, and users only notice if too many fail at once.
• -One address: clients see one name no matter how many servers exist behind it, so you can add and remove servers without telling anyone.
• -Painless deploys: take one server out, update it, put it back, then do the next (a rolling deploy), with no downtime.
• -Optional extras: many balancers also end the TLS encryption on behalf of the servers, and can compress or cache responses.

The simplest rule is round robin: send request 1 to server A, request 2 to B, request 3 to C, request 4 to A again, and so on. It is fair in count, but count is not the same as work, and a small example shows why.

Three servers, and six requests arrive. Request 1 is a slow report that takes 5 seconds; the other five are quick lookups that take 50 milliseconds each. Round robin sends requests 1 and 4 to server A, so request 4 now waits behind a 5-second job even though B and C are idle. A smarter rule, least connections, always picks the server with the fewest requests in progress right now. A is busy with the report, so request 4 goes to B or C and finishes in 50 ms.

• -Round robin: simple and fair in count. Good when requests are all similar and servers are identical.
• -Weighted round robin: stronger servers get a bigger share (a server with weight 3 gets three requests for every one that a weight-1 server gets). Good for mixed hardware.
• -Least connections: send to whoever is least busy. Good when request times vary a lot, as in the example.
• -IP hash: the same client address always reaches the same server. It gives sticky sessions without a cookie, but it breaks down when thousands of users share one address, and it brings back the state problem the next chapter warns about.

Play through the simulation below to watch round robin make that decision request by request, and notice that it hands out requests in turn without looking at how busy each server is.

Load balancers come in two main kinds, named after layers of the network stack. Go back to the marshal. A Layer-4 marshal only counts heads: they see that a passenger arrived (an IP address and a port) and send them to a counter, without ever looking at the ticket in their hand. It is fast, and it works for any kind of traffic, not only web traffic. A Layer-7 marshal reads the ticket: they see the HTTP path, the headers, even the body, and can say that anyone holding a refund form goes to the refunds desk.

• -L4 (transport level): sees IP addresses and ports only. Very fast and protocol-agnostic. Use it for raw TCP or UDP traffic, such as game servers or database connections, or when you simply need speed.
• -L7 (application level): reads the HTTP request. It can send /api/orders to one group of servers and /api/search to another, send 5% of traffic to a new version to test it (a canary), or route by a header or cookie. It costs a little extra latency and CPU for that intelligence.

There is one catch. If all traffic goes through one load balancer and that load balancer fails, nothing gets through: you have created a new single point of failure at the very front door. In practice load balancers are run in pairs or groups (managed cloud load balancers do this for you), so that if one fails, another takes its place within seconds.

In the lab's palette the load balancer is a single box on the canvas. The card below is its reference:

For the full set of balancing algorithms and when each one fits, the Wiki card below goes further: