One server is at 95% while the other two sit idle

See the problem

Your shop runs on three app servers behind a load balancer that takes turns: server 1, 2, 3, 1, 2, 3. Most requests take about a second. Every third request is a sales report that takes 20 seconds.

The graphs look odd. Server 3 is at 95% and its users wait. Servers 1 and 2 are nearly idle. Then server 2 crashes, and one request in three fails, even though two servers are fine.

Define the task

• What you have: three app servers and a load balancer in front of them. Requests take very different times.
• What you want: every request goes to a healthy server that has room for it.
• The hard part: “taking turns” counts requests, not how heavy they are. And it doesn’t notice a server that has died.

Pause and think

⏸ Pause and think

If you could see how many requests each server is working on right now, which server would you pick for the next one? How would you notice a dead server?

Give it two minutes with pen and paper before you open a hint.

Hint 1 of 3

Every third request is a report, and every third turn is server 3.

Hint 2 of 3

The load balancer already knows how many open requests it has sent to each server.

Hint 3 of 3

It could ask each server "are you OK?" every few seconds.

Understand the idea

Think of the checkout lines in a supermarket. A guard who sends shoppers to lines 1, 2, 3 in turn will put every full trolley in line 3 if they come every third. A smarter guard looks at the lines and picks the shortest. And if a till breaks, the guard stops sending people to it.

A load balancer is that guard. It has one address that clients know, and a list of servers behind it.

• Health checks: every few seconds it calls each server (for example GET /health). A server that fails a few in a row is taken out of the list until it passes again.
• Algorithms: round robin takes turns. Least connections picks the server with the fewest requests in flight. Hashing picks by a key, like the user id, so the same user always lands on the same server. Power of two choices picks two servers at random and takes the less busy one.
• Layer 4 or layer 7: a layer 4 load balancer sees only addresses and ports (TCP). It is very fast. A layer 7 load balancer reads the HTTP request: the path, headers and cookies. It can send /reports/ to other servers, and end HTTPS for them.

★Key insight

A load balancer is only as good as what it knows. Health checks tell it who is alive. Least connections tells it who is busy. Layer 7 tells it what each request is, so slow work can go to its own servers.

Watch the reports pile up, then spread out. Change how long a report takes below the player.

Round robin, then health checks and least connections

⤢ Full view

v1: round robin

Parts:

• Users
• Load balancer
• Server 1
• Server 2
• Server 3

Arrows:

• Users sends requests to Load balancer
• Load balancer sends requests to Server 1
• Load balancer sends requests to Server 2
• Load balancer sends requests to Server 3

Step 1 of 6. Three app servers behind a load balancer. 3 requests arrive each second. Every third one is a report that takes 20 seconds; the rest take 1.

◀ BackPlayNext ▶Reset Ask me first Show as text

Try your own input

Seconds a slow request takesShow it

The tests run the same stream through each algorithm:

Load-balancing algorithms on a fixed stream

"""Load-balancing algorithms on a small, fixed stream of requests.

Every second, 3 requests arrive. Every third request is a slow report that
takes 20 seconds; the rest take 1 second. We count how many requests each
server is working on at once (its "in flight" count) at the busiest moment.
"""

from collections.abc import Callable

SLOW_EVERY = 3
SLOW_SECONDS = 20
FAST_SECONDS = 1


def duration(i: int) -> int:
    return SLOW_SECONDS if i % SLOW_EVERY == SLOW_EVERY - 1 else FAST_SECONDS


Picker = Callable[[int, list[int], list[bool]], int]


def round_robin(i: int, in_flight: list[int], healthy: list[bool]) -> int:
    """Next server in turn, healthy or not."""
    return i % len(in_flight)


def round_robin_checked(i: int, in_flight: list[int], healthy: list[bool]) -> int:
    """Next healthy server in turn: a health check takes failed servers out."""
    up = [s for s, ok in enumerate(healthy) if ok]
    return up[i % len(up)]


def least_connections(i: int, in_flight: list[int], healthy: list[bool]) -> int:
    """The healthy server with the fewest requests in flight (lowest number on a tie)."""
    up = [s for s, ok in enumerate(healthy) if ok]
    return min(up, key=lambda s: (in_flight[s], s))


def run(
    pick: Picker,
    servers: int = 3,
    seconds: int = 60,
    per_second: int = 3,
    healthy: list[bool] | None = None,
) -> dict[str, object]:
    """Send the stream; return the peak in flight per server, and requests sent to a down server."""
    healthy = healthy or [True] * servers
    ends: list[list[int]] = [[] for _ in range(servers)]  # When each request in flight finishes.
    peak = [0] * servers
    lost = 0
    i = 0
    for t in range(seconds):
        for s in range(servers):
            ends[s] = [e for e in ends[s] if e > t]
        for _ in range(per_second):
            s = pick(i, [len(e) for e in ends], healthy)
            if not healthy[s]:
                lost += 1
            else:
                ends[s].append(t + duration(i))
                peak[s] = max(peak[s], len(ends[s]))
            i += 1
    return {"peak": peak, "lost": lost, "sent": i}


def route(path: str) -> str:
    """A layer 7 balancer reads the path to pick a pool; layer 4 sees only addresses and ports."""
    return "reports pool" if path.startswith("/reports/") else "web pool"

Climb the ladder

Each rung serves 3 requests a second for a minute; every third is a 20-second report.

Okay

Round robin, no health checks

The idea: Hand requests to servers 1, 2, 3 in turn, whatever state they are in.

Round robin is fine when every request costs about the same, which is often true for simple web pages.

Busiest server

server 3 holds 20 at once; others 1

Weak spot

a dead server still gets its turn

Why

Every report lands on the same server, because reports and turns repeat every three.

Where it breaks: uneven requests pile up on one server. And when server 2 crashes, round robin keeps sending it every third request: 60 of 180 fail in the tests.

Better

Health checks and least connections

The idea: Skip servers that fail their health check, and send each request to the server with the fewest in flight.

The waste in Okay: it ignored what it could easily know: which servers are alive and how busy each is.

Busiest server

about 7 or 8 at once on each

Weak spot

reports still slow shoppers on the same servers

Why

A server busy with reports is skipped until it catches up, so the work evens out.

Where it breaks: every server now holds some reports. A shopper on a server deep in reports still waits behind them.

Best

Layer 7 rules: slow work gets its own pool

The idea: Read the path: send /reports/ to a reports pool, everything else to the web pool, each with least connections and health checks.

The waste in Better: it mixed two kinds of work. Layer 7 can also end HTTPS once, add headers, and send a small share of traffic to a new version.

Busiest server

shoppers never queue behind a report

Weak spot

more rules to keep right

Why

Different kinds of work never share a server, so each pool can be sized for its own job.

When Best isn’t best: layer 7 does more work per request than layer 4. For huge volumes of plain TCP (a game server, a database proxy), a layer 4 balancer in front is faster and simpler.

Draw it yourself

Step 1 puts three servers behind one address. Step 2 removes the last single point of failure: the load balancer itself. Try Make a part fail on each.

Step 1 Three servers, one addressStep 2 No single load balancer

Open full page

Step 1: Three servers, one address

• Three app servers run the shop.
• Users must reach whichever servers are healthy, through one address.

+ Client+ DNS+ CDN+ Load balancer+ API gateway+ App server+ Cache+ Database+ Read replica+ Queue+ Worker+ Object storage+ Search index+ Service

Bin

Add parts from the list above. Then press the + on a part to draw an arrow.

Solid arrow: sends requests toDashed arrow: copies data to

Watch the traffic

Send 10 requests through your board and see where each one goes. Then make a part fail and send them again.

Send

ReadsWritesBackground jobs

Send 10 reads

How requests move

Requests start at the Client and follow your arrows. Reads try a cache first (4 in 5 are found there), then a read replica, then the database. Writes go to the database. Background jobs go on a queue, and a worker saves them. A load balancer skips a part that is down; nothing else can.

Check my designShow an example

Compare

Algorithm	Picks	Good when	Weak when
Round robin	the next server in turn	requests cost about the same	requests vary a lot
Weighted round robin	in turn, bigger servers more often	servers differ in size	requests vary a lot
Least connections	fewest requests in flight	request times vary	very short requests, many balancers
Power of two choices	the less busy of two random servers	many balancers, little shared view	very few servers
Hash of a key	the server for that key	a server holds data for a key (a cache)	servers come and go often

Common mistakes and edge cases

!A health check that always says OK

A /health page that returns 200 without checking anything says nothing. Check what the server needs to work, like its database connection, but keep it cheap: it runs every few seconds on every server.

• One load balancer. It becomes the single point of failure. Run two, with a shared floating address or DNS pointing at both. Cloud load balancers do this for you.
• Pulling a server out mid-request. Before a deploy, stop sending new requests to the server, let open ones finish (draining), then stop it.
• hash(user) % N. Adding one server moves almost every user to a new server. Consistent hashing moves only a few.
• Retries that double the load. If the balancer retries a failed request on another server, a sick system gets twice the traffic. Limit retries.
• Long connections. WebSockets stay open for hours, so least connections and draining matter even more.

Say it in the interview

Questions to ask first

• Do requests take about the same time, or very different times?
• Must a user stay on one server, or are the servers stateless?
• Is it HTTP, or plain TCP?

A 60-second walkthrough

“Clients know one address: the load balancer. It health-checks each server every few seconds and drops one that fails. Our requests vary a lot, so I’d use least connections rather than round robin, which piles slow requests on one server. It’s a layer 7 balancer, so it ends HTTPS and can route by path: slow reports go to their own pool. And I’d run two balancers, or use the cloud’s managed one, so the balancer isn’t a single point of failure.”

Likely follow-ups

• Layer 4 or layer 7? Layer 4 is faster and sees only TCP. Layer 7 reads HTTP, so it can route by path, add headers and end HTTPS.
• Sticky sessions? Possible with a cookie or a hash, but better to make servers stateless.
• How do you deploy with no downtime? Drain one server, update it, let it pass its health check, then move to the next.

Red flags interviewers notice

• Drawing a load balancer without health checks.
• One load balancer for a system that must never go down.
• Hashing users to servers with % N and no plan for adding a server.

Practice

1. A chat app with WebSockets (easy)

Each user keeps one WebSocket open for hours. You run 4 chat servers. After a deploy, server 4 has no users while servers 1 to 3 are full. Why, and what helps?

Hint 1 of 3

Connections are made when users open the app, then stay for hours.

Hint 2 of 3

A restarted server starts empty, and nobody moves.

Hint 3 of 3

Which algorithm sends new connections to the emptiest server?

Show the answers

Okay

Round robin

The idea: New connections are shared in turn.

After each deploy, the newest server stays light for hours.

Gives

new users spread evenly

Weak spot

old users stay where they are

Why

Server 4 gets only a quarter of new connections, so it fills slowly.

Better

Least connections

The idea: Send each new connection to the server with the fewest open.

The right default for long connections.

Gives

server 4 fills first

Weak spot

evens out only as users reconnect

Why

It looks at what is open now, not at turns.

Best

Least connections, plus slow rebalancing

The idea: Also ask a few users on full servers to reconnect, a little at a time.

Never drop everyone at once: they would all reconnect in the same second.

Gives

even within minutes

Weak spot

some users reconnect once

Why

Clients reconnect on their own, so moving a few is cheap.

2. A new version to 5% of users (medium)

You want to try a new version of the checkout on 5% of users first, and send everyone back if errors go up.

Hint 1 of 3

Which kind of load balancer can tell one user from another?

Hint 2 of 3

A user should not flip between versions on every click.

Hint 3 of 3

What should decide whether to go back?

Show the answers

Okay

One new server in twenty

The idea: Put the new version on 1 of 20 servers behind round robin.

Fine for a backend change users can’t see.

Gives

about 5% of requests

Weak spot

users flip between versions

Why

Each request picks a server, so one user sees both versions.

Better

Weighted, by user

The idea: A layer 7 rule sends 5% of users, picked by a hash of their id or a cookie, to the new pool.

This is a canary release: a small group sees it first.

Gives

5% of users, always the same ones

Weak spot

someone must watch the graphs

Why

Hashing the user id keeps each user on one version.

Best

Weighted by user, rolled back by numbers

The idea: Raise the share step by step (5%, 25%, 100%), and roll back at once if the new pool’s error rate or p95 is worse.

Compare the new pool with the old one at the same time, not with yesterday.

Gives

safe releases

Weak spot

needs good metrics per version

Why

A rule, not a person, decides, so a bad version reaches few users.

3. Cache servers by key (hard)

You have 4 cache servers. Each product should live on one of them, so the app always asks the same server for the same product. Then you add a fifth server.

Hint 1 of 3

Round robin would put the same product on every server.

Hint 2 of 3

hash(product) % 4 picks a server per product. What happens to that number when 4 becomes 5?

Hint 3 of 3

Is there a way where adding a server moves only some keys?

Show the answers

Okay

Round robin

The idea: Take turns.

Each server holds a quarter of the hits and all of the keys.

Gives

even load

Weak spot

each product cached 4 times; many misses

Why

The same key lands on a different server each time.

Better

hash(key) % 4

The idea: Pick the server by the key’s hash.

A cache that suddenly misses sends a flood to the database.

Gives

each key on one server

Weak spot

adding a server moves about 4 in 5 keys

Why

Most keys get a new remainder when 4 becomes 5, so most of the cache misses at once.

Best

Consistent hashing

The idea: Place servers and keys on a ring; a key goes to the next server round the ring.

The partitioning lesson in Core builds the ring step by step.

Gives

each key on one server

Weak spot

a little more code

Why

A new server takes keys only from its neighbour on the ring: about 1 in 5 keys move.

Check yourself

1. Multiple choiceRequests take from 10 ms to 30 seconds. Which algorithm spreads them best?

   Round robinLeast connectionsAlways the first serverRandom, without looking

   Check answer
2. Multiple choiceWhat can a layer 7 load balancer do that a layer 4 one can’t?

   Send traffic to several serversRoute by the URL path or a cookieUse TCPHave an IP address

   Check answer
3. Spot the bugThis health check runs every 5 seconds. What is wrong with it?

   @app.get("/health")
   def health():
       return {"ok": True}

   It should be POSTIt says OK even when the server can’t reach its databaseHealth checks must return HTMLNothing

   Check answer
4. Multiple choiceKeys are placed with hash(key) % 4. You add a fifth server. About how many keys move?

   About 1 in 5About halfAbout 4 in 5None

   Check answer

Cheat sheet

• A load balancer gives clients one address and spreads requests over healthy servers.
• Health checks every few seconds; drain a server before you stop it.
• Round robin for even requests; least connections when times vary; a hash when a key must stay on one server.
• Layer 4 sees TCP; layer 7 reads HTTP and can route by path, header or cookie.
• Run two load balancers, or a managed one: it must not be a single point of failure.

Go deeper

• NGINX: HTTP load balancing
• AWS: Elastic Load Balancing features
• Before this: Vertical and horizontal scaling.

Can you do these now?

If you’re not sure of any of these, scroll back to that part before moving on.

• Explain what a load balancer does, and why it needs health checks
• Compare round robin, least connections and hashing, and pick one for a workload
• Say what a layer 4 and a layer 7 load balancer can see and do
• Remove the load balancer as a single point of failure

✓Mark as completeI already know this

Status: Not started