On sale day, our one server hit 100% and the shop went down

System Design · Foundations
15 min read, about 1 h with practice

Concept
Asked often: high
Reviewed 2026-11

See the problem

Your shop runs on one app server. On a normal day it uses 20% of its CPU. On sale day, traffic jumps to 6,000 requests a second at peak. The server hits 100%, pages time out, and the shop is down during its best hour.

You add a second server and a load balancer. Now shoppers complain that their carts empty at random. What went wrong?

Define the task

What you have: one app server that handles about 1,000 requests a second, and a database.
What you want: 6,000 requests a second at peak, and a shop that stays up when any one server fails.
The hard part: a second server only helps if any server can answer any request. Data kept inside one server breaks that.

Pause and think

Hint 1 of 3

There is always a biggest machine you can buy.

Hint 2 of 3

With one machine, a restart for an update means the shop is down.

Hint 3 of 3

If carts live in a server’s memory, what happens when the next click goes to another server?

Understand the idea

Think of a café at rush hour. You can hire a faster barista (vertical scaling, or scaling up: a bigger machine). Or you can add more baristas at more counters (horizontal scaling, or scaling out: more machines). A faster barista only goes so far, and if they are ill, the café closes. More baristas need one thing: any of them must be able to serve any customer. So orders go on a shared board, not in one barista’s head.

That shared board is the key. A server that keeps user data in its own memory is stateful. A stateless server keeps nothing between requests: carts and logins live in a shared store, and the database holds the rest.

Watch the shop go from one server to many. Change the peak traffic below the player.

One server, a bigger one, then many

v1: one server

One ordinary server
Part	At peak	Handles
App server	6,000 a second (over the limit)	about 1,000

Step 1 of 6. A shop on one app server. On sale day, 6,000 requests a second arrive at peak. One ordinary server handles about 1,000. Question: The quickest fix?

What happens next? The quickest fix?

Speed Ask me first Show as text

The tests run this small model of carts kept on each server, and in a shared store:

"""Scaling out an app server: why sessions kept in one server's memory break
behind a load balancer, and how a shared store fixes it.

Each server handles requests; a round-robin load balancer hands them out in turn.
"""

from math import ceil


class AppServer:
    """Keeps each user's cart in its own memory (a "stateful" server)."""

    def __init__(self, name: str, store: dict[str, list[str]] | None = None) -> None:
        self.name = name
        # No shared store: this server's own dict. A shared store: the same dict for all.
        self.carts = store if store is not None else {}

    def add(self, user: str, item: str) -> list[str]:
        self.carts.setdefault(user, []).append(item)
        return self.carts[user]

    def cart(self, user: str) -> list[str]:
        return self.carts.get(user, [])


class RoundRobin:
    """Sends each request to the next server in turn."""

    def __init__(self, servers: list[AppServer]) -> None:
        self.servers = servers
        self.turn = 0

    def pick(self) -> AppServer:
        server = self.servers[self.turn % len(self.servers)]
        self.turn += 1
        return server


def servers_needed(peak_per_second: int, one_server_handles: int, spare: int = 1) -> int:
    """Enough servers to keep up at peak, plus spares so one can fail."""
    return ceil(peak_per_second / one_server_handles) + spare

Climb the ladder

Each rung serves the same sale: 6,000 requests a second at peak.

The idea: Move the app to a machine with about 8 times the cores and memory.

In the cloud this is a few clicks and a restart. It is the right first answer when the shop is small and the sale is next week.

Handles: up to about 8,000 a second
Weak spot: one machine; a ceiling
Why: More cores and memory serve more requests at once, with no code change.

Where it breaks: there is a biggest machine, and next year’s sale may need more. Every restart, crash or update takes the whole shop down. And you pay for the big machine all year, for a few busy hours.

The idea: Run 7 ordinary servers behind a load balancer: 6 to keep up at peak, plus a spare.

The waste in Okay: one machine carried everything, so it was both the limit and the single point of failure.

Handles: add servers to grow
Weak spot: carts disappear between servers
Why: 6,000 ÷ 1,000 is 6 servers to keep up, and one spare covers a crash.

Where it breaks: each server keeps carts in its own memory. Asha adds tea on server 1, her next click lands on server 2, and her cart is empty. “Sticky sessions” (always the same server for one user) hide this, until that server crashes and takes its carts with it.

The idea: Keep carts and logins in a shared in-memory store (like Redis), so any server can answer any request. Add and remove servers as traffic changes.

The waste in Better: the state was in the wrong place. Now servers are all the same, so autoscaling can add them when CPU passes, say, 70%, and remove them at night.

Handles: grows by adding servers
Weak spot: the shared stores now need care
Why: Nothing lives on one server, so a crash loses nothing and new servers help at once.

When Best isn’t best: for an internal tool with 50 users, one server and a nightly backup is cheaper and simpler. And scaling out the app servers moves the pressure to the database; scaling that is the job of replication and partitioning, in Core.

Draw it yourself

Step 1 sizes the servers for the peak, with a spare. Step 2 moves carts into a shared store, so the servers are stateless.

Step 1: Sale day

At peak, 6,000 requests a second. One app server handles about 1,000.
The shop must stay up if one server crashes.

Solid arrow: sends requests toDashed arrow: copies data to

Watch the traffic

Send 10 requests through your board and see where each one goes. Then make a part fail and send them again.

SendReadsWritesBackground jobs

Make a part fail

How requests move

Requests start at the Client and follow your arrows. Reads try a cache first (4 in 5 are found there), then a read replica, then the database. Writes go to the database. Background jobs go on a queue, and a worker saves them. A load balancer skips a part that is down; nothing else can.

Compare

Way	Grows to	A server crashes	Code change	Use it when
One server	about 1,000 a second	Shop is down	None	Small sites, prototypes
One bigger server	the biggest machine	Shop is down	None	A quick fix, or a database
Many servers, state on each	any size	Its users lose carts	Load balancer	Never on purpose
Many stateless servers	any size	Nobody notices	Move state to shared stores	Real traffic, real uptime

Common mistakes and edge cases

Forgetting the spare. Sizing for exactly the peak means one crash at peak overloads the rest. Add at least one spare, or about 20% headroom.
In-memory caches and counters. A per-server cache is fine for data that can be a bit out of date. A per-server rate limit or counter is wrong: each server sees only part of the traffic.
Scaling out the database like the app. App servers are easy to copy because they are stateless. Databases hold state, so they scale with replicas and shards instead.
Autoscaling too slowly. New servers take a minute or more to start. Scale before the sale, or scale on a lower threshold.

Say it in the interview

Questions to ask first

What is the peak traffic, and how much higher is it than normal?
How bad is a few minutes of downtime?
What do the servers keep in memory between requests?

A 60-second walkthrough

“One server handles about 1,000 requests a second, and the peak is 6,000. I could scale up to a bigger machine, which is quick, but it has a ceiling and is a single point of failure. So I’d scale out: 7 stateless app servers behind a load balancer, 6 for the peak plus a spare. Stateless means carts and sessions live in a shared store like Redis, and uploads go to object storage. Then any server can answer any request, a crash loses nothing, and autoscaling can add servers before the peak.”

Likely follow-ups

Why not sticky sessions? They break even spreading and lose a user’s data when that server dies. They are a stopgap, not a design.
Is the session store now a single point of failure? Yes, so run it with a replica that takes over. If it is lost, users sign in again; their orders are safe in the database.
When is vertical scaling the right answer? For a database that is hard to split, or for a quick fix before a planned redesign.

Red flags interviewers notice

Adding servers without saying where the sessions go.
One of anything with no spare: one load balancer, one database, one cache.
No numbers for how many servers, or why.

Practice

1. The nightly report (easy)

Every night a report job runs for 2 hours on the app server and slows down the shop for night owls. Fix it.

Hint 1 of 3

Does the report need to run on the same machine that serves shoppers?

Hint 2 of 3

Which machine has to answer users fast, and which can take its time?

Hint 3 of 3

Reading a big report hard also loads the main database.

Show the answers

The idea: Scale up so the report and the shop both fit.

Quick, but the cost runs 24 hours a day.

Gives: the shop is fast again
Weak spot: you pay all day for a 2-hour job
Why: More cores leave room for both.

The idea: Run the report on its own worker server, started only for the job.

The worker can be large and cheap, because it runs 2 hours a night.

Gives: the shop never shares a machine with it
Weak spot: the database still feels it
Why: Different jobs on different machines, sized for each.

The idea: The worker reads from a read replica of the database, not the primary.

For a report about yesterday, a few seconds of lag doesn’t matter.

Gives: nothing the shopper uses is touched
Weak spot: the replica may be a few seconds behind
Why: Heavy reads land on a copy, so the primary stays fast for orders.

2. Uploads on one server (medium)

Users upload profile photos. After you add a second app server, half the photos show as broken. Why, and what is the fix?

Hint 1 of 3

Where does the upload code save the file?

Hint 2 of 3

The next request may go to a different server.

Hint 3 of 3

Is there a store every server can reach, built for files?

Show the answers

The idea: Send each user to the same server every time.

Everyone else viewing that profile may land on the other server.

Gives: the uploader sees their photo
Weak spot: other users don’t; a crash loses photos
Why: The file is still on one server’s disk.

The idea: Mount one shared file system on every server.

It works, but shared file systems are hard to scale and to keep fast.

Gives: every server sees every file
Weak spot: one more thing to keep up and fast
Why: The files leave the servers’ own disks.

The idea: Save photos to object storage (like S3), keep only the photo’s key in the database, and serve photos through a CDN.

Give each new photo a new key, so the CDN never shows an old one.

Gives: any number of servers and photos
Weak spot: a short delay before a CDN shows a changed photo
Why: Object storage is built for many files and many readers; the servers stay stateless.

3. Now the database is at 100% (hard)

You scaled out to 20 stateless app servers. Now the one database is at 100% CPU at peak. 90% of its queries are reads. What do you do, in order?

Hint 1 of 3

Many reads ask for the same data again and again.

Hint 2 of 3

A copy of the database can answer reads.

Hint 3 of 3

Scaling up a database is often the easiest first step, because splitting it is hard.

Show the answers

The idea: Move it to a bigger machine.

Also check for missing indexes and N+1 queries: they are cheaper to fix than any machine.

Gives: quick room to grow
Weak spot: the same ceiling, later
Why: Databases are hard to split, so a bigger machine is often the first, right step.

The idea: Cache hot reads, and send other reads to read replicas; the primary takes the writes.

Read your own writes from the primary, so a user always sees what they just saved.

Gives: reads grow with replicas
Weak spot: replicas lag a little behind
Why: 90% of the load is reads, and reads can be answered by copies.

The idea: Split the data across several primaries by a key, like customer id.

Do this last: it is the most work. The partitioning lesson in Core covers it.

Gives: writes and storage grow too
Weak spot: queries across shards get hard
Why: Each shard holds part of the data and takes part of the writes.

Check yourself

Cheat sheet

Vertical (scale up): a bigger machine. Quick, no code change, has a ceiling, one point of failure.
Horizontal (scale out): more machines behind a load balancer. Grows further and survives crashes.
Servers needed = peak ÷ one server’s limit, rounded up, plus a spare.
Scale out only works with stateless servers: sessions in a shared store, files in object storage.
Scaling the app moves the load to the database: cache, replicas, then shards.

Go deeper

Can you do these now?

If you’re not sure of any of these, scroll back to that part before moving on.

Say what vertical and horizontal scaling are, and the limits of each
Work out how many servers a peak needs, with a spare
Explain why servers must be stateless to scale out, and where the state goes instead
Spot the single points of failure in a design

Status: Not started

On sale day, our one server hit 100% and the shop went down

See the problem

Define the task

Pause and think

Understand the idea

Climb the ladder

Scale up: one bigger server

Scale out: many servers, carts on each

Stateless servers and a shared session store

Draw it yourself

Step 1: Sale day

Watch the traffic

Compare

Common mistakes and edge cases

Say it in the interview

Practice

1. The nightly report (easy)

A bigger app server

A separate worker machine

A worker reading a replica

2. Uploads on one server (medium)

Sticky sessions

A shared network disk

Object storage and a CDN

3. Now the database is at 100% (hard)

Scale the database up

Cache and read replicas

Shard when writes or size grow too big for one machine

Check yourself

Cheat sheet

Go deeper

Can you do these now?