200 open chats froze my Flask server

See the problem

Your Flask app runs on Gunicorn with 8 threads. It is fast. Then you add a chat page. Each open chat sends a request that waits up to 30 seconds for a new message.

On launch day, 200 people open the chat. Every other page stops loading. The CPU is almost idle. The server is not busy, it is waiting, and it has nothing left to wait with.

To fix this, you need to know what Gunicorn does with a request, and what it expects your app to be.

Define the task

Answer three questions, then write a small app the way the servers expect it.

• Who reads HTTP? The web server (Gunicorn, Uvicorn) reads the bytes. Your app never should.
• What does the server call? A WSGI app is app(environ, start_response). An ASGI app is async def app(scope, receive, send).
• What does waiting cost? Under WSGI, a waiting request holds a thread. Under ASGI, it holds only a little memory.
• The hard part: keep 200 slow requests open without 200 threads.

Example: a WSGI server gets GET /hello?name=Ana. It calls your app with environ["PATH_INFO"] == "/hello". Your app calls start_response("200 OK", headers) and returns [b"Hello, Ana"].

Pause and think

⏸ Pause and think

Before you read on: each chat request spends 30 seconds waiting and a few milliseconds working. What is the thread doing for those 30 seconds?

Give it two minutes with pen and paper before you open a hint.

Hints open one at a time:

Hint 1 of 3

A normal function call holds its thread until it returns. Even time.sleep(30) keeps the thread.

Hint 2 of 3

What if the function could pause, give the thread back, and go on when the message comes?

Hint 3 of 3

That is what await does. A server that awaits your app can run other requests while one waits.

Understand the idea

The server and the app are two programs with a deal. The server owns the network: it reads HTTP bytes, and it writes them back. The app owns the logic: given a request, what is the answer? The deal is the shape of the call between them.

WSGI is the classic deal (PEP 3333, from 2003). Your app is a plain function, or anything you can call:

• The server puts the request in a dict, environ: REQUEST_METHOD, PATH_INFO, QUERY_STRING, the headers as HTTP_* keys, and the body in wsgi.input.
• Your app calls start_response(status, headers), then returns the body as a list of bytes.
• Flask and Django are WSGI apps. Gunicorn, uWSGI, Waitress and mod_wsgi are WSGI servers.

ASGI is the async deal (2018). Your app is an async def with three arguments:

• scope is a dict like environ: type ("http", "websocket" or "lifespan"), method, path and headers.
• await receive() gets the body, in pieces. await send({...}) sends the start of the response, then the body.
• FastAPI and Starlette are ASGI apps, and Django can be too. Uvicorn, Hypercorn and Daphne are ASGI servers.

The difference is waiting. A WSGI call holds its thread until it returns. An ASGI app that reaches await pauses, and the server’s event loop (one thread that switches between paused tasks) runs another request. A paused request is just an object in memory, so one process can keep thousands open.

A request through the server, then 200 waiting chats

⤢ Full view

WSGI: one request

Parts:

• Browser (the request is here)
• Gunicorn
• Your app

Arrows:

• Browser sends requests to Gunicorn
• Gunicorn sends requests to Your app

Step 1 of 9. A browser sends GET /hello. It arrives at the server as bytes on a network connection.

◀ BackPlayNext ▶Reset Ask me first Show as text

★WSGI holds a thread per request; ASGI holds a thread only while it works

Both deals let any server run any app. ASGI adds await, so a request that waits gives its thread back. That is what lets one process keep thousands of slow requests, and WebSockets, open.

Climb the ladder

Three ways to answer GET /hello with Hello. Each one hands more of the work to the server.

Okay

Read and write raw HTTP yourself

The idea: Open a socket, read the request’s bytes, find the method and path, and write the response bytes back.

It shows what every server does underneath, and it does answer a real browser. The tests check that.

import socket


def serve(port: int) -> None:
    """Okay: the app reads and writes raw HTTP itself, one request at a time."""
    with socket.create_server(("127.0.0.1", port)) as server:
        while True:
            conn, _ = server.accept()
            with conn:
                head = conn.recv(65536).decode("latin-1")
                method, path, _ = head.split("\r\n", 1)[0].split(" ")
                if method == "GET" and path == "/hello":
                    status, body = "200 OK", b"Hello"
                else:
                    status, body = "404 Not Found", b"Not found"
                conn.sendall(
                    f"HTTP/1.1 {status}\r\nContent-Length: {len(body)}\r\n\r\n".encode() + body
                )

Runs on

Only its own loop

At once

1 request

Why

The app is its own server. It answers one connection at a time, and parses HTTP itself.

Where it breaks: one slow client blocks everyone. It ignores headers, bodies and keep-alive, and it can’t move to a better server without a rewrite. The previous lesson’s practice showed how much HTTP there is to get right.

Better

A WSGI app

The idea: Write a function that takes environ and start_response. A WSGI server like Gunicorn does all the HTTP.

The waste in Okay: every app wrote its own half-correct HTTP code. Now the app has a single job, and the server can be swapped.

Highlighted lines are new in this rung.

def app(environ, start_response):
    """Better: a WSGI app. Any WSGI server (Gunicorn, Waitress) reads HTTP and calls it."""
    if environ["REQUEST_METHOD"] == "GET" and environ["PATH_INFO"] == "/hello":
        status, body = "200 OK", b"Hello"
    else:
        status, body = "404 Not Found", b"Not found"
    start_response(status, [("Content-Type", "text/plain"), ("Content-Length", str(len(body)))])
    return [body]

Runs on

Any WSGI serverwas Only its own loop

At once

1 per thread or processwas 1 request

Why

The server reads the request into environ, calls the app, and writes what it returns. It can run many copies in threads or processes.

What changed from the previous rung

import socket


def serve(port: int) -> None:
    """Okay: the app reads and writes raw HTTP itself, one request at a time."""
    with socket.create_server(("127.0.0.1", port)) as server:
        while True:
            conn, _ = server.accept()
            with conn:
                head = conn.recv(65536).decode("latin-1")
                method, path, _ = head.split("\r\n", 1)[0].split(" ")
                if method == "GET" and path == "/hello":
                    status, body = "200 OK", b"Hello"
                else:
                    status, body = "404 Not Found", b"Not found"
                conn.sendall(
                    f"HTTP/1.1 {status}\r\nContent-Length: {len(body)}\r\n\r\n".encode() + body
                )
def app(environ, start_response):
    """Better: a WSGI app. Any WSGI server (Gunicorn, Waitress) reads HTTP and calls it."""
    if environ["REQUEST_METHOD"] == "GET" and environ["PATH_INFO"] == "/hello":
        status, body = "200 OK", b"Hello"
    else:
        status, body = "404 Not Found", b"Not found"
    start_response(status, [("Content-Type", "text/plain"), ("Content-Length", str(len(body)))])
    return [body]

Where it breaks: the chat. Each waiting request holds its thread for 30 seconds. With 8 threads, chat 9 waits in line, and so does every other page. WSGI also has no way to carry WebSockets.

Best

An ASGI app

The idea: Write an async def that takes scope, receive and send. An ASGI server like Uvicorn awaits it on an event loop.

The waste in Better: a thread spent 30 seconds doing nothing but waiting. Now a waiting request costs a few kilobytes, not a thread.

The tests run 50 requests that each wait 0.05 seconds. Four threads take about 0.65 seconds. One event loop takes about 0.05.

Highlighted lines are new in this rung.

async def app(scope, receive, send):
    """Best: an ASGI app. The server (Uvicorn) awaits it, so one process can wait on many."""
    if scope["type"] != "http":
        return  # "lifespan" and "websocket" messages come through here too
    if scope["method"] == "GET" and scope["path"] == "/hello":
        status, body = 200, b"Hello"
    else:
        status, body = 404, b"Not found"
    await send(
        {
            "type": "http.response.start",
            "status": status,
            "headers": [
                (b"content-type", b"text/plain"),
                (b"content-length", str(len(body)).encode()),
            ],
        }
    )
    await send({"type": "http.response.body", "body": body})

Runs on

Any ASGI serverwas Any WSGI server

At once

Thousands, while they waitwas 1 per thread or process

Why

A request paused at await gives the thread back. One event loop runs whichever request is ready.

What changed from the previous rung

def app(environ, start_response):
    """Better: a WSGI app. Any WSGI server (Gunicorn, Waitress) reads HTTP and calls it."""
    if environ["REQUEST_METHOD"] == "GET" and environ["PATH_INFO"] == "/hello":
        status, body = "200 OK", b"Hello"
async def app(scope, receive, send):
    """Best: an ASGI app. The server (Uvicorn) awaits it, so one process can wait on many."""
    if scope["type"] != "http":
        return  # "lifespan" and "websocket" messages come through here too
    if scope["method"] == "GET" and scope["path"] == "/hello":
        status, body = 200, b"Hello"
    else:
        status, body = "404 Not Found", b"Not found"
    start_response(status, [("Content-Type", "text/plain"), ("Content-Length", str(len(body)))])
    return [body]
        status, body = 404, b"Not found"
    await send(
        {
            "type": "http.response.start",
            "status": status,
            "headers": [
                (b"content-type", b"text/plain"),
                (b"content-length", str(len(body)).encode()),
            ],
        }
    )
    await send({"type": "http.response.body", "body": body})

When Best isn’t best: ASGI only helps while code awaits. A slow time.sleep() or a blocking database call inside async def freezes the whole loop. CPU-heavy work gains nothing either. A plain Django or Flask site with quick pages runs well on WSGI.

Write it yourself

Write a WSGI app by hand. No framework: just the deal. Press Run tests to check your code.

• GET /hello?name=Ana answers 200 OK and Hello, Ana. With no name, Hello, world.
• Any other method on /hello answers 405 Method Not Allowed with Allow: GET.
• Any other path answers 404 Not Found.

Each test calls your app with an environ and a start_response that keeps what it was given. Then it checks the status, a header and the body.

A WSGI app by hand

Ctrl + Enter runs

from urllib.parse import parse_qs def app(environ, start_response): # environ["REQUEST_METHOD"], environ["PATH_INFO"], environ["QUERY_STRING"] start_response("200 OK", [("Content-Type", "text/plain; charset=utf-8")]) return [b"Hello"]

Run testsReset codeLoads about 6 MB the first time. Best on Wi-Fi.

1. ○Not run yet: GET /hello?name=Ana gives 200 and "Hello, Ana" → ['200 OK', 'Hello, Ana']
2. ○Not run yet: GET /hello with no name gives "Hello, world" → 'Hello, world'
3. ○Not run yet: GET /hello?name=Jos%C3%A9 decodes the name: "Hello, José" → 'Hello, José'
4. ○Not run yet: POST /hello gives 405 with Allow: GET → ['405 Method Not Allowed', 'GET']
5. ○Not run yet: GET / gives 404 → '404 Not Found'
6. ○Not run yet: Extra test

Extra tests check tricky cases. You see their input only if they fail.

Try your own input

Arguments, separated by commas as in Python

app()Run onceVisualize

Compare

Way	Server	At once	Good	Not so good	Use it when
Okay: raw HTTP	Itself	1	Shows what servers do	Half-correct HTTP, one request at a time	Learning only
Better: WSGI	Gunicorn, uWSGI, Waitress	1 per thread or process	Simple, mature, any WSGI server	A waiting request holds a thread; no WebSockets	Flask, Django, quick pages
Best: ASGI	Uvicorn, Hypercorn, Daphne	Thousands while they wait	Many slow requests, WebSockets, streaming	Blocking code inside freezes everything	FastAPI, chats, many slow outside calls

Which server for which framework? Flask: Gunicorn (WSGI). Django: Gunicorn with wsgi.py, or Uvicorn with asgi.py for async views. FastAPI: Uvicorn, often run by Gunicorn with Uvicorn workers.

Common mistakes and edge cases

!Blocking code inside async def freezes every request

Under ASGI, time.sleep(1) or requests.get(...) inside an async def holds the one event loop. Every other request waits too. Use await asyncio.sleep(1) and an async client like httpx.AsyncClient. The FastAPI async lesson covers this in full.

• The body must be bytes. A WSGI app that returns ["Hello"] (a str) breaks the deal. Encode it.
• start_response comes first. Call it before you return the body, and only once. A second call replaces the first.
• environ holds headers with a prefix. User-Agent arrives as environ["HTTP_USER_AGENT"]. Two are special: CONTENT_TYPE and CONTENT_LENGTH have no HTTP_.
• flask run and manage.py runserver are for your laptop. They are slow, single servers with debug tools. In production, use Gunicorn or Uvicorn.
• ASGI apps get lifespan messages too. At start-up and shut-down, the server sends scope["type"] == "lifespan". Ignore it or handle it, but don’t treat it like HTTP.

Say it in the interview

Questions to ask first

• Are the requests quick, or do many of them wait on slow things like other APIs or chats?
• Do we need WebSockets or streaming responses?
• Is the code we call async, like httpx, or blocking, like requests or a sync database driver?

A 60-second walkthrough

“WSGI and ASGI are the deals between a Python web server and the app. Under WSGI, the server, say Gunicorn, reads the HTTP, builds an environ dict, and calls app(environ, start_response). The app returns the body as bytes. Flask and classic Django are WSGI apps. Each request holds a thread or process until it returns, which is fine for quick pages. ASGI is the async version: async def app(scope, receive, send), run by Uvicorn on an event loop. When the app awaits a database or another API, it gives the thread back. So one process can keep thousands of slow requests and WebSockets open. FastAPI is ASGI. But blocking code inside async def stalls everything.”

Likely follow-ups

• Is ASGI always faster? No. For quick, CPU-bound pages, it’s about the same. It wins when requests spend most of their time waiting.
• Can Django do ASGI? Yes: run asgi.py with Uvicorn, and write async def views. Its ORM is still mostly sync, so Django runs sync parts in a thread.
• What does Gunicorn add in front of Uvicorn? It manages processes: starts several Uvicorn workers, and restarts any that die.
• What is middleware in WSGI terms? A WSGI app that wraps another one: it can change environ on the way in and the headers on the way out. The second practice problem builds one.

Practice

Three problems, from easy to medium. Each has hints, tests you can run here, and its own Okay, Better and Best answers.

1. Read a query string (easy)

Write get_param(environ, name, default=""). It gives back the first value of name in environ["QUERY_STRING"], decoded. If there is no such name, give back default.

Example: with QUERY_STRING set to "name=Ana+Lee&city=S%C3%A3o%20Paulo", get_param(environ, "city") is "São Paulo".

Hint 1 of 3

Spaces arrive as + or %20, and other letters as %XX bytes. They must be decoded.

Hint 2 of 3

urllib.parse has unquote_plus for one value, and parse_qs for a whole query string.

Hint 3 of 3

parse_qs gives a list for each name, because a name can appear twice. Take the first.

Read a query string

Ctrl + Enter runs

from urllib.parse import parse_qs def get_param(environ: dict, name: str, default: str = "") -> str: return default

Run testsVisualizeReset codeLoads about 6 MB the first time. Best on Wi-Fi.

1. ○Not run yet: get_param({'QUERY_STRING': 'name=Ana'}, 'name') → 'Ana'
2. ○Not run yet: A + is a space: "Ana+Lee" → 'Ana Lee'
3. ○Not run yet: %C3%A3 is ã, %20 is a space → 'São Paulo'
4. ○Not run yet: No QUERY_STRING at all → 'world'
5. ○Not run yet: Extra test
6. ○Not run yet: Extra test

Extra tests check tricky cases. You see their input only if they fail.

Try your own input

Arguments, separated by commas as in Python

get_param()Run onceVisualize

Show the Okay, Better and Best answers

Okay

Split on & and =

The idea: Split the query string into pairs, and each pair at its =.

It finds the right pair.

def get_param(environ: dict, name: str, default: str = "") -> str:
    """Okay: split the query string on & and = by hand."""
    for pair in environ.get("QUERY_STRING", "").split("&"):
        key, _, value = pair.partition("=")
        if key == name:
            return value
    return default

Reads

Plain letters

Misses

+, %20, %C3%A3

Why

The values come back still encoded.

Where it breaks: Ana+Lee comes back with its +, and São comes back as S%C3%A3o.

Better

Decode each value

The idea: The same loop, with unquote_plus on the name and the value.

The waste in Okay: the value is still in its encoded form.

Highlighted lines are new in this rung.

from urllib.parse import unquote_plus


def get_param(environ: dict, name: str, default: str = "") -> str:
    """Better: decode %20 and + too."""
    for pair in environ.get("QUERY_STRING", "").split("&"):
        key, _, value = pair.partition("=")
        if unquote_plus(key) == name:
            return unquote_plus(value)
    return default

Reads

Every valuewas Plain letters

Misses

Nothing in the taskwas +, %20, %C3%A3

Why

unquote_plus turns + into a space and %XX bytes into letters.

What changed from the previous rung

from urllib.parse import unquote_plus


def get_param(environ: dict, name: str, default: str = "") -> str:
    """Okay: split the query string on & and = by hand."""
    """Better: decode %20 and + too."""
    for pair in environ.get("QUERY_STRING", "").split("&"):
        key, _, value = pair.partition("=")
        if key == name:
            return value
        if unquote_plus(key) == name:
            return unquote_plus(value)
    return default

Where it breaks: nothing in the tests, but you wrote a parser the standard library already has.

Best

parse_qs

The idea: Let urllib.parse.parse_qs read the whole string, and take the first value.

The waste in Better: a hand-written loop to keep and test.

Highlighted lines are new in this rung.

from urllib.parse import parse_qs


def get_param(environ: dict, name: str, default: str = "") -> str:
    """Best: let the standard library read it. parse_qs decodes, and keeps every value."""
    values = parse_qs(environ.get("QUERY_STRING", ""), keep_blank_values=True).get(name)
    return values[0] if values else default

Reads

Every value, and repeatswas Every value

Misses

Nothingwas Nothing in the task

Why

One tested call, which keeps every value of a repeated name.

What changed from the previous rung

from urllib.parse import unquote_plus
from urllib.parse import parse_qs


def get_param(environ: dict, name: str, default: str = "") -> str:
    """Better: decode %20 and + too."""
    for pair in environ.get("QUERY_STRING", "").split("&"):
        key, _, value = pair.partition("=")
        if unquote_plus(key) == name:
            return unquote_plus(value)
    return default
    """Best: let the standard library read it. parse_qs decodes, and keeps every value."""
    values = parse_qs(environ.get("QUERY_STRING", ""), keep_blank_values=True).get(name)
    return values[0] if values else default

When Best isn’t best: inside a framework, use its own reader: request.args in Flask, request.GET in Django, a function argument in FastAPI.

2. Middleware that adds a header (medium)

Write add_header(app, name, value). It gives back a new WSGI app that runs app, and adds the header (name, value) to every response. Keep the app’s own status and headers.

This is middleware: an app that wraps another app. Every framework has it, and the next lesson is about it.

Example: add_header(app, "X-Server", "notes") answers like app, plus X-Server: notes.

Hint 1 of 3

The inner app tells the server its headers by calling start_response. You can’t change them after that call.

Hint 2 of 3

So give the inner app your own start_response. It adds the header, then calls the real one.

Hint 3 of 3

A start_response can get a third argument, exc_info, when the app reports an error. Pass it on.

Middleware that adds a header

Ctrl + Enter runs

def add_header(app, name: str, value: str): def wrapped(environ, start_response): return app(environ, start_response) return wrapped

Run testsReset codeLoads about 6 MB the first time. Best on Wi-Fi.

1. ○Not run yet: The header is added, and the app’s own headers stay → ['200 OK', [['A', '1'], ['X-Server', 'notes']], 'hi', 1]
2. ○Not run yet: A 404 from the app stays a 404 → ['404 Not Found']
3. ○Not run yet: The app reports an error with exc_info, a third argument → ['500 Internal Server Error', ['err']]

Try your own input

Arguments, separated by commas as in Python

add_header()Run onceVisualize

Show the Okay, Better and Best answers

Okay

Call start_response again afterwards

The idea: Let the app answer, then call start_response a second time with the new header.

It does add the header, in a way.

def add_header(app, name: str, value: str):
    """Okay: let the app answer, then call start_response again with the new header."""

    def wrapped(environ, start_response):
        body = app(environ, start_response)
        start_response("200 OK", [(name, value)])  # a second call replaces the first
        return body

    return wrapped

Keeps

The body

Loses

The app’s status and headers

Why

The second call replaces the first: every answer becomes 200 with one header.

Where it breaks: a 404 turns into a 200, and the app’s own headers disappear. Real servers refuse a second call outright.

Better

Wrap start_response

The idea: Hand the app your own start_response, which adds the header and calls the real one.

The waste in Okay: it fights the deal instead of using it.

Highlighted lines are new in this rung.

def add_header(app, name: str, value: str):
    """Better: hand the app our own start_response, which adds the header."""

    def wrapped(environ, start_response):
        def start(status, headers):
            return start_response(status, [*headers, (name, value)])

        return app(environ, start)

    return wrapped

Keeps

Status, headers, bodywas The body

Loses

Error reportswas The app’s status and headers

Why

The app still calls start_response once, and the header joins on the way through.

What changed from the previous rung

def add_header(app, name: str, value: str):
    """Okay: let the app answer, then call start_response again with the new header."""
    """Better: hand the app our own start_response, which adds the header."""

    def wrapped(environ, start_response):
        body = app(environ, start_response)
        start_response("200 OK", [(name, value)])  # a second call replaces the first
        return body
        def start(status, headers):
            return start_response(status, [*headers, (name, value)])

        return app(environ, start)

    return wrapped

Where it breaks: an app that calls start_response(status, headers, exc_info) crashes with a TypeError, because the wrapper takes two arguments.

Best

Wrap it, and pass exc_info on

The idea: Like Better, with an optional third argument passed to the real start_response.

The waste in Better: half of the deal.

Highlighted lines are new in this rung.

def add_header(app, name: str, value: str):
    """Best: like Better, and pass exc_info on, so an app can still report an error."""

    def wrapped(environ, start_response):
        def start(status, headers, exc_info=None):
            return start_response(status, [*headers, (name, value)], exc_info)

        return app(environ, start)

    return wrapped

Keeps

Everythingwas Status, headers, body

Loses

Nothingwas Error reports

Why

It matches the full start_response from PEP 3333.

What changed from the previous rung

def add_header(app, name: str, value: str):
    """Better: hand the app our own start_response, which adds the header."""
    """Best: like Better, and pass exc_info on, so an app can still report an error."""

    def wrapped(environ, start_response):
        def start(status, headers):
            return start_response(status, [*headers, (name, value)])
        def start(status, headers, exc_info=None):
            return start_response(status, [*headers, (name, value)], exc_info)

        return app(environ, start)

    return wrapped

When Best isn’t best: in a framework, use its own hook: @app.after_request in Flask, a middleware class in Django, @app.middleware("http") in FastAPI.

3. How many workers? (medium)

Write workers_needed(per_second, seconds_each, threads). Requests come at per_second on average, each takes seconds_each, and each worker has threads threads. Size for busy times, 2.5 times the average, and add one spare worker for a crash or a restart.

Example: workers_needed(200, 0.5, 4) is 64. At peak, 500 requests a second, each held for half a second, means 250 in progress. That needs 63 workers of 4 threads, plus 1 spare.

Hint 1 of 3

How many requests are in progress at once? Requests a second, times seconds each. This is Little’s law.

Hint 2 of 3

Each worker holds `threads` requests at once. Round up: half a worker is a whole worker.

Hint 3 of 3

Size for the peak, 2.5 times the average, then add one spare.

How many workers?

Ctrl + Enter runs

import math def workers_needed(per_second: float, seconds_each: float, threads: int) -> int: return 1

Run testsVisualizeReset codeLoads about 6 MB the first time. Best on Wi-Fi.

1. ○Not run yet: workers_needed(200, 0.5, 4) → 64
2. ○Not run yet: Light traffic still needs 1, plus 1 spare → 2
3. ○Not run yet: Chats: 2,000 a second, each waiting 30 s → 37501
4. ○Not run yet: Extra test

Extra tests check tricky cases. You see their input only if they fail.

Try your own input

Arguments, separated by commas as in Python

workers_needed()Run onceVisualize

Look at the third test. Under WSGI, 2,000 chats a second that each wait 30 seconds need 37,501 workers. Under ASGI, the waiting is free, so a handful of processes will do.

Show the Okay, Better and Best answers

Okay

One thread per request a second

The idea: Divide the requests a second by the threads per worker.

It grows with traffic, which is right.

import math


def workers_needed(per_second: float, seconds_each: float, threads: int) -> int:
    """Okay: one thread per request a second."""
    return math.ceil(per_second / threads)

Counts

Requests a second

Misses

How long each one takes

Why

It assumes every request takes exactly one second.

Where it breaks: a request that takes 30 seconds holds a thread 30 times longer than one that takes 1. Okay can’t see that.

Better

Requests in progress at once

The idea: Requests a second times seconds each, divided by threads, rounded up.

The waste in Okay: it counts arrivals, but threads are held for a duration.

Highlighted lines are new in this rung.

import math


def workers_needed(per_second: float, seconds_each: float, threads: int) -> int:
    """Better: requests in progress at once = per second x seconds each (Little's law)."""
    return math.ceil(per_second * seconds_each / threads)

Counts

Requests in progresswas Requests a second

Misses

Busy times, crasheswas How long each one takes

Why

Little’s law: in progress = arrival rate × time in the system.

What changed from the previous rung

import math


def workers_needed(per_second: float, seconds_each: float, threads: int) -> int:
    """Okay: one thread per request a second."""
    return math.ceil(per_second / threads)
    """Better: requests in progress at once = per second x seconds each (Little's law)."""
    return math.ceil(per_second * seconds_each / threads)

Where it breaks: it is just enough for an average minute. At lunchtime, traffic doubles and requests queue. One crash, and you are short.

Best

Size for the peak, plus a spare

The idea: Like Better, at 2.5 times the average, plus one spare worker.

The waste in Better: it plans for the one minute that never causes trouble.

Highlighted lines are new in this rung.

import math

PEAK = 2.5  # busy times run 2 to 3 times the average


def workers_needed(per_second: float, seconds_each: float, threads: int) -> int:
    """Best: size for the peak, and keep one spare for a crash or a restart."""
    busy = per_second * PEAK * seconds_each
    return math.ceil(busy / threads) + 1

Counts

In progress at peakwas Requests in progress

Misses

Nothing in the taskwas Busy times, crashes

Why

Busy times run 2 to 3 times the average, and one worker can always be down.

What changed from the previous rung

import math

PEAK = 2.5  # busy times run 2 to 3 times the average


def workers_needed(per_second: float, seconds_each: float, threads: int) -> int:
    """Better: requests in progress at once = per second x seconds each (Little's law)."""
    return math.ceil(per_second * seconds_each / threads)
    """Best: size for the peak, and keep one spare for a crash or a restart."""
    busy = per_second * PEAK * seconds_each
    return math.ceil(busy / threads) + 1

When Best isn’t best: measure your real peak instead of using 2.5. And check memory: each worker process holds its own copy of the app.

Check yourself

A few quick questions. Ones you miss are remembered, so you can find your weak spots later.

1. Multiple choiceWhat does a WSGI server call?

   async def app(scope, receive, send)app(environ, start_response)app.run(host, port)app(request) -> response

   Check answer
2. Pick the patternWhich app gains the most from moving to ASGI?

   A site whose pages each take 5 ms of CPUAn image resizer that uses the CPU for 2 seconds a requestAn API whose requests each wait 3 seconds for another company’s APIA static blog

   Check answer
3. Spot the bugThis FastAPI app runs on Uvicorn. Under load, every request gets slow. Why?

   @app.get("/weather")
   async def weather(city: str):
       r = requests.get(f"https://api.example.com/{city}")
       return r.json()

   requests.get blocks the event loop, so no other request runs while it waitsThe route needs a trailing slashFastAPI can’t return JSON from an async functioncity should be an int

   Check answer
4. Predict the outputA WSGI app returns ["Hello"] (a str inside the list). What happens?

   The browser shows HelloIt breaks the WSGI rules: the body must be bytesThe server encodes it as UTF-8 for youThe browser shows ["Hello"]

   Check answer

Flashcards

Add these to your review deck. Each card comes back a little later every time you get it right.

• Card 1WSGI in one line?Show answer
• Card 2ASGI in one line?Show answer
• Card 3Which servers run which?Show answer
• Card 4Why can ASGI keep thousands of slow requests open?Show answer
• Card 5What breaks an ASGI app?Show answer

Add 5 cards to my review deck

Cheat sheet

• The server reads and writes HTTP. The app only answers requests.
• WSGI: app(environ, start_response) returns a list of bytes. Gunicorn runs Flask and Django this way.
• ASGI: async def app(scope, receive, send). Uvicorn runs FastAPI, and Django’s asgi.py, this way.
• Under WSGI, a waiting request holds a thread. Under ASGI, it gives the thread back at each await.
• Never block inside async def: it stalls every request.

Go deeper

• PEP 3333, the WSGI standard, and the ASGI specification.
• In Python: wsgiref, with a tiny server and a checker that tests an app against the rules.
• The servers: Gunicorn and Uvicorn.
• Next on the Frameworks roadmap: The request lifecycle, what happens inside the app between the server’s call and your view.

Can you do these now?

If you’re not sure of any of these, scroll back to that part before moving on.

• Say what a WSGI app and an ASGI app are, and what the server does for each
• Write a small WSGI app by hand, and wrap it in middleware
• Explain why waiting requests hold a thread in WSGI but not in ASGI
• Pick Gunicorn or Uvicorn for Flask, Django and FastAPI

✓Mark as completeI already know this

Status: Not started