p95 latency tripled after a deploy

New to how these answers are judged? Open the quick recap. It takes about two minutes.

Quick recapHow scenario answers are graded· 2 min

A weak answer jumps to a fix with no evidence. A decent answer is sound, but stops once the fire is out. A strong answer shows four things interviewers listen for: ownership (you act and follow through), calm triage (the worst harm first), evidence over guesses, and prevention (it won’t quietly happen again).

See the problem

It’s 9:45 on a Tuesday. Your team deployed at 9:30. Now an alert fires:

ALERT  p95 latency  shop-api  790 ms  (normal: about 250 ms)

Support says a few customers are complaining that checkout “hangs”. Your lead turns to you: “What do we do?”

In an interview, this question has no single right fix. It tests how you think under pressure: what you do first, what you look at, and what you do after the graph turns green.

Define the task

What you know

• A deploy went out at 9:30. The alert started at about 9:35.
• p95 latency went from about 250 ms to 790 ms: more than 3 times higher.
• p50 is still about 180 ms. The error rate is normal.

The words

• Latency: how long one request takes, from arriving to answering.
• p50 (the median): half the requests are this fast or faster.
• p95: 95 in 100 requests are this fast or faster. The slowest 1 in 20 are slower.
• p99: only 1 request in 100 is slower.
• Rolling back: putting the previous version back, so the new code stops running.

What the interviewer listens for: ownership, calm triage, evidence over guesses, and prevention.

Pause and think

⏸ Pause and think

Write down your first three steps, in order. For each one, say why it comes before the next.

Give it two minutes with pen and paper before you open a hint.

Hint 1 of 3

What changed just before it got slow?

Hint 2 of 3

Customers are hurting while you investigate. What stops that fastest?

Hint 3 of 3

p50 didn’t move, but p95 did. What does that tell you about which requests got slow?

Understand the idea

Each dot is one request, placed by how long it took. Step through to see where p50, p95 and p99 fall before the deploy, then what changes after it. Edit the numbers to try your own.

Where the slow requests land

⤢ Full view

Before the deploy, fastest first: 120, 135, 140, 150, 155, 160, 165, 170, 175, 180, 185, 190, 195, 200, 205, 210, 220, 230, 250, 260.

Step 1 of 8. 20 requests from before the deploy, one dot each, placed by how long it took: from 120 ms to 260 ms.

◀ BackPlayNext ▶Reset Ask me first Show as text

Try your own input

Before the deploy (ms)After the deploy (ms)Show it

Three requests out of 20 got about 3 times slower. That was enough to triple p95, while p50 stayed put and the average rose by less than half. Watch percentiles, not averages: the average hides the users who suffer most.

Climb the ladder

Here are three answers to “p95 latency tripled after this morning’s deploy. What do you do?”, with what the interviewer hears in each one.

Weak

Restart the servers

The idea: “I’d restart the servers and see if it goes away.”

• ✗ No impact check. Who is slow, and how slow? Is checkout affected?
• ✗ Ignores the obvious suspect. Something was deployed 15 minutes before the alert.
• ✗ A guess, not evidence. If things look better for a minute (fresh servers, empty queues), you learn the wrong lesson.

Customers hurt for

until someone finds the bug

Happens again?

yes, likely

Why

A restart doesn't undo the deploy, so the slow code keeps running.

What the interviewer hears: no ownership, no evidence, and no plan for next time.

Decent

Check the dashboards, roll back

The idea: “I’d check the dashboards, compare with the last release, and roll back if the deploy is to blame.”

• ✓ Looks at evidence. The timing points at the deploy.
• ✓ Stops the harm. Rolling back is the fastest fix there is.
• ✗ Doesn’t size the impact. Which requests, which customers, how much money?
• ✗ Stops when the graph is green. The bug is still in the code. Deploy it again tomorrow and it’s back.

Customers hurt for

minutes, until the rollback

Happens again?

maybe: nothing stops it

Why

Rolling back stops the harm fast, but the cause is still unknown and nothing guards against it.

What’s missing: follow-through. Find the cause, fix it with a test, and make sure the next one is caught before customers notice.

Strong

Stop the harm, find the cause, fix it, prevent it

The idea: Confirm the impact, roll back first, find the cause with evidence, fix it with a test, write a blameless postmortem and add a guardrail.

1. Confirm the impact. p95 went from 250 to 790 ms at 9:30; p50 and errors didn’t change. Grouped by endpoint, only POST /checkout got slower. Checkout is where the money comes in, so it’s urgent.
2. Stop the harm first. Roll back the 9:30 deploy, or turn off its feature flag. Watch p95 fall back to about 250 ms. Tell the team what you did and why.
3. Find the cause with evidence. A trace of one slow checkout shows 41 database queries where there used to be 1. The diff: the new code loads each cart item’s product in its own query. A 40-item cart makes 40 extra queries of about 15 ms each: about 600 ms. That’s the N+1 pattern. Only big carts were slow, which is why p50 didn’t move.
4. Fix it, with a test. Load the cart in one query with a JOIN, and add a test that counts the queries: 1 for any cart, not 1 + N. The code below has both, and the guardrail count_queries.
5. Write a blameless postmortem. What happened, when, who was affected, why the tests missed it (the test carts had 3 items), and what changes. It’s about the system, not the person who deployed.
6. Add a guardrail. A canary deploy sends 5% of traffic to the new version first and rolls back on its own if p95 jumps. Add a p95 alert for each endpoint, and run the query-count test on every change.

"""The deploy's bug, shrunk: checkout went from 1 query to 1 + N.

`load_cart_slow` looks up each cart line's product in its own query (the
N+1 pattern), so a 40-item cart makes 41 queries. `load_cart` gets the same
rows in one query with a JOIN. `count_queries` is the guardrail: a test can
assert how many queries checkout makes."""

import sqlite3
from collections.abc import Iterator
from contextlib import contextmanager


def load_cart_slow(db: sqlite3.Connection, cart_id: int) -> list[dict]:
    lines = db.execute(
        "SELECT product_id, quantity FROM cart_lines WHERE cart_id = ? ORDER BY product_id",
        (cart_id,),
    ).fetchall()
    cart = []
    for product_id, quantity in lines:
        name, price = db.execute(  # one more query for every line: N+1
            "SELECT name, price FROM products WHERE product_id = ?", (product_id,)
        ).fetchone()
        cart.append({"name": name, "price": price, "quantity": quantity})
    return cart


def load_cart(db: sqlite3.Connection, cart_id: int) -> list[dict]:
    rows = db.execute(
        "SELECT p.name, p.price, l.quantity"
        " FROM cart_lines AS l JOIN products AS p ON p.product_id = l.product_id"
        " WHERE l.cart_id = ? ORDER BY l.product_id",
        (cart_id,),
    )
    return [{"name": name, "price": price, "quantity": qty} for name, price, qty in rows]


@contextmanager
def count_queries(db: sqlite3.Connection) -> Iterator[list[int]]:
    """Count the SQL statements run inside the block.

    Use it as `with count_queries(db) as n: ...`, then read n[0]."""
    seen = [0]

    def trace(_statement: str) -> None:
        seen[0] += 1

    db.set_trace_callback(trace)
    try:
        yield seen
    finally:
        db.set_trace_callback(None)

Customers hurt for

minutes, and the next one is caught early

Happens again?

unlikely: a test and an alert guard it

Why

Each step answers one question in order: how bad is it, how do we stop it, why did it happen, and how do we stop it coming back?

Keep it short in the interview: say the order out loud, spend most of your time on stopping the harm and finding the cause, and name one or two guardrails. Don’t list every step in detail.

The fix in load_cart is the same one Django’s select_related makes for you: fetch the related rows in one query instead of one per row.

Write it yourself

The first clue in step 3 was “only checkout got slower”. Write the function that finds it: given requests from before and after the deploy, return the endpoint whose p95 grew the most.

Find the endpoint that got slower

Ctrl + Enter runs

import math def percentile(times, p): """Nearest rank: the smallest time that p% of requests were at or under.""" ordered = sorted(times) rank = max(1, math.ceil(p / 100 * len(ordered))) return ordered[rank - 1] def regressed_endpoint(before, after): # before and after are lists of [endpoint, ms] pairs. # Return [endpoint, p95 before, p95 after] for the endpoint # whose p95 grew the most (after ÷ before), or None if # no endpoint got slower. Skip endpoints that appear on # only one side. return None

Run testsVisualizeReset codeLoads about 6 MB the first time. Best on Wi-Fi.

1. ○Not run yet: regressed_endpoint([['GET /home', 90], ['GET /home', 110], ['POST /checkout', 200], ['POST /checkout', 240]], [['GET /home', 95], ['GET /home', 105], ['POST /checkout', 210], ['POST /checkout', 780]]) → ['POST /checkout', 240, 780]
2. ○Not run yet: regressed_endpoint([['GET /home', 90], ['GET /home', 110]], [['GET /home', 85], ['GET /home', 110]]) → None
3. ○Not run yet: Extra test
4. ○Not run yet: Extra test
5. ○Not run yet: Extra test

Extra tests check tricky cases. You see their input only if they fail.

Try your own input

Arguments, separated by commas as in Python

regressed_endpoint()Run onceVisualize

Compare

What interviewers listen for	Weak	Decent	Strong
Ownership	No: hopes it goes away	Partly: acts, then stops	Yes: acts, follows through, tells the team
Calm triage	No: no impact check	Partly: rolls back	Yes: sizes the impact, stops the harm first
Evidence over guesses	No	Partly: dashboards and timing	Yes: by endpoint, a trace, the diff
Prevention	No	No	Yes: a test, a canary, an alert, a postmortem

Common mistakes and edge cases

!Averages hide the slowest users

In the example, the average went from 185 to 268 ms. It looks like a small change, but 3 customers in 20 waited nearly a second. Alert on p95 or p99 for each endpoint, not on the average.

• Rolling back can break things too. If the deploy changed the database (a new column, a renamed table), the old code may not work with it. Make database changes that both versions can live with, and remove the old parts in a later deploy.
• The deploy might not be the cause. Check what else changed at 9:30: a config push, a traffic spike, a slow third-party service, a nightly job running late. If the rollback doesn’t help, look again.
• Don’t average percentiles. The p95 of all servers isn’t the average of each server’s p95. Combine the raw times or histograms, then take the percentile.
• Too few requests mislead. With 20 requests, p99 is just the slowest one. Real dashboards use thousands of requests over a few minutes.
• Save evidence before you restart. A restart clears memory, open connections and in-flight work. Take a trace, a thread dump or the slow-query log first if it’s quick.

Say it in the interview

Questions to ask first

• Who is affected: every request, one endpoint, one region, some customers?
• Did errors go up too, or only latency?
• What else changed around 9:30, besides the deploy?
• Can we roll back safely? Did the deploy change the database?

A 60-second answer

“First I’d confirm the impact: p95 tripled from 250 to 790 ms right after the 9:30 deploy, while p50 stayed at 180. So a small group of requests got much slower. If it’s checkout, it’s urgent. I’d stop the harm first: roll back, or turn off the feature flag, and watch p95 recover. Then I’d find the cause with evidence. Latency by endpoint shows only checkout moved; a trace shows 41 queries instead of 1; the diff added a query per cart item, an N+1. I’d fix it with a join and a test that counts the queries. Then a blameless postmortem, and guardrails: canary deploys that roll back on a p95 jump, and latency alerts per endpoint.”

Likely follow-ups

• The rollback didn’t help? Then the deploy probably isn’t the cause. Look at what else changed (traffic, a dependency, the database), and compare slow requests with fast ones.
• You can’t roll back? Turn off the feature flag, or fix forward with a small, reviewed change. Next time, make database changes that the old code can still use.
• No tracing tool? Log the time around each database call, count queries per request, or read the database’s slow-query log.
• Why p95 and not the average? The average mixes 17 normal requests with 3 slow ones, so it barely moves. p95 shows what the slowest 1 in 20 customers feel.

Red flags interviewers notice

• Restarting or adding servers before looking at any data.
• Debugging for an hour while customers are still suffering.
• Blaming the person who deployed.
• Stopping when the graph is green, with no test and no guardrail.

Practice

Four more incidents, from easy to hard. There’s no code to run: say your answer out loud, in order, then compare it with the weak, decent and strong answers.

1. The error rate jumped (easy)

At 14:05, someone changes a config value. By 14:07, 8% of requests fail with 500 errors. Normally it’s 0.1%. No code was deployed.

Hint 1 of 3

What changed right before it started?

Hint 2 of 3

Which is faster to undo: a config change or a code fix?

Hint 3 of 3

Do the failing requests share something: an endpoint, a server, a dependency?

Show the weak, decent and strong answers

Weak

Search the code for a bug

The idea: “I’d look through the recent code changes to find the bug.”

✗ Ignores the change that lines up with the errors. ✗ Doesn’t stop the harm first.

Customers hurt for

until the bug is found

Happens again?

yes

Why

No code changed. The config did, and every minute of searching is 8% of requests failing.

Decent

Revert the config

The idea: “I’d revert the config change and check that errors go back to normal.”

✓ Evidence (the timing) and fast mitigation. ✗ No cause, no guardrail.

Customers hurt for

minutes

Happens again?

maybe

Why

It stops the harm fast, but nobody learns why one value could break 8% of requests.

Strong

Revert, read the errors, check config before it ships

The idea: Revert the config and watch errors fall, then read the error logs to see what failed, and make config changes as safe as code changes.

The logs show “connection refused” on the payment call: the new value pointed at the wrong host. Fix: check config against a schema, test connections when the app starts, and roll config out to one server first. Then a blameless postmortem.

Customers hurt for

minutes

Happens again?

unlikely

Why

Config gets the same care as code: checked, reviewed and rolled out a little at a time.

2. Database CPU at 100% every morning (medium)

Every weekday at 9:00, the database’s CPU hits 100% for about 20 minutes, and every page gets slow. Nothing was deployed this week.

Hint 1 of 3

What happens at 9:00 every weekday that doesn’t happen at 3:00?

Hint 2 of 3

The database can list the queries that took the most time in total.

Hint 3 of 3

Is it one expensive query, or a cheap one run millions of times?

Show the weak, decent and strong answers

Weak

Buy a bigger database

The idea: “I’d move to a bigger database server.”

✗ No evidence of what uses the CPU. ✗ The same query will soon be too slow for the bigger machine too.

Customers hurt for

less, for a while

Happens again?

yes, as data grows

Why

Bigger hardware hides the cause and costs more every month.

Decent

Fix the slowest query

The idea: “I’d find the slowest query in the slow-query log and add an index for it.”

✓ Looks at data. ✗ Doesn’t ask what is special about 9:00.

Customers hurt for

maybe less

Happens again?

maybe

Why

Evidence-based, but the slowest single query isn't always the one using the most CPU.

Strong

Find what runs at 9:00, by total time

The idea: Rank queries by total time during 9:00 to 9:20 (calls × time per call). Find the cause, fix it, and move heavy reports off the main database.

The top query by total time is a morning report that scans every order for each of 5,000 users. Add an index, run the report from a read replica or at night, and cache its result. Alert on database CPU and on new slow queries.

Customers hurt for

no more 9:00 slowdowns

Happens again?

unlikely

Why

Total time shows the real cost; keeping reports away from the primary protects customers.

3. Intermittent 502s (medium)

About 2% of requests get 502 Bad Gateway from the load balancer. It happens all day, on every server, and you can’t make it happen on your laptop.

Hint 1 of 3

502 means the load balancer got a bad answer, or no answer, from an app server.

Hint 2 of 3

Do the 502s line up with deploys, restarts, or idle connections being reused?

Hint 3 of 3

Compare how long the load balancer keeps an idle connection open with how long the app server does.

Show the weak, decent and strong answers

Weak

Retry in the client

The idea: “I’d make the client retry failed requests.”

✗ No evidence. ✗ Treats the symptom, and can cause new harm.

Customers hurt for

hidden, not fixed

Happens again?

yes

Why

Retries hide the bug, add load, and can charge a customer twice if a payment is retried.

Decent

Find the bad servers and restart them

The idea: “I’d check the load balancer logs for which servers return 502s, and restart them.”

✓ Looks at the logs. ✗ Stops before finding the pattern.

Customers hurt for

until it comes back

Happens again?

yes

Why

Uses evidence, but the logs show every server is affected, so a restart changes nothing.

Strong

Find the pattern, fix the timeouts

The idea: Group the 502s by server and by connection: they hit every server, on idle connections being reused. Match the timeouts, and make deploys drain connections.

The app server closes idle connections after 5 seconds, but the load balancer keeps them for 60. Now and then it sends a request down a connection the app just closed. Set the app’s keep-alive longer than the load balancer’s, check that deploys finish in-flight requests before stopping, and alert on the 502 rate.

Customers hurt for

fixed

Happens again?

unlikely

Why

With the app's idle timeout longer than the load balancer's, the race goes away.

4. A cache stampede on the hour (hard)

Every hour, on the hour, database load spikes for a minute and p99 jumps to 5 seconds. Every cache entry is set to expire after exactly 1 hour.

Hint 1 of 3

What happens to all the cache entries at the same moment?

Hint 2 of 3

When a popular entry expires, how many requests try to rebuild it at once?

Hint 3 of 3

Could entries expire at slightly different times, or be refreshed before they expire?

Show the weak, decent and strong answers

Weak

Cache for longer

The idea: “I’d raise the cache time to 24 hours.”

✗ Doesn’t explain why the spike happens. ✗ Trades one problem for two.

Customers hurt for

once a day instead

Happens again?

yes, and data can be out of date

Why

It moves the spike, makes it bigger, and serves old data for longer.

Decent

Spread out the expiry times

The idea: “I’d add a random few minutes to each entry’s expiry, so they don’t all expire at once.”

✓ Finds the cause: everything expires together. ✗ Misses the hot-key stampede.

Customers hurt for

much smaller spikes

Happens again?

a popular key still can

Why

It spreads the expiries out, but one popular entry still sends thousands of misses to the database at the same moment.

Strong

Spread expiries, rebuild each key once

The idea: Add jitter to expiry times. Let only one request rebuild a key while the others get the old value. Refresh popular keys before they expire.

The “only one rebuilds” part is a short lock per key (sometimes called single flight). Alert on the cache hit rate, so a drop shows up before the database feels it.

Customers hurt for

no spikes

Happens again?

unlikely

Why

No moment when everything expires, and no key is rebuilt by thousands of requests at once.

Check yourself

A few quick questions. Ones you miss are remembered, so you can find your weak spots later.

1. Fill in the answerYou line up 20 request times from fastest to slowest. Which one is p95? (Type its position, like 7.)

   Your answerCheck answer
2. Multiple choiceAfter a deploy, p50 stays at 180 ms but p95 goes from 250 to 790 ms. What does that tell you?

   Every request got a little slowerMost requests are fine, and a small group got much slowerThe servers are overloadedThe dashboard is broken

   Check answer
3. Multiple choiceCheckout is slow right after this morning's deploy, and customers are complaining. What do you do first?

   Read the diff until you find the bugRestart the serversRoll back the deploy, or turn off its feature flag, then investigateAdd more servers

   Check answer
4. Spot the bugThis code was fine in tests, where carts had 3 items. Why is it slow for a cart with 40?

   def load_cart(db, cart_id):
       lines = db.execute(
           "SELECT product_id, quantity FROM cart_lines"
           " WHERE cart_id = ?", (cart_id,)
       ).fetchall()
       cart = []
       for product_id, quantity in lines:
           name, price = db.execute(
               "SELECT name, price FROM products"
               " WHERE product_id = ?", (product_id,)
           ).fetchone()
           cart.append({"name": name, "price": price,
                        "quantity": quantity})
       return cart

   fetchall() loads too many rowsIt runs one more query for every item: 41 queries for 40 itemsThe dict is slow to buildproduct_id needs an index

   Check answer
5. Pick the patternWhich guardrail would have caught this N+1 before customers did?

   A bigger databaseA test that asserts checkout makes 1 query, whatever the cart sizeRestarting the servers every nightAn alert on average latency across the whole site

   Check answer

Flashcards

Add these to your review deck. Each card comes back a little later every time you get it right.

• Card 1What do p50, p95 and p99 mean?Show answer
• Card 2Why watch p95 instead of the average?Show answer
• Card 3The order of a strong incident answer?Show answer
• Card 4What is a canary deploy?Show answer
• Card 5What makes a postmortem blameless?Show answer

Add 5 cards to my review deck

Cheat sheet

• Order: confirm the impact, stop the harm, find the cause, fix with a test, postmortem, guardrail.
• Stop the harm first: roll back, or turn off the feature flag. Investigate after.
• p95 moves when a few requests get much slower. p50 and the average barely notice.
• Narrow it down: which endpoint, which requests, what they share, what the deploy changed.
• Guardrails: canary deploys, per-endpoint p95 alerts, a test that counts queries.
• Blameless: the postmortem is about the system, not the person.

Go deeper

• Monitoring distributed systems, from Google’s SRE book: the four golden signals, and why to watch percentiles rather than averages.
• Postmortem culture, from the same book: how to write a blameless postmortem.
• Next on the Scenarios roadmap: The error rate jumped and Production debugging.

Can you do these now?

If you’re not sure of any of these, scroll back to that part before moving on.

• Read p50, p95 and p99, and say why p95 jumped when the average barely moved
• Stop the harm first, by rolling back or turning off a flag, before you hunt for the cause
• Find the cause with evidence, from which requests got slow to what the deploy changed
• Fix it with a test, then add a guardrail so it can't quietly happen again

✓Mark as completeI already know this

Status: Not started