I removed the bad readings, but one is still there
New to lists, sets and dicts? Open the quick recap. It takes about two minutes.
Quick recapLists, sets and dicts in two minutes· 2 min
A list keeps items in order, and each has a position, starting at 0. A set keeps each item once, in no
order, and answers x in s in about one step. A dict maps keys to values.
readings = [5, -1, 7]readings.append(2) # [5, -1, 7, 2]readings.remove(-1) # takes out the first -1: [5, 7, 2]seen = {"a", "b"}"a" in seen # True, in about one stepages = {"ann": 31}ages["bo"] = 28 # {"ann": 31, "bo": 28}See the problem
Section titled “See the problem”You’re cleaning sensor readings. A reading below 0 means the sensor failed, so you remove those:
readings = [5, -1, -2, 7]for r in readings: if r < 0: readings.remove(r)print(readings)You expect [5, 7]. Python prints:
[5, -2, 7]No error, no warning. -1 is gone, but -2 is still there. With [5, -1, 7, -2] it works, which makes it worse: the bug hides until two errors sit side by side. Why?
Define the task
Section titled “Define the task”Write drop_errors(readings).
- Input: a list of whole-number readings, up to 100,000 of them. Below 0 is an error.
- Output: a new list of the readings that are 0 or more, in the same order.
- The hard part:
readingsitself must not change. Other code still uses it, for example to count the errors.
Example: drop_errors([5, -1, -2, 7]) gives [5, 7], and readings is still [5, -1, -2, 7].
Pause and think
Section titled “Pause and think”Hints open one at a time:
Hint 1 of 3
A for loop over a list keeps a position: 0, then 1, then 2. It doesn’t notice when items move.
Hint 2 of 3
remove() moves every item after the removed one a place to the left. Where does -2 land?
Hint 3 of 3
Don’t change the list you’re looping over. Build a new list of the readings you keep.
Understand the idea
Section titled “Understand the idea”A for loop over a list keeps its place by position. It takes the item at position 0, then 1, then 2, and stops when the position passes the end. It never notices that items moved.
readings.remove(-1) takes -1 out of position 1, and every reading after it moves one place left. -2 lands in position 1, which the loop has already done. The loop moves on to position 2 and takes 7. -2 is never checked.
Watch it happen. The loop arrow is the loop’s place in the list. The player stops to ask what comes next.
- 50
- -11
- -22
- 73
Step 1 of 10. readings holds 4 readings. 2 are below 0: errors. The loop should remove them.
readings = [5, -1, -2, 7] for r in readings: if r < 0: readings.remove( r) print( readings)
A list comprehension is a loop that builds a list, folded into one expression. These two do the same:
kept = []for r in readings: if r >= 0: kept.append(r)
kept = [r for r in readings if r >= 0]It has three parts: what to put in (r), where items come from (for r in readings), and which ones to keep (if r >= 0). The part written first runs last. For each reading, for runs, then if, and only then the part before for.
Below, the comprehension is written over several lines, one part per line, so you can see which part runs. Type your own readings under the picture.
- 50
- -11
- -22
- 73
kept = []
Step 1 of 13. The comprehension starts a new, empty list. It reads readings, but never changes it.
readings = [5, -1, -2, 7] kept = [ r for r in readings if r >= 0 ] print( kept, readings)
It looks at each reading once, never moves anything, and leaves readings as it was.
The brackets say what you get:
| Written | Makes | Example |
|---|---|---|
[x for x in items] | a list | [n * n for n in range(4)] gives [0, 1, 4, 9] |
{x for x in items} | a set: each item once | {w.lower() for w in words} |
{k: v for x in items} | a dict | {w: len(w) for w in words} |
(x for x in items) | a generator: one item at a time, no list | sum(n * n for n in nums) |
Climb the ladder
Section titled “Climb the ladder”All three give back a new list, so readings never changes. What changes is how much work each one does.
Remove from a copy
The idea: Copy readings, loop over the original, and remove each error from the copy.
The loop runs over readings, which never changes, so nothing is skipped.
def drop_errors(readings: list[int]) -> list[int]: """A new list of the readings that are 0 or more, in order. readings doesn't change.""" kept = readings.copy() for r in readings: if r < 0: kept.remove(r) # searches kept from the front, then moves the rest left return kept- Time
O(n²)- Space
O(n)- Why
- Each remove() searches the copy from the front, then moves every reading after it one place left.
Where it breaks: speed. With 100,000 readings, half of them errors, each remove() walks past thousands of readings: billions of steps in all. The speed test stops it.
A loop that appends
The idea: Start with an empty list, and append each reading that is 0 or more.
The waste in Okay: it copies every reading, then searches and shifts the copy for each error, when it could add only the readings it keeps.
Nothing moves, so each reading costs one step.
Highlighted lines are new in this rung.
def drop_errors(readings: list[int]) -> list[int]: """A new list of the readings that are 0 or more, in order. readings doesn't change.""" kept: list[int] = [] for r in readings: if r >= 0: kept.append(r) return kept- Time
O(n)was O(n²)- Space
O(n)- Why
- One look at each reading, and one append for each one kept.
What changed from the previous rung
def drop_errors(readings: list[int]) -> list[int]: """A new list of the readings that are 0 or more, in order. readings doesn't change.""" kept = readings.copy() kept: list[int] = [] for r in readings: if r < 0: kept.remove(r) # searches kept from the front, then moves the rest left if r >= 0: kept.append(r) return keptWhere it breaks: nothing is wrong when it runs. But it takes four lines to say “the readings that are 0 or more”, and kept is only half built until the loop ends.
A list comprehension
The idea: Write [r for r in readings if r >= 0].
The waste in Better: four lines, and a look-up of kept.append for every reading, when one line says what you mean.
The comprehension adds each item with a built-in step instead of calling append, so it’s a little faster too. And r stays inside the comprehension: it doesn’t exist after the line.
Highlighted lines are new in this rung.
def drop_errors(readings: list[int]) -> list[int]: """A new list of the readings that are 0 or more, in order. readings doesn't change.""" return [r for r in readings if r >= 0]- Time
O(n)- Space
O(n)- Why
- The same single pass, in one line.
What changed from the previous rung
def drop_errors(readings: list[int]) -> list[int]: """A new list of the readings that are 0 or more, in order. readings doesn't change.""" kept: list[int] = [] for r in readings: if r >= 0: kept.append(r) return kept return [r for r in readings if r >= 0]When Best isn’t best: if other code holds the same list and must see it change, replace what’s in it: readings[:] = [r for r in readings if r >= 0]. If you loop over the result only once, drop the brackets for a generator: (r for r in readings if r >= 0) gives one reading at a time and builds no list. And when each item needs several steps, or a try, write the loop.
Write it yourself
Section titled “Write it yourself”Now write it in the page. Press Run tests to check your code. Python runs in your browser, so nothing is sent anywhere. Your code is saved in this browser for next time.
Try Okay from the ladder first: it passes every test but the speed one.
Drop the error readings
Ctrl + Enter runs- Not run yet:
drop_errors([5, -1, -2, 7])→[5, 7] - Not run yet:
drop_errors([0, 3, 0])→[0, 3, 0] - Not run yet:
drop_errors([-4, -4])→[] - Not run yet:
readings = [2, -1]; drop_errors(readings); readings→[2, -1] - Not run yet: Extra test
- Not run yet: Extra test
- Not run yet: Extra test
Extra tests check tricky cases. You see their input only if they fail.
Try your own input
Compare
Section titled “Compare”| Way | Time | Extra memory | Good | Not so good | Use it when |
|---|---|---|---|---|---|
| Okay: remove from a copy | O(n²) | O(n) | Fixes the skipping | Each remove() searches and shifts the list | Hardly ever: a few items, in a hurry |
| Better: a loop that appends | O(n) | O(n) | Fast; room for several steps per item | Four lines for a simple rule | Each item needs several steps, a try or a print |
| Best: a list comprehension | O(n) | O(n) | One clear line; a little faster | Hard to read with a long condition or many fors | You build a list from a simple rule |
Common mistakes and edge cases
Section titled “Common mistakes and edge cases”- Two
fors read like the loops:[x for row in grid for x in row]meansfor row in grid:, thenfor x in row:. Written the other way round,rowdoesn’t exist yet, and Python stops with aNameError. ifat the end filters,if/elseat the front chooses:[n for n in nums if n > 0]drops items.[n if n > 0 else 0 for n in nums]keeps them all.[n for n in nums if n > 0 else 0]is aSyntaxError.- Not for side effects:
[print(r) for r in readings]prints, then builds a list ofNonethat nobody uses. When you want the effect, not the list, write aforloop. - A dict comprehension keeps the last value:
{e.lower(): e for e in emails}keeps the last spelling for each key, though the key stays where it first appeared. - A generator runs out:
squares = (n * n for n in nums)can be looped over once. A secondsum(squares)gives 0. - The loop variable stays inside: after
[n * n for n in nums],ndoesn’t exist. A comprehension has its own scope, unlike aforloop. The scope lesson shows why. - Two
fors at most: past that, or with a long condition, write the loop. Clear beats short.
Say it in the interview
Section titled “Say it in the interview”Questions to ask first
- May I return a new list, or must this same list change, because other code holds it?
- Should the order stay the same?
- How many items can there be, and do they all fit in memory?
A 60-second walkthrough
“Removing from a list while looping over it skips items: the loop keeps a position, and
remove()moves the next item back into a place the loop has done. So I don’t change the list. I build a new one:[r for r in readings if r >= 0]looks at each reading once, so it’s O(n) time, plus O(n) for the new list. Removing from a copy also works, but eachremove()is O(n), so that’s O(n²). If the caller needs the same list changed, I assign the result toreadings[:]. If I only loop over the result once, a generator expression avoids building the list.”
Likely follow-ups
- List comprehension or generator expression? A list when you need it more than once, or its length. A generator when you loop over it once, above all for big data.
- Why is a comprehension faster than a loop with append? It adds items with a built-in step, without looking up and calling
appendeach time. The gain is small: say it reads better first. - Does the loop variable leak out? Not in Python 3: a comprehension has its own scope. In Python 2, a list comprehension’s variable did leak.
- How else could you remove items in place? Loop backwards over the positions and
deleach error: a removal then only moves items the loop has already done. It doesn’t skip, but eachdelstill shifts the list, so many removals cost O(n²).
Practice
Section titled “Practice”Three problems, from easy to hard. Each has hints, tests you can run here, and its own Okay, Better and Best answers. Try before you open them.
1. Flatten the pages (easy)
Section titled “1. Flatten the pages (easy)”A search API gives results in pages: a list of pages, each a list of titles. Write flatten(pages), which gives back one new list with every title, in order. There may be up to 50,000 pages.
Example: flatten([["a", "b"], ["c"]]) gives ["a", "b", "c"].
Hint 1 of 3
Write it as two loops first: for each page, then for each title in that page.
Hint 2 of 3
A comprehension can have two fors. They go in the same order as the loops.
Hint 3 of 3
sum(pages, []) works, but each + copies every title so far.
Flatten the pages
Ctrl + Enter runs- Not run yet:
flatten([["a", "b"], ["c"]])→['a', 'b', 'c'] - Not run yet:
flatten([])→[] - Not run yet:
flatten([[], ["a"], []])→['a'] - Not run yet: Extra test
- Not run yet: Extra test
- Not run yet: Extra test
Extra tests check tricky cases. You see their input only if they fail.
Try your own input
Show the Okay, Better and Best answers
Add the pages up with sum
The idea: sum(pages, []) joins the lists with +, starting from an empty list.
One short line, and it gives the right answer.
def flatten(pages: list[list[str]]) -> list[str]: """Every title from every page, in order, in one new list.""" return sum(pages, []) # noqa: RUF017 (each + copies every title so far)- Time
O(n × p), p = pages- Space
O(n)- Why
- Each + makes a new list and copies every title so far, once per page.
Where it breaks: speed. a + b makes a new list and copies both, so each page copies every title before it. 50,000 pages of 2 titles copy about 2.5 billion titles. Ruff flags it as RUF017.
A loop with extend
The idea: Start with an empty list, and extend it with each page.
The waste in Okay: it copies the titles it already has again for every page, when each title needs adding only once.
extend adds a page’s titles to the end of the same list.
Highlighted lines are new in this rung.
def flatten(pages: list[list[str]]) -> list[str]: """Every title from every page, in order, in one new list.""" titles: list[str] = [] for page in pages: titles.extend(page) return titles- Time
O(n)was O(n × p)- Space
O(n)- Why
- Each title is copied once.
What changed from the previous rung
def flatten(pages: list[list[str]]) -> list[str]: """Every title from every page, in order, in one new list.""" return sum(pages, []) # noqa: RUF017 (each + copies every title so far) titles: list[str] = [] for page in pages: titles.extend(page) return titlesA comprehension with two fors
The idea: One comprehension with two fors: for each page, then for each title in it, put in the title.
The waste in Better: a list to set up and a loop to follow, when one line says it.
The fors go in the same order as the loops: each page first, then each title in that page.
Highlighted lines are new in this rung.
def flatten(pages: list[list[str]]) -> list[str]: """Every title from every page, in order, in one new list.""" return [title for page in pages for title in page]- Time
O(n)- Space
O(n)- Why
- Each title is copied once.
What changed from the previous rung
def flatten(pages: list[list[str]]) -> list[str]: """Every title from every page, in order, in one new list.""" titles: list[str] = [] for page in pages: titles.extend(page) return titles return [title for page in pages for title in page]When Best isn’t best: if you loop over the titles only once, itertools.chain.from_iterable(pages) gives them one at a time without building a list.
2. Rows to columns (medium)
Section titled “2. Rows to columns (medium)”A spreadsheet gives a table as rows, but a chart needs it as columns. Write to_columns(rows): column c holds item c of every row, top to bottom. Every row has the same length, and a table with no rows gives [].
Example: to_columns([[1, 2, 3], [4, 5, 6]]) gives [[1, 4], [2, 5], [3, 6]].
Hint 1 of 3
[[0] * 2] * 3 makes one list with three names for it. Change one column, and they all change.
Hint 2 of 3
Build each column whole: [row[c] for row in rows] is column c.
Hint 3 of 3
zip(*rows) walks the rows together: item 0 of every row, then item 1, and so on.
Rows to columns
Ctrl + Enter runs- Not run yet:
to_columns([[1, 2, 3], [4, 5, 6]])→[[1, 4], [2, 5], [3, 6]] - Not run yet:
to_columns([[1, 2]])→[[1], [2]] - Not run yet:
to_columns([])→[] - Not run yet: Extra test
- Not run yet: Extra test
- Not run yet: Extra test
- Not run yet: Extra test
Extra tests check tricky cases. You see their input only if they fail.
Try your own input
Show the Okay, Better and Best answers
Fill a grid made with *
The idea: Make the columns with [[0] * len(rows)] * len(rows[0]), then fill them in with two loops.
It looks right: the right number of columns, each the right length.
def to_columns(rows: list[list[int]]) -> list[list[int]]: """Column c holds item c of every row, top to bottom.""" columns = [[0] * len(rows)] * len(rows[0]) # one list, repeated: every column is the same list for r, row in enumerate(rows): for c, value in enumerate(row): columns[c][r] = value return columns- Time
O(r × c)- Space
O(r × c)- Why
- Each of the r × c values is set once.
Where it breaks: * len(rows[0]) repeats one list, so every column is the same list. Each value lands in all of them, and the last row’s values win: [[3, 6], [3, 6], [3, 6]]. And rows[0] fails on a table with no rows.
A comprehension inside a comprehension
The idea: Column c is [row[c] for row in rows]. Make one for each c.
The waste in Okay: it makes the columns first and fills them in later, when each column can be built whole.
The outer comprehension makes a new list for each column, so no two columns share one.
Highlighted lines are new in this rung.
def to_columns(rows: list[list[int]]) -> list[list[int]]: """Column c holds item c of every row, top to bottom.""" return [[row[c] for row in rows] for c in range(len(rows[0]))]- Time
O(r × c)- Space
O(r × c)- Why
- Each value is copied once.
What changed from the previous rung
def to_columns(rows: list[list[int]]) -> list[list[int]]: """Column c holds item c of every row, top to bottom.""" columns = [[0] * len(rows)] * len(rows[0]) # one list, repeated: every column is the same list for r, row in enumerate(rows): for c, value in enumerate(row): columns[c][r] = value return columns return [[row[c] for row in rows] for c in range(len(rows[0]))]Where it breaks: rows[0] still fails on a table with no rows, with an IndexError. It needs an extra if not rows: return [].
zip(*rows)
The idea: zip(*rows, strict=True) gives each column as a tuple. Turn each one into a list.
The waste in Better: it counts positions by hand and needs a special case for no rows, when zip walks the rows together.
*rows passes each row to zip as its own argument. With no rows, zip() gives nothing, so an empty table just works. strict=True stops with a ValueError if the rows differ in length, instead of quietly dropping items.
Highlighted lines are new in this rung.
def to_columns(rows: list[list[int]]) -> list[list[int]]: """Column c holds item c of every row, top to bottom.""" return [list(column) for column in zip(*rows, strict=True)]- Time
O(r × c)- Space
O(r × c)- Why
- Each value is copied once.
What changed from the previous rung
def to_columns(rows: list[list[int]]) -> list[list[int]]: """Column c holds item c of every row, top to bottom.""" return [[row[c] for row in rows] for c in range(len(rows[0]))] return [list(column) for column in zip(*rows, strict=True)]When Best isn’t best: if the chart reads each column only once, zip(*rows, strict=True) on its own is enough: it gives tuples, and builds them one at a time.
3. Keep the first of each (hard)
Section titled “3. Keep the first of each (hard)”A sign-up form collected emails, and some people sent theirs more than once, with different capitals. Write first_of_each(emails): each email once, where two emails are the same if they match in lowercase. Keep the first spelling of each, in the order they first appeared. There may be up to 50,000 emails.
Example: first_of_each(["ann@x.com", "Bo@x.com", "ANN@x.com"]) gives ["ann@x.com", "Bo@x.com"].
Hint 1 of 3
Checking each email against every email before it is a loop inside a loop.
Hint 2 of 3
A set answers “seen this before?” in about one step.
Hint 3 of 3
Each answer depends on the emails before it. A comprehension is best when each item is decided on its own.
Keep the first of each
Ctrl + Enter runs- Not run yet:
first_of_each(["ann@x.com", "Bo@x.com", "ANN@x.com"])→['ann@x.com', 'Bo@x.com'] - Not run yet:
first_of_each(["Bo@x.com", "bo@x.com"])→['Bo@x.com'] - Not run yet:
first_of_each([])→[] - Not run yet: Extra test
- Not run yet: Extra test
- Not run yet: Extra test
Extra tests check tricky cases. You see their input only if they fail.
Try your own input
Show the Okay, Better and Best answers
Look back through the list
The idea: Keep an email if its lowercase isn’t among the lowercased emails before it.
One comprehension, and it gives the right answer.
def first_of_each(emails: list[str]) -> list[str]: """Each email once, ignoring case: the first spelling, in the order first seen.""" return [e for i, e in enumerate(emails) if e.lower() not in [x.lower() for x in emails[:i]]]- Time
O(n²)- Space
O(n)- Why
- For each email, it lowercases and searches every email before it.
Where it breaks: speed. For the 50,000th email, it lowercases and checks the 49,999 before it: over a billion steps in all.
A set, filled inside the comprehension
The idea: Keep a set of lowercased emails seen so far, and add to it inside the if.
The waste in Okay: it searches the list before each email, when a set answers “seen this before?” in about one step.
seen.add() gives back None, which counts as false. So e.lower() in seen or seen.add(e.lower()) checks the set, and adds a new email to it.
Highlighted lines are new in this rung.
def first_of_each(emails: list[str]) -> list[str]: """Each email once, ignoring case: the first spelling, in the order first seen.""" seen: set[str] = set() return [e for e in emails if not (e.lower() in seen or seen.add(e.lower()))]- Time
O(n)was O(n²)- Space
O(n)- Why
- One set look-up, and at most one add, for each email.
What changed from the previous rung
def first_of_each(emails: list[str]) -> list[str]: """Each email once, ignoring case: the first spelling, in the order first seen.""" return [e for i, e in enumerate(emails) if e.lower() not in [x.lower() for x in emails[:i]]] seen: set[str] = set() return [e for e in emails if not (e.lower() in seen or seen.add(e.lower()))]Where it breaks: nothing fails, but it hides a change inside a comprehension, through a trick with or. A reader expects a comprehension only to build a list. And it lowercases each new email twice.
A plain loop with a set
The idea: Loop over the emails, keep a set of lowercased ones seen, and append each new one.
The waste in Better: a trick the reader has to work out, when a short loop says each step plainly.
Here a loop beats a comprehension: each answer depends on the emails before it, and the loop changes seen where everyone can see it.
Highlighted lines are new in this rung.
def first_of_each(emails: list[str]) -> list[str]: """Each email once, ignoring case: the first spelling, in the order first seen.""" seen: set[str] = set() kept: list[str] = [] for email in emails: key = email.lower() if key not in seen: seen.add(key) kept.append(email) return kept- Time
O(n)- Space
O(n)- Why
- One set look-up, and at most one add, for each email.
What changed from the previous rung
def first_of_each(emails: list[str]) -> list[str]: """Each email once, ignoring case: the first spelling, in the order first seen.""" seen: set[str] = set() return [e for e in emails if not (e.lower() in seen or seen.add(e.lower()))] kept: list[str] = [] for email in emails: key = email.lower() if key not in seen: seen.add(key) kept.append(email) return keptWhen Best isn’t best: if case matters and you only drop exact repeats, list(dict.fromkeys(emails)) does it in one line: a dict keeps each key once, in the order it was first added.
Check yourself
Section titled “Check yourself”A few quick questions. Ones you miss are remembered, so you can find your weak spots later.
Flashcards
Section titled “Flashcards”Add these to your review deck. Each card comes back a little later every time you get it right.
Cheat sheet
Section titled “Cheat sheet”- Don’t change a list while you loop over it. Build a new one, or assign the new one to
items[:]. - In
[expr for x in items if test],forruns first, thenif, thenexpr. [ ]makes a list,{ }a set,{k: v}a dict and( )a generator, which gives one item at a time.ifat the end filters.a if test else bat the front chooses.- Two
fors go in the same order as the loops. - Make a grid with
[[0] * n for _ in range(m)], never[[0] * n] * m. - For side effects, several steps or more than two
fors, write a loop.
Go deeper
Section titled “Go deeper”- In the Python tutorial: list comprehensions and nested list comprehensions.
- In the Python reference: comprehensions and generator expressions.
- From the Python FAQ: How do you remove multiple items from a list?, How do you remove duplicates from a list? and How do I create a multidimensional list?
- Draw your own: the comprehension on the Python visualizer.
- Next on the Python roadmap: My server ignored the port I set, on exceptions. Later, Count errors in a 10 GB log builds pipelines out of generator expressions.
Can you do these now?
If you’re not sure of any of these, scroll back to that part before moving on.
Status: Not started