My script can’t find its file when I run it from another folder

See the problem

Your shop project has a script that reads the price list:

shop/
├── data/
│   └── prices.csv
└── tools/
    └── report.py

with open("data/prices.csv") as f:
    ...

From the shop folder, python tools/report.py works. Then you set it to run every night, and the job starts it from your home folder. Now it stops with FileNotFoundError: [Errno 2] No such file or directory: 'data/prices.csv'. The file is still there. Only the folder you ran it from changed.

Then a product called "Pens, blue" shows up, with a comma in its name, and the price list breaks again. How do you find a file from any folder, and read it safely?

Define the task

Write load_prices(script).

• Input: script, the running script’s own path, as __file__ holds it: "/shop/tools/report.py".
• Output: a dict of each product’s name to its price, as a float, from data/prices.csv in the folder above the script’s folder.
• The hard part: it must work whichever folder it’s run from. The file is UTF-8, and Excel may have saved it with a mark at the start (a BOM). A name can hold a comma, inside quotes: "Pens, blue",2.50.

Example: for this prices.csv, load_prices("/shop/tools/report.py") gives {"pen": 1.5, "cup": 4.25}.

name,price
pen,1.50
cup,4.25

Pause and think

⏸ Pause and think

open("data/prices.csv") works when you run it from the shop folder, and fails from your home folder. What decides where Python looks for the file?

Give it two minutes with pen and paper before you open a hint.

Hints open one at a time:

Hint 1 of 3

A path that doesn’t start with / (or a drive like C:\ on Windows) is relative. Python starts it in the current folder: where the program was run from.

Hint 2 of 3

__file__ holds the script’s own path. Path(__file__).resolve().parent is the script’s folder, whichever folder you ran it from.

Hint 3 of 3

A line like "Pens, blue",2.50 has a comma inside quotes. The csv module knows the quoting rules. split(",") doesn’t.

Understand the idea

Every running program has a current folder, also called the working directory. It’s the folder the program was started from, and Path.cwd() gives it. A relative path like data/prices.csv starts there. An absolute path starts at the top: / on Linux and macOS, a drive like C:\ on Windows.

Follow a path one part at a time. Pick another folder to run from, or another path: try running from /shop, or ../data/prices.csv from /shop/tools.

Where a path leads

⤢ Full view

The path, part by part

1. data
2. prices.csv

• /
• home/
• sam/you ran it from here
• shop/
• data/
• prices.csv
• tools/
• report.pythe script

Step 1 of 3. You ran python /shop/tools/report.py from /home/sam. That’s the program’s current folder, the one Path.cwd() gives. Question: open() gets 'data/prices.csv', which doesn’t start with /. Where does it start looking?

What happens next? open() gets 'data/prices.csv', which doesn’t start with /. Where does it start looking?

/shop/tools, the folder report.py is in/home/sam, the folder you ran it from/shop, the project’s top folderJust show me

◀ BackPlayNext ▶Reset Ask me first Show as text

1# In a terminal, in /home/sam:
2#   python /shop/tools/report.py
3# In report.py:
4with open("data/prices.csv", encoding="utf-8") as f:
5    first = f.readline()

Try your own input

Run fromPathShow it

★Key insight

The current folder belongs to whoever runs the program: you, a teammate, or a scheduled job. Your script’s own folder belongs to your code. Build paths from __file__, not from the current folder.

pathlib makes paths into objects. Path joins parts with /, and writes the right separator for the system: \ on Windows, / elsewhere.

from pathlib import Path

p = Path("/shop/tools/report.py")
p.parent                  # Path('/shop/tools'): drops the last part
p.parent.parent / "data"  # Path('/shop/data'): / adds a part
p.name, p.stem, p.suffix  # ('report.py', 'report', '.py')
p.with_suffix(".txt")     # Path('/shop/tools/report.txt')
p.exists()                # True if something is there

.parent and / only change the text of the path. Nothing is looked up until you open it. Here’s the path built from __file__. Run it from any folder: the steps are the same.

A path from the script

⤢ Full view

The path, part by part

1. .parent
2. .parent
3. / "data"
4. / "prices.csv"

• /
• home/
• sam/you ran it from here
• shop/
• data/
• prices.csv
• tools/
• report.pythe script

Step 1 of 8. You ran python /shop/tools/report.py from /home/sam. That’s the current folder, but this path never uses it.

◀ BackPlayNext ▶Reset Ask me first Show as text

1# In a terminal, in /home/sam:
2#   python /shop/tools/report.py
3# In report.py:
4from pathlib import Path
5 
6here = Path(__file__).resolve().parent
7path = here.parent / "data" / "prices.csv"
8with path.open(encoding="utf-8") as f:
9    first = f.readline()

Try your own input

Run fromShow it

Text is bytes plus an encoding. A file holds bytes. To read it as text, Python needs the encoding that turns bytes into letters. Say encoding="utf-8" every time. Before Python 3.15, open() without it used the system’s own encoding, which on Windows is often not UTF-8: there, Café can come back as Café.

with closes the file for you, even when an error goes through, like the finally from the last lesson. The mode says what you’ll do:

Mode	Does	If the file is there	If it isn’t
"r" (the default)	read	reads it	FileNotFoundError
"w"	write	empties it first	makes it
"a"	add to the end	keeps what’s there	makes it
"x"	write a new file	FileExistsError	makes it

Add "b" ("rb", "wb") for bytes instead of text, as for images. path.read_text() and path.write_text() open, read or write, and close, in one call.

Climb the ladder

All three give {"pen": 1.5, "cup": 4.25} for the example, when run from shop. What changes is where else they work.

Okay

Open it from the current folder

The idea: open("data/prices.csv"), skip the header, and split each line on commas.

It works when you run it from the shop folder.

def load_prices(script: str) -> dict[str, float]:
    """Prices from data/prices.csv, in the folder above the script's folder."""
    prices = {}
    with open("data/prices.csv") as f:  # from the current folder
        next(f)  # the header
        for line in f:
            name, price = line.strip().split(",")
            prices[name] = float(price)
    return prices

Time

O(n)

Space

O(r)

Why

One pass over the file's n characters, and a dict of its r rows.

Where it breaks: from any other folder, FileNotFoundError. A name with a comma in quotes splits into three parts, so name, price = ... raises ValueError. And without an encoding, Windows may read Café wrong on Python before 3.15.

Better

Start from the script’s folder

The idea: Path(script).resolve().parent.parent / "data" / "prices.csv", opened with encoding="utf-8".

The waste in Okay: it depends on a folder it doesn’t control, the one someone ran it from.

Now the path starts at the script, so it’s the same from any folder. resolve() makes it absolute, and follows a link to the script to the real file, so .parent is the folder the script really lives in.

Highlighted lines are new in this rung.

from pathlib import Path


def load_prices(script: str) -> dict[str, float]:
    """Prices from data/prices.csv, in the folder above the script's folder."""
    path = Path(script).resolve().parent.parent / "data" / "prices.csv"
    prices = {}
    with path.open(encoding="utf-8") as f:
        next(f)  # the header
        for line in f:
            name, price = line.strip().split(",")
            prices[name] = float(price)
    return prices

Time

O(n)

Space

O(r)

Why

One pass over the file's n characters, and a dict of its r rows.

What changed from the previous rung

from pathlib import Path


def load_prices(script: str) -> dict[str, float]:
    """Prices from data/prices.csv, in the folder above the script's folder."""
    path = Path(script).resolve().parent.parent / "data" / "prices.csv"
    prices = {}
    with open("data/prices.csv") as f:  # from the current folder
    with path.open(encoding="utf-8") as f:
        next(f)  # the header
        for line in f:
            name, price = line.strip().split(",")
            prices[name] = float(price)
    return prices

Where it breaks: "Pens, blue",2.50 still splits into '"Pens', ' blue"' and '2.50', and the line raises ValueError.

Best

Let csv read it

The idea: The same path, opened with encoding="utf-8-sig" and newline="", read by csv.DictReader.

The waste in Better: it writes its own CSV rules, and they’re wrong.

csv.DictReader knows the quoting rules, and names each field by the header: row["name"]. utf-8-sig reads UTF-8, and drops the BOM Excel puts at the start, which would otherwise stick to the first header as "\ufeffname". newline="" leaves line endings to csv, as its docs ask, so a line break inside quotes comes back exactly as written.

Highlighted lines are new in this rung.

import csv
from pathlib import Path


def load_prices(script: str) -> dict[str, float]:
    """Prices from data/prices.csv, in the folder above the script's folder."""
    path = Path(script).resolve().parent.parent / "data" / "prices.csv"
    # utf-8-sig also reads the mark Excel can put at the start; newline="" leaves line ends to csv.
    with path.open(encoding="utf-8-sig", newline="") as f:
        return {row["name"]: float(row["price"]) for row in csv.DictReader(f)}

Time

O(n)

Space

O(r)

Why

One pass over the file's n characters, and a dict of its r rows.

What changed from the previous rung

import csv
from pathlib import Path


def load_prices(script: str) -> dict[str, float]:
    """Prices from data/prices.csv, in the folder above the script's folder."""
    path = Path(script).resolve().parent.parent / "data" / "prices.csv"
    prices = {}
    with path.open(encoding="utf-8") as f:
        next(f)  # the header
        for line in f:
            name, price = line.strip().split(",")
            prices[name] = float(price)
    return prices
    # utf-8-sig also reads the mark Excel can put at the start; newline="" leaves line ends to csv.
    with path.open(encoding="utf-8-sig", newline="") as f:
        return {row["name"]: float(row["price"]) for row in csv.DictReader(f)}

When Best isn’t best: when the data file ships inside an installed package, maybe as a zip, importlib.resources.files() finds it where __file__ can’t. And for big tables you’ll filter and sum, pandas’ read_csv() does more in one line.

Write it yourself

Now write it in the page. Press Run tests to check your code. Python runs in your browser, with its own small set of folders, so /shop here is a folder in your browser, not on your computer. Your code is saved in this browser for next time.

Each test says which folder it runs from. Try Okay from the ladder: it passes only from /shop. Python in your browser numbers system errors its own way, so you’ll see [Errno 44] where Linux, macOS and Windows say [Errno 2]. The error’s name, FileNotFoundError, is the same.

Load the prices

Ctrl + Enter runs

def load_prices(script: str) -> dict[str, float]: # script is the running script's own path, like __file__: "/shop/tools/report.py". # Give back {name: price} from data/prices.csv, in the folder above the script's folder. return {}

Run testsVisualizeReset codeLoads about 6 MB the first time. Best on Wi-Fi.

1. ○Not run yet: Run from /shop: load_prices("/shop/tools/report.py") → {'pen': 1.5, 'cup': 4.25}
   What’s in /shop/data/prices.csv

   name,price
   pen,1.50
   cup,4.25
   

2. ○Not run yet: Run from /shop/tools: load_prices("/shop/tools/report.py") → {'pen': 1.5, 'cup': 4.25}
   What’s in /shop/data/prices.csv

   name,price
   pen,1.50
   cup,4.25
   

3. ○Not run yet: Run from /home/sam: load_prices("/shop/tools/report.py") → {'pen': 1.5, 'cup': 4.25}
   What’s in /shop/data/prices.csv

   name,price
   pen,1.50
   cup,4.25
   

4. ○Not run yet: Run from /shop, with an accent in a name: load_prices("/shop/tools/report.py") → {'Café au lait': 3.2, 'pen': 1.5}
   What’s in /shop/data/prices.csv

   name,price
   Café au lait,3.20
   pen,1.50
   

5. ○Not run yet: Extra test
6. ○Not run yet: Extra test

Extra tests check tricky cases. You see their input only if they fail.

Try your own input

Arguments, separated by commas as in Python

load_prices()Run onceVisualize

Compare

n is the size of the file, and r its number of rows.

Way	Time	Extra memory	Good	Not so good	Use it when
Okay: from the current folder, split on commas	O(n)	O(r)	Short	Works from one folder only; breaks on a comma in quotes	A one-off you run yourself
Better: from the script, split on commas	O(n)	O(r)	Works from any folder	Breaks on a comma in quotes	Your own simple files, with no quotes
Best: from the script, with csv	O(n)	O(r)	Any folder, quotes and a BOM	A few more names to learn	Reading CSV

Common mistakes and edge cases

!"w" empties the file before you write

open(path, "w") empties the file the moment it opens, even if you never write. To add to the end, use "a". To make sure you don’t overwrite anything, use "x", which raises FileExistsError if the file is there.

• A backslash in a plain string can be an escape: "C:\new\table.csv" holds a newline and a tab, not two folders. Use /, which Windows takes too, a raw string r"C:\new\table.csv", or Path.
• An absolute part wins: Path("data") / "/prices.csv" is /prices.csv. Joining a part that starts with / throws away everything before it.
• .parent only works on the text: Path("report.py").parent is ., and so is its parent. Call .resolve() first, as the ladder does.
• Each line keeps its line end: for line in f gives "pen,1.50\n". Strip it before you split.
• A blank row after every row on Windows: csv ends rows with \r\n, and on Windows text mode turns each \n into \r\n too. Open CSV files with newline="", for reading and writing.
• Checking first can still fail: a file can disappear between exists() and open(). Open it, and catch FileNotFoundError if you can do something about it, as in the last lesson.
• os.chdir() changes the current folder for the whole program, every module included. Build paths instead of moving around.

Say it in the interview

Questions to ask first

• Where does the file live: next to the code, or wherever it’s run?
• How big can it get? Does it fit in memory?
• What encoding is it, and who writes it: a person in Excel, or another program?
• What should a missing file do: stop with an error, or give an empty result?

A 60-second walkthrough

“A relative path like data/prices.csv starts in the current folder, which is wherever the program was started. So it breaks when a scheduled job or a teammate runs it from somewhere else. I build the path from the script’s own location instead: Path(file).resolve().parent, then .parent and / to reach data/prices.csv, and pathlib writes the right separator on each system. I open it in a with block, so it’s always closed, with encoding=‘utf-8-sig’, which reads UTF-8 and drops the mark Excel adds. For CSV I use the csv module with newline=”, because a field can hold a comma inside quotes, which split(’,’) gets wrong. It reads row by row, so memory stays small however long the file.”

Likely follow-ups

• Why pathlib over os.path? A path is an object: / joins, and .parent, .suffix, .exists() and .read_text() read clearly. The same code writes the right separators on Windows.
• How would you read a 10 GB file? A line at a time with for line in f, or in pieces with f.read(65536). Never f.read() with no size, which loads it all.
• How do you save a file so a crash can’t leave half of it? Write a temporary file in the same folder, then os.replace() it over the old one. The swap happens all at once.
• Why with instead of f.close()? If an error goes through before close(), the file stays open. with closes it either way.

Practice

Four problems, from easy to hard. Each has hints, tests you can run here, and its own Okay, Better and Best answers. Try before you open them.

1. Save a report (easy)

Write save_report(folder, name, lines). It writes lines, one per line, to <folder>/reports/<name>.txt, as UTF-8, and makes any folder that’s missing. Saving the same name again replaces the old report.

Example: save_report("/r1", "day1", ["a", "b"]) leaves /r1/reports/day1.txt holding "a\nb\n".

Hint 1 of 3

open() can make a file, but not the folder it goes in. Make the folder first.

Hint 2 of 3

mkdir() raises FileExistsError if the folder is there. exist_ok=True makes that fine.

Hint 3 of 3

If folder itself is missing too, mkdir() needs parents=True to make every folder on the way.

Save a report

Ctrl + Enter runs

def save_report(folder: str, name: str, lines: list[str]) -> None: # Write lines, one per line, to <folder>/reports/<name>.txt, making missing folders. pass

Run testsReset codeLoads about 6 MB the first time. Best on Wi-Fi.

1. ○Not run yet: In /r1, with no reports folder: save_report("/r1", "day1", ["a", "b"]), then what day1.txt holds → 'a\nb\n'
2. ○Not run yet: In /r2, reports is there already: save_report("/r2", "day1", ["c"]), then what day1.txt holds → 'c\n'
3. ○Not run yet: Saved twice: save_report("/r3", "day1", ["old", "x"]), then ["new"], then what day1.txt holds → 'new\n'
4. ○Not run yet: No /r4 at all: save_report("/r4/shop", "day1", ["a"]), then what day1.txt holds → 'a\n'
5. ○Not run yet: Extra test
6. ○Not run yet: Extra test

Extra tests check tricky cases. You see their input only if they fail.

Try your own input

Arguments, separated by commas as in Python

save_report()Run onceVisualize

Show the Okay, Better and Best answers

Okay

Open the file and write

The idea: open(folder + "/reports/" + name + ".txt", "w"), and write each line with "\n".

It writes the report when the reports folder is already there.

def save_report(folder: str, name: str, lines: list[str]) -> None:
    """Write lines, one per line, to <folder>/reports/<name>.txt, replacing an older one."""
    with open(folder + "/reports/" + name + ".txt", "w") as f:
        for line in lines:
            f.write(line + "\n")

Time

O(n)

Space

O(1)

Why

Writes the n characters of the lines once.

Where it breaks: open() makes the file, not the folder, so with no reports folder it raises FileNotFoundError.

Better

Make the reports folder first

The idea: (Path(folder) / "reports").mkdir(exist_ok=True), then open the file in it.

The waste in Okay: it counts on a folder someone else had to make.

mkdir(exist_ok=True) makes reports when it’s missing, and does nothing when it’s there.

Highlighted lines are new in this rung.

from pathlib import Path


def save_report(folder: str, name: str, lines: list[str]) -> None:
    """Write lines, one per line, to <folder>/reports/<name>.txt, replacing an older one."""
    reports = Path(folder) / "reports"
    reports.mkdir(exist_ok=True)
    with (reports / f"{name}.txt").open("w", encoding="utf-8") as f:
        for line in lines:
            f.write(line + "\n")

Time

O(n)

Space

O(1)

Why

Writes the n characters of the lines once.

What changed from the previous rung

from pathlib import Path


def save_report(folder: str, name: str, lines: list[str]) -> None:
    """Write lines, one per line, to <folder>/reports/<name>.txt, replacing an older one."""
    with open(folder + "/reports/" + name + ".txt", "w") as f:
    reports = Path(folder) / "reports"
    reports.mkdir(exist_ok=True)
    with (reports / f"{name}.txt").open("w", encoding="utf-8") as f:
        for line in lines:
            f.write(line + "\n")

Where it breaks: mkdir() makes only the last folder. With no /r4 either, it raises FileNotFoundError.

Best

Make every folder, then write_text()

The idea: path.parent.mkdir(parents=True, exist_ok=True), then path.write_text(..., encoding="utf-8").

The waste in Better: it stops one folder short.

parents=True makes every missing folder on the way, like mkdir -p. write_text() opens, writes and closes in one call, and replaces an older report.

Highlighted lines are new in this rung.

from pathlib import Path


def save_report(folder: str, name: str, lines: list[str]) -> None:
    """Write lines, one per line, to <folder>/reports/<name>.txt, replacing an older one."""
    path = Path(folder) / "reports" / f"{name}.txt"
    path.parent.mkdir(parents=True, exist_ok=True)
    path.write_text("".join(line + "\n" for line in lines), encoding="utf-8")

Time

O(n)

Space

O(n)

Why

Builds the n characters of the report, then writes them once.

What changed from the previous rung

from pathlib import Path


def save_report(folder: str, name: str, lines: list[str]) -> None:
    """Write lines, one per line, to <folder>/reports/<name>.txt, replacing an older one."""
    reports = Path(folder) / "reports"
    reports.mkdir(exist_ok=True)
    with (reports / f"{name}.txt").open("w", encoding="utf-8") as f:
        for line in lines:
            f.write(line + "\n")
    path = Path(folder) / "reports" / f"{name}.txt"
    path.parent.mkdir(parents=True, exist_ok=True)
    path.write_text("".join(line + "\n" for line in lines), encoding="utf-8")

2. Read the orders, skip the bad rows (easy)

Write read_orders(path). The file at path is a CSV export of orders: a header row, then one order per row. Give back a list with a dict for each order, {"customer": ..., "item": ..., "qty": ...}, with qty as an int. The header names the columns, and the file may have other columns too, in any order. Skip blank lines, and skip a row whose qty is missing or isn’t a whole number. A field may be in double quotes, with a comma or a line break inside. The file may start with the BOM that Excel adds.

Example: for a file holding customer,item,qty, then Ana,tea,2, then Bo,cake,two, read_orders(path) gives [{"customer": "Ana", "item": "tea", "qty": 2}].

Hint 1 of 3

split(",") cuts "Smith, Jane" in two. The csv module knows the quoting rules, line breaks inside quotes too, when the file is opened with newline="".

Hint 2 of 3

csv.DictReader names each field by the header, so row["qty"] works wherever qty is. It skips blank lines, and encoding="utf-8-sig" drops Excel’s BOM.

Hint 3 of 3

int("two") raises ValueError: catch it, and go on to the next row. A row with no qty gives None, and int(None) raises TypeError, so pass DictReader restval="".

Read the orders, skip the bad rows

Ctrl + Enter runs

def read_orders(path: str) -> list[dict]: # Each order in the CSV file at path: {"customer": ..., "item": ..., "qty": int}. # Skip blank lines, and rows whose qty isn't a whole number. return []

Run testsVisualizeReset codeLoads about 6 MB the first time. Best on Wi-Fi.

1. ○Not run yet: Two orders: read_orders("/orders/o1.csv") → [{'customer': 'Ana', 'item': 'tea', 'qty': 2}, {'customer': 'Bo', 'item': 'cake', 'qty': 1}]
   What’s in /orders/o1.csv

   customer,item,qty
   Ana,tea,2
   Bo,cake,1
   

2. ○Not run yet: A comma inside quotes: read_orders("/orders/o2.csv") → [{'customer': 'Smith, Jane', 'item': 'tea', 'qty': 3}]
   What’s in /orders/o2.csv

   customer,item,qty
   "Smith, Jane",tea,3
   

3. ○Not run yet: Bad numbers and a blank line: read_orders("/orders/o3.csv") → [{'customer': 'Bo', 'item': 'cake', 'qty': 1}]
   What’s in /orders/o3.csv

   customer,item,qty
   Ana,tea,two
   
   Bo,cake,1
   Cy,pie,
   

4. ○Not run yet: More columns, in another order: read_orders("/orders/o4.csv") → [{'customer': 'Ana', 'item': 'tea', 'qty': 2}]
   What’s in /orders/o4.csv

   id,qty,customer,note,item
   7,2,Ana,ring twice,tea
   

5. ○Not run yet: Extra test
6. ○Not run yet: Extra test
7. ○Not run yet: Extra test

Extra tests check tricky cases. You see their input only if they fail.

Try your own input

Arguments, separated by commas as in Python

read_orders()Run onceVisualize

Show the Okay, Better and Best answers

Okay

Split each line on commas

The idea: Skip the header line, then split(",") each line into customer, item and qty.

It skips blank lines and bad numbers, and reads a simple file right.

def read_orders(path: str) -> list[dict]:
    """Each order in the CSV file at path, as {"customer", "item", "qty"}, skipping bad rows."""
    orders = []
    with open(path, encoding="utf-8") as f:
        next(f)  # skip the header
        for line in f:
            if not line.strip():
                continue  # a blank line
            customer, item, qty = line.strip().split(",")
            try:
                count = int(qty)
            except ValueError:
                continue  # qty isn't a whole number
            orders.append({"customer": customer, "item": item, "qty": count})
    return orders

Time

O(n)

Space

O(n)

Why

One pass over the file's n characters; the list holds every order.

Where it breaks: "Smith, Jane",tea,3 splits into four parts, so customer, item, qty = ... raises ValueError, and the whole read stops. A line break inside quotes, or one more column, does the same.

Better

csv.reader, fields by place

The idea: csv.reader reads each row as a list; customer, item, qty = row.

The waste in Okay: it splits at every comma, even inside quotes.

csv.reader knows the quoting rules: a comma or a line break inside quotes stays in its field.

Highlighted lines are new in this rung.

import csv


def read_orders(path: str) -> list[dict]:
    """Each order in the CSV file at path, as {"customer", "item", "qty"}, skipping bad rows."""
    orders = []
    with open(path, encoding="utf-8", newline="") as f:
        rows = csv.reader(f)
        next(rows)  # skip the header
        for row in rows:
            if not row:
                continue  # a blank line
            customer, item, qty = row  # fields by place: customer, item, qty
            try:
                count = int(qty)
            except ValueError:
                continue  # qty isn't a whole number
            orders.append({"customer": customer, "item": item, "qty": count})
    return orders

Time

O(n)

Space

O(n)

Why

One pass over the file; the list holds every order.

What changed from the previous rung

import csv


def read_orders(path: str) -> list[dict]:
    """Each order in the CSV file at path, as {"customer", "item", "qty"}, skipping bad rows."""
    orders = []
    with open(path, encoding="utf-8") as f:
        next(f)  # skip the header
        for line in f:
            if not line.strip():
    with open(path, encoding="utf-8", newline="") as f:
        rows = csv.reader(f)
        next(rows)  # skip the header
        for row in rows:
            if not row:
                continue  # a blank line
            customer, item, qty = line.strip().split(",")
            customer, item, qty = row  # fields by place: customer, item, qty
            try:
                count = int(qty)
            except ValueError:
                continue  # qty isn't a whole number
            orders.append({"customer": customer, "item": item, "qty": count})
    return orders

Where it breaks: it counts on the order and number of the columns. When the export adds an id column, or a row is missing its qty, customer, item, qty = row raises ValueError. Worse, if a column only moves, every field quietly gets the wrong value.

Best

csv.DictReader, fields by name

The idea: csv.DictReader with encoding="utf-8-sig" and restval="", then row["qty"] wherever qty is.

The waste in Better: it guesses where each column is, when the header already says.

DictReader names each field by the header, and skips blank lines. utf-8-sig drops the BOM Excel puts first. Without it, the BOM sticks to the first name, and row["customer"] raises KeyError. A short row’s missing fields are None, and int(None) raises TypeError, not ValueError. restval="" makes them "", so the except ValueError skips that row too.

Highlighted lines are new in this rung.

import csv


def read_orders(path: str) -> list[dict]:
    """Each order in the CSV file at path, as {"customer", "item", "qty"}, skipping bad rows."""
    orders = []
    # utf-8-sig drops the BOM Excel may put first; newline="" lets csv see line breaks in quotes.
    with open(path, encoding="utf-8-sig", newline="") as f:
        # Fields by the header's names; blank lines skipped; a missing field reads as "".
        for row in csv.DictReader(f, restval=""):
            try:
                count = int(row["qty"])
            except ValueError:
                continue  # qty isn't a whole number
            orders.append({"customer": row["customer"], "item": row["item"], "qty": count})
    return orders

Time

O(n)

Space

O(n)

Why

One pass over the file; the list holds every order.

What changed from the previous rung

import csv


def read_orders(path: str) -> list[dict]:
    """Each order in the CSV file at path, as {"customer", "item", "qty"}, skipping bad rows."""
    orders = []
    with open(path, encoding="utf-8", newline="") as f:
        rows = csv.reader(f)
        next(rows)  # skip the header
        for row in rows:
            if not row:
                continue  # a blank line
            customer, item, qty = row  # fields by place: customer, item, qty
    # utf-8-sig drops the BOM Excel may put first; newline="" lets csv see line breaks in quotes.
    with open(path, encoding="utf-8-sig", newline="") as f:
        # Fields by the header's names; blank lines skipped; a missing field reads as "".
        for row in csv.DictReader(f, restval=""):
            try:
                count = int(qty)
                count = int(row["qty"])
            except ValueError:
                continue  # qty isn't a whole number
            orders.append({"customer": customer, "item": item, "qty": count})
            orders.append({"customer": row["customer"], "item": row["item"], "qty": count})
    return orders

When Best isn’t best: skipping bad rows quietly hides them. Real code logs each one with reader.line_num, the line its row ends on, which stays right when a field has a line break. For a file too big for memory, yield each order instead of building a list. And for tables you’ll filter and sum, pandas’ read_csv() does it in one call.

3. Rotate the logs (medium)

A server writes to app.log. Each night, rotate(log, keep) moves app.log to app.log.1, app.log.1 to app.log.2, and so on, so the newest old log is always .1. At most keep old logs stay (keep is 1 or more): the oldest goes. If there’s no app.log, nothing changes.

Example: with app.log and app.log.1, rotate("/logs/app.log", 3) leaves app.log.1 (yesterday’s app.log) and app.log.2.

Hint 1 of 3

Moving app.log.1 to app.log.2 first, then app.log.2 to app.log.3, moves the same log twice.

Hint 2 of 3

Start from the oldest, so each move goes to a name that’s free.

Hint 3 of 3

Delete the oldest first, with unlink(missing_ok=True). replace() moves a file and overwrites, on Windows too.

Rotate the logs

Ctrl + Enter runs

def rotate(log: str, keep: int) -> None: # Move log to log.1, log.1 to log.2, ..., keeping at most keep old logs. pass

Run testsReset codeLoads about 6 MB the first time. Best on Wi-Fi.

1. ○Not run yet: app.log = "mon", nothing older: rotate("/l1/app.log", 3), then the files → [['app.log.1', 'mon']]
2. ○Not run yet: app.log = "tue", app.log.1 = "mon": rotate("/l2/app.log", 3), then the files → [['app.log.1', 'tue'], ['app.log.2', 'mon']]
3. ○Not run yet: app.log = "thu", .1 = "wed", .2 = "tue", .3 = "mon": rotate("/l3/app.log", 3), then the files → [['app.log.1', 'thu'], ['app.log.2', 'wed'], ['app.log.3', 'tue']]
4. ○Not run yet: No app.log, app.log.1 = "mon": rotate("/l4/app.log", 3), then the files → [['app.log.1', 'mon']]
5. ○Not run yet: Extra test
6. ○Not run yet: Extra test

Extra tests check tricky cases. You see their input only if they fail.

Try your own input

Arguments, separated by commas as in Python

rotate()Run onceVisualize

Show the Okay, Better and Best answers

Okay

Move each log up, newest first

The idea: For i from 1 up, move app.log.i to app.log.i+1. Then move app.log to app.log.1.

The first night, with nothing older, it works.

from pathlib import Path


def rotate(log: str, keep: int) -> None:
    """Move app.log to app.log.1, app.log.1 to app.log.2, and so on, keeping keep copies."""
    if not Path(log).exists():
        return
    for i in range(1, keep):  # newest first
        old = Path(f"{log}.{i}")
        if old.exists():
            old.rename(f"{log}.{i + 1}")
    Path(log).rename(f"{log}.1")

Time

O(k)

Space

O(1)

Why

At most k moves, for k old logs kept.

Where it breaks: app.log.1 moves to .2, then the loop moves .2 to .3: the same log, again. With tue and mon, it leaves .1 = tue and .3 = mon. On Linux, rename() also overwrites silently, so an old log can disappear.

Better

Move each log up, oldest first

The idea: For i from keep down to 1, move app.log.i to app.log.i+1. Then app.log to app.log.1.

The waste in Okay: it moves a log into a name that’s still taken, then moves it again.

Oldest first, each move goes to a name that was just freed, so no log is moved twice or lost.

Highlighted lines are new in this rung.

from pathlib import Path


def rotate(log: str, keep: int) -> None:
    """Move app.log to app.log.1, app.log.1 to app.log.2, and so on, keeping keep copies."""
    path = Path(log)
    if not path.exists():
        return
    for i in range(keep, 0, -1):  # oldest first, so nothing is overwritten
        old = Path(f"{log}.{i}")
        if old.exists():
            old.rename(f"{log}.{i + 1}")
    path.rename(f"{log}.1")

Time

O(k)

Space

O(1)

Why

At most k + 1 moves, for k old logs kept.

What changed from the previous rung

from pathlib import Path


def rotate(log: str, keep: int) -> None:
    """Move app.log to app.log.1, app.log.1 to app.log.2, and so on, keeping keep copies."""
    if not Path(log).exists():
    path = Path(log)
    if not path.exists():
        return
    for i in range(1, keep):  # newest first
    for i in range(keep, 0, -1):  # oldest first, so nothing is overwritten
        old = Path(f"{log}.{i}")
        if old.exists():
            old.rename(f"{log}.{i + 1}")
    Path(log).rename(f"{log}.1")
    path.rename(f"{log}.1")

Where it breaks: the oldest log moves up too, to app.log.4, so it keeps one log too many. With keep = 1, two old logs stay.

Best

Delete the oldest, then move up with replace()

The idea: unlink(missing_ok=True) app.log.keep, move the rest up oldest first with replace(), then app.log to .1.

The waste in Better: it keeps a log nobody asked for.

The oldest goes first, so exactly keep stay. missing_ok=True makes deleting a log that isn’t there fine. replace() overwrites on every system, where rename() raises FileExistsError on Windows if the name is taken.

Highlighted lines are new in this rung.

from pathlib import Path


def rotate(log: str, keep: int) -> None:
    """Move app.log to app.log.1, app.log.1 to app.log.2, and so on, keeping keep copies."""
    path = Path(log)
    if not path.exists():
        return
    Path(f"{log}.{keep}").unlink(missing_ok=True)  # the oldest copy goes
    for i in range(keep - 1, 0, -1):  # oldest first, so nothing is overwritten
        old = Path(f"{log}.{i}")
        if old.exists():
            old.replace(f"{log}.{i + 1}")  # replace() overwrites on Windows too; rename() doesn't
    path.replace(f"{log}.1")

Time

O(k)

Space

O(1)

Why

One delete and at most k moves, for k old logs kept.

What changed from the previous rung

from pathlib import Path


def rotate(log: str, keep: int) -> None:
    """Move app.log to app.log.1, app.log.1 to app.log.2, and so on, keeping keep copies."""
    path = Path(log)
    if not path.exists():
        return
    for i in range(keep, 0, -1):  # oldest first, so nothing is overwritten
    Path(f"{log}.{keep}").unlink(missing_ok=True)  # the oldest copy goes
    for i in range(keep - 1, 0, -1):  # oldest first, so nothing is overwritten
        old = Path(f"{log}.{i}")
        if old.exists():
            old.rename(f"{log}.{i + 1}")
    path.rename(f"{log}.1")
            old.replace(f"{log}.{i + 1}")  # replace() overwrites on Windows too; rename() doesn't
    path.replace(f"{log}.1")

When Best isn’t best: in a real server, logging.handlers.RotatingFileHandler rotates by size for you, and TimedRotatingFileHandler by time.

4. Find duplicate files (hard)

Your photos folder has copies everywhere. Write find_duplicates(folder): every group of two or more files, in folder or any folder inside it, with exactly the same bytes. Give each file as its path from folder, like "b/c.txt". Sort each group, then sort the groups. Some files are big, so never hold a whole file in memory: the big tests allow 1 MB.

Example: with a.txt and b/c.txt both holding hi, and d.txt holding yo!, the answer is [["a.txt", "b/c.txt"]].

Hint 1 of 3

Two files with different sizes can’t be the same. Group by size first: it costs no reading at all.

Hint 2 of 3

Inside a group of the same size, a hash like sha256 of each file finds the twins.

Hint 3 of 3

read_bytes() loads the whole file. Hash it a piece at a time instead: hashlib.file_digest(f, "sha256") does that.

Find duplicate files

Ctrl + Enter runs

def find_duplicates(folder: str) -> list[list[str]]: # Groups of 2+ files under folder with the same bytes, as sorted paths from folder. return []

Run testsVisualizeReset codeLoads about 6 MB the first time. Best on Wi-Fi.

1. ○Not run yet: a.txt = "hi", b/c.txt = "hi", d.txt = "yo!": find_duplicates("/d1") → [['a.txt', 'b/c.txt']]
2. ○Not run yet: x.txt = "1", y.txt = "22": find_duplicates("/d2") → []
3. ○Not run yet: Same size, different bytes: p.txt = "ab", q.txt = "cd", r.txt = "zz", s/t.txt = "zz": find_duplicates("/d3") → [['r.txt', 's/t.txt']]
4. ○Not run yet: One 4 MB file, and s1.txt = s2.txt = "same": find_duplicates("/d4"), in 1 MB → [['s1.txt', 's2.txt']]
5. ○Not run yet: Extra test
6. ○Not run yet: Extra test

Extra tests check tricky cases. You see their input only if they fail.

Try your own input

Arguments, separated by commas as in Python

find_duplicates()Run onceVisualize

Show the Okay, Better and Best answers

Okay

Hash every file

The idea: For every file under folder, sha256 of read_bytes(). Files with the same hash are a group.

It finds every group, at any depth.

import hashlib
from pathlib import Path


def find_duplicates(folder: str) -> list[list[str]]:
    """Groups of files under folder with the same bytes, as sorted paths from folder."""
    root = Path(folder)
    groups: dict[str, list[str]] = {}
    for path in root.rglob("*"):
        if path.is_file():
            digest = hashlib.sha256(path.read_bytes()).hexdigest()
            groups.setdefault(digest, []).append(path.relative_to(root).as_posix())
    return sorted(sorted(group) for group in groups.values() if len(group) > 1)

Time

O(b)

Space

O(f + m)

Why

Reads all b bytes, keeps f paths, and holds the biggest file, m bytes, at once.

Where it breaks: it reads every file, even one with a size no other file has, and read_bytes() holds the whole file at once. A 4 MB file needs 4 MB, over the 1 MB limit.

Better

Group by size first

The idea: Group files by size. Only files that share a size are hashed, still with read_bytes().

The waste in Okay: it reads files that can’t have a twin.

stat().st_size costs no reading at all, so a file with a size of its own is never opened.

Highlighted lines are new in this rung.

import hashlib
from pathlib import Path


def find_duplicates(folder: str) -> list[list[str]]:
    """Groups of files under folder with the same bytes, as sorted paths from folder."""
    root = Path(folder)
    by_size: dict[int, list[Path]] = {}
    for path in root.rglob("*"):
        if path.is_file():
            by_size.setdefault(path.stat().st_size, []).append(path)
    groups: dict[str, list[str]] = {}
    for same_size in by_size.values():
        if len(same_size) < 2:
            continue  # a file with a size of its own has no twin: never read it
        for path in same_size:
            digest = hashlib.sha256(path.read_bytes()).hexdigest()
            groups.setdefault(digest, []).append(path.relative_to(root).as_posix())
    return sorted(sorted(group) for group in groups.values() if len(group) > 1)

Time

O(b′)

Space

O(f + m′)

Why

Reads only the b′ bytes of files that share a size, and holds the biggest of them, m′ bytes.

What changed from the previous rung

import hashlib
from pathlib import Path


def find_duplicates(folder: str) -> list[list[str]]:
    """Groups of files under folder with the same bytes, as sorted paths from folder."""
    root = Path(folder)
    groups: dict[str, list[str]] = {}
    by_size: dict[int, list[Path]] = {}
    for path in root.rglob("*"):
        if path.is_file():
            by_size.setdefault(path.stat().st_size, []).append(path)
    groups: dict[str, list[str]] = {}
    for same_size in by_size.values():
        if len(same_size) < 2:
            continue  # a file with a size of its own has no twin: never read it
        for path in same_size:
            digest = hashlib.sha256(path.read_bytes()).hexdigest()
            groups.setdefault(digest, []).append(path.relative_to(root).as_posix())
    return sorted(sorted(group) for group in groups.values() if len(group) > 1)

Where it breaks: two big files of the same size are still read whole, so two 4 MB copies need 4 MB at once.

Best

Group by size, then hash in pieces

The idea: Group by size, then hashlib.file_digest(f, "sha256") on each file that shares a size.

The waste in Better: it holds a whole file to work out a 32-byte hash.

file_digest() reads the file a piece at a time and feeds each piece to the hash. So memory stays the same for a 4 MB or a 4 GB file. Before Python 3.11, write the loop yourself: while chunk := f.read(65536): h.update(chunk).

Highlighted lines are new in this rung.

import hashlib
from pathlib import Path


def find_duplicates(folder: str) -> list[list[str]]:
    """Groups of files under folder with the same bytes, as sorted paths from folder."""
    root = Path(folder)
    by_size: dict[int, list[Path]] = {}
    for path in root.rglob("*"):
        if path.is_file():
            by_size.setdefault(path.stat().st_size, []).append(path)
    groups: dict[str, list[str]] = {}
    for same_size in by_size.values():
        if len(same_size) < 2:
            continue  # a file with a size of its own has no twin: never read it
        for path in same_size:
            with path.open("rb") as f:
                digest = hashlib.file_digest(f, "sha256").hexdigest()  # reads a piece at a time
            groups.setdefault(digest, []).append(path.relative_to(root).as_posix())
    return sorted(sorted(group) for group in groups.values() if len(group) > 1)

Time

O(b′)

Space

O(f)

Why

Reads only files that share a size, a fixed-size piece at a time.

What changed from the previous rung

import hashlib
from pathlib import Path


def find_duplicates(folder: str) -> list[list[str]]:
    """Groups of files under folder with the same bytes, as sorted paths from folder."""
    root = Path(folder)
    by_size: dict[int, list[Path]] = {}
    for path in root.rglob("*"):
        if path.is_file():
            by_size.setdefault(path.stat().st_size, []).append(path)
    groups: dict[str, list[str]] = {}
    for same_size in by_size.values():
        if len(same_size) < 2:
            continue  # a file with a size of its own has no twin: never read it
        for path in same_size:
            digest = hashlib.sha256(path.read_bytes()).hexdigest()
            with path.open("rb") as f:
                digest = hashlib.file_digest(f, "sha256").hexdigest()  # reads a piece at a time
            groups.setdefault(digest, []).append(path.relative_to(root).as_posix())
    return sorted(sorted(group) for group in groups.values() if len(group) > 1)

When Best isn’t best: with millions of files, hash only the first 4 KB of each first, then fully hash the files that still match. And to be certain, compare the matching files byte by byte: sha256 has no known collisions, but a check costs little.

Check yourself

A few quick questions. Ones you miss are remembered, so you can find your weak spots later.

1. Predict the outputYou run python /shop/tools/report.py from /home/sam. What does it print?

   from pathlib import Path
   
   print(Path.cwd())
   print(Path(__file__).resolve().parent)

   /home/sam, then /shop/tools/shop/tools, then /shop/tools/home/sam, then /home/sam/shop/tools, then /home/sam

   Check answer
2. Predict the outputWhat does this print?

   from pathlib import PurePosixPath
   
   p = PurePosixPath("/shop/tools/report.py")
   print(p.parent.parent / "data")

   /shop/data/shop/tools/data/datashop/data

   Check answer
3. Spot the bugadd_line should add a line to the end of the log. Why does the log only ever hold the last line?

   def add_line(path, line):
       with open(path, "w", encoding="utf-8") as f:
           f.write(line + "\n")

   "w" empties the file as it opens: "a" adds to the endwrite() replaces all the text in the filewith closes the file before write() runsThe "\n" makes the file start over

   Check answer
4. Multiple choiceA line of prices.csv is "Pens, blue",2.50. What does line.split(",") give?

   ['"Pens', ' blue"', '2.50']['Pens, blue', '2.50']['"Pens, blue"', '2.50']An error

   Check answer
5. Multiple choiceYou read a CSV that Excel saved, and the first column comes out as "\ufeffname", not "name". What fixes it?

   open() it with encoding="utf-8-sig"open() it with encoding="ascii"open() it with newline=""Call .strip() on each line

   Check answer

Flashcards

Add these to your review deck. Each card comes back a little later every time you get it right.

• Card 1Where does a relative path like data/prices.csv start?Show answer
• Card 2How do you build a path from the script's own folder?Show answer
• Card 3Why say encoding="utf-8" when you open a text file?Show answer
• Card 4What do the modes "w", "a" and "x" do?Show answer
• Card 5Why read CSV with the csv module, not split(",")?Show answer

Add 5 cards to my review deck

Cheat sheet

• A relative path starts in the current folder, where the program was run. Path.cwd() shows it.
• Build paths from the script: Path(__file__).resolve().parent, then .parent and /.
• Open with with and encoding="utf-8" ("utf-8-sig" for files from Excel).
• "w" empties first, "a" adds, "x" fails if it’s there. "b" for bytes.
• path.parent.mkdir(parents=True, exist_ok=True) makes every missing folder.
• Read CSV with the csv module and newline="", never split(",").
• Read big files a line or a piece at a time, never all at once.

Go deeper

• In the Python tutorial: reading and writing files.
• In the Python library: pathlib, open(), with its modes and encodings, and csv.
• Draw your own: where a path leads, on the Python visualizer.
• Next on the Python roadmap: Modules, packages and virtual environments, where __file__ comes up again.

Can you do these now?

If you’re not sure of any of these, scroll back to that part before moving on.

• Say where a relative path starts, and why a script that works from one folder fails from another
• Build a path from the script's own folder with Path(__file__), .parent and /
• Open files with with, the right mode and an encoding, and make folders that may be missing
• Read CSV with the csv module instead of split(",")

✓Mark as completeI already know this

Status: Not started