I keep writing tiny scripts that crawl directory trees. Rename files. Hash them. Find duplicates. My favorite discovery was fourteen copies of the same CSV scattered across folders. I called them "backups," but really I was just a digital hoarder with commitment issues.
Python gives you two common tools for recursive file listing: os.walk() and pathlib. Both work. One makes me feel like I'm writing 2009-era glue code because I forgot pathlib exists.
The os.walk() way (and the classic foot-gun)
os.walk() yields a 3-tuple for each directory:
dirpath: the current directory (full path)dirnames: subdirectory names (names only, not full paths)filenames: file names indirpath(names only)
That "names only" part is where bugs live. It's why your script works on your laptop and then explodes in production when someone runs it from a different directory.
A working os.walk() pattern that returns full paths
If you want a flat list of full file paths, you must join dirpath with each filename.
Here's a solid pattern. I'm using a queue so it's obvious what gets visited next, and I break after one level so each queue item is handled once:
import os
def list_files_walk(root):
files = []
dirlist = [root]
while dirlist:
current = dirlist.pop()
for dirpath, dirnames, filenames in os.walk(current):
# Subdirectories: join to get full paths
dirlist.extend([os.path.join(dirpath, d) for d in dirnames])
# Files: join here too
files.extend([os.path.join(dirpath, name) for name in filenames])
# One level per queue item
break
return files
Two real-world notes:
os.walk()is already recursive. The queue +breakisn't required. It's just explicit.- If you don't need custom traversal, don't build it. Let
os.walk(root)do its job.
The simpler version most people should use:
import os
def list_files_walk_simple(root):
files = []
for dirpath, dirnames, filenames in os.walk(root):
files.extend([os.path.join(dirpath, name) for name in filenames])
return files
This works. It's also where people start "optimizing" and break everything.
The common bug
People do this:
dirlist.extend(dirnames) # nope
Then they spend an hour debugging why their script is crawling /home instead of /data.
dirnames are not full paths. They're names relative to dirpath.
Concrete example:
- You're walking
/data/projects os.walk()yieldsdirpath="/data/projects"anddirnames=["clientA", "clientB"]- If you push
"clientA"into your queue, your script will next try to walk./clientArelative to your current working directory
Best case: empty results. Worst case: you rename half your Downloads folder before you notice.
If you want to be defensive, normalize the root up front:
root = os.path.abspath(root)
And verify it exists:
if not os.path.isdir(root):
raise ValueError(f"Not a directory: {root}")
That won't fix the dirnames mistake, but it will catch typos like /dat/projects before they waste your time.
Things that will bite you (anti-patterns I still see)
I see these everywhere. They're not clever.
-
[dirpath] * len(filenames)
This creates a list of repeated references. It works with strings because strings are immutable. Copy this pattern with mutable objects and you'll get a bug that makes you question your career choices. -
zip([dirpath] * len(filenames), filenames)
Now you have tuples instead of full paths. You still need to join them. You've invented extra work for yourself. -
map(lambda ...)
Use a list comprehension. It's clearer and you won't spend five minutes decoding a lambda.
The clean way to join paths is this:
[os.path.join(dirpath, name) for name in filenames]
It's boring. That's the point.
os.walk() wins when you need traversal control
Two times I actually use it:
- Skip directories early
You can edit dirnames in-place to stop os.walk() from descending:
import os
def list_files_skip_stuff(root):
files = []
for dirpath, dirnames, filenames in os.walk(root):
dirnames[:] = [d for d in dirnames if d not in {".git", "venv", "__pycache__","node_modules"}]
files.extend([os.path.join(dirpath, f) for f in filenames])
return files
This matters on big repos. Skipping .git and node_modules turns a five-minute crawl into ten seconds.
- Follow symlinks (carefully)
os.walk(..., followlinks=True) follows symlinks. This is occasionally useful and frequently how you create infinite loops. If you turn it on, add cycle protection (visited inodes) or be ready for a bad time.
The pathlib way (shorter and harder to break)
If you want "give me all files under this directory," pathlib is the clean answer:
from pathlib import Path
def list_files_pathlib(root):
return [str(p) for p in Path(root).rglob("*") if p.is_file()]
Done. It walks recursively, yields Path objects, and you filter with is_file(). Convert to strings if you need them.
Why Path objects are nicer than strings
File scripts always grow. You start with "list files" and end up grouping by extension, filtering by size, and generating reports.
Path makes that less annoying:
p.name,p.suffix,p.stemp.stat().st_sizep.open("rb")/p.read_bytes()(don'tread_bytes()on huge files)
Example: find duplicate CSVs by hashing their content:
from pathlib import Path
import hashlib
def sha256_file(path: Path, chunk_size: int = 1024 * 1024) -> str:
h = hashlib.sha256()
with path.open("rb") as f:
for chunk in iter(lambda: f.read(chunk_size), b""):
h.update(chunk)
return h.hexdigest()
def find_duplicate_csvs(root):
hashes = {}
for p in Path(root).rglob("*.csv"):
if not p.is_file():
continue
digest = sha256_file(p)
hashes.setdefault(digest, []).append(p)
return {h: paths for h, paths in hashes.items() if len(paths) > 1}
That's how you go from "I probably have duplicates" to "I have fourteen copies of this CSV and here's where they are."
rglob("*") vs rglob("*.ext")
Filter early if you know what you want:
from pathlib import Path
def list_images(root):
exts = {".jpg", ".jpeg", ".png", ".webp"}
return [p for p in Path(root).rglob("*") if p.is_file() and p.suffix.lower() in exts]
I returned Path objects here. I only convert to strings at the edges-printing, JSON output, CLI args. Keeping Path objects internal means less path-joining code.
The error handling problem
File trees are messy. You'll hit permission errors, broken symlinks, and files that vanish mid-walk because something else is cleaning up.
With pathlib, p.is_file() can raise exceptions in edge cases like permission errors. If you're crawling system directories, wrap it:
from pathlib import Path
def list_files_pathlib_safe(root):
files = []
for p in Path(root).rglob("*"):
try:
if p.is_file():
files.append(p)
except OSError:
continue
return files
os.walk() also hits errors, but you can pass an onerror callback. Either way, crawling system directories means planning for broken stuff.
Which one should you use?
- Use
Path(...).rglob()if you want readable code you won't hate later. - Use
os.walk()if you need fine control over traversal (like pruning directories), or you're in a codebase that already uses it.
Both work. The difference is how many ways you can screw it up.
With os.walk(), you keep forgetting that dirnames are just names and you need os.path.join(). With pathlib, you write what you mean.
Most of my scripts start as "I'll write something quick." Then I find fourteen duplicate CSVs and realize I need something that's hard to break. pathlib wins.
-Sethers