sethserver.com Subscribe
A black tree with circuit board-style roots grows from a laptop displaying the Python logo, with colorful folders in pink, orange, blue, and green as leaves against a light mint background, surrounded by mathematical symbols and geometric shapes.

Recursively List Files in Python: os.walk vs pathlib

By Seth Black • Updated: September 17, 2026

Python · 6 min read

Like this kind of writing? Get one email a week with notes on startups, AI, and the occasional strong opinion about Python: subscribe to the newsletter.

I keep writing tiny scripts that crawl directory trees. Rename files. Hash them. Find duplicates. My favorite discovery was fourteen copies of the same CSV scattered across folders. I called them "backups," but really I was just a digital hoarder with commitment issues.

Python gives you two common tools for recursive file listing: os.walk() and pathlib. Both work. One makes me feel like I'm writing 2009-era glue code because I forgot pathlib exists.

The os.walk() way (and the classic foot-gun)

os.walk() yields a 3-tuple for each directory:

That "names only" part is where bugs live. It's why your script works on your laptop and then explodes in production when someone runs it from a different directory.

A working os.walk() pattern that returns full paths

If you want a flat list of full file paths, you must join dirpath with each filename.

Here's a solid pattern. I'm using a queue so it's obvious what gets visited next, and I break after one level so each queue item is handled once:

import os

def list_files_walk(root):
    files = []
    dirlist = [root]

    while dirlist:
        current = dirlist.pop()

        for dirpath, dirnames, filenames in os.walk(current):
            # Subdirectories: join to get full paths
            dirlist.extend([os.path.join(dirpath, d) for d in dirnames])

            # Files: join here too
            files.extend([os.path.join(dirpath, name) for name in filenames])

            # One level per queue item
            break

    return files

Two real-world notes:

The simpler version most people should use:

import os

def list_files_walk_simple(root):
    files = []
    for dirpath, dirnames, filenames in os.walk(root):
        files.extend([os.path.join(dirpath, name) for name in filenames])
    return files

This works. It's also where people start "optimizing" and break everything.

The common bug

People do this:

dirlist.extend(dirnames)  # nope

Then they spend an hour debugging why their script is crawling /home instead of /data.

dirnames are not full paths. They're names relative to dirpath.

Concrete example:

Best case: empty results. Worst case: you rename half your Downloads folder before you notice.

If you want to be defensive, normalize the root up front:

root = os.path.abspath(root)

And verify it exists:

if not os.path.isdir(root):
    raise ValueError(f"Not a directory: {root}")

That won't fix the dirnames mistake, but it will catch typos like /dat/projects before they waste your time.

Things that will bite you (anti-patterns I still see)

I see these everywhere. They're not clever.

The clean way to join paths is this:

[os.path.join(dirpath, name) for name in filenames]

It's boring. That's the point.

os.walk() wins when you need traversal control

Two times I actually use it:

  1. Skip directories early

You can edit dirnames in-place to stop os.walk() from descending:

import os

def list_files_skip_stuff(root):
    files = []
    for dirpath, dirnames, filenames in os.walk(root):
        dirnames[:] = [d for d in dirnames if d not in {".git", "venv", "__pycache__","node_modules"}]
        files.extend([os.path.join(dirpath, f) for f in filenames])
    return files

This matters on big repos. Skipping .git and node_modules turns a five-minute crawl into ten seconds.

  1. Follow symlinks (carefully)

os.walk(..., followlinks=True) follows symlinks. This is occasionally useful and frequently how you create infinite loops. If you turn it on, add cycle protection (visited inodes) or be ready for a bad time.

The pathlib way (shorter and harder to break)

If you want "give me all files under this directory," pathlib is the clean answer:

from pathlib import Path

def list_files_pathlib(root):
    return [str(p) for p in Path(root).rglob("*") if p.is_file()]

Done. It walks recursively, yields Path objects, and you filter with is_file(). Convert to strings if you need them.

Why Path objects are nicer than strings

File scripts always grow. You start with "list files" and end up grouping by extension, filtering by size, and generating reports.

Path makes that less annoying:

Example: find duplicate CSVs by hashing their content:

from pathlib import Path
import hashlib

def sha256_file(path: Path, chunk_size: int = 1024 * 1024) -> str:
    h = hashlib.sha256()
    with path.open("rb") as f:
        for chunk in iter(lambda: f.read(chunk_size), b""):
            h.update(chunk)
    return h.hexdigest()

def find_duplicate_csvs(root):
    hashes = {}
    for p in Path(root).rglob("*.csv"):
        if not p.is_file():
            continue
        digest = sha256_file(p)
        hashes.setdefault(digest, []).append(p)

    return {h: paths for h, paths in hashes.items() if len(paths) > 1}

That's how you go from "I probably have duplicates" to "I have fourteen copies of this CSV and here's where they are."

rglob("*") vs rglob("*.ext")

Filter early if you know what you want:

from pathlib import Path

def list_images(root):
    exts = {".jpg", ".jpeg", ".png", ".webp"}
    return [p for p in Path(root).rglob("*") if p.is_file() and p.suffix.lower() in exts]

I returned Path objects here. I only convert to strings at the edges-printing, JSON output, CLI args. Keeping Path objects internal means less path-joining code.

The error handling problem

File trees are messy. You'll hit permission errors, broken symlinks, and files that vanish mid-walk because something else is cleaning up.

With pathlib, p.is_file() can raise exceptions in edge cases like permission errors. If you're crawling system directories, wrap it:

from pathlib import Path

def list_files_pathlib_safe(root):
    files = []
    for p in Path(root).rglob("*"):
        try:
            if p.is_file():
                files.append(p)
        except OSError:
            continue
    return files

os.walk() also hits errors, but you can pass an onerror callback. Either way, crawling system directories means planning for broken stuff.

Which one should you use?

Both work. The difference is how many ways you can screw it up.

With os.walk(), you keep forgetting that dirnames are just names and you need os.path.join(). With pathlib, you write what you mean.

Most of my scripts start as "I'll write something quick." Then I find fourteen duplicate CSVs and realize I need something that's hard to break. pathlib wins.

-Sethers

Share this post
Newsletter

One email, once a week.

Notes on databases, systems, and the occasional strong opinion about Python. No spam, unsubscribe anytime.

Seth Black
Written by

Seth Black

Engineer and founder based in Texas. Writes about databases, AI, and running things in production. Embeds in small teams as lead engineer.

More from Python

View all →
Python

OpenAI Bought Astral - and my fav tool uv

Mar 23, 2026
Python

Python Pydantic Validation: Stop Writing Manual Checks

Mar 04, 2026
Python

Python asyncio Recipes

Mar 04, 2026