---
title: "Recursively List Files in Python: os.walk vs pathlib"
description: "How to recursively list files in Python with os.walk() and pathlib.rglob(). Covers when to use each, the dirpath mistake that breaks results, and code examples."
image: "https://cdn.sethserver.com/blog-images/recursively-list-files-social-1767028872.webp"
---

# Recursively List Files in Python: os.walk vs pathlib

I keep writing tiny scripts that crawl directory trees. Rename files. Hash them. Find duplicates. My favorite discovery was **fourteen copies of the same CSV** scattered across folders. I called them "backups," but really I was just a digital hoarder with commitment issues.

Python gives you two common tools for recursive file listing: `os.walk()` and `pathlib`. Both work. One makes me feel like I'm writing 2009-era glue code because I forgot `pathlib` exists.

## The `os.walk()` way (and the classic foot-gun)

`os.walk()` yields a 3-tuple for each directory:

- `dirpath`: the current directory (full path)
- `dirnames`: subdirectory names (**names only**, not full paths)
- `filenames`: file names in `dirpath` (**names only**)

That "names only" part is where bugs live. It's why your script works on your laptop and then explodes in production when someone runs it from a different directory.

### A working `os.walk()` pattern that returns full paths

If you want a flat list of full file paths, you must join `dirpath` with each filename.

Here's a solid pattern. I'm using a queue so it's obvious what gets visited next, and I `break` after one level so each queue item is handled once:

```python
import os

def list_files_walk(root):
    files = []
    dirlist = [root]

    while dirlist:
        current = dirlist.pop()

        for dirpath, dirnames, filenames in os.walk(current):
            # Subdirectories: join to get full paths
            dirlist.extend([os.path.join(dirpath, d) for d in dirnames])

            # Files: join here too
            files.extend([os.path.join(dirpath, name) for name in filenames])

            # One level per queue item
            break

    return files
```

Two real-world notes:

- `os.walk()` is already recursive. The queue + `break` isn't required. It's just explicit.
- If you don't need custom traversal, don't build it. Let `os.walk(root)` do its job.

The simpler version most people should use:

```python
import os

def list_files_walk_simple(root):
    files = []
    for dirpath, dirnames, filenames in os.walk(root):
        files.extend([os.path.join(dirpath, name) for name in filenames])
    return files
```

This works. It's also where people start "optimizing" and break everything.

### The common bug

People do this:

```python
dirlist.extend(dirnames)  # nope
```

Then they spend an hour debugging why their script is crawling `/home` instead of `/data`.

`dirnames` are not full paths. They're names relative to `dirpath`.

Concrete example:

- You're walking `/data/projects`
- `os.walk()` yields `dirpath="/data/projects"` and `dirnames=["clientA", "clientB"]`
- If you push `"clientA"` into your queue, your script will next try to walk `./clientA` relative to your current working directory

Best case: empty results. Worst case: you rename half your Downloads folder before you notice.

If you want to be defensive, normalize the root up front:

```python
root = os.path.abspath(root)
```

And verify it exists:

```python
if not os.path.isdir(root):
    raise ValueError(f"Not a directory: {root}")
```

That won't fix the `dirnames` mistake, but it *will* catch typos like `/dat/projects` before they waste your time.

### Things that will bite you (anti-patterns I still see)

I see these everywhere. They're not clever.

- `[dirpath] * len(filenames)`
  This creates a list of repeated references. It works with strings because strings are immutable. Copy this pattern with mutable objects and you'll get a bug that makes you question your career choices.
- `zip([dirpath] * len(filenames), filenames)`
  Now you have tuples instead of full paths. You still need to join them. You've invented extra work for yourself.
- `map(lambda ...)`
  Use a list comprehension. It's clearer and you won't spend five minutes decoding a lambda.

The clean way to join paths is this:

```python
[os.path.join(dirpath, name) for name in filenames]
```

It's boring. That's the point.

### `os.walk()` wins when you need traversal control

Two times I actually use it:

1. **Skip directories early**

You can edit `dirnames` in-place to stop `os.walk()` from descending:

```python
import os

def list_files_skip_stuff(root):
    files = []
    for dirpath, dirnames, filenames in os.walk(root):
        dirnames[:] = [d for d in dirnames if d not in {".git", "venv", "__pycache__","node_modules"}]
        files.extend([os.path.join(dirpath, f) for f in filenames])
    return files
```

This matters on big repos. Skipping `.git` and `node_modules` turns a five-minute crawl into ten seconds.

1. **Follow symlinks (carefully)**

`os.walk(..., followlinks=True)` follows symlinks. This is occasionally useful and frequently how you create infinite loops. If you turn it on, add cycle protection (visited inodes) or be ready for a bad time.

## The `pathlib` way (shorter and harder to break)

If you want "give me all files under this directory," `pathlib` is the clean answer:

```python
from pathlib import Path

def list_files_pathlib(root):
    return [str(p) for p in Path(root).rglob("*") if p.is_file()]
```

Done. It walks recursively, yields `Path` objects, and you filter with `is_file()`. Convert to strings if you need them.

### Why `Path` objects are nicer than strings

File scripts always grow. You start with "list files" and end up grouping by extension, filtering by size, and generating reports.

`Path` makes that less annoying:

- `p.name`, `p.suffix`, `p.stem`
- `p.stat().st_size`
- `p.open("rb")` / `p.read_bytes()` (don't `read_bytes()` on huge files)

Example: find duplicate CSVs by hashing their content:

```python
from pathlib import Path
import hashlib

def sha256_file(path: Path, chunk_size: int = 1024 * 1024) -> str:
    h = hashlib.sha256()
    with path.open("rb") as f:
        for chunk in iter(lambda: f.read(chunk_size), b""):
            h.update(chunk)
    return h.hexdigest()

def find_duplicate_csvs(root):
    hashes = {}
    for p in Path(root).rglob("*.csv"):
        if not p.is_file():
            continue
        digest = sha256_file(p)
        hashes.setdefault(digest, []).append(p)

    return {h: paths for h, paths in hashes.items() if len(paths) > 1}
```

That's how you go from "I probably have duplicates" to "I have fourteen copies of this CSV and here's where they are."

### `rglob("*")` vs `rglob("*.ext")`

Filter early if you know what you want:

```python
from pathlib import Path

def list_images(root):
    exts = {".jpg", ".jpeg", ".png", ".webp"}
    return [p for p in Path(root).rglob("*") if p.is_file() and p.suffix.lower() in exts]
```

I returned `Path` objects here. I only convert to strings at the edges-printing, JSON output, CLI args. Keeping `Path` objects internal means less path-joining code.

### The error handling problem

File trees are messy. You'll hit permission errors, broken symlinks, and files that vanish mid-walk because something else is cleaning up.

With `pathlib`, `p.is_file()` can raise exceptions in edge cases like permission errors. If you're crawling system directories, wrap it:

```python
from pathlib import Path

def list_files_pathlib_safe(root):
    files = []
    for p in Path(root).rglob("*"):
        try:
            if p.is_file():
                files.append(p)
        except OSError:
            continue
    return files
```

`os.walk()` also hits errors, but you can pass an `onerror` callback. Either way, crawling system directories means [planning for broken stuff](https://www.sethserver.com/python/python-logging-done-right.html).

## Which one should you use?

- Use **`Path(...).rglob()`** if you want readable code you won't hate later.
- Use **`os.walk()`** if you need fine control over traversal (like pruning directories), or you're in a codebase that already uses it.

Both work. The difference is how many ways you can screw it up.

With `os.walk()`, you keep forgetting that `dirnames` are just names and you need `os.path.join()`. With `pathlib`, you write what you mean.

Most of my scripts start as "I'll write something quick." Then I find fourteen duplicate CSVs and realize I need something that's hard to break. `pathlib` wins.

-Sethers


```json
{"@context": "https://schema.org", "@type": "BlogPosting", "headline": "Recursively List Files in Python: os.walk vs pathlib", "datePublished": "2020-12-01T00:00:00", "dateModified": "2026-09-17T19:47:36", "image": {"@type": "ImageObject", "url": "https://cdn.sethserver.com/blog-images/recursively-list-files-social-1767028872.webp", "width": 400, "height": 400}, "url": "https://www.sethserver.com/python/recursively-list-files.html", "author": {"@type": "Person", "name": "Seth Black", "url": "https://www.linkedin.com/in/seth-black-tx"}, "publisher": {"@type": "Organization", "name": "Sethserver", "logo": {"@type": "ImageObject", "url": "https://www.sethserver.com/static/images/PaperAirplane512.png"}}, "description": "How to recursively list files in Python with os.walk() and pathlib.rglob(). Covers when to use each, the dirpath mistake that breaks results, and code examples.", "articleSection": "Technology"}
{"@context": "https://schema.org", "@type": "BreadcrumbList", "itemListElement": [{"@type": "ListItem", "position": 1, "name": "sethserver", "item": "https://www.sethserver.com/"}, {"@type": "ListItem", "position": 2, "name": "Python", "item": "https://www.sethserver.com/python/"}, {"@type": "ListItem", "position": 3, "name": "Recursively List Files in Python: os.walk vs pathlib", "item": "https://www.sethserver.com/python/recursively-list-files.html"}]}
```
