I like tools that feel like cheating, but aren't. itertools is one of those. It's just code, but it saves you from writing the same sad loop for the 900th time. I first ran into it while refactoring a data pipeline and had the very normal reaction of, "wait, Python ships with this?"
The mental shift is simple: stop building big lists just to immediately loop over them. Start streaming values. Keep the pipeline moving. I spent way too long early on wondering why my scripts were eating all my RAM before I finally internalized this.
If you've ever stared at a nested loop and thought "there has to be a cleaner way," you're ready. I've been there. More than once. (Usually after a long night, not a short one.)
Iterators, generators, and why they matter
An iterator gives you values one at a time. A generator is a convenient way to make an iterator. The win is that you can process data without loading it all at once. No fluff.
Here's the classic "I accidentally made a huge list" problem:
## Oops: builds a list of 10 million ints
data = [i for i in range(10_000_000)]
total = sum(data)
Now the streaming version:
total = sum(range(10_000_000)) # range is already lazy-ish
itertools takes that idea and turns it into a full set of LEGO bricks for iteration.
The parts I actually reach for
cycle() and repeat(): controlled repetition
cycle() loops forever over an iterable. Great for round-robin behavior.
from itertools import cycle
workers = ["gpu0", "gpu1", "gpu2"]
assign = cycle(workers)
for job_id in range(7):
print(job_id, next(assign))
repeat() repeats a single value. Often paired with map().
from itertools import repeat
## Apply the same threshold to many items without writing a loop
thresholds = repeat(0.8, 5)
If you don't cap these, they'll happily run forever. Try it with cycle(). It'll sit there for days, doing exactly what you told it to do.
chain(): stop concatenating lists
If you're doing a + b + c just to iterate, you're paying for copies.
from itertools import chain
for x in chain([1, 2], [3, 4], [5]):
print(x)
I use this constantly to stitch together multiple sources (files, shards, batches) into one stream you can process the same way.
Cartesian products and search space explosions
product(): grid search without the ugly loops
from itertools import product
lrs = [1e-3, 1e-4]
batches = [16, 32, 64]
for lr, bs in product(lrs, batches):
print(lr, bs)
Perfect for small experiments.
Ordering matters (sometimes): permutations vs combinations
permutations(): order matters
from itertools import permutations
for p in permutations(["A", "B", "C"], 2):
print(p)
combinations(): order doesn't
from itertools import combinations
for c in combinations(["A", "B", "C"], 2):
print(c)
I use these a lot for testing. If you're validating a system that should behave the same under different input orders, permutations() gives you the test cases without writing custom logic.
A concrete pattern: building a small, fast pipeline
Say you're processing logs and want to scan for "interesting" lines across multiple sources. Chain the sources, filter on the fly, and take a sample, no giant intermediate list required.
from itertools import chain, islice
def interesting(line: str) -> bool:
return "ERROR" in line or "timeout" in line
sources = [open("app.log"), open("worker.log"), open("db.log")]
stream = chain.from_iterable(sources)
hits = (line for line in stream if interesting(line))
for line in islice(hits, 20):
print(line.strip())
This style scales down nicely too. I'm a fan of getting smaller systems running on constrained hardware: phones, Jetson-class boxes, little manufacturing rigs. Streaming patterns matter a lot more when you can't buy your way out with more RAM.
Best practices (aka: ways to avoid pain)
- Prefer lazy pipelines until you truly need a list.
- Be careful with infinite iterators (
cycle,count,repeat). Add limits likeislice. - If debugging gets weird, materialize a small slice:
python from itertools import islice preview = list(islice(stream, 10)) - Read the docs. Seriously. They're dry, but dense in the good way. I've learned more from the standard library docs than from most blog posts.
itertools is one of those modules that makes you feel like you got better at Python without changing languages or buying a new framework. It's already in the box. I wish I'd picked it up earlier.
-Sethers