Python Generators and Single-Pass Iteration

A function that contains yield is a generator function. Calling it doesn’t run the body. It builds a generator object that will run the body later, a piece at a time, each time someone asks for the next value. That laziness is what makes generators good at streaming large files and building pipelines, and it’s also behind their three classic surprises: errors that show up late, data that’s gone on the second pass, and cleanup that depends on who stops iterating. This guide works through each with code you can run (output is from Python 3.14).

Calling a generator function runs nothing

Here’s a reader that’s supposed to reject non-CSV paths:

def read_orders(path):
    if not path.endswith(".csv"):
        raise ValueError(f"expected a CSV file, got {path}")
    print("opening", path)
    yield "order-1"
    yield "order-2"

orders = read_orders("orders.json")
print("generator created:", orders)
try:
    next(orders)
except ValueError as e:
    print("ValueError raised on first next():", e)

generator created: <generator object read_orders at 0x105009ff0>
ValueError raised on first next(): expected a CSV file, got orders.json

The bad path was accepted without complaint. The check ran only when something first asked for a value. In a real program that “something” might be a background job three modules away, so the traceback points at the consumer instead of the code that passed the wrong path. Interviewers like asking “when does this ValueError happen?” because many candidates assume a function’s first lines run when it’s called.

The fix is to split validation from iteration. A plain function checks its arguments immediately and returns a generator:

def read_orders(path):
    if not path.endswith(".csv"):
        raise ValueError(f"expected a CSV file, got {path}")
    return _read(path)

def _read(path):
    print("opening", path)
    yield "order-1"

try:
    read_orders("orders.json")
except ValueError as e:
    print("raised immediately:", e)

raised immediately: expected a CSV file, got orders.json

Because read_orders no longer contains yield, it’s an ordinary function, and its checks run on the call.

How a lazy pipeline actually moves data

Generator expressions chain into pipelines where each stage pulls from the previous one. Adding a print to the source makes the order visible:

def numbers():
    for n in range(1, 6):
        print(f"  produce {n}")
        yield n

evens = (n for n in numbers() if n % 2 == 0)
squares = (n * n for n in evens)
print("pipeline built, nothing produced yet")
print(next(squares))
print(list(squares))
print(list(squares))

pipeline built, nothing produced yet
  produce 1
  produce 2
4
  produce 3
  produce 4
  produce 5
[16]
[]

Read it top to bottom. Building evens and squares produced nothing. The first next(squares) asked evens for a value, which asked numbers(); 1 was produced, failed the filter, so evens asked again, got 2, passed it along, and squares returned 4. Only two source items were touched. That’s the point of laziness: work happens on demand, one item at a time, and memory use stays flat no matter how big the source is. A pipeline like (parse(line) for line in open("access.log")) can process a 20 GB log while holding one line in memory.

Then list(squares) drained the rest (16), and the second list(squares) returned []. No error, no warning. The generator is exhausted and will stay that way. If you need the data twice, either keep a list (results = list(squares)) or recreate the pipeline by calling the generator function again. The iterator exhaustion guide covers how this bites functions that receive “an iterable” and loop over it twice.

A common misconception is that a generator is “faster” than a list. It isn’t, per item; it often adds a little overhead. Its advantages are memory and latency to the first result. If you’re going to need every item, in memory, more than once, a list is simpler.

StopIteration, close() and cleanup

Inside a generator, a stray StopIteration used to end iteration silently, hiding bugs. Since Python 3.7 (PEP 479), it’s converted into a RuntimeError:

def first_item():
    it = iter([])
    yield next(it)

try:
    list(first_item())
except RuntimeError as e:
    print("RuntimeError:", e, "| cause:", repr(e.__cause__))

RuntimeError: generator raised StopIteration | cause: StopIteration()

If a generator wants to stop early, it should return. If it calls next() on something that might be empty, use the default form next(it, None) or catch StopIteration explicitly.

Generators also have a lifecycle that matters for resources. A generator that opens a file inside a with block keeps it open while it’s paused at a yield. If the consumer stops early (a break after finding a match), the generator stays paused until it’s closed. gen.close() raises GeneratorExit at the paused yield, which runs finally blocks and exits with blocks:

def countdown(n):
    try:
        while n:
            yield n
            n -= 1
    finally:
        print("cleanup ran")

g = countdown(3)
print(next(g))
g.close()

3
cleanup ran

In CPython, a generator that’s garbage collected is also closed, which is why most code gets away without calling close(). Other implementations (PyPy) collect later, and even in CPython a generator stored in a long-lived variable stays open. For anything holding a file, socket or database cursor, use contextlib.closing(gen) or have the caller own the resource and pass in the open file.

send() and yield from in one paragraph each

yield is also an expression. gen.send(value) resumes the generator and makes the paused yield evaluate to value. That turns a generator into a small stateful consumer, like a running average:

def average():
    total = count = 0
    avg = None
    while True:
        value = yield avg
        total += value
        count += 1
        avg = total / count

a = average()
next(a)                 # advance to the first yield
print(a.send(10), a.send(20), a.send(60))

10.0 15.0 30.0

You must call next() once first to reach the initial yield; sending a value into a just-started generator raises TypeError. In modern code this pattern is rare (it was the basis of pre-async coroutines), but it still appears in interview questions.

yield from sub delegates to another generator, forwarding every value and send(), and evaluates to the subgenerator’s return value. A generator’s return "summary" becomes StopIteration("summary") internally, and result = yield from task() receives "summary". That’s how you split a long generator into helpers without losing return values.

Interview practice: the empty report

A function computes statistics like this, and the second number is always zero:

def report(rows):
    valid = (r for r in rows if r["amount"] > 0)
    count = sum(1 for _ in valid)
    total = sum(r["amount"] for r in valid)
    return count, total

Explain the bug and give two fixes, saying when you’d choose each.

Answer

valid is a generator expression, so it can be iterated once. The count line consumes it entirely; when the total line iterates it again, it’s already exhausted and sum of nothing is 0. No exception tells you. One fix is to materialise it: valid = [r for r in rows if r["amount"] > 0]. That’s the right choice when the filtered rows fit comfortably in memory, which is most of the time, and it keeps the code obvious. The other fix is a single pass that computes both values together, a plain loop that increments count and adds to total for each valid row. That’s what you’d pick when rows itself is a stream (a database cursor or a file too big to hold), because it never stores the rows and also works if rows is a one-shot iterator. A good answer also mentions that itertools.tee exists but buffers everything that one copy has seen and the other hasn’t, so for this access pattern it’s no cheaper than a list.

Keep learning

• Python interview questions
• Python MCQs
• Iterators and exhaustion
• Comprehensions and readable transformations
• Official docs: Generators in the functional programming HOWTO