Python Iterators and Exhaustion

A for loop in Python works on anything iterable, and that flexibility hides a distinction that matters: some objects can be looped over as many times as you like, and some only once. A list is the first kind. The object returned by map() is the second. When code written for lists receives the second kind, it doesn’t crash; it quietly works on empty data. Interviewers ask about this because the bug is common, silent, and easy to explain once you know the protocol.

Iterable and iterator are different roles

The iteration protocol has two parts.

• An iterable has __iter__(), which returns an iterator. Lists, tuples, dicts, sets, strings and range are iterables.
• An iterator has __next__(), which returns the next item or raises StopIteration when there are none left. It also has __iter__(), which returns itself.

That last detail is the key. Calling iter() on a list builds a new iterator each time, starting from the beginning. Calling iter() on an iterator gives you back the same object, wherever it currently is:

nums = [3, 1, 2]
it = iter(nums)
print(iter(nums) is nums, iter(it) is it)
print(next(it), next(it), next(it))
next(it)

False True
3 1 2
StopIteration

A for loop calls iter() once, then next() until StopIteration. So looping over a list twice gets two fresh iterators. Looping over an iterator twice continues the same one, and the second loop starts at the end.

Which objects are one-shot? More than people expect: map, filter, zip, enumerate, reversed, generator objects and generator expressions, open files, csv.reader and csv.DictReader, itertools results, database cursors, and dict.items()’s iterator (though dict.items() itself is a reusable view). A useful habit is to ask, for any value you loop over, “if I loop again, do I get the data again?”

The function that works for lists only

Here’s a normaliser that divides each value by the total:

def normalise(values):
    total = sum(values)
    return [v / total for v in values]

print(normalise([1, 3]))
print(normalise(map(int, ["1", "3"])))

[0.25, 0.75]
[]

With a list, sum and the comprehension each get their own pass. With a map, sum consumes every item, and the comprehension finds the iterator exhausted. The result is an empty list instead of an error. Type hints don’t catch it if the parameter is annotated Iterable[int], because map is an iterable; the hint that says “I’ll iterate more than once” is Collection[int] or Sequence[int].

The same thing happens with files. A CSV reader can only be read once:

import io, csv
f = io.StringIO("id,qty\n1,2\n2,5\n")
rows = csv.DictReader(f)
print(sum(int(r["qty"]) for r in rows), sum(int(r["qty"]) for r in rows))

7 0

The second total is 0. In a report that computes a sum and then a count, the count will be wrong in exactly the way that looks plausible.

Operations that consume items

Exhaustion isn’t only about full loops. Several common operations advance an iterator as a side effect.

in on an iterator walks forward until it finds a match, and those items are gone:

m = map(str.upper, ["a", "b", "c"])
print("B" in m, list(m))

True ['C']

"A" and "B" were consumed by the membership test. On a list, in doesn’t change anything.

zip with uneven inputs stops at the shortest, but it may have already pulled an item from a longer iterator before noticing:

prices = iter([10, 20, 30, 40, 50])
names = ["a", "b"]
print(list(zip(prices, names)), list(prices))

[(10, 'a'), (20, 'b')] [40, 50]

30 vanished. On the third round, zip took 30 from prices, then asked names for a third item, got StopIteration, and threw 30 away. If the shorter input comes first in the argument list, nothing is lost from the longer one, but relying on argument order is fragile. Python 3.10’s zip(..., strict=True) raises ValueError when lengths differ, which is usually what you want when the inputs are supposed to line up.

next() and islice are the deliberate versions. Skipping a header row is a good use:

from itertools import islice
lines = iter(["header", "r1", "r2", "r3"])
header = next(lines)
print(header, list(islice(lines, 2)), list(lines))

header ['r1', 'r2'] ['r3']

Here, the shared position is a feature: each step picks up where the previous one stopped.

Designing functions that survive iterators

You have three reasonable choices when a function needs more than one pass.

Materialise once at the top. values = list(values) costs memory proportional to the input but makes the rest of the function behave as if it got a list. It’s the right default for small to moderate data.

Do the work in one pass. A loop that accumulates total and count together, or that writes results as it goes, works for any iterable including streams too large to store. It’s the right choice for files and cursors.

Reject one-shot iterators loudly. If the function truly needs a re-iterable collection and the data might be large, fail fast instead of silently producing an empty result:

def normalise_safe(values):
    if iter(values) is values:
        raise TypeError("normalise() needs a re-iterable collection, not a one-shot iterator")
    total = sum(values)
    return [v / total for v in values]

normalise_safe(map(int, ["1", "3"]))

TypeError: normalise() needs a re-iterable collection, not a one-shot iterator

The iter(values) is values check uses the protocol rule from earlier: only iterators return themselves.

And if you control the data source, make it re-iterable. A class whose __iter__ is a generator method produces a fresh iterator on every loop:

class Orders:
    def __init__(self, path):
        self.path = path
    def __iter__(self):
        yield from ("order-1", "order-2")   # real code would open self.path here

o = Orders("orders.csv")
print(list(o), list(o))

['order-1', 'order-2'] ['order-1', 'order-2']

Each loop re-reads the source. That trades repeat I/O for memory, which is often the right trade for big files.

A misconception worth correcting: itertools.tee doesn’t make an iterator replayable for free. It buffers every item one copy has consumed and the other hasn’t, so if you fully consume one copy before the other, it stores everything, the same as a list.

Interview practice: the duplicate checker

This function is supposed to report whether a sequence has duplicates and how many items it has. Called with a list it works; called with a generator of user ids from a database cursor, it returns (True, 0) for every non-empty input, even when all the ids are unique. Why, and what would you write instead?

def summarise(ids):
    has_dupes = len(set(ids)) != len(list(ids))
    return has_dupes, len(list(ids))

Answer

set(ids) consumes the generator completely. list(ids) on the same line then gets an empty list, and so does the second list(ids). For three unique ids you’d expect (False, 3); running it with (x for x in [1, 2, 3]) returns (True, 0), because the comparison becomes 3 != 0. An empty generator happens to return the right-looking (False, 0), so a test with no data would pass. The function makes three passes over input that supports one. A single-pass version keeps a seen set and a counter: for each id, increment the count, and if the id is already in seen, record a duplicate; otherwise add it. That’s O(n) time, uses memory only for distinct ids, and works for lists, generators and cursors alike. If the caller needs the ids afterwards, materialise once with ids = list(ids) at the top and say so in the docstring.

Keep going

• Python interview questions
• Python MCQs
• Generators and single-pass iteration
• Comprehensions and readable transformations
• Official docs: Iterator types