Python Comprehensions and Readable Transformations

A comprehension is a loop written as an expression: [f(x) for x in items if keep(x)]. Used well, it says “build this collection from that one” in a single readable line. Used badly, it hides three nested loops, a side effect and a filter inside a line nobody wants to review. Interviewers ask about comprehensions for both reasons: to check you know the scoping rules, and to see whether you know when to stop using one.

What a comprehension does with its loop variable

In Python 3, the loop variable belongs to the comprehension, not to the surrounding code:

x = "outer"
squares = [x * x for x in range(4)]
print(squares, x)

[0, 1, 4, 9] outer

x inside the brackets is a different variable from the x outside, so the outer value survives. Python 2 leaked list-comprehension variables into the enclosing scope, and you’ll still find older blog posts warning about it. In Python 3 that’s fixed for list, set and dict comprehensions and generator expressions.

Under the hood, a comprehension used to be compiled as a hidden nested function, which is how it got its own scope. Since Python 3.12 (PEP 709) list, dict and set comprehensions are inlined for speed, but the scoping behaviour was deliberately kept the same. You shouldn’t be able to tell the difference except in tracebacks and benchmarks.

The walrus exception

An assignment expression (:=) inside a comprehension binds in the enclosing scope, on purpose, so you can keep a value you computed while filtering:

orders = [{"id": 1, "total": "19.90"}, {"id": 2, "total": "n/a"}, {"id": 3, "total": "250.00"}]
big = [t for o in orders if (t := o["total"]).replace(".", "").isdigit() and float(t) > 100]
print(big, "| t leaked:", t)

['250.00'] | t leaked: 250.00

t is visible after the comprehension and holds the last value assigned. That’s useful for “find the first match and keep it” with any(), but in a long function it’s a variable appearing out of nowhere, so use it sparingly.

The class-body trap

This one catches experienced developers. Inside a class body, a comprehension can’t see class attributes:

class Config:
    base = 10
    values = [base + i for i in range(3)]

NameError: name 'base' is not defined

Class bodies don’t create an enclosing scope for nested functions, and the comprehension behaves like a nested function. The one exception is the first iterable, which is evaluated in the class body before the comprehension starts. So this works, if you need it:

class Config:
    base = 10
    values = [b + i for b in [base] for i in range(3)]
print(Config.values)

[10, 11, 12]

In real code it’s cleaner to compute such values outside the class or in a classmethod. The point for an interview is being able to explain why the first one fails: it’s the class-scope rule, not a comprehension bug.

Reading nested comprehensions

Multiple for clauses run in the same order you’d write nested loops, left to right, outermost first:

matrix = [[1, 2, 3], [4, 5, 6]]
print([n for row in matrix for n in row])
print([[row[i] for row in matrix] for i in range(3)])

[1, 2, 3, 4, 5, 6]
[[1, 4], [2, 5], [3, 6]]

The first line flattens: for row in matrix is the outer loop, for n in row the inner one. The second is a comprehension inside a comprehension, which transposes the matrix. People often reverse the clause order in the flattening case and get NameError: name 'row' is not defined, because the inner clause refers to a name the outer clause hasn’t introduced yet. A trick that helps: write the nested for loops on separate lines, then delete the colons and line breaks.

List, generator, dict or set

The bracket type decides what you get, and choosing the right one is part of writing readable code:

• [...] builds a list in memory. Use it when you need indexing, len(), or more than one pass.
• (...) is a generator expression. It produces items one at a time and stores none of them.
• {k: v ...} builds a dict, {x ...} a set.

The memory difference is large on big inputs:

import sys
nums = range(1_000_000)
print(sys.getsizeof([n * 2 for n in nums]), sys.getsizeof((n * 2 for n in nums)))
print(sum(n * 2 for n in nums))

8448728 208
999999000000

The list costs about 8 MB for its pointer array alone (the int objects themselves are extra); the generator object is 208 bytes no matter how many items it will yield. When a comprehension feeds straight into sum, max, any, "".join or a for loop, drop the square brackets and pass a generator expression. You can omit the extra parentheses when it’s the only argument, as in sum(n * 2 for n in nums). The catch is that a generator can only be consumed once (see iterator exhaustion).

A dict comprehension is the idiomatic way to index records: {u["id"]: u for u in users}. Remember that duplicate keys silently keep the last value, which can hide data problems if ids aren’t unique.

When a loop reads better

Comprehensions are for building a collection. They’re a poor fit when:

• The point is a side effect. [print(x) for x in items] or [db.save(r) for r in rows] builds a throwaway list of None values and tells the reader you wanted a list. Write a for loop.
• The logic needs more than one condition and a transformation that won’t fit on a line. Once you’re nesting a conditional expression inside a filter inside two for clauses, a loop with named intermediate variables is easier to review and to debug with a breakpoint.
• You need error handling. You can’t put try/except inside a comprehension. Moving the risky part into a small helper function (parse_total(o)) keeps the comprehension readable and the error handling explicit.

A reasonable rule: if you have to read a comprehension twice to know what it produces, rewrite it.

Interview exercise: clean up a transformation

A teammate wrote this to get the email domains of active users, lower-cased and de-duplicated:

domains = list(set([u["email"].split("@")[1].lower() for u in users if u["active"] == True and "@" in u["email"]]))

What would you change, and why?

Answer

Several small things. Use a set comprehension directly, {... for u in users if ...}, instead of building a list and converting it, which allocates twice. Drop == True; if u["active"] is the idiom, and == True also rejects truthy values such as the string "yes" (though 1 == True passes, which makes the check inconsistent as well as noisy). Use rpartition("@")[2] rather than split("@")[1] so an address with an unusual local part doesn’t pick the wrong segment. Finally, decide whether the caller needs a list at all: if they only check membership or iterate, return the set. If order matters, return sorted(...), because converting a set to a list gives an arbitrary order that can change between runs. The result is domains = {u["email"].rpartition("@")[2].lower() for u in users if u["active"] and "@" in u["email"]}, and if that still feels long, a helper like email_domain(u) makes the comprehension read like a sentence.

Further practice

• Python interview questions
• Python MCQs
• Closures and late binding
• Generators and single-pass iteration
• Official docs: List comprehensions in the tutorial