Comprehension Syntax

The previous two lessons measured the difference between square brackets and parentheses, and separated by number what laziness reduces and what it does not. But the square brackets themselves were never opened up: in every measurement, comprehension stood only as “the form producing everything now.”

Yet the same notation can build three separate containers. Square brackets produce a list, curly braces a set, the same braces with a colon a dict. All three take the same items from the same source, but not all three retain the same number of them. This lesson asks two questions: of the same thousand items, how many does each of the three comprehension forms retain, and does where a filter condition is written change the number of materialized objects?

Three Containers, One Notation

A comprehension is a loop and a container setup squeezed into a single statement. The syntax of the three forms is nearly identical, and the only difference between them is the brace and the item expression. Comprehension notation differs from a generator expression only by the brace, and that single character entirely changes setup behavior — the previous lesson measured this.

Where the three containers split is how they accept an item. A list accepts every item in order and rejects none. A set places an item by its hash value and does not take a new one if an equal one already exists. A dict does the same job on the key, but replaces the value: when the same key arrives a second time, the old value drops.

The Data Structures course built these three containers’ mechanism — the hash table, collisions, placement — and it is not repeated here. What is measured is how many items the same source comes down to in the three containers.

Two Places for a Condition

A comprehension accepts a condition in two separate places, and they are not the same thing.

A filter clause comes at the end of the notation and operates on the loop variable: a turn that does not pass the condition never runs the item expression at all. A conditional expression sits inside the item expression itself and picks a result on every turn; it filters no turn out.

Confused with each other, the result can look the same, but the number does not. The measurement separates them.

The measurement’s assumptions:

• IF28 — The shared reference’s Item class is used as is, with its counting arrangement; the oracle is the rig itself.
• IF29 — All three comprehensions run on the same thousand-item source, and the source was set up before the measurement; this is why all three “materialized” columns are zero. What is measured is retention.
• IF30 — The set and dict comprehensions group items by value % 100; the number is arbitrary and chosen only so a collision can be observed.
• IF31 — “Shared Item” is whether the objects inside the container are the source’s own objects; this column is zero for a container holding a derived number.
• IF32 — The condition measurement is done on ten items, and the condition lets half of them through; what is measured is not size, it is the production difference between the two notations.
• IF33 — In the nested measurement, the table has four rows, three columns; the total item count is 12 in all three notations. “Containers built” includes the outer container too.
• IF34 — In the scope measurement, the comprehension’s loop name is deliberately made to collide with an outer variable of the same name.
• IF35 — In the last measurement, the two notations are character-for-character identical; the only thing that changes is whether the inner source is a list or a generator expression. The outer loop runs three turns and the inner source has two items.

Measurement

"""Comprehension: three containers, same source, separate retention; nested comprehension's intermediate object."""

COUNTER = {"produced": 0}


class Item:
    """An item that counts every production of itself."""

    def __init__(self, value):
        COUNTER["produced"] += 1
        self.value = value

    def __repr__(self):
        return f"Item({self.value})"


def reset():
    COUNTER["produced"] = 0


def produced():
    return COUNTER["produced"]


N = 1000
SOURCE = [Item(i) for i in range(N)]
COMPREHENSIONS = (
    ("list comprehension", lambda: [o for o in SOURCE]),
    ("set comprehension", lambda: {o.value % 100 for o in SOURCE}),
    ("dict comprehension", lambda: {o.value % 100: o for o in SOURCE}))

print(f"{'comprehension':<20s} {'materialized':>12s} {'held in container':>18s}"
      f" {'shared Item':>12s}")
for name, build in COMPREHENSIONS:
    reset()
    container = build()
    items = list(container.values()) if isinstance(container, dict) else list(container)
    shared = sum(1 for o in items if isinstance(o, Item))
    print(f"  {name:<18s} {produced():12d} {len(container):18d} {shared:12d}")


def filter_before(n=10):
    reset()
    items = [Item(i) for i in range(n) if i % 2 == 0]
    return produced(), len(items)


def filter_after(n=10):
    reset()
    items = [o for o in (Item(i) for i in range(n)) if o.value % 2 == 0]
    return produced(), len(items)


def conditional_expression(n=10):
    reset()
    items = [Item(i) if i % 2 == 0 else Item(-i) for i in range(n)]
    return produced(), len(items)


print()
print(f"{'condition location':<36s} {'materialized':>12s} {'retained':>8s}")
for name, f in (("filtering before production", filter_before),
              ("filtering after production", filter_after),
              ("conditional expression (no filter)", conditional_expression)):
    p, r = f()
    print(f"  {name:<34s} {p:12d} {r:8d}")


def nested(m=4, k=3):
    reset()
    table = [[Item(i * k + j) for j in range(k)] for i in range(m)]
    return produced(), 1 + len(table), sum(len(row) for row in table)


def flat(m=4, k=3):
    reset()
    items = [Item(i * k + j) for i in range(m) for j in range(k)]
    return produced(), 1, len(items)


def flatten(m=4, k=3):
    reset()
    table = [[Item(i * k + j) for j in range(k)] for i in range(m)]
    items = [o for row in table for o in row]
    return produced(), 2 + len(table), len(items)


print()
print(f"{'nested notation':<28s} {'materialized':>12s} {'containers built':>16s}"
      f" {'items':>5s}")
for name, f in (("row-by-row table", nested), ("single-level comprehension", flat),
              ("flattening the table", flatten)):
    p, containers, items = f()
    print(f"  {name:<26s} {p:12d} {containers:16d} {items:5d}")

i = "outer value"
squares = [i * i for i in range(4)]
print()
print(f"outer name after comprehension: {i!r}, result {squares}")

reset()
mapping = {o.value: o for o in (Item(i % 4) for i in range(12))}
print(f"key collision: produced {produced()}, retained {len(mapping)},"
      f" last value won -> {mapping[0].value}")


def inner_source_list():
    reset()
    inner = [0, 1]
    items = [Item(i * 2 + j) for i in range(3) for j in inner]
    return produced(), len(items)


def inner_source_stream():
    reset()
    inner = (x for x in [0, 1])
    items = [Item(i * 2 + j) for i in range(3) for j in inner]
    return produced(), len(items)


print()
print(f"{'inner source type':<36s} {'materialized':>12s} {'retained':>8s}")
for name, f in (("list (re-walkable)", inner_source_list),
              ("generator expression (single-pass)", inner_source_stream)):
    p, r = f()
    print(f"  {name:<34s} {p:12d} {r:8d}")

comprehension        materialized  held in container  shared Item
  list comprehension            0               1000         1000
  set comprehension             0                100            0
  dict comprehension            0                100          100

condition location                   materialized retained
  filtering before production                   5        5
  filtering after production                   10        5
  conditional expression (no filter)           10       10

nested notation              materialized containers built items
  row-by-row table                     12                5    12
  single-level comprehension           12                1    12
  flattening the table                 12                6    12

outer name after comprehension: 'outer value', result [0, 1, 4, 9]
key collision: produced 12, retained 4, last value won -> 0

inner source type                    materialized retained
  list (re-walkable)                            6        6
  generator expression (single-pass)            2        2

What the Three Containers Retain

In the top table, “materialized” is 0 in all three rows. This is not an oddity, it is the rig itself: the thousand items were set up before the measurement, and none of the three comprehensions produces a new Item — they only take what already exists. What is measured here is not production, it is retention.

The list comprehension retains 1000 items, and 1000 of them are the source’s own objects. A new container was built, a thousand references were written into it, and no item was duplicated — the very container-building operation the first lesson measured.

The set comprehension retains 100 items and shared is 0. The two numbers have to be read together: the item expression o.value % 100 produces a number, not an Item; the container never takes the source’s objects at all. A thousand turns yield only a hundred distinct values, because a set does not accept an equal one a second time. A thousand items were read, a hundred were retained — the gap between read and retained is 900.

The dict comprehension also retains 100, but shared is 100. The key is a derived number, the value is the source’s own object. So the dict both filtered and shared: each of the hundred keys is bound to the last Item that fell to that key.

The last line shows this consequence directly. Twelve items were produced, four retained, and on colliding keys the last value won. What filters in a dict comprehension is silent: dropped items get neither a warning nor an exception. A thousand-row source turning into a hundred-row dict leaves the notation unchanged; the loss’s trace is only visible if it is counted.

Wherever the Condition Is Written

The middle table runs the same ten turns in all three rows, but production numbers split.

Filtering before production — the condition at the notation’s end, operating on the loop variable — materializes 5 objects. The five turns that fail the condition never reach the item expression at all; Item is never called.

Filtering after production retains the same five items but materializes 10. Here the condition looks at the item itself, so the item has to exist first. Five objects are born, get filtered, and become unreachable since they enter no container.

The two rows retain the same number: 5. The result is the same too. The only thing that differs is which side does the filtering. If the filter condition can be written in the source’s own language, production is avoided; if only a produced object’s property can be checked, it cannot. This is the previous lesson’s early-exit measurement, restated inside a comprehension — laziness is again a possibility, tied to where the condition can be written.

The conditional expression row shows the third behavior: 10 production, 10 retention. The if here is not a filter, it is a selector; it filters no turn out, it only picks which object gets produced. The two notations look alike and sit in different places: the selector inside the item expression, the filter at the notation’s end. The number gives the distinction — a filter reduces what is retained, a selector does not.

Nested Comprehension’s Intermediate Containers

The bottom table builds the same twelve items with three notations. “Materialized” is 12 in all three, and item count is 12 in all three; the column that splits is container count.

Row-by-row table — one comprehension inside another — builds 5 containers: four row lists and the outer list carrying them. The inner comprehension reruns on every outer turn and produces a new list each time. These lists are not intermediate objects, they are the result itself; they are needed if a table structure is wanted.

Single-level comprehension — two fors side by side, one item expression — builds the same twelve items with 1 container. Two loops run nested inside a single comprehension, and no intermediate list is ever born. The order is the same too: the leftmost for is the outer loop.

Flattening the table shows the third route: build the table first, then flatten it. The result matches the second row, but container count is 6. Five containers were built, one more added, and the four row lists built along the way serve no purpose after flattening — these are real intermediate objects. Where the same result can be built with one container, five are extra.

The distinction settles here: nested comprehension builds a structure, side-by-side for builds a walk. If the result is going to be a flat sequence, building the structure and then tearing it down costs more than never building it.

How Many Times the Inner Source Is Walked

The last table measures the most overlooked side of side-by-side for notation: the two notations are identical, the results are 6 against 2.

The difference comes from the inner source’s type. The right-hand for is re-walked on every outer turn; for three outer turns, the inner source has to be passed through from the start three times. If the inner source is a list, this is fine — a list gives a fresh iterator every time one is requested, and three turns of two items each give 6 items total.

If the inner source is a generator expression, it does not work. The stream is consumed on the first outer turn; the second and third turns want to pass through the same stream again, and the stream comes back empty. The result is 2 items, and there is neither an exception nor a warning. The single-pass behavior measured in the first lesson produces a consequence here: because an exhausted stream cannot be told apart from an empty one, the comprehension silently builds an incomplete list.

This is the practical counterpart of the distinction the three lessons have built up. If a stream is placed somewhere it will be re-walked — a nested loop’s inner source, a variable read twice — it has to be collected into a list first. If not, the second pass produces not an error but a silent shortfall.

A Comprehension’s Own Scope

The last two lines measure a detail specific to Python. A comprehension’s loop variable does not overwrite the outer variable of the same name: after the comprehension, i still carries its own value, while the comprehension produces [0, 1, 4, 9] with its own i.

The reason is that a comprehension has its own scope. The Python Fundamentals course’s scope lesson built which level a name is searched at; what is added here is that a comprehension opens one of these levels itself. The loop name is born inside the comprehension and dies there.

An ordinary for loop is not like this: there, the loop variable is written into the enclosing scope and lives on after the loop ends. Of two notations doing the same job, one leaves a name behind, the other does not. This shows a comprehension is not just a shorthand — it is a separate mechanism where name visibility is concerned.

Summary

• From the same thousand-item source, a list comprehension retains 1000, a set and a dict comprehension 100; shared Item count is 1000, 0, and 100 respectively. A container decides for itself how much it keeps, and filtering is silent.
• If a filter condition can be written before production, materialized stays at 5; if only a produced object can be checked, it becomes 10. Retained is 5 in both notations.
• A conditional expression inside the item expression is not a filter: it filters no turn out, it produces 10 and retains 10.
• A nested comprehension builds 5 containers for twelve items, two side-by-side fors give the same items with 1 container; building then flattening makes 6 containers, four of them intermediate objects.
• A comprehension has its own scope: the loop name does not overwrite an outer variable of the same name and disappears once the comprehension ends; an ordinary for loop leaves the name behind.
• In side-by-side for notation, the right-hand source is re-walked on every outer turn: a list gives 6 items, a generator expression in the same notation gives 2. Because an exhausted stream behaves silently empty, the shortfall produces no exception.

Next Step

In this lesson, both the filter condition and the item expression were written inside the comprehension; both stayed embedded where they were written and could not be reused in another comprehension. Yet both express a rule — “let this through,” “turn this into that” — and rules can be named. The next lesson treats a rule as an object: can a function be passed as an argument to another function, and if so, how many times is it called when passed? In sorting, is the key computed once per item, or freshly on every comparison?