--- title: 'Generator Expressions' source: 'https://academia.sh/en/courses/python-data-structures/generator-expressions' course: 'Data Structures and Functional Tools' language: en updated: '2026-08-17T18:10:26+00:00' license: 'CC BY-SA 4.0' --- # Generator Expressions Summing a thousand items, a generator and a comprehension both produce 1000 objects and both give 499500; the only place they split is what stays held in a list — 0 against 1000. Laziness defers production, it does not avoid it. The previous lesson measured only the moment of setup, and there was a gap there: a comprehension materializes **1000** objects on its first line, a generator materializes **0**. But a stream is not written to be set up and left; it is consumed sooner or later. If a thousand items are really going to be collected, the generator has to produce those thousand objects too. This lesson's question: for two notations that carry the same job to its end, how many objects has each produced once done, and how many does each still hold? Half the answer will be expected, the other half will not — and only there does it become visible what laziness actually reduces. The lesson also builds the generator expression's own notation, measured here on its own. ## Notation A generator expression is a list comprehension with its square brackets replaced by parentheses, and what it produces is not a list but a **generator object**. Given as the sole argument to a function, the parentheses are not written separately; the expression in `sum(o.value for o in stream)` is itself a generator expression on its own. The object it produces is the same kind the previous lesson's generator function returned: an iterator, single-pass, `__iter__` returning itself. The difference is only in notation. A generator function takes a name, can carry multiple statements, can branch; a generator expression is nameless and states a single transformation. The choice between them is not one of cost but of expression width — the measured numbers are the same in both. The Programming Fundamentals course introduced lazy evaluation as a concept at the entrance to functional programming: a value is not computed until it is used. This lesson does not repeat that concept; it counts **how many objects** its Python counterpart costs — and the number turns out different from what the name suggests. The measurement's assumptions: - **IF19** — The shared reference's `Item` class and the number `N = 1000` are used as is; the counting arrangement is unchanged. The oracle is the rig itself. - **IF20** — The two consumption forms target the **same** result: the value sum of a thousand items. The result column is the measurement's correctness check; if the two forms gave different results, the comparison would be invalid. - **IF21** — "Held in list" is the number of items still alive in a container once the sum is done; it is zero in the generator form because no such container is built. - **IF22** — The chain measurement has three links: produce, filter, transform. The two forms apply the same links in the same order; the only thing that changes is whether each link is a container or a stream. - **IF23** — The chain's "held in containers" count is the sum of the three lists' lengths. **The memory claim is built on the object count, not measured bytes; duration is not measured.** - **IF24** — In the early-exit measurement, the sought condition is satisfied at the fifth item; the condition's position is fixed and decides when the consumer stops. - **IF25** — The single-pass measurement is done on a small ten-item stream; what is measured is not size, it is the value the second summation returns. - **IF26** — In the source-evaluation measurement, `give_source` counts its own call count; this count is the measure of when the outer iterable is evaluated. - **IF27** — In the last table, every operation is tried separately on two containers, and setup is reset before each trial; the exception class of a failing operation is also taken from the run. Only the standard library is used. ## Measurement ```python """Generator expression: same result, same production, different retention.""" import itertools COUNTER = {"produced": 0} class Item: """An item that counts every production of itself.""" def __init__(self, value): COUNTER["produced"] += 1 self.value = value def __repr__(self): return f"Item({self.value})" def reset(): COUNTER["produced"] = 0 def produced(): return COUNTER["produced"] N = 1000 def generator_consumption(n=N): reset() stream = (Item(i) for i in range(n)) total = sum(o.value for o in stream) return produced(), total, 0 def comprehension_consumption(n=N): reset() items = [Item(i) for i in range(n)] total = sum(o.value for o in items) return produced(), total, len(items) print(f"{'consumption form':<27s} {'produced':>9s} {'total':>8s}" f" {'held in list':>13s}") for name, f in (("summing via generator", generator_consumption), ("summing via comprehension", comprehension_consumption)): p, t, held = f() print(f" {name:<25s} {p:9d} {t:8d} {held:13d}") def lazy_chain(n=N): reset() evens = (o for o in (Item(i) for i in range(n)) if o.value % 2 == 0) squares = (o.value * o.value for o in evens) at_setup = produced() total = sum(squares) return at_setup, produced(), total, 0 def list_chain(n=N): reset() all_items = [Item(i) for i in range(n)] evens = [o for o in all_items if o.value % 2 == 0] squares = [o.value * o.value for o in evens] at_setup = produced() total = sum(squares) return at_setup, produced(), total, len(all_items) + len(evens) + len(squares) print() print(f"{'chain form':<29s} {'at setup':>10s} {'produced':>9s}" f" {'total':>10s} {'held in containers':>19s}") for name, f in (("three generator expressions", lazy_chain), ("three list comprehensions", list_chain)): s, p, t, held = f() print(f" {name:<27s} {s:10d} {p:9d} {t:10d} {held:19d}") def early_exit(n=N): reset() stream = (Item(i) for i in range(n)) found = any(o.value > 4 for o in stream) return produced(), found def early_exit_list(n=N): reset() items = [Item(i) for i in range(n)] found = any(o.value > 4 for o in items) return produced(), found print() p1, f1 = early_exit() p2, f2 = early_exit_list() print(f"when the consumer stops at the fifth item: generator expression produced {p1} objects" f" (answer {f1}), comprehension {p2} (answer {f2})") reset() stream = (Item(i) for i in range(10)) first = sum(o.value for o in stream) second = sum(o.value for o in stream) print(f"same generator expression summed twice: {first} and {second}," f" produced {produced()}") CALLS = {"source": 0} def give_source(): CALLS["source"] += 1 return [1, 2, 3] stream = (x for x in give_source()) at_setup = CALLS["source"] list(stream) print(f"was the outer iterable evaluated at setup: {at_setup} calls," f" after walking {CALLS['source']}") remaining = [5, 6, 7] stream = (x * 10 for x in remaining) remaining.append(8) print(f"did an item added to the source afterward enter the stream: {list(stream)}") reset() a_set = {o.value % 3 for o in (Item(i) for i in range(9))} print(f"generator expression given to a set constructor: produced {produced()}," f" held {len(a_set)}") OPERATIONS = (("len(n)", lambda n: len(n)), ("n[2]", lambda n: n[2].value), ("n[:3]", lambda n: len(n[:3])), ("islice(n, 3)", lambda n: len(list(itertools.islice(n, 3))))) print() print(f"{'operation':<16s} {'list':>25s} {'generator expression':>28s}") for name, operation in OPERATIONS: row = [] for build in (lambda: [Item(i) for i in range(N)], lambda: (Item(i) for i in range(N))): reset() n = build() try: result = f"{operation(n)} ({produced()} productions)" except TypeError as e: result = f"{type(e).__name__} ({produced()} productions)" row.append(result) print(f" {name:<14s} {row[0]:>25s} {row[1]:>28s}") reset() stream = (Item(i) for i in range(N)) exists = any(o.value == 2 for o in stream) at_test = produced() left = sum(1 for _ in stream) print() print(f"membership test on a stream: answer {exists}, produced during test {at_test}," f" remaining in stream {left}") ``` ``` consumption form produced total held in list summing via generator 1000 499500 0 summing via comprehension 1000 499500 1000 chain form at setup produced total held in containers three generator expressions 0 1000 166167000 0 three list comprehensions 1000 1000 166167000 2000 when the consumer stops at the fifth item: generator expression produced 6 objects (answer True), comprehension 1000 (answer True) same generator expression summed twice: 45 and 0, produced 10 was the outer iterable evaluated at setup: 1 calls, after walking 1 did an item added to the source afterward enter the stream: [50, 60, 70, 80] generator expression given to a set constructor: produced 9, held 3 operation list generator expression len(n) 1000 (1000 productions) TypeError (0 productions) n[2] 2 (1000 productions) TypeError (0 productions) n[:3] 3 (1000 productions) TypeError (0 productions) islice(n, 3) 3 (1000 productions) 3 (3 productions) membership test on a stream: answer True, produced during test 3, remaining in stream 997 ``` ## Same Production, Separate Retention The top table completes the shared reference's second claim, and the first column breaks the expectation. **Produced is 1000 in both.** The lazy form does not produce fewer objects; a thousand items were requested, a thousand were born. The total column matches too: both **499500**. The two notations do the same job, give the same result, pay the same cost. The only column where they split is the last. Once summing is done, **1000** items still sit in a list in the comprehension form; **0** in the generator form. Each of the thousand produced objects was born, contributed to the sum, and became unreachable because nothing wrote it into a container. In the comprehension all of them stayed in the list, because that is exactly the list's job. From this comes the shared reference's sentence: **laziness does not reduce production, it reduces retention.** This is narrower than the name "lazy evaluation" suggests. The name implies work will not be done; the measurement shows it is done, and only **when** and **what is left afterward** change. **Laziness defers production, it does not avoid it.** The previous lesson's **0** has to be read together with this table. Zero at setup is not a promise of savings, it is a record of **deferral**. Whether the promise is kept depends on how the stream is consumed, not on how it is written. ## The Chain's Intermediate Containers The middle table measures not a single transformation but a three-link chain: produce, filter, transform. The total is the same again — **166167000** — and produced is again **1000**. Two columns split. The setup column repeats the previous lesson's result: the lazy chain **0**, the list chain **1000**. But the real difference is in the last column. Once the list chain finishes, **2000** references sit in containers: a thousand-item source list, a five-hundred-item filtered list, a five-hundred-item transformed list. In the lazy chain this number is **0**. Intermediate-container count grows with chain length. A four-link list chain builds four containers, a five-link one five; each link wants to see all of what came before and stores what it saw. In the lazy chain, container count does not grow with link count, because no link ever wants all of what came before — each wants **one** item at a time. Together with the previous lesson's last row, this gives the chain's breaking point: the moment a single `list()` call enters the chain, a container is built at that point, and the rest of the chain's laziness cannot undo it. ## Early Exit's Source The next line measures the most natural objection to the claim. When the consumer stops at the fifth item, the generator expression produces **6** objects, the comprehension **1000**. Here laziness really does look like it avoided production. The distinction: what enables the avoidance is not laziness, it is the **consumer stopping early**. `any` stops asking once it finds what it is looking for; the lazy stream produces nothing that was not asked for. In the comprehension, even if the consumer stops early, production already finished — the list was filled before `any` asked its first question. Together, the two cases give the claim's full shape: laziness **ties production to the consumer**. Wanting everything produces everything (first table: 1000); exiting early leaves what was not needed unproduced (this line: 6). In the comprehension, production is independent of the consumer and is **1000** either way. What laziness gives is not avoidance, it is the **possibility** of avoidance. Why the sixth object gets produced is part of the same pattern: the condition is satisfied at the fifth item, but `any` can only know this **after** taking it, and taking it costs one more production. ## When the Source Is Evaluated The last four lines measure two subtle sides of a generator expression. The first is single-pass behavior. Summed twice, the same generator expression gives **45** and **0**; produced stays at **10**. The second sum runs on an empty stream, and `sum` returns zero on an empty stream — no error, a silent zero. This is single-pass behavior's most treacherous consequence: an exhausted stream cannot be told apart from an empty one. The second is when the source is read. `give_source` is called **once** at setup, and stays at **1** after the stream is walked. The outermost iterable in a generator expression is evaluated **immediately**; what is deferred is pulling items, not finding the source. A wrong source name or an erroring call shows up at setup. The next line shows this rule's limit. The source list was evaluated at setup, but an item added afterward **entered** the stream: the result is `[50, 60, 70, 80]`. What got evaluated was the list **object**, not a snapshot of its content — the same distinction as the previous topic's view measurement: a binding was established, no copy taken. If a lazy stream's source changes before consumption, the stream sees the changed state. The last line shows where a generator expression can be given: a stream given to a set constructor makes it produce **9** objects, and **3** items are retained afterward. The gap between materialized and retained is visible here too — the container decides for itself how much it keeps. ## What a Stream Cannot Do The last table measures what is given up in exchange for not retaining, and the same exception sits in three of its rows. `len(n)` gives **1000** on a list, `TypeError` on a stream. A stream does not know its length, because knowing it would require going all the way to the end and remembering that it did. `n[2]` and `n[:3]` give the same answer: a stream cannot be indexed or sliced. Everything the previous topic's slicing lesson measured — a slice building a new container, sharing inner items — can never even be asked of a stream, because a slice requires looking backward, and a stream has nowhere to look back to. The shared reason: **every one of these questions requires retention.** Length, index, and slice all assume items sit together; a stream answers none of the three because it refuses to do exactly that. Zero retention costs three built-in operations. The fourth row shows where the loss can be made up. The standard library's `islice` tool takes the first three items from a stream and causes only **3** productions; the same result on a list comes only after **1000**. A job that looks like a slice, defined without looking backward, becomes possible on a stream too — but the result is not a slice, it is another stream, and it too is single-pass. The last row is single-pass behavior's most attention-demanding consequence. The membership test on a stream answers **True**, only **3** objects produced during the test; but **997** items remain in the stream after. The test **consumed** the stream: the three items asked about do not come back. A second test on the same stream would search from the third item onward, and the first three would never be seen. A list has no such side effect; asking on a stream is reading. ## Summary - Summing a thousand items, a generator and a comprehension both produce **1000** objects and both give **499500**; the only place they split is what stays held — **0** against **1000**. Laziness defers production, it does not avoid it. - In a three-link chain, produced is still **1000**, but the list form holds **2000** references in containers at the end, the lazy form **0**. Intermediate container count grows with chain length in a list, not in a stream. - When the consumer stops at the fifth item, a generator expression produces **6**, a comprehension **1000**. What enables the avoidance is not laziness but the consumer stopping; laziness only makes it **possible**. - A generator expression is single-pass and behaves silently empty once exhausted: a second sum gives not an error but **0**. - The outermost iterable is evaluated at setup (**1** call), but what gets evaluated is the object itself: an item added to the source afterward enters the stream. ## Next Step These two lessons measured the difference between square brackets and parentheses, but never opened up the square brackets themselves. Yet the same notation can build three separate containers — list, set, and dict — and all three take the same items from the same source and hold a **different** number of them. What is more, the notation can nest, and one comprehension inside another produces an unbuilt container. The next lesson counts these: of the same thousand items, how many does each of the three comprehension forms retain, and does where a filter clause is written change the number of objects produced?