--- title: Generators source: 'https://academia.sh/en/courses/python-data-structures/generators' course: 'Data Structures and Functional Tools' language: en updated: '2026-08-17T18:10:26+00:00' license: 'CC BY-SA 4.0' --- # Generators Calling a generator function does not run a single line of its body: on a thousand-item stream, setup materializes 1000 objects for a comprehension, 0 for a generator, and stays at 3 once three items are requested. The previous lesson measured the iterator contract and built a class writing it by hand: `Counting` needed two special methods, a bound, and a position field in five lines — all for one job, remembering where it left off. That job repeats in every iterator, and when the position is not a single number, the class grows fast. A nested walk needs multiple counters, a branched search needs a stack; each has to be carried by hand. Yet an ordinary function already carries this information. While a function runs, which line it is on and which values its local names are bound to are known; they only vanish when the function ends. This lesson's question is: if a function could **pause instead of ending**, would the position ever need to be written separately? Python's answer is the **generator** function, and the number to measure is this — calling such a function, how many of the thousand productions in its body does it do at call time? ## A Pausing Function If `yield` appears in a function body instead of `return`, that function is no longer an ordinary function. When called, its body does not run; what comes back is a **generator object**. The body only starts running once an item is requested from that object, and **stops** when it reaches the first `yield` statement. Local names, loop counters, and the call frame stay exactly as they were where it stopped. When the next request arrives, execution resumes right after the `yield`. The distinction is sharp here: `return` tears down the frame, `yield` **suspends** it. The Programming Fundamentals course introduced lazy evaluation as a paradigm choice — not computing a value until the moment it is used. What is measured here is not the name of that choice, it is which mechanism Python gives it through, and **how many objects it costs**. ## A Generator Is an Iterator A generator object signs the previous lesson's contract: `__iter__` and `__next__` are defined, `__iter__` returns itself, and `StopIteration` is thrown once the body ends. That is, a generator is the language-written form of what the `Counting` class wrote by hand — the **writing side** of the contract, without building a class. Two consequences follow directly. A generator is **single-pass**, because `__iter__` returns itself. And once exhausted it does not renew, because its body has reached its end. Both were already established as consequences of the contract in the previous lesson; here they are measured once more. The measurement's assumptions: - **IF10** — The shared reference's `Item` class and the number `N = 1000` are used as is; the counting arrangement is unchanged. - **IF11** — All three forms define the **same** job: a thousand-item stream. The only thing that changes is notation; the oracle is the rig itself, because the object counts its own production. - **IF12** — The "at setup" measurement is taken right after the line defining the form runs. The "after three items" measurement is taken after three items are requested; in the comprehension this request is `list[:3]`, in the other two it is `next`. - **IF13** — The trace log is a list placed inside the body; every line writes which point was reached. This list is the measure for "did the body run." - **IF14** — In the trace measurement, item count is three; what is measured is not the count, it is the relationship between calls and the body's progress. - **IF15** — The endless-generator measurement is deliberate: setup producing zero does not come from item count being finite. - **IF16** — In the joining measurement, the two wings have **400** and **600** items; the total is still **1000**, and all three forms give the same item sequence. - **IF17** — The "up to first item" column is the production count after only one item is taken from the join; this column is what separates the forms. - **IF18** — In the closing measurement, the stream has a thousand items but is released with `close` after three are taken; the trace line placed in the `finally` block also writes the production count at that moment, so when the closing runs can be measured. ## Measurement ```python """Generator: when does the body run, how many objects does setup materialize.""" COUNTER = {"produced": 0} TRACE = [] class Item: """An item that counts every production of itself.""" def __init__(self, value): COUNTER["produced"] += 1 self.value = value def __repr__(self): return f"Item({self.value})" def reset(): COUNTER["produced"] = 0 def produced(): return COUNTER["produced"] N = 1000 def generator_function(n=N): for i in range(n): yield Item(i) def comprehension(n=N): reset() items = [Item(i) for i in range(n)] at_setup = produced() first_three = items[:3] return at_setup, produced(), len(first_three) def generator_expression(n=N): reset() stream = (Item(i) for i in range(n)) at_setup = produced() first_three = [next(stream) for _ in range(3)] return at_setup, produced(), len(first_three) def generator_call(n=N): reset() stream = generator_function(n) at_setup = produced() first_three = [next(stream) for _ in range(3)] return at_setup, produced(), len(first_three) print(f"{'form':<24s} {'at setup':>10s} {'after three items':>18s} {'got':>5s}") for name, f in (("comprehension", comprehension), ("generator expression", generator_expression), ("generator function call", generator_call)): a, b, c = f() print(f" {name:<22s} {a:10d} {b:18d} {c:5d}") def traced(n=3): TRACE.append("body started") for i in range(n): TRACE.append(f"before yield {i}") yield Item(i) TRACE.append(f"after yield {i}") TRACE.append("body ended") return "done" TRACE.clear() g = traced() print() print(f"trace after the call: {TRACE}") next(g) print(f"after first next: {TRACE}") next(g) print(f"after second next: {TRACE}") print(f"is the generator an iterator: iter(g) is g -> {iter(g) is g}") TRACE.clear() one = traced(1) next(one) try: next(one) except StopIteration as e: print(f"when the body ends: {TRACE[-1]!r}, value carried by the exception {e.value!r}") single = generator_function(3) print() print(f"generator function is single-pass: {len(list(single))} and {len(list(single))} items") def endless(): i = 0 while True: yield Item(i) i += 1 reset() stream = endless() setup = produced() [next(stream) for _ in range(3)] print(f"endless generator: at setup {setup}, once three items are requested {produced()}") def branch(a, b): yield from generator_function(a) yield from generator_function(b) def manual_branch(a, b): for o in generator_function(a): yield o for o in generator_function(b): yield o def intermediate_list(a, b): return list(generator_function(a)) + list(generator_function(b)) print() print(f"{'joining form':<22s} {'at setup':>10s} {'up to first item':>16s}" f" {'once fully walked':>17s}") for name, f in (("yield from", branch), ("nested for + yield", manual_branch), ("appending two lists", intermediate_list)): reset() n = f(400, 600) at_setup = produced() y = iter(n) next(y) first = produced() list(y) print(f" {name:<20s} {at_setup:10d} {first:16d} {produced():17d}") def accumulator(): total = 0 try: while True: incoming = yield total total += incoming finally: TRACE.append("accumulator closed") def source(n=N): try: for i in range(n): yield Item(i) finally: TRACE.append(f"source closed, produced by then {produced()}") TRACE.clear() acc = accumulator() first = next(acc) print() print(f"yield is an expression: first value {first}, send(5) -> {acc.send(5)}," f" send(7) -> {acc.send(7)}") acc.close() print(f"trace after close call: {TRACE}") TRACE.clear() reset() stream = source() three = [next(stream) for _ in range(3)] stream.close() print(f"three items taken from a thousand-item stream, then left: got {len(three)}," f" produced {produced()}") print(f"closing trace: {TRACE}") ``` ``` form at setup after three items got comprehension 1000 1000 3 generator expression 0 3 3 generator function call 0 3 3 trace after the call: [] after first next: ['body started', 'before yield 0'] after second next: ['body started', 'before yield 0', 'after yield 0', 'before yield 1'] is the generator an iterator: iter(g) is g -> True when the body ends: 'body ended', value carried by the exception 'done' generator function is single-pass: 3 and 0 items endless generator: at setup 0, once three items are requested 3 joining form at setup up to first item once fully walked yield from 0 1 1000 nested for + yield 0 1 1000 appending two lists 1000 1000 1000 yield is an expression: first value 0, send(5) -> 5, send(7) -> 12 trace after close call: ['accumulator closed'] three items taken from a thousand-item stream, then left: got 3, produced 3 closing trace: ['source closed, produced by then 3'] ``` ## The Cost of Setup The top table is the first half of the shared reference's second claim, and all three rows define the same job: a thousand-item stream. The **comprehension** materializes **1000** objects on the very line where it is written. Once three items are requested the number does not change — because all of them had already been produced; `list[:3]` only shows three of what already exists. The **generator expression** and **generator function call** materialize **0** at setup. Once three items are requested it rises to **3**. The remaining 997 objects are never born, because nobody requested them. The gap between the two numbers is invisible in the notation. All three rows are a single statement, all three name a thousand-item stream, and all three can be fed into a `for` loop afterward. The only thing that says whether setup is **1000** or **0** is the difference between square brackets and parentheses, and the `yield` keyword. The limit of the laziness here has to be read correctly. **0** does not mean the work was not done; it means it was **not done yet**. Once three items are requested, three objects are born; if a thousand were requested, a thousand would be born. The number this lesson measures only answers **when**; the answer to **how many in total** is the subject of the next lesson. The generator expression row's notation — parentheses instead of square brackets — is not this lesson's subject; the next lesson measures it on its own. It sits in this table only to give the setup count. ## Where the Body Stops The middle block shows the body's progress through the trace log, and the first line is the most surprising: after the generator function is called, the trace is **empty**. Not a single line of the body has run. The call did not start a job, it **built an object**. After the first `next` call the trace has two lines: the body started and reached the first `yield`. It stopped there. After the second call two more lines were added — execution resumed **right after** the `yield`, moved to the loop's next turn, and stopped again at the second `yield`. The local name `i` was preserved between the two calls; nobody wrote it anywhere, the frame stayed suspended exactly as it was. This is the exact counterpart of what the `Counting` class did by hand. There, position was an attribute, advanced by hand on every call; here, position **is the body itself**, and the language preserves it. The `iter(g) is g` line confirms the other end of the contract: a generator is its own iterator. Single-pass behavior comes from the same place — `3` then `0`. ## The Value Carried by the Ending When the body reaches its end, the trace writes its last line and `StopIteration` is born. What is interesting is that the exception is **not empty**: the value of the `return "done"` statement is carried on top of the exception. This shows how `return` and `yield` split roles in a generator. `yield` gives the stream's items; `return` gives the stream's **result**, and is not part of the stream. A `for` loop walking a generator never sees this value, because the loop catches `StopIteration` and discards it. Whoever wants the value has to catch the exception themselves. The endless-generator line shows laziness does not depend on finiteness: a generator whose body is a loop that never ends still materializes **0** at setup and stays at **3** once three items are requested. Writing such a stream with a comprehension is impossible — a comprehension tries to exhaust an endless source and never leaves its first line. Laziness here is not an optimization, it is a condition for **expressibility**. ## Delegation The last table measures three ways of joining two streams, and all three give the same **1000** items. `yield from` assigns a generator the job of delivering another generator's items: the outer generator suspends its own body and hands requests off to the inner one. At setup **0**, once the first item is taken **1**. The hand-written form with a `for` loop gives the same numbers; `yield from` is that loop's shorthand, and it also carries through the inner generator's ending value. The **appending two lists** row stands apart. At setup **1000** — because the `list()` calls exhaust both wings all the way through. A thousand objects are born and two intermediate lists are built before the first item is even reached; the `+` operator then produces a third list on top. The result is the same, the order is the same, the items are the same; the only thing that differs is **how many productions** the first item comes after. The measurement also says where a chain breaks: the moment a single `list()` call enters a lazy stream, laziness ends at that point in the chain. Even if the links after it are written lazily, they cannot undo the thousand productions that already happened before them. ## Two Directions of the Channel, and Closing The last four lines show `yield` is not a one-way exit. `yield` is not a statement but an **expression**, and it has a value: when a suspended generator is woken with `send`, the waiting `yield` expression returns the sent value and the body continues from there. The accumulator in the measurement gives **5** with `send(5)`, **12** with `send(7)`; the running total sits in the body's local name between the two calls. The first `next` call is mandatory in this arrangement: if the body has not yet reached its first `yield`, there is no expression to receive a sent value. This is why every generator fed with `send` is first advanced once; the **0** in the measurement is the starting value that first `yield` gives. The last two lines measure the most overlooked side of lazy streams. Taking three items from a thousand-item stream and then leaving it, produced stays at **3** — expected so far. What is unexpected is that the `finally` block **runs**: the `close` call wakes the suspended body at the point it stopped, raises an exception there, and runs the `finally` blocks. The trace line proves this, and it also writes the production count at that moment. The practical counterpart: if a generator is operating on an open resource, cutting the stream short does not leave that resource hanging. The Programming Fundamentals course's context managers lesson established that release happens even under an exception; the `try`/`finally` in a generator's body gives the same guarantee for a **suspended** frame. Laziness defers the work, not the responsibility. ## Summary - Calling a function containing `yield` runs not a single line of its body; what comes back is a generator object. `return` tears down the frame, `yield` suspends it. - On a thousand-item stream, a comprehension materializes **1000** objects at setup; a generator expression and a generator function call materialize **0**; once three items are requested, both stay at **3**. The notations are single statements and look alike; the numbers do not. - A generator signs the previous lesson's contract: `iter(g) is g` is true, the stream is single-pass, and a second pass gives **0** items. The language does what a hand-written iterator class did. - The body stops at every `yield` and resumes from that point on the next request; local names are preserved in between. Once the end is reached, the `return` value is carried on `StopIteration`, and a `for` loop walking it never sees this value. - `yield from` delegates without building an intermediate container: **1** production up to the first item. Doing the same job by appending two lists takes **1000** productions before the first item and needs three containers. - `yield` is an expression and returns the value sent with `send`; when a thousand-item stream is left with `close` after three items, production stays at **3**, but the body's `finally` block still runs. ## Next Step This lesson only measured the moment of setup, and there was a gap there: **1000** against **0**. But a stream is consumed sooner or later — if a thousand items are really going to be collected, the generator has to produce those thousand objects too. The next lesson asks exactly this question: for two notations that carry the same job to its end, how many objects has each produced by the end of the job, and how many does each still hold on to? The first half of the answer will be expected, the second half will not — and only there does it become visible what laziness actually reduces.