Generators

The previous lesson measured the iterator contract and built a class writing it by hand: Counting needed two special methods, a bound, and a position field in five lines — all for one job, remembering where it left off. That job repeats in every iterator, and when the position is not a single number, the class grows fast. A nested walk needs multiple counters, a branched search needs a stack; each has to be carried by hand.

Yet an ordinary function already carries this information. While a function runs, which line it is on and which values its local names are bound to are known; they only vanish when the function ends. This lesson’s question is: if a function could pause instead of ending, would the position ever need to be written separately? Python’s answer is the generator function, and the number to measure is this — calling such a function, how many of the thousand productions in its body does it do at call time?

A Pausing Function

If yield appears in a function body instead of return, that function is no longer an ordinary function. When called, its body does not run; what comes back is a generator object. The body only starts running once an item is requested from that object, and stops when it reaches the first yield statement. Local names, loop counters, and the call frame stay exactly as they were where it stopped. When the next request arrives, execution resumes right after the yield.

The distinction is sharp here: return tears down the frame, yield suspends it. The Programming Fundamentals course introduced lazy evaluation as a paradigm choice — not computing a value until the moment it is used. What is measured here is not the name of that choice, it is which mechanism Python gives it through, and how many objects it costs.

A Generator Is an Iterator

A generator object signs the previous lesson’s contract: __iter__ and __next__ are defined, __iter__ returns itself, and StopIteration is thrown once the body ends. That is, a generator is the language-written form of what the Counting class wrote by hand — the writing side of the contract, without building a class.

Two consequences follow directly. A generator is single-pass, because __iter__ returns itself. And once exhausted it does not renew, because its body has reached its end. Both were already established as consequences of the contract in the previous lesson; here they are measured once more.

The measurement’s assumptions:

• IF10 — The shared reference’s Item class and the number N = 1000 are used as is; the counting arrangement is unchanged.
• IF11 — All three forms define the same job: a thousand-item stream. The only thing that changes is notation; the oracle is the rig itself, because the object counts its own production.
• IF12 — The “at setup” measurement is taken right after the line defining the form runs. The “after three items” measurement is taken after three items are requested; in the comprehension this request is list[:3], in the other two it is next.
• IF13 — The trace log is a list placed inside the body; every line writes which point was reached. This list is the measure for “did the body run.”
• IF14 — In the trace measurement, item count is three; what is measured is not the count, it is the relationship between calls and the body’s progress.
• IF15 — The endless-generator measurement is deliberate: setup producing zero does not come from item count being finite.
• IF16 — In the joining measurement, the two wings have 400 and 600 items; the total is still 1000, and all three forms give the same item sequence.
• IF17 — The “up to first item” column is the production count after only one item is taken from the join; this column is what separates the forms.
• IF18 — In the closing measurement, the stream has a thousand items but is released with close after three are taken; the trace line placed in the finally block also writes the production count at that moment, so when the closing runs can be measured.

Measurement

"""Generator: when does the body run, how many objects does setup materialize."""

COUNTER = {"produced": 0}
TRACE = []


class Item:
    """An item that counts every production of itself."""

    def __init__(self, value):
        COUNTER["produced"] += 1
        self.value = value

    def __repr__(self):
        return f"Item({self.value})"


def reset():
    COUNTER["produced"] = 0


def produced():
    return COUNTER["produced"]


N = 1000


def generator_function(n=N):
    for i in range(n):
        yield Item(i)


def comprehension(n=N):
    reset()
    items = [Item(i) for i in range(n)]
    at_setup = produced()
    first_three = items[:3]
    return at_setup, produced(), len(first_three)


def generator_expression(n=N):
    reset()
    stream = (Item(i) for i in range(n))
    at_setup = produced()
    first_three = [next(stream) for _ in range(3)]
    return at_setup, produced(), len(first_three)


def generator_call(n=N):
    reset()
    stream = generator_function(n)
    at_setup = produced()
    first_three = [next(stream) for _ in range(3)]
    return at_setup, produced(), len(first_three)


print(f"{'form':<24s} {'at setup':>10s} {'after three items':>18s} {'got':>5s}")
for name, f in (("comprehension", comprehension), ("generator expression", generator_expression),
              ("generator function call", generator_call)):
    a, b, c = f()
    print(f"  {name:<22s} {a:10d} {b:18d} {c:5d}")


def traced(n=3):
    TRACE.append("body started")
    for i in range(n):
        TRACE.append(f"before yield {i}")
        yield Item(i)
        TRACE.append(f"after yield {i}")
    TRACE.append("body ended")
    return "done"


TRACE.clear()
g = traced()
print()
print(f"trace after the call: {TRACE}")
next(g)
print(f"after first next: {TRACE}")
next(g)
print(f"after second next: {TRACE}")
print(f"is the generator an iterator: iter(g) is g -> {iter(g) is g}")

TRACE.clear()
one = traced(1)
next(one)
try:
    next(one)
except StopIteration as e:
    print(f"when the body ends: {TRACE[-1]!r}, value carried by the exception {e.value!r}")

single = generator_function(3)
print()
print(f"generator function is single-pass: {len(list(single))} and {len(list(single))} items")


def endless():
    i = 0
    while True:
        yield Item(i)
        i += 1


reset()
stream = endless()
setup = produced()
[next(stream) for _ in range(3)]
print(f"endless generator: at setup {setup}, once three items are requested {produced()}")


def branch(a, b):
    yield from generator_function(a)
    yield from generator_function(b)


def manual_branch(a, b):
    for o in generator_function(a):
        yield o
    for o in generator_function(b):
        yield o


def intermediate_list(a, b):
    return list(generator_function(a)) + list(generator_function(b))


print()
print(f"{'joining form':<22s} {'at setup':>10s} {'up to first item':>16s}"
      f" {'once fully walked':>17s}")
for name, f in (("yield from", branch), ("nested for + yield", manual_branch),
              ("appending two lists", intermediate_list)):
    reset()
    n = f(400, 600)
    at_setup = produced()
    y = iter(n)
    next(y)
    first = produced()
    list(y)
    print(f"  {name:<20s} {at_setup:10d} {first:16d} {produced():17d}")


def accumulator():
    total = 0
    try:
        while True:
            incoming = yield total
            total += incoming
    finally:
        TRACE.append("accumulator closed")


def source(n=N):
    try:
        for i in range(n):
            yield Item(i)
    finally:
        TRACE.append(f"source closed, produced by then {produced()}")


TRACE.clear()
acc = accumulator()
first = next(acc)
print()
print(f"yield is an expression: first value {first}, send(5) -> {acc.send(5)},"
      f" send(7) -> {acc.send(7)}")
acc.close()
print(f"trace after close call: {TRACE}")

TRACE.clear()
reset()
stream = source()
three = [next(stream) for _ in range(3)]
stream.close()
print(f"three items taken from a thousand-item stream, then left: got {len(three)},"
      f" produced {produced()}")
print(f"closing trace: {TRACE}")

form                       at setup  after three items   got
  comprehension                1000               1000     3
  generator expression            0                  3     3
  generator function call          0                  3     3

trace after the call: []
after first next: ['body started', 'before yield 0']
after second next: ['body started', 'before yield 0', 'after yield 0', 'before yield 1']
is the generator an iterator: iter(g) is g -> True
when the body ends: 'body ended', value carried by the exception 'done'

generator function is single-pass: 3 and 0 items
endless generator: at setup 0, once three items are requested 3

joining form             at setup up to first item once fully walked
  yield from                    0                1              1000
  nested for + yield            0                1              1000
  appending two lists        1000             1000              1000

yield is an expression: first value 0, send(5) -> 5, send(7) -> 12
trace after close call: ['accumulator closed']
three items taken from a thousand-item stream, then left: got 3, produced 3
closing trace: ['source closed, produced by then 3']

The Cost of Setup

The top table is the first half of the shared reference’s second claim, and all three rows define the same job: a thousand-item stream.

The comprehension materializes 1000 objects on the very line where it is written. Once three items are requested the number does not change — because all of them had already been produced; list[:3] only shows three of what already exists.

The generator expression and generator function call materialize 0 at setup. Once three items are requested it rises to 3. The remaining 997 objects are never born, because nobody requested them.

The gap between the two numbers is invisible in the notation. All three rows are a single statement, all three name a thousand-item stream, and all three can be fed into a for loop afterward. The only thing that says whether setup is 1000 or 0 is the difference between square brackets and parentheses, and the yield keyword.

The limit of the laziness here has to be read correctly. 0 does not mean the work was not done; it means it was not done yet. Once three items are requested, three objects are born; if a thousand were requested, a thousand would be born. The number this lesson measures only answers when; the answer to how many in total is the subject of the next lesson.

The generator expression row’s notation — parentheses instead of square brackets — is not this lesson’s subject; the next lesson measures it on its own. It sits in this table only to give the setup count.

Where the Body Stops

The middle block shows the body’s progress through the trace log, and the first line is the most surprising: after the generator function is called, the trace is empty. Not a single line of the body has run. The call did not start a job, it built an object.

After the first next call the trace has two lines: the body started and reached the first yield. It stopped there. After the second call two more lines were added — execution resumed right after the yield, moved to the loop’s next turn, and stopped again at the second yield. The local name i was preserved between the two calls; nobody wrote it anywhere, the frame stayed suspended exactly as it was.

This is the exact counterpart of what the Counting class did by hand. There, position was an attribute, advanced by hand on every call; here, position is the body itself, and the language preserves it. The iter(g) is g line confirms the other end of the contract: a generator is its own iterator. Single-pass behavior comes from the same place — 3 then 0.

The Value Carried by the Ending

When the body reaches its end, the trace writes its last line and StopIteration is born. What is interesting is that the exception is not empty: the value of the return "done" statement is carried on top of the exception.

This shows how return and yield split roles in a generator. yield gives the stream’s items; return gives the stream’s result, and is not part of the stream. A for loop walking a generator never sees this value, because the loop catches StopIteration and discards it. Whoever wants the value has to catch the exception themselves.

The endless-generator line shows laziness does not depend on finiteness: a generator whose body is a loop that never ends still materializes 0 at setup and stays at 3 once three items are requested. Writing such a stream with a comprehension is impossible — a comprehension tries to exhaust an endless source and never leaves its first line. Laziness here is not an optimization, it is a condition for expressibility.

Delegation

The last table measures three ways of joining two streams, and all three give the same 1000 items.

yield from assigns a generator the job of delivering another generator’s items: the outer generator suspends its own body and hands requests off to the inner one. At setup 0, once the first item is taken 1. The hand-written form with a for loop gives the same numbers; yield from is that loop’s shorthand, and it also carries through the inner generator’s ending value.

The appending two lists row stands apart. At setup 1000 — because the list() calls exhaust both wings all the way through. A thousand objects are born and two intermediate lists are built before the first item is even reached; the + operator then produces a third list on top. The result is the same, the order is the same, the items are the same; the only thing that differs is how many productions the first item comes after.

The measurement also says where a chain breaks: the moment a single list() call enters a lazy stream, laziness ends at that point in the chain. Even if the links after it are written lazily, they cannot undo the thousand productions that already happened before them.

Two Directions of the Channel, and Closing

The last four lines show yield is not a one-way exit. yield is not a statement but an expression, and it has a value: when a suspended generator is woken with send, the waiting yield expression returns the sent value and the body continues from there. The accumulator in the measurement gives 5 with send(5), 12 with send(7); the running total sits in the body’s local name between the two calls.

The first next call is mandatory in this arrangement: if the body has not yet reached its first yield, there is no expression to receive a sent value. This is why every generator fed with send is first advanced once; the 0 in the measurement is the starting value that first yield gives.

The last two lines measure the most overlooked side of lazy streams. Taking three items from a thousand-item stream and then leaving it, produced stays at 3 — expected so far. What is unexpected is that the finally block runs: the close call wakes the suspended body at the point it stopped, raises an exception there, and runs the finally blocks. The trace line proves this, and it also writes the production count at that moment.

The practical counterpart: if a generator is operating on an open resource, cutting the stream short does not leave that resource hanging. The Programming Fundamentals course’s context managers lesson established that release happens even under an exception; the try/finally in a generator’s body gives the same guarantee for a suspended frame. Laziness defers the work, not the responsibility.

Summary

• Calling a function containing yield runs not a single line of its body; what comes back is a generator object. return tears down the frame, yield suspends it.
• On a thousand-item stream, a comprehension materializes 1000 objects at setup; a generator expression and a generator function call materialize 0; once three items are requested, both stay at 3. The notations are single statements and look alike; the numbers do not.
• A generator signs the previous lesson’s contract: iter(g) is g is true, the stream is single-pass, and a second pass gives 0 items. The language does what a hand-written iterator class did.
• The body stops at every yield and resumes from that point on the next request; local names are preserved in between. Once the end is reached, the return value is carried on StopIteration, and a for loop walking it never sees this value.
• yield from delegates without building an intermediate container: 1 production up to the first item. Doing the same job by appending two lists takes 1000 productions before the first item and needs three containers.
• yield is an expression and returns the value sent with send; when a thousand-item stream is left with close after three items, production stays at 3, but the body’s finally block still runs.

Next Step

This lesson only measured the moment of setup, and there was a gap there: 1000 against 0. But a stream is consumed sooner or later — if a thousand items are really going to be collected, the generator has to produce those thousand objects too. The next lesson asks exactly this question: for two notations that carry the same job to its end, how many objects has each produced by the end of the job, and how many does each still hold on to? The first half of the answer will be expected, the second half will not — and only there does it become visible what laziness actually reduces.