Generator Expressions

The previous lesson measured only the moment of setup, and there was a gap there: a comprehension materializes 1000 objects on its first line, a generator materializes 0. But a stream is not written to be set up and left; it is consumed sooner or later. If a thousand items are really going to be collected, the generator has to produce those thousand objects too.

This lesson’s question: for two notations that carry the same job to its end, how many objects has each produced once done, and how many does each still hold? Half the answer will be expected, the other half will not — and only there does it become visible what laziness actually reduces. The lesson also builds the generator expression’s own notation, measured here on its own.

Notation

A generator expression is a list comprehension with its square brackets replaced by parentheses, and what it produces is not a list but a generator object. Given as the sole argument to a function, the parentheses are not written separately; the expression in sum(o.value for o in stream) is itself a generator expression on its own.

The object it produces is the same kind the previous lesson’s generator function returned: an iterator, single-pass, __iter__ returning itself. The difference is only in notation. A generator function takes a name, can carry multiple statements, can branch; a generator expression is nameless and states a single transformation. The choice between them is not one of cost but of expression width — the measured numbers are the same in both.

The Programming Fundamentals course introduced lazy evaluation as a concept at the entrance to functional programming: a value is not computed until it is used. This lesson does not repeat that concept; it counts how many objects its Python counterpart costs — and the number turns out different from what the name suggests.

The measurement’s assumptions:

• IF19 — The shared reference’s Item class and the number N = 1000 are used as is; the counting arrangement is unchanged. The oracle is the rig itself.
• IF20 — The two consumption forms target the same result: the value sum of a thousand items. The result column is the measurement’s correctness check; if the two forms gave different results, the comparison would be invalid.
• IF21 — “Held in list” is the number of items still alive in a container once the sum is done; it is zero in the generator form because no such container is built.
• IF22 — The chain measurement has three links: produce, filter, transform. The two forms apply the same links in the same order; the only thing that changes is whether each link is a container or a stream.
• IF23 — The chain’s “held in containers” count is the sum of the three lists’ lengths. The memory claim is built on the object count, not measured bytes; duration is not measured.
• IF24 — In the early-exit measurement, the sought condition is satisfied at the fifth item; the condition’s position is fixed and decides when the consumer stops.
• IF25 — The single-pass measurement is done on a small ten-item stream; what is measured is not size, it is the value the second summation returns.
• IF26 — In the source-evaluation measurement, give_source counts its own call count; this count is the measure of when the outer iterable is evaluated.
• IF27 — In the last table, every operation is tried separately on two containers, and setup is reset before each trial; the exception class of a failing operation is also taken from the run. Only the standard library is used.

Measurement

"""Generator expression: same result, same production, different retention."""

import itertools

COUNTER = {"produced": 0}


class Item:
    """An item that counts every production of itself."""

    def __init__(self, value):
        COUNTER["produced"] += 1
        self.value = value

    def __repr__(self):
        return f"Item({self.value})"


def reset():
    COUNTER["produced"] = 0


def produced():
    return COUNTER["produced"]


N = 1000


def generator_consumption(n=N):
    reset()
    stream = (Item(i) for i in range(n))
    total = sum(o.value for o in stream)
    return produced(), total, 0


def comprehension_consumption(n=N):
    reset()
    items = [Item(i) for i in range(n)]
    total = sum(o.value for o in items)
    return produced(), total, len(items)


print(f"{'consumption form':<27s} {'produced':>9s} {'total':>8s}"
      f" {'held in list':>13s}")
for name, f in (("summing via generator", generator_consumption),
              ("summing via comprehension", comprehension_consumption)):
    p, t, held = f()
    print(f"  {name:<25s} {p:9d} {t:8d} {held:13d}")


def lazy_chain(n=N):
    reset()
    evens = (o for o in (Item(i) for i in range(n)) if o.value % 2 == 0)
    squares = (o.value * o.value for o in evens)
    at_setup = produced()
    total = sum(squares)
    return at_setup, produced(), total, 0


def list_chain(n=N):
    reset()
    all_items = [Item(i) for i in range(n)]
    evens = [o for o in all_items if o.value % 2 == 0]
    squares = [o.value * o.value for o in evens]
    at_setup = produced()
    total = sum(squares)
    return at_setup, produced(), total, len(all_items) + len(evens) + len(squares)


print()
print(f"{'chain form':<29s} {'at setup':>10s} {'produced':>9s}"
      f" {'total':>10s} {'held in containers':>19s}")
for name, f in (("three generator expressions", lazy_chain), ("three list comprehensions", list_chain)):
    s, p, t, held = f()
    print(f"  {name:<27s} {s:10d} {p:9d} {t:10d} {held:19d}")


def early_exit(n=N):
    reset()
    stream = (Item(i) for i in range(n))
    found = any(o.value > 4 for o in stream)
    return produced(), found


def early_exit_list(n=N):
    reset()
    items = [Item(i) for i in range(n)]
    found = any(o.value > 4 for o in items)
    return produced(), found


print()
p1, f1 = early_exit()
p2, f2 = early_exit_list()
print(f"when the consumer stops at the fifth item: generator expression produced {p1} objects"
      f" (answer {f1}), comprehension {p2} (answer {f2})")

reset()
stream = (Item(i) for i in range(10))
first = sum(o.value for o in stream)
second = sum(o.value for o in stream)
print(f"same generator expression summed twice: {first} and {second},"
      f" produced {produced()}")

CALLS = {"source": 0}


def give_source():
    CALLS["source"] += 1
    return [1, 2, 3]


stream = (x for x in give_source())
at_setup = CALLS["source"]
list(stream)
print(f"was the outer iterable evaluated at setup: {at_setup} calls,"
      f" after walking {CALLS['source']}")

remaining = [5, 6, 7]
stream = (x * 10 for x in remaining)
remaining.append(8)
print(f"did an item added to the source afterward enter the stream: {list(stream)}")

reset()
a_set = {o.value % 3 for o in (Item(i) for i in range(9))}
print(f"generator expression given to a set constructor: produced {produced()},"
      f" held {len(a_set)}")

OPERATIONS = (("len(n)", lambda n: len(n)),
            ("n[2]", lambda n: n[2].value),
            ("n[:3]", lambda n: len(n[:3])),
            ("islice(n, 3)", lambda n: len(list(itertools.islice(n, 3)))))

print()
print(f"{'operation':<16s} {'list':>25s} {'generator expression':>28s}")
for name, operation in OPERATIONS:
    row = []
    for build in (lambda: [Item(i) for i in range(N)],
                lambda: (Item(i) for i in range(N))):
        reset()
        n = build()
        try:
            result = f"{operation(n)} ({produced()} productions)"
        except TypeError as e:
            result = f"{type(e).__name__} ({produced()} productions)"
        row.append(result)
    print(f"  {name:<14s} {row[0]:>25s} {row[1]:>28s}")

reset()
stream = (Item(i) for i in range(N))
exists = any(o.value == 2 for o in stream)
at_test = produced()
left = sum(1 for _ in stream)
print()
print(f"membership test on a stream: answer {exists}, produced during test {at_test},"
      f" remaining in stream {left}")

consumption form             produced    total  held in list
  summing via generator          1000   499500             0
  summing via comprehension      1000   499500          1000

chain form                      at setup  produced      total  held in containers
  three generator expressions          0      1000  166167000                   0
  three list comprehensions         1000      1000  166167000                2000

when the consumer stops at the fifth item: generator expression produced 6 objects (answer True), comprehension 1000 (answer True)
same generator expression summed twice: 45 and 0, produced 10
was the outer iterable evaluated at setup: 1 calls, after walking 1
did an item added to the source afterward enter the stream: [50, 60, 70, 80]
generator expression given to a set constructor: produced 9, held 3

operation                             list         generator expression
  len(n)           1000 (1000 productions)    TypeError (0 productions)
  n[2]                2 (1000 productions)    TypeError (0 productions)
  n[:3]               3 (1000 productions)    TypeError (0 productions)
  islice(n, 3)        3 (1000 productions)            3 (3 productions)

membership test on a stream: answer True, produced during test 3, remaining in stream 997

Same Production, Separate Retention

The top table completes the shared reference’s second claim, and the first column breaks the expectation.

Produced is 1000 in both. The lazy form does not produce fewer objects; a thousand items were requested, a thousand were born. The total column matches too: both 499500. The two notations do the same job, give the same result, pay the same cost.

The only column where they split is the last. Once summing is done, 1000 items still sit in a list in the comprehension form; 0 in the generator form. Each of the thousand produced objects was born, contributed to the sum, and became unreachable because nothing wrote it into a container. In the comprehension all of them stayed in the list, because that is exactly the list’s job.

From this comes the shared reference’s sentence: laziness does not reduce production, it reduces retention. This is narrower than the name “lazy evaluation” suggests. The name implies work will not be done; the measurement shows it is done, and only when and what is left afterward change. Laziness defers production, it does not avoid it.

The previous lesson’s 0 has to be read together with this table. Zero at setup is not a promise of savings, it is a record of deferral. Whether the promise is kept depends on how the stream is consumed, not on how it is written.

The Chain’s Intermediate Containers

The middle table measures not a single transformation but a three-link chain: produce, filter, transform. The total is the same again — 166167000 — and produced is again 1000. Two columns split.

The setup column repeats the previous lesson’s result: the lazy chain 0, the list chain 1000. But the real difference is in the last column. Once the list chain finishes, 2000 references sit in containers: a thousand-item source list, a five-hundred-item filtered list, a five-hundred-item transformed list. In the lazy chain this number is 0.

Intermediate-container count grows with chain length. A four-link list chain builds four containers, a five-link one five; each link wants to see all of what came before and stores what it saw. In the lazy chain, container count does not grow with link count, because no link ever wants all of what came before — each wants one item at a time.

Together with the previous lesson’s last row, this gives the chain’s breaking point: the moment a single list() call enters the chain, a container is built at that point, and the rest of the chain’s laziness cannot undo it.

Early Exit’s Source

The next line measures the most natural objection to the claim. When the consumer stops at the fifth item, the generator expression produces 6 objects, the comprehension 1000. Here laziness really does look like it avoided production.

The distinction: what enables the avoidance is not laziness, it is the consumer stopping early. any stops asking once it finds what it is looking for; the lazy stream produces nothing that was not asked for. In the comprehension, even if the consumer stops early, production already finished — the list was filled before any asked its first question.

Together, the two cases give the claim’s full shape: laziness ties production to the consumer. Wanting everything produces everything (first table: 1000); exiting early leaves what was not needed unproduced (this line: 6). In the comprehension, production is independent of the consumer and is 1000 either way. What laziness gives is not avoidance, it is the possibility of avoidance.

Why the sixth object gets produced is part of the same pattern: the condition is satisfied at the fifth item, but any can only know this after taking it, and taking it costs one more production.

When the Source Is Evaluated

The last four lines measure two subtle sides of a generator expression.

The first is single-pass behavior. Summed twice, the same generator expression gives 45 and 0; produced stays at 10. The second sum runs on an empty stream, and sum returns zero on an empty stream — no error, a silent zero. This is single-pass behavior’s most treacherous consequence: an exhausted stream cannot be told apart from an empty one.

The second is when the source is read. give_source is called once at setup, and stays at 1 after the stream is walked. The outermost iterable in a generator expression is evaluated immediately; what is deferred is pulling items, not finding the source. A wrong source name or an erroring call shows up at setup.

The next line shows this rule’s limit. The source list was evaluated at setup, but an item added afterward entered the stream: the result is [50, 60, 70, 80]. What got evaluated was the list object, not a snapshot of its content — the same distinction as the previous topic’s view measurement: a binding was established, no copy taken. If a lazy stream’s source changes before consumption, the stream sees the changed state.

The last line shows where a generator expression can be given: a stream given to a set constructor makes it produce 9 objects, and 3 items are retained afterward. The gap between materialized and retained is visible here too — the container decides for itself how much it keeps.

What a Stream Cannot Do

The last table measures what is given up in exchange for not retaining, and the same exception sits in three of its rows.

len(n) gives 1000 on a list, TypeError on a stream. A stream does not know its length, because knowing it would require going all the way to the end and remembering that it did. n[2] and n[:3] give the same answer: a stream cannot be indexed or sliced. Everything the previous topic’s slicing lesson measured — a slice building a new container, sharing inner items — can never even be asked of a stream, because a slice requires looking backward, and a stream has nowhere to look back to.

The shared reason: every one of these questions requires retention. Length, index, and slice all assume items sit together; a stream answers none of the three because it refuses to do exactly that. Zero retention costs three built-in operations.

The fourth row shows where the loss can be made up. The standard library’s islice tool takes the first three items from a stream and causes only 3 productions; the same result on a list comes only after 1000. A job that looks like a slice, defined without looking backward, becomes possible on a stream too — but the result is not a slice, it is another stream, and it too is single-pass.

The last row is single-pass behavior’s most attention-demanding consequence. The membership test on a stream answers True, only 3 objects produced during the test; but 997 items remain in the stream after. The test consumed the stream: the three items asked about do not come back. A second test on the same stream would search from the third item onward, and the first three would never be seen. A list has no such side effect; asking on a stream is reading.

Summary

• Summing a thousand items, a generator and a comprehension both produce 1000 objects and both give 499500; the only place they split is what stays held — 0 against 1000. Laziness defers production, it does not avoid it.
• In a three-link chain, produced is still 1000, but the list form holds 2000 references in containers at the end, the lazy form 0. Intermediate container count grows with chain length in a list, not in a stream.
• When the consumer stops at the fifth item, a generator expression produces 6, a comprehension 1000. What enables the avoidance is not laziness but the consumer stopping; laziness only makes it possible.
• A generator expression is single-pass and behaves silently empty once exhausted: a second sum gives not an error but 0.
• The outermost iterable is evaluated at setup (1 call), but what gets evaluated is the object itself: an item added to the source afterward enters the stream.

Next Step

These two lessons measured the difference between square brackets and parentheses, but never opened up the square brackets themselves. Yet the same notation can build three separate containers — list, set, and dict — and all three take the same items from the same source and hold a different number of them. What is more, the notation can nest, and one comprehension inside another produces an unbuilt container. The next lesson counts these: of the same thousand items, how many does each of the three comprehension forms retain, and does where a filter clause is written change the number of objects produced?