Loops

The previous lesson measured the truthiness rule: the object after if is put through a question, and that question’s answer ends with a branch. The trial happens once, the result is read once.

A loop does not repeat the same question, it repeats a different question: is there an item next in line? The Programming Fundamentals course’s loops lesson built this structure as a control structure — the number of turns depended either on a collection’s length or on a condition, and the loop’s termination was guaranteed by a change on each turn that moved toward falsifying the condition. In Python a loop is a protocol, and that lesson’s reason for termination does not apply here: a for loop has no condition it tests. So what decides the turns have ended? This is what this lesson measures — and the answer turns out to be an exception class.

What a Loop Calls

The notation for x in n has three steps, and all three are hidden. First an iterator is requested from the object: __iter__ is called. Then on every turn the next item is requested from that iterator: __next__ is called. The third step is termination — and what announces it is not a return value.

It could not have been a return value. __next__ can return any object; whatever value we chose, that value could be a real item of the collection, and “done” would get mixed up with “here is your item.” This is why termination is taken out of the channel: __next__ throws an exception when it has no items left to give — StopIteration. This is what ends the loop.

There is a direct consequence: one last call, a call producing no item, is always made. Regardless of item count, __next__ is called one extra time, and that extra call is the loop’s termination announcement.

Writing the Same Loop by Hand

The way to show the protocol is really a shorthand is to do the same job without for: take the iterator with iter(), set up an infinite loop, call next() on every turn, and exit when StopIteration is caught. If the two notations produce the same protocol sequence, for really is the shorthand for these three steps.

Two more objects enter the measurement. Tracker defines both __iter__ and __getitem__; either is enough for iteration, but only counting tells which one is used. Indexed defines only __getitem__ and has no __iter__ — whether such an object can even enter a loop, and if so which protocol it follows, is the second question.

The measurement’s assumptions:

• CF8 — The oracle is the rig itself: it logs the name every time a special method is called; the measure for “how many times was it called” is this log.
• CF9 — Tracker keeps the behavior from the shared reference: __iter__ resets the counter and returns itself, __next__ throws StopIteration once items run out.
• CF10 — The item count is varied from zero to four; what is measured is not the count itself but the gap between it and the call count.
• CF11 — break is run on the second item; the break point’s location is fixed and chosen to isolate the effect early termination has on the call count.
• CF12 — The measurement uses no list-building notation, only an explicit for body; its purpose is guaranteeing the counted calls come only from the iteration protocol.
• CF13 — Indexed defines only __getitem__; the absence of __iter__ is deliberate and needed so the second protocol can be measured.
• CF14 — In the while measurement, the body removes one item every turn; the guarantee of termination comes from this, and the trial count and turn count are read separately.
• CF15 — The nested-loop measurement is done in two states: the two loops on the same object, and on separate objects. Tracker.__iter__ returns itself and resets the counter as in the shared reference; what is measured is the effect these two choices have on the turn count.

Measurement

"""Iteration protocol: how many times does a loop call __next__, and what ends it."""

LOG = []


def record(name):
    LOG.append(name)


class Tracker:
    """Both __iter__ and __getitem__ defined."""

    def __init__(self, items=(1, 2, 3)):
        self.items = list(items)

    def __iter__(self):
        record("__iter__")
        self._i = 0
        return self

    def __next__(self):
        record("__next__")
        if self._i >= len(self.items):
            raise StopIteration
        value = self.items[self._i]
        self._i += 1
        return value

    def __getitem__(self, k):
        record("__getitem__")
        return self.items[k]

    def __len__(self):
        record("__len__")
        return len(self.items)


class Indexed:
    """Only __getitem__ defined: no __iter__."""

    def __init__(self, items=(1, 2, 3)):
        self.items = list(items)

    def __getitem__(self, k):
        record("__getitem__")
        return self.items[k]


def measure(action):
    LOG.clear()
    try:
        action()
    except Exception as e:
        LOG.append(f"!{type(e).__name__}")
    return list(LOG)


LAPS = []


def complete():
    for x in Tracker():
        LAPS.append(x)


def interrupted():
    for x in Tracker():
        LAPS.append(x)
        if x == 2:
            break


def finished_else():
    for x in Tracker():
        LAPS.append(x)
    else:
        record("else")


def interrupted_else():
    for x in Tracker():
        LAPS.append(x)
        if x == 2:
            break
    else:
        record("else")


def manual():
    y = iter(Tracker())
    while True:
        try:
            LAPS.append(next(y))
        except StopIteration:
            break


def fourth():
    y = iter(Tracker())
    for _ in range(4):
        next(y)


FORMS = (("for x in n", complete), ("for + break", interrupted),
         ("for + else", finished_else), ("for + break + else", interrupted_else),
         ("iter/next manual", manual), ("fourth next", fourth))

print(f"{'form':<20s} {'laps':>4s} {'__next__':>9s} {'else':>5s}  protocol sequence")
for name, f in FORMS:
    LAPS.clear()
    c = measure(f)
    p = [a for a in c if a != "else"]
    print(f"  {name:<18s} {len(LAPS):4d} {c.count('__next__'):9d}"
          f" {('yes' if 'else' in c else '-'):>5s}  {' '.join(p)}")

print()
print(f"{'item':>4s} {'laps':>4s} {'__next__':>9s}  diff")
for n in range(5):
    LAPS.clear()

    def count(n=n):
        for x in Tracker(range(n)):
            LAPS.append(x)
    c = measure(count)
    print(f"  {n:2d} {len(LAPS):4d} {c.count('__next__'):9d}"
          f"  {c.count('__next__') - n:+d}")

print()
print(f"{'object':<10s} {'defines':<24s} {'calls':>6s}  protocol sequence")
for name, build, defines in (
        ("Tracker", Tracker, "__iter__ and __getitem__"),
        ("Indexed", Indexed, "only __getitem__")):
    def walk(k=build):
        for _ in k():
            pass
    c = measure(walk)
    print(f"  {name:<8s} {defines:<24s} {len(c):6d}  {' '.join(c)}")
print("  fourth index directly:", " ".join(measure(lambda: Indexed()[3])))


OUTER = []


def nested_same():
    n = Tracker()
    for x in n:
        OUTER.append(x)
        for y in n:
            LAPS.append(y)


def nested_separate():
    for x in Tracker():
        OUTER.append(x)
        for y in Tracker():
            LAPS.append(y)


print()
print(f"{'nested loop':<18s} {'outer laps':>10s} {'inner laps':>10s}"
      f" {'__iter__':>9s} {'__next__':>9s}")
for name, f in (("same object", nested_same), ("separate objects", nested_separate)):
    LAPS.clear()
    OUTER.clear()
    c = measure(f)
    print(f"  {name:<16s} {len(OUTER):10d} {len(LAPS):10d}"
          f" {c.count('__iter__'):9d} {c.count('__next__'):9d}")


def while_loop():
    n = Tracker()
    while n:
        n.items.pop()


print()
c = measure(while_loop)
print(f"while n  (until 3 items empty): calls {len(c)}  {' '.join(c)}")

form                 laps  __next__  else  protocol sequence
  for x in n            3         4     -  __iter__ __next__ __next__ __next__ __next__
  for + break           2         2     -  __iter__ __next__ __next__
  for + else            3         4   yes  __iter__ __next__ __next__ __next__ __next__
  for + break + else    2         2     -  __iter__ __next__ __next__
  iter/next manual      3         4     -  __iter__ __next__ __next__ __next__ __next__
  fourth next           0         4     -  __iter__ __next__ __next__ __next__ __next__ !StopIteration

item laps  __next__  diff
   0    0         1  +1
   1    1         2  +1
   2    2         3  +1
   3    3         4  +1
   4    4         5  +1

object     defines                   calls  protocol sequence
  Tracker  __iter__ and __getitem__      5  __iter__ __next__ __next__ __next__ __next__
  Indexed  only __getitem__              4  __getitem__ __getitem__ __getitem__ __getitem__
  fourth index directly: __getitem__ !IndexError

nested loop        outer laps inner laps  __iter__  __next__
  same object               1          3         2         6
  separate objects          3          9         4        16

while n  (until 3 items empty): calls 4  __len__ __len__ __len__ __len__

One Extra Call

The top table’s first row is the shared reference’s third claim. On a three-item object, the loop takes three laps, but __next__ is called four times. The extra fourth call produces no item; it exists to announce that it produced none.

The second table shows this is not tied to a single instance. As item count rises from zero to four, the diff column does not change: +1 on every row. There is a call even on a zero-item object — even a loop that never takes a lap has to ask once, because it cannot know it is empty without asking. Call count is not a function of item count, it is item count plus the termination announcement.

The last row shows that announcement directly. When the iterator is taken by hand and next() is called four times, the fourth does not return a value, it throws StopIteration. We do not see this exception in a for loop because the loop catches it itself and silently carries flow out of the body. Not seeing it does not mean it is absent: every normal termination of a loop gives birth to an exception, and that exception is caught.

The reading that follows is the shared reference’s third claim: the exception is not an error, it is part of the protocol. If it were an error it would announce an exceptional state; instead this exception announces the most ordinary state — the collection ran out.

The Shorthand Expanded

The fifth row proves the for notation really is a shorthand. The hand-written form — iter(), an infinite loop, next(), catching StopIteration — produces exactly the same protocol sequence: one __iter__, four __next__. There is no execution difference between the two notations; the only difference is that the try block is visible.

The second and fourth rows measure a loop cut off by break, and the count here stands apart from all the rest: two laps, two calls. There is no extra call. break ends the loop from outside the protocol; the third item is never requested, StopIteration is never born. A cut-off loop is the only form where the one extra call disappears.

The same topic’s other control statement, continue, needs no separate row in this table, because it does not touch the protocol at all: it skips the rest of the body and carries the loop to the next __next__ call. The lap count does not change, the extra call stays in place. This is exactly the difference between it and break — one skips a step inside the protocol, the other leaves the protocol.

This also explains the else column. A loop’s attached else block runs when the loop ends by its own protocol; it does not run when cut off by break. In the third row else runs, in the fourth it does not. In other words, else is the answer to the question “did it finish with no break at all?” — put differently, did StopIteration really get thrown? The search pattern that breaks on finding a sought item and behaves in the else block when not found is built directly on this distinction.

The Second Protocol

The third table measures which protocol gets chosen, and it repeats the previous lesson’s pattern.

Tracker defines both __iter__ and __getitem__. The loop chooses __iter__, and __getitem__ is never called. Just as __bool__ disabled __len__ in the truthiness rule, here __iter__ disables __getitem__; the two methods do not compete, the first shuts off the second.

The Indexed row shows the second rung. This object has no __iter__, but the loop still runs: __getitem__ is called four times with rising indices starting from zero. For an object with no iterator, the language recognizes a second path of iteration — request indices in order, until an index turns out invalid.

The two roles have to be separated. An iterable object is one that defines __iter__; it is not walked itself, it gives what will walk it. An iterator is one that defines __next__, and it is the one that actually does the walking. It is not forbidden for one object to define both — Tracker does exactly this and returns itself in the body of __iter__. This is why Tracker is both iterable and an iterator; what happens when that distinction collapses is measured below.

The count is again four, one more than the item count. But the exception ending the extra call is different this time: the last row calls Indexed()[3] directly, and what comes out is not StopIteration but IndexError. Two protocols, two separate termination exceptions, the same +1 pattern. In both paths, what tells the loop it has ended is not a return value but a thrown exception.

Where the Multiplication Rule Breaks

The fourth table measures two nested loops and tests the rule the Programming Fundamentals course gave for nested loops — that the total lap count is the product of the two. On separate objects the rule holds: outer 3 laps, inner 9 laps, that is, three times three.

It does not hold on the same object. The outer loop takes not 3 but 1 lap; the inner takes not 9 but 3. The multiplication rule collapses, and the reason is not the loop itself but what __iter__ returns.

Tracker.__iter__ does not produce a new object; it returns itself and resets the counter. After the outer loop takes its first item, the inner loop calls __iter__ on the same object, the counter resets, the inner loop consumes all three items and drives the counter to the end. When the outer loop asks for its next item, the counter is already at the end: StopIteration arrives and the outer loop ends on its first lap. The two loops share the same counter.

The distinction here is that an iterable object and an iterator are separate things. An object returning a fresh iterator on every __iter__ call behaves as expected in nested loops; one returning itself cannot be walked more than once at the same time. The syntax is identical in both cases, the lap count is not — and the only thing that says which behavior applies is the body of __iter__. It is impossible to see this distinction by looking at the object’s type, name, or item count.

The last row gives the same pattern for while. A loop running until three items are emptied calls __len__ four times: three trials for three laps, plus a fourth trial that falsifies the condition. for calls __next__ one extra time, while does one extra truthiness trial. In both cases, the extra one is the trial that ends the loop.

Summary

• In Python a loop is not a control structure, it is a protocol: for x in n first calls __iter__, then __next__ for every turn; it has no condition it tests.
• On a three-item object, __next__ is called four times, and the difference is independent of item count — a call happens even on zero items, because emptiness can only be learned by asking.
• The extra call does not return a value, it throws StopIteration; this is what ends the loop, and it goes unseen because for catches this exception itself. The exception is not an error, it is part of the protocol.
• The for notation is shorthand for iter, next, and catching StopIteration; the hand-written expansion produces exactly the same call sequence. break ends the loop from outside the protocol and removes the extra call — the else block reads exactly this distinction.
• While __iter__ is defined, __getitem__ takes no part in iteration at all; on an object defining only __getitem__, the loop calls it with rising indices, and the exception ending this path is IndexError.
• The multiplication rule for nested loops breaks when __iter__ returns itself: two loops on the same object share a single counter, and the outer loop takes 1 lap instead of 3; on separate objects the rule still holds (3 and 9).

Next Step

A loop body most often calls a function, and that function is passed different values on every turn. The Programming Fundamentals course built the parameter-versus- argument distinction and the call frame; in Python, the forms of passing an argument go beyond what that lesson assumed — positional and keyword arguments, collecting a variable number of arguments, default values. One of these is about timing, and that is what the next lesson measures: when is a parameter’s default value computed — on every call, or once? The answer, together with a mutable default, produces a surprising number.