Python's Execution Model

The Software Development Practice curriculum built a team discipline across five courses and, at its close, said that everything it measured was language-independent: what was counted was never the code itself, but the work around it. This curriculum turns from the surroundings back to the center. It’s the language’s own turn.

The Programming Fundamentals course wrote its examples in Python; assignment as a binding, the is/== distinction, basic data types, operators, and type conversion were all built there. But using a language as an example and learning the language itself are different things: there, Python was a vehicle for explanation; here, it is the object being measured. This course looks beneath the syntax and carries a single question across sixteen lessons — which machinery does a written form invoke while it runs, and how many times?

Two Phases: Compiling and Executing

Python runs with an interpreter, but it does not read the source line by line and run it immediately. When a file is run, the text is first split into tokens, then turned into an abstract syntax tree, then translated into bytecode. What the interpreter runs is the bytecode, not the source text.

The boundary between the two phases is observable. Compilation is done over the whole file: a syntax defect on the source’s last line also blocks the execution of the statement on the first line. An error born during execution, by contrast, only surfaces once its own turn comes, and everything before it has already run. The difference can be measured with two built-in functions: compile compiles the source and returns a code object, exec runs that object.

"""Two phases: the source is first compiled as a whole, then executed."""

SOURCE = {
    "missing parenthesis": 'write = print\nwrite("one")\nwrite(\n',
    "undefined name": 'write = print\nwrite("one")\nwritee("two")\n',
}

for name, source in SOURCE.items():
    print(f"--- {name}")
    try:
        code_obj = compile(source, "<lesson>", "exec")
    except SyntaxError as e:
        print(f"compile: {type(e).__name__} — no statement executed")
        continue
    print(f"compile: clean, obtained a {type(code_obj).__name__} object")
    try:
        exec(code_obj, {})
    except Exception as e:
        print(f"execution: {type(e).__name__} — statements before it were executed")

--- missing parenthesis
compile: SyntaxError — no statement executed
--- undefined name
compile: clean, obtained a code object
one
execution: NameError — statements before it were executed

In the second source, the line one was printed, and the call after it fell with a NameError. In the first source, nothing was printed at all — even though its first two lines are flawless. The distinction is this: a syntax defect is found at compile time, a name defect at execution time. Whether a name exists is not known at compile time, because a name is only bound to an object during execution.

The same two steps also work in the interactive shell; there, the unit compiled is not the whole file but the block entered. If the block’s syntax is broken, the block never runs at all.

The layout of the bytecode that compilation produces is not part of the language’s definition; it is the interpreter’s internal business, and reading it is not this course’s subject. What is worth reading is which methods the bytecode calls.

Syntax Is a Shorthand

The course’s measurement axis comes from a single observation: most forms of Python’s syntax are a shorthand. When n + m is written, what runs is not an addition but a call — the n object’s __add__ method is called, and the returned value is the expression’s value. for x in n is not a looping keyword but two methods called in sequence: first __iter__, then __next__ until it runs out.

The names of these methods are written into the language’s definition; each one is called a special method, and the set of methods a form requires is called a protocol. Their names are written between two underscores, and this is meant to set them apart from ordinary methods: special methods do not exist to be called directly, they exist to be called by syntax.

Not every syntax form is a shorthand, and this distinction has to be set up from the start. Binding a value to a name, defining a function, or returning a value asks the object no question at all; these forms are concerned with names and flow, not with what is inside the object. The ones that are a shorthand are forms that consume an object: they sum it, walk through it, test it in a condition, slice it, turn it into text. All eleven of the forms measured here belong to this second set.

This has a direct consequence: syntax does not look at the type’s name, it looks at whether the method exists. That the + operator adds numbers and concatenates strings is not a special privilege but the fact that two separate types define a method with the same name in different ways. The Programming Fundamentals course built this observation as a concept under the heading operator overloading; what is measured here is not the concept but the name that carries it out — which form calls which method, and how many times.

The Shared Setup: an Object That Logs Its Participation

A single class is enough for the measurement. Tracker defines a large number of special methods, and every time a method is called, it writes its name to a shared list. Ordinary syntax is run on it — for, in, if, +, +=, slicing, str(), with, == — and which method each form calls, how many times, is read from the list.

A second class, NonEmpty, derives from Tracker and adds only one thing: __bool__. The same if expression will now call a different method on this object. The oracle is the setup itself: because the object records which method got called, we know it directly — the count is a log, not an inference.

"""Protocols called by syntax: every special method logs itself when called."""

LOG = []


def record(name):
    LOG.append(name)


class Tracker:
    """An object that takes part in many protocols; records each participation."""

    def __init__(self, items=(1, 2, 3)):
        self.items = list(items)

    def __iter__(self):
        record("__iter__")
        self._i = 0
        return self

    def __next__(self):
        record("__next__")
        if self._i >= len(self.items):
            raise StopIteration
        value = self.items[self._i]
        self._i += 1
        return value

    def __len__(self):
        record("__len__")
        return len(self.items)

    def __contains__(self, x):
        record("__contains__")
        return x in self.items

    def __add__(self, o):
        record("__add__")
        return Tracker(self.items + list(o.items))

    def __iadd__(self, o):
        record("__iadd__")
        self.items.extend(o.items)
        return self

    def __getitem__(self, k):
        record("__getitem__")
        return self.items[k]

    def __eq__(self, o):
        record("__eq__")
        return isinstance(o, Tracker) and self.items == o.items

    def __str__(self):
        record("__str__")
        return f"Tracker{tuple(self.items)}"

    def __repr__(self):
        record("__repr__")
        return self.__str__()

    def __enter__(self):
        record("__enter__")
        return self

    def __exit__(self, kind, value, tb):
        record("__exit__")
        return False          # does not swallow the exception


class NonEmpty(Tracker):
    """__bool__ defined: the truthiness test no longer falls back to __len__."""

    def __bool__(self):
        record("__bool__")
        return len(self.items) > 0


def measure(function):
    """Runs a syntax form and returns the protocols it called."""
    LOG.clear()
    try:
        function()
    except Exception as e:
        LOG.append(f"!{type(e).__name__}")
    return list(LOG)


def loop():
    for _ in Tracker():
        pass


def membership():
    9 in Tracker()


def truthiness_len():
    if Tracker():
        pass


def truthiness_bool():
    if NonEmpty():
        pass


def addition():
    Tracker() + Tracker()


def in_place_addition():
    a = Tracker()
    a += Tracker()


def slicing():
    Tracker()[0:2]


def to_string():
    str(Tracker())


def context():
    with Tracker():
        pass


def context_with_exception():
    with Tracker():
        raise ValueError("example")


def equality():
    Tracker() == Tracker()


FORMS = (
    ("for x in n", loop),
    ("x in n", membership),
    ("if n  (__len__)", truthiness_len),
    ("if n  (__bool__)", truthiness_bool),
    ("n + m", addition),
    ("n += m", in_place_addition),
    ("n[0:2]", slicing),
    ("str(n)", to_string),
    ("with n", context),
    ("with n + exception", context_with_exception),
    ("n == m", equality),
)

The measurement’s assumptions:

• LF1 — The forms measured are eleven; not the language’s entire syntax, but the core this course pays off lesson by lesson. The remaining forms are measured in their own lessons.
• LF2 — The oracle is the object’s own log: it writes its name to the list when a method is called, so the measurement is a count, not an inference.
• LF3 — Tracker is built with three items; the call count read in the loop measurement depends on this three.
• LF4 — __exit__ returns False on every run; an exception born inside the context block is not swallowed, and it enters the measurement as an exception name with an exclamation mark in front of it.
• LF5 — The measurement reads no environment data: memory address, object identity count, and duration are not used; the only thing counted is the names of the methods called.
• LF6 — Every form is measured twice — once for the table, once for the paired comparison. Because Tracker is rebuilt on every measurement, the two runs give the same sequence.

Measurement

print(f"{'syntax':<20s} {'calls':>6s}  protocol sequence")
for name, function in FORMS:
    c = measure(function)
    print(f"{name:<20s} {len(c):6d}  {' '.join(c)}")

print()
by_form = {name: measure(f) for name, f in FORMS}
unique = sorted({p for c in by_form.values() for p in c if not p.startswith("!")})
print(f"distinct protocols {len(unique)}: {', '.join(unique)}")
print(f"total calls {sum(len(c) for c in by_form.values())}, "
      f"forms ending in an exception "
      f"{sum(1 for c in by_form.values() if any(p.startswith('!') for p in c))}")

print()
print("pairs that look the same but call different protocols:")
for a, b in (("n + m", "n += m"),
             ("if n  (__len__)", "if n  (__bool__)")):
    print(f"  {a:<18s} -> {' '.join(by_form[a])}")
    print(f"  {b:<18s} -> {' '.join(by_form[b])}")

print()
loop_log = by_form["for x in n"]
print(f"loop over a three-item object: __next__ called {loop_log.count('__next__')} times "
      f"— one more than the item count; the last one ended with StopIteration")

syntax                calls  protocol sequence
for x in n                5  __iter__ __next__ __next__ __next__ __next__
x in n                    1  __contains__
if n  (__len__)           1  __len__
if n  (__bool__)          1  __bool__
n + m                     1  __add__
n += m                    1  __iadd__
n[0:2]                    1  __getitem__
str(n)                    1  __str__
with n                    2  __enter__ __exit__
with n + exception        3  __enter__ __exit__ !ValueError
n == m                    1  __eq__

distinct protocols 12: __add__, __bool__, __contains__, __enter__, __eq__, __exit__, __getitem__, __iadd__, __iter__, __len__, __next__, __str__
total calls 18, forms ending in an exception 1

pairs that look the same but call different protocols:
  n + m              -> __add__
  n += m             -> __iadd__
  if n  (__len__)    -> __len__
  if n  (__bool__)   -> __bool__

loop over a three-item object: __next__ called 4 times — one more than the item count; the last one ended with StopIteration

Reading the Numbers

Eleven forms produce 12 distinct protocols and 18 calls in total. This is the measure the course will carry forward: a lesson’s number is which protocol the syntax it covers calls, and how many times. Every row of the table is a syntax form, and the sequence next to it shows the calls that really ran underneath that form. None of it is a guess; all of it was read from the object’s own log.

Two expressions that look alike call different protocols. n + m and n += m sit side by side as if one were shorthand for the other, but one calls __add__, the other __iadd__. The difference is not just a name: __add__ produces and returns a new object, __iadd__ mutates the existing object and returns itself. In the same way, if n calls __len__ on Tracker, __bool__ on NonEmpty — a single syntax, two separate methods. What creates the difference is which method the object defines. The look of the syntax does not tell you which protocol gets called. The first half of this claim will be paid off in this topic’s operators lesson, the second half in the flow topic’s conditionals lesson.

An exception is not an error, it is part of the protocol. The loop over the three-item object called __next__ 4 times — one more than the item count. The fourth call does not return a value, it raises StopIteration; this is exactly what ends the loop. The word for does not know the items ran out by counting, it knows by catching an exception. The line with n + exception is of the same family: even though an exception is born inside the block, __exit__ is still called, and its name stands in the log as the third entry. Because __exit__ returns False, the exception made it out — had it returned True, it would not have.

No Protocol, No Syntax

The table above is full because Tracker has these methods. What happens if the same forms are tried on an object that defines no special method at all? The measurement’s assumption:

• LF7 — The object being compared defines no special method; the only thing it carries is what comes from every object’s base. Forms that fail are written with the exception type, forms that do not fail are written with “ran”; values are not printed, because the text the base definition produces carries an environment-dependent identity.

class Bare:
    """Defines no special methods; carries only what comes from the object's base."""


def bare_loop():
    for _ in Bare():
        pass


def bare_context():
    with Bare():
        pass


BARE_FORMS = (
    ("for x in n", bare_loop),
    ("x in n", lambda: 9 in Bare()),
    ("n + m", lambda: Bare() + Bare()),
    ("n[0:2]", lambda: Bare()[0:2]),
    ("len(n)", lambda: len(Bare())),
    ("with n", bare_context),
    ("str(n)", lambda: str(Bare())),
    ("n == m", lambda: Bare() == Bare()),
)

print("the same forms on an object with no special methods:")
failing = 0
for name, function in BARE_FORMS:
    try:
        function()
    except Exception as e:
        failing += 1
        print(f"  {name:12s} {type(e).__name__}")
    else:
        print(f"  {name:12s} ran — used the definition inherited from the base")
print(f"failing forms {failing} / {len(BARE_FORMS)}")

the same forms on an object with no special methods:
  for x in n   TypeError
  x in n       TypeError
  n + m        TypeError
  n[0:2]       TypeError
  len(n)       TypeError
  with n       TypeError
  str(n)       ran — used the definition inherited from the base
  n == m       ran — used the definition inherited from the base
failing forms 6 / 8

6 of eight forms fail, and all with the same exception: TypeError. The name fits — the defect is not in the value used, it is that the object carries no method that answers that syntax. The source text is flawless and passes the compile phase without issue; when it comes to execution, the method to call cannot be found.

The 2 forms that do not fail are the other side of the same rule. str(n) and n == m work because every object inherits these two methods from its base: the first produces a text that identifies the object, the second answers equality by identity. So the protocol is still there; only its definition comes not from the class itself but from its base. When a class writes __str__, it does not add a capability from scratch, it takes the place of the inherited definition.

The result is the course’s fourth reading: in Python, what an object “can do” is determined not by its type’s name but by the special methods it can reach. If __iter__ can be reached, a loop can be built over it; if __len__ can be reached, its length can be asked — there is no requirement linking the two. The language does not look at the type’s name before running a form — it looks for the method, and raises TypeError if it cannot find it.

Summary

• The source text is first turned into tokens and an abstract syntax tree, then into bytecode; what the interpreter runs is the bytecode.
• Compilation happens over the whole file: a syntax defect runs no statement at all, while a name defect only surfaces once its own turn comes, after everything before it has run.
• Syntax is a shorthand; every form calls a specific special method, and the set of methods is called a protocol.
• Eleven forms use 12 distinct protocols with 18 calls in total; for x in n alone produces 5 calls.
• __next__ is called 4 times on a three-item object, and the last one raises StopIteration; here the exception is not a defect but the mechanism that ends the loop.
• On an object defining no special method, 6 of the same eight forms fail with TypeError; the remaining 2 work with definitions inherited from the base object.

Next Step

This lesson counted the calls underneath syntax, but never questioned the syntax itself: what decides where the body under a for line starts and ends? Python does this not with braces but with whitespace, and whitespace here is not a style preference but part of the grammar. The next lesson measures this: how the same sequence of tokens turns into a different tree and a different result just by changing its indentation — and exactly where the boundary between a statement and an expression lies.