---
title: 'Python Fundamentals'
source: 'https://academia.sh/en/courses/python-fundamentals'
language: en
updated: '2026-08-17T04:42:19+00:00'
license: 'CC BY-SA 4.0'
lessons: 16
outcomes:
    - 'Writes programs with the basic syntax and data types'
    - 'Uses the indentation-based block structure and the statement/expression distinction'
    - 'Applies file reading and writing together with exception handling'
---

# Python Fundamentals

In Python, syntax is shorthand; the course counts how many times each form calls which special method, and builds the language's own machinery from the execution model to the exception hierarchy.

## Language Fundamentals

The interpreter and bytecode, indentation's grammatical role, name binding and object identity, the special methods operators call, string formatting, and type conversion's results tied to dynamic typing.

### Python's Execution Model

Source: https://academia.sh/en/courses/python-fundamentals/python-execution-model

The source is compiled as a whole before it is executed; syntax is a shorthand — eleven forms call 12 distinct special methods 18 times in total, and a loop over a three-item object calls `__next__` one more time than the item count, 4 times.

The Software Development Practice curriculum built a team discipline across five courses
and, at its close, said that everything it measured was **language-independent**: what
was counted was never the code itself, but the work around it. This curriculum turns
from the surroundings back to the center. It's the language's own turn.

The Programming Fundamentals course wrote its examples in Python; assignment as a
binding, the `is`/`==` distinction, basic data types, operators, and type conversion were
all built there. But **using** a language as an example and learning the language
**itself** are different things: there, Python was a vehicle for explanation; here, it
is the object being measured. This course looks **beneath** the syntax and carries a
single question across sixteen lessons — which machinery does a written form invoke
while it runs, and how many times?

#### Two Phases: Compiling and Executing

Python runs with an **interpreter**, but it does not read the source line by line and
run it immediately. When a file is run, the text is first split into **tokens**, then
turned into an abstract syntax tree, then translated into **bytecode**. What the
interpreter runs is the bytecode, not the source text.

The boundary between the two phases is observable. Compilation is done over the
**whole** file: a syntax defect on the source's last line also blocks the execution of
the statement on the first line. An error born during execution, by contrast, only
surfaces once its own turn comes, and everything before it has already run. The
difference can be measured with two built-in functions: `compile` compiles the source
and returns a code object, `exec` runs that object.

```python
"""Two phases: the source is first compiled as a whole, then executed."""

SOURCE = {
    "missing parenthesis": 'write = print\nwrite("one")\nwrite(\n',
    "undefined name": 'write = print\nwrite("one")\nwritee("two")\n',
}

for name, source in SOURCE.items():
    print(f"--- {name}")
    try:
        code_obj = compile(source, "<lesson>", "exec")
    except SyntaxError as e:
        print(f"compile: {type(e).__name__} — no statement executed")
        continue
    print(f"compile: clean, obtained a {type(code_obj).__name__} object")
    try:
        exec(code_obj, {})
    except Exception as e:
        print(f"execution: {type(e).__name__} — statements before it were executed")
```

```
--- missing parenthesis
compile: SyntaxError — no statement executed
--- undefined name
compile: clean, obtained a code object
one
execution: NameError — statements before it were executed
```

In the second source, the line `one` was printed, and the call after it fell with a
`NameError`. In the first source, nothing was printed at all — even though its first two
lines are flawless. The distinction is this: **a syntax defect is found at compile time,
a name defect at execution time.** Whether a name exists is not known at compile time,
because a name is only bound to an object during execution.

The same two steps also work in the **interactive shell**; there, the unit compiled is
not the whole file but the block entered. If the block's syntax is broken, the block
never runs at all.

The layout of the bytecode that compilation produces is not part of the language's
definition; it is the interpreter's internal business, and reading it is not this
course's subject. What is worth reading is which methods the bytecode **calls**.

#### Syntax Is a Shorthand

The course's measurement axis comes from a single observation: most forms of Python's
syntax are a **shorthand**. When `n + m` is written, what runs is not an addition but a
**call** — the `n` object's `__add__` method is called, and the returned value is the
expression's value. `for x in n` is not a looping keyword but two methods called in
sequence: first `__iter__`, then `__next__` until it runs out.

The names of these methods are written into the language's definition; each one is
called a **special method**, and the set of methods a form requires is called a
**protocol**. Their names are written between two underscores, and this is meant to set
them apart from ordinary methods: special methods do not exist to be called directly,
they exist to be **called by syntax**.

Not every syntax form is a shorthand, and this distinction has to be set up from the
start. Binding a value to a name, defining a function, or returning a value asks the
object no question at all; these forms are concerned with names and flow, not with what
is inside the object. The ones that are a shorthand are forms that **consume** an
object: they sum it, walk through it, test it in a condition, slice it, turn it into
text. All eleven of the forms measured here belong to this second set.

This has a direct consequence: **syntax does not look at the type's name, it looks at
whether the method exists.** That the `+` operator adds numbers and concatenates strings
is not a special privilege but the fact that two separate types define a method with the
same name in different ways. The Programming Fundamentals course built this observation
as a concept under the heading **operator overloading**; what is measured here is not
the concept but the **name** that carries it out — which form calls which method, and
how many times.

#### The Shared Setup: an Object That Logs Its Participation

A single class is enough for the measurement. `Tracker` defines a large number of
special methods, and every time a method is called, it writes its name to a shared list.
Ordinary syntax is run on it — `for`, `in`, `if`, `+`, `+=`, slicing, `str()`, `with`,
`==` — and which method each form calls, how many times, is read from the list.

A second class, `NonEmpty`, derives from `Tracker` and adds only one thing: `__bool__`.
The same `if` expression will now call a different method on this object. The oracle is
the setup itself: because the object records which method got called, we know it
directly — the count is a log, not an inference.

```python
"""Protocols called by syntax: every special method logs itself when called."""

LOG = []


def record(name):
    LOG.append(name)


class Tracker:
    """An object that takes part in many protocols; records each participation."""

    def __init__(self, items=(1, 2, 3)):
        self.items = list(items)

    def __iter__(self):
        record("__iter__")
        self._i = 0
        return self

    def __next__(self):
        record("__next__")
        if self._i >= len(self.items):
            raise StopIteration
        value = self.items[self._i]
        self._i += 1
        return value

    def __len__(self):
        record("__len__")
        return len(self.items)

    def __contains__(self, x):
        record("__contains__")
        return x in self.items

    def __add__(self, o):
        record("__add__")
        return Tracker(self.items + list(o.items))

    def __iadd__(self, o):
        record("__iadd__")
        self.items.extend(o.items)
        return self

    def __getitem__(self, k):
        record("__getitem__")
        return self.items[k]

    def __eq__(self, o):
        record("__eq__")
        return isinstance(o, Tracker) and self.items == o.items

    def __str__(self):
        record("__str__")
        return f"Tracker{tuple(self.items)}"

    def __repr__(self):
        record("__repr__")
        return self.__str__()

    def __enter__(self):
        record("__enter__")
        return self

    def __exit__(self, kind, value, tb):
        record("__exit__")
        return False          # does not swallow the exception


class NonEmpty(Tracker):
    """__bool__ defined: the truthiness test no longer falls back to __len__."""

    def __bool__(self):
        record("__bool__")
        return len(self.items) > 0


def measure(function):
    """Runs a syntax form and returns the protocols it called."""
    LOG.clear()
    try:
        function()
    except Exception as e:
        LOG.append(f"!{type(e).__name__}")
    return list(LOG)


def loop():
    for _ in Tracker():
        pass


def membership():
    9 in Tracker()


def truthiness_len():
    if Tracker():
        pass


def truthiness_bool():
    if NonEmpty():
        pass


def addition():
    Tracker() + Tracker()


def in_place_addition():
    a = Tracker()
    a += Tracker()


def slicing():
    Tracker()[0:2]


def to_string():
    str(Tracker())


def context():
    with Tracker():
        pass


def context_with_exception():
    with Tracker():
        raise ValueError("example")


def equality():
    Tracker() == Tracker()


FORMS = (
    ("for x in n", loop),
    ("x in n", membership),
    ("if n  (__len__)", truthiness_len),
    ("if n  (__bool__)", truthiness_bool),
    ("n + m", addition),
    ("n += m", in_place_addition),
    ("n[0:2]", slicing),
    ("str(n)", to_string),
    ("with n", context),
    ("with n + exception", context_with_exception),
    ("n == m", equality),
)
```

The measurement's assumptions:

- **LF1** — The forms measured are eleven; not the language's entire syntax, but the
  core this course pays off lesson by lesson. The remaining forms are measured in their
  own lessons.
- **LF2** — The oracle is the object's own log: it writes its name to the list when a
  method is called, so the measurement is a count, not an inference.
- **LF3** — `Tracker` is built with three items; the call count read in the loop
  measurement depends on this three.
- **LF4** — `__exit__` returns `False` on every run; an exception born inside the
  context block is not swallowed, and it enters the measurement as an exception name
  with an exclamation mark in front of it.
- **LF5** — The measurement reads no environment data: memory address, object identity
  count, and duration are not used; the only thing counted is the names of the methods
  called.
- **LF6** — Every form is measured twice — once for the table, once for the paired
  comparison. Because `Tracker` is rebuilt on every measurement, the two runs give the
  same sequence.

#### Measurement

```python
print(f"{'syntax':<20s} {'calls':>6s}  protocol sequence")
for name, function in FORMS:
    c = measure(function)
    print(f"{name:<20s} {len(c):6d}  {' '.join(c)}")

print()
by_form = {name: measure(f) for name, f in FORMS}
unique = sorted({p for c in by_form.values() for p in c if not p.startswith("!")})
print(f"distinct protocols {len(unique)}: {', '.join(unique)}")
print(f"total calls {sum(len(c) for c in by_form.values())}, "
      f"forms ending in an exception "
      f"{sum(1 for c in by_form.values() if any(p.startswith('!') for p in c))}")

print()
print("pairs that look the same but call different protocols:")
for a, b in (("n + m", "n += m"),
             ("if n  (__len__)", "if n  (__bool__)")):
    print(f"  {a:<18s} -> {' '.join(by_form[a])}")
    print(f"  {b:<18s} -> {' '.join(by_form[b])}")

print()
loop_log = by_form["for x in n"]
print(f"loop over a three-item object: __next__ called {loop_log.count('__next__')} times "
      f"— one more than the item count; the last one ended with StopIteration")
```

```
syntax                calls  protocol sequence
for x in n                5  __iter__ __next__ __next__ __next__ __next__
x in n                    1  __contains__
if n  (__len__)           1  __len__
if n  (__bool__)          1  __bool__
n + m                     1  __add__
n += m                    1  __iadd__
n[0:2]                    1  __getitem__
str(n)                    1  __str__
with n                    2  __enter__ __exit__
with n + exception        3  __enter__ __exit__ !ValueError
n == m                    1  __eq__

distinct protocols 12: __add__, __bool__, __contains__, __enter__, __eq__, __exit__, __getitem__, __iadd__, __iter__, __len__, __next__, __str__
total calls 18, forms ending in an exception 1

pairs that look the same but call different protocols:
  n + m              -> __add__
  n += m             -> __iadd__
  if n  (__len__)    -> __len__
  if n  (__bool__)   -> __bool__

loop over a three-item object: __next__ called 4 times — one more than the item count; the last one ended with StopIteration
```

#### Reading the Numbers

Eleven forms produce **12 distinct protocols** and **18 calls** in total. This is the
measure the course will carry forward: a lesson's number is which protocol the syntax it
covers calls, and how many times. Every row of the table is a syntax form, and the
sequence next to it shows the calls that really ran underneath that form. None of it is
a guess; all of it was read from the object's own log.

**Two expressions that look alike call different protocols.** `n + m` and `n += m` sit
side by side as if one were shorthand for the other, but one calls `__add__`, the other
`__iadd__`. The difference is not just a name: `__add__` produces and returns **a new
object**, `__iadd__` **mutates the existing object** and returns itself. In the same way,
`if n` calls `__len__` on `Tracker`, `__bool__` on `NonEmpty` — a single syntax, two
separate methods. What creates the difference is which method the object defines.
**The look of the syntax does not tell you which protocol gets called.** The first half
of this claim will be paid off in this topic's operators lesson, the second half in the
flow topic's conditionals lesson.

**An exception is not an error, it is part of the protocol.** The loop over the
three-item object called `__next__` **4 times** — one more than the item count. The
fourth call does not return a value, it raises `StopIteration`; this is exactly what
ends the loop. The word `for` does not know the items ran out by counting, it knows by
catching an exception. The line `with n + exception` is of the same family: even though
an exception is born inside the block, `__exit__` is **still called**, and its name
stands in the log as the third entry. Because `__exit__` returns `False`, the exception
made it out — had it returned `True`, it would not have.

#### No Protocol, No Syntax

The table above is full **because** `Tracker` has these methods. What happens if the
same forms are tried on an object that defines no special method at all? The
measurement's assumption:

- **LF7** — The object being compared defines no special method; the only thing it
  carries is what comes from every object's base. Forms that fail are written with
  the exception type, forms that do not fail are written with "ran"; **values are not
  printed**, because the text the base definition produces carries an
  environment-dependent identity.

```python
class Bare:
    """Defines no special methods; carries only what comes from the object's base."""


def bare_loop():
    for _ in Bare():
        pass


def bare_context():
    with Bare():
        pass


BARE_FORMS = (
    ("for x in n", bare_loop),
    ("x in n", lambda: 9 in Bare()),
    ("n + m", lambda: Bare() + Bare()),
    ("n[0:2]", lambda: Bare()[0:2]),
    ("len(n)", lambda: len(Bare())),
    ("with n", bare_context),
    ("str(n)", lambda: str(Bare())),
    ("n == m", lambda: Bare() == Bare()),
)

print("the same forms on an object with no special methods:")
failing = 0
for name, function in BARE_FORMS:
    try:
        function()
    except Exception as e:
        failing += 1
        print(f"  {name:12s} {type(e).__name__}")
    else:
        print(f"  {name:12s} ran — used the definition inherited from the base")
print(f"failing forms {failing} / {len(BARE_FORMS)}")
```

```
the same forms on an object with no special methods:
  for x in n   TypeError
  x in n       TypeError
  n + m        TypeError
  n[0:2]       TypeError
  len(n)       TypeError
  with n       TypeError
  str(n)       ran — used the definition inherited from the base
  n == m       ran — used the definition inherited from the base
failing forms 6 / 8
```

**6** of eight forms fail, and all with the same exception: `TypeError`. The name fits —
the defect is not in the value used, it is that the object **carries no method that
answers that syntax**. The source text is flawless and passes the compile phase without
issue; when it comes to execution, the method to call cannot be found.

The **2** forms that do not fail are the other side of the same rule. `str(n)` and
`n == m` work because every object inherits these two methods from its **base**: the
first produces a text that identifies the object, the second answers equality by
identity. So the protocol is still there; only its definition comes not from the class
itself but from its base. When a class writes `__str__`, it does not add a capability
from scratch, it **takes the place of** the inherited definition.

The result is the course's fourth reading: in Python, what an object "can do" is
determined not by its type's name but by **the special methods it can reach**. If
`__iter__` can be reached, a loop can be built over it; if `__len__` can be reached, its
length can be asked — there is no requirement linking the two. The language does not
look at the type's name before running a form — it looks for the method, and raises
`TypeError` if it cannot find it.

#### Summary

- The source text is first turned into tokens and an abstract syntax tree, then into
  bytecode; what the interpreter runs is the bytecode.
- Compilation happens over the whole file: a syntax defect runs no statement at all,
  while a name defect only surfaces once its own turn comes, after everything before it
  has run.
- Syntax is a shorthand; every form calls a specific special method, and the set of
  methods is called a protocol.
- Eleven forms use **12 distinct protocols** with **18 calls** in total; `for x in n`
  alone produces 5 calls.
- `__next__` is called **4 times** on a three-item object, and the last one raises
  `StopIteration`; here the exception is not a defect but the mechanism that ends the
  loop.
- On an object defining no special method, **6** of the same eight forms fail with
  `TypeError`; the remaining **2** work with definitions inherited from the base object.

#### Next Step

This lesson counted the calls underneath syntax, but never questioned the syntax itself:
what decides where the body under a `for` line starts and ends? Python does this not
with braces but with **whitespace**, and whitespace here is not a style preference but
part of the grammar. The next lesson measures this: how the same sequence of tokens
turns into a different tree and a different result just by changing its indentation —
and exactly where the boundary between a statement and an expression lies.

### Syntax and Indentation

Source: https://academia.sh/en/courses/python-fundamentals/syntax-and-indentation

Indentation is not a style preference but part of the grammar: the same 21 content tokens produce two different trees and two different results under two different indentations, four different widths give a single tree, and only 5 of ten parts are expressions.

The previous lesson showed that the source is compiled as a whole before anything runs,
and said that the first step of compilation is splitting the text into **tokens**. That
step left a question open: what happens to the whitespace at the start of a line?

In most languages that build a block with braces, nothing — the whitespace is discarded,
braces say where the body starts and ends, and indentation is there only for the reader.
In Python, the whitespace is not discarded. **Indentation** adds its own tokens to the
token sequence, and it is exactly those tokens that build the block structure. The
observable consequence is this: the same token sequence turns into a different tree and
a different result just by changing its indentation.

#### Whitespace Builds the Block, Not Braces

The tokenizing step measures the whitespace at the start of every line and keeps it on a
stack. If a line sits deeper than the one before it, an `INDENT` token is placed in the
stream; if it sits shallower, a `DEDENT` token is placed for every level that closes. If
several levels close at once, that many `DEDENT` tokens are produced, so the number of
`DEDENT` tokens in the sequence gives the number of blocks that closed. These tokens have
no character in the source text that corresponds to them — they are produced from the
whitespace itself.

The two sources below are indistinguishable if you only look at the tokens' names. Only
one line's indentation differs: the `print` call sits outside the loop body in one,
inside it in the other.

The measurement's assumptions:

- **LF8** — The two sources differ only in the last line's indentation; no character was
  added, removed, or changed.
- **LF9** — The comparison is made through the tokens' **names**; `INDENT`, `DEDENT`,
  `NEWLINE`, `NL`, and the end-of-file token are counted as "layout tokens" and set
  aside, the rest count as "content tokens."
- **LF10** — The tree is written with the class names of the statements at the parsed
  structure's top level; the loop statement's body is given in square brackets. What is
  read is not the tree's full dump but the shell that is enough for this measurement.
- **LF11** — Execution output is captured into a separate buffer and written as a list of
  printed lines; no environment-dependent value is read.

```python
"""Same token sequence, two indentations: three views."""
import ast
import contextlib
import io
import tokenize

OUTSIDE = "total = 0\nfor x in (1, 2, 3):\n    total += x\nprint(total)\n"
INSIDE = "total = 0\nfor x in (1, 2, 3):\n    total += x\n    print(total)\n"
SOURCE = {"outside the body": OUTSIDE, "inside the body": INSIDE}
LAYOUT = {"INDENT", "DEDENT", "NEWLINE", "NL", "ENDMARKER"}


def tokens(source):
    reader = io.StringIO(source).readline
    return [tokenize.tok_name[t.type] for t in tokenize.generate_tokens(reader)]


def tree(source):
    parts = []
    for d in ast.parse(source).body:
        name = type(d).__name__
        if isinstance(d, ast.For):
            name += "[" + ",".join(type(g).__name__ for g in d.body) + "]"
        parts.append(name)
    return " ".join(parts)


def run(source):
    buffer = io.StringIO()
    with contextlib.redirect_stdout(buffer):
        exec(compile(source, "<lesson>", "exec"), {})
    return buffer.getvalue().split()


for name, source in SOURCE.items():
    all_tokens = tokens(source)
    content = [t for t in all_tokens if t not in LAYOUT]
    print(f"{name:17s} tokens {len(all_tokens)}  non-layout {len(content)}  "
          f"INDENT/DEDENT {[i for i, t in enumerate(all_tokens) if t in ('INDENT', 'DEDENT')]}")

a, b = (tokens(k) for k in SOURCE.values())
print(f"equal once layout tokens are dropped: "
      f"{[x for x in a if x not in LAYOUT] == [x for x in b if x not in LAYOUT]}")
print()
for name, source in SOURCE.items():
    print(f"{name:17s} tree: {tree(source)}")
    print(f"{'':17s} printed: {run(source)}")
```

```
outside the body  tokens 28  non-layout 21  INDENT/DEDENT [16, 21]
inside the body   tokens 28  non-layout 21  INDENT/DEDENT [16, 26]
equal once layout tokens are dropped: True

outside the body  tree: Assign For[AugAssign] Expr
                  printed: ['6']
inside the body   tree: Assign For[AugAssign,Expr]
                  printed: ['1', '3', '6']
```

Both sources produce **28** tokens, and **21** of them are content tokens. Once layout
tokens are dropped, the two remaining sequences are **exactly equal** — same names, same
order. The difference sits at a single token's position: `DEDENT` stands at position
**21** in one, position **26** in the other. The only thing that changed in the source is
four spaces, and those four spaces moved one token five positions over.

That single difference changes the tree. In the first source, the call is the loop's
**sibling**: in the tree, `For[AugAssign]` finishes and a separate `Expr` follows. In the
second, the call is the loop's **child**: `For[AugAssign,Expr]`. The result splits
accordingly — one prints a single line, the other prints three.

The rule that follows from this closes one of the most expensive misunderstandings when
approaching the language: **breaking indentation is not a style defect, it is changing
the program.** In a brace-based language, these same two sources would be formatted
versions of each other and both would do the same thing. Here indentation is the sole
determinant; no other marker states the body's boundary.

#### Width Is Free, Consistency Is Mandatory

Indentation being grammatical does not mean a particular **width** is grammatical. The
language does not impose a number; what it imposes is that a block be internally
consistent. The boundary is observable: the same structure can be written with four
different widths and their trees compared, then four inconsistent sources can be tried
for compilation.

- **LF12** — In valid sources, width changes but stays consistent within a block; in
  broken sources, consistency breaks in four different ways. The sources are compiled,
  **not executed** — what is measured is whether the compile phase accepts them.

```python
"""The rule of indentation: width is free, consistency is mandatory."""
import ast

WIDTH = {"two spaces": "if 1:\n  a = 1\n  b = 2\n",
         "four spaces": "if 1:\n    a = 1\n    b = 2\n",
         "eight spaces": "if 1:\n        a = 1\n        b = 2\n",
         "tab": "if 1:\n\ta = 1\n\tb = 2\n"}

BROKEN = {"no indentation at all": "for x in (1, 2):\nb = 1\n",
          "unexpected indent": "a = 0\n    b = 1\n",
          "misaligned dedent": "if 1:\n        a = 1\n    b = 2\n",
          "tab mixed with spaces": "if 1:\n\ta = 1\n        b = 2\n"}


def tree(source):
    d = ast.parse(source).body[0]
    return f"{type(d).__name__}[{','.join(type(g).__name__ for g in d.body)}]"


print("same structure, different width:")
for name, source in WIDTH.items():
    print(f"  {name:12s} tree: {tree(source)}")
print(f"  are all four trees the same: "
      f"{len({tree(k) for k in WIDTH.values()}) == 1}")

print("\ninconsistent indentation:")
for name, source in BROKEN.items():
    try:
        compile(source, "<lesson>", "exec")
        print(f"  {name:24s} compiled")
    except SyntaxError as e:
        print(f"  {name:24s} {type(e).__name__}")
print(f"  is IndentationError a SyntaxError: "
      f"{issubclass(IndentationError, SyntaxError)}")
print(f"  is TabError an IndentationError: "
      f"{issubclass(TabError, IndentationError)}")
```

```
same structure, different width:
  two spaces   tree: If[Assign,Assign]
  four spaces  tree: If[Assign,Assign]
  eight spaces tree: If[Assign,Assign]
  tab          tree: If[Assign,Assign]
  are all four trees the same: True

inconsistent indentation:
  no indentation at all    IndentationError
  unexpected indent        IndentationError
  misaligned dedent        IndentationError
  tab mixed with spaces    TabError
  is IndentationError a SyntaxError: True
  is TabError an IndentationError: True
```

Four widths give a single tree. What is grammatical is not the **amount** of
indentation but its **structure**: a level opens and continues at the same alignment,
and when it closes it lands on an alignment that was already opened. A program written
with two spaces and one written with eight spaces are the same program; the choice
between them is a matter of team convention, not the language.

When consistency breaks, compilation stops in all four cases. Three give
`IndentationError`: a block whose body is never opened, a line that slides inward
without opening anything, and a dedent that lands on no previously opened alignment.
The fourth stands apart — a source mixing tabs and spaces gives `TabError`, because the
width of these two characters cannot be resolved just by looking at the source. The
output's last two lines show all three come from a single lineage: `TabError` is an
`IndentationError`, which is in turn a `SyntaxError`.

This lineage also settles the previous lesson's boundary. Indentation defects are found
at **compile time**; none of them make it to execution. A broken indentation on the
file's last line keeps even the assignment on the first line from running.

#### Logical Line vs. Physical Line

Indentation being grammatical stops somewhere, and that limit is worth knowing. What
builds the block structure is the whitespace at the start of a **logical line**; not
every line start on screen begins a logical line. While an expression sits inside an
open bracket, a line ending does not finish the statement — an `NL` is placed in the
stream instead of a `NEWLINE`, and **the indentation of those lines produces no token at
all**. The same continuation can also be built with a backslash.

- **LF13** — Four forms write the same sum and differ only in how they split across
  lines; what is measured is the physical line count, the logical line ending
  (`NEWLINE`), the continuation line ending (`NL`), and the `INDENT` count. The value
  column is the sum the source itself produces.

```python
"""Logical line vs physical line: indentation is not grammatical inside parens."""
import ast
import io
import tokenize

FORM = {
    "single line": "total = 1 + 2 + 3\n",
    "aligned inside parens": "total = (1 +\n          2 +\n          3)\n",
    "ragged inside parens": "total = (1 +\n2 +\n              3)\n",
    "with backslash": "total = 1 + \\\n    2 + \\\n    3\n",
}


def count(source):
    reader = io.StringIO(source).readline
    names = [tokenize.tok_name[t.type] for t in tokenize.generate_tokens(reader)]
    return names.count("NEWLINE"), names.count("NL"), names.count("INDENT")


print(f"{'form':<22s} {'physical':>8s} {'NEWLINE':>7s} {'NL':>3s} {'INDENT':>6s}"
      f"  tree     value")
for name, source in FORM.items():
    logical, nl, indent = count(source)
    env = {}
    exec(compile(source, "<lesson>", "exec"), env)
    tree = " ".join(type(d).__name__ for d in ast.parse(source).body)
    print(f"{name:<22s} {len(source.splitlines()):8d} {logical:7d} {nl:3d} {indent:6d}"
          f"  {tree:8s} {env['total']}")
```

```
form                   physical NEWLINE  NL INDENT  tree     value
single line                   1       1   0      0  Assign   6
aligned inside parens         3       1   2      0  Assign   6
ragged inside parens          3       1   2      0  Assign   6
with backslash                3       1   0      0  Assign   6
```

All four forms give a single `Assign` node and a single value. Even though three of
them spread across **3** physical lines, the `NEWLINE` count is **1** in every one:
there is exactly one logical line. In the two forms inside parentheses, the line
endings in between pass through as `NL`, meaning "no statement ended here." In the
backslash form, the line ending does not turn into a token at all; the continuation is
resolved before the text is even split into tokens.

The most telling column is `INDENT`: **0** in all four. The "ragged" form's second line
starts at column zero, its third starts deep in, and neither changes anything. So
indentation stops being grammatical **inside brackets**; the alignment done there is
entirely for the reader, and inconsistency there produces no defect. The rule can be
narrowed like this: indentation only carries meaning at the start of a logical line.

#### Statement and Expression

Indentation builds a block; a block, in turn, is made of **statements**. This requires
the language's second fundamental distinction. An **expression** produces a value: it is
computed and leaves an object behind. A **statement** does a job: it binds a name,
enters a block, directs the flow; it leaves no value behind.

The distinction does not have to be left to intuition, because the compile mode states
it directly. `eval` mode accepts only an expression; `exec` mode accepts a sequence of
statements.

- **LF14** — Every part is attempted in both modes; what is measured is whether it is
  accepted. Parts whose value is printed are evaluated in an environment where a single
  name is bound, so the value column depends on that binding, not on the environment.

```python
"""Statement or expression: 'eval' mode only compiles an expression, 'exec' compiles both."""

PARTS = (
    "2 + 3",
    "total",
    "len('value')",
    "[k for k in (1, 2, 3)]",
    "total if total else 0",
    "total = 5",
    "total += 1",
    "if total: pass",
    "for k in (1, 2): pass",
    "pass",
)

print(f"{'part':<26s} {'expr':>5s} {'stmt':>5s}  value")
expr_count = stmt_count = 0
for p in PARTS:
    try:
        code_obj = compile(p, "<lesson>", "eval")
        is_expr = True
    except SyntaxError:
        is_expr = False
    try:
        compile(p, "<lesson>", "exec")
        is_stmt = True
    except SyntaxError:
        is_stmt = False
    expr_count += is_expr
    stmt_count += is_stmt
    value = repr(eval(code_obj, {"total": 3})) if is_expr else "—"
    print(f"{p:<26s} {str(is_expr):>5s} {str(is_stmt):>5s}  {value}")

print(f"\n{len(PARTS)} parts: expression {expr_count}, statement {stmt_count}; "
      f"statement-only {stmt_count - expr_count}")
```

```
part                        expr  stmt  value
2 + 3                       True  True  5
total                       True  True  3
len('value')                True  True  5
[k for k in (1, 2, 3)]      True  True  [1, 2, 3]
total if total else 0       True  True  3
total = 5                  False  True  —
total += 1                 False  True  —
if total: pass             False  True  —
for k in (1, 2): pass      False  True  —
pass                       False  True  —

10 parts: expression 5, statement 10; statement-only 5
```

**5** of ten parts are expressions, **10** are statements. The statement column being
entirely filled gives the direction of the distinction: **every expression is also a
statement** — placed alone on a line, its value is computed and discarded. The reverse
does not hold; the **5** statement-only parts cannot be placed anywhere an expression
could stand.

The distinction has a direct consequence for the course. All eleven forms the previous
lesson measured **consumed** an object, and what they consumed was an expression's
value; special methods operate on expressions. This is also why it matters that the row
`total += 1` does not come out as an expression: the Programming Fundamentals course
said that a compound assignment is a statement; what is measured here is **which
method** that statement calls and what it does with the returned value. The result of
the `__iadd__` call does not sit around as a value, it is bound straight back to the
name — the detail is paid off in the operators lesson.

One last observation hides in the table: `if total: pass`, even though written on a
single line, is a valid statement and uses no indentation at all. Indentation is not
the **only** way to write a block; what makes a block mandatory is the statement itself,
and indentation is the way to write a body with more than one statement.

#### Summary

- Indentation is not discarded; the whitespace at the start of a line is turned into
  `INDENT` and `DEDENT` tokens, and it is these tokens that build the block structure.
- Two sources differing only in indentation carry the same **21** content tokens in the
  same order; the position of a single `DEDENT` token changes the tree and the result.
- Indentation's width is free — four different widths give a single tree — but it must
  be consistent within a block.
- Inconsistent indentation stops at compile time; `TabError` is an `IndentationError`,
  which is in turn a `SyntaxError`.
- Indentation is only grammatical at the start of a logical line: **3** physical lines
  inside an open bracket produce a single `NEWLINE`, and the `INDENT` count stays **0**.
- **5** of ten parts are expressions, **10** are statements: every expression can be
  written as a statement, no statement can stand in for an expression.

#### Next Step

In this lesson, assignment — the most commonly written statement — showed up only as
"not an expression" in the table; what it does was never examined. The Programming
Fundamentals course established that assignment is a binding — a name is not a box, it
is a label attached to an object. The next lesson measures this bond from the object's
side: what does it mean when two names are bound to the same object, under what
condition do two equal values turn out to be the same object, and how does a type being
"immutable" affect binding?

### Variables and Data Types

Source: https://academia.sh/en/courses/python-fundamentals/variables-and-data-types

A name carries no type, the object does: a single name binds to seven different types without a declaration, the `+=` notation rebinds the name in three of four types and mutates the object in one, and whether two equal values turn out to be the same object depends on the implementation.

The previous lesson showed that indentation carries blocks and blocks carry statements,
and assignment — the most commonly written statement — showed up in the table only as
"not an expression." This lesson looks at what that statement does.

The Programming Fundamentals course already established that assignment is a
**binding**, that a variable can be modeled either as a named memory cell or as a label
attached to an object, the distinction between `is` and `==`, and aliasing. **None of
that is repeated.** What was built there was a concept: a language picks one of two
models. What is measured here is the **observable consequences** of the model Python
picked — that a name carries no type at all, whether `+=` rebinds the name or mutates
the object depending on the type, and what determines whether two equal values turn out
to be the same object.

#### A Name Carries No Type

In Python, every value is an **object**, and type is a property of the object. A name is
not a declaration, only a bond; it neither declares a type nor locks onto one. This is
why the `type` built-in is asked of the object, not the name.

- **LF15** — A value of each of seven distinct types is bound to a single name in turn;
  the source carries no type declaration at all. What is measured is the type name the
  object reports after every binding.
- **LF16** — The last binding is to a function; because a function is itself a bindable
  object, it shows up in the same table.

```python
"""A name is a label: the object carries the type, the name carries none."""

VALUES = (12, 3.5, "value", (1, 2), [1, 2], {"north": 1}, None)


def length(text):
    return len(text)


measurement = None
types = []
for value in VALUES:
    measurement = value                      # single name, seven objects in sequence
    types.append(type(measurement).__name__)

print("types bound to a single name in sequence:", " ".join(types))
print("binding count", len(VALUES), "— none of them declares a type")
print("the name's type after the last binding:", type(measurement).__name__)
measurement = length                        # same name, now a function
print("the same name can also be bound to a function:",
      type(measurement).__name__, "->", measurement("value"))
```

```
types bound to a single name in sequence: int float str tuple list dict NoneType
binding count 7 — none of them declares a type
the name's type after the last binding: NoneType
the same name can also be bound to a function: function -> 5
```

A single name was bound to **7** distinct types, and to a function on the eighth. No
declaration was written at any binding, because there is nowhere to write one:
`measurement` is not a container, it is a name pointing at whatever object it is
currently attached to. There is no operation that asks the name's type either; a `type`
call goes to the object the name **is bound to**, and that object gives the answer.

The last line closes one more class of case. A function is an object too; defining one
is nothing more than binding a name to an object. That is why a function can be placed
in a name, a list, or a dictionary, and called from there.

Python has a notation that **looks like** a declaration and is prone to confusion: a
type name can be written next to a parameter or a return value. This notation is not a
declaration, it is an **annotation**; the interpreter stores it but does not enforce it.

```python
"""An annotation is not a declaration: the interpreter does not enforce what is written."""


def double(n: int) -> int:
    return n * 2


print("integer:", double(12))
print("string: ", repr(double("ab")))
print("list:   ", double([1, 2]))
print("where the annotations are stored:", list(double.__annotations__))
```

```
integer: 24
string:  'abab'
list:    [1, 2, 1, 2]
where the annotations are stored: ['n', 'return']
```

Even though the function's annotation says it expects an integer, it worked with a
string and a list too, with no warning at all. The written annotations are not lost
either; they sit on the function under two names. So the annotation is written not for
execution but for **the reader and outside checkers that run separately**. Looking at an
annotation and assuming "the type is checked here" is one of the most expensive
misreadings of the language.

This is called **dynamic typing**, and it needs to be read correctly: it does not mean
there is no type, it means type checking is **deferred to execution time**. This is the
same limit as in the first lesson. Whether a name exists could not be known at compile
time; whether an object supports an operation cannot be known either, because the object
only comes into being while running. The exception that appears when a defect is found
is also the same one measured in the first lesson — `TypeError` if the special method
sought does not exist.

#### Built-in Types and the Protocol They Carry

Type belongs to the object; so what is it that tells types apart? The Programming
Fundamentals course built the basic data types as a classification — number, text,
boolean, collection. **That classification is not repeated.** What is asked here is
different, and it grows out of the first lesson's measure: a built-in type announces
which syntax forms it can support through **the special methods it defines**. The
type's name says nothing; the method list says everything.

- **LF17** — Each type is represented by a single example, and the probe is made not on
  the example itself but on its **type**; what is asked is whether the type carries that
  method.
- **LF18** — The "mutable" column is a known trait of the type and is not measured; only
  the method columns in the table are measurements.

```python
"""What built-in types can do: look at the special methods they carry."""

EXAMPLES = (("int", 12, False), ("float", 3.5, False), ("bool", True, False),
            ("str", "value", False), ("tuple", (1, 2), False),
            ("list", [1, 2], True), ("dict", {"north": 1}, True),
            ("set", {1, 2}, True), ("NoneType", None, False))
PROTOCOL = ("__len__", "__iter__", "__contains__", "__getitem__", "__add__")

print(f"{'type':<9s} {'mutable':>11s}  "
      + " ".join(f"{p.strip('_'):>10s}" for p in PROTOCOL))
for name, example, mutable in EXAMPLES:
    supported = " ".join(f"{'yes' if hasattr(type(example), p) else '—':>10s}"
                        for p in PROTOCOL)
    print(f"{name:<9s} {str(mutable):>11s}  {supported}")

print()
print("is bool an int:", isinstance(True, int), "| True + True =", True + True)
big = 2 ** 100
print("an integer's width is not fixed:", big.bit_length(), "bits;",
      "its square", (big * big).bit_length(), "bits")
```

```
type          mutable         len       iter   contains    getitem        add
int             False           —          —          —          —        yes
float           False           —          —          —          —        yes
bool            False           —          —          —          —        yes
str             False         yes        yes        yes        yes        yes
tuple           False         yes        yes        yes        yes        yes
list             True         yes        yes        yes        yes        yes
dict             True         yes        yes        yes        yes          —
set              True         yes        yes        yes          —          —
NoneType        False           —          —          —          —          —

is bool an int: True | True + True = 2
an integer's width is not fixed: 101 bits; its square 201 bits
```

The table groups types not by name but by capability. The three numeric types carry
only `__add__`: they can be added, but no loop can be built over them, no length can be
asked. `str`, `tuple`, and `list` fill all five columns; the only difference between
them is not in the measured columns but in the mutability column. `dict` and `set`
carry the top three protocols but not `__add__` — the reason two dictionaries cannot be
merged with `+` is not a prohibition, it is that the method is **not defined**.
The `NoneType` row is empty: none of these forms works on `None`, and it gives the same
result as the bare object in the first lesson.

The single gap in the `set` row is instructive too. A set carries `__contains__` but not
`__getitem__`: you can ask whether an item is in it, you cannot ask which position it
sits at. This is the code-level counterpart of a set being "unordered."

The bottom two rows close two Python-specific points. `bool` is a subtype of `int` —
which is why `True + True` is not a `TypeError`, it gives **2**. And an integer has no
fixed width: a hundred-bit value stops at **101** bits, and its square grows to **201**
bits. The How Computers Work course established the overflow behavior of a fixed-width
type; the difference measured here is that Python's integer has **no such limit**.
There is no such thing as not fitting, only a representation that grows.

#### The Binding's Identity

More than one name can be bound to the same object, and the `is` operator asks exactly
that. This lesson **does not print identity numbers**: the number depends on the
environment, changes from run to run, and no program's correctness should depend on it.
The question to ask is not "which object" but "the same object?" — and only `is`
answers that.

The Python-specific version of this question shows up in compound assignment. `x +=
extra` is a single notation, but it has two different outcomes: it either rebinds the
name to a new object, or mutates the object it is bound to in place. Which one happens
can be measured by attaching a second name to the object.

- **LF19** — In every measurement, a value is bound to a name, a second name is
  attached to that same object, and `+=` is run only on the first name. `is` is then
  used to check whether the first name still points at the same object.
- **LF20** — Four types are chosen so that three are immutable and one is mutable; the
  result column reflects this trait of the type, not the particular values chosen.

```python
"""Same notation, two different results: was the binding rebuilt, or the object mutated?"""


def mutated_in_place(value, extra):
    x = value
    witness = x                  # second name attached to the same object
    x += extra
    return x is witness, witness


print(f"{'type':<9s} {'after +=':<22s} what the second name sees")
for name, value, extra in (("integer", 10, 1),
                            ("string", "ab", "cd"),
                            ("tuple", (1, 2), (3,)),
                            ("list", [1, 2], [3])):
    same, witness = mutated_in_place(value, extra)
    print(f"{name:<9s} {'object mutated' if same else 'name rebound':<22s} {witness!r}")
```

```
type      after +=               what the second name sees
integer   name rebound           10
string    name rebound           'ab'
tuple     name rebound           (1, 2)
list      object mutated         [1, 2, 3]
```

In **three** of four types the name was rebound, in **one** the object was mutated.
What creates the difference is whether the type is **mutable**. Integer, string, and
tuple are **immutable** types: once built, an object's content cannot be changed by any
route. `+=` cannot perform an in-place operation on such an object; it produces a new
object and binds **the name** to it, and the second name stays on the old object. The
right-hand column shows this directly — in three rows, the second name still sees the
starting value.

For the list, the result flips. The object itself grows; the name is not touched,
because there is no need to touch it. The second name, being bound to the same object,
sees the grown version too. The same line meaning two different things for two types is
not an inconsistency, it is one rule with two outcomes: **an in-place operation cannot
be defined on an immutable object.**

This is the first sign of the course's second claim. The notations `x = x + extra` and
`x += extra` do not give the same result here, and the difference is not accidental:
the two notations call **two different special methods**. Their names are set in the
next lesson.

#### The Interned Object

One question remains: are two equal values the same object? The answer carries far less
guarantee than the question suggests.

- **LF21** — The compared values are produced **at run time**, inside a function call;
  this way, two constants sitting side by side in the source being merged at compile
  time does not interfere with the measurement.
- **LF22** — The "guarantee" column says what the result rests on: the `language
  definition` row is the same in every interpreter, the `implementation` row is
  **subject to change**.
- **LF23** — Identity is only asked with `is`; no identity number is printed.

```python
"""Interned object: the result is IMPLEMENTATION-DEPENDENT."""


def produce(a, b):
    return a + b                 # the value is produced at run time


def nothing():
    return None


def true_value():
    return 1 == 1


PAIRS = (
    ("small integer", produce(2, 3), produce(1, 4)),
    ("large integer", produce(500, 1), produce(499, 2)),
    ("short string", produce("va", "lue"), produce("val", "ue")),
    ("list", produce([1], [2]), produce([1], [2])),
)

print(f"{'pair':<16s} {'==':>6s} {'is':>6s}   guarantee")
for name, x, y in PAIRS:
    guarantee = "language definition" if isinstance(x, list) else "implementation"
    print(f"{name:<16s} {str(x == y):>6s} {str(x is y):>6s}   {guarantee}")

print("\nsingleton objects the language definition guarantees:")
print("  is a None returned from elsewhere the same object:", nothing() is None)
print("  is a boolean returned from elsewhere the same object:", true_value() is True)
```

```
pair                 ==     is   guarantee
small integer      True   True   implementation
large integer      True  False   implementation
short string       True  False   implementation
list               True  False   language definition

singleton objects the language definition guarantees:
  is a None returned from elsewhere the same object: True
  is a boolean returned from elsewhere the same object: True
```

`==` gives **True** for all four pairs: the values are equal. The `is` column is where
it scatters. For the small integer, two separate computations gave the same object; for
the large integer and the short string, they did not.

These three rows are **implementation-dependent and have to be read that way.** An
interpreter can build certain frequently used immutable objects ahead of time and hand
out the same one every time it is requested; this is called **interning**. Which values
fall under this is not written in the language's definition. Getting `True` on all three
of these rows in a different interpreter — or even a different run of the same
interpreter — or `False` on all three, breaks nothing in the language's definition. The
table **describes** a behavior, it does not **establish** a rule.

The fourth row is a different class and does carry a guarantee: a list literal builds a
**new object** every time it is evaluated. Interning is not even in question here,
because the object is mutable — if it were shared, a change made in one would show up
in the other.

A single practical rule follows from this, and it has no exception: **equality is asked
with `==`, `is` is only for identity.** Code that tests whether two values are the same
using `is` works on small values and silently gives the wrong answer on large ones — a
defect class that raises no error message and is among the hardest to find.

The last two lines show the rule's legitimate use. `None` is a single object by the
language's definition; wherever it comes from, it is the same object. The same holds
for boolean values. This is why whether a value is empty is asked not with `== None`
but with **`is None`**: what is really being asked here is identity, and the language
guarantees the answer.

#### Summary

- Type is a property of the object, not the name; a single name can be bound to **7**
  distinct types and a function with no declaration at all.
- Dynamic typing is not the absence of type but type checking deferred to execution
  time; an unsupported operation falls with `TypeError`.
- The `+=` notation rebinds the name to a new object in **3** of four types and mutates
  the object in place in **1**; what makes the difference is whether the type is
  mutable.
- An in-place operation cannot be defined on an immutable object; this is why a second
  name keeps seeing the old value.
- Whether two equal values turn out to be the same object depends on the
  implementation, and a program's correctness must not rest on it; a list literal
  building a new object every time, by contrast, is the language definition's
  guarantee.
- Equality is asked with `==`; `is` is only for identity — which is why a `None` test is
  written as `is None`.

#### Next Step

This lesson measured that the notation `x += extra` does two different jobs depending
on the type, but never gave that job a **name**. The next lesson names it and pays off
the course's second claim: `n + m` and `n += m` are not two spellings of the same
operator, they are calls to two different special methods. Which method does each
arithmetic, comparison, logical, and membership operator call, where does a membership
test fall when it cannot find the method it is looking for, and what happens when an
operator finds no match at all?

### Operators

Source: https://academia.sh/en/courses/python-fundamentals/operators

Eleven operator forms call 10 distinct special methods and one goes unmatched: `n + m` calls `__add__`, `n += m` calls `__iadd__`; if `x in n` cannot find `__contains__`, it falls back to the iteration protocol and pays five calls instead of one.

The previous lesson measured that the notation `x += extra` does two different jobs
depending on the type: in three types it rebound the name to a new object, in one it
mutated the object in place. The job it does was never named. This lesson names it.

The Programming Fundamentals course built operators under the headings of precedence,
associativity, and short-circuit evaluation, and also showed that the same operator can
carry different meaning across different types. **None of that is repeated.** What was
built there was the concept — an operator's meaning depends on its operand's type. What
is measured here is the **machinery** of that link: which name each operator looks for,
where it falls back to if it cannot find what it is looking for, and what happens if it
finds nothing anywhere.

#### An Operator Is a Method Call

The first lesson's setup is extended for this lesson. `Tracker` keeps its existing
methods; subtraction, multiplication, right-hand multiplication, less-than, negation,
and intersection are added on top. Every method still logs its own name when called.

- **LF24** — All eleven forms measured run on two `Tracker` objects or one `Tracker`
  and an integer; the measurement looks not at the operator's **result** but at the name
  of the method it calls.
- **LF25** — The last form (`n / m`) is deliberately left unmatched: `Tracker` defines no
  method for division. The exclamation-prefixed entry on that line marks an exception,
  not a call.

```python
"""Special methods called by operators; Tracker logs each participation."""

LOG = []


def record(name):
    LOG.append(name)


class Tracker:
    """The operators portion of the shared setup; six methods are added in this lesson."""

    def __init__(self, items=(1, 2, 3)):
        self.items = list(items)

    def __iter__(self):
        record("__iter__")
        self._i = 0
        return self

    def __next__(self):
        record("__next__")
        if self._i >= len(self.items):
            raise StopIteration
        value = self.items[self._i]
        self._i += 1
        return value

    def __contains__(self, x):
        record("__contains__")
        return x in self.items

    def __add__(self, o):
        record("__add__")
        return Tracker(self.items + list(o.items))

    def __iadd__(self, o):
        record("__iadd__")
        self.items.extend(o.items)
        return self

    def __eq__(self, o):
        record("__eq__")
        return isinstance(o, Tracker) and self.items == o.items

    def __sub__(self, o):                      # added
        record("__sub__")
        return Tracker([x for x in self.items if x not in o.items])

    def __mul__(self, k):                      # added
        record("__mul__")
        return Tracker(self.items * k)

    def __rmul__(self, k):                     # added
        record("__rmul__")
        return Tracker(self.items * k)

    def __lt__(self, o):                       # added
        record("__lt__")
        return len(self.items) < len(o.items)

    def __neg__(self):                         # added
        record("__neg__")
        return Tracker([-x for x in self.items])

    def __and__(self, o):                      # added
        record("__and__")
        return Tracker([x for x in self.items if x in o.items])


def measure(function):
    """Runs an operator form and returns the protocols it called."""
    LOG.clear()
    try:
        function()
    except Exception as e:
        LOG.append(f"!{type(e).__name__}")
    return list(LOG)


def in_place_addition():
    a = Tracker()
    a += Tracker()


FORMS = (
    ("n + m", lambda: Tracker() + Tracker()),
    ("n += m", in_place_addition),
    ("n - m", lambda: Tracker() - Tracker()),
    ("n * 2", lambda: Tracker() * 2),
    ("2 * n", lambda: 2 * Tracker()),
    ("-n", lambda: -Tracker()),
    ("n < m", lambda: Tracker() < Tracker()),
    ("n == m", lambda: Tracker() == Tracker()),
    ("n & m", lambda: Tracker() & Tracker()),
    ("x in n", lambda: 9 in Tracker()),
    ("n / m", lambda: Tracker() / Tracker()),
)

print(f"{'operator form':<14s} {'calls':>5s}  protocol")
for name, function in FORMS:
    c = measure(function)
    print(f"{name:<14s} {len(c):5d}  {' '.join(c)}")

logs = [measure(f) for _, f in FORMS]
unique = sorted({p for c in logs for p in c if not p.startswith("!")})
print(f"\n{len(FORMS)} forms, distinct protocols {len(unique)}, "
      f"total calls {sum(len(c) for c in logs)}, "
      f"unmatched forms {sum(1 for c in logs if any(p.startswith('!') for p in c))}")
```

```
operator form  calls  protocol
n + m              1  __add__
n += m             1  __iadd__
n - m              1  __sub__
n * 2              1  __mul__
2 * n              1  __rmul__
-n                 1  __neg__
n < m              1  __lt__
n == m             1  __eq__
n & m              1  __and__
x in n             1  __contains__
n / m              1  !TypeError

11 forms, distinct protocols 10, total calls 11, unmatched forms 1
```

Eleven forms call **10 distinct protocols**, and **1** form goes unmatched. The table
reads one way: the symbol written on the left is shorthand for the name on the right.
The `-` operator means `__sub__`, `*` means `__mul__`, `<` means `__lt__`. The `&` row is
the clearest example of this reading — what the operator does is left to the type: on
integers it does a bitwise AND, here `Tracker` has defined it as intersection. An
operator does not carry a meaning, it carries **a name**.

The last line is the other side of this. Because `Tracker` defines no method for
division, the `n / m` form falls with `TypeError`. The source is flawless and passes the
compile phase; the defect is that the name to call cannot be found. There is no such
thing as an operator being "unsupported" — there is only an undefined method. This is
also why there is exactly one way to give a type division: write a method named
`__truediv__`.

#### Addition and In-Place Addition

The table's first two rows pay off the course's second claim. `n + m` calls
**`__add__`**, `n += m` calls **`__iadd__`**. They are not shorthand for each other;
separate names, separate contracts.

`__add__` does not touch its operands, it produces and returns **a new object**.
`__iadd__` mutates the existing object **in place and returns itself**. This is exactly
the distinction the previous lesson measured: if a type has defined `__iadd__`, `+=`
mutates the object and a second name bound to it sees the change too; if it has not,
the `+=` notation falls back to `__add__`, a new object is produced, and **only the
name on the left** is bound to it. This is exactly why integer, string, and tuple came
out as "name rebound" in the last lesson — these types are immutable and so cannot
define `__iadd__`.

A practical consequence follows: if `a += b` is written where `a` is bound to an object
shared with others, **it also changes the value they see**. The notation `a = a + b`
never does. The two lines do not do the same job, and the choice between them is not a
matter of style.

#### Falling Back to the Right-Hand Operand

The lines `n * 2` and `2 * n` call different methods: `__mul__` and `__rmul__`. The
reason is this: when `2 * n` is written, **the left-hand** operand's type is asked
first. An integer does not know how to multiply by a `Tracker`, and it answers "I do not
know." At that point **the right-hand** operand is asked the reverse direction of the
operation — the name sought is not `__mul__` but the one prefixed with `r`,
`__rmul__`.

This fallback is the basis of the language's extensibility. A built-in type cannot have
new behavior added to it; but a new type can join expressions where a built-in type
stands on the left, through methods like `__radd__`, `__rmul__`. If the sought method is
absent from both operands, the result is what the `n / m` line showed.

#### The Call Order Precedence Decides

If an expression has more than one operator, **precedence** decides which is called
first. The Programming Fundamentals course built the precedence and associativity
table; what is measured here is that table's code-level counterpart. Precedence is not
a reading convenience: it shapes the tree the second lesson showed, and that tree, in
turn, gives **the order methods are called in**.

- **LF26** — Three expressions are built on the same three objects and differ only by
  their operators and parentheses; what is measured is the **order** of the log
  sequence, not its length.

```python
def multiplication_first():
    Tracker((1,)) + Tracker((2,)) * 2


def addition_first():
    (Tracker((1,)) + Tracker((2,))) * 2


def comparison_after():
    Tracker((1,)) + Tracker((2,)) == Tracker((1, 2))


print(f"{'expression':<16s} call order")
for name, function in (("n + m * 2", multiplication_first),
                  ("(n + m) * 2", addition_first),
                  ("n + m == k", comparison_after)):
    print(f"{name:<16s} {' '.join(measure(function))}")
```

```
expression       call order
n + m * 2        __mul__ __add__
(n + m) * 2      __add__ __mul__
n + m == k       __add__ __eq__
```

The first two lines call the same two methods, in reverse order. Because multiplication
comes before addition, `__mul__` runs first, and the object it produces enters
`__add__` as an operand; once parentheses are added, the order flips. The third line
shows that arithmetic comes before comparison: equality cannot be asked before the
addition finishes, because the object to compare does not exist yet. Precedence rules
decide which method gets to see which object as its operand.

#### When a Membership Test Cannot Find Its Method

The second claim's most visible example is the membership test. The form `x in n` first
looks for `__contains__`; if it cannot find it, **it falls back to the iteration
protocol** and scans the object from the start, comparing items one by one. The syntax
is the same; the cost is not.

- **LF27** — The two compared objects carry the same three items; the only difference
  is that `Tracker` defines `__contains__` and `Iterable` does not.
- **LF28** — Every test is run once with a value that is found and once with a value
  that is not; what is measured is the result together with the call count.

```python
class Iterable:
    """Does not define __contains__; carries only the iteration protocol."""

    def __init__(self, items=(1, 2, 3)):
        self.items = list(items)

    def __iter__(self):
        record("__iter__")
        self._i = 0
        return self

    def __next__(self):
        record("__next__")
        if self._i >= len(self.items):
            raise StopIteration
        value = self.items[self._i]
        self._i += 1
        return value


MEMBERSHIP = (
    ("2 in Tracker()    ", lambda: 2 in Tracker()),
    ("9 in Tracker()    ", lambda: 9 in Tracker()),
    ("2 in Iterable()   ", lambda: 2 in Iterable()),
    ("9 in Iterable()   ", lambda: 9 in Iterable()),
)

print(f"{'membership test':<19s} {'result':>6s} {'calls':>5s}  protocol")
for name, function in MEMBERSHIP:
    LOG.clear()
    result = function()
    c = list(LOG)
    print(f"{name:<19s} {str(result):>6s} {len(c):5d}  {' '.join(c)}")
```

```
membership test     result calls  protocol
2 in Tracker()        True     1  __contains__
9 in Tracker()       False     1  __contains__
2 in Iterable()       True     3  __iter__ __next__ __next__
9 in Iterable()      False     5  __iter__ __next__ __next__ __next__ __next__
```

`Tracker` pays **1** call in both cases: the question asked goes straight to the object,
and the object knows the answer itself. `Iterable` gives the same results, but at a
different cost — **3** calls for a value that is found, **5** for one that is not.

The distribution of the numbers also shows how the fallback works. For the value that
is found, the scan stops at the second item: one `__iter__` and two `__next__`. For the
value that is not found, all three items are read, and the fourth `__next__` ends the
scan by raising `StopIteration` — the same "one more than the item count" pattern from
the first lesson. **A value that is not found always costs a full scan.**

That `Tracker` also defines `__iter__` closes one more point: if both methods are
defined, the `in` form **picks `__contains__`**, because the fallback only happens when
the sought method is absent. The result can come out the same either way; the call
count paid does not.

#### Chained Comparison and Logical Operators

Two notations remain, and neither fits the table, because the number of methods they
call is not fixed.

- **LF29** — The middle operand is produced by a function call, and that function also
  joins the log; this way, **how many times** the operand gets produced can be counted.
- **LF30** — `Tracker` defines no method for a truthiness test; an empty log on the
  logical-notation lines is meaningful for exactly this reason. Which method a
  truthiness test calls is measured in the flow topic's first lesson.

```python
def middle():
    record("middle()")                 # how many times is the middle operand produced?
    return Tracker((1, 2))


def chain_both_true():
    Tracker((1,)) < middle() < Tracker((1, 2, 3))


def explicit_form():
    Tracker((1,)) < middle() and middle() < Tracker((1, 2, 3))


def chain_first_false():
    Tracker((1, 2, 3)) < middle() < Tracker((1, 2, 3, 4))


print(f"{'notation':<32s} {'calls':>5s}  protocol")
for name, function in (("a < middle() < c  (both true)", chain_both_true),
                  ("a < middle() and middle() < c", explicit_form),
                  ("a < middle() < c  (first false)", chain_first_false)):
    c = measure(function)
    print(f"{name:<32s} {len(c):5d}  {' '.join(c)}")

print()
print(f"{'logical notation':<12s} {'calls':>5s}  protocol")
for name, function in (("n and m", lambda: Tracker() and Tracker()),
                  ("n or m", lambda: Tracker() or Tracker()),
                  ("not n", lambda: not Tracker()),
                  ("n & m", lambda: Tracker() & Tracker())):
    c = measure(function)
    print(f"{name:<12s} {len(c):5d}  {' '.join(c) if c else '(none)'}")

print()
print("what and/or return:", repr("ab" and "cd"), repr("" or "cd"),
      repr(0 or []), "| what not returns:", repr(not "ab"))
```

```
notation                         calls  protocol
a < middle() < c  (both true)        3  middle() __lt__ __lt__
a < middle() and middle() < c        4  middle() __lt__ middle() __lt__
a < middle() < c  (first false)      2  middle() __lt__

logical notation calls  protocol
n and m          0  (none)
n or m           0  (none)
not n            0  (none)
n & m            1  __and__

what and/or return: 'cd' 'cd' [] | what not returns: False
```

The chained comparison calls two `__lt__`, but produces the middle operand only
**once**. The manually spelled-out notation performs the same two comparisons and
produces the middle operand **twice**: four calls against three. The difference is not
just in count — it can be in correctness too: if the middle expression carries a side
effect, the two notations do not give the same program. The third line shows the
chain's second half: when the first comparison comes out false, the second is never
performed at all, and the call count stays at **2**.

The lower table draws a boundary. `and`, `or`, and `not` call no special method at all;
their logs are empty. These cannot be overloaded, because what they do is not an
operation but **a choice**: they run a truthiness test on an operand and return **one of
the operands as-is**. The last line confirms this — `"ab" and "cd"` does not give a
truth value, it gives the string `'cd'`; `0 or []` gives the empty list. Only `not`
produces a truth value. The pair most prone to confusion is `and` and `&`: the first is
a choice, the second is an operator that calls `__and__`.

#### Summary

- Every operator's name is shorthand for a specific special method; eleven forms call
  **10 distinct protocols**, and a form with no counterpart falls with `TypeError`.
  Precedence decides the order these methods are called in.
- `n + m` calls **`__add__`**, `n += m` calls **`__iadd__`**; the first produces a new
  object, the second mutates the existing one, and this difference is observable on a
  shared object.
- If the left-hand operand cannot support the operation, the right-hand operand's
  reverse-direction method (`__rmul__`) is sought; if neither has it, the operator goes
  unmatched.
- `x in n` first looks for `__contains__` (**1** call); if it cannot find it, it falls
  back to the iteration protocol and pays **5** calls for a value that is not found.
- A chained comparison produces the middle operand once, and skips the second
  comparison entirely if the first one is false.
- `and`, `or`, and `not` call no special method; they return one of the operands as-is,
  and are distinct from `&` and `|`, which have bitwise counterparts.

#### Next Step

This lesson saw that the `+` operator produces a new object on `Tracker`. The same
operator works on strings too, and there is not even a choice there: a string is an
immutable type, so it cannot define `__iadd__`. The next lesson counts the cost of
this — why turning a string to lowercase, stripping its whitespace, or replacing a part
of it has to produce **a new object** every time, how formatting notations reduce this
production to a single step, and how many objects come out of a chain of successive
changes?

### Working with Strings

Source: https://academia.sh/en/courses/python-fundamentals/working-with-strings

Because a string is immutable, every producing method returns a new object: joining seven parts with `+` builds 7 intermediate objects, `join` builds 1, and eight formatting notations call 3 distinct special methods.

The previous lesson measured that the `+` operator calls `__add__`, the `+=` notation
calls `__iadd__`, and said that immutable types cannot define `__iadd__`. A string is
exactly such a type; every operation on it derives from this single constraint.

This lesson's question is not the list of string methods. The question is: how does a
type being immutable **force** the methods that can be defined on it, and how much does
that forcing cost, counted?

#### There Is No Such Thing as Writing in Place

There is no way to change a single character of a string. The syntax exists — writing
with square brackets — but no method backing it is defined.

- **LF31** — A single starting string is used in the measurement, and four operations
  are run on it in sequence; at every step, whether the result is the **same object** as
  the previous one is asked with `is`. No identity number is printed.
- **LF32** — The four chosen operations really change the content. Whether an operation
  whose result is byte-for-byte identical to its input (`strip` on a string with no
  whitespace, for instance) returns the same object or not is
  **implementation-dependent** and is not brought into the measurement.
- **LF33** — The methods in the second table do not transform the string, they ask it a
  question; what is measured is the **type** of the returned value.

```python
"""A string is immutable: producing methods return a new object, query methods return another type."""

RAW = "  12; north; 18  "

try:
    RAW[0] = "x"
except TypeError as e:
    print("writing in place to a string:", type(e).__name__)

STEPS = (("start", lambda s: s),
           ("strip()", lambda s: s.strip()),
           ("upper()", lambda s: s.upper()),
           ("replace(';', ',')", lambda s: s.replace(";", ",")),
           ("split(',')", lambda s: s.split(",")))

print(f"\n{'step':<20s} {'type':<5s} {'same as previous':>17s}  value")
previous = RAW
for name, operation in STEPS:
    current = operation(previous)
    print(f"{name:<20s} {type(current).__name__:<5s} "
          f"{str(current is previous):>17s}  {current!r}")
    previous = current
print("starting string after the measurement:", repr(RAW))

TEXT = "12; north; 18"
QUERIES = (("startswith('12')", lambda s: s.startswith("12")),
            ("find('north')", lambda s: s.find("north")),
            ("count(';')", lambda s: s.count(";")),
            ("isdigit()", lambda s: s.isdigit()),
            ("len(s)", lambda s: len(s)))

print(f"\n{'query method':<20s} {'return type':<5s} value")
for name, function in QUERIES:
    value = function(TEXT)
    print(f"{name:<20s} {type(value).__name__:<5s} {value!r}")
print("string after the queries:", repr(TEXT))
```

```
writing in place to a string: TypeError

step                 type   same as previous  value
start                str                True  '  12; north; 18  '
strip()              str               False  '12; north; 18'
upper()              str               False  '12; NORTH; 18'
replace(';', ',')    str               False  '12, NORTH, 18'
split(',')           list              False  ['12', ' NORTH', ' 18']
starting string after the measurement: '  12; north; 18  '

query method         return type value
startswith('12')     bool  True
find('north')        int   4
count(';')           int   2
isdigit()            bool  False
len(s)               int   13
string after the queries: '12; north; 18'
```

The first line states the constraint by name: writing with square brackets gives
`TypeError`, because the string type has no method backing that operation. It is the
exact same result taken for the division operator in the previous lesson — the defect
is not a prohibition, it is an absence.

In all four steps of the chain, the "same as previous" column reads `False`. Every
method built a new object and did not touch its input; the last line confirms this —
after four operations, the starting string still stands as it was. It matters to see
that this is not a choice. A string method could not mutate its input **even if it
wanted to**; the only thing it can do is build and return a new object. Immutability
writes the method's contract on its own, which is why the string type carries no
method that works in place, like a list's `append` or `sort`.

A direct rule follows from this: **if a string method's return value is not used,
nothing has happened.** The line `text.strip()` written alone builds a new object, does
not bind it to a name, and discards it; `text` stays as it was. This is the single most
common silent defect in code written by someone new to the language.

The lower table sets apart the second class of method. `startswith`, `find`, `count`,
`isdigit`, and `len` do not produce a string; they give a boolean, a position, a count,
in turn. These do not transform the string, they ask it a question — and so they sit
outside the new-object discussion. The `find` method returns `-1` when it cannot find
the piece sought; it does not raise an exception. The `index` method, which does the
same job, raises `ValueError` when it cannot find it. The choice between the two
methods is not a matter of style: when not finding something is an expected situation,
the notation that works with a return value is chosen; when it is a situation that
should not happen, the notation that raises an exception is chosen. The counterpart of
this same distinction on the conversion side is the next lesson's subject.

#### A String Is a Sequence

In the third lesson's built-in type table, `str` filled all five columns: its length
could be asked, a loop could be built over it, membership could be tested, it could be
sliced, and it could be added. This means a string is not only text but also a
**sequence** — and being a sequence has two observable consequences.

- **LF34** — In the slicing measurement, every result is compared to the source string
  with `is`; a slice that asks for the whole source is not used, because whether it
  returns the same object is implementation-dependent.
- **LF35** — In the escaping measurement, the **character count** of the three written
  notations is read; the number shows how many characters the notation produces, not
  how many characters were written in the source.

```python
"""A string is a sequence: slicing, iteration, and membership rest on the same protocols."""

TEXT = "12; north; 18"

SLICES = (("[0]", lambda s: s[0]),
            ("[-1]", lambda s: s[-1]),
            ("[4:9]", lambda s: s[4:9]),
            ("[:2]", lambda s: s[:2]),
            ("[::-1]", lambda s: s[::-1]),
            ("[::4]", lambda s: s[::4]))

print(f"{'slice':<8s} {'same object as source':>22s}  value")
for name, function in SLICES:
    piece = function(TEXT)
    print(f"{name:<8s} {str(piece is TEXT):>22s}  {piece!r}")

print(f"\nlength {len(TEXT)}, first four items {list(TEXT)[:4]}")
print("membership:", "'nor' in text ->", "nor" in TEXT,
      "| 'rth' in text ->", "rth" in TEXT,
      "| 'thr' in text ->", "thr" in TEXT)
print("same question on a list:", "[1, 2] in [1, 2, 3] ->", [1, 2] in [1, 2, 3],
      "| 2 in [1, 2, 3] ->", 2 in [1, 2, 3])

ESCAPES = (("'a\\tb'", "a\tb"), ("r'a\\tb'", r"a\tb"),
         ("'''two\\nlines'''", "two\nlines"))
print(f"\n{'notation':<18s} {'characters':>10s}  actual content")
for name, value in ESCAPES:
    print(f"{name:<18s} {len(value):10d}  {value!r}")
```

```
slice     same object as source  value
[0]                       False  '1'
[-1]                      False  '8'
[4:9]                     False  'north'
[:2]                      False  '12'
[::-1]                    False  '81 ;htron ;21'
[::4]                     False  '1nh8'

length 13, first four items ['1', '2', ';', ' ']
membership: 'nor' in text -> True | 'rth' in text -> True | 'thr' in text -> False
same question on a list: [1, 2] in [1, 2, 3] -> False | 2 in [1, 2, 3] -> True

notation           characters  actual content
'a\tb'                      3  'a\tb'
r'a\tb'                     4  'a\\tb'
'''two\nlines'''            9  'two\nlines'
```

All six slices give an object separate from the source. Slicing is the `__getitem__`
call measured in the first lesson, and it carries the same rule on a string: a new
object is built, the source stays as it is. A negative position counts from the end,
the third number sets the step; the `[::-1]` notation reversing the text is not a
separate method's job, it is the consequence of the step being negative.

The iteration line shows that a string's items are **characters**: a thirteen-character
text gives thirteen items, and every item is itself a string. Python has no separate
type representing a single character; a character is a string whose length is one.

The membership line, in turn, sets a string apart from other sequences. `"nor" in text`
gives **True** — even though what is being asked for is not one item but a **sequence**
of items. When the same question is asked of a list, `[1, 2] in [1, 2, 3]` gives
**False**, because `in` on a list only looks for an item. A string's `__contains__`
method is deliberately written differently: it searches for a substring. The syntax is
the same across the two types, the contract is not.

The last table sets apart notation forms. Backslash escape sequences are written with
two characters in the source but produce a **single** character: the notation
`'a\tb'` is a three-character string. In the raw form, prefixed with `r`, the escape is
not resolved, and the same source gives four characters. The triple-quoted form carries
a line ending as-is. In all three, the character count written in the source and the
length of the string produced are different; this is the reason a raw notation is
preferred when writing a file path or a pattern string.

#### The Cost of Concatenation

Immutability's measurable cost shows up when building text piece by piece. Because
every `+` operation builds a new object, a string grown inside a loop leaves behind
intermediate objects built and discarded along the way.

- **LF36** — Two methods build the same text from the same seven parts; what is
  measured is the **number of intermediate objects built** along the way, not the time
  elapsed. In the incremental method, every step's result is compared to the previous
  one with `is` and counted; because joining builds a single object, its count is
  written directly.

```python
"""The cost of concatenation: how many intermediate objects get built?"""

PARTS = ("north", "; ", "12", "; ", "slope", "; ", "18")


def incremental(parts):
    text = ""
    intermediate = 0
    for p in parts:
        previous = text
        text = text + p          # a new string on every step
        intermediate += text is not previous
    return text, intermediate


def joining(parts):
    text = "".join(parts)      # one object in one step
    return text, 1


print(f"{'method':<18s} {'parts':>5s} {'intermediate objects':>21s}  result")
for name, function in (("incrementing with +", incremental), ("with join", joining)):
    result, intermediate = function(PARTS)
    print(f"{name:<18s} {len(PARTS):5d} {intermediate:21d}  {result!r}")

print("are the two results equal:", incremental(PARTS)[0] == joining(PARTS)[0])
```

```
method             parts  intermediate objects  result
incrementing with +     7                     7  'north; 12; slope; 18'
with join              7                     1  'north; 12; slope; 18'
are the two results equal: True
```

Seven parts build **7** intermediate objects with the incremental method; `join` builds
**1**. The results are byte-for-byte equal. The pattern grows linearly with the part
count: when the part count doubles, the number of objects the incremental method
builds doubles too, while `join`'s stays at **1**.

The reason for the difference is not that one method is written better, it is that
**it already has the information**. `join` sees all the parts up front, can compute the
total length, and builds the result in one pass. The `+` operator, by contrast, only
sees two parts on each call; not knowing what comes at the next step, it has to fully
build that step's result.

The same reasoning also explains why collecting into a list and joining at the end is
a common pattern: a list is a **mutable** type, appending to it builds no new object.
The text is produced only once, at the very end.

The `join` method has one condition that is easy to overlook: **all the parts have to
be strings.** If a number slips in among them, the method does not convert it to text
on its own — it raises `TypeError`. The string the method is called on is not one of
the parts either, it is the **separator**: in the notation `"; ".join(...)`, the text
placed in between is the string the call is made on.

#### Which Protocol Formatting Calls

There is more than one notation for producing text, and they look alike. The course's
measure applies here too: each notation calls a different method, and the text each
produces can therefore diverge.

- **LF37** — The measured object takes part in all three text protocols: `__str__`,
  `__repr__`, and `__format__`. All three produce different text, so which one was
  called can also be read from the output.
- **LF38** — The `__format__` method itself calls no other protocol; if the spec is
  empty it builds its own text, if it is filled it applies the spec to the number it
  carries. This way, the log shows only what the notation called.

```python
"""Which special method do formatting notations call?"""

LOG = []


def record(name):
    LOG.append(name)


class Measurement:
    """Takes part in all three text protocols and logs each participation."""

    def __init__(self, value=12):
        self.value = value

    def __str__(self):
        record("__str__")
        return f"measurement {self.value}"

    def __repr__(self):
        record("__repr__")
        return f"Measurement({self.value})"

    def __format__(self, spec):
        record("__format__")
        return format(self.value, spec) if spec else f"measurement {self.value}"


FORMS = (
    ('f"{n}"', lambda n: f"{n}"),
    ('f"{n:>6}"', lambda n: f"{n:>6}"),
    ('f"{n!s}"', lambda n: f"{n!s}"),
    ('f"{n!r}"', lambda n: f"{n!r}"),
    ('"{}".format(n)', lambda n: "{}".format(n)),
    ('"%s" % n', lambda n: "%s" % n),
    ('str(n)', lambda n: str(n)),
    ('repr(n)', lambda n: repr(n)),
)

unique = set()
print(f"{'notation':<16s} {'protocol':<12s} text produced")
for name, function in FORMS:
    LOG.clear()
    text = function(Measurement())
    unique.update(LOG)
    print(f"{name:<16s} {' '.join(LOG):<12s} {text!r}")

print(f"\n{len(FORMS)} notations, distinct protocols {len(unique)}: "
      f"{', '.join(sorted(unique))}")
```

```
notation         protocol     text produced
f"{n}"           __format__   'measurement 12'
f"{n:>6}"        __format__   '    12'
f"{n!s}"         __str__      'measurement 12'
f"{n!r}"         __repr__     'Measurement(12)'
"{}".format(n)   __format__   'measurement 12'
"%s" % n         __str__      'measurement 12'
str(n)           __str__      'measurement 12'
repr(n)          __repr__     'Measurement(12)'

8 notations, distinct protocols 3: __format__, __repr__, __str__
```

Eight notations spread across **3 distinct protocols**. The curly-brace notation and
the `format` method call `__format__`; the percent-sign notation and the `str` built-in
call `__str__`; the `repr` built-in and the `!r` marker call `__repr__`. The same
object gives three different pieces of text across three notations, and none stands in
for another.

Two points deserve a closer read. First, `__format__` **also receives the spec**: on
the second line, the spec `>6` was passed to the method, and the method applied it to
the number it carries. Alignment, digit count, and width rules are handed to the
object itself this way. Second, the markers are not a shortcut, they are a **protocol
selector**: the `!r` notation switches the method called from `__format__` to
`__repr__`.

The curly-brace notation has one more distinct feature: what goes inside the braces
does not have to be a value, it can be an **expression**, and the expression is
evaluated right there. This is why the second lesson's statement/expression distinction
applies here too; an assignment cannot be placed inside the braces, because an
assignment is not an expression. The `format` method, by contrast, takes values as
arguments and keeps the spec separate.

The three methods' contracts differ too. `__str__` gives readable text; `__repr__`
gives text that identifies the object, usually written in a form that could rebuild
it — the measurement's `'Measurement(12)'` is an example of this. If an object defines
only `__repr__`, a `str` call falls to it too; this is the source of the behavior seen
on the bare object in the first lesson.

#### Summary

- Writing to a single character of a string gives `TypeError`; no method backs that
  operation.
- In all four producing methods, the result is a new object and the input stays as it
  is; if the return value is not used, nothing has happened.
- Query methods do not produce a string, they give a response of another type; `find`
  returns `-1` when it cannot find something, it does not raise an exception.
- A string is a sequence: slicing calls `__getitem__` and gives a new object, its items
  are characters, and unlike a list, the `in` test looks for a **substring**.
- Concatenating seven parts incrementally builds **7** intermediate objects, `join`
  builds **1**; the difference comes from `join` seeing all the parts up front.
- Eight formatting notations spread across **3 distinct protocols**: `__format__`,
  `__str__`, and `__repr__`; markers like `!r` change which method gets called.

#### Next Step

This lesson measured that the notation `str(n)` calls `__str__` — meaning turning an
object into text is itself a protocol call. The same question can be asked in the
reverse direction: what does `int("12")` call, what does `float(n)` ask of an object,
and what happens when a conversion with no counterpart, like `int("north")`, is
requested? The next lesson counts which special methods conversion rests on, and shows
that a failed conversion produces not a return value but **an exception**.

### Type Conversion

Source: https://academia.sh/en/courses/python-fundamentals/type-conversion

Conversion built-ins call distinct special methods across eight notations, and two notations with no counterpart go unmatched; twenty string attempts split into 7 `ValueError` and 4 `TypeError`, and numeric promotion is shown not to be magic through `NotImplemented`.

The previous lesson measured that the notation `str(n)` calls `__str__`: turning an
object into text is itself a protocol call. The same question can be asked in the
reverse direction. What does `int(n)` call, what does `float(n)` ask of the object, and
what happens when a conversion with no counterpart is requested?

The Programming Fundamentals course already established explicit and implicit
conversion, lossy narrowing, and validating outside input before converting it; it also
showed that the move from a real number to an integer truncates rather than rounds.
**None of that is repeated.** What is measured here is the conversion's **machinery**:
which built-in looks for which name, which exception it raises when it cannot find what
it is looking for, and whether what the language does when two different numeric types
are added is really a "conversion."

#### Conversion Is a Protocol Call Too

`int`, `float`, `str`, and `list` are built-in functions, and all of them work the same
way: they ask the given object for a method carrying a specific name. If the name is
absent, there is no conversion either.

- **LF39** — The measured object takes part in four conversion protocols at once and
  logs each participation; the exclamation-prefixed entry in the log marks an
  exception, not a call.
- **LF40** — Two counter-examples are used: an object defining no conversion method at
  all, and one defining only `__int__` while not carrying the index protocol.
- **LF41** — A fixed five-letter string is used in the indexing measurement; the letter
  returned depends on the number the object gives as an index.

```python
"""Which special method do conversion built-ins call?"""

LOG = []


def record(name):
    LOG.append(name)


class Measurement:
    """Takes part in the conversion protocols and logs each participation."""

    def __init__(self, items=(1, 2, 3)):
        self.items = list(items)

    def __int__(self):
        record("__int__")
        return sum(self.items)

    def __float__(self):
        record("__float__")
        return float(sum(self.items))

    def __str__(self):
        record("__str__")
        return f"measurement {self.items}"

    def __index__(self):
        record("__index__")
        return len(self.items)

    def __iter__(self):
        record("__iter__")
        self._i = 0
        return self

    def __next__(self):
        record("__next__")
        if self._i >= len(self.items):
            raise StopIteration
        value = self.items[self._i]
        self._i += 1
        return value


class Plain:
    """Defines no conversion method."""


class IntOnly:
    """Defines only __int__; does not carry the index protocol."""

    def __int__(self):
        record("__int__")
        return 2


def measure(function):
    LOG.clear()
    try:
        result = repr(function())
    except Exception as e:
        LOG.append(f"!{type(e).__name__}")
        result = "—"
    return list(LOG), result


FORMS = (
    ("int(n)", lambda: int(Measurement())),
    ("float(n)", lambda: float(Measurement())),
    ("str(n)", lambda: str(Measurement())),
    ("list(n)", lambda: list(Measurement())),
    ("'north'[n]", lambda: "north"[Measurement((1, 2))]),
    ("int(Plain())", lambda: int(Plain())),
    ("'north'[m]", lambda: "north"[IntOnly()]),
    ("int(m)", lambda: int(IntOnly())),
)

print(f"{'notation':<14s} {'calls':>5s} {'protocol':<32s} result")
for name, function in FORMS:
    c, result = measure(function)
    print(f"{name:<14s} {len(c):5d} {' '.join(c):<32s} {result}")
```

```
notation       calls protocol                         result
int(n)             1 __int__                          6
float(n)           1 __float__                        6.0
str(n)             1 __str__                          'measurement [1, 2, 3]'
list(n)            5 __iter__ __next__ __next__ __next__ __next__ [1, 2, 3]
'north'[n]         1 __index__                        'r'
int(Plain())       1 !TypeError                       —
'north'[m]         1 !TypeError                       —
int(m)             1 __int__                          2
```

The first three lines establish the rule: the `int` built-in calls `__int__`, `float`
calls `__float__`, `str` calls `__str__`. The correspondence between the built-in's
name and the method's name is not a coincidence, it is the contract. The built-in is
not what performs the conversion — it only **knows who will do it**; the object itself
does the work and returns the result.

The fourth line sets apart one more conversion. `list(n)` does not look for a `__list__`
method; it uses **the iteration protocol** and builds a new list by collecting items one
at a time. **5** calls are paid for three items: one `__iter__` and four `__next__` —
the exact "one more than the item count" pattern from the first lesson. A general rule
follows from this: conversion to a collection asks for no separate protocol, it is
built on the one that already exists. For the same reason, `tuple(n)` and `set(n)` seek
no new method either; all three consume the same iteration protocol and differ only in
the type of container they build.

The fifth and seventh lines should be read together. Using an object as an index does
not ask for `__int__`, it asks for **`__index__`**. The two are separate contracts:
`__int__` means "this object **can be converted** to an integer," `__index__` means
"this object already **is** an integer position." The distinction is not arbitrary — a
real number carries `__int__` but cannot serve as an index, because truncating during
indexing would be a silent defect. The object that defines only `__int__` shows exactly
this in the last two lines: `int(m)` works, `'north'[m]` falls with `TypeError`.

The sixth line is the same result as in the first lesson. On an object defining no
conversion method, the `int` call raises `TypeError`. **6** of eight notations find a
method, **2** cannot and fall. **Conversion is not an entitlement, it is a matter of
having the method.**

#### A Failed Conversion Is an Exception

The table above measured conversion from an object to a number. Data coming from the
outside world, though, is a string, and the situation there is different: the `str`
type does not define `__int__`. When `int("12")` is written, what happens is not a
protocol call but **parsing** the text — and parsing can fail.

- **LF42** — Ten inputs are tried: eight are strings, two are of another type. Every
  input is tried with both `int` and `float`; what is measured is the **type** of the
  returned value or the exception.
- **LF43** — The count collects only exceptions; successful conversions do not enter the
  count.

```python
"""Failed conversion: which input produces which exception?"""

INPUTS = ("12", " 12 ", "+12", "1_2", "12.5", "", "north", "0x1f", None, [1])


def try_convert(function, value):
    try:
        return repr(function(value)), "—"
    except Exception as e:
        return "—", type(e).__name__


print(f"{'input':<10s} {'int(...)':>10s} {'exception':<12s}"
      f" {'float(...)':>12s} {'exception':<12s}")
counts = {}
for value in INPUTS:
    i_value, i_error = try_convert(int, value)
    f_value, f_error = try_convert(float, value)
    for h in (i_error, f_error):
        if h != "—":
            counts[h] = counts.get(h, 0) + 1
    print(f"{repr(value):<10s} {i_value:>10s} {i_error:<12s}"
          f" {f_value:>12s} {f_error:<12s}")

print(f"\ntotal attempts {2 * len(INPUTS)}, "
      + ", ".join(f"{h} {n}" for h, n in sorted(counts.items())))
print("with a base given:", "int('0x1f', 16) ->", int("0x1f", 16))
```

```
input        int(...) exception      float(...) exception   
'12'               12 —                    12.0 —           
' 12 '             12 —                    12.0 —           
'+12'              12 —                    12.0 —           
'1_2'              12 —                    12.0 —           
'12.5'              — ValueError           12.5 —           
''                  — ValueError              — ValueError  
'north'             — ValueError              — ValueError  
'0x1f'              — ValueError              — ValueError  
None                — TypeError               — TypeError   
[1]                 — TypeError               — TypeError   

total attempts 20, TypeError 4, ValueError 7
with a base given: int('0x1f', 16) -> 31
```

**11** of twenty attempts end in an exception, and the exceptions split into two kinds:
**7** `ValueError`, **4** `TypeError`. The split is not arbitrary, and this is exactly
what needs to be learned.

**`TypeError`** is born when the input's **type** cannot be converted. In the `None` and
list rows, the problem is not what the value is, it is that the type carries no method
for converting to a number — the same defect class as the `Plain` row in the first
table. **`ValueError`**, by contrast, is born when the type is right but the **value**
is not suitable. `'north'` is a string, and a string can be converted to a number; what
cannot be converted is the text that string carries.

The accepted inputs are instructive too. Leading and trailing whitespace is dropped, a
plus sign is accepted, an underscore is read as a digit separator. But the string
`'12.5'` is invalid for `int` and valid for `float`: `int` expects an integer
representation and does not silently truncate a fractional text. The truncating
behavior measured in the previous course happens on the move from a **real number
object** to an integer, not from a string. The `'0x1f'` row is of the same class too;
hexadecimal notation is only read when a base is explicitly given, and the last line
shows this.

The rule that follows from this is the same as the `find`/`index` distinction from the
previous lesson: **a failed conversion produces not a return value but an exception.**
No built-in returns a special number that says "I could not convert this"; had it done
so, that value could not be told apart from a real result. A noisy exception was chosen
over a silent defect.

The table's accepted column carries one more warning. The Programming Fundamentals
course gave validating the format before conversion as one of two patterns; the
measurement shows the limit of that pattern. A pre-check that only tests for being made
of digits rejects the inputs `' 12 '` and `'+12'` — yet both convert without a problem.
The set a pre-check accepts and the set a conversion accepts are not the same, and
keeping the two sets consistent by hand means writing the conversion rules a second
time. Trying and catching the fallen conversion is therefore the narrower contract: it
leaves the acceptance rule in a single place, the conversion itself.

#### Is Conversion Reversible

A failed conversion makes noise and gets found. The real danger is in a conversion
that **looks successful**: a value is converted, no exception is born, and converting
it back does not give the original. The Programming Fundamentals course showed this
loss; what is measured here is the loss's **boundary** — which path is guaranteed, and
which is not.

- **LF44** — Five integers go and come back by two separate paths: through a string and
  through a real number. What is measured is whether the returned value is **equal** to
  the starting one.
- **LF45** — In the real-number measurement, the return path is built through the text
  that identifies the object; the four chosen values are picked from opposite ends in
  magnitude and fraction.

```python
"""Is conversion reversible: does the round trip give back the same value?"""

INTEGERS = (12, -3, 2 ** 53, 2 ** 53 + 1, 10 ** 23)
REALS = (0.1, 3.5, 1 / 3, 1e300)

print(f"{'integer':<26s} {'via string':>16s} {'via float':>17s}")
for n in INTEGERS:
    print(f"{n!s:<26s} {str(int(str(n)) == n):>16s} "
          f"{str(int(float(n)) == n):>17s}")

print(f"\n{'real number':<26s} {'via string':>16s}")
for x in REALS:
    print(f"{x!r:<26s} {str(float(repr(x)) == x):>16s}")
```

```
integer                          via string         via float
12                                     True              True
-3                                     True              True
9007199254740992                       True              True
9007199254740993                       True             False
100000000000000000000000               True             False

real number                      via string
0.1                                    True
3.5                                    True
0.3333333333333333                     True
1e+300                                 True
```

The string column reads `True` in all five rows. Going and coming back through text is
**always** reversible for an integer, because — as measured in the third lesson — an
integer has no fixed width: however many digits it has, the text stretches to match.
The same guarantee holds for a real number too — all four rows of the lower table read
`True`. The text that identifies a real number is produced carrying enough digits to
give that number back.

The upper table's **via float** column, though, splits in two: the first three values
come back with no problem, the last two do not. The boundary shows up right in the
table: two to the fifty-third works fine, one more than that does not. The reason is
that a real number's representation can only carry a limited number of significant
digits; above that limit, two adjacent integers collapse onto the same real number, and
on the way back the two cannot be told apart. This is the Python-side counterpart of
the representation limit established in the How Computers Work course. The equality
test reading `False` in the table also shows that the conversion raised no exception:
both conversions completed successfully, only the result was not the starting value.

The practical consequence is singular: **if an integer's value needs to be preserved, it
is not routed through a real number.** Neither an exception is born nor a warning
appears; only an equality test comes out wrong. The exceptions measured throughout this
lesson are the exact opposite of this class — an exception makes the defect audible.

#### Numeric Promotion Is Not Magic

Implicit conversion remains. When `1 + 2.5` is written, the integer is "promoted" to a
real number and the result comes out real. This gives the impression that the language
carries a separate rule for numbers. The measurement shows this is not true.

- **LF46** — The methods behind the operator are called directly; what is measured is
  the returned value. When the method cannot produce a result, what it returns is not
  an exception but a special marker.
- **LF47** — Four additions are tried; in two, two distinct numeric types meet, in two,
  a number meets a string.

```python
"""Numeric promotion is not magic, it is the fallback in operators."""

print("(1).__add__(2.5) ->", (1).__add__(2.5))
print("(2.5).__radd__(1) ->", (2.5).__radd__(1))
print("1 + 2.5 ->", 1 + 2.5, "| type", type(1 + 2.5).__name__)

print()
print(f"{'expression':<14s} {'result':>8s}  exception")
for left, right in (("3", 4), (3, "4"), (3, 2.5), (True, 1)):
    try:
        result, error = repr(left + right), "—"
    except TypeError as e:
        result, error = "—", type(e).__name__
    print(f"{repr(left) + ' + ' + repr(right):<14s} {result:>8s}  {error}")
```

```
(1).__add__(2.5) -> NotImplemented
(2.5).__radd__(1) -> 3.5
1 + 2.5 -> 3.5 | type float

expression       result  exception
'3' + 4               —  TypeError
3 + '4'               —  TypeError
3 + 2.5             5.5  —
True + 1              2  —
```

The first line is decisive. The integer's `__add__` method, given a real number,
returns not a result but `NotImplemented` — a special value meaning "I cannot do this
job." At that point, the fallback measured in the fourth lesson kicks in: the
right-hand operand's reverse-direction method is tried, and as seen in the second line,
the real number takes over the job. The third line gives the reason the result comes
out real — the side that does the arithmetic is the real-number type.

So there is no separate mechanism called "promotion." Numeric types work together
because they support each other's reverse-direction methods; this is the exact same
thing as the `__rmul__` fallback from the fourth lesson. The lower table draws the
boundary too: a string and a number support neither direction of each other's method,
so both notations give `TypeError`. Which one stands on the left and which on the right
does not change the result. Code that wants to add a string and a number has to write
the conversion **itself**; the language not filling this gap on its own is not a
shortcoming, it is a decision made so as not to produce a silent result.

The last line closes a Python detail. The expression `True + 1` gives **2**, because,
as measured in the third lesson, booleans are a subtype of integer. No conversion
happens here; the object already is an integer.

#### Summary

- Conversion built-ins call specially named methods: `int` calls `__int__`, `float`
  calls `__float__`, `str` calls `__str__`; if the method is absent, the conversion
  falls with `TypeError`.
- `list(n)` asks for no separate method, it uses the iteration protocol and pays **5**
  calls for three items.
- Being used as an index asks for `__index__`, not `__int__`; the first states
  convertibility, the second states that the object already is a position.
- **11** of twenty string attempts end in an exception: **4** `TypeError` when the type
  cannot be converted, **7** `ValueError` when the type is right but the value is not
  suitable.
- A failed conversion produces not a special return value but an exception; had it not,
  that value could not be told apart from a real result. The silent defect lives in a
  large integer that is routed through a real number and comes back different.
- Numeric promotion is not a separate rule: the integer's `__add__` method returns
  `NotImplemented`, and the real number's reverse-direction method takes over the job.

#### Next Step

This lesson closed a topic while leaving one question open. When conversion fails, an
**exception** appears, and an exception tears the flow away from where it was and
carries it somewhere else — meaning it is not a defect report, it is a flow tool. The
next topic takes up flow itself and opens its first lesson with this question: when a
condition is written, what exactly is being tested? A number, a string, or an empty
list can all be placed in an `if` line, and all are accepted. So who decides whether an
object counts as **true or false** — and what happens when there is no method to make
that decision?

## Control Flow and Functions

Comparison chains and truthiness rules, loops through the iteration protocol, positional and keyword arguments, the four scope levels, and the contract of built-ins.

### Conditionals

Source: https://academia.sh/en/courses/python-fundamentals/conditionals

A condition does not expect a logical value, it puts the object through the truthiness rule: eight protocol calls are measured across twelve trials, four return true without calling any method, and a comparison chain produces its middle term once.

The previous lesson measured type conversion and showed that where conversion fails,
an **exception** is born: when an object cannot answer for the type demanded of it,
flow is interrupted. Conversion asks the object "what do you correspond to as an
integer?" and that question has the right to go unanswered.

A condition also asks the object a question, but a different one: **"are you counted
as true?"** Unlike conversion's, the answer here is almost never an exception — if
the object does not give the answer itself, the language produces one in its place.
So who gives that answer? Does `if` look inside the object and count its items, check
its type, or ask the object? This is what this lesson measures.

#### A Condition Does Not Expect a Logical Value

The Programming Fundamentals course's conditional-branching lesson defined a
condition as **an expression producing a logical value**; that definition was
language-independent and built branching as a **control structure**. In Python the
definition is broader: **any object** can be written after the `if` keyword, and the
language reduces it to a truth value itself. What is measured here is not branching
itself, but **which special method** that reduction calls.

The reduction rule is called **truthiness** and has three rungs:

1. If `__bool__` is defined on the object, **it is called**; the value it returns is
   the result itself.
2. If not defined, `__len__` **is called**; if the result is greater than zero, the
   object is counted true.
3. If neither is defined, no method is called at all and the object is **always
   true**.

The third rung is the most overlooked part of the rule. An empty object being counted
false is not a language rule, it is **the object's own declaration**: if there is no
method declaring it, there is no concept of emptiness either. An ordinary object,
whatever it contains, is true.

#### A Chain Is a Shorthand

Comparison operators are chainable in Python: `a < b < c` is a valid expression. This
notation does **not** mean a left-to-right pairwise comparison — if it did, `a < b`
would produce a truth value, that value would be compared with `c`, and the result
would come out meaningless.

The chain means the expression `a < b and b < c`, with two differences. First, the
middle term `b` is evaluated **once**. In the expanded notation `b` is written twice,
and if it is a function call, it runs twice. Second, like `and`, a chain also
**short-circuits**: if the first comparison comes out false, the second is never even
built, and the right-hand term is not even produced.

The chaining rule applies not only to `<` and `>` but to **every comparison
operator**; membership and identity operators are in the same class. This is why
`a is b is c` is as valid a chain as `a < b <= c` and carries the same two
guarantees. A chain has no length limit: each additional term builds its own
comparison with the one before it, and never comes up unless everything to its left
has come out true.

Neither of these is visible to the eye. The only way to see how many times terms are
produced is to build a rig that counts production.

#### What Short-Circuiting Returns

The `and` and `or` operators do not return a logical value; they return **one of the
operands**. The expression `x or y` gives `x` if `x` is true, `y` otherwise. What is
returned is the object itself, not its truth value.

The protocol the two call is separate too: `and` and `or` do not call their own
special method on the operands the way an arithmetic operator does. All they do is put
**the left operand through the truthiness rule** and decide, based on the result,
which operand to return. The right operand is returned if needed — it is not tested.

The measurement's assumptions:

- **CF1** — The oracle is the rig itself: which special method gets called is known
  because the object records it itself; the measurement is not an outside
  observation, it is the object's own declaration.
- **CF2** — Three classes are used. `Tracker` defines only `__len__`, `NonEmpty` adds
  `__bool__` to it, `Silent` defines neither. The three classes correspond exactly to
  the three rungs of the truthiness rule.
- **CF3** — Each class is tested in an empty and a three-item state; the item count
  changes not to change the result, but to distinguish **who determines the result**.
- **CF4** — The chain's middle term is produced by a function call, and every
  production is logged; this log is the measure for the question "how many times was
  it evaluated."
- **CF5** — `__lt__` compares by length. **What** the comparison is based on does not
  enter the measurement; what is measured is **how many times** it is called.
- **CF6** — `Tracker` defines only `__lt__`, its counterpart `__gt__` is deliberately
  left undefined, so that reflection can be measured.
- **CF7** — The calls counted come only from the forms this lesson builds; other forms
  in the shared reference are not part of this measurement.

#### Measurement

```python
"""Truthiness: which special method does `if n` call, in what order."""

LOG = []


def record(name):
    LOG.append(name)


class Tracker:
    """__len__ defined, __bool__ undefined."""

    def __init__(self, items=(1, 2, 3)):
        self.items = list(items)

    def __len__(self):
        record("__len__")
        return len(self.items)

    def __lt__(self, other):
        record("__lt__")
        return len(self.items) < len(other.items)


class NonEmpty(Tracker):
    """__bool__ defined: truthiness does not fall to __len__."""

    def __bool__(self):
        record("__bool__")
        return len(self.items) > 0


class Silent:
    """Neither __bool__ nor __len__ defined."""

    def __init__(self, items=()):
        self.items = list(items)


def measure(action):
    LOG.clear()
    try:
        value = action()
    except Exception as e:
        LOG.append(f"!{type(e).__name__}")
        value = None
    return list(LOG), value


OBJECTS = (
    ("Tracker(3 items)", lambda: Tracker((1, 2, 3))),
    ("Tracker(0 items)", lambda: Tracker(())),
    ("NonEmpty(3 items)", lambda: NonEmpty((1, 2, 3))),
    ("NonEmpty(0 items)", lambda: NonEmpty(())),
    ("Silent(3 items)", lambda: Silent((1, 2, 3))),
    ("Silent(0 items)", lambda: Silent(())),
)

print(f"{'object':<19s} {'if n':>6s} {'not n':>6s}  protocol called")
total = trials = silent = 0
for name, build in OBJECTS:
    n = build()
    c1, d1 = measure(lambda: bool(n))
    c2, d2 = measure(lambda: not n)
    total += len(c1) + len(c2)
    trials += 2
    silent += (len(c1) == 0) + (len(c2) == 0)
    print(f"  {name:<17s} {str(d1):>6s} {str(d2):>6s}  {' '.join(c1) or '(no method)'}")
print(f"\ntrials {trials}, calls {total}, calling no method {silent}")

PRODUCED = []


def produce(name, n):
    PRODUCED.append(name)
    return Tracker(range(n))


def chained():
    return produce("a", 1) < produce("b", 2) < produce("c", 3)


def expanded():
    return (produce("a", 1) < produce("b", 2)) and (produce("b", 2) < produce("c", 3))


def short_circuit():
    return produce("a", 3) < produce("b", 2) < produce("c", 9)


print()
print(f"{'form':<17s} {'result':>6s} {'__lt__':>7s} {'produced':>9s}  order")
for name, f in (("a<b<c", chained), ("a<b and b<c", expanded),
                ("a<b<c  (false)", short_circuit)):
    PRODUCED.clear()
    c, d = measure(f)
    print(f"  {name:<15s} {str(d):>6s} {c.count('__lt__'):7d}"
          f" {len(PRODUCED):9d}  {' '.join(PRODUCED)}")

print()
print(f"{'comparison':<25s} {'result':>6s}  protocol called")
for name, f in (("Tracker(1) < Tracker(2)", lambda: Tracker((1,)) < Tracker((1, 2))),
              ("Tracker(2) > Tracker(1)", lambda: Tracker((1, 2)) > Tracker((1,))),
              ("Silent(1) < Silent(2)", lambda: Silent((1,)) < Silent((1, 2)))):
    c, d = measure(f)
    print(f"  {name:<23s} {str(d):>6s}  {' '.join(c)}")

print()
print(f"{'expression':<32s} {'calls':>6s}  {'returned type':<16s} protocol")
EMPTY, FULL = Tracker(()), NonEmpty((1, 2))
for name, f in (("Tracker(0) or NonEmpty(2)", lambda: EMPTY or FULL),
              ("NonEmpty(2) or Tracker(0)", lambda: FULL or EMPTY),
              ("Tracker(0) and NonEmpty(2)", lambda: EMPTY and FULL),
              ("bool(Tracker(0) or Tracker(0))", lambda: bool(Tracker(()) or Tracker(())))):
    c, d = measure(f)
    print(f"  {name:<30s} {len(c):6d}  {type(d).__name__:<16s} {' '.join(c)}")
```

```
object                if n  not n  protocol called
  Tracker(3 items)    True  False  __len__
  Tracker(0 items)   False   True  __len__
  NonEmpty(3 items)   True  False  __bool__
  NonEmpty(0 items)  False   True  __bool__
  Silent(3 items)     True  False  (no method)
  Silent(0 items)     True  False  (no method)

trials 12, calls 8, calling no method 4

form              result  __lt__  produced  order
  a<b<c             True       2         3  a b c
  a<b and b<c       True       2         4  a b b c
  a<b<c  (false)   False       1         2  a b

comparison                result  protocol called
  Tracker(1) < Tracker(2)   True  __lt__
  Tracker(2) > Tracker(1)   True  __lt__
  Silent(1) < Silent(2)     None  !TypeError

expression                        calls  returned type    protocol
  Tracker(0) or NonEmpty(2)           1  NonEmpty         __len__
  NonEmpty(2) or Tracker(0)           1  NonEmpty         __bool__
  Tracker(0) and NonEmpty(2)          1  Tracker          __len__
  bool(Tracker(0) or Tracker(0))      2  bool             __len__ __len__
```

#### The Cost of the Three Rungs

The top table lays the truthiness rule's three rungs side by side. There are **eight
calls across twelve trials**; the remaining four trials produce a result **without
calling any method**.

The first two rows are the second rung: because `Tracker` does not define `__bool__`,
the trial falls to `__len__`, and the item count determines the result. The third and
fourth rows are the first rung: the moment `__bool__` is added to the same class,
`__len__` is **never called at all**. When both are defined, there is no question of
which method gets called — if `__bool__` exists, `__len__` takes no part in the
trial.

The last two rows are the third rung, and this is where the table's real result
shows. `Silent(0 items)` is **empty**, but the trial gives **true** — and it does this
not by calling a method, but by **calling none at all**. Whether the object has three
items or zero does not change the result, because there is no method to produce the
result. The practical consequence: a program testing an empty instance of its own
class with `if`, if it forgot to write `__len__` or `__bool__`, always takes the
**true** branch, and nothing signals that this happened.

The `not n` column shows the same chain running once more: negation is not a separate
protocol, it is the reverse of the same trial's result. The only difference between
the columns shows up in the `Silent` rows — `not` returns not an object, but a real
logical value.

#### How Many Terms a Chain Produces

The middle table shows in numbers that a chain is a shorthand. `a<b<c` and the
expanded notation call `__lt__` the same number of times: **both 2**. There is no
gain in comparison count.

The difference is in the produced column. The chain produces three terms, the
expanded notation **four**: `b` is produced twice. This is a difference not of
readability but of **meaning**. If the middle term is a function call, that function
runs twice; if it has a side effect, the side effect happens twice; if it returns a
different value on each call, the two comparisons test **two separate values unaware
of each other**. The expanded notation does not give the guarantee the chain gives.

The third row measures short-circuiting. Because the first comparison comes out
false, `__lt__` is called **1** time and production stops at **2**: `c` is never
built at all. The term at the chain's right end is something that will not be
produced without looking to its left.

#### The Protocol Found in the Reverse Direction

The third table shows comparison's own fallback rule, and the middle row is the
lesson's sharpest result. The `Tracker` class defines `__lt__`, not `__gt__`. Yet
`Tracker(2) > Tracker(1)` runs — and the method called is **`__lt__`**.

The rule is this: if the left operand's method cannot be found in a comparison, the
language tries the **right operand's counterpart method**. For `a > b`, `__gt__` is
first sought on `a`; if absent, `__lt__` is called on `b` and the operands swap
places. The result comes out correct, because "`b` is less than `a`" and "`a` is
greater than `b`" are the same claim.

The direct consequence of this is the second claim itself: **the operator written
does not say the name of the method called.** Two rows give `True`, both call the
same method, but one writes `<` and the other `>`. It is impossible to guess which
method ran by looking at the syntax; only measurement tells you.

The third row shows where the rule ends. `Silent` defines neither `__lt__` nor
`__gt__`; no method is found on either the left or the right operand, and the
comparison produces not a result but a `TypeError`. While the third rung of the
truthiness trial silently returns true, comparison's last rung ends in an
**exception**: one syntax has a default behavior, the other does not. An object with
no protocol does not carry that syntax at all.

#### The Condition Statement and the Conditional Expression

The truthiness rule does not run only in the `if` statement. The same reduction runs
in a `while` condition, in the `and` and `or` operators, in the `not` operator, and in
the **conditional expression** too.

The conditional expression uses the statement-versus-expression distinction from the
previous topic: `if` is a **statement**, it produces no value and cannot be written on
the right side of an assignment. Its expression counterpart is the `a if k else b`
notation, and it **produces a value** — so it can be placed on the right side of an
assignment, in place of an argument, or inside a list.

The difference between the two is only in notation; the truthiness rule runs the same
three rungs in both, and the object written in place of `k` goes through the same
measurement. Short-circuiting is preserved too: in a conditional expression, only
**one** of the `a` and `b` branches is evaluated. This is what separates a conditional
expression from `and`/`or` chains — there, the returned value was one of the
operands; here it is the chosen branch itself, and the branch not chosen is never
built at all.

#### The Type of the Returned Object

The bottom table shows what `and` and `or` return, and in all three of its rows, what
comes back is **not a logical value**.

First row: the left operand `Tracker(0)` is empty, the trial comes out false via
`__len__`, and `or` returns the right operand — the returned type is `NonEmpty`.
Second row: this time the left operand is true, tested via `__bool__`, and `or`
returns **the left operand**. Third row: `and` does the reverse: because the left
operand is false, the result is directly **it**, type `Tracker`.

All three rows' call count is **1**. The right operand is never tested; it is only
returned. The consequence of this is exactly what the default-value pattern written
with `or` does in Python: if the left operand comes out false by the truthiness rule,
the right one is given — not because it is **empty**, but because it is **counted
false**.

The last row separates the two. When the `bool()` built-in steps in, the returned type
really becomes `bool` and the call count rises to **2**: once `or` tests the left
operand, once `bool()` tests the result. The cost of turning the same expression into
a logical value is one extra protocol call.

#### Summary

- In Python a condition does not expect a logical value; every object after `if` is
  put through the **truthiness rule**, and what is measured is which special method
  this reduction calls.
- The rule has three rungs: if `__bool__` exists it is called, otherwise it falls to
  `__len__`, and if that is also absent the object is counted true **without calling
  any method** — four of the twelve trials fall on this rung, and even an empty
  object comes back true.
- While `__bool__` is defined, `__len__` takes no part in the trial at all; the two
  methods do not compete, the first disables the second.
- A comparison chain `a<b<c` and the expanded notation call `__lt__` the same number
  of times (**2**), but the chain produces the middle term **once**, the expanded
  notation **twice**; when the chain comes out false, the right term is never
  produced.
- Comparison has its own fallback rule: if the left operand has no method, the right
  operand's counterpart is called — an expression written with `>` can run `__lt__`;
  if neither exists, the result is not a value but a `TypeError`.
- `and` and `or` do not return a logical value, they return **one of the operands**;
  they put only the left operand through the truthiness rule and never test the
  right one.

#### Next Step

The truthiness rule asks one question once and ends with one branch. A loop, on the
other hand, asks not the same question but **another question over and over**: is
there an item next in line? In the Programming Fundamentals course a loop was a
control structure; how many turns it took depended on a condition or a collection's
length. In Python a loop is a **protocol**, and what decides the turns are ending is
not a condition. The next lesson counts how many times a loop calls `__next__` on a
three-item object — and why that count is one more than the item count.

### Loops

Source: https://academia.sh/en/courses/python-fundamentals/loops

A loop is not a control structure, it is a protocol: on a three-item object, __next__ is called four times, the last one throws StopIteration, and what ends the loop is not a condition but that exception.

The previous lesson measured the truthiness rule: the object after `if` is put
through a question, and that question's answer ends with a branch. The trial happens
once, the result is read once.

A loop does not repeat the same question, it repeats **a different question**: is
there an item next in line? The Programming Fundamentals course's loops lesson built
this structure as a **control structure** — the number of turns depended either on a
collection's length or on a condition, and the loop's termination was guaranteed by a
change on each turn that moved toward falsifying the condition. In Python a loop is a
**protocol**, and that lesson's reason for termination does not apply here: a `for`
loop has no condition it tests. So what decides the turns have ended? This is what
this lesson measures — and the answer turns out to be an exception class.

#### What a Loop Calls

The notation `for x in n` has three steps, and all three are hidden. First an
**iterator** is requested from the object: `__iter__` is called. Then on every turn
the next item is requested from that iterator: `__next__` is called. The third step
is termination — and what announces it is not a return value.

It could not have been a return value. `__next__` can return any object; whatever
value we chose, that value could be a real item of the collection, and "done" would
get mixed up with "here is your item." This is why termination is taken out of the
channel: `__next__` throws an **exception** when it has no items left to give —
`StopIteration`. This is what ends the loop.

There is a direct consequence: one last call, a call producing no item, is always
made. Regardless of item count, `__next__` is called **one extra time**, and that
extra call is the loop's termination announcement.

#### Writing the Same Loop by Hand

The way to show the protocol is really a shorthand is to do the same job without
`for`: take the iterator with `iter()`, set up an infinite loop, call `next()` on
every turn, and exit when `StopIteration` is caught. If the two notations produce the
same protocol sequence, `for` really is the shorthand for these three steps.

Two more objects enter the measurement. `Tracker` defines both `__iter__` and
`__getitem__`; either is enough for iteration, but only counting tells which one is
used. `Indexed` defines only `__getitem__` and has no `__iter__` — whether such an
object can even enter a loop, and if so which protocol it follows, is the second
question.

The measurement's assumptions:

- **CF8** — The oracle is the rig itself: it logs the name every time a special
  method is called; the measure for "how many times was it called" is this log.
- **CF9** — `Tracker` keeps the behavior from the shared reference: `__iter__` resets
  the counter and returns itself, `__next__` throws `StopIteration` once items run
  out.
- **CF10** — The item count is varied from zero to four; what is measured is not the
  count itself but **the gap between it and the call count**.
- **CF11** — `break` is run on the second item; the break point's location is fixed
  and chosen to isolate the effect early termination has on the call count.
- **CF12** — The measurement uses no list-building notation, only an explicit `for`
  body; its purpose is guaranteeing the counted calls come only from the iteration
  protocol.
- **CF13** — `Indexed` defines only `__getitem__`; the absence of `__iter__` is
  deliberate and needed so the second protocol can be measured.
- **CF14** — In the `while` measurement, the body removes one item every turn; the
  guarantee of termination comes from this, and the trial count and turn count are
  read separately.
- **CF15** — The nested-loop measurement is done in two states: the two loops on the
  **same** object, and on **separate** objects. `Tracker.__iter__` returns itself and
  resets the counter as in the shared reference; what is measured is the effect these
  two choices have on the turn count.

#### Measurement

```python
"""Iteration protocol: how many times does a loop call __next__, and what ends it."""

LOG = []


def record(name):
    LOG.append(name)


class Tracker:
    """Both __iter__ and __getitem__ defined."""

    def __init__(self, items=(1, 2, 3)):
        self.items = list(items)

    def __iter__(self):
        record("__iter__")
        self._i = 0
        return self

    def __next__(self):
        record("__next__")
        if self._i >= len(self.items):
            raise StopIteration
        value = self.items[self._i]
        self._i += 1
        return value

    def __getitem__(self, k):
        record("__getitem__")
        return self.items[k]

    def __len__(self):
        record("__len__")
        return len(self.items)


class Indexed:
    """Only __getitem__ defined: no __iter__."""

    def __init__(self, items=(1, 2, 3)):
        self.items = list(items)

    def __getitem__(self, k):
        record("__getitem__")
        return self.items[k]


def measure(action):
    LOG.clear()
    try:
        action()
    except Exception as e:
        LOG.append(f"!{type(e).__name__}")
    return list(LOG)


LAPS = []


def complete():
    for x in Tracker():
        LAPS.append(x)


def interrupted():
    for x in Tracker():
        LAPS.append(x)
        if x == 2:
            break


def finished_else():
    for x in Tracker():
        LAPS.append(x)
    else:
        record("else")


def interrupted_else():
    for x in Tracker():
        LAPS.append(x)
        if x == 2:
            break
    else:
        record("else")


def manual():
    y = iter(Tracker())
    while True:
        try:
            LAPS.append(next(y))
        except StopIteration:
            break


def fourth():
    y = iter(Tracker())
    for _ in range(4):
        next(y)


FORMS = (("for x in n", complete), ("for + break", interrupted),
         ("for + else", finished_else), ("for + break + else", interrupted_else),
         ("iter/next manual", manual), ("fourth next", fourth))

print(f"{'form':<20s} {'laps':>4s} {'__next__':>9s} {'else':>5s}  protocol sequence")
for name, f in FORMS:
    LAPS.clear()
    c = measure(f)
    p = [a for a in c if a != "else"]
    print(f"  {name:<18s} {len(LAPS):4d} {c.count('__next__'):9d}"
          f" {('yes' if 'else' in c else '-'):>5s}  {' '.join(p)}")

print()
print(f"{'item':>4s} {'laps':>4s} {'__next__':>9s}  diff")
for n in range(5):
    LAPS.clear()

    def count(n=n):
        for x in Tracker(range(n)):
            LAPS.append(x)
    c = measure(count)
    print(f"  {n:2d} {len(LAPS):4d} {c.count('__next__'):9d}"
          f"  {c.count('__next__') - n:+d}")

print()
print(f"{'object':<10s} {'defines':<24s} {'calls':>6s}  protocol sequence")
for name, build, defines in (
        ("Tracker", Tracker, "__iter__ and __getitem__"),
        ("Indexed", Indexed, "only __getitem__")):
    def walk(k=build):
        for _ in k():
            pass
    c = measure(walk)
    print(f"  {name:<8s} {defines:<24s} {len(c):6d}  {' '.join(c)}")
print("  fourth index directly:", " ".join(measure(lambda: Indexed()[3])))


OUTER = []


def nested_same():
    n = Tracker()
    for x in n:
        OUTER.append(x)
        for y in n:
            LAPS.append(y)


def nested_separate():
    for x in Tracker():
        OUTER.append(x)
        for y in Tracker():
            LAPS.append(y)


print()
print(f"{'nested loop':<18s} {'outer laps':>10s} {'inner laps':>10s}"
      f" {'__iter__':>9s} {'__next__':>9s}")
for name, f in (("same object", nested_same), ("separate objects", nested_separate)):
    LAPS.clear()
    OUTER.clear()
    c = measure(f)
    print(f"  {name:<16s} {len(OUTER):10d} {len(LAPS):10d}"
          f" {c.count('__iter__'):9d} {c.count('__next__'):9d}")


def while_loop():
    n = Tracker()
    while n:
        n.items.pop()


print()
c = measure(while_loop)
print(f"while n  (until 3 items empty): calls {len(c)}  {' '.join(c)}")
```

```
form                 laps  __next__  else  protocol sequence
  for x in n            3         4     -  __iter__ __next__ __next__ __next__ __next__
  for + break           2         2     -  __iter__ __next__ __next__
  for + else            3         4   yes  __iter__ __next__ __next__ __next__ __next__
  for + break + else    2         2     -  __iter__ __next__ __next__
  iter/next manual      3         4     -  __iter__ __next__ __next__ __next__ __next__
  fourth next           0         4     -  __iter__ __next__ __next__ __next__ __next__ !StopIteration

item laps  __next__  diff
   0    0         1  +1
   1    1         2  +1
   2    2         3  +1
   3    3         4  +1
   4    4         5  +1

object     defines                   calls  protocol sequence
  Tracker  __iter__ and __getitem__      5  __iter__ __next__ __next__ __next__ __next__
  Indexed  only __getitem__              4  __getitem__ __getitem__ __getitem__ __getitem__
  fourth index directly: __getitem__ !IndexError

nested loop        outer laps inner laps  __iter__  __next__
  same object               1          3         2         6
  separate objects          3          9         4        16

while n  (until 3 items empty): calls 4  __len__ __len__ __len__ __len__
```

#### One Extra Call

The top table's first row is the shared reference's third claim. On a three-item
object, the loop takes **three laps**, but `__next__` is called **four times**. The
extra fourth call produces no item; it exists to announce that it produced none.

The second table shows this is not tied to a single instance. As item count rises
from zero to four, the diff column does not change: **+1 on every row**. There is a
call even on a zero-item object — even a loop that never takes a lap has to ask once,
because it cannot know it is empty without asking. Call count is not a function of
item count, it is **item count plus the termination announcement**.

The last row shows that announcement directly. When the iterator is taken by hand and
`next()` is called four times, the fourth does not return a value, **it throws
`StopIteration`**. We do not see this exception in a `for` loop because the loop
catches it itself and silently carries flow out of the body. Not seeing it does not
mean it is absent: **every normal termination of a loop gives birth to an exception,
and that exception is caught.**

The reading that follows is the shared reference's third claim: the exception is not
an error, **it is part of the protocol**. If it were an error it would announce an
exceptional state; instead this exception announces the most ordinary state — the
collection ran out.

#### The Shorthand Expanded

The fifth row proves the `for` notation really is a shorthand. The hand-written
form — `iter()`, an infinite loop, `next()`, catching `StopIteration` — produces
**exactly the same protocol sequence**: one `__iter__`, four `__next__`. There is no
execution difference between the two notations; the only difference is that the
`try` block is visible.

The second and fourth rows measure a loop cut off by `break`, and the count here
stands apart from all the rest: **two laps, two calls**. There is **no** extra call.
`break` ends the loop from outside the protocol; the third item is never requested,
`StopIteration` is never born. A cut-off loop is the only form where the one extra
call disappears.

The same topic's other control statement, `continue`, needs no separate row in this
table, because it does not touch the protocol at all: it skips the rest of the body
and carries the loop to the **next `__next__` call**. The lap count does not change,
the extra call stays in place. This is exactly the difference between it and
`break` — one skips a step inside the protocol, the other leaves the protocol.

This also explains the `else` column. A loop's attached `else` block runs when the
loop ends **by its own protocol**; it does not run when cut off by `break`. In the
third row `else` runs, in the fourth it does not. In other words, `else` is the
answer to the question "did it finish with no `break` at all?" — put differently,
**did `StopIteration` really get thrown?** The search pattern that `break`s on
finding a sought item and behaves in the `else` block when not found is built
directly on this distinction.

#### The Second Protocol

The third table measures which protocol gets chosen, and it repeats the previous
lesson's pattern.

`Tracker` defines both `__iter__` and `__getitem__`. The loop **chooses `__iter__`**,
and `__getitem__` is never called. Just as `__bool__` disabled `__len__` in the
truthiness rule, here `__iter__` disables `__getitem__`; the two methods do not
compete, the first shuts off the second.

The `Indexed` row shows the second rung. This object has no `__iter__`, but the loop
still runs: `__getitem__` is called **four times** with rising indices starting from
zero. For an object with no iterator, the language recognizes a second path of
iteration — request indices in order, until an index turns out invalid.

The two roles have to be separated. An **iterable** object is one that defines
`__iter__`; it is not walked itself, it gives what will walk it. An **iterator** is
one that defines `__next__`, and it is the one that actually does the walking. It is
not forbidden for one object to define both — `Tracker` does exactly this and returns
itself in the body of `__iter__`. This is why `Tracker` is both iterable and an
iterator; what happens when that distinction collapses is measured below.

The count is again **four**, one more than the item count. But the exception ending
the extra call is different this time: the last row calls `Indexed()[3]` directly,
and what comes out is not `StopIteration` but **`IndexError`**. Two protocols, two
separate termination exceptions, the same **+1** pattern. In both paths, what tells
the loop it has ended is not a return value but a thrown exception.

#### Where the Multiplication Rule Breaks

The fourth table measures two nested loops and tests the rule the Programming
Fundamentals course gave for nested loops — that the total lap count is the
**product** of the two. On separate objects the rule holds: outer **3** laps, inner
**9** laps, that is, three times three.

It does not hold on the same object. The outer loop takes not 3 but **1** lap; the
inner takes not 9 but **3**. The multiplication rule collapses, and the reason is not
the loop itself but what `__iter__` returns.

`Tracker.__iter__` does not produce a new object; it returns **itself** and resets
the counter. After the outer loop takes its first item, the inner loop calls
`__iter__` on the same object, the counter resets, the inner loop consumes all three
items and drives the counter to the end. When the outer loop asks for its next item,
the counter is already at the end: `StopIteration` arrives and the outer loop ends on
its first lap. The two loops **share the same counter**.

The distinction here is that an iterable object and an iterator are separate things.
An object returning a **fresh iterator** on every `__iter__` call behaves as expected
in nested loops; one returning itself cannot be walked more than once at the same
time. The syntax is identical in both cases, the lap count is not — and the only
thing that says which behavior applies is the body of `__iter__`. It is impossible to
see this distinction by looking at the object's type, name, or item count.

The last row gives the same pattern for `while`. A loop running until three items are
emptied calls `__len__` **four times**: three trials for three laps, plus a fourth
trial that falsifies the condition. `for` calls `__next__` one extra time, `while`
does one extra truthiness trial. In both cases, the extra one is the trial **that
ends the loop**.

#### Summary

- In Python a loop is not a control structure, it is a protocol: `for x in n` first
  calls `__iter__`, then `__next__` for every turn; it has no condition it tests.
- On a three-item object, `__next__` is called **four times**, and the difference is
  independent of item count — a call happens even on zero items, because emptiness
  can only be learned by asking.
- The extra call does not return a value, **it throws `StopIteration`**; this is what
  ends the loop, and it goes unseen because `for` catches this exception itself. The
  exception is not an error, it is part of the protocol.
- The `for` notation is shorthand for `iter`, `next`, and catching `StopIteration`;
  the hand-written expansion produces exactly the same call sequence. `break` ends the
  loop from outside the protocol and removes the extra call — the `else` block reads
  exactly this distinction.
- While `__iter__` is defined, `__getitem__` takes no part in iteration at all; on an
  object defining only `__getitem__`, the loop calls it with rising indices, and the
  exception ending this path is `IndexError`.
- The multiplication rule for nested loops breaks when `__iter__` returns itself: two
  loops on the same object share a single counter, and the outer loop takes **1** lap
  instead of **3**; on separate objects the rule still holds (**3** and **9**).

#### Next Step

A loop body most often calls a function, and that function is passed different
values on every turn. The Programming Fundamentals course built the parameter-versus-
argument distinction and the call frame; in Python, the forms of passing an argument
go beyond what that lesson assumed — positional and keyword arguments, collecting a
variable number of arguments, default values. One of these is about timing, and that
is what the next lesson measures: **when** is a parameter's default value computed —
on every call, or once? The answer, together with a mutable default, produces a
surprising number.

### Function Definition

Source: https://academia.sh/en/courses/python-fundamentals/function-definition

A default value is evaluated once when the definition is read, not on every call: five calls make one evaluation, a mutable default accumulates five items across five calls, and four of six parameters can be given only one way.

The previous lesson measured that a loop is a protocol and showed that an exception
decides when the turns end. A loop body most often calls a function; it passes that
function different values on every turn.

The Programming Fundamentals course's function lesson built the parameter-versus-
argument distinction, the return value, and the call frame; it explained **what a
function is** and reduced argument passing to a single question — which value goes
to which parameter. In Python the answer is not one-line: a parameter can be given
only by position, only by name, or by both; surplus arguments can be collected;
parameters can be bound to a default value. One of these is not about the value but
about **timing**, and that is what this lesson measures: **when** is a default value
computed — on every call, or once?

#### A Default Is an Expression

`def` is a statement, and when run it produces a function object. During that
production the body is not run — that lesson was already built — but **the
default-value expressions in the signature are run**. Their results are computed and
written onto the function object.

The meaning of this is: a default value is part of the **definition**, not the call.
It is not recomputed at call time; the object computed once when the definition is
read is **rebound** on every call. Five calls do not produce five separate defaults;
they use the same object five times.

"Once" does not mean once over the program's lifetime; it means **once every time the
definition statement runs**. A top-level definition runs once while the file is read,
so its default is computed once too. A definition written inside another function's
body, however, reruns **on every call of the outer function** and recomputes its own
default. The measured rule does not change — the default is still part of the
definition — but how many times the definition runs does.

If the value is an immutable object, this distinction is invisible. A number or a
string being the same object on every call has no consequence, because there is no
way to change it. The distinction appears once the default is **mutable**: if the
body changes it, the change carries over to the next call.

#### Sentinel Value

The pattern set against this behavior is making the default `None` and producing the
real value in the body. `None` is immutable, so sharing it has no consequence; the
body builds a fresh object on every call. This role is called a **sentinel value**:
it is not used itself, it only carries the information "no argument was given."

The sentinel need not be `None`, but there is a reason it usually is. If `None` is
also among a parameter's valid values, it cannot be chosen as the sentinel — then
"not given" and "`None` was given" cannot be told apart, and a separate object is
made the sentinel instead.

#### How Many Ways an Argument Can Be Given

The second measure is the number of argument forms. Python recognizes two markers in
a signature. A slash announces that the parameters to its left can be given **only by
positional argument**. A star announces that the parameters to its right can be given
**only by keyword argument**. What falls between the two can be given either way.

Two collectors are added to this. A starred parameter collects unmatched
**positional** arguments into a tuple; a double-starred parameter collects unmatched
**keyword** arguments into a dictionary. Together the two are called a **variadic**
argument, and they free the signature from an upper bound on how many arguments it
can take.

The order of these pieces in a signature is not free: first the positional-only ones,
then the ones giveable either way, then the positional collector, then the
keyword-only ones, and last the keyword collector. This order exists to make binding
**unambiguous**. Arguments at the call site are read left to right; positional ones
fill the signature's positional parameters in order, the ones they cannot fill go
into the collector, keyword ones match by name, and the ones that do not match fall
into the keyword collector. Break this order and the same call could have more than
one valid binding, with no way to tell from what was written which one gets chosen.

The same reasoning also requires parameters carrying a default to come after ones
that do not — on the positional side. No such constraint applies keyword-only: since
matching is by name there, parameters with and without a default can mix freely.

The measurement's assumptions:

- **CF16** — The oracle is the rig itself: the default expression is a function call
  and logs every evaluation of itself; the measure for "how many times was it
  evaluated" is this log.
- **CF17** — The evaluation count is read at two points: right after the definition
  statement runs, and after five calls are made. The difference between the two is
  the evaluation the calls added.
- **CF18** — Two functions do the same job and differ only in their default: one an
  empty list, the other a `None` sentinel. The rest of their bodies is identical.
- **CF19** — Sharing is shown by comparing the object two consecutive calls return
  with `is`; **no identity number is ever printed**. The object compared is a list;
  caching of small immutable values does not enter the measurement.
- **CF20** — The signature carries six parameters and two collectors; these counts
  are specific to this signature and do not carry over to another one.
- **CF21** — For calls that fail to bind, only the **exception's class** is printed,
  not its message; the message text is not part of the measurement.
- **CF22** — Where the default sits is shown by reading the field on the function
  object that carries defaults; what is read is not a memory fact, it is a field the
  language itself defines.
- **CF23** — The unpacking notation at a call site uses the same markers as the
  collectors in a signature but works in the reverse direction; the measurement
  compares the two on the same signature.

#### Measurement

```python
"""When is a default value evaluated, and how do arguments bind."""

EVALUATIONS = []


def stamp(name):
    """Records every time it is called; used as a default-value expression."""
    EVALUATIONS.append(name)
    return len(EVALUATIONS)


def order(x, d=stamp("default expression")):
    return d


print(f"evaluations right after definition: {len(EVALUATIONS)}")
results = [order(i) for i in range(5)]
print(f"evaluations after five calls: {len(EVALUATIONS)}")
print(f"what five calls returned: {results}")


def accumulate(x, bucket=[]):
    bucket.append(x)
    return bucket


def sentinel(x, bucket=None):
    if bucket is None:
        bucket = []
    bucket.append(x)
    return bucket


print()
print(f"{'call':>6s} {'bucket=[] length':>17s} {'bucket=None length':>19s}")
previous_a = previous_b = None
for i in range(1, 6):
    a, b = accumulate(i), sentinel(i)
    print(f"  {i:4d} {len(a):17d} {len(b):19d}")
    if i > 1:
        same_a, same_b = a is previous_a, b is previous_b
    previous_a, previous_b = a, b
print(f"is two consecutive calls' bucket the same object: "
      f"bucket=[] {same_a}, bucket=None {same_b}")
print(f"the default sitting on the function object: "
      f"accumulate {accumulate.__defaults__}, sentinel {sentinel.__defaults__}")
print(f"is the returned list the very default sitting on the function: "
      f"{accumulate(6) is accumulate.__defaults__[0]}")


def register(a, b, /, c, d=40, *extra, e, f=60, **rest):
    """a,b positional-only; e,f keyword-only; extra and rest collect the surplus."""
    return {"a": a, "b": b, "c": c, "d": d, "extra": extra, "e": e, "f": f,
            "rest": rest}


CALLS = (
    "register(1, 2, 3, e=5)",
    "register(1, 2, c=3, e=5)",
    "register(1, 2, 3, 4, 7, 8, e=5, f=9, g=0)",
    "register(1, 2, 3)",
    "register(a=1, b=2, c=3, e=5)",
)

print()
print(f"{'call':<44s} binding")
for text in CALLS:
    try:
        result = eval(text)
        bound = (f"a={result['a']} b={result['b']} c={result['c']} d={result['d']} "
                 f"extra={result['extra']} e={result['e']} f={result['f']} rest={result['rest']}")
    except TypeError as error:
        bound = type(error).__name__
    print(f"  {text:<42s} {bound}")

POSITIONAL = (1, 2, 3)
KEYWORD = {"e": 5, "g": 0}
unpacked = register(*POSITIONAL, **KEYWORD)
print(f"  {'register(*POSITIONAL, **KEYWORD)':<42s} "
      f"a={unpacked['a']} b={unpacked['b']} c={unpacked['c']} d={unpacked['d']} "
      f"extra={unpacked['extra']} e={unpacked['e']} f={unpacked['f']} "
      f"rest={unpacked['rest']}")

print()
print("parameter  positional-giveable  keyword-giveable")
for name, positional, keyword in (("a", True, False), ("b", True, False),
                                ("c", True, True), ("d", True, True),
                                ("e", False, True), ("f", False, True)):
    print(f"  {name:<9s} {('yes' if positional else 'no'):>17s}"
          f" {('yes' if keyword else 'no'):>18s}")
```

```
evaluations right after definition: 1
evaluations after five calls: 1
what five calls returned: [1, 1, 1, 1, 1]

  call  bucket=[] length  bucket=None length
     1                 1                   1
     2                 2                   1
     3                 3                   1
     4                 4                   1
     5                 5                   1
is two consecutive calls' bucket the same object: bucket=[] True, bucket=None False
the default sitting on the function object: accumulate ([1, 2, 3, 4, 5],), sentinel (None,)
is the returned list the very default sitting on the function: True

call                                         binding
  register(1, 2, 3, e=5)                     a=1 b=2 c=3 d=40 extra=() e=5 f=60 rest={}
  register(1, 2, c=3, e=5)                   a=1 b=2 c=3 d=40 extra=() e=5 f=60 rest={}
  register(1, 2, 3, 4, 7, 8, e=5, f=9, g=0)  a=1 b=2 c=3 d=4 extra=(7, 8) e=5 f=9 rest={'g': 0}
  register(1, 2, 3)                          TypeError
  register(a=1, b=2, c=3, e=5)               TypeError
  register(*POSITIONAL, **KEYWORD)           a=1 b=2 c=3 d=40 extra=() e=5 f=60 rest={'g': 0}

parameter  positional-giveable  keyword-giveable
  a                       yes                 no
  b                       yes                 no
  c                       yes                yes
  d                       yes                yes
  e                        no                yes
  f                        no                yes
```

#### An Expression Evaluated Once

The first three lines close the timing question. Right after the definition
statement runs, **before any call is made yet**, the evaluation count is **1**. The
default expression ran while the definition was read.

After five calls the count is still **1**. The calls added no evaluation. The third
line shows this directly: all five calls return the same value — every one of them
sees the single object computed at definition time.

There is **no link** between how many times a function is called and how many times
its default is evaluated. This is why it is wrong to read an expression written in
the signature as if it were written in the body: one runs once, the other on every
call.

#### The Shared Bucket

The middle table makes this single evaluation pay its cost. In the left column the
default is an empty list; over five calls the length grows **1, 2, 3, 4, 5**. Every
call finds the list the previous one left behind.

In the right column, the sentinel pattern; the length stays **1** across all five
calls. Every call builds its own list and inherits nothing from the one before.

The bottom line states the reason through identity. The object two consecutive calls
return in the `bucket=[]` form comes out **true** under the `is` test: the same
object. In the sentinel pattern, **false**: separate objects. This is where the
assumption that a function gets "a new list" on every call breaks — the list was
built once at definition time, bound to the function object, and from that moment on
lives as a single object that every call mutates.

The second-to-last line shows **where** the default sits, and this is the lesson's
most direct proof. When the field on the function object carrying defaults is read,
it holds `[1, 2, 3, 4, 5]` — the list five calls accumulated, sitting, after the
calls are done, **on the function itself**. In the sentinel pattern the same field
holds only `None`; nothing accumulates there, because the accumulating lists were
built in the body and left behind with the call.

The last line closes the loop: a call's returned list and the default sitting on the
function object come out **true** under the `is` test. The two are a single object.
The sentence "the default is part of the definition" stops being a comment here; it
becomes something readable off the function itself.

The result itself is not a bug, it is the rule's direct consequence. The Programming
Fundamentals course wrote this trap as a warning; the numbers here give the reason
for it. For someone who knows the rule, the behavior is not surprising: if the body
changes the default, what it changes **is part of the definition**.

#### Six Parameters, Two Markers

The bottom two tables measure binding. The first two calls give the same result: `c`
is given once by position, once by name, and the binding is exactly the same. Because
`d`, `e`, and `f` are not given, `d` and `f` fall to their defaults; `e` has to be
written at the call site regardless, because it has no default and can only be given
by keyword.

The third call fills both collectors. Six positional arguments are given; the first
four satisfy `a`, `b`, `c`, `d`, and the surplus **two** go into the `extra` tuple.
On the keyword side, `e` and `f` find their own parameters, and `g`, which has no
match in the signature, falls into the `rest` dictionary. Collectors are parameters
that **collect** an argument that cannot bind, instead of turning it into an error.

The last two calls fail to bind, and both give `TypeError`. The fourth is missing
`e`: a call cannot be built without a keyword-only parameter that has no default. The
fifth has the opposite problem — `a` and `b` are given by name, but sit left of the
slash and can only be taken by position. Given by name, they are seen not as
satisfying the parameters but as names that would fall into `rest`, leaving the two
positional parameters empty.

The sixth line shows what the same markers do **at the call site**. While a
signature **collects** with the star markers, a call does the exact opposite: it
**unpacks** a tuple into positional arguments and a dictionary into keyword
arguments. The three-item tuple satisfies `a`, `b`, `c`; `e` in the dictionary finds
its own parameter, and `g` falls into the `rest` dictionary.

That the two directions use the same markers is not a coincidence: collecting and
unpacking are each other's inverse, and passing a function's received arguments on to
another function unchanged means using both side by side. This is the basic pattern
for a wrapping function — it collects arguments without knowing the signature and
unpacks them into the function it wraps.

The last table summarizes the whole signature. **Four of the six** parameters can be
given only one way — `a` and `b` only by position, `e` and `f` only by name. Only
**two** accept both ways. This is what the markers do: they narrow how many ways a
parameter can be given. The payoff of narrowing is that the call notation does not
change along with the signature — when a parameter kept positional-only is renamed,
no call breaks, and the same holds when a parameter kept keyword-only changes order.

#### Summary

- A default-value expression is part of the **definition**, not the call: it is
  evaluated once while the definition is read, and five calls do not change that
  count — there is no link between call count and evaluation count.
- This is invisible with an immutable default; with a **mutable** one, the body
  changes part of the definition, and five calls accumulate **five items** in a
  single list.
- The `None`-sentinel pattern builds a fresh object in the body; the bucket of two
  consecutive calls comes out as separate objects under the `is` test, and the
  length stays **1** on every call.
- A slash makes the parameters to its left positional-only, a star makes the ones to
  its right keyword-only; in the measured signature, **four of six** parameters can
  be given only one way, two either way.
- Starred and double-starred collectors gather unmatched positional and keyword
  arguments; the same markers work in reverse at a call site, unpacking a tuple and
  a dictionary into arguments. An argument that cannot be collected and a parameter
  left unfilled both produce `TypeError`.
- The default sits in the function object's own field: after five calls, the entire
  list accumulated in that field can be read, and the returned list comes out as the
  same object under the `is` test.

#### Next Step

A function body uses names beyond the ones arriving with arguments: ones built in the
body, ones coming from the definition that encloses it, ones sitting at the outermost
level of the file, and ones the language itself supplies. The Programming
Fundamentals course built this search order as a language-independent concept. In
Python the order has **four levels**, and what decides which level a name is found in
is not where it is **read**, but where it is **assigned**. The next lesson defines the
same name at all four levels at once and measures which one wins — and how adding a
single assignment line to a body moves that same name to a different level.

### Scope Rules

Source: https://academia.sh/en/courses/python-fundamentals/scope-rules

Name lookup runs through four levels, and what decides which level it stops at is not where the name is read but where it is assigned: adding a single assignment line to a body turns the same read from a valid value into UnboundLocalError.

The previous lesson measured how arguments bind to parameters. Every bound argument
establishes a **name** in the body; but the body does not only use those names. Names
it builds itself, ones coming from the enclosing definition, ones sitting at the
outermost level of the file, and ones the language itself supplies are there too —
and all of them are read with the same notation, a bare name.

The Programming Fundamentals course's scope lesson built this search order as a
language-independent **concept**: start local, go to enclosing then global, end with
built-ins, first match wins. That lesson defined scope through visibility, lifetime
through duration, and explained shadowing. Here the same order is measured as a
**search chain**, focused on one question: what decides which level a name is
searched at? The answer runs against intuition — not where the name is **read**, but
whether it is **assigned** somewhere in the body.

#### Four Levels

In Python, name lookup runs through four levels, and their order is fixed:

1. **Local** — the function body currently executing.
2. **Enclosing** — the function bodies surrounding it, inside out.
3. **Global** — the file's outermost level.
4. **Built-in** — names the language itself supplies.

The lookup stops at the first place it finds the name and never looks at the levels
below. If none of the four have it, the result is not a value but `NameError`.

The fourth level differs from the others: the program does not write its names, the
language does. But its place in the search order is not special — because it is
last, any of the three levels above defining the same name **shadows** the built-in
and makes it unreachable in that body.

#### What Makes a Name Local

This is the real rule, and it concerns the entire body. Python decides which level a
name in a body belongs to **while reading the body, not while running it**. The rule
is one sentence: **if the name is assigned to anywhere in the body, that name is
local for the entire body.**

"Anywhere" has to be taken literally. Even if the assignment is on the body's very
last line, the name is local from the body's very first line onward. This is why a
read done **before** the assignment does not find the outer value — it finds a local
name that has not yet received a value, and the result is an exception:
`UnboundLocalError`.

This is the measurable form of shadowing. The Programming Fundamentals course defined
shadowing as "the outer name becomes unreachable"; the measurement here says
**where** that unreachability starts — not from the assignment line, from the body's
first line.

#### Writing Outward

A direct consequence of the rule: **assigning** to an outer name is impossible with
bare notation. Assignment always establishes a local name; it does not change the
outer one, it shadows it.

The language therefore recognizes two declarations. `global` declares that the name
belongs to the global level and routes assignments in that body there. `nonlocal`
declares that the name belongs to an **enclosing** body and routes assignments there;
it does not look at the global level, and errors if it cannot find one.

Reading needs no declaration; a declaration only changes the **writing** direction,
since the rule governs writing, not reading. The asymmetry is deliberate: reading an
outer value changes it for no one, but writing to it leaves a result visible outside
the body. Requiring a declaration keeps that result from being born by accident —
every body writing outward announces it in a single line.

The measurement's assumptions:

- **CF24** — The oracle is a separate string bound at each level naming that level;
  the returned value directly states which level the lookup stopped at.
- **CF25** — The same name `BOUNDARY` is measured in four states, and the states
  differ only in **which levels define that name**; the reading line in the bodies
  is character-for-character identical.
- **CF26** — The built-in level is measured with a separate name, because the
  program cannot write the names at that level. The language-supplied name `min` is
  used for the measurement and is shadowed at the levels above it.
- **CF27** — The assignment rule is measured with three bodies: one that only reads,
  one that reads and then assigns, one that only assigns. The reading line is
  identical in all three; the only thing that changes is whether the body **contains**
  an assignment, and where.
- **CF28** — The writing direction is measured in three states: undeclared, with a
  `nonlocal` declaration, and with a `global` declaration. Because the global-level
  name changes at the end of the measurement, that line is run last.
- **CF29** — Exceptions are written only by **class name**, their messages are not
  written.
- **CF30** — In the closure measurement, three functions are produced in a loop and
  called **after the loop ends**. Leaving the call for after the loop is deliberate;
  what is measured is **when** the name resolves, not which value it carries.

#### Measurement

```python
"""Four scope levels: at which level is a name found, where does assignment move it."""

BOUNDARY = "global"


def measure(action):
    try:
        return action()
    except Exception as e:
        return f"!{type(e).__name__}"


def none_defined():
    return UNDEFINED_NAME


def only_global():
    return BOUNDARY


def enclosing_too():
    BOUNDARY = "enclosing"

    def inner():
        return BOUNDARY
    return inner()


def local_too():
    BOUNDARY = "enclosing"

    def inner():
        BOUNDARY = "local"
        return BOUNDARY
    return inner()


SEARCH = (
    ("no level", none_defined, "-"),
    ("only global", only_global, "global"),
    ("global + enclosing", enclosing_too, "global, enclosing"),
    ("global + enclosing + local", local_too, "global, enclosing, local"),
)

print(f"{'levels where the name is defined':<30s} {'found':<26s} result")
for name, f, defined in SEARCH:
    print(f"  {name:<28s} {defined:<26s} {measure(f)}")


def uses_builtin():
    return min([3, 1, 2])


def enclosing_shadows_builtin():
    min = "enclosing shadowed"

    def inner():
        return min
    return inner()


def local_shadows_builtin():
    def inner():
        min = "local shadowed"
        return min
    return inner()


print()
print(f"{'builtin name min':<32s} result")
for name, f in (("no level defines it", uses_builtin),
              ("enclosing level defines it", enclosing_shadows_builtin),
              ("local level defines it", local_shadows_builtin)):
    print(f"  {name:<30s} {measure(f)}")


def only_reads():
    return BOUNDARY


def reads_then_assigns():
    before = BOUNDARY
    BOUNDARY = "local"
    return before


def only_assigns():
    BOUNDARY = "local"
    return BOUNDARY


print()
print(f"{'what is done with BOUNDARY in body':<34s} {'reading in body':<24s} result")
for name, f, reading in (("only read", only_reads, "none before assignment"),
                     ("read, then assigned", reads_then_assigns, "before assignment"),
                     ("only assigned", only_assigns, "after assignment")):
    print(f"  {name:<32s} {reading:<24s} {measure(f)}")


def declares_global():
    global BOUNDARY
    BOUNDARY = "written from inside"


def declares_nonlocal():
    BOUNDARY = "enclosing"

    def inner():
        nonlocal BOUNDARY
        BOUNDARY = "written from inside"
    inner()
    return BOUNDARY


def writes_undeclared():
    BOUNDARY = "enclosing"

    def inner():
        BOUNDARY = "written from inside"
    inner()
    return BOUNDARY


print()
print(f"{'writing outward from body':<28s} {'target level':<13s} outer name's value after")
for name, f, target in (("no declaration", writes_undeclared, "enclosing"),
                     ("nonlocal declaration", declares_nonlocal, "enclosing")):
    print(f"  {name:<26s} {target:<13s} {measure(f)}")

before = BOUNDARY
declares_global()
print(f"  {'global declaration':<26s} {'global':<13s} {before} -> {BOUNDARY}")


def late_bound():
    functions = []
    for i in range(3):
        def produce():
            return i
        functions.append(produce)
    return [f() for f in functions]


def captured_by_default():
    functions = []
    for i in range(3):
        def produce(i=i):
            return i
        functions.append(produce)
    return [f() for f in functions]


def name_after_loop():
    for i in range(3):
        pass
    return i


print()
print(f"{'three functions produced in a loop':<34s} returned when called")
print(f"  {'bare name in body':<32s} {late_bound()}")
print(f"  {'name captured by default':<32s} {captured_by_default()}")
print(f"  {'name after loop ends':<32s} {name_after_loop()}")
```

```
levels where the name is defined found                      result
  no level                     -                          !NameError
  only global                  global                     global
  global + enclosing           global, enclosing          enclosing
  global + enclosing + local   global, enclosing, local   local

builtin name min                 result
  no level defines it            1
  enclosing level defines it     enclosing shadowed
  local level defines it         local shadowed

what is done with BOUNDARY in body reading in body          result
  only read                        none before assignment   global
  read, then assigned              before assignment        !UnboundLocalError
  only assigned                    after assignment         local

writing outward from body    target level  outer name's value after
  no declaration             enclosing     enclosing
  nonlocal declaration       enclosing     written from inside
  global declaration         global        global -> written from inside

three functions produced in a loop returned when called
  bare name in body                [2, 2, 2]
  name captured by default         [0, 1, 2]
  name after loop ends             2
```

#### The Chain's Four Rungs

The top table builds the search chain rung by rung. The **reading line** of the four
rows is **character-for-character identical**; the only thing that changes is which
levels define that name.

When no level defines it, the result is `NameError`. In the second row the name
exists only at the global level, and that is what is found. In the third row the
enclosing level also defines the same name, and **enclosing wins** — the global-level
name is still sitting there, but never even enters the search. In the fourth, the
local level is added too, and this time **local wins**.

The pattern: every new level shuts out everything **outside** it. The search runs
**inside out**, not outside in, and stops at the first match. When three levels
define the same name, they do not compete — the innermost never even tests the other
two.

The middle table adds the fourth level. In the first row, no level defines `min`, and
the search descends to the built-in level and finds the language-supplied name; the
result is **1**, the smallest of the three-item list. In the next two rows the same
name is defined first at the enclosing, then the local level, and both shadow the
built-in. The built-in level has no privilege in search order; being last makes it
**the easiest to shadow**.

The practical consequence: giving a built-in's name to a local variable makes that
built-in unusable in that body. No error, no warning; the name now points to
something else.

#### A Single Assignment Line

The third table is the lesson's main measurement, and its three rows again have the
**same reading line**.

First row: the body only reads the name `BOUNDARY`, never assigns to it anywhere.
The search chain runs, finds it at the global level, result **global**.

Second row: the body does the same read first, **then** makes an assignment to the
name. The reading line comes before the assignment and is character-for-character
identical to the one in the first row. But the result is no longer a value — it is
**`UnboundLocalError`**.

The only difference is that an assignment exists somewhere **further down** in the
body. That assignment makes the name local for the entire body; by the time the
reading line is reached, the search never goes out to the global level at all, it
stops at the local level and finds a name there that has not received a value yet.
The chain does not break — **it is never built at all**.

The order relationship here is critical: what decides a name is local is not
execution order, it is a decision made by **looking at the whole body**. This is why
the error-producing line does not contain the cause; the cause is the assignment
line below it. The third row completes this — when the body only assigns and then
reads, there is no problem, the result is **local**.

#### Three Ways to Write

The bottom table measures the writing direction. In the first row the inner body
makes an undeclared assignment, and the outer name **does not change**: the result
is still `enclosing`. The assignment did not update the outer one, it established a
new name belonging to the inner body, and that name vanished once the body ended. The
line written looks like an update; what it does is a shadowing.

In the second row the same body writes with a `nonlocal` declaration, and the outer
name **does** change. In the third row a `global` declaration changes the
global-level name; the value that was `global` before the measurement becomes
`written from inside` afterward.

What the three rows say: writing outward from a body needs **a declaration**, not
bare assignment. The declaration adds no new capability; it only states which level
the assignment targets. With no declaration, the target is always local — the other
face of the same rule as the previous table's `UnboundLocalError`: one shows up in
reading, the other in writing.

This distinction also explains the difference between **mutating** an outer object
and **rebinding** an outer name. The shared bucket in the previous lesson grew with
no declaration at all, because what happened there was not assignment to a name, it
was mutating the object the name pointed to. Scope rules govern the binding of names,
not the content of objects.

#### When a Name Resolves

The last table measures **when** the chain runs, and gathers everything up to here
into a single result. Three functions are produced in a loop; each reads the loop
variable in its body with a bare name. All three are called after the loop ends.

The expected result is `[0, 1, 2]`, the measured result is **`[2, 2, 2]`**. All three
functions return the same value.

The reason is the scope rule itself. The name in the functions' bodies does not
resolve when the function is **produced**; it resolves when it is **called**. What
gets written into the body during production is not a value but a name — and that
name points to a single variable in the enclosing body. The three functions do not
carry three separate values, they carry **the same name**. Called, the search chain
runs, finds the name at the enclosing level, and finds there the last value the loop
left behind.

The third row shows what that last value is: the name is still alive after the loop
ends, and its value is **2**. In Python a loop does not open its own scope; the loop
variable belongs to the enclosing body and does not vanish when the loop ends. The
Programming Fundamentals course said the loop variable is **local**, and that is
true — the measurement narrows it: the unit it is local to is not the loop, it is
**the entire function body**. In languages where a brace- or keyword-closed block
opens its own scope, this line would raise an error; here it gives a value.

The second row gives the resolution, and the resolution is the previous lesson's
rule. When the name is written as a default value, the result becomes **`[0, 1,
2]`**. A default expression is evaluated once **while the definition is read**;
because the definition sits inside the loop body here, it reruns on every turn and
captures that turn's value. Rather than being left to the search chain, the value
gets copied at definition time.

The two lessons' rules meet here: scope says **where** a name will be searched for,
while a default's evaluation timing changes **when** it will be searched for. The
same three-line loop gives two different results with a single change to a
signature.

#### Summary

- Name lookup runs through four levels in a fixed order — local, enclosing, global,
  built-in — stopping at the first match, and gives `NameError` if none of the four
  have it.
- Every level shuts out everything outside it: if the same name is defined at three
  levels, the innermost wins and the others never enter the search.
- The built-in level has no privilege in search order; being last, the same name
  defined at any level above makes the built-in unreachable in that body.
- What decides a name is local is not where it is read, but that it is assigned to
  **somewhere** in the body; even if the assignment is last, the name is local from
  the body's start, and a read before the assignment produces `UnboundLocalError`.
- Writing outward from a body needs a declaration: `nonlocal` routes the assignment
  to the enclosing level, `global` to the global level. An undeclared assignment does
  not update the outer name, it establishes a new local one that shadows it.
- A name in a body resolves not when the function is **produced** but when it is
  **called**: three functions produced in a loop share the same name, so all three
  return **`[2, 2, 2]`**; capturing the name as a default value makes the result
  **`[0, 1, 2]`**.

#### Next Step

Up to here, the names at the search chain's last rung only ever showed up as
something to be shadowed. Yet the names sitting at that level are the language's
most heavily used functions, and each one connects directly to the protocols the
first two lessons measured: one asks for length, one converts to text, one requests
an iterator, one tests truthiness, one sorts. The next lesson measures these
built-ins by their **contracts** — which special method each one calls, and what
happens when that method is not defined. What emerges is that built-ins are not
functions doing the work themselves, but **thin shells calling a protocol**.

### Built-in Functions

Source: https://academia.sh/en/courses/python-fundamentals/built-in-functions

Built-ins are not functions doing the work themselves, they are thin shells calling a protocol: eight built-ins produce twenty-one calls, every collector builds the same five-call sequence, and on an object defining no protocol, three return a value while five throw TypeError.

The previous lesson measured the four levels of name lookup and showed the names at
the last rung only as something to be shadowed. Yet the names sitting at that level
are the language's most heavily used, and they connect directly to every protocol
measured so far.

This lesson's question is not what those names do — it is **whether they do any work
themselves**. Does `len` compute the length itself, or does it ask the object? Does
`sorted` know how to sort itself, or does it leave comparison to the object? The
course's measurement axis suggests an answer: if syntax is a shorthand, built-ins can
be a shorthand too. The measurement tests this.

#### A Built-in Is a Shell

The four lessons up to here measured syntax forms: `if`, `for`, `while`, call
notation. Built-ins are not syntax; they are ordinary functions called with a bare
name. But the measurement axis is the same for both — a built-in's number is also
which special method it calls, and how many times.

The contract is this: a built-in does not do the work itself. It looks for a
special method of a given name on the object, calls it if found, and gives back what
it returns. `len` does not count the length, it asks `__len__`; `str` does not build
the text, it asks `__str__`; `iter` does not produce an iterator, it requests one from
`__iter__`. A built-in's contribution is not the work, it is **a single call name**:
the same name is used no matter what the object's type is.

The consequence is that built-ins operate not by type but by **protocol**. What
decides whether an object can be measured with `len` is not its class's name, it is
whether it defines `__len__`.

The practical payoff: a new class needs nothing to **inherit** or **register**
anywhere to work with built-ins. It only defines a method carrying the expected
name; the built-in finds it. The shared reference's fourth reading gets paid a
second time here — what an object can do is decided not by its type's name, but by
the special methods it defines. All a built-in knows about an object is whether the
method it is looking for is there.

#### Collecting Built-ins

A second cluster is not satisfied with calling a single method. `list`, `sum`, and
`sorted` want **all of the object's items**; to do this they run the iteration
protocol start to finish. The expected sequence has to be the same one the second
lesson measured: one `__iter__`, followed by one more `__next__` than the item count.

One consequence: collecting items is always work proportional to item count. The
built-in cannot know how many items there are in advance, because the iteration
protocol announces no count. Collecting ends only when `StopIteration` arrives — it
continues until items run out.

`sorted` stands apart within this cluster, because after collecting it also has to
**order** the items. Collecting is one protocol, ordering another; the second does
comparisons between items. This is where the assumption that a built-in maps to a
single special method gets tested.

#### When There Is No Protocol

The third measure is the case where the method **cannot be found**. The shared
reference's fourth reading already said this: no protocol, no syntax either. Does the
same hold for built-ins?

It is clear from the start that the answer is not uniform, because there is a case
already measured in the first lesson: the truthiness trial did not error when it
found no method at all, it counted the object **true**. So at least one built-in has
a default behavior. How many do, how many do not — the measurement separates this.

The measurement's assumptions:

- **CF31** — The oracle is the rig itself: it logs the name every time a special
  method is called, and which method a built-in calls is read from this log.
- **CF32** — Two classes are used. `Flow` defines only the iteration and comparison
  methods; `Tracker` adds length and text methods to it. Collecting built-ins are
  measured on **`Flow`** — so the measured sequence comes only from the iteration
  protocol.
- **CF33** — `Silent` defines no special method at all and is the measure of the
  "no protocol" state.
- **CF34** — Results are **not printed as values**, only "produced a value" or the
  exception's class is written. The text form of an object defining no method
  carries environment-dependent information; it does not enter the measurement.
- **CF35** — `Tracker.__repr__`'s body calls `__str__`; this is the shared
  reference's behavior and is the reason for the two calls.
- **CF36** — In the sorting measurement, **not the call count but the set of
  protocols called** is written. Comparison count is a detail of the sort order and
  is not part of the measure.
- **CF37** — The total call count is specific to these eight built-ins and a
  three-item object.
- **CF38** — In the key-function measurement, the `len` built-in is given as `key`,
  so every place the key is called is logged and how many times per item can be
  counted.

#### Measurement

```python
"""The built-ins' contract: which special method does each one call."""

LOG = []


def record(name):
    LOG.append(name)


class Flow:
    """Only iteration and ordering protocol; does not define __len__."""

    def __init__(self, items=(1, 2, 3)):
        self.items = list(items)

    def __iter__(self):
        record("__iter__")
        self._i = 0
        return self

    def __next__(self):
        record("__next__")
        if self._i >= len(self.items):
            raise StopIteration
        value = self.items[self._i]
        self._i += 1
        return value

    def __lt__(self, other):
        record("__lt__")
        return len(self.items) < len(other.items)


class Tracker(Flow):
    """Also defines the length and text protocols."""

    def __len__(self):
        record("__len__")
        return len(self.items)

    def __str__(self):
        record("__str__")
        return f"Tracker{tuple(self.items)}"

    def __repr__(self):
        record("__repr__")
        return self.__str__()


class NonEmpty(Tracker):
    """__bool__ defined: truthiness does not fall to __len__."""

    def __bool__(self):
        record("__bool__")
        return len(self.items) > 0


class Silent:
    """Defines no special method at all."""


def measure(action):
    LOG.clear()
    try:
        action()
        status = "produced value"
    except Exception as e:
        status = f"!{type(e).__name__}"
    return list(LOG), status


SINGLE = (("len(n)", len, Tracker), ("str(n)", str, Tracker), ("repr(n)", repr, Tracker),
         ("bool(n)", bool, Tracker), ("iter(n)", iter, Tracker))
COLLECTING = (("list(n)", list, Flow), ("sum(n)", sum, Flow),
            ("sorted(n)", sorted, Flow))

print(f"{'builtin':<12s} {'calls':>6s}  protocol sequence called")
total = 0
for name, f, cls in SINGLE + COLLECTING:
    c, _ = measure(lambda f=f, s=cls: f(s()))
    total += len(c)
    print(f"  {name:<10s} {len(c):6d}  {' '.join(c)}")
print(f"\neight built-ins, total {total} calls")

print()
print(f"{'builtin':<12s} {'object defining protocol':<27s} object not defining it")
for name, f, cls in SINGLE + COLLECTING:
    _, d1 = measure(lambda f=f, s=cls: f(s()))
    _, d2 = measure(lambda f=f: f(Silent()))
    print(f"  {name:<10s} {d1:<27s} {d2}")

print()
c, _ = measure(lambda: bool(Tracker()))
print(f"bool(__bool__ undefined) -> {' '.join(c)}")
c, _ = measure(lambda: bool(NonEmpty()))
print(f"bool(__bool__ defined)   -> {' '.join(c)}")

print()
c, _ = measure(lambda: sorted(Flow()))
print(f"sorted(single object, items are numbers) -> {' '.join(sorted(set(c)))}")
TRIO = [Flow((1, 2, 3)), Flow((1,)), Flow((1, 2))]
c, _ = measure(lambda: sorted(TRIO))
print(f"sorted(three-object list)                -> {' '.join(sorted(set(c)))}")

print()
TRIO_TRACKER = [Tracker((1, 2, 3)), Tracker((1,)), Tracker((1, 2))]
print(f"{'three-object list':<28s} {'protocol called':<20s} is comparison asked of object")
for name, f in (("sorted(list)", lambda: sorted(TRIO_TRACKER)),
              ("sorted(list, key=len)", lambda: sorted(TRIO_TRACKER, key=len))):
    c, _ = measure(f)
    print(f"  {name:<26s} {' '.join(sorted(set(c))):<20s}"
          f" {'yes' if '__lt__' in c else 'no'}")
c, _ = measure(lambda: sorted(TRIO_TRACKER, key=len))
print(f"  key=len calls per item: {c.count('__len__')} / {len(TRIO_TRACKER)} items")
```

```
builtin       calls  protocol sequence called
  len(n)          1  __len__
  str(n)          1  __str__
  repr(n)         2  __repr__ __str__
  bool(n)         1  __len__
  iter(n)         1  __iter__
  list(n)         5  __iter__ __next__ __next__ __next__ __next__
  sum(n)          5  __iter__ __next__ __next__ __next__ __next__
  sorted(n)       5  __iter__ __next__ __next__ __next__ __next__

eight built-ins, total 21 calls

builtin      object defining protocol    object not defining it
  len(n)     produced value              !TypeError
  str(n)     produced value              produced value
  repr(n)    produced value              produced value
  bool(n)    produced value              produced value
  iter(n)    produced value              !TypeError
  list(n)    produced value              !TypeError
  sum(n)     produced value              !TypeError
  sorted(n)  produced value              !TypeError

bool(__bool__ undefined) -> __len__
bool(__bool__ defined)   -> __bool__

sorted(single object, items are numbers) -> __iter__ __next__
sorted(three-object list)                -> __lt__

three-object list            protocol called      is comparison asked of object
  sorted(list)               __lt__               yes
  sorted(list, key=len)      __len__              no
  key=len calls per item: 3 / 3 items
```

#### A One-to-One Match

The top table's first five rows confirm the contract. `len` makes **one** call and
what it calls is `__len__`; `str` one call, `__str__`; `iter` one call, `__iter__`.
There is no code in the built-in's body that counts a length, builds text, or
produces an iterator — if there were, the call count would be zero, because there
would be no need to ask the object at all.

The **2** in the `repr` row looks like an exception, but is not. `repr` still calls a
single method: `__repr__`. The second call does not come from the built-in — the body
of `__repr__` itself calls `__str__`. The built-in's contract is not broken; the
measured sequence also shows the work the called method does **itself**.

The `bool` row reproduces the first lesson's result: because `__bool__` is not
defined on the object, the trial falls to `__len__`. The two lines further down put
the pair side by side — while `__bool__` is defined, **only it** is called,
`__len__` never enters the trial. This is the one real exception to the rule that a
built-in maps to a single method: `bool` looks for not one but **two, in order**.

#### The Collectors' Shared Sequence

The last three rows give the same sequence: one `__iter__`, four `__next__`, total
**5**. Three separate built-ins — one builds a list, one takes a sum, one sorts —
and all three collect items in exactly the same way.

The sequence is character-for-character the one the second lesson measured, and for
the same reason it is **one more**: four `__next__` for three items, the last one
announcing collection is done by throwing `StopIteration`. `for`, `list`, `sum`, and
`sorted` — all four call the same protocol the same number of times. The difference
between them is **what they do** with the items collected, not how they collect
them.

The practical consequence: an object defining the iteration protocol works with all
of these built-ins automatically. `list` support, `sum` support, `sorted` support are
not added one by one — the moment `__iter__` and `__next__` are written, all three
arrive.

#### Sorting's Two Protocols

The bottom two lines set `sorted` apart and show it does not map to a single method.

In the first line, the sorted object's **items are numbers**. The protocols called
are `__iter__` and `__next__` — collecting only. Comparison happens too, of course,
but between numbers; the rig's object never enters that comparison and nothing lands
in the log.

In the second line, what is sorted is **a list of three objects**. This time the log
holds only `__lt__`. Because the list is already a list, the rig's protocol is not
called during collection; during sorting, every comparison is asked of the object.

Read together, the two lines show `sorted` calls two separate protocols: the
iteration protocol to **collect** items, the comparison protocol to **order** them.
Which one lands in the log depends on which role the rig's object is playing — the
outer container, or an item inside.

The reading that follows: `sorted` does not know how to sort, it asks the object
**to compare**. What decides the sort order is not the built-in's body, it is the
items' `__lt__` body. The built-in only asks which item comes before which and
follows the answer it gets. The same list is arranged in a different order just by
changing the items' `__lt__` definition, without a single character changing in the
`sorted` call.

#### A Key Changes the Protocol

The last table measures how the same call switches to **a different protocol** with
a keyword argument. The two rows sort the same list, the same objects; the only
thing that changes is whether the `key` argument is given.

In the call with no key, the protocol landing in the log is `__lt__` — comparison is
asked of the objects. In the call with a key, the protocol in the log is `__len__`
and `__lt__` is **never called at all**. The right-hand column says this directly:
comparison is no longer asked of the object.

What happens: given a `key`, `sorted` first applies the key function to every item,
then does the ordering on the **returned key values**. The objects are never
compared with each other; what gets compared is their keys. This is why even objects
with no `__lt__` can be sorted with a suitable key — sortability becomes a property
of the **key**, not the object.

The last line gives how many times the key is called: **3** calls for three items,
exactly once per item. The key is not recomputed on every comparison; it is computed
once and stored. The result is that an expensive key function is paid for once over
the whole sort — no matter the comparison count.

#### Having a Default and Not

The middle table measures the case where the protocol cannot be found, and the eight
built-ins **split in two**.

Five define no default at all: `len`, `iter`, `list`, `sum`, and `sorted` produce not
a value but `TypeError` when the method cannot be found. For these built-ins the
shared reference's fourth reading holds literally — no protocol, no built-in either.

Three produce a value. `str` and `repr` give a text form even for an object with no
method defined; the language defines a default text representation for every object.
`bool` applies the first lesson's third rung and counts the object true.

The reason for the split is whether the question has an answerable default. "What is
this object's length?" has no reasonable answer if the object does not say — any
number made up would be wrong. "What is this object's text form?" does have an
answer producible for every object. Whether a built-in has a default depends on
whether the question has an answer **independent of the object**.

This split has a cost that does not show in the table: built-ins with a default run
**silently**. A class that forgot to write `__len__` shows it immediately on a `len`
call; a class that forgot `__bool__` returns true from every `if` trial with no
signal at all. The error surfacing early is a guarantee the `TypeError`-producing
built-ins provide.

#### Summary

- Built-ins are not functions doing the work themselves, they are **thin shells**
  calling a special method on the object; `len` does not count the length, `str` does
  not build the text, `iter` does not produce an iterator — all three ask.
- Eight built-ins produce a total of **21** calls on a three-item object; `len`, `str`,
  `repr`, and `iter` each map to a single method — the second call in the `repr` row
  comes from the body of `__repr__`, not from the built-in — while `bool` looks for two
  methods in order and falls to `__len__` if `__bool__` is absent.
- `list`, `sum`, and `sorted` build the same five-call sequence — one `__iter__`,
  four `__next__` — character-for-character the same as the `for` loop's sequence; a
  class writing the iteration protocol gains all three at once.
- `sorted` calls not one but **two** protocols: iteration to collect, comparison to
  order. The sort order is decided not by the built-in's body but by the items'
  `__lt__` body.
- With a keyword argument given, `sorted` never asks the object for comparison at
  all: the key function is called instead of `__lt__` — exactly **once per item** —
  and sortability becomes a property of the key, not the object.
- On an object defining no protocol, **five** of eight built-ins give `TypeError`,
  **three** produce a value; the ones with a default are the ones whose question has
  an answer independent of the object, and they pay the cost of passing over an
  omission silently.

#### Next Step

Every protocol measured up to here followed the **expected** path. Syntax looked for
a method, found it, and called it; when not found it fell to a second method; when
that too was absent it either applied a default or threw a well-formed `TypeError`.
Even the `StopIteration` ending a loop was not an accident, it was the protocol's
planned last step.

So what if a protocol **fails**? The method called is found, it runs, but cannot
finish its job — a file is missing, a number cannot convert, an index falls outside
the bound. In this lesson `TypeError` showed up once; in earlier lessons `NameError`,
`UnboundLocalError`, `IndexError`, and `StopIteration` showed up. All five were
classes, and all can be caught with the same notation. The next lesson builds how
these classes relate to each other: which one catching also catches which others, and
what a broad catch swallows.

## Error Handling and Files

The exception hierarchy and catching behavior, custom exception placement, text and binary file modes with encoding, the context manager protocol, and regular expressions.

### Exceptions

Source: https://academia.sh/en/courses/python-fundamentals/exceptions

An exception is an instance of a class, and catching looks at the class lineage: eleven events and seven catch classes match in 28 of 77 pairs, Exception catches ten events and swallows three program defects, and BaseException catches all eleven and swallows the exit request too.

Every protocol measured up to this point followed the expected path. `for` asked the
iterator for something and got it, `len` returned a number, `sorted` gave back a sorted
list. Syntax called, a special method answered, a value came back. So what happens when
a protocol **fails**?

Python's answer already showed up in the previous lesson's loop measurement: on a
three-item object, the fourth `__next__` call returns no value, it **raises
`StopIteration`**. What ends the loop is not a return value, it is an exception. This
lesson looks at that machinery itself. **The concept of an exception** and error
handling were built in the Programming Fundamentals course; the **concept** was
established there, what is measured here is the machinery Python realizes it with: an
exception is an object, the object has a class, and **catching looks at that class's
lineage**.

#### An Exception Is an Instance of a Class

What is raised is not a message, it is an **object**. The `raise` statement takes an
object, the `except` clause takes a **class**, and the two are compared with
`isinstance`. This one sentence determines the rest of the lesson: because `except`
writes a class name, **all of that class's subclasses** get caught too.

The lineage starts from two ends. Every exception class derives from `BaseException`.
Right below it stands `Exception`, and everything expected to be handled inside the
program descends from there. Below `Exception` sit intermediate classes —
`ArithmeticError`, `LookupError`, `OSError` — and their leaves are concrete classes like
`ZeroDivisionError`, `KeyError`, `FileNotFoundError`.

These intermediate classes are not empty decoration. Catching a leaf catches only that
leaf; catching an intermediate class catches **every leaf beneath it**. How many it
catches is a measurable number, and that number is exactly what this lesson measures.

#### The Measurement's Setup

The measurement builds eleven **events**: small functions each of which really raises an
exception. Against them sit seven **catch classes** — two leaves, three intermediate
classes, and two roots. Every event is tried against every catch class; the result is a
77-pair table.

The events split into three tags, and the tag is the **oracle**: because we built the
setup ourselves, we know which event is really a **domain error**, which is a **program
defect**, which is an **interrupt**. A domain error is something expected to be
handled — a missing key, an unparseable string, a file that cannot be found. A program
defect is a mistake in the code itself, and it needs not to be handled but **to escape**.
An interrupt is a request to end the program, and it must not be swallowed.
`StopIteration` is a fourth tag: not an error, **part of the protocol**.

The measurement's assumptions:

- **EF1** — All eleven events are really run, and the object they raise is caught and
  kept; no class name is written by hand — every class name in the table is read
  **from the raised object**.
- **EF2** — The tag is an oracle and comes from the setup itself; it is not derived by
  looking at the message or the class name.
- **EF3** — A pair "matching" means the catch class covers the event's object through
  `isinstance`; this is exactly the test the `except` clause performs.
- **EF4** — The lineage chain is read from the class's resolution order; `object` is not
  counted, because every class descends from it and it is not distinctive.
- **EF5** — The measurement is not a performance measurement; what is counted is the
  **number of events caught**, not time.

#### Measurement — the Catch Matrix

```python
"""Exception hierarchy: catching which class also catches which events."""
import math

LOG = []


class Tracker:
    """The core of the shared setup; only the iteration part is used here."""

    def __init__(self, items=(1, 2, 3)):
        self.items = list(items)

    def __iter__(self):
        LOG.append("__iter__")
        self._i = 0
        return self

    def __next__(self):
        LOG.append("__next__")
        if self._i >= len(self.items):
            raise StopIteration
        value = self.items[self._i]
        self._i += 1
        return value


def zero_division():
    return 1 // 0


def overflowing_power():
    return math.exp(10000)


def missing_key():
    return {"north": 1}["slope"]


def overflowing_index():
    return [1, 2, 3][9]


def unconvertible_string():
    return int("north")


def missing_file():
    return open("missing/measurement.txt")


def exhausted_iterator():
    return next(iter(Tracker(())))


def wrong_type():
    return len(5)


def undefined_name():
    return unknown_name


def missing_attribute():
    return (1).north


def exit_request():
    raise SystemExit


EVENTS = (
    ("zero division",        zero_division,        "domain"),
    ("overflowing power",    overflowing_power,    "domain"),
    ("missing key",          missing_key,          "domain"),
    ("overflowing index",    overflowing_index,    "domain"),
    ("unconvertible string", unconvertible_string, "domain"),
    ("missing file",         missing_file,         "domain"),
    ("exhausted iterator",   exhausted_iterator,   "protocol"),
    ("wrong type",           wrong_type,           "defect"),
    ("undefined name",       undefined_name,       "defect"),
    ("missing attribute",    missing_attribute,    "defect"),
    ("exit request",         exit_request,         "interrupt"),
)

CATCHES = (
    ("ZeroDivisionError", ZeroDivisionError),
    ("ArithmeticError",   ArithmeticError),
    ("LookupError",       LookupError),
    ("ValueError",        ValueError),
    ("OSError",           OSError),
    ("Exception",         Exception),
    ("BaseException",     BaseException),
)


def throw(function):
    try:
        function()
    except BaseException as e:
        return e
    return None


print("event                   class              place in the hierarchy")
for name, function, _ in EVENTS:
    s = type(throw(function))
    chain = " < ".join(k.__name__ for k in s.__mro__ if k is not object)
    print(f"  {name:21s} {s.__name__:18s} {chain}")

print()
print("catch class         catches  domain  protocol  defect  interrupt  escapes")
for cname, cls in CATCHES:
    counts = {"domain": 0, "protocol": 0, "defect": 0, "interrupt": 0}
    for name, function, kind in EVENTS:
        if isinstance(throw(function), cls):
            counts[kind] += 1
    caught = sum(counts.values())
    print(f"  {cname:18s} {caught:7d} {counts['domain']:5d} {counts['protocol']:9d}"
          f" {counts['defect']:6d} {counts['interrupt']:8d} {len(EVENTS) - caught:6d}")

print()
matches = sum(isinstance(throw(f), cls)
            for _, f, _ in EVENTS for _, cls in CATCHES)
print(f"events {len(EVENTS)} | catch classes {len(CATCHES)} | "
      f"pairs {len(EVENTS) * len(CATCHES)} | matching pairs {matches}")
for kind in ("defect", "interrupt"):
    items_list = [name for name, _, k in EVENTS if k == kind]
    print(f"{kind} {len(items_list)}: {', '.join(items_list)}")

LOG.clear()
throw(exhausted_iterator)
print(f"empty Tracker: {LOG.count('__iter__')} times __iter__, "
      f"{LOG.count('__next__')} times __next__ — that one call returned no value, "
      f"it raised StopIteration")
```

```
event                   class              place in the hierarchy
  zero division         ZeroDivisionError  ZeroDivisionError < ArithmeticError < Exception < BaseException
  overflowing power     OverflowError      OverflowError < ArithmeticError < Exception < BaseException
  missing key           KeyError           KeyError < LookupError < Exception < BaseException
  overflowing index     IndexError         IndexError < LookupError < Exception < BaseException
  unconvertible string  ValueError         ValueError < Exception < BaseException
  missing file          FileNotFoundError  FileNotFoundError < OSError < Exception < BaseException
  exhausted iterator    StopIteration      StopIteration < Exception < BaseException
  wrong type            TypeError          TypeError < Exception < BaseException
  undefined name        NameError          NameError < Exception < BaseException
  missing attribute     AttributeError     AttributeError < Exception < BaseException
  exit request          SystemExit         SystemExit < BaseException

catch class         catches  domain  protocol  defect  interrupt  escapes
  ZeroDivisionError        1     1         0      0        0     10
  ArithmeticError          2     2         0      0        0      9
  LookupError              2     2         0      0        0      9
  ValueError               1     1         0      0        0     10
  OSError                  1     1         0      0        0     10
  Exception               10     6         1      3        0      1
  BaseException           11     6         1      3        1      0

events 11 | catch classes 7 | pairs 77 | matching pairs 28
defect 3: wrong type, undefined name, missing attribute
interrupt 1: exit request
empty Tracker: 1 times __iter__, 1 times __next__ — that one call returned no value, it raised StopIteration
```

#### Reading the Numbers

The upper table gives the lineage and shows the depth is not fixed. `ValueError`
reaches the root in two steps, `FileNotFoundError` in three. The difference between
them is not a detail: every intermediate step corresponds to an `except` clause that
would catch at that step.

The lower table counts the cost of this. Catching `ZeroDivisionError` matches **1**
event and misses **10**. One step up, `ArithmeticError` matches **2** — zero division
and the overflowing power. `LookupError` also matches **2**, the missing key and the
overflowing index. The way to write a single `except` line that handles two separate
events is to write their **common ancestor**.

The `OSError` row shows **1**, because only one event in the setup descends from it.
The number is the setup's count, not the class's scope: many leaves sit beneath
`OSError`, and all of them would land on this row. The lesson writes **the count it
itself measured**.

#### What a Broad Catch Swallows

The two root rows are the table's central finding. Catching `Exception` matches **10**
events and misses only **1**. **6** of these ten are domain errors — exactly what is
expected to be handled. But the same row also swallows **3** program defects: wrong
type, undefined name, missing attribute. None of these three come from user input or
the file system; they come from **a mistake in the code itself**. A block written as
`except Exception` catches these too, silently, and the program keeps running in its
broken form.

It also swallows **1** protocol event: `StopIteration`. The last line states where this
comes from: on an empty `Tracker`, `__iter__` was called **1** time, `__next__` **1**
time, and that single call raised `StopIteration` instead of returning a value — one
more than the item count. A broad catch can mistake a loop's **normal end** for an error
and fall into its own branch.

`BaseException` goes one step further: it matches all **11** events, missing **0**. The
one extra thing it catches is the **exit request**, and this is the lesson's sharpest
number. `Exception` misses it — missing it is **by design**. A request to end the
program is not a domain error and is not something to be handled; writing `except
BaseException` means swallowing that request too. This is the only difference between
the two root rows, and in the table it stands as **11 − 10 = 1**.

The pattern is this: **as the catch class climbs, the events it matches grow, but only
part of what it matches is something that should be handled.** **28** of 77 pairs match,
and **21** of these 28 matches come from the two root rows. Narrowing means shrinking
what is matched and zeroing out what is swallowed.

#### Re-Raising and the Cause Chain

Catching an exception does not mean it has to be handled. Catching it, logging it, and
then **re-raising** it is a common need, and Python offers a few distinct forms for
this. What differs between the forms is the **identity** of the object that escapes and
whether the inner object survives.

Every exception object carries two fields. `__context__` points to **the other**
exception if this one was born while that other one was being handled, and the
interpreter sets this up on its own. `__cause__`, by contrast, is only filled in when
`from` is written, and it is the declaration "this exception's cause is that one."
`from None` **suppresses** the context.

- **EF6** — Five re-raising forms are produced from the same source; because the source
  produces **a new object** on every call, the forms do not share each other's object.
- **EF7** — Whether the objects are the same is tested with `is`; **no identity number
  is printed**.
- **EF8** — The chain length is counted by following unsuppressed context and cause
  links; because no exception escapes in the swallowing form, its length is zero.
- **EF9** — The branch table runs the three states separately, and whether each branch
  ran is read from a log placed inside the branch; the order is not guessed, it is
  measured.

```python
"""Re-raising and the chain: which object escapes, does the inner one survive."""

STEPS = []


def source():
    """Produces a domain error; a new object forms on every call."""
    return {"north": 1}["slope"]


def bare():
    try:
        source()
    except KeyError:
        raise


def wrap():
    try:
        source()
    except KeyError:
        raise RuntimeError("could not read the measurement")


def with_cause():
    try:
        source()
    except KeyError as e:
        raise RuntimeError("could not read the measurement") from e


def suppress():
    try:
        source()
    except KeyError:
        raise RuntimeError("could not read the measurement") from None


def swallow():
    try:
        source()
    except KeyError:
        return None


FORMS = (("bare raise", bare), ("wrapping", wrap),
            ("with from", with_cause), ("with from None", suppress),
            ("swallowing", swallow))

print(f"{'form':<17s}{'escapes as':<15s}{'cause':<14s}"
      f"{'context':<14s}chain")
for name, function in FORMS:
    try:
        function()
    except BaseException as exc:
        escapes = type(exc).__name__
        cause = type(exc.__cause__).__name__ if exc.__cause__ else "-"
        context = type(exc.__context__).__name__ if exc.__context__ else "-"
        length, cur = 0, exc
        while cur is not None:
            length += 1
            cur = cur.__cause__ or (None if cur.__suppress_context__
                                else cur.__context__)
    else:
        escapes = cause = context = "-"
        length = 0
    print(f"  {name:<15s}{escapes:<15s}{cause:<14s}{context:<14s}{length}")

print()
try:
    try:
        source()
    except KeyError as e:
        inner = e
        raise
except KeyError as exc:
    print("does a bare raise send the same object out:", exc is inner)

try:
    try:
        source()
    except KeyError as e:
        inner = e
        raise RuntimeError("could not read the measurement") from e
except RuntimeError as exc:
    print("with from, is the outgoing object the same:", exc is inner)
    print("with from, is the cause the same object:", exc.__cause__ is inner)

print()
BRANCHES = ("try", "except", "else", "finally")
print(f"{'state':<16s}" + "".join(f"{b:<9s}" for b in BRANCHES))
for state, function in (("no error", lambda: 1),
                     ("caught", source),
                     ("uncaught", lambda: 1 // 0)):
    STEPS.clear()
    try:
        try:
            STEPS.append("try")
            function()
        except KeyError:
            STEPS.append("except")
        else:
            STEPS.append("else")
        finally:
            STEPS.append("finally")
    except ZeroDivisionError:
        pass
    print(f"  {state:<14s}"
          + "".join(f"{('ran' if b in STEPS else 'skipped'):<9s}"
                    for b in BRANCHES))
```

```
form             escapes as     cause         context       chain
  bare raise     KeyError       -             -             1
  wrapping       RuntimeError   -             KeyError      2
  with from      RuntimeError   KeyError      KeyError      2
  with from None RuntimeError   -             KeyError      1
  swallowing     -              -             -             0

does a bare raise send the same object out: True
with from, is the outgoing object the same: False
with from, is the cause the same object: True

state           try      except   else     finally  
  no error      ran      skipped  ran      ran      
  caught        ran      ran      skipped  ran      
  uncaught      ran      skipped  skipped  ran
```

#### What the Chain Costs

**Bare `raise`** sends `KeyError` out, and the line below states it is the **same
object**. Because no new object is produced, the chain is **1**. Catching and
re-raising loses no information.

The **wrapping** form sends `RuntimeError` out; `__cause__` is empty but `__context__`
holds the `KeyError`. The chain is **2**. Nobody wrote this link — it formed on its own
because the exception was born while another exception was being handled.

In the form written **with `from`**, `__cause__` is filled and the chain is again **2**.
The difference is in the output's lower lines: the escaping object is **not the same**
as the inner one, but `__cause__` points to **the same object**. Wrapping and writing
`from` give the same chain length; where they part ways is whether the link was
**declared** or formed on its own.

**`from None`** brings the chain down to **1**. `__context__` is still filled, but
because it is suppressed it does not enter the count — someone looking from outside
cannot see the `KeyError`. For a boundary that wants to hide a domain error, the right
tool is one used knowing the inner detail is lost.

The **swallowing** form gives **0**: nothing escapes at all. Combined with the broad
catch from the previous section, this zero is dangerous — this is exactly where the
three program defects got swallowed.

The last table gives which of the four branches ran in which state. `try` ran in all
three states; `finally` ran in **all three too** — even while an uncaught exception was
escaping. `else` only ran when no exception **occurred**, `except` only when one was
**caught**. Keeping the source call inside `try` and moving the success path into
`else` keeps the `except` clause from accidentally catching too much.

#### Summary

- An exception is an object, the object has a class, and `except` takes a **class**;
  catching covers all of that class's subclasses.
- Eleven events and seven catch classes produce **77** pairs, and **28** of them match;
  leaf classes match **1**–**2** events, while `Exception` matches **10** and
  `BaseException` matches **11**.
- Catching `Exception` swallows **6** domain errors alongside **3** program defects and
  **1** protocol event; `BaseException` additionally swallows the **exit request**.
- A bare `raise` sends **the same object** out (chain **1**); wrapping and writing
  `from` make the chain **2**, `from None` brings it down to **1**, swallowing to **0**.
- `finally` runs in all three states, `else` only when no exception occurs; keeping the
  success path inside `else` narrows the `except` clause's scope.

#### Next Step

Every class in this lesson came ready-made: `KeyError`, `ValueError`, `OSError`. The
measurement could group them by common ancestor because the lineage was already built.
But what if what is raised is **the domain's own error** — a measurement device
reporting an out-of-range value, a field missing from a record? Then it is us who write
the class, and where we hang it in the lineage decides which of the later catches will
match. The next lesson measures a custom exception class's place in the hierarchy and
what that place costs in catching behavior.

### Custom Exceptions

Source: https://academia.sh/en/courses/python-fundamentals/custom-exceptions

A custom exception class's place in the hierarchy decides who catches it: five classes and six catches match in 13 of 30 pairs, the domain root handles seven errors with a single name, the borrowed base catches only three of them, and writing the broad clause first runs the narrow clause zero times.

Every class in the previous lesson came ready-made. `KeyError`, `ValueError`,
`FileNotFoundError` — their lineages were already built, and the measurement could
group them by common ancestor. We saw that catching an intermediate class catches every
leaf beneath it; someone else had done the placing.

But what if what is raised is **the domain's own error**? A measurement device
reporting a value outside the accepted range, a required field missing from a record, a
raw piece of data that cannot be converted to a number. For these, a class has to be
written, and where we hang that class in the lineage is not a matter of style: its place
decides **who will catch it**. That is this lesson's question.

#### The Domain's Own Error Is a Type

There is a measurable difference between describing an error with a message string and
describing it with a type. A message is a string; code reading it has to parse the
string, and the parsing breaks the moment the message changes. A type, by contrast, is
exactly what the `except` clause tests directly, and the test is done with `isinstance`.

Writing a custom exception amounts to nothing more than a class definition. The class
can hold the domain data it carries as **attributes** — which field, which value — and
catching code reads these without parsing a message.

The real decision is the base. A **root** class is defined for domain errors, and
concrete errors descend from it; this way a single `except` line covers all of them.
Beyond that, a concrete class can **also** descend from a built-in class that fits its
meaning: an out-of-range value is a `ValueError`, a missing field is a `KeyError`. This
two-base placement looks like a convenience, and its cost is measured.

#### Choosing the Root

The measurement builds four domain classes. `MeasurementError` is the root and descends
directly from `Exception`. `OutOfRangeValue` descends from the root **and** from
`ValueError`, `MissingField` from the root **and** from `KeyError`. `BrokenRecord`
descends only from the root. The fifth class, `DeviceError`, does not descend from the
root — it is a separate tree, and it exists to show what the root does not cover.

The root side of the placement looks like this as a tree; because the borrowed bases sit
on the tree's other branches, they are noted at the end of the line:

```text
# illustrative dump, not executed

Exception
├── MeasurementError          domain's root
│   ├── OutOfRangeValue       also ValueError
│   ├── MissingField          also KeyError
│   └── BrokenRecord          root only
└── DeviceError               does not descend from root
```

The measurement's assumptions:

- **EF10** — The lineage chain is read from the class's own resolution order, not
  written by hand; `object` is not counted because every class descends from it.
- **EF11** — A pair "matching" means the catch class covers the raised class through
  `issubclass`; this is exactly the test the `except` clause performs, and it can be
  known in advance at the class level.
- **EF12** — A "foreign catch" is an `except ValueError` or `except KeyError` clause
  written by code unaware of the domain, for its own purposes; a domain class landing
  there is a **leak**.
- **EF13** — The classes are defined purely for placement; none of them add behavior, so
  the only thing measured is **the place in the hierarchy**.

```python
"""Custom exception: the class's place in the hierarchy decides catching behavior."""


class MeasurementError(Exception):
    """The domain's root: every error handling a measurement record descends from here."""

    def __init__(self, field, value=None):
        super().__init__(field)
        self.field = field
        self.value = value

    def __str__(self):
        return f"{type(self).__name__}(field={self.field!r}, value={self.value!r})"


class OutOfRangeValue(MeasurementError, ValueError):
    """The value was read but is outside the field's accepted range."""


class MissingField(MeasurementError, KeyError):
    """The record is missing a required field."""


class BrokenRecord(MeasurementError):
    """The record could not be converted to a number; it has no built-in counterpart."""


class DeviceError(Exception):
    """Does not descend from the domain root: a separate tree."""


CLASSES = (("MeasurementError", MeasurementError), ("OutOfRangeValue", OutOfRangeValue),
            ("MissingField", MissingField), ("BrokenRecord", BrokenRecord),
            ("DeviceError", DeviceError))

CATCHES = (("MeasurementError", MeasurementError), ("OutOfRangeValue", OutOfRangeValue),
               ("ValueError", ValueError), ("KeyError", KeyError),
               ("LookupError", LookupError), ("Exception", Exception))

print("class               place in the hierarchy")
for name, s in CLASSES:
    chain = " < ".join(k.__name__ for k in s.__mro__ if k is not object)
    print(f"  {name:18s}{chain}")

print()
header = "".join(f"{y:<17s}" for y, _ in CATCHES)
print(f"{'raised':<20s}{header}")
for name, s in CLASSES:
    cell = "".join(f"{('matches' if issubclass(s, ys) else '-'):<17s}"
                    for _, ys in CATCHES)
    print(f"  {name:<18s}{cell}")

print()
matches = sum(issubclass(s, ys) for _, s in CLASSES for _, ys in CATCHES)
print(f"classes {len(CLASSES)} | catches {len(CATCHES)} | "
      f"pairs {len(CLASSES) * len(CATCHES)} | matching pairs {matches}")
rooted = sum(issubclass(s, MeasurementError) for _, s in CLASSES)
leaked = sum(issubclass(s, (ValueError, KeyError)) for _, s in CLASSES)
print(f"descend from the domain root {rooted} | leak into a foreign catch {leaked}")
```

```
class               place in the hierarchy
  MeasurementError  MeasurementError < Exception < BaseException
  OutOfRangeValue   OutOfRangeValue < MeasurementError < ValueError < Exception < BaseException
  MissingField      MissingField < MeasurementError < KeyError < LookupError < Exception < BaseException
  BrokenRecord      BrokenRecord < MeasurementError < Exception < BaseException
  DeviceError       DeviceError < Exception < BaseException

raised              MeasurementError OutOfRangeValue  ValueError       KeyError         LookupError      Exception        
  MeasurementError  matches          -                -                -                -                matches          
  OutOfRangeValue   matches          matches          matches          -                -                matches          
  MissingField      matches          -                -                matches          matches          matches          
  BrokenRecord      matches          -                -                -                -                matches          
  DeviceError       -                -                -                -                -                matches          

classes 5 | catches 6 | pairs 30 | matching pairs 13
descend from the domain root 4 | leak into a foreign catch 2
```

#### What the Borrowed Base Costs

The upper table gives the placement's result. The `MeasurementError` column matches in
**4** rows — all four classes descending from the root. The `DeviceError` row is empty;
the root does not cover it, because it does not descend from it. Writing the domain's
root is writing the domain's **boundary**.

What is interesting is the `ValueError` and `KeyError` columns. Each matches in **1**
row, and those rows are exactly the classes carrying a borrowed base. `MissingField`
even matches in the `LookupError` column too: because it descends from `KeyError`, it
also inherited that class's ancestor. A single placement decision put the class within
the reach of **three separate foreign clauses**.

The bottom row counts this: **2** of the four domain classes leak into a foreign catch.
This means the `except ValueError` clause code unaware of the domain writes for its
`int()` call will also catch an `OutOfRangeValue` object — a measurement error mistaken
for and handled as a conversion error. A borrowed base gives the class **meaning**, and
in exchange, **scope**.

**13** of thirty pairs match. **5** of these come from the `Exception` column; every
class descends from it, and that column is not distinctive. The distinguishing columns
are the root and the borrowed bases.

#### Catch Forms and Clause Order

The second measurement runs the classes. Twelve measurement records are processed; part
of them are valid, part produce a domain error, and one produces a **program defect** —
the coefficient field is written with the wrong type, and the multiplication raises
`TypeError`. Four catch forms are tried, and three numbers are read for each: how many
names were written, how many domain errors were handled, how many program defects were
swallowed.

- **EF14** — Each of the twelve records is really processed; the result class is not
  labeled by hand, it is read from the raised object.
- **EF15** — The oracle is the setup: which record is a domain error and which is a
  program defect is known from how the records were built.
- **EF16** — "Handled" means the caught object is an instance of `MeasurementError`;
  "swallowed defect" means an object that was caught but does not descend from the
  domain root.
- **EF17** — "Escaped" is the count of records that do not fall into the form's clause
  and pass to the next clause.
- **EF18** — In the clause-order measurement, two blocks write the **same** two classes
  and only change the order; there is no other difference.

```python
"""Catch forms: how many names are written, how many domain errors handled, how many defects swallowed."""


class MeasurementError(Exception):
    def __init__(self, field, value=None):
        super().__init__(field)
        self.field = field
        self.value = value

    def __str__(self):
        return f"{type(self).__name__}(field={self.field!r}, value={self.value!r})"


class OutOfRangeValue(MeasurementError, ValueError):
    pass


class MissingField(MeasurementError, KeyError):
    pass


class BrokenRecord(MeasurementError):
    pass


LOWER_BOUND, UPPER_BOUND = -50.0, 60.0

RECORDS = (
    {"value": "12.5", "coefficient": 2},
    {"value": "-3", "coefficient": 1},
    {"value": "99", "coefficient": 1},
    {"coefficient": 1},
    {"value": "north", "coefficient": 1},
    {"value": "0", "coefficient": 3},
    {"value": "-80", "coefficient": 1},
    {"value": None, "coefficient": 1},
    {"value": "45", "coefficient": "2"},
    {"coefficient": 5},
    {"value": "60", "coefficient": 1},
    {"value": "60.1", "coefficient": 1},
)


def process(record):
    if "value" not in record:
        raise MissingField("value")
    raw = record["value"]
    try:
        number = float(raw)
    except (TypeError, ValueError):
        raise BrokenRecord("value", raw) from None
    if not LOWER_BOUND <= number <= UPPER_BOUND:
        raise OutOfRangeValue("value", number)
    return record["coefficient"] * number


def class_name(record):
    try:
        process(record)
    except BaseException as e:
        return type(e).__name__
    return "-"


results = [class_name(r) for r in RECORDS]
print("record results:", ", ".join(f"{i}:{s}" for i, s in enumerate(results, 1)))
domain_count = sum(s in ("OutOfRangeValue", "MissingField", "BrokenRecord") for s in results)
defect_count = sum(s == "TypeError" for s in results)
print(f"records {len(RECORDS)} | successful {results.count('-')} | "
      f"domain error {domain_count} | program defect {defect_count}")

FORMS = (
    ("three names", (OutOfRangeValue, MissingField, BrokenRecord), 3),
    ("domain root", (MeasurementError,), 1),
    ("borrowed base", (ValueError,), 1),
    ("broad", (Exception,), 1),
)

print()
print(f"{'catch form':<17s}{'names written':>14s}{'handled':>12s}"
      f"{'swallowed defect':>18s}{'escaped':>12s}")
for name, cls, count in FORMS:
    handled = swallowed = escaped = 0
    for r in RECORDS:
        try:
            process(r)
        except cls as e:
            if isinstance(e, MeasurementError):
                handled += 1
            else:
                swallowed += 1
        except Exception:
            escaped += 1
    print(f"  {name:<15s}{count:>14d}{handled:>12d}{swallowed:>18d}{escaped:>12d}")

print()
root_first = {"MeasurementError": 0, "OutOfRangeValue": 0}
for r in RECORDS:
    try:
        process(r)
    except MeasurementError:
        root_first["MeasurementError"] += 1
    except OutOfRangeValue:
        root_first["OutOfRangeValue"] += 1
    except Exception:
        pass

narrow_first = {"OutOfRangeValue": 0, "MeasurementError": 0}
for r in RECORDS:
    try:
        process(r)
    except OutOfRangeValue:
        narrow_first["OutOfRangeValue"] += 1
    except MeasurementError:
        narrow_first["MeasurementError"] += 1
    except Exception:
        pass

print("clause order    first clause              second clause")
print(f"  root first    MeasurementError {root_first['MeasurementError']:<13d}"
      f" OutOfRangeValue {root_first['OutOfRangeValue']}")
print(f"  narrow first  OutOfRangeValue {narrow_first['OutOfRangeValue']:<9d}"
      f" MeasurementError {narrow_first['MeasurementError']}")

print()
print("domain errors reported through their attributes:")
for i, r in enumerate(RECORDS, 1):
    try:
        process(r)
    except MeasurementError as e:
        print(f"  record {i:2d}: {e} -> field {e.field!r}, value {e.value!r}")
    except Exception:
        pass
```

```
record results: 1:-, 2:-, 3:OutOfRangeValue, 4:MissingField, 5:BrokenRecord, 6:-, 7:OutOfRangeValue, 8:BrokenRecord, 9:TypeError, 10:MissingField, 11:-, 12:OutOfRangeValue
records 12 | successful 4 | domain error 7 | program defect 1

catch form        names written     handled  swallowed defect     escaped
  three names                 3           7                 0           1
  domain root                 1           7                 0           1
  borrowed base               1           3                 0           5
  broad                       1           7                 1           0

clause order    first clause              second clause
  root first    MeasurementError 7             OutOfRangeValue 0
  narrow first  OutOfRangeValue 3         MeasurementError 4

domain errors reported through their attributes:
  record  3: OutOfRangeValue(field='value', value=99.0) -> field 'value', value 99.0
  record  4: MissingField(field='value', value=None) -> field 'value', value None
  record  5: BrokenRecord(field='value', value='north') -> field 'value', value 'north'
  record  7: OutOfRangeValue(field='value', value=-80.0) -> field 'value', value -80.0
  record  8: BrokenRecord(field='value', value=None) -> field 'value', value None
  record 10: MissingField(field='value', value=None) -> field 'value', value None
  record 12: OutOfRangeValue(field='value', value=60.1) -> field 'value', value 60.1
```

#### Reading the Numbers

**4** of the twelve records are successful, **7** are domain errors, **1** is a program
defect. Four forms split this distribution in different ways.

**Three names** and **domain root** give the same result: **7** handled, **0**
swallowed defects, **1** escaped. The two are indistinguishable — but one writes **3**
names, the other **1**. This is exactly what the root pays for: as the domain grows, new
concrete classes will descend from the root, and no clause written as `except
MeasurementError` will need to change. A clause counting three names has to be updated
with every new class.

**Borrowed base** handles only **3** errors and lets **5** records escape. Writing
`except ValueError` catches `OutOfRangeValue`, but `MissingField` and `BrokenRecord` do
not fall into it. A borrowed base is not a **group name**; each concrete class's own
borrowed base is separate, and there is no shared one among them. The domain's group
name is only the root.

**Broad** catching lets **0** records escape — and swallows **1** program defect. The
record whose coefficient field was written with the wrong type is handled as if it were
a domain error. The previous lesson's finding takes concrete shape here: what hides the
defect is not the defect itself, it is the clause that covers it.

The **clause order** table gives the sharpest number. When the root is written first,
the `MeasurementError` clause runs **7** times, the `OutOfRangeValue` clause **0** times.
The second clause is written, its syntax is correct, it raises no warning, and it **never
runs** — because the first matching clause wins, and the root already covers everything.
When the order is reversed, the narrow clause gets **3**, the root clause **4**; the
total is still **7**. Catch clauses are written **narrow to broad**; the reverse
silently produces dead code.

The last dump shows what the attribute earns. Every line gives the field name and the
value **without parsing a message**; code reading the difference between `-80.0` and
`60.1` works with a number, not a string.

#### Translation at the Boundary

The root's second job does not show up in the table, but it stands in the measurement's
code. The `process` function tries to convert the raw value with `float`, and that call
can raise either `ValueError` or `TypeError` depending on the data's shape. Neither of
these two classes is the domain's own class; both are the conversion layer's classes.
The code catches them and raises `BrokenRecord` in their place:

`except (TypeError, ValueError): raise BrokenRecord("value", raw) from None`

These three lines are a **boundary translation**. Code that knows what happened inside
speaks the domain's language to the outside. The eighth record in the output is the
result of this: the `value` field is not a number, it is not a value at all, and `float`
produces a `TypeError` from it — but the class that escapes is `BrokenRecord`, and the
`except MeasurementError` clause catches it. Without the translation, that record would
fall into the same class as the one set apart as a "program defect" in the previous
section, and the two could not be told apart.

The previous lesson's chain measurement also gives the cost of the `from None` used
here: because the context is suppressed, someone looking from outside cannot see the
`TypeError`. This is a deliberate choice — the conversion detail does not leak outside
the domain layer. If keeping the detail is wanted, `from` is written and the chain
becomes two; the decision is which side of the boundary gets the information.

Translation has a limit too. The `TypeError` in the coefficient field was not
translated, because that error does not come from the conversion layer, it comes from
the code that **builds** the record. Translating a program defect into the domain's
class would make it look like something that can be handled.

#### Summary

- A custom exception class's place in the hierarchy is not a matter of style; it decides
  **who will catch it**.
- Five classes and six catches produce **30** pairs, **13** of which match; the domain
  root covers **4** classes while it does not cover the class that does not descend
  from it.
- A borrowed base gives the class meaning and, in exchange, scope: **2** of four domain
  classes leak into a foreign `except ValueError` or `except KeyError` clause.
- The domain root handles **7** errors with **1** name; the clause counting three names
  gives the same result with **3** names, the borrowed base catches only **3** errors,
  the broad clause swallows **1** program defect.
- Clause order must go narrow to broad: when the root is written first, the narrow
  clause runs **0** times and silently becomes dead code.

#### Next Step

This lesson's twelve records sat ready in memory; the `value` field was already a
string, and the only problem was converting it to a number. When records come from a
file, one more layer gets inserted: what sits in a file is neither a string nor a
number, it is **bytes**. Converting bytes to a string is an encoding decision, and if
that decision is wrong, the error never reaches the domain layer at all — it shows up
while reading. The next lesson measures what the same bytes give in text mode versus
binary mode, and exactly where a wrong encoding produces its exception.

### File Operations

Source: https://academia.sh/en/courses/python-fundamentals/file-operations

The same 41 bytes give a byte sequence of length 41 in binary mode, and a 34-character string in text mode with utf-8; a wrong encoding breaks at read time, not open time, and latin-1 never breaks at all, silently producing 41 wrong characters.

The previous lesson's twelve records sat ready in memory. The `value` field was already
a string; the only job was converting it to a number and checking it against the
field's bounds. The domain root paid off because the error's source was the domain
itself.

When records come from a file, one more layer gets inserted. What sits in a file is
neither a string nor a number — it is **bytes**. Converting bytes to a string is an
**encoding** decision, and if that decision is wrong, the error never reaches the
domain layer at all; it shows up while reading. The concepts of file, encoding, and
byte were built in the How Computers Work course; the **concept** was built there, what
is measured here is what Python's two file modes give on the same bytes.

#### Two Modes, Two Types

`open` works in two modes, and the difference between them is not an option, it is
**the type of the returned object**.

**Binary mode** (`"rb"`, `"wb"`) gives a byte sequence. Whatever is in the file is what
you get; no translation happens. The length is **the byte count**.

**Text mode** (`"r"`, `"w"`) gives a string. A decoder steps in between: bytes are
turned into characters according to an **encoding**, and line endings are translated.
The length is **the character count**.

If `encoding` is not given in text mode, the interpreter picks a default, and that
default **depends on the environment it runs in**. Same file, same program, a different
environment — a different result. Every measurement in this lesson writes the encoding
explicitly; that is the condition for it to be measurable.

#### Same Bytes, Different Encodings

The measurement starts with a single piece of text: a measurement line carrying Turkish
letters, a degree sign, and an em dash. This text is encoded with utf-8, and the
resulting bytes are tried for decoding with **five separate encodings**.

The measurement's assumptions:

- **EF19** — The entire measurement runs on a single byte sequence; the sequence is
  produced by **encoding** the text, it is not written by hand.
- **EF20** — The oracle is the source text itself: whether the decoded string is
  **identical** to the source can be asked, because we produced the source ourselves.
- **EF21** — The "broken byte position" is read from the exception object's own field;
  the error message is not parsed.
- **EF22** — The encoding names come from the language's own decoder catalog; none of
  them belong to an outside tool.
- **EF23** — The chunk-size scan splits the same bytes at different sizes and decodes
  every chunk **separately**; no state is carried between chunks.

```python
"""Same bytes, different encodings: does it decode, is it identical, where does it break."""

TEXT = "kuzey yamaç: 12,5 °C — ölçüm tamam"
RAW = TEXT.encode("utf-8")

print(f"characters {len(TEXT)} | utf-8 bytes {len(RAW)} | "
      f"multi-byte characters {sum(len(c.encode('utf-8')) > 1 for c in TEXT)}")

ENCODINGS = ("utf-8", "utf-16", "latin-1", "ascii", "cp1254")

print()
print(f"{'encoding':<12s}{'decoded':<10s}{'characters':>11s}{'identical':>10s}"
      f"   broken class / byte position")
for name in ENCODINGS:
    try:
        decoded = RAW.decode(name)
    except UnicodeDecodeError as e:
        print(f"  {name:<10s}{'no':<10s}{'-':>11s}{'-':>10s}"
              f"   {type(e).__name__} / {e.start}")
    else:
        print(f"  {name:<10s}{'yes':<10s}{len(decoded):>11d}"
              f"{('yes' if decoded == TEXT else 'no'):>10s}   -")

print()
print(f"{'error mode':<20s}{'characters':>11s}{'identical':>10s}   first seven characters")
for mode in ("ignore", "replace", "backslashreplace"):
    c = RAW.decode("ascii", errors=mode)
    print(f"  {mode:<18s}{len(c):>11d}{('yes' if c == TEXT else 'no'):>10s}"
          f"   {c[:7]!r}")

print()
print(f"{'chunk size':<12s}{'chunks':>7s}{'broken chunks':>15s}")
for size in range(4, 25, 2):
    chunks = [RAW[i:i + size] for i in range(0, len(RAW), size)]
    broken = 0
    for chunk in chunks:
        try:
            chunk.decode("utf-8")
        except UnicodeDecodeError:
            broken += 1
    print(f"  {size:<10d}{len(chunks):>7d}{broken:>15d}")
whole = 0
try:
    RAW.decode("utf-8")
except UnicodeDecodeError:
    whole = 1
print(f"broken when the same bytes are decoded whole {whole}")
```

```
characters 34 | utf-8 bytes 41 | multi-byte characters 6

encoding    decoded    characters identical   broken class / byte position
  utf-8     yes                34       yes   -
  utf-16    no                  -         -   UnicodeDecodeError / 40
  latin-1   yes                41        no   -
  ascii     no                  -         -   UnicodeDecodeError / 10
  cp1254    yes                41        no   -

error mode           characters identical   first seven characters
  ignore                     28        no   'kuzey y'
  replace                    41        no   'kuzey y'
  backslashreplace           80        no   'kuzey y'

chunk size   chunks  broken chunks
  4              11              4
  6               7              2
  8               6              2
  10              5              2
  12              4              2
  14              3              2
  16              3              0
  18              3              0
  20              3              2
  22              2              0
  24              2              2
broken when the same bytes are decoded whole 0
```

#### The Silent Wrong Answer

The first line gives the lesson's core: **34**-character text holds **41** bytes. The
**7**-byte difference comes from the **6** multi-byte characters. Character count and
byte count are independent of each other, and which one you are counting depends on
which mode you read in.

The encoding table shows three separate behaviors. **utf-8** decodes, gives **34**
characters, and is **identical** to the source. This is the definition of the correct
encoding.

**ascii** and **utf-16** cannot decode and raise `UnicodeDecodeError`. The positions
differ: ascii breaks at the **10th** byte — the first byte of the first Turkish letter.
utf-16, by contrast, breaks at the **40th** byte, nearly at the end; because utf-16
reads bytes in pairs and **41** is an odd number, the last byte is left unpaired. Both
break, but **where** they break comes from the encoding's own rule.

The real finding is the **latin-1** and **cp1254** rows. Both **decode** — raising no
exception at all — and give **41** characters. That is, they turn every byte into a
character. But they are **not identical**. A single-byte encoding accepts every byte
that comes its way as valid; there is nothing there to break on. This is the
measurement's most dangerous result: **a wrong encoding does not always produce an
exception.** An encoding that raises one warns you; one that does not silently gives
back the wrong text, and the mistake is only noticed by a human reading it.

Error modes are the deliberate version of this. `ignore` drops bytes it cannot decode
and **28** characters remain — the bytes of **6** characters fell away. `replace` puts
one character in place of every faulty byte and gives **41**; the number matches the
byte count because the replacement is done byte by byte. `backslashreplace` describes
every faulty byte with a four-character notation, and the length grows to **80**. All
three have `identical` reading **no**: giving an error mode does not recover the data,
it only prevents **breaking**.

#### The Chunk Boundary

The last table gives the cost of reading in chunks in binary mode. The same **41**
bytes are split into different sizes, and each chunk is decoded **separately**.

The result is irregular. At **4**-byte chunks, **4** of **11** chunks break; at sizes
**16** and **18**, none break; at **20**, **2** chunks break again, **0** at **22**,
**2** at **24**. The break count does not depend on the chunk size's magnitude, it
depends on whether a chunk boundary **falls in the middle of a multi-byte character**.
When the same bytes are decoded **whole**, the break count is **0**.

The direct consequence of this: if you are reading in binary mode and decoding the
bytes yourself, carrying **state** between chunks is your job. Text mode already does
this work — the decoder holds back a character left incomplete at a chunk boundary.
Binary mode is fast and impartial; because it does not take on the translation, it does
not take on translation's problems either, and leaves them to the caller.

#### Mode, Type, and the Break Point

The second measurement works with a real file. The file is built in a **temporary
directory**, and the directory is deleted when the measurement ends; the measurement
leaves no permanent trace.

- **EF24** — The file is produced in a temporary directory, and its bytes come from the
  previous measurement's text; no path is printed, because a path depends on the
  environment.
- **EF25** — "Length" is bytes in binary mode, characters in text mode; the two are
  measured under the same name but are not the same thing.
- **EF26** — The break-point measurement tries opening and reading as **separate
  steps**; which one produces the exception is read from the run.
- **EF27** — In the newline measurement, the file carries **two separate line-ending
  forms**, one two characters long, one a single character; the translation difference
  arises from this.

```python
"""Text mode vs binary mode: same file, different type, different length, different break point."""
import tempfile
from pathlib import Path

TEXT = "kuzey yamaç: 12,5 °C — ölçüm tamam"
LINED = b"kuzey\r\nyama\xc3\xa7\nolcum\r\n"

with tempfile.TemporaryDirectory() as temp_dir:
    data_file = Path(temp_dir) / "measurement.dat"
    data_file.write_bytes(TEXT.encode("utf-8"))
    lines_file = Path(temp_dir) / "lines.dat"
    lines_file.write_bytes(LINED)

    print(f"{'read mode':<22s}{'type':<8s}{'length':>8s}{'identical':>10s}")
    with open(data_file, "rb") as f:
        binary = f.read()
    print(f"  {'binary':<20s}{type(binary).__name__:<8s}{len(binary):>8d}{'-':>10s}")
    for name, encoding, mode in (("text utf-8", "utf-8", "strict"),
                             ("text latin-1", "latin-1", "strict"),
                             ("text ascii replace", "ascii", "replace")):
        with open(data_file, "r", encoding=encoding, errors=mode) as f:
            text = f.read()
        print(f"  {name:<20s}{type(text).__name__:<8s}{len(text):>8d}"
              f"{('yes' if text == TEXT else 'no'):>10s}")

    print()
    print("step                result")
    try:
        f = open(data_file, "r", encoding="ascii")
    except UnicodeDecodeError as e:
        print(f"  {'open':<18s}{type(e).__name__} / byte {e.start}")
    else:
        print(f"  {'open':<18s}succeeded, file opened")
        try:
            f.read()
        except UnicodeDecodeError as e:
            print(f"  {'read':<18s}{type(e).__name__} / byte {e.start}")
        finally:
            f.close()

    print()
    print(f"{'read form':<22s}{'length':>8s}{'CR count':>11s}{'lines':>7s}")
    with open(lines_file, "rb") as f:
        raw = f.read()
    print(f"  {'binary':<20s}{len(raw):>8d}{raw.count(13):>11d}"
          f"{len(raw.splitlines()):>7d}")
    for name, newline_arg in (("text (default)", None), ('text newline=""', "")):
        with open(lines_file, "r", encoding="utf-8", newline=newline_arg) as f:
            text = f.read()
        with open(lines_file, "r", encoding="utf-8", newline=newline_arg) as f:
            lines = f.readlines()
        print(f"  {name:<20s}{len(text):>8d}{text.count(chr(13)):>11d}"
              f"{len(lines):>7d}")

print("temp directory deleted:", not Path(temp_dir).exists())
```

```
read mode             type      length identical
  binary              bytes         41         -
  text utf-8          str           34       yes
  text latin-1        str           41        no
  text ascii replace  str           41        no

step                result
  open              succeeded, file opened
  read              UnicodeDecodeError / byte 10

read form               length   CR count  lines
  binary                    21          2      3
  text (default)            18          0      3
  text newline=""           20          2      3
temp directory deleted: True
```

#### Reading the Numbers

The first table gives the two modes' difference by type. Binary mode returns `bytes`
and the length is **41**; text mode returns `str`, and with the correct encoding the
length is **34**. Same file, same content, two different numbers — and both are
correct. `latin-1` again gives **41**, and the `identical` column reads **no**: the
silent wrong answer seen in the in-memory measurement repeats exactly the same way when
reading from a file.

The second table is the lesson's sharpest finding. The `open` call with `ascii`
encoding **succeeds** — the file opens, no exception comes out. The exception comes at
the `read` call, at byte **10**. `open` only prepares the file and stores the encoding
name; the decoding work is done at the first read. An encoding error is **not an open
error, it is a read error**, and the practical consequence is this: a `try` block
wrapping the `open` call cannot catch this error. The block that catches it has to wrap
the read.

The third table counts newline translation. The file is **21** bytes and carries **2**
carriage-return characters inside it, because two lines end with a two-character line
ending and one with a single-character line ending. When text mode reads with the
default setting, the length drops to **18** and the carriage-return count becomes
**0**: the decoder turned every line-ending form into a single character. When
`newline=""` is given, the translation turns off, and the length is **20**, the
carriage-return count **2**.

The line count is **3** across all three reads. Translation does not change **where**
the lines end, it only changes how many characters the line ending is written with.
Comparing a file's length when read in text mode to the file's byte size therefore
gives the wrong result: both encoding and newline translation get in between.

#### The Write Side and Open Modes

The read side **assumes** the encoding; the write side **establishes** it. The third
measurement tries writing the same text with five encodings and does a round-trip test
on each: when the encoded bytes are decoded back with the same encoding, does the
source text come back?

- **EF28** — The round-trip test is done with **its own** encoding in every case; what
  is measured is whether the encoding can carry the text.
- **EF29** — The open modes are tried on the same file **in sequence**; at every step
  the file's byte count is read from the file system, not computed.
- **EF30** — The file is again built in a temporary directory; no path is printed, and
  the directory is deleted when the measurement ends.

```python
"""Write side: unencodable characters and file open modes."""
import tempfile
from pathlib import Path

TEXT = "kuzey yamaç: 12,5 °C — ölçüm tamam"

print(f"{'encoding':<12s}{'encoded':<10s}{'bytes':>6s}"
      f"{'round trip identical':>22s}   broken class / position")
for name in ("utf-8", "utf-16", "latin-1", "ascii", "cp1254"):
    try:
        raw = TEXT.encode(name)
    except UnicodeEncodeError as e:
        print(f"  {name:<10s}{'no':<10s}{'-':>6s}{'-':>22s}"
              f"   {type(e).__name__} / {e.start}")
    else:
        identical = "yes" if raw.decode(name) == TEXT else "no"
        print(f"  {name:<10s}{'yes':<10s}{len(raw):>6d}{identical:>22s}   -")

print()
with tempfile.TemporaryDirectory() as temp_dir:
    log_file = Path(temp_dir) / "measurement.log"
    print(f"{'mode':<6s}{'call':<8s}{'file bytes':>12s}   result")
    for step, mode in (("1.", "w"), ("2.", "w"), ("3.", "a"), ("4.", "x")):
        try:
            with open(log_file, mode, encoding="utf-8") as f:
                f.write(TEXT + "\n")
        except FileExistsError as e:
            print(f"  {mode:<4s}{step:<8s}{log_file.stat().st_size:>12d}"
                  f"   {type(e).__name__}")
        else:
            print(f"  {mode:<4s}{step:<8s}{log_file.stat().st_size:>12d}   written")
    with open(log_file, "r", encoding="utf-8") as f:
        lines = f.readlines()
    print(f"final state lines {len(lines)}, "
          f"characters {sum(len(s) for s in lines)}, "
          f"bytes {log_file.stat().st_size}")
```

```
encoding    encoded    bytes  round trip identical   broken class / position
  utf-8     yes           41                   yes   -
  utf-16    yes           70                   yes   -
  latin-1   no             -                     -   UnicodeEncodeError / 21
  ascii     no             -                     -   UnicodeEncodeError / 10
  cp1254    yes           34                   yes   -

mode  call      file bytes   result
  w   1.                42   written
  w   2.                42   written
  a   3.                84   written
  x   4.                84   FileExistsError
final state lines 2, characters 70, bytes 84
```

The `cp1254` row is the counterpart of the read table and the lesson's most instructive
pair. There, `cp1254` **silently misread** utf-8 bytes; here, that same encoding writes
the text into **34** bytes, and the round trip comes back **identical**. Encoding is
not a property of a file; it is **the agreement between the writer and the reader**,
and the file itself does not carry that agreement. What was wrong was not `cp1254`, it
was mistaking bytes written with utf-8 for `cp1254`.

The write side parts ways with the read side at one point: `latin-1` and `ascii`
**break** here. `latin-1` breaks at the **21st** character, at the em dash; `ascii`
breaks at the **10th** character, at the first Turkish letter. When writing, a
character that cannot be encoded cannot silently pass through, because that character
has **no counterpart** in the encoding. When reading, a silent wrong answer was
possible, because every byte had some counterpart. `utf-16` writes the text into **70**
bytes — against utf-8's **41** — and it too comes back identical; choosing an encoding
is a **size** decision as much as a correctness one.

The open-modes table sets apart three separate behaviors. `"w"` writes **42** bytes on
the first call; after the second `"w"` call, the file is still **42** — the mode
**truncated** the file and wrote it from scratch, the second record erased the first.
The `"a"` call brings it up to **84**: append mode does not truncate. The `"x"` call
raises `FileExistsError` without writing anything, and the byte count stays at **84**.
The last line says the file holds **2** lines, **70** characters, and **84** bytes; the
gap between character count and byte count stands here too.

#### Summary

- Binary mode gives `bytes` and the length is the byte count; text mode gives `str`,
  the length is the character count — the same file reads as **41** and **34**.
- A wrong encoding does not always break: `ascii` breaks at the **10th** byte,
  `utf-16` at the **40th**, while `latin-1` and `cp1254` never break and silently
  produce **41** wrong characters.
- Error modes prevent breaking, they do not recover the data: `ignore` gives **28**,
  `replace` **41**, `backslashreplace` **80** characters, and none is identical to the
  source.
- Decoding bytes chunk by chunk depends on the chunk boundary: the same **41** bytes
  break in **0**, **2**, or **4** chunks depending on size, and in **0** when decoded
  whole.
- An encoding error shows up not at the `open` call but at the `read` call; newline
  translation brings the length down from **21** to **18** but keeps the line count at
  **3**.
- Encoding is a property not of the file but of the writer and the reader: `cp1254`
  gives an identical round trip writing in **34** bytes while silently corrupting
  utf-8 bytes on read; an unencodable character breaks at the **21st** character in
  `latin-1` when writing, the **10th** in `ascii`.

#### Next Step

Every file example in this lesson was opened inside a `with` block, and the file
closed when the block ended — the temporary directory could even be deleted at the end
of the measurement, because no open handle was left behind. `with` has been used as a
habit up to now; but it too is syntax, and there is a protocol underneath it. The next
lesson measures that protocol: which special method `with` calls on entry, which on
exit, and what changes when **an exception occurs** inside the block. The shared
setup's `with n + exception` line's three calls get paid off there.

### Context Managers

Source: https://academia.sh/en/courses/python-fundamentals/context-managers

The with block calls __enter__ on entry and __exit__ on exit; an exception-free block produces 2 calls, an exception-carrying one 3, and __exit__ sees the exception — when the return value is True, the count drops back to 2 because the exception does not escape.

Every file example in the previous lesson was opened inside a `with` block, and the
file closed when the block ended. The temporary directory could even be deleted at the
end of the measurement, because no open handle was left behind. `with` has been used as
a habit up to now.

But `with` is syntax too, and this course's rule is: every syntax form is a
**shorthand** and calls a specific **special method** by name. This lesson measures
that protocol. Resource lifetime and releasing a resource were built in the Operating
System Concepts course; the **concept** was built there, what is measured here is which
protocol Python realizes it with: `with` calls `__enter__` on entry, `__exit__` on
exit — and when **an exception occurs** inside the block, `__exit__` **is still
called**.

#### The Protocol Is Just Two Methods

There is exactly one condition for an object to be usable in a `with` statement:
defining `__enter__` and `__exit__`. Its type's name, which class it descends from,
what it does — none of that matters. No protocol, no syntax; if the protocol is there,
the object enters the `with`.

`__enter__` takes no arguments, and the value it returns is the name bound with `as`. It
does not have to return the resource itself; it can return a different object.

`__exit__` takes **three arguments**: the raised exception's **type**, **object**, and
**traceback**. If the block ended without an exception, all three are empty. And
`__exit__` **returns** a value; this return value decides whether the exception gets
swallowed. A return counted as true swallows the exception, one counted as false lets
it escape.

The shared setup's `Tracker` object defines these two methods, and its `__exit__`
returns `False` — meaning it **does not swallow**. The measurement adds a sibling to
it: `Swallower`, which only changes the return value.

#### The Measurement's Setup

Three syntax forms are run: `with` with no exception, `with` with an exception, and
`with` with an exception but on an object that swallows. For each form, how many times
which special method got called is counted. The measurement log writes the exception
as an event too: if a form ends with an exception, the class name drops into the log
with a `!` prefix.

- **EF31** — `Tracker`'s `__enter__` and `__exit__` methods are taken unchanged from the
  shared setup; `__exit__` still returns `False`.
- **EF32** — `Swallower` only changes the `__exit__` return and calls it **without
  disturbing** the base class's log; the log's layout is the shared setup's layout.
- **EF33** — `Witness` also does not disturb the base class's log; it additionally
  writes the triple `__exit__` sees into a separate list. `Tracker`'s behavior is not
  touched.
- **EF34** — The call count is the number of items in the log; `!ValueError` is an item
  too, because what is measured is **the event sequence a form produces**, not only
  method calls.
- **EF35** — The return-value test is read by calling `__exit__` directly; whether it
  swallows is separately tested with a real `with` block. The two paths confirm each
  other.

```python
"""Context manager protocol: which method with calls on entry and exit."""

LOG = []
SEEN = []


def record(name):
    LOG.append(name)


class Tracker:
    """The core of the shared setup; the context part is used here."""

    def __init__(self, items=(1, 2, 3)):
        self.items = list(items)

    def __enter__(self):
        record("__enter__")
        return self

    def __exit__(self, kind, value, tb):
        record("__exit__")
        return False          # does not swallow the exception


class Swallower(Tracker):
    """__exit__ returns True: the exception does not escape."""

    def __exit__(self, kind, value, tb):
        super().__exit__(kind, value, tb)
        return True


class Witness(Tracker):
    """Separately logs the triple __exit__ sees; does not disturb Tracker's log."""

    def __init__(self, name, swallows=False, items=(1, 2, 3)):
        super().__init__(items)
        self.name = name
        self.swallows = swallows

    def __exit__(self, kind, value, tb):
        SEEN.append((self.name, kind.__name__ if kind else "-",
                        str(value) if value else "-", tb is not None))
        super().__exit__(kind, value, tb)
        return self.swallows


def measure(function):
    """Runs a syntax form and returns the protocols it called."""
    LOG.clear()
    try:
        function()
    except Exception as e:
        LOG.append(f"!{type(e).__name__}")
    return list(LOG)


def context():
    with Tracker():
        pass


def context_with_exception():
    with Tracker():
        raise ValueError("example")


def context_swallowing():
    with Swallower():
        raise ValueError("example")


FORMS = (
    ("with n", context),
    ("with n + exception", context_with_exception),
    ("with swallower + exception", context_swallowing),
)

print(f"{'syntax':<27s} {'calls':>6s}  protocol sequence")
for name, function in FORMS:
    c = measure(function)
    print(f"{name:<27s} {len(c):6d}  {' '.join(c)}")

print()
print(f"{'class':<12s}{'__exit__ return':<18s}exception escaped")
for name, cls in (("Tracker", Tracker), ("Swallower", Swallower)):
    n = cls()
    return_value = n.__exit__(ValueError, ValueError("example"), None)
    LOG.clear()
    try:
        with cls():
            raise ValueError("example")
    except ValueError:
        outcome = "yes"
    else:
        outcome = "no"
    print(f"  {name:<10s}{str(return_value):<18s}{outcome}")

print()
SEEN.clear()
with Witness("no exception"):
    pass
try:
    with Witness("exception"):
        raise ValueError("measurement out of range")
except ValueError:
    pass
print(f"{'witness':<16s}{'type it saw':<14s}{'value it saw':<26s}trace object present")
for name, kind, value, tb in SEEN:
    print(f"  {name:<14s}{kind:<14s}{value:<26s}{'yes' if tb else 'no'}")
```

```
syntax                       calls  protocol sequence
with n                           2  __enter__ __exit__
with n + exception               3  __enter__ __exit__ !ValueError
with swallower + exception       2  __enter__ __exit__

class       __exit__ return   exception escaped
  Tracker   False             yes
  Swallower True              no

witness         type it saw   value it saw              trace object present
  no exception  -             -                         no
  exception     ValueError    measurement out of range  yes
```

#### What the Three Lines Say

The first two lines are the shared setup's `with` lines. **`with n` produces two
calls:** `__enter__` and `__exit__`. **`with n + exception` produces three events:**
`__enter__`, `__exit__`, and `!ValueError`. The difference needs to be read correctly —
the extra one **is not a method call**. `__enter__` ran once, `__exit__` ran once, in
both forms. What is extra is that the form **ends with an exception**.

This gives the lesson's central sentence: **when an exception occurs, `__exit__` still
runs.** The block was cut short, the remaining lines did not run, control jumped out of
the `with` statement — and `__exit__` still ran. This is what makes the resource close,
and this is the reason to write `with` at all.

The third line tries the same exception with `Swallower`, and the call count drops to
**2**. The sequence is still `__enter__ __exit__` — the method calls did not change.
The only thing that changed is that `!ValueError` did not drop into the log: the
exception **did not escape**. The middle table confirms this. `Tracker`'s `__exit__`
returns `False` and the exception **escaped**; `Swallower`'s returns `True` and it
**did not**. A single-line return value brought a three-event form down to two.

The last table gives **what `__exit__` sees**. In the exception-free block, all three
arguments are seen empty: type `-`, value `-`, trace **no**. In the exception-carrying
block, the type is `ValueError`, the value is the exception's message, and the trace
object **is present**. `__exit__` does not merely swallow or not swallow the exception;
it **knows which exception it is** and can decide by looking at it. A context manager
that swallows one specific class and lets the others through is written with this
triple.

#### Nested Blocks and try/finally

The second measurement answers two questions at once. First: when two context managers
are nested, which one sees the exception, and if the inner one swallows it, what does
the outer one see? Second: can the difference between `with` and `try/finally` be
measured?

- **EF36** — Four combinations are tried in the nested measurement: the outer and inner
  managers each separately swallow or do not swallow. Nothing else changes.
- **EF37** — The "saw" column is read from the type argument passed to `__exit__`; if it
  was passed empty, `-` is written.
- **EF38** — The exit-form table tries four paths: normal ending, `return`, `break`, and
  an exception. For `return` and `break`, the block is placed inside a loop and a
  function, because both only make sense there.
- **EF39** — The `with` arm and the `try/finally` arm try **the same four paths**; the
  only difference between them is that one has a context manager, the other a
  `finally` block.

```python
"""Nested with, exit forms, and comparison with try/finally."""

LOG = []
SEEN = []


def record(name):
    LOG.append(name)


class Tracker:
    def __init__(self, items=(1, 2, 3)):
        self.items = list(items)

    def __enter__(self):
        record("__enter__")
        return self

    def __exit__(self, kind, value, tb):
        record("__exit__")
        return False


class Witness(Tracker):
    """Logs by name and takes whether it swallows at construction."""

    def __init__(self, name, swallows=False, items=(1, 2, 3)):
        super().__init__(items)
        self.name = name
        self.swallows = swallows

    def __exit__(self, kind, value, tb):
        SEEN.append((self.name, kind.__name__ if kind else "-"))
        super().__exit__(kind, value, tb)
        return self.swallows


print(f"{'outer':<10s}{'inner':<10s}{'calls':>6s}  {'sequence':<40s}"
      f"{'outer saw':<12s}{'inner saw':<12s}exception")
for outer_swallows in (False, True):
    for inner_swallows in (False, True):
        LOG.clear()
        SEEN.clear()
        try:
            with Witness("outer", outer_swallows), Witness("inner", inner_swallows):
                raise ValueError("measurement out of range")
        except ValueError:
            outcome = "escaped"
        else:
            outcome = "swallowed"
        seen = dict(SEEN)
        print(f"  {('swallows' if outer_swallows else 'passes'):<10s}"
              f"{('swallows' if inner_swallows else 'passes'):<9s}{len(LOG):6d}  "
              f"{' '.join(LOG):<40s}{seen['outer']:<12s}"
              f"{seen['inner']:<12s}{outcome}")

print()
FINALLY = []


def with_form(form):
    LOG.clear()
    SEEN.clear()
    for _ in range(1):
        with Witness("single"):
            if form == "return":
                return
            if form == "break":
                break
            if form == "exception":
                raise ValueError("measurement out of range")


def finally_form(form):
    FINALLY.clear()
    for _ in range(1):
        try:
            if form == "return":
                return
            if form == "break":
                break
            if form == "exception":
                raise ValueError("measurement out of range")
        finally:
            FINALLY.append("finally")


print(f"{'exit form':<16s}{'__exit__ called':<19s}{'finally ran':<17s}"
      f"__exit__ saw the exception")
for form in ("normal", "return", "break", "exception"):
    try:
        with_form(form)
    except ValueError:
        pass
    called = "yes" if "__exit__" in LOG else "no"
    saw = "yes" if SEEN and SEEN[0][1] != "-" else "no"
    try:
        finally_form(form)
    except ValueError:
        pass
    ran = "yes" if FINALLY else "no"
    print(f"  {form:<14s}{called:<19s}{ran:<17s}{saw}")
```

```
outer     inner      calls  sequence                                outer saw   inner saw   exception
  passes    passes        4  __enter__ __enter__ __exit__ __exit__   ValueError  ValueError  escaped
  passes    swallows      4  __enter__ __enter__ __exit__ __exit__   -           ValueError  swallowed
  swallows  passes        4  __enter__ __enter__ __exit__ __exit__   ValueError  ValueError  swallowed
  swallows  swallows      4  __enter__ __enter__ __exit__ __exit__   -           ValueError  swallowed

exit form       __exit__ called    finally ran      __exit__ saw the exception
  normal        yes                yes              no
  return        yes                yes              no
  break         yes                yes              no
  exception     yes                yes              yes
```

#### Reading the Numbers

All **four** rows of the nested table give a call count of **4** and the same
sequence: `__enter__ __enter__ __exit__ __exit__`. Entry goes outer to inner, exit
inner to outer. The swallowing decision **cancels no call** — the four methods run in
all four combinations.

What changes are the "saw" columns. The inner manager sees `ValueError` in **all four**
rows; the exception forms closest to it and reaches it first. The outer one sees it
**only when the inner does not swallow**, and gets `-` when it does. The second and
fourth rows show this: once the inner one has swallowed it, there is no exception left
for the outer one, and `__exit__` is called with an empty triple.

The third row shows the reverse: the inner one did not swallow, the outer one saw it
and swallowed it. The exception did not escape. Swallowing authority travels **outward
in order**, and the first one to swallow cuts the chain.

The second table sets apart what `with` and `try/finally` share and where they differ.
What they share: in **all four** exit forms, both `__exit__` gets called and `finally`
runs — however the block ends, the cleanup code runs. This is why `with` is not an
"error-catching" tool; it is an **exit** tool, active on all four paths.

The difference is in the last column. `__exit__` **sees the exception** only in the
exception form; in the other three it is called with an empty triple. A `finally`
block, by contrast, is never given a triple in any form — while it runs, it does not
know which exception is flying and cannot decide by looking at it. The only way to
swallow an exception inside `finally` is to exit from there with a new statement, and
that swallows **every** exception regardless of what happened in the block. `__exit__`,
by contrast, can look at the type and choose.

The choice follows from this: if cleanup is needed once, in one place, `finally` is
enough. If the same cleanup repeats in more than one place, or the decision depends on
the exception's **type**, the protocol is written.

#### The Bound Object and Cleanup's Own Error

Two details remain, both sitting at the protocol's edge. The first is the name bound
with `as`: this name is **not the manager itself**, it is what `__enter__` returned.
The second is `__exit__` breaking on its own — cleanup can fail too.

- **EF40** — `Wrapper` only changes `__enter__`'s return; it takes its log from the base
  class. `Broken` only adds an exception during cleanup.
- **EF41** — The bound object's identity is tested with `is`; **no identity number is
  printed**.
- **EF42** — Chain length is counted with **the same method** as the first lesson's
  measurement: unsuppressed context and cause links are followed.

```python
"""What __enter__ returns and an exception born inside __exit__."""

LOG = []


class Tracker:
    def __init__(self, items=(1, 2, 3)):
        self.items = list(items)

    def __enter__(self):
        LOG.append("__enter__")
        return self

    def __exit__(self, kind, value, tb):
        LOG.append("__exit__")
        return False


class Wrapper(Tracker):
    """__enter__ returns not itself but another object for the block to use."""

    def __enter__(self):
        super().__enter__()
        return self.items


class Broken(Tracker):
    """Produces an exception of its own during cleanup."""

    def __exit__(self, kind, value, tb):
        super().__exit__(kind, value, tb)
        raise RuntimeError("could not close the resource")


n = Tracker()
with n as bound:
    pass
print("Tracker: is the object bound with as the manager itself:", bound is n)

k = Wrapper()
with k as bound:
    pass
print("Wrapper: is the object bound with as the manager itself:", bound is k)
print("Wrapper: is the object bound with as what __enter__ returned:",
      bound is k.items)

print()
print(f"{'state':<36s}{'escapes as':<15s}{'context':<14s}chain")
for name, cls, raises in (("block clean, __exit__ clean", Tracker, False),
                          ("block raises, __exit__ clean", Tracker, True),
                          ("block clean, __exit__ breaks", Broken, False),
                          ("block raises, __exit__ breaks", Broken, True)):
    try:
        with cls():
            if raises:
                raise ValueError("measurement out of range")
    except BaseException as exc:
        escapes = type(exc).__name__
        context = type(exc.__context__).__name__ if exc.__context__ else "-"
        length, cur = 0, exc
        while cur is not None:
            length += 1
            cur = cur.__cause__ or (None if cur.__suppress_context__
                                        else cur.__context__)
    else:
        escapes = context = "-"
        length = 0
    print(f"  {name:<34s}{escapes:<15s}{context:<14s}{length}")
```

```
Tracker: is the object bound with as the manager itself: True
Wrapper: is the object bound with as the manager itself: False
Wrapper: is the object bound with as what __enter__ returned: True

state                               escapes as     context       chain
  block clean, __exit__ clean       -              -             0
  block raises, __exit__ clean      ValueError     -             1
  block clean, __exit__ breaks      RuntimeError   -             1
  block raises, __exit__ breaks     RuntimeError   ValueError    2
```

The first three lines settle the `as` binding. Because `Tracker` returns itself, the
bound name is the manager. `Wrapper` returns a different object, and the bound name is
**not** the manager anymore. This explains why the thing bound with `as` when you open
a file with `with` is the file object: `__enter__` returns it.

The lower table counts cleanup's own error, and it ties into the first lesson's chain
measurement. When both the block and `__exit__` are clean, nothing escapes, chain
**0**. When only the block breaks, `ValueError` escapes, chain **1**. When only
`__exit__` breaks, `RuntimeError` escapes, chain also **1**.

The last row is the dangerous one. The block raised `ValueError`, `__exit__` was
called, and it broke too. What escapes is **`RuntimeError`** — that is, **cleanup's own
error** takes the domain error's place. The real error is not lost: it sits in the
`__context__` field, and the chain is **2**. But an outside `except ValueError` clause
no longer matches it, because the escaping class changed. This is why the work inside
`__exit__` **must not break**; if it can, it has to be handled inside itself.

#### Summary

- `with` is a shorthand: it calls `__enter__` on entry, `__exit__` on exit, and an
  object not defining the protocol cannot enter a `with` statement.
- An exception-free block produces **2** events, an exception-carrying one **3**; the
  extra is not a method call, it is that the form ends with an exception —
  `__exit__` is called **once** in both.
- `__exit__` takes three arguments and sees the type, object, and trace in an
  exception-carrying block; when the return value counts as true, the exception is
  swallowed and the event count drops from **3** to **2**.
- In two nested managers, the call count is **4** across all four combinations; the
  inner one always sees the exception, the outer one only when the inner does not
  swallow it.
- `__exit__` and `finally` both run in all four exit forms; where they differ is that
  `__exit__` can **see the exception's type** and choose, while `finally` cannot.
- The name bound with `as` is not the manager, it is `__enter__`'s return value; if
  `__exit__` breaks on its own, **cleanup's error** escapes, the real error stays in
  the context, and the chain becomes **2**.

#### Next Step

The last syntax form this course measures is now behind us. Up to now every
measurement was a statement or expression calling the language's own protocols: `for`,
`if`, `+`, `with`. The next lesson's subject is not the language's syntax but a small,
separate language the **standard library** offers: regular expressions. There, a
pattern is written in a string, interpreted by a separate rule set, and a match comes
back as an object. What gets measured is what that object carries and in what order the
parentheses in the pattern get numbered.

### Regular Expressions

Source: https://academia.sh/en/courses/python-fundamentals/regular-expressions

Two of the pattern's seven opening parentheses do not capture, leaving five numbered groups; the number is given by the opening parenthesis's order, a non-match is None rather than an exception, and the match object carries every group's text, span, and name.

The previous lesson left behind the last syntax form this course measures. Every
measurement up to now was a statement or expression calling the language's own
protocols: `for`, `if`, `+`, `with`.

This lesson's subject is not the language's syntax. A **regular expression** is a
separate small language the standard library offers: a pattern is written in a string,
interpreted by a separate rule set, and the result comes back as an **object**. The
pattern-matching algorithms themselves — brute force search, hash-based search, prefix
table search — were built in the Algorithms course's text algorithms topic and are not
repeated here. What gets measured is something else: **what the match object carries**
and **in what order the parentheses in the pattern get numbered**.

#### A Parenthesis's Two Jobs in a Pattern

A parenthesis in a pattern has two separate jobs, and both appear with the same
notation. The first is **grouping**: it turns the piece inside it into a single unit
so a repetition or alternation operator can apply to it. The second is **capturing**:
it stores the matched text separately and numbers it.

An ordinary parenthesis does **both jobs at once**. A parenthesis opened with `(?:`
only groups, it does not capture — it gets no number. One opened with `(?P<name>` both
captures and puts a name next to its number.

The numbering rule is one sentence: **the number is given by the opening
parenthesis's left-to-right order.** Even in nested parentheses, the outer one is
numbered first, because its opening sits further left. Closing order is never looked
at.

#### Numbering the Groups

The measurement builds a single pattern that parses a measurement-record line: station
name, year, optional month, value, and unit. The pattern contains both a named and a
non-capturing parenthesis.

- **EF43** — Parenthesis counts are counted from the pattern string itself; the number
  of numbered groups is read from the compiled pattern's own field. The two are
  separate sources and confirm each other.
- **EF44** — The oracle is the setup that wrote the pattern: which piece should fall
  into which group is known in advance.
- **EF45** — Spans are read from the match object, not counted by hand; it is a
  **character** span, not a byte one.
- **EF46** — An optional group that does not match returns `None`; this is not an
  error, it is the declaration that the group did not take part.
- **EF47** — The nested-pattern measurement is kept separate purely to test numbering;
  it measures nothing else.

```python
"""Capture groups: numbering the parentheses and what the match object carries."""
import re

PATTERN = r"(?P<station>[a-z]+)-(\d{4})(?:-(\d{2}))?:\s*(-?\d+(?:[.,]\d+)?)\s*(°C|%)"
COMPILED = re.compile(PATTERN)

print(f"opening parens in pattern {PATTERN.count('(')} | "
      f"non-capturing parens {PATTERN.count('(?:')} | "
      f"named group {PATTERN.count('(?P<')} | numbered groups {COMPILED.groups}")
print("name -> number:", COMPILED.groupindex)

LINES = ("north-2031: 12,5 °C", "slope-2031-07: -3.5 °C",
            "summary-2031: 88 %", "broken record")

print()
print(f"{'line':<26s}{'matched':<9s}{'span':<10s}{'last group':>10s}   groups")
for s in LINES:
    e = COMPILED.search(s)
    if e is None:
        print(f"  {s:<24s}{'no':<9s}{'-':<10s}{'-':>10s}   -")
    else:
        print(f"  {s:<24s}{'yes':<9s}{str(e.span()):<10s}"
              f"{e.lastindex:>10d}   {e.groups()}")

print()
e = COMPILED.search(LINES[1])
print("second line's match object:")
print(f"  group(0) (the whole match): {e.group(0)!r}")
for i in range(1, COMPILED.groups + 1):
    print(f"  group({i}): {e.group(i)!r}  span {e.span(i)}")
print(f"  access by name: {e.group('station')!r} | "
      f"named group dict: {e.groupdict()}")

print()
NESTED = re.compile(r"((\d{4})-(\d{2}))")
i = NESTED.search("slope-2031-07: -3.5 °C")
print(f"numbered groups in nested pattern {NESTED.groups}; "
      f"number is given by opening-paren order:")
for n in range(1, NESTED.groups + 1):
    print(f"  group({n}) = {i.group(n)!r}")
```

```
opening parens in pattern 7 | non-capturing parens 2 | named group 1 | numbered groups 5
name -> number: {'station': 1}

line                      matched  span      last group   groups
  north-2031: 12,5 °C     yes      (0, 19)            5   ('north', '2031', None, '12,5', '°C')
  slope-2031-07: -3.5 °C  yes      (0, 22)            5   ('slope', '2031', '07', '-3.5', '°C')
  summary-2031: 88 %      yes      (0, 18)            5   ('summary', '2031', None, '88', '%')
  broken record           no       -                  -   -

second line's match object:
  group(0) (the whole match): 'slope-2031-07: -3.5 °C'
  group(1): 'slope'  span (0, 5)
  group(2): '2031'  span (6, 10)
  group(3): '07'  span (11, 13)
  group(4): '-3.5'  span (15, 19)
  group(5): '°C'  span (20, 22)
  access by name: 'slope' | named group dict: {'station': 'slope'}

numbered groups in nested pattern 3; number is given by opening-paren order:
  group(1) = '2031-07'
  group(2) = '2031'
  group(3) = '07'
```

#### What the Match Object Carries

The first line gives the arithmetic: the pattern has **7** opening parentheses, **2**
of them non-capturing, leaving **5** numbered groups. One number came from the pattern
string, the other from the compiled pattern, and they agree. The named group's number
is not lost: the name `station` corresponds to number **1**. A name is an alias, it
does not replace the number.

The second table resolves four lines. Three match, one does not. The matching ones'
`groups` column gives a five-item tuple, and the optional month group returns `None`
on lines where no month is written. `None` is not an error here: it means the group
**did not take part**, and it is distinct from an empty string — an empty string
would mean "it matched, but its content was empty."

The third block opens up everything the match object carries. `group(0)` is **the
whole match** and is not a group; numbers start after it. Every group gives both its
text and its **span**: `2031` sits between characters **6** and **10** of the record.
This span lets you point at the matched text inside the source without pulling it out
of place.

The last block settles the numbering. The nested pattern has **3** groups, and
`group(1)` gives the outer whole, `group(2)` the year, `group(3)` the month. Even
though the outer parenthesis closes later, it is numbered **first**, because it
**opens first**.

#### Forms of Searching

The second measurement runs the same pattern with different search forms and tests
what a non-match produces.

- **EF48** — Three search forms are tried with the **same** compiled pattern and the
  same lines; only the method called changes.
- **EF49** — Multi-match counts are read from the length of the list; the line count
  is counted from the text's own lines.
- **EF50** — In the non-matching case, the returned value is printed directly; that the
  result is not an exception is shown by the run.
- **EF51** — The position measurement searches the same content once as a string and
  once as a byte sequence; the difference comes purely from the content's type.

```python
"""Forms of matching search, the non-matching case, and position on a string vs bytes."""
import re

PATTERN = r"(?P<station>[a-z]+)-(\d{4})(?:-(\d{2}))?:\s*(-?\d+(?:[.,]\d+)?)\s*(°C|%)"
COMPILED = re.compile(PATTERN)

ATTEMPTS = ("north-2031: 12,5 °C", "  north-2031: 12,5 °C",
             "north-2031: 12,5 °C extra", "broken record")

print(f"{'line':<30s}{'search':<9s}{'match':<8s}fullmatch")
for s in ATTEMPTS:
    results = [("yes" if method(s) else "no")
         for method in (COMPILED.search, COMPILED.match, COMPILED.fullmatch)]
    print(f"  {repr(s):<28s}{results[0]:<9s}{results[1]:<8s}{results[2]}")

RECORDS = ("north-2031: 12,5 °C\n"
            "slope-2031-07: -3.5 °C\n"
            "summary-2031: 88 %\n"
            "broken record\n"
            "south-2032: 41 °C")

print()
found = COMPILED.findall(RECORDS)
iterated = list(COMPILED.finditer(RECORDS))
print(f"text lines {len(RECORDS.splitlines())} | findall {len(found)} | "
      f"finditer {len(iterated)} | length of a findall item "
      f"{len(found[0])} | group count {COMPILED.groups}")
print("finditer spans:", [e.span() for e in iterated])

print()
none_result = COMPILED.search("broken record")
print("result of a non-matching search:", none_result, "| did it raise an exception: no")
try:
    none_result.group(0)
except AttributeError as e:
    print(f"calling group on the result: {type(e).__name__} — "
          f"a program defect in the first lesson's classification")

print()
print("substitution reuses the groups:")
print(" ", COMPILED.sub(r"\g<station>=\4\5", RECORDS).replace("\n", " | "))

print()
TEXT = "kuzey yamaç: 12,5 °C"
RAW = TEXT.encode("utf-8")
print(f"characters {len(TEXT)} | bytes {len(RAW)}")
print(f"position of 'C' in the string {re.search(r'C', TEXT).start()} | "
      f"in the byte sequence {re.search(rb'C', RAW).start()}")
```

```
line                          search   match   fullmatch
  'north-2031: 12,5 °C'       yes      yes     yes
  '  north-2031: 12,5 °C'     yes      no      no
  'north-2031: 12,5 °C extra' yes      yes     no
  'broken record'             no       no      no

text lines 5 | findall 4 | finditer 4 | length of a findall item 5 | group count 5
finditer spans: [(0, 19), (20, 42), (43, 61), (76, 93)]

result of a non-matching search: None | did it raise an exception: no
calling group on the result: AttributeError — a program defect in the first lesson's classification

substitution reuses the groups:
  north=12,5°C | slope=-3.5°C | summary=88% | broken record | south=41°C

characters 20 | bytes 22
position of 'C' in the string 19 | in the byte sequence 21
```

#### Reading the Numbers

The first table sets apart three search forms. `search` looks for the pattern
**anywhere** and finds it in three of four lines. `match` only tries **from the
start**: it fails on the line with leading whitespace. `fullmatch` requires **the
whole** line to match: it fails on the line with extra text at the end. Same pattern,
same line, three different answers — depending on which question you ask.

The second line counts multi-matching: a **5**-line text has **4** matches, and
`findall` and `finditer` give the same count. The difference is in what they carry.
`findall` gives a **5**-length tuple per match — only the groups' text; span and name
information are lost. `finditer` gives the match objects themselves, and the spans can
be read. When groups are present, `findall` is a simplification, and what it
simplifies away is exactly the information measured in the previous section.

The third block says what a non-match is: the result is **`None`** and **there is no
exception**. A regular expression does not count non-matching as an error, because
non-matching is an expected outcome. An exception is only born if `group` is called on
the `None`, and the class that comes then is `AttributeError` — a **program defect** in
the first lesson's classification, meaning something to be prevented by testing, not
handled with `except`. A search result is tested before it is used.

The substitution line shows the group's second job: captured pieces can be reused
**by number or by name** in the replacement text. A non-matching line is left as it
is.

The last two lines tie back to this course's third lesson. The same content is **20**
characters as a string, **22** bytes as a byte sequence; the position of the letter
searched for is **19** in one, **21** in the other. A regular expression counts
**characters** on a string, **bytes** on a byte sequence. The pattern and the thing it
is searched against must be the same type, and the spans it returns are in that
type's unit.

#### Summary

- A parenthesis has two jobs: grouping and capturing; `(?:` only groups, `(?P<name>`
  both captures and names — of the pattern's **7** opening parentheses, **2** do not
  capture, leaving **5** groups.
- The number is given by **the opening parenthesis's order**; in a nested pattern the
  outer group is numbered first, and `group(0)` is not a group, it is the whole match.
- The match object carries every group's text, **span**, and name if it has one; an
  optional group that did not take part returns `None`, distinct from an empty string.
- `search`, `match`, and `fullmatch` give different answers on the same line;
  `findall` returns **5**-item tuples and drops span information, `finditer` keeps the
  match object.
- Non-matching is not an exception, it is `None`; an exception is only born if `group`
  is called without testing first, and it comes as `AttributeError`.

#### Course Wrap-Up

This course measured a single claim across sixteen lessons: **in Python, syntax is a
shorthand.** Every written form calls either a specific **special method** by name or a
specific rule by name; a lesson's number is which protocol the syntax it covers calls,
and how many times.

| lesson | syntax covered | protocol called |
|---|---|---|
| Python's Execution Model | eleven syntax forms | 12 special methods, 18 calls total |
| Syntax and Indentation | indentation, block, statement, and expression | no special method: indentation tokens and the tree |
| Variables and Data Types | name binding, `x += extra` | the `__add__`/`__iadd__` split, object identity |
| Operators | `n + m`, `n += m`, `x in n` | `__add__`, `__iadd__`, `__contains__` or `__iter__` if absent |
| Working with Strings | string methods and formatting notations | `__format__`, `__str__`, `__repr__`; immutability forces a new object |
| Type Conversion | `int(n)`, `float(n)`, `str(n)`, use as an index | `__int__`, `__float__`, `__str__`, `__index__`; `TypeError` if the method is absent |
| Conditionals | `if n` | `__bool__`, or `__len__` if absent, or true if neither |
| Loops | `for x in n` | `__iter__` and `__next__`; `StopIteration` ends it |
| Function Definition | `def`, default value, argument forms | the default is evaluated once, at definition time |
| Scope Rules | name reading and assignment | the four-level local, enclosing, global, built-in lookup |
| Built-in Functions | `len`, `str`, `iter`, `bool`, `sorted`, and three more built-ins | eight built-ins, 21 calls total; for `bool`, `__len__` if `__bool__` is absent |
| Exceptions | `try`, `except`, `raise`, `finally` | exception hierarchy; catching looks at the class lineage |
| Custom Exceptions | `class ...(Exception)` | the class's place in the hierarchy decides catching behavior |
| File Operations | `open`'s text mode and binary mode | the encoding decoder and newline translation |
| Context Managers | `with n` | `__enter__`, `__exit__`, and the return value's swallowing decision |
| Regular Expressions | pattern, group, substitution | the match object and capture-group numbering |

The distance between the second and third columns is the course's entire finding. Two
notations that look alike call different protocols — `n + m` and `n += m`, for
instance. The same notation calls a different protocol depending on the object — `if
n`, for instance. And a protocol **failing** is part of the protocol too:
`StopIteration` ends a loop, `__exit__` sees the exception inside a `with` block and
decides, through its return value, whether to swallow it.

The reading that follows from this is: **what an object can do is decided by the
special methods it defines, not by its type's name.** The object measured throughout
the course descended from no built-in collection; the only reason it could enter a
`for`, be tested with `in`, or be used in a `with` block was that it **defined** the
relevant methods.

Throughout this course, protocols were always **used**. The next course, Data
Structures and Functional Tools, takes the other half of the work: **writing** the
protocol. The two are different things. Someone writing `for` is calling the iteration
protocol; someone writing their own iterator, their own generator, their own sort key
is building **the other side** of the protocol. There, lists, tuples, dictionaries, and
sets are treated not as data structures but as **protocol implementations** — and the
condition for your own object to stand in their place is defining, yourself, the
methods counted in this course.