Special Methods

The previous three lessons called classes through methods we wrote ourselves: add, format, source. But the Python Fundamentals course measured that syntax is a shorthand — n + m is a method call, x in n is a method call, for is a protocol. Those lists are not repeated here. What was measured there was the calling side, and the answering side was always ready: a list, a string, a dict, a generator. The objects joining that protocol were given by the language.

This lesson measures the case where we write the answering side ourselves. There are two questions. First: how many of these syntax forms can a class join without writing a single special method, and when it does join, who gives the answer? Second: does every method we write open a syntax form — and does it ever close one too?

What an Unwritten Class Can Do

Eleven syntax forms are chosen; each looks for a specific special method name. Two classes carry the same data. Bare writes no special method at all. Written writes all eleven of the eleven methods. The measurement asks two things separately: did the syntax work, and which class provides that name. The two questions have to be asked separately, because their answers do not line up on every row — a syntax can fail even with the name found, and can work even without it being found.

• CD42 — Two classes carry the same three items, and their __init__ bodies are identical; the only difference between them is which special methods are written. Written does not derive from Bare.
• CD43 — Each form is run on two fresh instances; the measurement looks not at the result’s value but at whether the call was answered. An unanswered call falls with TypeError.
• CD44 — The “answering” column gives the first class carrying the name during resolution; if no class carries it, None is printed.
• CD45 — No syntax’s result is printed. Since Bare writes no representation method, the default representation carries environment-dependent data; the measurement therefore prints only the status and the answering class.

"""We write the side that joins the syntax: an unwritten class and a written class."""


def answering_class(cls, name):
    for s in cls.__mro__:
        if name in s.__dict__:
            return s.__name__
    return None


class Bare:
    """Writes no special method at all; only carries its items."""

    def __init__(self, items=(1, 2, 3)):
        self.items = list(items)


class Written:
    """Carries the same data, writes one method for every syntax it joins."""

    def __init__(self, items=(1, 2, 3)):
        self.items = list(items)

    def __repr__(self):
        return f"Written({self.items})"

    def __str__(self):
        return "-".join(str(x) for x in self.items)

    def __len__(self):
        return len(self.items)

    def __eq__(self, o):
        return isinstance(o, Written) and self.items == o.items

    def __lt__(self, o):
        return len(self.items) < len(o.items)

    def __bool__(self):
        return bool(self.items)

    def __contains__(self, x):
        return x in self.items

    def __iter__(self):
        return iter(self.items)

    def __enter__(self):
        return self

    def __exit__(self, *k):
        return False

    def __getitem__(self, i):
        return self.items[i]

    def __hash__(self):
        return hash(tuple(self.items))


def use_context(n):
    with n:
        return True


FORMS = (
    ("repr(n)", "__repr__", lambda n, m: repr(n)),
    ("str(n)", "__str__", lambda n, m: str(n)),
    ("len(n)", "__len__", lambda n, m: len(n)),
    ("n == m", "__eq__", lambda n, m: n == m),
    ("n < m", "__lt__", lambda n, m: n < m),
    ("bool(n)", "__bool__", lambda n, m: bool(n)),
    ("2 in n", "__contains__", lambda n, m: 2 in n),
    ("for x in n", "__iter__", lambda n, m: [x for x in n]),
    ("with n", "__enter__", lambda n, m: use_context(n)),
    ("n[0]", "__getitem__", lambda n, m: n[0]),
    ("hash(n)", "__hash__", lambda n, m: hash(n)),
)


def measure(cls, action):
    try:
        action(cls(), cls())
    except TypeError:
        return "fell"
    return "worked"


print(f"{'syntax':<12s} {'special method':<15s} {'Bare':>7s} "
      f"{'answering':>11s} {'Written':>9s} {'answering':>11s}")
for name, method, action in FORMS:
    print(f"{name:<12s} {method:<15s} {measure(Bare, action):>7s} "
          f"{str(answering_class(Bare, method)):>11s} {measure(Written, action):>9s} "
          f"{str(answering_class(Written, method)):>11s}")

for cls in (Bare, Written):
    worked = sum(1 for _, _, f in FORMS if measure(cls, f) == "worked")
    own = sum(1 for _, m, _ in FORMS if answering_class(cls, m) == cls.__name__)
    print(f"\n{cls.__name__}: worked {worked}/{len(FORMS)}, "
          f"answered by itself {own}/{len(FORMS)}")

syntax       special method     Bare   answering   Written   answering
repr(n)      __repr__         worked      object    worked     Written
str(n)       __str__          worked      object    worked     Written
len(n)       __len__            fell        None    worked     Written
n == m       __eq__           worked      object    worked     Written
n < m        __lt__             fell      object    worked     Written
bool(n)      __bool__         worked        None    worked     Written
2 in n       __contains__       fell        None    worked     Written
for x in n   __iter__           fell        None    worked     Written
with n       __enter__          fell        None    worked     Written
n[0]         __getitem__        fell        None    worked     Written
hash(n)      __hash__         worked      object    worked     Written

Bare: worked 5/11, answered by itself 0/11

Written: worked 11/11, answered by itself 11/11

Bare joins 5 of eleven forms, and the number of forms it answers itself is 0. Not writing a method does not close the syntax; it hands it off to a silent default. In four of the five working rows the answer comes from the common base of every class; on the bool(n) row no class gives it at all — there, instead of falling when it cannot find a method, the syntax switches to the rule “every object is true.”

What the defaults say matters, because all of them rest on identity. The equality the common base gives does not count two separate objects equal; the hash it gives is also produced from identity. If a Bare object needs to be compared by its items, or used as a key by its items, these defaults give the wrong answer — they do not raise an error, they give a wrong answer. This is the most expensive form of a bug: two Bare objects carrying the same content, put in a set, count as two separate items, and the program keeps running without a single warning.

The n < m row is a separate reading. The answering column is not empty, it shows the common base; the name was found. Even so, the syntax fell. The common base defines this name, but its body says “this job is unknown to me,” and the comparison goes unanswered. A name being found does not mean an answer was given. The correct way to look for a capability is not whether the name exists, it is whether the call gets answered.

Written joins all eleven of eleven forms and answers all eleven itself. A class “resembling the language’s objects” is not a separate capability; it is the methods written. Joining comes not from the type’s name or its base, but from the names placed in the class’s dict.

One detail of the number does not show in the table. Written writes twelve bodies for eleven syntax forms: the context syntax is a single line, but its entry and exit are answered by two separate methods, and the table only lists the entry. The mapping between syntax and method is not one to one; a syntax can ask for more than one body, or none at all, as seen on the bool(n) row.

Some of the written methods also open syntax forms adjacent to themselves. A class writing equality does not additionally write inequality; a class writing less-than also answers a greater-than comparison between two objects of the same type, because the language flips the operation and asks the other operand. These were measured from the calling side in the Python Fundamentals course; here the same drops appear from the writing side — writing one body answers every spelling that falls to that body at once.

Writing a Method Can Close a Syntax Form

Written methods do not just open a door. There is a contract the language upholds between equality and hash value: two objects counted equal must also have equal hash values. When a class defines equality itself, this contract can no longer be guaranteed automatically, and the language does not leave the decision to the class — it removes the hashing capability.

• CD46 — Three classes carry the same single field; they differ only in which method they write. The third class derives from the second and only adds the hashing method.
• CD47 — The hash test does not print the value itself, only whether the call was answered; a hash value can be environment-dependent.
• CD48 — The last two columns separately show whether the __hash__ name is found in the class’s own dict, and whether its value is empty.

class Plain:
    """Writes neither equality nor hash."""

    def __init__(self, value=1):
        self.value = value


class Equatable:
    """Writes only equality."""

    def __init__(self, value=1):
        self.value = value

    def __eq__(self, o):
        return isinstance(o, Equatable) and self.value == o.value


class EquatableHashable(Equatable):
    """Inherits equality, writes hashing itself."""

    def __hash__(self):
        return hash(self.value)


def attempt(action):
    try:
        return str(action())
    except TypeError as e:
        return f"!{type(e).__name__}"


print(f"{'class':<19s} {'a == b':>8s} {'hash(a)':>16s} {'key use':>16s} "
      f"{'__hash__ written':>17s} {'value empty':>12s}")
for cls in (Plain, Equatable, EquatableHashable):
    a, b = cls(), cls()
    print(f"{cls.__name__:<19s} {attempt(lambda: a == b):>8s} "
          f"{attempt(lambda: hash(a) is not None):>16s} "
          f"{attempt(lambda: len({a: 1})):>16s} "
          f"{str('__hash__' in cls.__dict__):>17s} "
          f"{str(cls.__hash__ is None):>12s}")

class                 a == b          hash(a)          key use  __hash__ written  value empty
Plain                  False             True                1             False        False
Equatable               True       !TypeError       !TypeError              True         True
EquatableHashable       True             True                1              True        False

The first row shows the default state: two separate objects are not equal, but a hash value exists and the object can be used as a key in a container. In the second row, a single method was written, and two syntax forms closed at once — both hash(a) and use as a key fell. The method written was __eq__; both of the closing syntax forms were tied to __hash__.

The last two columns say how. The name __hash__ is written into Equatable’s own dict, and its value is empty. The language silently adds an entry to a class that writes equality and leaves that entry empty; lookup can no longer reach the common base, it hits an empty value at the first step. This is not a side effect, it is a deliberate decision: an object whose equality has changed using its old hash value would turn it into a lost key inside a container.

The third row shows the fix. EquatableHashable inherits equality and writes hashing itself; the entry is no longer empty, and both syntax forms reopen. The rule is: a class writing equality also writes hashing — or, by not writing it, accepts that the object cannot be a key. When hashing is written, it has to use the same fields used in equality, or the contract breaks again.

The second half of the contract is also the writing side’s responsibility: a hash value must not change over the object’s lifetime. A class producing equality from a mutable field makes the object unfindable in its own container if that field changes while it is a key. This is why classes meant to be hashable are expected to produce equality from unchanging fields. It is also why Written’s hash body in the first measurement turns its items into a tuple: a mutable container cannot be hashed on its own.

Two Representations

The remaining distinction is between two representation methods. Both produce a string from an object, but their targets differ: one introduces the object for the interpreter, the other presents it to the reader. Which one gets called is decided by where it is used.

• CD49 — Two classes return the same two strings; they differ only in that one does not write the text method.
• CD50 — Six usage forms are tried, each run on a fresh instance; the measurement looks at which string comes back.
• CD51 — The last lines count how many of the six forms go to the text method; printing inside a container is counted too.

class ReprOnly:
    """Writes only __repr__."""

    def __repr__(self):
        return "repr"


class BothReprs:
    """Writes both."""

    def __repr__(self):
        return "repr"

    def __str__(self):
        return "text"


USAGE = (
    ("str(n)", lambda n: str(n)),
    ("repr(n)", lambda n: repr(n)),
    ("n in f-string", lambda n: f"{n}"),
    ("n!r in f-string", lambda n: f"{n!r}"),
    ("printing [n]", lambda n: str([n])),
    ("format()", lambda n: "{}".format(n)),
)

print(f"{'form':<17s} {'only __repr__ written':>22s} {'both written':>14s}")
for name, action in USAGE:
    print(f"{name:<17s} {action(ReprOnly()):>22s} {action(BothReprs()):>14s}")

for cls in (ReprOnly, BothReprs):
    text = sum(1 for _, f in USAGE if "text" in f(cls()))
    print(f"\n{cls.__name__}: form giving the __str__ result {text}/{len(USAGE)}")

form               only __repr__ written   both written
str(n)                              repr           text
repr(n)                             repr           repr
n in f-string                       repr           text
n!r in f-string                     repr           repr
printing [n]                      [repr]         [repr]
format()                            repr           text

ReprOnly: form giving the __str__ result 0/6

BothReprs: form giving the __str__ result 3/6

In the left column, all six forms give the same result, and the forms reaching the text method are 0. When the text method is not written, the request does not go unanswered, it falls back to the representation method. The fall runs in one direction only: when the representation method is not written, there is no falling back to the text method. If only one of the two is going to be written, it should be the representation.

In the right column, 3 of six forms go to the text method. The split is clear: conversion to a string and formatting present to the reader; the explicit call and the conversion marker introduce to the interpreter. The fifth row is the most surprising — while an object sits inside a container, it is always printed with the representation method, even if the container itself has been converted to text from outside. When a container produces its own text, it asks every item inside it for its representation, because what the interpreter will see is not the item’s presentation, it is its identity.

A writing rule follows from this: the representation method should carry enough information to rebuild the object; the text method should be readable. Written‘s two bodies in the first measurement follow exactly this split: the representation body gives the class’s name together with its items, the text body lays out only the items in a readable form.

This distinction’s counterpart in debugging is a directly measurable convenience. When a hundred objects sitting in a container are dumped, what is shown is that class’s representation body; if representation is not written, the dump says nothing across a hundred lines. A single written body earns readability across every container the object appears in.

Summary

• A class writing no special method joins 5 of eleven syntax forms and answers 0 of them itself; not writing does not close the syntax, it hands it off to a silent default, and every default rests on identity.
• A name being found does not mean an answer is given: n < m finds a name on the common base, but the syntax still falls.
• A class writing all eleven methods joins all eleven syntax forms and answers all of them itself; joining comes from the methods written, not the type’s name.
• Writing one method can close a syntax form: the class writing __eq__ gets an empty __hash__ name added to its dict, and both hashing and key use fall together. A class writing equality also writes hashing.
• When the text method is not written, the request falls back to the representation method, never the reverse; in a class writing both, 3 of six forms go to the text method, and printing inside a container always uses the representation.

Next Step

In this lesson, every written method was triggered by a call: writing len(n) ran a body, writing n[0] ran a body. Reading an attribute, on the other hand, ran no body at all — n.items was a dict lookup and nothing stepped in between. This is exactly the gap the third lesson left open: a write that bypasses the body protecting the invariant stays unsupervised. The next lesson measures catching access itself by putting a body in the way: for the same number of reads and writes, how many bodies does a plain field run, how many does an intercepting definition run, and where does the intervening code hold its data before three instances start sharing a single value?