Context Managers

Every file example in the previous lesson was opened inside a with block, and the file closed when the block ended. The temporary directory could even be deleted at the end of the measurement, because no open handle was left behind. with has been used as a habit up to now.

But with is syntax too, and this course’s rule is: every syntax form is a shorthand and calls a specific special method by name. This lesson measures that protocol. Resource lifetime and releasing a resource were built in the Operating System Concepts course; the concept was built there, what is measured here is which protocol Python realizes it with: with calls __enter__ on entry, __exit__ on exit — and when an exception occurs inside the block, __exit__ is still called.

The Protocol Is Just Two Methods

There is exactly one condition for an object to be usable in a with statement: defining __enter__ and __exit__. Its type’s name, which class it descends from, what it does — none of that matters. No protocol, no syntax; if the protocol is there, the object enters the with.

__enter__ takes no arguments, and the value it returns is the name bound with as. It does not have to return the resource itself; it can return a different object.

__exit__ takes three arguments: the raised exception’s type, object, and traceback. If the block ended without an exception, all three are empty. And __exit__ returns a value; this return value decides whether the exception gets swallowed. A return counted as true swallows the exception, one counted as false lets it escape.

The shared setup’s Tracker object defines these two methods, and its __exit__ returns False — meaning it does not swallow. The measurement adds a sibling to it: Swallower, which only changes the return value.

The Measurement’s Setup

Three syntax forms are run: with with no exception, with with an exception, and with with an exception but on an object that swallows. For each form, how many times which special method got called is counted. The measurement log writes the exception as an event too: if a form ends with an exception, the class name drops into the log with a ! prefix.

• EF31 — Tracker’s __enter__ and __exit__ methods are taken unchanged from the shared setup; __exit__ still returns False.
• EF32 — Swallower only changes the __exit__ return and calls it without disturbing the base class’s log; the log’s layout is the shared setup’s layout.
• EF33 — Witness also does not disturb the base class’s log; it additionally writes the triple __exit__ sees into a separate list. Tracker’s behavior is not touched.
• EF34 — The call count is the number of items in the log; !ValueError is an item too, because what is measured is the event sequence a form produces, not only method calls.
• EF35 — The return-value test is read by calling __exit__ directly; whether it swallows is separately tested with a real with block. The two paths confirm each other.

"""Context manager protocol: which method with calls on entry and exit."""

LOG = []
SEEN = []


def record(name):
    LOG.append(name)


class Tracker:
    """The core of the shared setup; the context part is used here."""

    def __init__(self, items=(1, 2, 3)):
        self.items = list(items)

    def __enter__(self):
        record("__enter__")
        return self

    def __exit__(self, kind, value, tb):
        record("__exit__")
        return False          # does not swallow the exception


class Swallower(Tracker):
    """__exit__ returns True: the exception does not escape."""

    def __exit__(self, kind, value, tb):
        super().__exit__(kind, value, tb)
        return True


class Witness(Tracker):
    """Separately logs the triple __exit__ sees; does not disturb Tracker's log."""

    def __init__(self, name, swallows=False, items=(1, 2, 3)):
        super().__init__(items)
        self.name = name
        self.swallows = swallows

    def __exit__(self, kind, value, tb):
        SEEN.append((self.name, kind.__name__ if kind else "-",
                        str(value) if value else "-", tb is not None))
        super().__exit__(kind, value, tb)
        return self.swallows


def measure(function):
    """Runs a syntax form and returns the protocols it called."""
    LOG.clear()
    try:
        function()
    except Exception as e:
        LOG.append(f"!{type(e).__name__}")
    return list(LOG)


def context():
    with Tracker():
        pass


def context_with_exception():
    with Tracker():
        raise ValueError("example")


def context_swallowing():
    with Swallower():
        raise ValueError("example")


FORMS = (
    ("with n", context),
    ("with n + exception", context_with_exception),
    ("with swallower + exception", context_swallowing),
)

print(f"{'syntax':<27s} {'calls':>6s}  protocol sequence")
for name, function in FORMS:
    c = measure(function)
    print(f"{name:<27s} {len(c):6d}  {' '.join(c)}")

print()
print(f"{'class':<12s}{'__exit__ return':<18s}exception escaped")
for name, cls in (("Tracker", Tracker), ("Swallower", Swallower)):
    n = cls()
    return_value = n.__exit__(ValueError, ValueError("example"), None)
    LOG.clear()
    try:
        with cls():
            raise ValueError("example")
    except ValueError:
        outcome = "yes"
    else:
        outcome = "no"
    print(f"  {name:<10s}{str(return_value):<18s}{outcome}")

print()
SEEN.clear()
with Witness("no exception"):
    pass
try:
    with Witness("exception"):
        raise ValueError("measurement out of range")
except ValueError:
    pass
print(f"{'witness':<16s}{'type it saw':<14s}{'value it saw':<26s}trace object present")
for name, kind, value, tb in SEEN:
    print(f"  {name:<14s}{kind:<14s}{value:<26s}{'yes' if tb else 'no'}")

syntax                       calls  protocol sequence
with n                           2  __enter__ __exit__
with n + exception               3  __enter__ __exit__ !ValueError
with swallower + exception       2  __enter__ __exit__

class       __exit__ return   exception escaped
  Tracker   False             yes
  Swallower True              no

witness         type it saw   value it saw              trace object present
  no exception  -             -                         no
  exception     ValueError    measurement out of range  yes

What the Three Lines Say

The first two lines are the shared setup’s with lines. with n produces two calls: __enter__ and __exit__. with n + exception produces three events: __enter__, __exit__, and !ValueError. The difference needs to be read correctly — the extra one is not a method call. __enter__ ran once, __exit__ ran once, in both forms. What is extra is that the form ends with an exception.

This gives the lesson’s central sentence: when an exception occurs, __exit__ still runs. The block was cut short, the remaining lines did not run, control jumped out of the with statement — and __exit__ still ran. This is what makes the resource close, and this is the reason to write with at all.

The third line tries the same exception with Swallower, and the call count drops to 2. The sequence is still __enter__ __exit__ — the method calls did not change. The only thing that changed is that !ValueError did not drop into the log: the exception did not escape. The middle table confirms this. Tracker’s __exit__ returns False and the exception escaped; Swallower’s returns True and it did not. A single-line return value brought a three-event form down to two.

The last table gives what __exit__ sees. In the exception-free block, all three arguments are seen empty: type -, value -, trace no. In the exception-carrying block, the type is ValueError, the value is the exception’s message, and the trace object is present. __exit__ does not merely swallow or not swallow the exception; it knows which exception it is and can decide by looking at it. A context manager that swallows one specific class and lets the others through is written with this triple.

Nested Blocks and try/finally

The second measurement answers two questions at once. First: when two context managers are nested, which one sees the exception, and if the inner one swallows it, what does the outer one see? Second: can the difference between with and try/finally be measured?

• EF36 — Four combinations are tried in the nested measurement: the outer and inner managers each separately swallow or do not swallow. Nothing else changes.
• EF37 — The “saw” column is read from the type argument passed to __exit__; if it was passed empty, - is written.
• EF38 — The exit-form table tries four paths: normal ending, return, break, and an exception. For return and break, the block is placed inside a loop and a function, because both only make sense there.
• EF39 — The with arm and the try/finally arm try the same four paths; the only difference between them is that one has a context manager, the other a finally block.

"""Nested with, exit forms, and comparison with try/finally."""

LOG = []
SEEN = []


def record(name):
    LOG.append(name)


class Tracker:
    def __init__(self, items=(1, 2, 3)):
        self.items = list(items)

    def __enter__(self):
        record("__enter__")
        return self

    def __exit__(self, kind, value, tb):
        record("__exit__")
        return False


class Witness(Tracker):
    """Logs by name and takes whether it swallows at construction."""

    def __init__(self, name, swallows=False, items=(1, 2, 3)):
        super().__init__(items)
        self.name = name
        self.swallows = swallows

    def __exit__(self, kind, value, tb):
        SEEN.append((self.name, kind.__name__ if kind else "-"))
        super().__exit__(kind, value, tb)
        return self.swallows


print(f"{'outer':<10s}{'inner':<10s}{'calls':>6s}  {'sequence':<40s}"
      f"{'outer saw':<12s}{'inner saw':<12s}exception")
for outer_swallows in (False, True):
    for inner_swallows in (False, True):
        LOG.clear()
        SEEN.clear()
        try:
            with Witness("outer", outer_swallows), Witness("inner", inner_swallows):
                raise ValueError("measurement out of range")
        except ValueError:
            outcome = "escaped"
        else:
            outcome = "swallowed"
        seen = dict(SEEN)
        print(f"  {('swallows' if outer_swallows else 'passes'):<10s}"
              f"{('swallows' if inner_swallows else 'passes'):<9s}{len(LOG):6d}  "
              f"{' '.join(LOG):<40s}{seen['outer']:<12s}"
              f"{seen['inner']:<12s}{outcome}")

print()
FINALLY = []


def with_form(form):
    LOG.clear()
    SEEN.clear()
    for _ in range(1):
        with Witness("single"):
            if form == "return":
                return
            if form == "break":
                break
            if form == "exception":
                raise ValueError("measurement out of range")


def finally_form(form):
    FINALLY.clear()
    for _ in range(1):
        try:
            if form == "return":
                return
            if form == "break":
                break
            if form == "exception":
                raise ValueError("measurement out of range")
        finally:
            FINALLY.append("finally")


print(f"{'exit form':<16s}{'__exit__ called':<19s}{'finally ran':<17s}"
      f"__exit__ saw the exception")
for form in ("normal", "return", "break", "exception"):
    try:
        with_form(form)
    except ValueError:
        pass
    called = "yes" if "__exit__" in LOG else "no"
    saw = "yes" if SEEN and SEEN[0][1] != "-" else "no"
    try:
        finally_form(form)
    except ValueError:
        pass
    ran = "yes" if FINALLY else "no"
    print(f"  {form:<14s}{called:<19s}{ran:<17s}{saw}")

outer     inner      calls  sequence                                outer saw   inner saw   exception
  passes    passes        4  __enter__ __enter__ __exit__ __exit__   ValueError  ValueError  escaped
  passes    swallows      4  __enter__ __enter__ __exit__ __exit__   -           ValueError  swallowed
  swallows  passes        4  __enter__ __enter__ __exit__ __exit__   ValueError  ValueError  swallowed
  swallows  swallows      4  __enter__ __enter__ __exit__ __exit__   -           ValueError  swallowed

exit form       __exit__ called    finally ran      __exit__ saw the exception
  normal        yes                yes              no
  return        yes                yes              no
  break         yes                yes              no
  exception     yes                yes              yes

Reading the Numbers

All four rows of the nested table give a call count of 4 and the same sequence: __enter__ __enter__ __exit__ __exit__. Entry goes outer to inner, exit inner to outer. The swallowing decision cancels no call — the four methods run in all four combinations.

What changes are the “saw” columns. The inner manager sees ValueError in all four rows; the exception forms closest to it and reaches it first. The outer one sees it only when the inner does not swallow, and gets - when it does. The second and fourth rows show this: once the inner one has swallowed it, there is no exception left for the outer one, and __exit__ is called with an empty triple.

The third row shows the reverse: the inner one did not swallow, the outer one saw it and swallowed it. The exception did not escape. Swallowing authority travels outward in order, and the first one to swallow cuts the chain.

The second table sets apart what with and try/finally share and where they differ. What they share: in all four exit forms, both __exit__ gets called and finally runs — however the block ends, the cleanup code runs. This is why with is not an “error-catching” tool; it is an exit tool, active on all four paths.

The difference is in the last column. __exit__ sees the exception only in the exception form; in the other three it is called with an empty triple. A finally block, by contrast, is never given a triple in any form — while it runs, it does not know which exception is flying and cannot decide by looking at it. The only way to swallow an exception inside finally is to exit from there with a new statement, and that swallows every exception regardless of what happened in the block. __exit__, by contrast, can look at the type and choose.

The choice follows from this: if cleanup is needed once, in one place, finally is enough. If the same cleanup repeats in more than one place, or the decision depends on the exception’s type, the protocol is written.

The Bound Object and Cleanup’s Own Error

Two details remain, both sitting at the protocol’s edge. The first is the name bound with as: this name is not the manager itself, it is what __enter__ returned. The second is __exit__ breaking on its own — cleanup can fail too.

• EF40 — Wrapper only changes __enter__’s return; it takes its log from the base class. Broken only adds an exception during cleanup.
• EF41 — The bound object’s identity is tested with is; no identity number is printed.
• EF42 — Chain length is counted with the same method as the first lesson’s measurement: unsuppressed context and cause links are followed.

"""What __enter__ returns and an exception born inside __exit__."""

LOG = []


class Tracker:
    def __init__(self, items=(1, 2, 3)):
        self.items = list(items)

    def __enter__(self):
        LOG.append("__enter__")
        return self

    def __exit__(self, kind, value, tb):
        LOG.append("__exit__")
        return False


class Wrapper(Tracker):
    """__enter__ returns not itself but another object for the block to use."""

    def __enter__(self):
        super().__enter__()
        return self.items


class Broken(Tracker):
    """Produces an exception of its own during cleanup."""

    def __exit__(self, kind, value, tb):
        super().__exit__(kind, value, tb)
        raise RuntimeError("could not close the resource")


n = Tracker()
with n as bound:
    pass
print("Tracker: is the object bound with as the manager itself:", bound is n)

k = Wrapper()
with k as bound:
    pass
print("Wrapper: is the object bound with as the manager itself:", bound is k)
print("Wrapper: is the object bound with as what __enter__ returned:",
      bound is k.items)

print()
print(f"{'state':<36s}{'escapes as':<15s}{'context':<14s}chain")
for name, cls, raises in (("block clean, __exit__ clean", Tracker, False),
                          ("block raises, __exit__ clean", Tracker, True),
                          ("block clean, __exit__ breaks", Broken, False),
                          ("block raises, __exit__ breaks", Broken, True)):
    try:
        with cls():
            if raises:
                raise ValueError("measurement out of range")
    except BaseException as exc:
        escapes = type(exc).__name__
        context = type(exc.__context__).__name__ if exc.__context__ else "-"
        length, cur = 0, exc
        while cur is not None:
            length += 1
            cur = cur.__cause__ or (None if cur.__suppress_context__
                                        else cur.__context__)
    else:
        escapes = context = "-"
        length = 0
    print(f"  {name:<34s}{escapes:<15s}{context:<14s}{length}")

Tracker: is the object bound with as the manager itself: True
Wrapper: is the object bound with as the manager itself: False
Wrapper: is the object bound with as what __enter__ returned: True

state                               escapes as     context       chain
  block clean, __exit__ clean       -              -             0
  block raises, __exit__ clean      ValueError     -             1
  block clean, __exit__ breaks      RuntimeError   -             1
  block raises, __exit__ breaks     RuntimeError   ValueError    2

The first three lines settle the as binding. Because Tracker returns itself, the bound name is the manager. Wrapper returns a different object, and the bound name is not the manager anymore. This explains why the thing bound with as when you open a file with with is the file object: __enter__ returns it.

The lower table counts cleanup’s own error, and it ties into the first lesson’s chain measurement. When both the block and __exit__ are clean, nothing escapes, chain 0. When only the block breaks, ValueError escapes, chain 1. When only __exit__ breaks, RuntimeError escapes, chain also 1.

The last row is the dangerous one. The block raised ValueError, __exit__ was called, and it broke too. What escapes is RuntimeError — that is, cleanup’s own error takes the domain error’s place. The real error is not lost: it sits in the __context__ field, and the chain is 2. But an outside except ValueError clause no longer matches it, because the escaping class changed. This is why the work inside __exit__ must not break; if it can, it has to be handled inside itself.

Summary

• with is a shorthand: it calls __enter__ on entry, __exit__ on exit, and an object not defining the protocol cannot enter a with statement.
• An exception-free block produces 2 events, an exception-carrying one 3; the extra is not a method call, it is that the form ends with an exception — __exit__ is called once in both.
• __exit__ takes three arguments and sees the type, object, and trace in an exception-carrying block; when the return value counts as true, the exception is swallowed and the event count drops from 3 to 2.
• In two nested managers, the call count is 4 across all four combinations; the inner one always sees the exception, the outer one only when the inner does not swallow it.
• __exit__ and finally both run in all four exit forms; where they differ is that __exit__ can see the exception’s type and choose, while finally cannot.
• The name bound with as is not the manager, it is __enter__‘s return value; if __exit__ breaks on its own, **cleanup’s error** escapes, the real error stays in the context, and the chain becomes 2.

Next Step

The last syntax form this course measures is now behind us. Up to now every measurement was a statement or expression calling the language’s own protocols: for, if, +, with. The next lesson’s subject is not the language’s syntax but a small, separate language the standard library offers: regular expressions. There, a pattern is written in a string, interpreted by a separate rule set, and a match comes back as an object. What gets measured is what that object carries and in what order the parentheses in the pattern get numbered.