Asynchronous Library Compatibility

The previous lesson’s table had three rows, and the third was this: a coroutine that never gives up control gives 0 yields, 80 ticks, 0 overlapping steps. It was written with async def, a real coroutine, and its result could not be told apart from the single-thread regime.

That row looks like a writing mistake, but in practice it never comes from that. The person writing the body rarely forgets to write await; what forgets is the function it calls. A call that does not know how to give up control also stops the coroutine calling it from giving up control. What this lesson measures is that call’s bill: how far overlapping steps drop when one call in the flow does not yield, what is left when no call yields, and how many steps it costs to bring a call that cannot yield back into the flow.

Blocking Code Is Code That Produces No Yield Point

Blocking code finishes its work without returning control to the event loop. The name can mislead: what gets blocked is not the calling body, it is the loop itself. The body was going to wait for the result anyway; the real loss is that while it waits, no other task can advance.

Whether a call is blocking comes not from what it does, but from how it is written. The same work — reading a file, waiting on a reply — can be written in a form that produces a yield point, or one that does not. In an asynchronous flow, the difference between the two is exactly the difference measured in the previous lesson’s table.

This gives rise to this lesson’s narrow question. Not every call in a program changes at once; some yield, some do not. What comes out of the mix?

One Call Site, Eight Tasks

The measurement separates two axes, because mixing them makes the result unreadable.

The first axis is call count: how many I/O calls in each task’s flow block. A task’s flow has ten steps, and part of them are I/O steps; what is measured is how many of these steps pass through a call that does not yield. The point to note here: in the code, a single call site can be the blocking one, but eight tasks pass through that call site. A one-line defect gets written into eight flows at once.

The second axis is task count: how many tasks use blocking calls from start to finish. When one part of a program is written asynchronously while another part stays in the old form, this axis measures it.

The measurement’s assumptions:

• CM48 — The task setup is the same as the previous lessons: eight tasks, ten steps per task, eighty steps total, I/O-bound load, single slot.
• CM49 — The coroutines and the loop that resumes them are the same as the previous lesson; the only thing that changes is at which step the body yields.
• CM50 — A compatible call is one that gives up control at an I/O step. A blocking call takes the same step without giving up control. Both do the same work and spend the same step.
• CM51 — CPU steps never yield in any regime; the only variable measured is the form of the I/O calls.
• CM52 — On the first axis, each task’s first k I/O calls block; which calls get chosen is written into the setup and does not change between runs.
• CM53 — On the second axis, the first k tasks block start to finish, the rest are compatible.
• CM54 — Offloading to an executor pool is an added CPU step in the model: the work runs in the pool, the loop regains its yield point, and one more step gets counted in exchange.
• CM55 — Two placements of the offload cost are measured: once per task and once per call. Both offload the same work; only where the cost falls changes.
• CM56 — The pool’s width is unbounded in this measurement; the effect of pool width is the next lesson’s subject.
• CM57 — Duration is never measured; the counted unit is steps, ticks, overlapping steps, and yields.
• CM58 — The measurement is a single run, and the seed is fixed.

The Measurement

"""Blocking code: how much overlap one call that never yields costs."""

SEED = 20260817
CPU, IO = "cpu", "io"


def make_rng(seed):
    state = seed % 2147483646 + 1

    def draw(n):
        nonlocal state
        state = (state * 48271) % 2147483647
        return state % n
    return draw


def tasks(count=8, steps=10, io_share=7, seed=SEED):
    """Each task is a step sequence; io_share/10 fraction are I/O steps."""
    draw, result = make_rng(seed), []
    for i in range(count):
        result.append([IO if draw(10) < io_share else CPU for _ in range(steps)])
    return result


class Suspend:
    """The point where a coroutine hands control back to the event loop."""

    def __init__(self, kind):
        self.kind = kind

    def __await__(self):
        return (yield self.kind)


def compatible_factory(blocking_calls=0):
    """Each task's first 'blocking_calls' I/O calls do not yield."""
    async def task(steps, log):
        remaining = blocking_calls
        for s in steps:
            if s == IO and remaining > 0:
                remaining -= 1              # blocking call: control not given up
            elif s == IO:
                await Suspend(s)            # compatible call: control given up
            log.append(s)
    return task


def mixed_factory(blocking_tasks=0):
    """The first 'blocking_tasks' tasks never yield; the rest are compatible."""
    compatible = compatible_factory(0)

    def factory(steps, log):
        factory.turn += 1
        if factory.turn <= blocking_tasks:
            return compatible_factory(len(steps))(steps, log)
        return compatible(steps, log)
    factory.turn = 0
    return factory


def loop(factory, jobs, cpu_slots=1):
    """Hand-written deterministic event loop. Returns: tick, overlap, yields."""
    log = [[] for _ in jobs]
    coro = [factory(j, log[i]) for i, j in enumerate(jobs)]
    pending = [None] * len(coro)
    alive = [True] * len(coro)
    debt = [0] * len(coro)
    yields = 0

    def resume(i, prepaid=0):
        nonlocal yields
        before = len(log[i])
        try:
            pending[i] = coro[i].send(None)
            yields += 1
        except StopIteration:
            pending[i] = None
            alive[i] = False
        debt[i] += max(0, len(log[i]) - before - prepaid)

    for i in range(len(coro)):
        resume(i)
    tick = overlap = 0
    while any(alive) or any(debt):
        busy = [i for i in range(len(coro)) if debt[i] > 0]
        if busy:
            debt[busy[0]] -= 1
            tick += 1
            continue
        slots_left, advanced = cpu_slots, 0
        for i in range(len(coro)):
            if not alive[i] or pending[i] is None:
                continue
            if pending[i] == CPU:
                if slots_left <= 0:
                    continue
                slots_left -= 1
            resume(i, 1)
            advanced += 1
        if advanced == 0:
            break
        tick += 1
        overlap += max(0, advanced - 1)
    return tick, overlap, yields


j = tasks(io_share=7)
print("eight tasks, eighty steps, I/O-bound load, single slot")
print()
print("A - how many I/O calls block in each task")
print(f"{'blocking calls':>17s} {'yields':>7s} {'tick':>5s} {'overlap':>8s}")
for k in range(9):
    tick, overlap, yields = loop(compatible_factory(k), j)
    print(f"{k:17d} {yields:7d} {tick:5d} {overlap:8d}")

print()
print("B - how many tasks block start to finish")
print(f"{'blocking tasks':>17s} {'yields':>7s} {'tick':>5s} {'overlap':>8s}")
for k in range(9):
    tick, overlap, yields = loop(mixed_factory(k), j)
    print(f"{k:17d} {yields:7d} {tick:5d} {overlap:8d}")

print()
print("C - offloading the blocking call to an executor pool")
offload_per_task = [[CPU] + list(t) for t in j]
offload_per_call = [[s for a in t for s in ([CPU, IO] if a == IO else [a])]
                     for t in j]
print(f"{'regime':<28s} {'steps':>5s} {'yields':>7s} {'tick':>5s} {'overlap':>8s}")
for label, seq, factory in (
        ("all block", j, compatible_factory(10)),
        ("all compatible", j, compatible_factory(0)),
        ("offload once per task", offload_per_task, compatible_factory(0)),
        ("offload per call", offload_per_call, compatible_factory(0))):
    tick, overlap, yields = loop(factory, seq)
    print(f"{label:<28s} {sum(len(t) for t in seq):5d} {yields:7d} "
          f"{tick:5d} {overlap:8d}")

eight tasks, eighty steps, I/O-bound load, single slot

A - how many I/O calls block in each task
   blocking calls  yields  tick  overlap
                0      54    36       44
                1      46    43       37
                2      38    50       30
                3      30    57       23
                4      22    64       16
                5      14    71        9
                6       8    76        4
                7       5    78        2
                8       2    80        0

B - how many tasks block start to finish
   blocking tasks  yields  tick  overlap
                0      54    36       44
                1      46    44       36
                2      40    50       30
                3      30    58       22
                4      22    64       16
                5      17    69       11
                6      11    75        5
                7       6    80        0
                8       0    80        0

C - offloading the blocking call to an executor pool
regime                       steps  yields  tick  overlap
all block                       80       0    80        0
all compatible                  80      54    36       44
offload once per task           88      54    44       44
offload per call               134      54    90       44

The Cost of a Single Call

Table A’s first two rows give this lesson’s shortest result. With every call compatible: 36 ticks / 44 overlapping steps. With a single I/O call blocking in every task: 43 ticks / 37 overlapping steps.

One call costs seven ticks and seven overlapping steps. What changes in the code is one line; what changes in the measurement is one-sixth of the overlap.

The loss ratio is larger than the blocking call’s share. Eight of fifty-four I/O calls block, roughly one call in seven; the loss in overlapping steps, though, is 7 of 44. The reason is that a blocking call does not stop just its own task, it stops everyone who could have advanced at that tick. While one task blocks, the other seven may sit ready and waiting; they do not advance.

The rest of the table follows this curve to its end: 30 overlapping steps at two calls, 16 at four, 4 at six. At eight calls — that is, when every I/O call in every task blocks — overlapping steps are 0 and ticks 80. The regime returns to single-thread.

This last row defines what compatibility in this lesson’s title means: an asynchronous flow’s numbers are decided by its most blocking part.

Seven Tasks Are Enough

Table B shows the same collapse on the other axis, and one row stands apart from the rest.

When seven tasks block, ticks are 80, overlapping steps 0. The eighth task is still compatible; it gives up control six times. The yields column writes this: 6 yields, 0 overlapping steps.

The previous lesson’s rule gets tested exactly here. Yielding was necessary for overlap but not sufficient; a second ready task was needed too. While the other seven tasks are stuck in their own blocking calls, the eighth gives up control, and there is no one to take the control it gives up. The loop resumes the same task again.

The practical reading of this is blunt: a single compatible part does not save an asynchronous flow. Writing one part of a codebase asynchronously may earn nothing measurable while the rest blocks. The first half of table B shows the same direction: with one task blocking, the loss is 8 overlapping steps; with two, 14; with three, 22. The loss accelerates with task count, because every blocking task takes away both its own overlap and others’ opportunity to overlap.

The Invisibility of the Blocker

Both tables share a property that explains why this defect slips through so often: no row has an error. All eight tasks take every one of their steps, none stops halfway, no exception comes up. The only thing that changes is which tick a step falls on.

It cannot be told apart by looking at the call site either. A compatible call and a blocking call do the same work in the measurement and spend the same step; where they part ways is inside the body. The only thing that says whether a function is blocking is whether calling it returns a resumable object — a function defined with async def returns a coroutine object, an ordinary function returns the result directly.

The direct consequence: in an asynchronous flow, every call without a written await is a suspect. A call that returns its result directly either does not wait at all, or waits while holding the loop; the two cannot be told apart at the call site.

What the measurement contributes here is naming the defect’s symptom. A blocking call produces no error, does not corrupt output, writes nothing to a log. The only symptom it produces is a drop in overlapping steps — and if overlapping steps are not counted, no symptom remains at all. Table B’s seventh row is this at its most extreme: the program is written asynchronously, six yield points genuinely run, and the result matches single-thread exactly.

Offloading to a Pool and Where the Cost Falls

Table C measures the fix. When a blocking call has no compatible counterpart, the call can be left as it is and run somewhere else instead: it gets offloaded to an executor pool, and the event loop regains its yield point meanwhile.

The rows at the two ends are known numbers: if everything blocks, 80 steps, 0 yields, 80 ticks, 0 overlapping steps; if everything is compatible, 80 steps, 54 yields, 36 ticks, 44 overlapping steps.

Offloading adds two rows, and both give overlapping steps of 44 — offloading recovers overlap exactly. Where they part ways is tick count.

When the offload cost falls once per task, total steps rise from 80 to 88, and ticks become 44. Against the blocking regime’s 80 ticks, that is half the gain back; against the compatible regime’s 36 ticks, eight ticks behind. The cost is eight steps, the loss eight ticks; the account matches exactly.

When the offload cost falls once per call, total steps rise to 134, and ticks become 90. This row holds this lesson’s most notable number: 90 is larger than the 80 ticks of doing nothing and just blocking. Even though overlapping steps rose from 0 to 44, the total got worse.

Not a contradiction, arithmetic. Every offloaded call adds one step, and fifty-four calls make fifty-four steps; the forty-four overlapping steps earned are smaller than this addition. Reading overlapping steps on their own misleads for this reason — the third lesson’s transfer table had the same trap. A measurement cannot be compared without saying which steps go into the count.

The guideline that follows is concrete: offload by the largest possible chunk, not per call. Offloading eight tasks one by one costs eight steps; offloading every call separately costs fifty-four. The size of the work being offloaded directly decides whether the offload pays off.

The order the three rows give together also suggests a priority rule. The best result is switching to a compatible call (36 ticks); second best is offloading in large chunks (44); worst is offloading every call separately (90). A pool is not a substitute for a compatible call, it is a solution reached for when no compatible counterpart exists, and its cost is always countable.

Summary

• Blocking code is code that does not return control to the event loop; what it blocks is not the calling body, it is the loop itself.
• One blocking I/O call in every task drops overlapping steps from 44 to 37 and raises ticks from 36 to 43; when every call blocks, the result is 80 ticks / 0 overlapping steps — single thread.
• When seven of eight tasks block, overlapping steps are 0; even though the eighth task gives up control 6 times, there is no ready task to take the control it gives up.
• Offloading the call to an executor pool restores overlapping steps to 44; when the offload cost falls once per task, ticks are 44, and once per call, 90.
• Offloading per call produces more ticks than not fixing anything at all (90 against 80): when the earned overlap stays smaller than the number of added steps, the measurement flips sign.

Next Step

This lesson’s last section used the pool as a fix and never asked about its width; the pool was treated as unbounded. A real pool has a width, and what sits behind it — a thread, or a process — changes the measurement. The next lesson asks this: distributing work through the same interface, what numbers does it give on the same task with a thread behind it versus a process behind it, and does the interface being the same mean the choice is the same too?