Executor Pools

The previous lesson offloaded a blocking call to an executor pool and never asked about the pool’s width; the measurement treated the pool as unbounded. A real pool has a width, and what sits behind it — a thread, or a process — changes the measurement.

This lesson asks that question. An executor pool is the common interface for distributing work: the same call form is written the same way whether a thread or a process sits behind it. What this lesson measures is the difference underneath the interface: same task, same width, two backends — what numbers come out?

The Common Interface

An executor pool gives three things. Work gets submitted, and a result object comes back in exchange; a function gets mapped over a sequence of inputs, and results return in input order; the pool gets closed, and submitted work runs to completion. All three operations are called with the same name and the same signature on both backends.

A guarantee the interface carries also gives this lesson its measurement condition: the mapping operation’s output depends on input order, not on scheduling. Even though which work runs on which worker at which moment is unspecified, the result list is deterministic. The first block below shows this.

Queue, background worker, and backpressure were built on the service side in the Caching, Queues, and Asynchronous Processing course and are not repeated here. The pool here is not a service component, it is a work-distribution interface inside a single program.

Same Interface, Two Numbers per Backend

In the model’s language, the difference between the two backends collapses into one place.

The thread backend gives as many workers as the width, but stays inside a single interpreter; the slot stays at one. The process backend gives as many workers as the width, and since each is a separate interpreter, the slot equals the width too.

The second and third lessons’ results sit inside this definition. What is new is that both now sit behind the same interface: the caller switches from one to the other by changing a single line, and the numbers change.

The measurement’s assumptions:

• CM59 — The task setup is the same as the previous lessons: eight tasks, ten steps per task, eighty steps total; the three loads differ only in I/O share.
• CM60 — The thread pool’s counterpart in the model is worker = width, slot = 1.
• CM61 — The process pool’s counterpart in the model is worker = width, slot = width.
• CM62 — No real process is started. A real thread is used only to demonstrate the interface, and the work handed to it is pure: it gives the same result for the same input, has no side effects, and the output’s order comes from the mapping operation’s contract, not from the schedule.
• CM63 — The measurement never counts duration; the unit the pools are compared on is ticks and overlapping steps.
• CM64 — Tasks are independent; work submitted to the pool does not wait on other work.
• CM65 — Submitting work to the pool is itself free in this measurement.
• CM66 — The process backend’s data-transfer cost is not included in this measurement; that cost was measured separately in the third lesson. The process column here is an upper bound for that reason.
• CM67 — The width sweep uses the same task sequence; only width changes.
• CM68 — Width never exceeds task count in any run.
• CM69 — The count in the last row is the number of distinct ticks the three loads give; it counts how many separate results come out, not their size.
• CM70 — The measurement is a single run, and the seed is fixed.

The Measurement

"""Executor pools: same interface, two backends, different overlap on the same task."""

from concurrent.futures import ThreadPoolExecutor

SEED = 20260817
CPU, IO = "cpu", "io"


def make_rng(seed):
    state = seed % 2147483646 + 1

    def draw(n):
        nonlocal state
        state = (state * 48271) % 2147483647
        return state % n
    return draw


def tasks(count=8, steps=10, io_share=7, seed=SEED):
    """Each task is a step sequence; io_share/10 fraction are I/O steps."""
    draw, result = make_rng(seed), []
    for i in range(count):
        result.append([IO if draw(10) < io_share else CPU for _ in range(steps)])
    return result


def run(jobs, workers, cpu_slots):
    """Returns: tick, overlapping steps, cpu steps, io steps."""
    remaining = [list(j) for j in jobs]
    tick = overlap = cpu_steps = io_steps = 0
    while any(remaining):
        active = [i for i, j in enumerate(remaining) if j][:workers]
        if not active:
            break
        slots_left, advanced = cpu_slots, 0
        for i in active:
            step = remaining[i][0]
            if step == CPU:
                if slots_left <= 0:
                    continue
                slots_left -= 1
                cpu_steps += 1
            else:
                io_steps += 1
            remaining[i].pop(0)
            advanced += 1
        tick += 1
        overlap += max(0, advanced - 1)
    return tick, overlap, cpu_steps, io_steps


def pool(backend, width):
    """The interface is the same, the backend is told by two settings: worker and slot.

    thread  : width workers, single slot   (single interpreter)
    process : width workers, width slots   (separate interpreters)
    """
    return (width, 1) if backend == "thread" else (width, width)


def square(n):
    """Work handed to the pool: pure, no side effects, order does not matter."""
    return n * n


LOADS = (("I/O bound", 7), ("balanced", 5), ("CPU bound", 1))
BACKENDS = ("thread", "process")

with ThreadPoolExecutor(max_workers=4) as pool_obj:
    print("same interface:", list(pool_obj.map(square, range(6))))
print("map output order comes from input, not from scheduling")

print()
print("width eight - same task, two backends")
print(f"{'load':<22s} " + " ".join(f"{label:>22s}" for label in BACKENDS))
print(f"{'':22s} " + " ".join(f"{'tick / overlap':>22s}" for _ in BACKENDS))
for label, share in LOADS:
    j = tasks(io_share=share)
    cells = []
    for backend in BACKENDS:
        workers, slots = pool(backend, 8)
        tick, overlap, _, _ = run(j, workers, slots)
        cells.append(f"{f'{tick} / {overlap}':>22s}")
    print(f"{label:<22s} " + " ".join(cells))

for backend in BACKENDS:
    print()
    print(f"{backend} pool, as width grows")
    print(f"{'width':>9s} " + " ".join(f"{label:>22s}" for label, _ in LOADS))
    print(f"{'':9s} " + " ".join(f"{'tick / overlap':>22s}" for _ in LOADS))
    for width in (1, 2, 4, 8):
        cells = []
        for label, share in LOADS:
            workers, slots = pool(backend, width)
            tick, overlap, _, _ = run(tasks(io_share=share), workers, slots)
            cells.append(f"{f'{tick} / {overlap}':>22s}")
        print(f"{width:9d} " + " ".join(cells))

print()
for backend in BACKENDS:
    counts = {run(tasks(io_share=share), *pool(backend, 8))[0] for _, share in LOADS}
    print(f"{backend} pool, distinct tick counts across three loads: {len(counts)} "
          f"({sorted(counts)})")

same interface: [0, 1, 4, 9, 16, 25]
map output order comes from input, not from scheduling

width eight - same task, two backends
load                                   thread                process
                               tick / overlap         tick / overlap
I/O bound                             31 / 49                10 / 70
balanced                              42 / 38                10 / 70
CPU bound                              73 / 7                10 / 70

thread pool, as width grows
    width              I/O bound               balanced              CPU bound
                  tick / overlap         tick / overlap         tick / overlap
        1                 80 / 0                 80 / 0                 80 / 0
        2                44 / 36                52 / 28                 73 / 7
        4                34 / 46                44 / 36                 73 / 7
        8                31 / 49                42 / 38                 73 / 7

process pool, as width grows
    width              I/O bound               balanced              CPU bound
                  tick / overlap         tick / overlap         tick / overlap
        1                 80 / 0                 80 / 0                 80 / 0
        2                40 / 40                40 / 40                40 / 40
        4                20 / 60                20 / 60                20 / 60
        8                10 / 70                10 / 70                10 / 70

thread pool, distinct tick counts across three loads: 3 ([31, 42, 73])
process pool, distinct tick counts across three loads: 1 ([10])

Same Task, Two Results

The first table gives this lesson’s claim in three rows. The only difference between the left and right columns is the backend; the tasks are the same object, width is the same number, the scheduler is the same function.

On the I/O-bound load, the thread pool gives 31 / 49, the process pool 10 / 70. On the balanced load, 42 / 38 against 10 / 70. On the CPU-bound load, 73 / 7 against 10 / 70.

The left column gets worse going down, the right column does not change at all. The source of this is in the two settings’ definition: the thread pool sees the load’s CPU share, because CPU steps queue up in a single slot. In the process pool, since the slot equals task count, CPU steps stop being the constraint, and load composition becomes invisible.

This gives this lesson’s sentence: behind the same interface, there is no same choice. Changing the pool object in a codebase is a one-line job; its payoff in the measurement ranges, depending on the load, from nothing to a sevenfold difference.

Width Earns Differently on the Two Backends

The lower two tables raise width from one to eight, giving two entirely different shapes.

In the thread pool, every column follows its own curve. The I/O-bound column drops from 80 to 44, to 34, to 31 — earning with diminishing returns. The CPU-bound column drops to 73 at the second width and stops there; widths four and eight add nothing. In this pool, what raising width pays back is exactly the load’s I/O share.

In the process pool, all three columns are the same: 80, 40, 20, 10. Every time width doubles, ticks halve, and this holds across all three loads. The reason is worker and slot growing together: at every width, every live task can advance, and the only constraint is the ten steps per task.

Both tables’ first row is also the same: at width one, both backends give 80 / 0. Whatever sits behind the pool, a single-worker pool is single-thread. A comparison is unreadable if it does not state pool width — the first row shows this.

The number to watch when choosing width is in these tables too: the difference between two consecutive rows. In the thread pool, for the CPU-bound column, the difference after the second row is 0; raising width to four or eight adds nothing measurable. In the I/O-bound column, that same difference from four to eight is 3 ticks. In the process pool, the difference stays large at every step and is the same across all three columns. There is no single correct width; what there is, is stopping where the difference between two measurements shrinks.

A warning sits outside these tables and is written in the assumptions: the process column does not count data transfer. The third lesson measured that cost separately; adding fifteen transfer steps to each task at both ends zeroed out four slots’ gain. The process column here therefore gives the best reachable number, an upper bound. The thread column carries no such outside cost, because there is nothing to transfer within the same address space.

The Shape of the Work Handed to the Pool

The interface being the same does not mean the work that can be handed to it is the same, and the choice often gets decided here, before the numbers.

The process backend requires the work to be transferable. The submitted function and its input get converted to a representation transferable to the other end, and the result gets transferred back. A value that cannot be transferred cannot enter the pool. This transfer, moreover, is exactly the transfer steps measured in the third lesson: when the work is small and the data is large, the gain melts away there.

The thread backend does not require transfer, but it requires something else: the work has to release the lock. Work that does not release it falls into the tables’ CPU-bound column, and raising width does not save it — that column stopped moving after the second width.

Both require the work to be independent. The eight tasks in the measurement do not wait on each other; if one task’s result were used by another, the waiting task would not advance in that tick, and overlapping steps would drop. The pool interface does not see this dependency, because every submitted piece of work gets submitted on its own.

The measurement’s numbers come out because these three conditions hold. When one of them breaks, the table shifts downward; when none breaks, the numbers above are the best reachable case. The first question to ask when choosing a pool, for this reason, is not width: is the work itself in the shape the backend requires?

What the Interface Does Not Say

The last two lines are this topic’s closing number: the thread pool gives 3 distinct tick counts across the three loads ([31, 42, 73]); the process pool gives 1 ([10]).

These two numbers define what a backend choice is. Switching to the thread pool means making the measurement load-sensitive: the result cannot be predicted without knowing which load gets handed in. Switching to the process pool — as long as transfer cost is not counted — makes the measurement load-independent.

The interface calls both the same way, and that is a convenience. But the same call form does not promise the same behavior. What the common interface hides is exactly what this lesson measures: which step can overlap. Changing the pool object in a codebase with two lines and expecting the result to stay the same means counting these two numbers — 3 and 1 — as the same.

Every regime in this topic has answered the same question so far: which regime earns what? Across five lessons, the answer came out in the same shape every time — gain depends on the number of overlappable steps and how many slots those steps can be spread across. The answer always had an input, and that input never got asked about: which step is which type. The setup handed us that; in a real program, nobody hands it over.

Summary

• An executor pool is the common interface for distributing work; the same call form works with both backends, and the mapping operation’s output order comes from input, not from scheduling.
• In the model, the thread backend means worker = width, slot = 1; the process backend means worker = width, slot = width. The entire difference sits in these two settings.
• At width eight, the same task gives 31 / 49, 42 / 38, and 73 / 7 across the three loads on the thread pool; on the process pool, 10 / 70 on all three.
• Raising width earns, on the thread pool, exactly the load’s I/O share, and stops at the second width on the CPU-bound load; on the process pool, it halves ticks on all three loads.
• The thread pool gives 3 distinct tick counts across the three loads, the process pool gives 1. Behind the same interface, there is no same choice; the process column is also an upper bound that does not count data transfer.

Next Step

What has been counted up to here was always the regime: which regime earns what, on which load. In every measurement, the tasks’ step sequence was in hand — the setup stated which step was a CPU step and which was an I/O step. In a real program, that sequence is not written down. What is in hand is a codebase, and which call is heavy, which line sits on the hot path, is invisible. The next lesson closes that gap: how is it found which step of a program is heavy, without measuring duration, by counting calls instead?