--- title: 'Concurrency and Performance' source: 'https://academia.sh/en/courses/python-concurrency' language: en updated: '2026-08-17T04:42:13+00:00' license: 'CC BY-SA 4.0' lessons: 10 outcomes: - 'Explains the effect of the global interpreter lock on the choice of concurrency' - 'Chooses between thread, process, and asynchronous models on a criterion' - 'Finds bottlenecks by profiling' --- # Concurrency and Performance This course does not measure time, it counts overlapping steps; it measures which step the global interpreter lock allows to overlap, what threads and processes each gain under which load, and that a measurement cannot be compared without saying what it counts. ## Concurrency Models What the global interpreter lock restricts, threads for I/O-bound work, multiprocessing for CPU-bound work, the event loop and coroutines, moving blocking code out of an asynchronous flow, and distributing work through a common interface. ### The Global Interpreter Lock Source: https://academia.sh/en/courses/python-concurrency/global-interpreter-lock This course counts overlapping steps, not duration; in an eighty-step setup, a single thread gives 80 ticks and 0 overlapping steps on every load, a single-slot regime takes at most 1 CPU step per tick, and the I/O-bound load's 26 CPU steps spread across 26 separate ticks. The previous course spent eleven lessons measuring a single question: who answered the call, who enforced the contract. Its close exposed an assumption those measurements never wrote down — single thread. When asked which class answered, a single resolution chain ran; when asked which layer caught the violation, calls ran one finishing before the next started. Under a single thread, "who answered" had exactly one answer every time. With more than one thread, the question changes shape. What is asked now is not who answered, but who could advance at the same time. This course counts that question over ten lessons, and starts the first at its narrowest point: how many threads can take a CPU step at once in Python's interpreter. #### The Lock Is a Number The **global interpreter lock (GIL)** is a restriction the interpreter places to keep its own internal structures consistent: only one thread executes interpreter bytecode at a time. A thread not holding the lock waits. The lock is given to one thread at a time, released, and passed to another. In this lesson's view, the lock is not a mechanism, it is a **number**: how many threads can take a CPU step at once. The course's measure name for this number is **slot**. With the lock in place, the slot count is one. Places where the lock gets released do not disturb this number, because a step where the lock is released is not a CPU step to begin with. A thread reading a file, waiting on a network reply, or sleeping does not execute interpreter bytecode; the lock passes to another thread meanwhile. This is the course's second step type: the **I/O step**. Slot count does not bound these. Process, thread, scheduling, and context switching were built as mechanisms in the Operating System Concepts course and are not repeated here. The question here is not how the operating system orders threads, it is which step Python can overlap. #### Why We Do Not Measure Duration The first instinct for a concurrency measurement is comparing duration. This course does not do that, and will not in any lesson. Real duration depends on three things at once: the machine itself, its load at the moment, and when the measurement started. All three are outside the writing. The same code run twice on the same machine gives two different numbers, and cannot be compared across two different machines. A claim written with duration cannot be reproduced where it is read. What is needed instead is something countable: if more than one task advances in a tick, one less than the number that advanced counts as overlapped. That is the course's measure, called the **overlapping step**. It comes from the setup, is read from the run, and gives the same number everywhere. The ban is not limited to seconds. A speedup factor is a duration claim too and carries the same problem; "N times faster" says nothing unless it also states which work was measured under which condition. A lesson near the course's end measures this from the other direction: same work, same regime, the only difference being what the measurement counts — and two different numbers come out. This lesson lays the ground for that claim; the counted unit gets named in every measurement. #### The Setup Eight tasks, ten steps each; 80 steps total. Every step is either a CPU step or an I/O step. A load is defined by its share of I/O steps: seven in ten for an I/O-bound load, five in ten for balanced, one in ten for CPU-bound. Tasks advance tick by tick in a deterministic scheduler. Two numbers decide how many tasks can advance in one tick: **worker**, the number of tasks that can advance at once; and **CPU slot**, how many tasks can take a CPU step in one tick. A single slot means CPU steps get taken one at a time. The measurement's assumptions: - **CM1** — The oracle is the setup itself: we generated the eight tasks' step sequences, so which step is which type is known. - **CM2** — Eight tasks, ten steps per task, eighty steps total. The three loads differ only in I/O share; step count is the same in all three. - **CM3** — The tick is the scheduler's unit. In one tick, a task takes at most one step. - **CM4** — Worker is the most tasks that can advance in one tick; order is given by task index. - **CM5** — CPU slot is how many tasks can take a CPU step in one tick. **A single slot is this model's counterpart to the global interpreter lock.** - **CM6** — An I/O step is unaffected by slots; this is the step where the lock gets released. - **CM7** — Overlapping steps are one less than the number of tasks that advance in a tick; zero if no task advances. - **CM8** — The single-thread regime has one worker, one slot: one task, one step per tick. - **CM9** — Duration is never measured or written. What is counted is steps, ticks, and overlapping steps. - **CM10** — `trace` is `run`'s report twin: it makes the same decisions in the same order and does not change behavior. The output's last line shows the two give the same number. - **CM11** — The measurement is a single run; the seed is fixed and the setup is deterministic. - **CM12** — This lesson looks only inside the tick. Regimes' total tick counts are the next lessons' measure and are not printed here. #### The Measurement ```python """Global interpreter lock: how many tasks can advance in one tick.""" SEED = 20260817 CPU, IO = "cpu", "io" def make_rng(seed): state = seed % 2147483646 + 1 def draw(n): nonlocal state state = (state * 48271) % 2147483647 return state % n return draw def tasks(count=8, steps=10, io_share=7, seed=SEED): """Each task is a step sequence; io_share/10 fraction are I/O steps.""" draw, result = make_rng(seed), [] for i in range(count): result.append([IO if draw(10) < io_share else CPU for _ in range(steps)]) return result def run(jobs, workers, cpu_slots): """Deterministic scheduler advancing tick by tick. workers : how many tasks can advance at the same time cpu_slots : how many tasks can take a CPU step in one tick Returns: tick count, overlapping steps, cpu steps, io steps """ remaining = [list(j) for j in jobs] tick = overlap = cpu_steps = io_steps = 0 while any(remaining): active = [i for i, j in enumerate(remaining) if j][:workers] if not active: break slots_left, advanced = cpu_slots, 0 for i in active: step = remaining[i][0] if step == CPU: if slots_left <= 0: continue slots_left -= 1 cpu_steps += 1 else: io_steps += 1 remaining[i].pop(0) advanced += 1 tick += 1 overlap += max(0, advanced - 1) return tick, overlap, cpu_steps, io_steps def trace(jobs, workers, cpu_slots): """run's report twin: dumps the steps taken each tick, turn by turn.""" remaining, rows = [list(j) for j in jobs], [] while any(remaining): active = [i for i, j in enumerate(remaining) if j][:workers] if not active: break slots_left, cpu, io = cpu_slots, 0, 0 for i in active: if remaining[i][0] == CPU: if slots_left <= 0: continue slots_left -= 1 cpu += 1 else: io += 1 remaining[i].pop(0) rows.append((cpu, io)) return rows LOADS = (("I/O bound", 7), ("balanced", 5), ("CPU bound", 1)) print(f"{'load':<20s} {'steps':>5s} {'cpu':>5s} {'io':>5s} " f"{'single-thread ticks':>20s} {'overlap':>8s}") for label, share in LOADS: j = tasks(io_share=share) tick, overlap, cpu_steps, io_steps = run(j, 1, 1) print(f"{label:<20s} {sum(len(t) for t in j):5d} {cpu_steps:5d} " f"{io_steps:5d} {tick:20d} {overlap:8d}") j = tasks(io_share=7) rows = trace(j, 8, 1) print() print("single slot, eight workers, I/O bound load — first twelve ticks") print(f"{'tick':>4s} {'advanced':>9s} {'cpu':>5s} {'io':>5s} {'overlap':>8s}") for i, (cpu, io) in enumerate(rows[:12], 1): print(f"{i:4d} {cpu + io:9d} {cpu:5d} {io:5d} {max(0, cpu + io - 1):8d}") print(f"steps taken in first twelve ticks {sum(cpu + io for cpu, io in rows[:12])}, " f"overlapping steps {sum(max(0, cpu + io - 1) for cpu, io in rows[:12])}") print() print(f"most cpu steps taken in one tick {max(cpu for cpu, _ in rows)}") print(f"ticks carrying a cpu step {sum(1 for cpu, _ in rows if cpu)}, " f"total cpu steps {sum(cpu for cpu, _ in rows)}") print(f"trace and run agree: cpu " f"{sum(cpu for cpu, _ in rows) == run(j, 8, 1)[2]}, io " f"{sum(io for _, io in rows) == run(j, 8, 1)[3]}") ``` ``` load steps cpu io single-thread ticks overlap I/O bound 80 26 54 80 0 balanced 80 40 40 80 0 CPU bound 80 73 7 80 0 single slot, eight workers, I/O bound load — first twelve ticks tick advanced cpu io overlap 1 5 1 4 4 2 5 1 4 4 3 5 1 4 4 4 5 1 4 4 5 3 1 2 2 6 4 1 3 3 7 4 1 3 3 8 5 1 4 4 9 5 1 4 4 10 3 1 2 2 11 2 1 1 1 12 1 1 0 0 steps taken in first twelve ticks 47, overlapping steps 35 most cpu steps taken in one tick 1 ticks carrying a cpu step 26, total cpu steps 26 trace and run agree: cpu True, io True ``` #### Single Thread: Load Composition Changes Nothing The upper table's last two columns are the same in all three rows: **80 ticks, 0 overlapping steps.** This is not a performance result, it is a **definition**. In the single-thread regime, worker is one; one task advances per tick, and one advancing task minus one is zero. Eighty steps make eighty ticks. Load composition has no effect: in the I/O-bound load, 54 steps could have released the lock, but with no other thread to release it to, releasing it pays off nothing. This gives the course's first reading: **overlap is a property of the regime, not the load.** The same eighty steps overlap on no load if there is no other task to advance. The third and fourth columns, though, are the load's own property, independent of regime. The I/O-bound load has **26** CPU, **54** I/O steps; the CPU-bound load has **73** CPU, **7** I/O steps. The total is 80 in all three. These two numbers stay fixed throughout the measurement, because the regime does not change a step's type, only which tick it gets taken in. #### Inside the Tick The lower table opens up the single-slot regime's first twelve ticks row by row, and this is the lesson's real measurement. The CPU column is **1** in all twelve rows. The most CPU steps taken in one tick is **1**, and this holds for the entire run. Whatever the slot count is, that is the upper bound on CPU steps per tick; here the slot is one, so the bound is one. The I/O column ranges between 0 and 4. Four tasks can take an I/O step in the same tick, because those steps do not use the slot. In the first tick, five tasks advance: four I/O steps, one CPU step. Four overlapping steps are earned in that tick. With eight tasks alive, only five advancing might look odd at first. The reason is in the columns: at that tick, four tasks' next step is a CPU step, and there is only one slot. Whichever task takes the slot advances, the other three **wait**. A waiting task takes no step at all that tick. The twelfth row shows the lock at its plainest: **1 advanced, 1 CPU step, 0 overlapping steps.** Only one task could advance at that tick, and its step was a CPU step. A tick like this is indistinguishable from the single-thread regime. #### Worker and Slot Are Two Separate Numbers The model has two settings, and confusing them is this topic's most common mistake. **Worker** says how many tasks can enter the queue in a tick. Entering the queue does not guarantee advancing: at the twelfth tick, two tasks were alive, one entered and advanced, the other's step was a CPU step and the slot was taken. **Slot** says how many of those in the queue can take a CPU step. Since the two are set independently, the model produces four separate regimes, and each of this course's remaining lessons changes one of these two numbers. Opening a thread grows the **worker** count; since the interpreter is single, it does not grow the slot. Starting a separate process, though, means a separate interpreter, and grows the **slot**. Asynchronous programming takes a third path: it asks the operating system for neither worker nor slot, and leaves the queueing decision to the code itself. This distinction matters because "concurrency" is not one thing, it is a set of choices. Adding a thread to a program does not speed it up under every load; what gets added is a **worker**, and the only thing a worker can earn is **overlap in I/O steps**. In the table above, the CPU-bound load has 7 steps that could overlap; giving it eight workers cannot push the gain past that number. How many workers earn how much is the next two lessons' measure, but the upper bound is already written in this lesson: **overlapping steps can never exceed the number of steps that could overlap.** #### The Step That Cannot Overlap The last two lines give this lesson's structural finding: **ticks carrying a CPU step 26, total CPU steps 26.** The two numbers being equal is not a coincidence, it is the direct result of a single slot. Since a tick can carry at most one CPU step, 26 CPU steps have to spread across at least **26 separate ticks**. The general rule: in a single-slot regime, **tick count can never drop below CPU step count.** With slot count `y`, the lower bound is CPU step count divided by `y`, rounded up. This is where the course's measurement axis comes from. Not all 80 steps of a load are equal: in the I/O-bound load, 54 steps can overlap, **26 cannot**. What sets the upper bound on the gain expected from concurrency is not task count or worker count, it is the **share of steps that can overlap**. Of the 47 steps taken in the first twelve ticks, 35 are overlapping. The difference between these two numbers is exactly 12 — the tick count. This is the identity that holds throughout the model: **tick count is found by subtracting overlapping steps from total steps.** Counting overlapping steps and counting ticks are two faces of the same measurement; this is why the course never needs duration. #### Summary - In this course, the global interpreter lock is not a mechanism, it is a number: how many tasks can take a CPU step in one tick. With the lock in place, this number — the slot — is one. - I/O steps do not use the slot; these are the steps where the lock gets released, and in a single-slot regime, four tasks can take one in the same tick. - The single-thread regime gives 80 ticks and 0 overlapping steps on all three loads. Overlap is a property of the regime, not the load; no load overlaps without a second task to advance. - With a single slot, the most CPU steps taken in one tick is 1; the I/O-bound load's 26 CPU steps spread across 26 separate ticks, and tick count cannot drop below that number. - Tick count is found by subtracting overlapping steps from total steps; this is why overlapping steps are enough to compare without measuring duration. #### Next Step This lesson looked inside the tick and never wrote the total tick count. The next lesson counts it: running eight workers with a single slot — the thread regime — finishes eighty steps in how many ticks? The answer is not one number, it is **two** — because the same mechanism behaves completely differently on an I/O-bound load versus a CPU-bound one, and where that difference comes from is already written in this lesson's last two lines. ### Threads Source: https://academia.sh/en/courses/python-concurrency/threads Eight workers and a single slot finish eighty steps in 31 ticks on the I/O-bound load (gain 49, ratio 0.6125), and only 73 ticks on the CPU-bound load (gain 7); same mechanism, a sevenfold difference, and the gain is bounded in every load by the number of steps that can overlap. The previous lesson looked inside the tick and never wrote the total tick count. In the single-slot regime's first twelve ticks, the CPU column was always **1**, while the I/O column climbed as high as four. From that came a lower bound: 26 CPU steps have to spread across at least 26 separate ticks. This lesson asks for the number itself. Queueing eight tasks at once — opening a **thread** — finishes eighty steps in how many ticks? The answer is not one number. The same mechanism gives two entirely different numbers depending on load composition, and what this lesson measures is the size of that difference. #### A Thread Adds a Worker, Not a Slot A thread is a separate execution flow within the same process. Process, thread, scheduling, and context switching were built as mechanisms in the Operating System Concepts course and are not repeated here. This lesson's question is not mechanism, it is a **number**: which setting of the model does opening a thread change? The answer is written in the previous lesson's two settings. Eight threads means eight tasks can enter the queue at once; that is, worker count rises to eight. But all eight threads are still inside the same interpreter, and that interpreter's lock is single. The slot stays at one. The direct consequence: adding a thread lets I/O steps overlap; it does not let CPU steps overlap. A thread releases the lock while waiting on a network reply, and another thread advances. If both want to execute bytecode at the same instant, one waits. This distinction corrects both of two common claims about threads. "A thread makes work parallel" is true only for I/O steps; CPU steps have no parallelism, only order. "Threads are useless because of the lock" misses the I/O steps. Both share the same gap: **not saying which step is meant.** The measurement below counts exactly this distinction. #### The Regime Is Modeled, Not Run This lesson does not start real threads, and that is not a convenience, it is the measurement's condition. A program that actually opens eight threads leaves which thread advances at which moment to the operating system's scheduler. The same program run twice produces two different orders; the written output will not reproduce on the reader's machine. What is meant to be measured — which step overlaps — stays buried under that noise. Instead, scheduling gets **modeled**: the ordering rule is written down, the seed is fixed, and the run gives the same number everywhere. The measurement's assumptions: - **CM13** — The core is the same as the previous lesson: eight tasks, ten steps per task, eighty steps total; the seed is fixed and the setup is deterministic. - **CM14** — The thread regime's counterpart in the model is worker 8, slot 1. - **CM15** — No real thread is started; if one were, the output would depend on the scheduler and would not reproduce. - **CM16** — The comparison baseline is the single-thread regime (worker 1, slot 1), and it gives 80 ticks on all three loads. - **CM17** — Gain is the regime's tick count subtracted from the single-thread tick count; ratio is gain divided by the single-thread tick count. **This is not a speedup factor**, it is the difference between two tick counts. - **CM18** — The lower bound is the rule built in the previous lesson: in a single slot, tick count cannot drop below CPU step count. - **CM19** — The worker sweep uses the same task sequence; only worker count changes. - **CM20** — Context-switch cost is zero in the model. A real system has this cost, and it is outside the model; what is measured is not the size of the cost, it is the bound on overlap. - **CM21** — Tasks are independent of each other; shared state, locking, and races are outside this measurement. - **CM22** — Duration is never measured. The counted unit is steps, ticks, and overlapping steps. - **CM23** — The measurement is a single run; all three loads are generated from the same core. #### The Measurement ```python """Threads: same mechanism, two loads, two separate gains.""" SEED = 20260817 CPU, IO = "cpu", "io" def make_rng(seed): state = seed % 2147483646 + 1 def draw(n): nonlocal state state = (state * 48271) % 2147483647 return state % n return draw def tasks(count=8, steps=10, io_share=7, seed=SEED): """Each task is a step sequence; io_share/10 fraction are I/O steps.""" draw, result = make_rng(seed), [] for i in range(count): result.append([IO if draw(10) < io_share else CPU for _ in range(steps)]) return result def run(jobs, workers, cpu_slots): """Returns: tick, overlapping steps, cpu steps, io steps.""" remaining = [list(j) for j in jobs] tick = overlap = cpu_steps = io_steps = 0 while any(remaining): active = [i for i, j in enumerate(remaining) if j][:workers] if not active: break slots_left, advanced = cpu_slots, 0 for i in active: step = remaining[i][0] if step == CPU: if slots_left <= 0: continue slots_left -= 1 cpu_steps += 1 else: io_steps += 1 remaining[i].pop(0) advanced += 1 tick += 1 overlap += max(0, advanced - 1) return tick, overlap, cpu_steps, io_steps LOADS = (("I/O bound", 7), ("balanced", 5), ("CPU bound", 1)) print("thread regime: eight workers, single slot") print(f"{'load':<22s} {'steps':>5s} {'single-thread':>13s} {'thread':>9s} " f"{'overlap':>8s} {'gain':>7s} {'ratio':>7s}") for label, share in LOADS: j = tasks(io_share=share) single = run(j, 1, 1)[0] tick, overlap, _, _ = run(j, 8, 1) print(f"{label:<22s} {sum(len(t) for t in j):5d} {single:13d} {tick:9d} " f"{overlap:8d} {single - tick:7d} {(single - tick) / single:7.4f}") print() print("lower bound versus reached tick") print(f"{'load':<22s} {'cpu steps':>9s} {'lower bound':>11s} {'reached':>9s} " f"{'diff':>5s}") for label, share in LOADS: j = tasks(io_share=share) tick, _, cpu_steps, _ = run(j, 8, 1) print(f"{label:<22s} {cpu_steps:9d} {cpu_steps:11d} {tick:9d} " f"{tick - cpu_steps:5d}") print() print("as worker count grows (slot stays one)") print(f"{'worker':>6s} " + " ".join(f"{label:>13s}" for label, _ in LOADS)) print(f"{'':6s} " + " ".join(f"{'tick / overlap':>13s}" for _ in LOADS)) for workers in (1, 2, 4, 8): cells = [] for label, share in LOADS: tick, overlap, _, _ = run(tasks(io_share=share), workers, 1) cells.append(f"{f'{tick} / {overlap}':>13s}") print(f"{workers:6d} " + " ".join(cells)) print() j_io, j_cpu = tasks(io_share=7), tasks(io_share=1) g_io = run(j_io, 1, 1)[0] - run(j_io, 8, 1)[0] g_cpu = run(j_cpu, 1, 1)[0] - run(j_cpu, 8, 1)[0] print(f"same mechanism, two loads: gain {g_io} ticks versus {g_cpu} ticks, " f"ratio {g_io / g_cpu:.1f}") print(f"CPU-bound load io steps {run(j_cpu, 8, 1)[3]}, " f"gain {g_cpu}; I/O-bound load io steps " f"{run(j_io, 8, 1)[3]}, gain {g_io}") ``` ``` thread regime: eight workers, single slot load steps single-thread thread overlap gain ratio I/O bound 80 80 31 49 49 0.6125 balanced 80 80 42 38 38 0.4750 CPU bound 80 80 73 7 7 0.0875 lower bound versus reached tick load cpu steps lower bound reached diff I/O bound 26 26 31 5 balanced 40 40 42 2 CPU bound 73 73 73 0 as worker count grows (slot stays one) worker I/O bound balanced CPU bound tick / overlap tick / overlap tick / overlap 1 80 / 0 80 / 0 80 / 0 2 44 / 36 52 / 28 73 / 7 4 34 / 46 44 / 36 73 / 7 8 31 / 49 42 / 38 73 / 7 same mechanism, two loads: gain 49 ticks versus 7 ticks, ratio 7.0 CPU-bound load io steps 7, gain 7; I/O-bound load io steps 54, gain 49 ``` #### Same Mechanism, a Sevenfold Difference The upper table's two ends carry this lesson's entire claim. On the **I/O-bound load**, eight workers and a single slot finish eighty steps in **31** ticks. Gain **49 ticks**, ratio **0.6125**. Overlapping steps are also **49**: the identity that tick count is found by subtracting overlapping steps from total steps shows up here exactly. On the **CPU-bound load**, the same mechanism finishes eighty steps in only **73** ticks. Gain **7 ticks**, ratio **0.0875**. The only thing that changes is step type. Task count is the same, step count is the same, worker count is the same, slot is the same, scheduler is the same. The last line's ratio is **7.0** — the difference between the gains is exactly **sevenfold**. The **balanced** load sitting in between shows both ends lie on the same curve: 80 ticks drops to **42**, gain **38**, ratio **0.4750**. As I/O share falls from seven in ten to five in ten, gain drops from 49 to 38; at one in ten, to 7. Gain behaves like a step function of I/O share, and all three measurement points point the same direction. The reading is this: **concurrency is not a speedup mechanism.** The same regime overlaps more than half the work on one load, and cannot get close to a tenth on the other. When a thread is added to a program and no gain shows up, what is missing may not be thread count, but **overlappable steps in that program**. #### The Gain's Upper Bound Is Written in the Code The last line also says where the gain comes from, placing two numbers side by side. On the CPU-bound load, I/O steps are **7**, gain is **7**. The two numbers are equal. On the I/O-bound load, I/O steps are **54**, gain is **49** — gain sits a bit below the number of overlappable steps, but never above it. The rule: **gain can never exceed the number of overlappable steps.** Since CPU steps have to be taken one at a time in a single-slot regime, the only thing that can be shaved off tick count is I/O steps. On the CPU-bound load, only seven steps could overlap; not eight, and certainly not eighty. The middle table shows the same bound from the other end. The lower bound was the rule built in the previous lesson: in a single slot, tick count cannot drop below CPU step count. On the I/O-bound load, lower bound is **26**, reached is **31**; diff **5**. On the CPU-bound load, lower bound is **73**, reached is **73**; diff **0**. The last row matters: on the CPU-bound load, the regime sits **exactly** on the lower bound. Nothing better is possible, because each of the 73 CPU steps demands its own tick. Anyone looking for improvement on this load gets nothing by playing with thread count; what needs to change is **slot count**, and that is not a number a thread can change. The **5**-tick gap on the I/O-bound load is meaningful too: **26** of the 31 ticks carry a CPU step, the remaining **5** carry only I/O steps. In those five ticks, the slot sits empty, because no live task's next step is a CPU step at that moment. The empty slot looks like a loss, but it is not fixable: which task wants a CPU step at which tick is written into the step sequence, not something the scheduler can choose. #### Where Adding Workers Stops The lower table raises worker count from 1 to 8, giving three different behaviors on the three loads. On the **CPU-bound load**, the second worker drops 80 ticks to **73**; the fourth and eighth workers add nothing. Tick count freezes at **73**, overlapping steps at **7**. Seven steps could overlap, and all seven overlapped with the second worker; nothing was left for a third. **Raising thread count has no countable payoff on this load.** On the **I/O-bound load**, the curve is not flat but it is declining: the second worker earns 36 ticks, the third and fourth together earn 10, the fifth through eighth together earn 3. Gain does not grow linearly with worker count; the first few workers already claim most of the overlappable steps. On the **balanced** load, the same shape flattens sooner: from 80 to 52, then 44, then 42. Going to eight workers over four only earns two more ticks. The table's first row carries information too: with a single worker, all three loads give **80 ticks / 0 overlapping steps**. Running eight tasks with one worker is single-threaded no matter what the regime is called. When a program's thread pool width is dropped to one, what is measured is not concurrency, only order. A comparison that does not write down pool width is unreadable for this reason. The common lesson across the three columns: **adding workers earns until overlappable steps run out, then stops.** What decides where it stops is not worker count, it is load composition. And nowhere in this table does the CPU-bound column drop below 73 — there is no way past that bound in this lesson's regime. #### What the Model Leaves Out A measurement is unreadable unless it states what it counts, so two things this model **does not** count need naming. **Context-switch cost counts as zero.** In a real system, switching from one thread to another is not free; there is state to save. The model never counts this cost, so the gain numbers above are written on the **optimistic side**. This does not invalidate the results: adding the cost only shrinks the gain, it does not flip its sign. The CPU-bound load's 7 ticks were already small; adding the cost shrinks them further, and reinforces that threads are pointless on that load. **Shared state is ignored.** The eight tasks are independent of each other; none reads what another writes. In a real program, writing to a shared dictionary from two threads requires a lock in between, and waiting on that lock adds a new kind of wait. This falls outside what the model measures, but its direction is clear: **waiting on a lock reduces overlapping steps**, because a waiting task does not advance in that tick. Both together mean this: the measured **49** and **7** are the upper bound on the gain that can be expected from threads. A real program can only approach these numbers. If a gain claim sits above this bound, something is uncounted in the measurement. A separate warning follows from this: a thread is a mechanism for **waiting** steps. If a program is computing instead of waiting, it does not fail to gain because of the lock — it fails because there is nothing to overlap. #### Summary - Opening a thread grows worker count in the model, not slot count; all eight threads still share the same interpreter's single lock. - Eight workers and a single slot drop 80 ticks to 31 on the I/O-bound load — gain 49 ticks, ratio 0.6125. The same mechanism drops 80 to only 73 on the CPU-bound load — gain 7 ticks. The difference is exactly sevenfold. - Gain can never exceed the number of overlappable steps: on the CPU-bound load, I/O steps are 7 and gain is 7. - On the CPU-bound load, the regime sits exactly on the lower bound (73 against 73); on the I/O-bound load, a 5-tick gap remains, and the slot sits empty in those five ticks. - Adding workers earns until overlappable steps run out: on the CPU-bound load, gain freezes at the second worker; on the I/O-bound load, it keeps earning, diminishingly, through the eighth. #### Next Step This lesson's last number shows a boundary: on the CPU-bound load, no matter how many workers get added, tick count does not drop below 73, because the setting that does not change is the slot. The next lesson changes that setting. Starting a separate process means a separate interpreter, and a separate interpreter means a separate slot. Its question: how far does four slots drop eighty steps on the CPU-bound load, and once slot count equals task count, does load composition still say anything? ### Multiprocessing Source: https://academia.sh/en/courses/python-concurrency/multiprocessing Four slots drop the CPU-bound load's eighty steps to 20 ticks (gain 60); at eight slots, all three loads give 10 ticks and 70 overlapping steps; against that, adding 15 transfer steps to each task at both ends drops four slots' gain from 60 ticks to 0. The previous lesson hit a wall. On the CPU-bound load, raising worker count from two to eight left tick count at 73; gain froze at the second worker and never moved again. The reason was written in the table: the setting that does not change is the slot. Seventy-three CPU steps demand seventy-three ticks as long as only one CPU step can be taken per tick. A thread cannot change that setting, because all eight threads sit inside the same interpreter. What can change it is a separate interpreter. This lesson measures that path and asks two questions: how far does four slots drop eighty steps, and once slot count equals task count, does load composition still say anything? #### Separate Process, Separate Interpreter A process is a unit of execution with its own address space. The difference between process and thread was built as mechanism in the Operating System Concepts course and is not repeated here. This lesson's question is again a number: which setting of the model does starting a separate process change? A separate process means a separate interpreter. The separate interpreter has its own global interpreter lock, and that lock has nothing to do with another's. Four processes, four independent locks — that is, **four slots**. Its counterpart in the model is direct: the CPU slot count rises to four. This is what a thread cannot do. A thread adds a worker; a process adds a slot. The previous lesson's wall was not a worker wall, it was a slot wall; this is why what clears it is a process. #### What Processes Pay Is Countable A separate address space has a cost, and that cost can be written in the course's measurement unit. Two threads in the same process see the same object; one reads what the other writes. Two separate processes do not. Handing a task to a process means **transferring** its input to that process; getting the result back means transferring it back. What gets transferred is converted to a transferable representation, then rebuilt at the other end. This work is not a duration, it is a **step sequence**. The model counts it exactly that way: a number of CPU steps added to both ends of every task. A transfer step is a CPU step too, and it uses the slot. The measurement's last section asks at what size the cost eats the gain. The measurement's assumptions: - **CM24** — The core is the same as the previous two lessons: eight tasks, ten steps per task, eighty steps total; the seed is fixed. - **CM25** — The process regime's counterpart in the model is the slot growing; worker stays fixed at eight so that slot is seen as the only variable. - **CM26** — No real process is started. If one were, which process advances when would depend on the operating system, and the output would not reproduce. - **CM27** — Four slots means four separate interpreters; eight slots means as many interpreters as tasks. - **CM28** — The lower-bound rule continues: tick count cannot drop below CPU step count divided by slot count, rounded up. - **CM29** — Transfer cost is the number of CPU steps added to both ends of every task: incoming data and returning result. The cost is an integer and is the same for every task. - **CM30** — A transfer step uses the slot; it has no separate resource of its own. - **CM31** — Runs with and without transfer cost are generated from the same task sequence; the comparison baseline is the single-thread regime that pays no cost (80 ticks). - **CM32** — Starting a process itself is free in the model; only data transfer is counted. In a real system, starting is costly too, and this only shrinks the measured gain. - **CM33** — Tasks are independent; shared state and coordination between processes are outside this measurement. - **CM34** — Duration is never measured; the counted unit is steps, ticks, and overlapping steps. - **CM35** — The measurement is a single run, and all three loads are generated from the same core. #### The Measurement ```python """Multiprocessing: adding slots, and the slot count reaching task count.""" SEED = 20260817 CPU, IO = "cpu", "io" def make_rng(seed): state = seed % 2147483646 + 1 def draw(n): nonlocal state state = (state * 48271) % 2147483647 return state % n return draw def tasks(count=8, steps=10, io_share=7, seed=SEED): """Each task is a step sequence; io_share/10 fraction are I/O steps.""" draw, result = make_rng(seed), [] for i in range(count): result.append([IO if draw(10) < io_share else CPU for _ in range(steps)]) return result def run(jobs, workers, cpu_slots): """Returns: tick, overlapping steps, cpu steps, io steps.""" remaining = [list(j) for j in jobs] tick = overlap = cpu_steps = io_steps = 0 while any(remaining): active = [i for i, j in enumerate(remaining) if j][:workers] if not active: break slots_left, advanced = cpu_slots, 0 for i in active: step = remaining[i][0] if step == CPU: if slots_left <= 0: continue slots_left -= 1 cpu_steps += 1 else: io_steps += 1 remaining[i].pop(0) advanced += 1 tick += 1 overlap += max(0, advanced - 1) return tick, overlap, cpu_steps, io_steps def with_transfer(jobs, cost): """Adds 'cost' cpu steps to both ends of every task: data transfer.""" return [[CPU] * cost + list(t) + [CPU] * cost for t in jobs] LOADS = (("I/O bound", 7), ("balanced", 5), ("CPU bound", 1)) print("as slot count grows (worker stays eight)") print(f"{'slot':>5s} " + " ".join(f"{label:>13s}" for label, _ in LOADS)) print(f"{'':5s} " + " ".join(f"{'tick / overlap':>13s}" for _ in LOADS)) for slot in range(1, 9): cells = [] for label, share in LOADS: tick, overlap, _, _ = run(tasks(io_share=share), 8, slot) cells.append(f"{f'{tick} / {overlap}':>13s}") print(f"{slot:5d} " + " ".join(cells)) j = tasks(io_share=1) single = run(j, 1, 1)[0] print() print(f"CPU-bound load, single thread {single} ticks") for slot in (1, 4, 8): tick, overlap, cpu_steps, _ = run(j, 8, slot) print(f" {slot} slot: {tick:3d} ticks, {overlap:3d} overlapping steps, " f"gain {single - tick:3d} ticks, lower bound " f"{-(-cpu_steps // slot):3d}") print() print("transfer cost: 'cost' cpu steps added to each task at both ends") print(f"{'cost':>6s} {'steps':>5s} {'four slots':>10s} {'overlap':>8s} " f"{'single-thread':>13s} {'gain':>7s}") for cost in (0, 1, 2, 4, 8, 15): jj = with_transfer(j, cost) tick, overlap, _, _ = run(jj, 8, 4) print(f"{cost:6d} {sum(len(t) for t in jj):5d} {tick:10d} {overlap:8d} " f"{single:13d} {single - tick:7d}") ``` ``` as slot count grows (worker stays eight) slot I/O bound balanced CPU bound tick / overlap tick / overlap tick / overlap 1 31 / 49 42 / 38 73 / 7 2 16 / 64 23 / 57 38 / 42 3 11 / 69 18 / 62 29 / 51 4 10 / 70 16 / 64 20 / 60 5 10 / 70 12 / 68 19 / 61 6 10 / 70 10 / 70 19 / 61 7 10 / 70 10 / 70 16 / 64 8 10 / 70 10 / 70 10 / 70 CPU-bound load, single thread 80 ticks 1 slot: 73 ticks, 7 overlapping steps, gain 7 ticks, lower bound 73 4 slot: 20 ticks, 60 overlapping steps, gain 60 ticks, lower bound 19 8 slot: 10 ticks, 70 overlapping steps, gain 70 ticks, lower bound 10 transfer cost: 'cost' cpu steps added to each task at both ends cost steps four slots overlap single-thread gain 0 80 20 60 80 60 1 96 24 72 80 56 2 112 28 84 80 52 4 144 36 108 80 44 8 208 52 156 80 28 15 320 80 240 80 0 ``` #### Four Slots: From Eighty to Twenty The middle block breaks down the previous lesson's wall. On the CPU-bound load, a single slot gave 73 ticks; four slots finish the same eighty steps in **20** ticks. Gain **60 ticks**, overlapping steps **60**. The comparison carries this lesson's entire point. Threads earned 7 ticks on this load; a process earns **60**. What changes is not task count, not worker count, not the scheduler — only the slot. The lower-bound column falls into place too: 73 CPU steps divided across four slots gives a lower bound of **19**, reached is **20**. A one-tick gap remains, because toward the run's end not enough live tasks are left to fill the slots. At eight slots, the lower bound is **10** and reached is also **10** — this time it sits exactly. This gives a practical reading: **on a CPU-bound load, gain comes from dividing by slot count.** CPU step count is fixed; how many slots they get spread across decides tick count. A thread cannot perform this division. #### When Slots Reach Task Count, the Load Disappears The upper table's last row is this lesson's second measurement, and the course's fourth reading: **at eight slots, all three loads give 10 ticks and 70 overlapping steps.** Across the seven rows above, the three columns stayed apart. At a single slot, there were three separate numbers, 31, 42, and 73; at four slots, 10, 16, and 20. In the eighth row, all three land in the same place, and every difference between them **disappears**. The reason is in the definition: once slot count equals task count, no task waits for lack of a slot. Every task alive at a tick takes a step. Since each of the eight tasks carries ten steps, ten ticks are required, and since eight tasks advance every tick, seven overlapping steps accumulate per tick: 10 ticks, 70 overlapping steps. Step **type** no longer decides anything. This is the measurement's most commonly misread spot. "I/O-bound work wants this regime, CPU-bound work wants that regime" is true, but **conditional**: it is true while the constraint holds. Once the constraint lifts, load composition produces no distinction. Asking about load composition is meaningful where slot count is **less than** task count — which, on a real machine, is always the case. #### A Process on the I/O-Bound Load The upper table's first column answers a separate question: what does a process do on a load where threads already work well? On the I/O-bound load, a single slot gave 31 ticks; four slots drop it to **10**. There is a gain, but two things stand out. First, on this load there is no gain past the fourth slot — four, five, six, seven, and eight slots all give the same **10 / 70** row. The constraint is no longer the slot, it is the ten steps per task. Second, and more important, this column does not include transfer cost. On this load, CPU steps are only **26**; the data to be transferred is the same as in the thread regime. Against the 21 ticks four slots earn, adding even three transfer steps to each task's two ends is enough to flip the account. Opening a process for waiting work means **trading the wait for a transfer cost**. What can be written as a general rule: what adding a slot pays back is decided by CPU step count, and what transfer costs is decided by the size of the transferred data. The two vary independently within the same load. A process earns when CPU steps are many and transferred data is small; when the two reverse, it is the most expensive regime. Spreading concurrency across multiple machines is the subject of the System Design and Distributed Systems curriculum and is not repeated here. Everything this course measures happens on one machine, inside one program; transfer cost is a cost within that boundary too. The difference between distributing to separate processes and distributing to separate machines collapses in the model to one number: the size of the transfer cost. The direction is the same; the scale is not. #### Adding Slots Does Not Earn Linearly The upper table's CPU-bound column deserves reading on its own: 73, 38, 29, 20, 19, 19, 16, 10. The second slot earns 35 ticks, the third 9, the fourth 9, the fifth 1, the sixth 0, the seventh 3, the eighth 6. The curve is neither linear nor smoothly declining. The sixth slot earns nothing; the seventh and eighth earn again. The source of this is in the setup's detail: gain depends on how many tasks want a CPU step at that tick. With five slots, if most ticks already have fewer than five tasks wanting one, the sixth slot sits empty. Toward the end, once tasks even out, a new slot finds work again. The general lesson: **doubling slot count does not halve tick count.** Gain depends on the step sequences' alignment at that moment, and is known only by measuring. What decides where to stop when raising slot count is not a formula, it is the difference between two consecutive measurements. #### Transfer Cost Eats the Gain The lower table writes out the process regime's bill. At zero cost, four slots gave 20 ticks and gain was 60. Adding one transfer step to each task at both ends raises total steps from 80 to **96**, tick count becomes **24**, gain drops to **56**. At two steps, gain is **52**; at four steps, **44**; at eight steps, **28**. At fifteen steps, gain is **0**: 320 steps of work finish in 80 ticks on four slots — exactly where the single thread paying no cost finished. Past this point, the process regime starts losing. The reading of this number: **the gain from adding a process races against the step count of the data transfer.** For a ten-step-per-task job, a thirty-step transfer cost wipes out the gain entirely. This ratio depends on the setup and is not a general threshold; what is general is the relationship itself. Distributing small tasks to processes means the transfer becomes bigger than the work. The table's overlapping-step column shows the trap here too. As cost rises, overlapping steps climb from **60** to **240** — the number grows while the gain shrinks. Overlapping steps mislead when read on their own: the added steps overlap too. Whatever a measurement's denominator is, that is what needs reading; here the denominator is the single thread's 80 ticks, and the comparison is against that. #### Summary - Adding a process grows slot count in the model: a separate process is a separate interpreter, and a separate interpreter is a separate lock. This is what a thread cannot do. - On the CPU-bound load, four slots drop eighty steps to 20 ticks; gain 60 ticks, overlapping steps 60. On the same load, a single slot's gain was 7 ticks. - At eight slots, all three loads give 10 ticks / 70 overlapping steps. Once slots reach task count, CPU steps stop being the constraint, and load composition does not decide the outcome. - The gain from adding slots is not linear: on the CPU-bound load, the second slot earns 35 ticks, the fifth 1, the sixth 0. Where to stop is found by measuring. - Adding 15 transfer steps to each task at both ends drops four slots' gain from 60 ticks to 0: 320 steps of work finishes in 80 ticks, the same place as the cost-free single thread. #### Next Step Up to here, both regimes got their overlap from outside: threads asked the operating system for workers, processes asked a separate interpreter for slots. In both, what decided when a task advances sat outside the code. The next lesson measures the third path: code that does not ask the operating system for overlap, and instead **gives up control on its own terms**. Its question: how much overlap results from how often a coroutine gives up control, and what happens if it never does? ### Asynchronous Programming Source: https://academia.sh/en/courses/python-concurrency/asynchronous-programming The source of overlap is not the operating system, it is the code itself: a coroutine that gives up control before every step gives 31 ticks and 49 overlapping steps with 80 yields, one that gives up control only at I/O steps gives 36 ticks and 44 overlapping steps with 54 yields, and one that never gives up control gives 80 ticks and 0 overlapping steps with 0 yields. The previous two lessons always got their overlap from outside. Threads asked the operating system for workers, processes asked a separate interpreter for slots. In both, what decided when a task advances sat outside the code; the program only stated how many flows it wanted. A third path exists, and this lesson measures it. In **asynchronous programming**, the source of overlap is neither the operating system nor a second interpreter: **it is the code itself.** A coroutine writes, on its own, the point where it gives up control. This lesson's question is narrow and countable for that reason: how much overlap comes from how often a coroutine gives up control, and what happens if it never does? #### A Coroutine Is a Resumable Body A **coroutine** is a function whose body does not run when called; the call returns an object instead, and the body advances only as the coroutine is **resumed**. It is defined with `async def`. The `await` inside its body is not a marker waiting for a result, it is the **point where control gets given up**: the coroutine stops there, returns to whatever resumed it, and lets something else advance. A pausing body is not new to this course. The Data Structures and Functional Tools course measured the generator and showed that the body stops at `yield` and keeps its local names; that measurement is not repeated here. The difference between the two can be written in one sentence: a **generator** stops to give a value, a **coroutine** stops to give up control — one's stop adds an item to a stream, the other's stop hands the turn to something else. The **event loop** is the party that manages these stops. It holds coroutines waiting to be resumed; it resumes one, and once that one reaches a yield point and returns, it resumes the next. Registering a coroutine with the loop and asking it to advance independently makes it a **task**. #### The Event Loop Adds No New Slot In the model's language: the asynchronous regime asks the operating system for neither worker count nor slot count. The event loop runs in a single flow; only one body advances at a time, and the slot is one. Where does overlap come from, then? From exactly one place: another body being able to advance when one gives up control. In this course's measure, without a **yield point**, there is no overlapping step either. This gives a direct upper bound, and the measurement is about to test it: **overlapping steps cannot exceed the number of yields.** The reverse also holds, and gets measured at the end of the lesson: yielding is **necessary** for overlap but **not sufficient**. There also has to be a second task, ready and waiting at that moment, to take the control that gets given up. #### The Measurement Writes Its Own Loop The measurement's scheduler here is a hand-written event loop; the coroutines, though, are real. Bodies defined with `async def` and containing real `await` produce real coroutine objects, resumed with `send`. The only thing changed is **who does the resuming**: a loop with a written-down order, instead of the standard event loop. The reason is the measurement's condition — overlapping steps cannot be counted unless which task gets resumed at which tick is fixed. The short block at the lesson's end runs the real event loop, and `await` itself fixes the order there. The measurement's assumptions: - **CM36** — The task setup is the same as the previous lessons: eight tasks, ten steps per task, eighty steps total, I/O-bound load. - **CM37** — The coroutines are real; they are defined with async def and stop at await. The loop that resumes them is written within the lesson. - **CM38** — A yield point is the number of times a coroutine returns to the event loop; every await is a yield. - **CM39** — The three regimes differ only in where the yield points sit: at every step, only at I/O steps, never. The step sequences are the same in all three. - **CM40** — The event loop is single-flow: the CPU slot is one and does not change. - **CM41** — If a coroutine takes k steps without giving up control, the loop stays busy for those k steps; the first is written to the tick it was resumed on, the remaining k-1 to separate ticks where no one advances. - **CM42** — Resume order follows task index and does not change through the run. - **CM43** — The comparison baseline is the previous lessons' scheduler: the same tasks, eight workers, a single slot. - **CM44** — The real event loop is run only with sequential await; output order is fixed by await itself, not left to a scheduler. - **CM45** — `sleep(0)` is not a wait in this measurement, it is a **yield** marker; what is counted is not duration, it is how many times control gets given up. - **CM46** — Duration is never measured; the counted unit is steps, ticks, overlapping steps, and yields. - **CM47** — The measurement is a single run, and the seed is fixed. #### The Measurement ```python """Coroutines: who gives up control, and how often.""" import asyncio SEED = 20260817 CPU, IO = "cpu", "io" def make_rng(seed): state = seed % 2147483646 + 1 def draw(n): nonlocal state state = (state * 48271) % 2147483647 return state % n return draw def tasks(count=8, steps=10, io_share=7, seed=SEED): """Each task is a step sequence; io_share/10 fraction are I/O steps.""" draw, result = make_rng(seed), [] for i in range(count): result.append([IO if draw(10) < io_share else CPU for _ in range(steps)]) return result def run(jobs, workers, cpu_slots): """The previous lessons' scheduler; comparison baseline.""" remaining = [list(j) for j in jobs] tick = overlap = 0 while any(remaining): active = [i for i, j in enumerate(remaining) if j][:workers] if not active: break slots_left, advanced = cpu_slots, 0 for i in active: if remaining[i][0] == CPU: if slots_left <= 0: continue slots_left -= 1 remaining[i].pop(0) advanced += 1 tick += 1 overlap += max(0, advanced - 1) return tick, overlap class Suspend: """The point where a coroutine hands control back to the event loop.""" def __init__(self, kind): self.kind = kind def __await__(self): return (yield self.kind) async def every_step(steps, log): """Gives up control before every step: yield points = step count.""" for s in steps: await Suspend(s) log.append(s) async def io_step_only(steps, log): """Gives up control only at I/O steps; runs CPU steps back to back.""" for s in steps: if s == IO: await Suspend(s) log.append(s) async def never(steps, log): """Never gives up control.""" for s in steps: log.append(s) def loop(factory, jobs, cpu_slots=1): """Hand-written deterministic event loop. Returns: tick, overlap, yields. If a coroutine takes k steps without giving up control, the loop stays busy for those k steps: k-1 more ticks are written as debt, and no task advances in those ticks. """ log = [[] for _ in jobs] coro = [factory(j, log[i]) for i, j in enumerate(jobs)] pending = [None] * len(coro) alive = [True] * len(coro) debt = [0] * len(coro) yields = 0 def resume(i, prepaid=0): nonlocal yields before = len(log[i]) try: pending[i] = coro[i].send(None) yields += 1 except StopIteration: pending[i] = None alive[i] = False debt[i] += max(0, len(log[i]) - before - prepaid) for i in range(len(coro)): resume(i) # up to the first yield point tick = overlap = 0 while any(alive) or any(debt): busy = [i for i in range(len(coro)) if debt[i] > 0] if busy: debt[busy[0]] -= 1 tick += 1 continue slots_left, advanced = cpu_slots, 0 for i in range(len(coro)): if not alive[i] or pending[i] is None: continue if pending[i] == CPU: if slots_left <= 0: continue slots_left -= 1 resume(i, 1) advanced += 1 if advanced == 0: break tick += 1 overlap += max(0, advanced - 1) return tick, overlap, yields j = tasks(io_share=7) print("I/O bound load, eighty steps, single slot") print(f"{'yield point':<24s} {'yields':>7s} {'tick':>5s} {'overlap':>8s}") for label, factory in (("every step", every_step), ("I/O step only", io_step_only), ("never", never)): tick, overlap, yields = loop(factory, j) print(f"{label:<24s} {yields:7d} {tick:5d} {overlap:8d}") tick, overlap = run(j, 8, 1) print(f"{'previous lesson schedule':<24s} {'':>7s} {tick:5d} {overlap:8d}") async def stage(label, times, log): """Gives up control to the event loop on every iteration.""" for i in range(times): await asyncio.sleep(0) log.append(f"{label}{i}") async def sequential(): """Sequential await: the second coroutine does not start until the first finishes.""" log = [] for label in ("a", "b"): await stage(label, 3, log) return log print() log = asyncio.run(sequential()) print("real event loop, sequential await:", " ".join(log)) print(f"yield points {len(log)}, overlapping steps 0 — " f"one yields, none receive") ``` ``` I/O bound load, eighty steps, single slot yield point yields tick overlap every step 80 31 49 I/O step only 54 36 44 never 0 80 0 previous lesson schedule 31 49 real event loop, sequential await: a0 a1 a2 b0 b1 b2 yield points 6, overlapping steps 0 — one yields, none receive ``` #### Yield Frequency Decides Overlap The table changes exactly one variable across three rows: where `await` gets written. The step sequences are the same in all three, task count is the same, the slot is the same. The coroutine that **yields at every step** gives up control **80** times and finishes eighty steps in **31** ticks; overlapping steps **49**. The coroutine that **yields only at I/O steps** gives up control **54** times — exactly the I/O step count. Ticks rise to **36**, overlapping steps drop to **44**. The five-tick loss comes from CPU steps being taken back to back: if a coroutine carries three CPU steps between two I/O steps, it does those three in a single resumption, and the loop cannot hand the turn to anyone else during that stretch. The coroutine that **never yields** gives up control **0** times, and the result is **80 ticks, 0 overlapping steps** — exactly the single-thread regime itself. Being written with `async def` changes nothing. The order the three rows give: yields **80 → 54 → 0**, overlapping steps **49 → 44 → 0**. The relationship runs one direction, and the upper bound holds in every row: **overlapping steps never exceed the number of yields.** This gives a direct design consequence: what decides overlap in an asynchronous program is not how many tasks get started, it is **where the bodies stop**. The word `async` produces no yield point; `await` does. The second row's five-tick loss deserves reading on its own, because most real bodies resemble that row. The loss grows with the length of the CPU steps: the more steps between two yield points, the longer the loop stays busy. This is the countable reason long computations get moved elsewhere in asynchronous code — left in place, they make every task wait at once. #### Same Schedule, Different Source The table's last row carries a verification, and this lesson's most notable number sits there. The previous lessons' scheduler, running the same tasks with eight workers and a single slot, gave **31 ticks / 49 overlapping steps**. The coroutine regime that yields at every step gives **31 ticks / 49 overlapping steps** too. **The two numbers match exactly.** The match is not a coincidence, it is a consequence of the definition. In the thread regime, once a task advanced, the turn passed to another; the scheduler made that call. In the coroutine regime, too, a task takes one step and stops, and the turn passes to another; the **code** makes that call. The schedule observed is the same, the party that writes the schedule is different. This also settles what asynchronous programming actually earns. It does not earn **more overlap** — the top row shows this, the number is the same. What it earns is a different **source** of overlap: getting the same schedule without asking the operating system for a thread, in a single flow, by writing the yield points into the code. The cost sits in the same place: a body that forgets to write its yield points falls to the third row. #### One Yields, None Receive The last two lines show that a yield point is not enough. Two coroutines run on the real event loop; each gives up control three times, for **6** yield points total. The output is `a0 a1 a2 b0 b1 b2`. The second coroutine does not advance at all until the first finishes, and overlapping steps are **0**. The reason is what `await` does: the `sequential` body waits on `stage("a", ...)`, and until that finishes, `stage("b", ...)` is **not started** at all. Control gets given up, but there is no ready task to take it; the loop resumes the same coroutine again. This gives a two-part rule. Overlap requires **both** that the body gives up control **and** that a second task ready to advance exists at that moment. The first is provided by `await`; the second, by registering coroutines with the loop **as tasks**. If either is missing, the result is the table's third row: no matter how many yields there are, overlapping steps stay zero. This distinction also explains the most common mistake in asynchronous code: the body is written `async`, `await` is in place, and nothing overlaps. The number that needs measuring is not the count of `await`, it is the count of **tasks ready to advance at the same time**. #### The Cost of Yielding the Model Does Not Count The table's first row gives the most yields and the most overlap together; from this, "the more `await`, the better" might be concluded. The model confirms this, but the model does not count one thing, and it needs naming. Every yield is bookkeeping: the coroutine's state gets saved, put back in the loop's queue, then resumed again. In the model, this work's cost counts as **zero**; the only thing counted is which tick a step gets taken on. In a real program, yielding is not free, and a body that yields at every step pays back part of its earned overlap to that bookkeeping. Its effect on the measurement has a direction and can be read: **31** ticks is the **upper bound** on the gain that can be expected from the asynchronous regime. Adding the cost only grows this number. The **36** in the second row sits between the two costs: it writes fewer yields, but loses five ticks by taking CPU steps back to back. Bodies written in practice resemble the second row, because `await` gets written in front of waiting calls, not scattered among computing lines. What the measurement says: the difference between the second and third row — **44** overlapping steps against **0** — does not depend on the length of the computing lines, it depends on whether the waiting call in between gives up control or not. A long CPU stretch shrinks overlap somewhat; a single call that never yields zeroes it out. #### Summary - A coroutine is a body that does not run when called and advances only as it is resumed; `await` is not a wait, it is the yield point where control gets given up. - The event loop adds neither worker nor slot; it runs in a single flow. The only source of overlap is bodies giving up control, and overlapping steps can never exceed the number of yields. - The same eighty steps give 80 yields / 31 ticks / 49 overlapping steps when yielding is written at every step, 54 / 36 / 44 when written only at I/O steps, and 0 / 80 / 0 when never written. - The coroutine regime that yields at every step gives exactly the same numbers as the eight-worker, single-slot schedule; what changes is not the schedule, it is who writes it. - A yield point is necessary but not sufficient for overlap: two coroutines running with sequential await give up control 6 times and give 0 overlapping steps. #### Next Step The table's third row is this lesson's result, but the next lesson's question. A body never giving up control is not always the writer's choice: if a called function does not know how to give up control, the body waits on it, and the loop stalls. The next lesson measures this — how far overlapping steps drop when one call in the flow does not yield, what is left when no call yields, and how many steps it costs to bring a call that cannot yield back into the flow. ### Asynchronous Library Compatibility Source: https://academia.sh/en/courses/python-concurrency/asynchronous-library-compatibility One blocking call in every task drops overlapping steps from 44 to 37, and when seven of eight tasks block, overlap drops to 0 despite 6 yields; offloading the call to an executor pool restores overlap to 44, but whether the cost falls once per task or once per call splits tick count between 44 and 90. The previous lesson's table had three rows, and the third was this: a coroutine that never gives up control gives 0 yields, 80 ticks, 0 overlapping steps. It was written with async def, a real coroutine, and its result could not be told apart from the single-thread regime. That row looks like a writing mistake, but in practice it never comes from that. The person writing the body rarely forgets to write await; what forgets is the **function it calls**. A call that does not know how to give up control also stops the coroutine calling it from giving up control. What this lesson measures is that call's bill: how far overlapping steps drop when one call in the flow does not yield, what is left when no call yields, and how many steps it costs to bring a call that cannot yield back into the flow. #### Blocking Code Is Code That Produces No Yield Point **Blocking code** finishes its work without returning control to the event loop. The name can mislead: what gets blocked is not the calling body, it is the **loop itself**. The body was going to wait for the result anyway; the real loss is that while it waits, no other task can advance. Whether a call is blocking comes not from what it does, but from **how it is written**. The same work — reading a file, waiting on a reply — can be written in a form that produces a yield point, or one that does not. In an asynchronous flow, the difference between the two is exactly the difference measured in the previous lesson's table. This gives rise to this lesson's narrow question. Not every call in a program changes at once; some yield, some do not. What comes out of the mix? #### One Call Site, Eight Tasks The measurement separates two axes, because mixing them makes the result unreadable. **The first axis is call count**: how many I/O calls in each task's flow block. A task's flow has ten steps, and part of them are I/O steps; what is measured is how many of these steps pass through a call that does not yield. The point to note here: in the code, a **single call site** can be the blocking one, but eight tasks pass through that call site. A one-line defect gets written into eight flows at once. **The second axis is task count**: how many tasks use blocking calls from start to finish. When one part of a program is written asynchronously while another part stays in the old form, this axis measures it. The measurement's assumptions: - **CM48** — The task setup is the same as the previous lessons: eight tasks, ten steps per task, eighty steps total, I/O-bound load, single slot. - **CM49** — The coroutines and the loop that resumes them are the same as the previous lesson; the only thing that changes is at which step the body yields. - **CM50** — A compatible call is one that gives up control at an I/O step. A blocking call takes the same step without giving up control. **Both do the same work and spend the same step.** - **CM51** — CPU steps never yield in any regime; the only variable measured is the form of the I/O calls. - **CM52** — On the first axis, each task's first k I/O calls block; which calls get chosen is written into the setup and does not change between runs. - **CM53** — On the second axis, the first k tasks block start to finish, the rest are compatible. - **CM54** — Offloading to an executor pool is an added CPU step in the model: the work runs in the pool, the loop regains its yield point, and one more step gets counted in exchange. - **CM55** — Two placements of the offload cost are measured: once per task and once per call. Both offload the same work; only where the cost falls changes. - **CM56** — The pool's width is unbounded in this measurement; the effect of pool width is the next lesson's subject. - **CM57** — Duration is never measured; the counted unit is steps, ticks, overlapping steps, and yields. - **CM58** — The measurement is a single run, and the seed is fixed. #### The Measurement ```python """Blocking code: how much overlap one call that never yields costs.""" SEED = 20260817 CPU, IO = "cpu", "io" def make_rng(seed): state = seed % 2147483646 + 1 def draw(n): nonlocal state state = (state * 48271) % 2147483647 return state % n return draw def tasks(count=8, steps=10, io_share=7, seed=SEED): """Each task is a step sequence; io_share/10 fraction are I/O steps.""" draw, result = make_rng(seed), [] for i in range(count): result.append([IO if draw(10) < io_share else CPU for _ in range(steps)]) return result class Suspend: """The point where a coroutine hands control back to the event loop.""" def __init__(self, kind): self.kind = kind def __await__(self): return (yield self.kind) def compatible_factory(blocking_calls=0): """Each task's first 'blocking_calls' I/O calls do not yield.""" async def task(steps, log): remaining = blocking_calls for s in steps: if s == IO and remaining > 0: remaining -= 1 # blocking call: control not given up elif s == IO: await Suspend(s) # compatible call: control given up log.append(s) return task def mixed_factory(blocking_tasks=0): """The first 'blocking_tasks' tasks never yield; the rest are compatible.""" compatible = compatible_factory(0) def factory(steps, log): factory.turn += 1 if factory.turn <= blocking_tasks: return compatible_factory(len(steps))(steps, log) return compatible(steps, log) factory.turn = 0 return factory def loop(factory, jobs, cpu_slots=1): """Hand-written deterministic event loop. Returns: tick, overlap, yields.""" log = [[] for _ in jobs] coro = [factory(j, log[i]) for i, j in enumerate(jobs)] pending = [None] * len(coro) alive = [True] * len(coro) debt = [0] * len(coro) yields = 0 def resume(i, prepaid=0): nonlocal yields before = len(log[i]) try: pending[i] = coro[i].send(None) yields += 1 except StopIteration: pending[i] = None alive[i] = False debt[i] += max(0, len(log[i]) - before - prepaid) for i in range(len(coro)): resume(i) tick = overlap = 0 while any(alive) or any(debt): busy = [i for i in range(len(coro)) if debt[i] > 0] if busy: debt[busy[0]] -= 1 tick += 1 continue slots_left, advanced = cpu_slots, 0 for i in range(len(coro)): if not alive[i] or pending[i] is None: continue if pending[i] == CPU: if slots_left <= 0: continue slots_left -= 1 resume(i, 1) advanced += 1 if advanced == 0: break tick += 1 overlap += max(0, advanced - 1) return tick, overlap, yields j = tasks(io_share=7) print("eight tasks, eighty steps, I/O-bound load, single slot") print() print("A - how many I/O calls block in each task") print(f"{'blocking calls':>17s} {'yields':>7s} {'tick':>5s} {'overlap':>8s}") for k in range(9): tick, overlap, yields = loop(compatible_factory(k), j) print(f"{k:17d} {yields:7d} {tick:5d} {overlap:8d}") print() print("B - how many tasks block start to finish") print(f"{'blocking tasks':>17s} {'yields':>7s} {'tick':>5s} {'overlap':>8s}") for k in range(9): tick, overlap, yields = loop(mixed_factory(k), j) print(f"{k:17d} {yields:7d} {tick:5d} {overlap:8d}") print() print("C - offloading the blocking call to an executor pool") offload_per_task = [[CPU] + list(t) for t in j] offload_per_call = [[s for a in t for s in ([CPU, IO] if a == IO else [a])] for t in j] print(f"{'regime':<28s} {'steps':>5s} {'yields':>7s} {'tick':>5s} {'overlap':>8s}") for label, seq, factory in ( ("all block", j, compatible_factory(10)), ("all compatible", j, compatible_factory(0)), ("offload once per task", offload_per_task, compatible_factory(0)), ("offload per call", offload_per_call, compatible_factory(0))): tick, overlap, yields = loop(factory, seq) print(f"{label:<28s} {sum(len(t) for t in seq):5d} {yields:7d} " f"{tick:5d} {overlap:8d}") ``` ``` eight tasks, eighty steps, I/O-bound load, single slot A - how many I/O calls block in each task blocking calls yields tick overlap 0 54 36 44 1 46 43 37 2 38 50 30 3 30 57 23 4 22 64 16 5 14 71 9 6 8 76 4 7 5 78 2 8 2 80 0 B - how many tasks block start to finish blocking tasks yields tick overlap 0 54 36 44 1 46 44 36 2 40 50 30 3 30 58 22 4 22 64 16 5 17 69 11 6 11 75 5 7 6 80 0 8 0 80 0 C - offloading the blocking call to an executor pool regime steps yields tick overlap all block 80 0 80 0 all compatible 80 54 36 44 offload once per task 88 54 44 44 offload per call 134 54 90 44 ``` #### The Cost of a Single Call Table A's first two rows give this lesson's shortest result. With every call compatible: **36 ticks / 44 overlapping steps.** With a single I/O call blocking in every task: **43 ticks / 37 overlapping steps.** One call costs seven ticks and seven overlapping steps. What changes in the code is one line; what changes in the measurement is one-sixth of the overlap. The loss ratio is **larger** than the blocking call's share. Eight of fifty-four I/O calls block, roughly one call in seven; the loss in overlapping steps, though, is 7 of 44. The reason is that a blocking call does not stop just its own task, it stops **everyone who could have advanced at that tick**. While one task blocks, the other seven may sit ready and waiting; they do not advance. The rest of the table follows this curve to its end: **30** overlapping steps at two calls, **16** at four, **4** at six. At eight calls — that is, when every I/O call in every task blocks — overlapping steps are **0** and ticks **80**. The regime returns to single-thread. This last row defines what compatibility in this lesson's title means: an asynchronous flow's numbers are decided by its **most blocking part**. #### Seven Tasks Are Enough Table B shows the same collapse on the other axis, and one row stands apart from the rest. When **seven** tasks block, ticks are **80**, overlapping steps **0**. The eighth task is still compatible; it gives up control six times. The yields column writes this: **6 yields, 0 overlapping steps.** The previous lesson's rule gets tested exactly here. Yielding was necessary for overlap but not sufficient; a second ready task was needed too. While the other seven tasks are stuck in their own blocking calls, the eighth gives up control, and there is no one to take the control it gives up. The loop resumes the same task again. The practical reading of this is blunt: **a single compatible part does not save an asynchronous flow.** Writing one part of a codebase asynchronously may earn nothing measurable while the rest blocks. The first half of table B shows the same direction: with one task blocking, the loss is **8** overlapping steps; with two, **14**; with three, **22**. The loss accelerates with task count, because every blocking task takes away both its own overlap and others' opportunity to overlap. #### The Invisibility of the Blocker Both tables share a property that explains why this defect slips through so often: **no row has an error**. All eight tasks take every one of their steps, none stops halfway, no exception comes up. The only thing that changes is which tick a step falls on. It cannot be told apart by looking at the call site either. A compatible call and a blocking call do the same work in the measurement and spend the same step; where they part ways is **inside the body**. The only thing that says whether a function is blocking is whether calling it returns a resumable object — a function defined with `async def` returns a coroutine object, an ordinary function returns the result directly. The direct consequence: in an asynchronous flow, **every call without a written `await` is a suspect.** A call that returns its result directly either does not wait at all, or waits while holding the loop; the two cannot be told apart at the call site. What the measurement contributes here is naming the defect's **symptom**. A blocking call produces no error, does not corrupt output, writes nothing to a log. The only symptom it produces is a drop in overlapping steps — and if overlapping steps are not counted, no symptom remains at all. Table B's seventh row is this at its most extreme: the program is written asynchronously, six yield points genuinely run, and the result matches single-thread exactly. #### Offloading to a Pool and Where the Cost Falls Table C measures the fix. When a blocking call has no compatible counterpart, the call can be left as it is and run somewhere else instead: it gets **offloaded to an executor pool**, and the event loop regains its yield point meanwhile. The rows at the two ends are known numbers: if everything blocks, **80 steps, 0 yields, 80 ticks, 0 overlapping steps**; if everything is compatible, **80 steps, 54 yields, 36 ticks, 44 overlapping steps**. Offloading adds two rows, and both give overlapping steps of **44** — offloading recovers overlap exactly. Where they part ways is tick count. When the offload cost falls **once per task**, total steps rise from 80 to **88**, and ticks become **44**. Against the blocking regime's 80 ticks, that is half the gain back; against the compatible regime's 36 ticks, eight ticks behind. The cost is eight steps, the loss eight ticks; the account matches exactly. When the offload cost falls **once per call**, total steps rise to **134**, and ticks become **90**. This row holds this lesson's most notable number: **90 is larger than the 80 ticks of doing nothing and just blocking.** Even though overlapping steps rose from 0 to 44, the total got worse. Not a contradiction, arithmetic. Every offloaded call adds one step, and fifty-four calls make fifty-four steps; the forty-four overlapping steps earned are smaller than this addition. Reading overlapping steps on their own misleads for this reason — the third lesson's transfer table had the same trap. **A measurement cannot be compared without saying which steps go into the count.** The guideline that follows is concrete: offload by the **largest possible chunk**, not per call. Offloading eight tasks one by one costs eight steps; offloading every call separately costs fifty-four. The size of the work being offloaded directly decides whether the offload pays off. The order the three rows give together also suggests a priority rule. The best result is switching to a compatible call (**36** ticks); second best is offloading in large chunks (**44**); worst is offloading every call separately (**90**). A pool is not a substitute for a compatible call, it is a solution reached for when no compatible counterpart exists, and its cost is always countable. #### Summary - Blocking code is code that does not return control to the event loop; what it blocks is not the calling body, it is the loop itself. - One blocking I/O call in every task drops overlapping steps from 44 to 37 and raises ticks from 36 to 43; when every call blocks, the result is 80 ticks / 0 overlapping steps — single thread. - When seven of eight tasks block, overlapping steps are 0; even though the eighth task gives up control 6 times, there is no ready task to take the control it gives up. - Offloading the call to an executor pool restores overlapping steps to 44; when the offload cost falls once per task, ticks are 44, and once per call, 90. - Offloading per call produces more ticks than not fixing anything at all (90 against 80): when the earned overlap stays smaller than the number of added steps, the measurement flips sign. #### Next Step This lesson's last section used the pool as a fix and never asked about its width; the pool was treated as unbounded. A real pool has a width, and what sits behind it — a thread, or a process — changes the measurement. The next lesson asks this: distributing work through the same interface, what numbers does it give on the same task with a thread behind it versus a process behind it, and does the interface being the same mean the choice is the same too? ### Executor Pools Source: https://academia.sh/en/courses/python-concurrency/executor-pools The same interface finishes the same task on two backends with separate numbers: at width eight, a thread pool gives 31, 42, and 73 ticks across the three loads, while a process pool gives 10 ticks and 70 overlapping steps on all three; the interface being the same does not mean the choice is the same. The previous lesson offloaded a blocking call to an executor pool and never asked about the pool's width; the measurement treated the pool as unbounded. A real pool has a width, and what sits behind it — a thread, or a process — changes the measurement. This lesson asks that question. An **executor pool** is the common interface for distributing work: the same call form is written the same way whether a thread or a process sits behind it. What this lesson measures is the difference underneath the interface: same task, same width, two backends — what numbers come out? #### The Common Interface An executor pool gives three things. Work gets submitted, and a **result object** comes back in exchange; a function gets **mapped** over a sequence of inputs, and results return **in input order**; the pool gets closed, and submitted work runs to completion. All three operations are called with the same name and the same signature on both backends. A guarantee the interface carries also gives this lesson its measurement condition: the mapping operation's output depends on input order, not on scheduling. Even though which work runs on which worker at which moment is unspecified, the result list is deterministic. The first block below shows this. Queue, background worker, and backpressure were built on the service side in the Caching, Queues, and Asynchronous Processing course and are not repeated here. The pool here is not a service component, it is a work-distribution interface inside a single program. #### Same Interface, Two Numbers per Backend In the model's language, the difference between the two backends collapses into one place. The **thread backend** gives as many workers as the width, but stays inside a single interpreter; the **slot stays at one**. The **process backend** gives as many workers as the width, and since each is a separate interpreter, **the slot equals the width too**. The second and third lessons' results sit inside this definition. What is new is that both now sit **behind the same interface**: the caller switches from one to the other by changing a single line, and the numbers change. The measurement's assumptions: - **CM59** — The task setup is the same as the previous lessons: eight tasks, ten steps per task, eighty steps total; the three loads differ only in I/O share. - **CM60** — The thread pool's counterpart in the model is worker = width, slot = 1. - **CM61** — The process pool's counterpart in the model is worker = width, slot = width. - **CM62** — No real process is started. A real thread is used only to demonstrate the interface, and the work handed to it is pure: it gives the same result for the same input, has no side effects, and the output's order comes from the mapping operation's contract, not from the schedule. - **CM63** — The measurement never counts duration; the unit the pools are compared on is ticks and overlapping steps. - **CM64** — Tasks are independent; work submitted to the pool does not wait on other work. - **CM65** — Submitting work to the pool is itself free in this measurement. - **CM66** — The process backend's data-transfer cost is not included in this measurement; that cost was measured separately in the third lesson. The process column here is an **upper bound** for that reason. - **CM67** — The width sweep uses the same task sequence; only width changes. - **CM68** — Width never exceeds task count in any run. - **CM69** — The count in the last row is the number of distinct ticks the three loads give; it counts how many separate results come out, not their size. - **CM70** — The measurement is a single run, and the seed is fixed. #### The Measurement ```python """Executor pools: same interface, two backends, different overlap on the same task.""" from concurrent.futures import ThreadPoolExecutor SEED = 20260817 CPU, IO = "cpu", "io" def make_rng(seed): state = seed % 2147483646 + 1 def draw(n): nonlocal state state = (state * 48271) % 2147483647 return state % n return draw def tasks(count=8, steps=10, io_share=7, seed=SEED): """Each task is a step sequence; io_share/10 fraction are I/O steps.""" draw, result = make_rng(seed), [] for i in range(count): result.append([IO if draw(10) < io_share else CPU for _ in range(steps)]) return result def run(jobs, workers, cpu_slots): """Returns: tick, overlapping steps, cpu steps, io steps.""" remaining = [list(j) for j in jobs] tick = overlap = cpu_steps = io_steps = 0 while any(remaining): active = [i for i, j in enumerate(remaining) if j][:workers] if not active: break slots_left, advanced = cpu_slots, 0 for i in active: step = remaining[i][0] if step == CPU: if slots_left <= 0: continue slots_left -= 1 cpu_steps += 1 else: io_steps += 1 remaining[i].pop(0) advanced += 1 tick += 1 overlap += max(0, advanced - 1) return tick, overlap, cpu_steps, io_steps def pool(backend, width): """The interface is the same, the backend is told by two settings: worker and slot. thread : width workers, single slot (single interpreter) process : width workers, width slots (separate interpreters) """ return (width, 1) if backend == "thread" else (width, width) def square(n): """Work handed to the pool: pure, no side effects, order does not matter.""" return n * n LOADS = (("I/O bound", 7), ("balanced", 5), ("CPU bound", 1)) BACKENDS = ("thread", "process") with ThreadPoolExecutor(max_workers=4) as pool_obj: print("same interface:", list(pool_obj.map(square, range(6)))) print("map output order comes from input, not from scheduling") print() print("width eight - same task, two backends") print(f"{'load':<22s} " + " ".join(f"{label:>22s}" for label in BACKENDS)) print(f"{'':22s} " + " ".join(f"{'tick / overlap':>22s}" for _ in BACKENDS)) for label, share in LOADS: j = tasks(io_share=share) cells = [] for backend in BACKENDS: workers, slots = pool(backend, 8) tick, overlap, _, _ = run(j, workers, slots) cells.append(f"{f'{tick} / {overlap}':>22s}") print(f"{label:<22s} " + " ".join(cells)) for backend in BACKENDS: print() print(f"{backend} pool, as width grows") print(f"{'width':>9s} " + " ".join(f"{label:>22s}" for label, _ in LOADS)) print(f"{'':9s} " + " ".join(f"{'tick / overlap':>22s}" for _ in LOADS)) for width in (1, 2, 4, 8): cells = [] for label, share in LOADS: workers, slots = pool(backend, width) tick, overlap, _, _ = run(tasks(io_share=share), workers, slots) cells.append(f"{f'{tick} / {overlap}':>22s}") print(f"{width:9d} " + " ".join(cells)) print() for backend in BACKENDS: counts = {run(tasks(io_share=share), *pool(backend, 8))[0] for _, share in LOADS} print(f"{backend} pool, distinct tick counts across three loads: {len(counts)} " f"({sorted(counts)})") ``` ``` same interface: [0, 1, 4, 9, 16, 25] map output order comes from input, not from scheduling width eight - same task, two backends load thread process tick / overlap tick / overlap I/O bound 31 / 49 10 / 70 balanced 42 / 38 10 / 70 CPU bound 73 / 7 10 / 70 thread pool, as width grows width I/O bound balanced CPU bound tick / overlap tick / overlap tick / overlap 1 80 / 0 80 / 0 80 / 0 2 44 / 36 52 / 28 73 / 7 4 34 / 46 44 / 36 73 / 7 8 31 / 49 42 / 38 73 / 7 process pool, as width grows width I/O bound balanced CPU bound tick / overlap tick / overlap tick / overlap 1 80 / 0 80 / 0 80 / 0 2 40 / 40 40 / 40 40 / 40 4 20 / 60 20 / 60 20 / 60 8 10 / 70 10 / 70 10 / 70 thread pool, distinct tick counts across three loads: 3 ([31, 42, 73]) process pool, distinct tick counts across three loads: 1 ([10]) ``` #### Same Task, Two Results The first table gives this lesson's claim in three rows. The only difference between the left and right columns is the backend; the tasks are the same object, width is the same number, the scheduler is the same function. On the I/O-bound load, the thread pool gives **31 / 49**, the process pool **10 / 70**. On the balanced load, **42 / 38** against **10 / 70**. On the CPU-bound load, **73 / 7** against **10 / 70**. The left column gets worse going down, the right column does not change at all. The source of this is in the two settings' definition: the thread pool **sees** the load's CPU share, because CPU steps queue up in a single slot. In the process pool, since the slot equals task count, CPU steps stop being the constraint, and load composition becomes invisible. This gives this lesson's sentence: **behind the same interface, there is no same choice.** Changing the pool object in a codebase is a one-line job; its payoff in the measurement ranges, depending on the load, from nothing to a sevenfold difference. #### Width Earns Differently on the Two Backends The lower two tables raise width from one to eight, giving two entirely different shapes. In the **thread pool**, every column follows its own curve. The I/O-bound column drops from 80 to 44, to 34, to 31 — earning with diminishing returns. The CPU-bound column drops to **73** at the second width and stops there; widths four and eight add nothing. In this pool, what raising width pays back is exactly the load's I/O share. In the **process pool**, all three columns are the same: 80, 40, 20, 10. Every time width doubles, ticks halve, and this holds across all three loads. The reason is worker and slot growing together: at every width, every live task can advance, and the only constraint is the ten steps per task. Both tables' first row is also the same: at width one, both backends give **80 / 0**. Whatever sits behind the pool, a single-worker pool is single-thread. A comparison is unreadable if it does not state pool width — the first row shows this. The number to watch when choosing width is in these tables too: the difference between two consecutive rows. In the thread pool, for the CPU-bound column, the difference after the second row is **0**; raising width to four or eight adds nothing measurable. In the I/O-bound column, that same difference from four to eight is **3** ticks. In the process pool, the difference stays large at every step and is the same across all three columns. There is no single correct width; what there is, is stopping where the difference between two measurements shrinks. A warning sits outside these tables and is written in the assumptions: the process column does **not** count data transfer. The third lesson measured that cost separately; adding fifteen transfer steps to each task at both ends zeroed out four slots' gain. The process column here therefore gives the best reachable number, an **upper bound**. The thread column carries no such outside cost, because there is nothing to transfer within the same address space. #### The Shape of the Work Handed to the Pool The interface being the same does not mean the work that can be handed to it is the same, and the choice often gets decided here, before the numbers. **The process backend requires the work to be transferable.** The submitted function and its input get converted to a representation transferable to the other end, and the result gets transferred back. A value that cannot be transferred cannot enter the pool. This transfer, moreover, is exactly the transfer steps measured in the third lesson: when the work is small and the data is large, the gain melts away there. **The thread backend does not require transfer**, but it requires something else: the work has to **release the lock**. Work that does not release it falls into the tables' CPU-bound column, and raising width does not save it — that column stopped moving after the second width. **Both require the work to be independent.** The eight tasks in the measurement do not wait on each other; if one task's result were used by another, the waiting task would not advance in that tick, and overlapping steps would drop. The pool interface does not see this dependency, because every submitted piece of work gets submitted on its own. The measurement's numbers come out because these three conditions hold. When one of them breaks, the table shifts downward; when none breaks, the numbers above are the best reachable case. The first question to ask when choosing a pool, for this reason, is not width: **is the work itself in the shape the backend requires?** #### What the Interface Does Not Say The last two lines are this topic's closing number: **the thread pool gives 3 distinct tick counts across the three loads ([31, 42, 73]); the process pool gives 1 ([10]).** These two numbers define what a backend choice is. Switching to the thread pool means making the measurement **load-sensitive**: the result cannot be predicted without knowing which load gets handed in. Switching to the process pool — as long as transfer cost is not counted — makes the measurement **load-independent**. The interface calls both the same way, and that is a convenience. But the same call form does not promise the same behavior. What the common interface hides is exactly what this lesson measures: **which step can overlap**. Changing the pool object in a codebase with two lines and expecting the result to stay the same means counting these two numbers — 3 and 1 — as the same. Every regime in this topic has answered the same question so far: **which regime earns what?** Across five lessons, the answer came out in the same shape every time — gain depends on the number of overlappable steps and how many slots those steps can be spread across. The answer always had an input, and that input never got asked about: **which step is which type.** The setup handed us that; in a real program, nobody hands it over. #### Summary - An executor pool is the common interface for distributing work; the same call form works with both backends, and the mapping operation's output order comes from input, not from scheduling. - In the model, the thread backend means worker = width, slot = 1; the process backend means worker = width, slot = width. The entire difference sits in these two settings. - At width eight, the same task gives 31 / 49, 42 / 38, and 73 / 7 across the three loads on the thread pool; on the process pool, 10 / 70 on all three. - Raising width earns, on the thread pool, exactly the load's I/O share, and stops at the second width on the CPU-bound load; on the process pool, it halves ticks on all three loads. - The thread pool gives 3 distinct tick counts across the three loads, the process pool gives 1. Behind the same interface, there is no same choice; the process column is also an upper bound that does not count data transfer. #### Next Step What has been counted up to here was always the regime: which regime earns what, on which load. In every measurement, the tasks' step sequence was in hand — the setup stated which step was a CPU step and which was an I/O step. In a real program, that sequence is not written down. What is in hand is a codebase, and which call is heavy, which line sits on the hot path, is invisible. The next lesson closes that gap: how is it found which step of a program is heavy, without measuring duration, by **counting calls** instead? ## Performance Profiling by call count, the memory effect of object overhead and data structure choice, moving the hot path to a lower level, and that a measurement cannot be compared without declaring what it counts. ### Profiling Source: https://academia.sh/en/courses/python-concurrency/profiling A deterministic profiler gives the call count, not time columns; the same 80-step job is advanced 80 times under both loads, but the hot-path function is called 136 times under the I/O-bound load and 322 times under the CPU-bound load. The previous topic laid four regimes side by side across six lessons and counted what each one earned under which load. Its last lesson measured executor pools and left a question behind: so far **which regime** earns what has been counted — but how do I know **which step** is heavy in a program? The question is not an empty one. Choosing a regime **assumes** you already know which kind the heavy step is — a CPU step or an I/O step. If the assumption is wrong, the choice is wrong too: adding a worker to a CPU-bound load only drops 80 ticks to 73. This lesson's answer is: you do not know, you measure. And what is measured in this course is not time, it is **call count**. #### The Deterministic Counter **Profiling** is the work of recording how much a run spends in which part. There are two families of tool. A sampling profiler polls the run at regular intervals and notes where it is; what it collects is a sample, and it does not come out the same across two runs. A deterministic profiler, on the other hand, sees every call and every return; it misses nothing. The standard library's deterministic profiler collects two kinds of data: **how many times** a function was called, and **how much** time passed during those calls. This lesson's table does not carry the second, and the reason fits in one sentence: when the same code runs twice on the same machine, the time columns come out different, the call count does not — **it is the number that can be compared.** This has a consequence: we do not print the tool's ready-made dump as is, because that dump also carries time columns. Instead, the collected record is opened and only the **call count** is read from each row. This looks like a restriction; as the lesson shows by the end, the number left over is enough to answer the whole question. The record's shape supports this directly. When the run ends, the tool holds one entry per function; the entry's key is the triple that identifies the function — file, line, and name — and its value is a tuple of numbers about that function. The tuple's first two fields are call counts: the non-recursive call count and the total call count. In a program without recursion the two are equal. The remaining fields concern time and are not read in this lesson. #### The Hot Path The **hot path** is the piece of code executed most often in a run. Its name can mislead: heat here has nothing to do with what the code does, it is about **how many times** it runs. A heavy function called once is not the hot path; a three-line function called hundreds of times is. Finding the hot path by intuition means confusing the part of the code that looks long with the part that runs a lot. Length is eyeballed, call count is not eyeballed — it is counted. In this lesson the hot path is tied to a single metric: **the most-called function.** This metric alone does not decide an optimization, but it narrows the search space. Most of a program's functions get called a few times in a run; a handful get called hundreds of times. Rewriting a function from the first group is limited in its effect on the total by however many times it is called. Every step gained in a function from the second group comes back multiplied by the call count. #### The Program Being Measured We are not writing a new program for this measurement; we are measuring the shared definition's scheduler. There is one obstacle: `run` is a single function, and it produces a single row in the profiler's table. One row does not show a hot path. So the body is split into named parts: `select_active`, `is_cpu_step`, `advance`, and `run_tick`, which runs a single tick. **Behavior does not change** — the split function returns the same quadruple as the original, across all twelve combinations of four regimes and three loads. The only thing that changes is the **boundary count** the tool can see. From here comes profiling's first rule: **the tool only sees call boundaries.** If a loop body is not a separate function, it has no row in the table. It is not the tool itself but the way the code is split that decides the measurement's resolution. We could have gathered the same numbers with hand-placed counters too: put an increment line at the top of every function and print it at the end. There are two differences. A hand-placed counter requires **knowing where to look in advance**; if we knew which one was the hot path, we would not need to measure in the first place. Second, counter lines get mixed into the code itself and have to be pulled out one by one when the measurement is done. The tool, by contrast, attaches from outside without changing the code and detaches the same way; the measured program stays identical to the written program. The measurement's assumptions: - **PE1** — The task set comes from the shared definition: eight tasks, ten steps per task, **80 steps** total. The oracle is the setup itself; we know the total step count without measuring. - **PE2** — The split `run` returns the **same** quadruple as the original function. The split is not a rewrite, it is opening boundaries for measurement; the behavior of `tasks` and `run` is preserved. - **PE3** — The measured regime is the eight-worker, single-slot regime. Two loads are tried — I/O-bound and CPU-bound — and **nothing but the load changes.** - **PE4** — Only the call count is read from the record. Time fields are collected by the tool but **not written down**; had they been, they would be the one column that changed between the two runs. - **PE5** — Only this lesson's own functions are taken into the table; calls to built-ins and the standard library are not counted. - **PE6** — The tool itself adds load to the run. This load does **not change** the call count, because the counter only counts calls, it does not produce new ones. #### Measurement ```python """Profiling: same work, two loads; call count shows the hot path.""" import cProfile import pstats SEED = 20260817 CPU, IO = "cpu", "io" def make_rng(seed): state = seed % 2147483646 + 1 def draw(n): nonlocal state state = (state * 48271) % 2147483647 return state % n return draw def tasks(count=8, steps=10, io_share=7, seed=SEED): draw, result = make_rng(seed), [] for i in range(count): result.append([IO if draw(10) < io_share else CPU for _ in range(steps)]) return result def select_active(remaining, workers): return [i for i, j in enumerate(remaining) if j][:workers] def is_cpu_step(step): return step == CPU def advance(remaining, i): remaining[i].pop(0) def run_tick(remaining, workers, cpu_slots): active = select_active(remaining, workers) slots_left, advanced, cpu_steps, io_steps = cpu_slots, 0, 0, 0 for i in active: if is_cpu_step(remaining[i][0]): if slots_left <= 0: continue slots_left -= 1 cpu_steps += 1 else: io_steps += 1 advance(remaining, i) advanced += 1 return advanced, cpu_steps, io_steps def run(jobs, workers, cpu_slots): """Same result as the shared definition; body split into named parts.""" remaining = [list(j) for j in jobs] tick = overlap = cpu_steps = io_steps = 0 while any(remaining): advanced, ia, ga = run_tick(remaining, workers, cpu_slots) if not advanced: break cpu_steps += ia io_steps += ga tick += 1 overlap += max(0, advanced - 1) return tick, overlap, cpu_steps, io_steps TRACKED = ("run", "run_tick", "select_active", "is_cpu_step", "advance") def count_calls(io_share, workers=8, slot=1): """Only call count is read; time columns are not taken.""" j = tasks(io_share=io_share) profiler = cProfile.Profile() profiler.enable() result = run(j, workers, slot) profiler.disable() counts = {name: 0 for name in TRACKED} for (_, _, name), data in pstats.Stats(profiler).stats.items(): if name in counts: counts[name] = data[1] return result, counts io_result, io_counts = count_calls(7) cpu_result, cpu_counts = count_calls(1) print(f"{'function':<18s} {'I/O-bound':>22s} {'CPU-bound':>19s}") for name in TRACKED: print(f"{name:<18s} {io_counts[name]:22d} {cpu_counts[name]:19d}") print(f"{'total calls':<18s} {sum(io_counts.values()):22d} " f"{sum(cpu_counts.values()):19d}") print() print(f"I/O-bound : {io_result[0]} ticks, {io_result[1]} overlapping steps") print(f"CPU-bound : {cpu_result[0]} ticks, {cpu_result[1]} overlapping steps") print(f"advance calls in both runs: {io_counts['advance']} and " f"{cpu_counts['advance']}; oracle's total step count " f"{sum(len(x) for x in tasks())}") print("is_cpu_step calls across three consecutive runs:", [count_calls(1)[1]["is_cpu_step"] for _ in range(3)]) ``` ``` function I/O-bound CPU-bound run 1 1 run_tick 31 73 select_active 31 73 is_cpu_step 136 322 advance 80 80 total calls 279 549 I/O-bound : 31 ticks, 49 overlapping steps CPU-bound : 73 ticks, 7 overlapping steps advance calls in both runs: 80 and 80; oracle's total step count 80 is_cpu_step calls across three consecutive runs: [322, 322, 322] ``` #### The Row That Stays Equal The most important row in the table is the one whose two columns are **equal**: `advance` is called **80** times under both loads. The oracle confirms this — the setup's total step count is 80 and every step is taken exactly once. **The work did not change.** The changing rows sit around this one. `run_tick` goes from **31** to **73**, `select_active` the same way from **31** to **73**, `is_cpu_step` rises from **136** to **322**. Total calls, **279** to **549**. The same 80 steps, nearly twice the calls. From here comes the lesson's central reading: **call count does not count the work done, it counts the effort spent reaching it.** 80 steps were taken in both runs; the difference is in how many times the question had to be asked to reach those 80 steps. The row that stays equal has a second job: it verifies the measurement. `advance` coming out to 80 means a number known independently of the setup matches the profiler table. If it did not match, the thing to question would not be the table, it would be the tool or the split. **Measuring a known number too** is the cheapest way to test the measurement itself; once a row like this coming from the oracle is present in a profiler table, the remaining rows can be read with confidence. #### Which One Is the Hot Path The most-called function in both columns is `is_cpu_step`. This is the hot path, and what draws attention is that the number is **larger than 80**: 136 and 322. That means more than one call per step. The reason is in the scheduler itself. **Every** task looked at in a tick calls this function — the one that gets to advance and the one turned back for lack of a slot alike. Under the I/O-bound load, most steps do not want a slot, tasks run out early, and the total looks stay at 136. Under the CPU-bound load, the single slot turns back everyone but one task every tick; the turned-back task sits without running out, gets looked at again on the next tick, and every look costs one call. The gap between them is **186 calls.** The split tests this explanation. Under the I/O-bound run, the average look per tick is 136/31, that is **4.39**; under the CPU-bound run, 322/73, that is **4.41**. The two numbers are nearly the same. So the number of tasks looked at in a tick is barely affected by the load; what produces the difference entirely is **tick count**. The hot path gets its heat not from something inside itself but from **how many rounds it runs.** This corrects a common mistake about the hot path: the most-called function itself does not have to be flawed. `is_cpu_step` is a three-line, single-comparison function; there is nothing in it to fix. What calls it 322 times is the slot constraint outside it. **The hot path is not the cause, it is the indicator.** The `run_tick` and `select_active` rows say something separate: their values are **exactly** equal to the tick count — 31 and 73. The profiler table gave the tick count without ever counting ticks. This is not a coincidence; both functions are called exactly once per tick. The rule is: **a function's call count is the count of the event that function represents.** What the measurement counts is decided not by the tool itself but by how the code is split — and that means the decision to write a function named `run_tick` was a measurement decision. The last row shows determinism: when the same run is repeated three times, `is_cpu_step` calls come out **322, 322, 322**. Had the time columns been taken, all three would have differed. #### The Observer's Effect The deterministic tool has a cost: at every call boundary the counter kicks in, updates the record, and hands control back. A program running under a profiler is not the same as the program running without one. This is a known flaw of measurement tools, and it leaks straight into the result in a table that measures time — small, frequently called functions look heavier than they are, because the counter's own load gets added to their share. Call count is not affected by this flaw, and the reason is in the definition itself: the counter **counts** calls, it does not produce them. `is_cpu_step` is called exactly 322 times whether the counter is present or not, because what calls it is not the tool's presence but the content of the task list. This is one of the rare cases where the measurement tool does not change what it observes — and it is the second reason this course counts steps instead of time. A practical result follows from this: a program's call breakdown can be taken once under a profiler and treated as valid for subsequent runs too. The same cannot be said for a time breakdown. #### What Call Count Does Not Say The table's first row shows this limit: `run` was called only **1** time. A reading that looks at call count alone would count it as the coldest function; yet the entire run passes inside it. Calling a function once does not mean it came cheap. The second limit is coverage. Everything outside the `TRACKED` tuple — built-in calls, list operations, formatting — does not enter the table. They ran too; they were not counted. A profiler table **cannot be read without saying what it tracks**, and this lesson's table tracks five functions. The third limit is the most important one. Call count gives how many times a function ran, not what that running cost. `advance` was called 80 times; each of those 80 calls removes one item from the front of a list, and that work's cost depends on the list's length. Call count does not show this. For that, a different number is needed — how many objects get created, and how many stay alive. #### Metric Design Is Not This Lesson's Subject The Observability and Reliability course and the Observability and Operations course established metric design: which metric to collect, how to name it, which threshold triggers an alert. **Not repeated here**, a reference is enough. The difference is one of direction. Metric design chooses **beforehand**: what will be collected is decided before the run starts, and what was not collected cannot be brought back later. Profiling reads **afterward**: once the run ends, the call breakdown is opened and which function was hot comes out of it. One is an operations tool, the other a diagnostic tool; both can exist in the same program and neither substitutes for the other. #### Summary - A deterministic profiler sees every call and every return; of the two kinds of data it collects, **call count** stays the same across repeated runs, the time fields change — this is why only call count is written into the table. - The hot path is the code that is **called** the most; it is defined by call count, not length, and cannot be known without measuring. - The tool only sees call boundaries; if a body is not a separate function, it has no row in the table. The way the code is split decides the measurement's resolution. - The same 80-step job ends with **80** `advance` calls under both loads, but total calls rise from **279** to **549**: what is counted is not the work, it is the effort spent reaching it. - The hot path is `is_cpu_step`; its calls rise from **136** to **322** because the single slot leaves every task it turns back to be looked at again on the next tick. #### Next Step Call count tells how many times a function ran. There is something it does not tell: how many objects were standing at the same time **during** that run. `advance` was called 80 times under both loads, but was the structure the task lists occupy in memory the same in both runs — and if the same 80 steps were held in a different container, how many objects would exist? The next lesson looks at this question, and again it counts not bytes but **objects**. ### Memory Usage Source: https://academia.sh/en/courses/python-concurrency/memory-usage When the same 80 steps are held in six shapes, the number of objects held ranges from 1 to 172; the shared string holds 11, the separate string 89, the dict-backed object 172, the slotted object 91, and all six give the scheduler the same 31 ticks / 49 overlapping steps. The previous lesson opened a run's call breakdown and found the hot path by call count. There was one thing the breakdown did not say: `advance` was called 80 times under both loads, but how many **objects** did the structure that those 80 calls worked on hold in memory? The question is not idle. Eight tasks' ten steps can be held as a list of lists, as a class that makes every step an object, or as a single byte string. All three carry the same 80 steps and make the scheduler give the same answer. The number of objects they hold, though, is not the same — and this lesson measures that number. #### Why Objects, Not Bytes Asking how much space a structure takes in memory, in bytes, seems natural, but that number is not this course's measure. The reason is in the measure itself: an object's byte equivalent depends on how the interpreter lays the object out, and it changes from environment to environment. A changing number cannot compare two shapes. Object count, by contrast, comes from the structure itself. How many items a list has, how many items point to the same object, whether an instance also carries a dictionary — all of this can be counted by walking the structure, and it comes out the same on every run. The count has to have a limit. Starting from one object and walking everything reachable from it reaches its type, the module the type is defined in, and from there the whole program; a number like that measures the environment, not the structure. So the walk tracks containers — list, tuple, dict, instance dictionary, and slot — and stops at type objects, functions, and modules. What is counted is **the structure the data holds**, not the whole program that uses that data. The number has meaning too. Every object has its own **identity**, its own type, and its own lifetime; carrying these brings a fixed overhead per object, and this overhead is not proportional to the data inside the object. Spreading a ten-step list across eighty separate objects means paying the overhead eighty times without growing the data. **This is why the memory claim is built on object count.** #### Creating Versus Holding The Data Structures and Functional Tools course counted the **created** object: with a class that counts its own instantiation, it measured how many objects a shape produces over the course of a run, and showed that the lazy shape does not reduce production, only holding. **This measurement is not repeated here**, it is used by reference. The question here is the other half. Once the run has ended, how many objects are **standing at the same time** inside the structure that is kept? Creating is a flow measure, holding is a state measure. Two shapes can create the same number of objects and hold wildly different numbers; two shapes can hold the same number of objects and have created wildly different numbers. #### Six Shapes The measured structures are derived from the same eight tasks the shared definition produces, and all of them carry the same 80 steps. **List, shared string** is the shared definition's own shape: steps point to two immutable strings, and all 80 slots bind to one of those two objects. **List, separate string** is the same structure, the only difference being that each step's string is rebuilt at run time — the values are equal, the objects are separate. **Dict, shared string** turns the outer list into a dict keyed by task number. **Object, dict-backed** wraps every step in a class instance; the instance holds its attributes in a dictionary. **Object, slotted** builds the same class with a `__slots__` declaration; because attribute names are fixed in advance, the instance does not carry a dictionary too. **Single bytes object** gathers the 80 steps into one immutable bytes object; task boundaries are recovered with arithmetic. The measurement's assumptions: - **PE7** — All six shapes are derived from the shared definition's `tasks` output and carry the same 80 steps; none of them changes the data. - **PE8** — The count walks the structure and counts **distinct** objects: a shared object gets counted **once** no matter how many places point to it. The distinction is made by identity, but **no identity number is printed.** - **PE9** — The walk tracks lists, tuples, dicts, instance dictionaries, and slots. Type objects, functions, and modules are not counted; what is counted is **the structure itself**, not the whole run. - **PE10** — What is measured is the object **held**. Intermediate objects created while the structure is being built do not enter the count; those belong to the Data Structures course's measure. - **PE11** — The measured byte count is **not written**. A structure's byte equivalent depends on the environment; object count does not, and it is object count that compares the two shapes. - **PE12** — Each shape is resolved to the common form before it is handed to the scheduler, and the table shows whether the resolved list is the same as the shared definition's; if it is not, the comparison is invalid. #### Measurement ```python """Memory: same 80 steps in six shapes; bytes are not counted, objects held are.""" SEED = 20260817 CPU, IO = "cpu", "io" def make_rng(seed): state = seed % 2147483646 + 1 def draw(n): nonlocal state state = (state * 48271) % 2147483647 return state % n return draw def tasks(count=8, steps=10, io_share=7, seed=SEED): draw, result = make_rng(seed), [] for i in range(count): result.append([IO if draw(10) < io_share else CPU for _ in range(steps)]) return result def run(jobs, workers, cpu_slots): remaining = [list(j) for j in jobs] tick = overlap = cpu_steps = io_steps = 0 while any(remaining): active = [i for i, j in enumerate(remaining) if j][:workers] if not active: break slots_left, advanced = cpu_slots, 0 for i in active: step = remaining[i][0] if step == CPU: if slots_left <= 0: continue slots_left -= 1 cpu_steps += 1 else: io_steps += 1 remaining[i].pop(0) advanced += 1 tick += 1 overlap += max(0, advanced - 1) return tick, overlap, cpu_steps, io_steps def children(n): if isinstance(n, (list, tuple)): return list(n) if isinstance(n, dict): return list(n.keys()) + list(n.values()) if hasattr(n, "__dict__"): return [n.__dict__] if hasattr(type(n), "__slots__"): return [getattr(n, a) for a in type(n).__slots__] return [] def count_objects(root): """Walks the structure; a shared object is counted once. No identity is printed.""" seen, held, stack, kinds = set(), [], [root], {} while stack: n = stack.pop() if id(n) in seen: continue seen.add(id(n)) held.append(n) name = type(n).__name__ kinds[name] = kinds.get(name, 0) + 1 stack.extend(children(n)) return len(seen), kinds def separate_string(s): return "".join([s[:2], s[2:]]) class Step: def __init__(self, kind): self.kind = kind class SlimStep: __slots__ = ("kind",) def __init__(self, kind): self.kind = kind G = tasks() SHAPES = { "list, shared string": ([list(g) for g in G], lambda y: [list(g) for g in y]), "list, separate string": ([[separate_string(a) for a in g] for g in G], lambda y: [list(g) for g in y]), "dict, shared string": ({i: list(g) for i, g in enumerate(G)}, lambda y: [y[i] for i in sorted(y)]), "object, dict-backed": ([[Step(a) for a in g] for g in G], lambda y: [[a.kind for a in g] for g in y]), "object, slotted": ([[SlimStep(a) for a in g] for g in G], lambda y: [[a.kind for a in g] for g in y]), "single bytes object": (bytes(1 if a == CPU else 0 for g in G for a in g), lambda y: [[CPU if b else IO for b in y[i * 10:i * 10 + 10]] for i in range(8)]), } print(f"{'shape':<26s} {'objects':>7s} {'resolved list same':>19s} {'tick / overlap':>14s}") for name, (structure, resolve) in SHAPES.items(): count, _ = count_objects(structure) d = resolve(structure) tick, overlap, _, _ = run(d, 8, 1) print(f"{name:<26s} {count:7d} {str(d == G):>19s} {f'{tick} / {overlap}':>14s}") print() for name in ("list, shared string", "object, dict-backed", "object, slotted"): print(f"{name:<26s} {count_objects(SHAPES[name][0])[1]}") print() p = SHAPES["list, shared string"][0] a = SHAPES["list, separate string"][0] print(f"in shared shape, first step same object as the setup's -> {p[0][0] is G[0][0]}") print(f"in separate shape, same -> {a[0][0] is G[0][0]}, equal -> {a[0][0] == G[0][0]}") b = SHAPES["single bytes object"][0] print(f"two slices taken from the bytes object are the same object -> {b[0:10] is b[0:10]}, " f"equal -> {b[0:10] == b[0:10]}") print(f"total steps {sum(len(g) for g in G)}, fewest-object shape " f"{min(count_objects(y)[0] for y, _ in SHAPES.values())}, most " f"{max(count_objects(y)[0] for y, _ in SHAPES.values())}") ``` ``` shape objects resolved list same tick / overlap list, shared string 11 True 31 / 49 list, separate string 89 True 31 / 49 dict, shared string 19 True 31 / 49 object, dict-backed 172 True 31 / 49 object, slotted 91 True 31 / 49 single bytes object 1 True 31 / 49 list, shared string {'list': 9, 'str': 2} object, dict-backed {'list': 9, 'Step': 80, 'dict': 80, 'str': 3} object, slotted {'list': 9, 'SlimStep': 80, 'str': 2} in shared shape, first step same object as the setup's -> True in separate shape, same -> False, equal -> True two slices taken from the bytes object are the same object -> False, equal -> True total steps 80, fewest-object shape 1, most 172 ``` #### The Result Did Not Change The table's right two columns are the measurement's validity condition. All six shapes, when resolved, come out **identical** to the shared definition's list, and all six make the scheduler give the same answer: **31 ticks / 49 overlapping steps.** This defines what the difference in the left column is. Object count ranges from 1 to 172 while the outcome of the work never changes. What is measured is **the price of holding the same result**, not the result itself. Without this distinction, the table would not be a comparison, it would be six separate measurements. The column's presence in the table is not a formality. The most common mistake in memory measurements is silently changing the data while changing the shape — dropping a field, shortening a string, building a task incompletely. In such a measurement, object count does genuinely drop, but the drop's cause is not the shape, it is the missing data. The `resolved list same` column tests this on every row; a row that was not **True** would be removed from the table. #### What Sharing Costs The first two rows are separated by a single difference. The structure is the same — one outer list, eight inner lists, eighty slots. The values are the same too; the `equal -> True` in the block below says so. The only thing that differs is the objects: in the shared shape, 80 slots point to **two** string objects; in the separate shape, to **eighty**. The count rises from **11** to **89**: **78 objects extra**, for zero information in return. The identity test in the block below shows why — in the first shape, the step is the very object the setup produced; in the second, an equal but separate object. This is the most common memory pattern found in real programs. Fields read from a file or a message are equal to each other even if they are separate objects; each one gets rebuilt during reading. Rebuilding sharing in a column where the same value repeats — producing a small set of immutable values once and pointing to them everywhere — drops object count directly. #### Who Pays the Per-Object Overhead The fourth and fifth rows are two constructions of the same class. Each step is an instance; the only difference is where the attributes are held. The dict-backed construction holds **172** objects, the slotted construction **91**. Where the **81** in between comes from is shown by the type breakdown: in the dict-backed shape, the `dict` count is **80** — one dictionary per instance — and an extra string for the attribute name is also counted, which is why `str` is **3** instead of 2. In the slotted shape, there is no `dict` row at all. When attribute names are declared in advance, the instance does not need to carry a dictionary too. The gain grows multiplied by the instance count: here eighty instances are freed from eighty dictionaries. The price is flexibility — an undeclared attribute cannot be added later. The third row shows another face of the same overhead. Turning the outer container from a list to a dict raises object count from **11** to **19**. The eight extra objects are the eight **keys** themselves: a list position is not an object, a dict key is. Access is the same, the result is the same, eight objects extra. #### The Price of a Single Object The last row holds 80 steps in a **single** object. This is the lowest number in the table, and the reason is clear: the 80 values inside the bytes object are not separate objects, they are the contents of a single object. Structure boundaries — which ten steps belong to which task — are recovered with arithmetic. The price sits in the last identity test in the block below: taking the same slice twice produces **two separate objects**. Reading a piece out of an immutable bytes object means building a new object on every read. So a structure that stands as one object **produces** objects as it is **read**; the number held drops while the number created rises. The two measures being separate is useful here. The question to ask when choosing a shape is not one: how much is held, and what does accessing what is held produce each time? The bytes object is the best choice for the first, the worst for the second; the shared-string list is above the middle on both and holds 80 steps directly accessible with **11** objects. The six rows give not a ranking but an axis. At one end, all of the structure's information sits in objects and access is free; at the other end, structure information has moved into code, the object count has dropped to one, and the price of access is spread across every read. The measurement does not say which end is right — it says what each end costs. What decides is how many times the same structure will be looked at. #### Summary - This course measures memory load not by **measured bytes** but by **object count held**; byte equivalents depend on the environment, object count comes from the structure itself and comes out the same on every run. - The same 80 steps can be held in six shapes ranging from **1** to **172** objects, and all six make the scheduler give the same **31 ticks / 49 overlapping steps.** - Losing sharing, without changing the structure, raises **11** objects to **89**: eighty equal but separate strings, for zero information, are **78** extra objects. - Holding a dictionary per instance means **172**, declaring attribute names in advance means **91** objects; the **81** in between are eighty instance dictionaries and one attribute name. - Turning the outer container from a list to a dict adds eight key objects; a list position is not an object, a dict key is. - A structure held in a single object drops what is held to **1** but produces a new object on every read; the holding measure and the producing measure can pull in different directions. #### Next Step Two numbers have been measured so far: how many times a function is called, and how many objects a structure holds. Both point the same direction — the hot path is where the `is_cpu_step` call runs 322 times. Can that path be brought down further? Moving a hot path out of the language, to a lower level, is a commonly used route, and its gain is usually told in terms of time. The next lesson measures this not in time but **as steps in a model**: when the hot path's step count drops, what happens to the total tick count, and how much does crossing the boundary itself take back? ### Native Extensions Source: https://academia.sh/en/courses/python-concurrency/native-extensions Moving the hot path to a lower level is measured in the model with three effects; under the CPU-bound load, twelve groups of four drop 80 steps to 44 and 73 ticks to 25, boundary cost pulls the gain from 48 down to 36, and the same mechanism earns nothing under the I/O-bound load. The previous two lessons framed the hot path with two separate numbers: under the CPU-bound load, `is_cpu_step` is called 322 times, and depending on the shape holding the 80-step structure, it occupies between 1 and 172 objects. Both point to the same place — the work's weight is in the CPU steps. At this point there is a commonly used route: move the hot path out of the language, to a lower level. Its gain is usually told in terms of time, and that telling is not accepted in this course. This lesson asks the same question in a single form: **when the hot path's step count drops, what happens to total ticks?** #### What Is a Native Extension A **native extension** is a component that holds code already translated into machine instructions rather than having the interpreter execute it step by step, and that appears to the language like a module. The rest of the program uses it with an ordinary function call; the call crosses a **boundary**, arguments are converted to the form the lower level expects, the work is done there, and the result is converted back. This lesson does not measure how to write an extension, it measures **what it earns**. No tool, compiler, or package is named; what is measured is the approach itself, not a particular mechanism that realizes it. Two properties defining the approach are enough: **the work is done outside the language's execution steps**, and **every entry and exit crosses a boundary.** Whatever tool provides these two properties, the measurement stays the same; the next table is the direct counterpart of these two properties. A real extension is not compiled either, and the reason is the same as the course's own measure. A compiled extension's gain depends on the environment, the compilation options, and the kind of work; it gives two different numbers on two machines. The step measured in the model, though, comes out the same on every run. **A number that cannot be reproduced cannot carry a gain claim.** #### Three Effects in the Model The native extension enters the model with three separate effects, and measuring the three separately is this lesson's real work. **The first is step count.** Several consecutive CPU steps on the hot path are handed off to the extension with a single call; the step count the interpreter has to see drops. Four steps collapsing into one call means one step in place of those four. **The second is boundary cost.** The call does not cross for free: converting arguments, transferring data to the other side of the boundary, and building the return value all do work. In the model this is a fixed number of extra steps per call. **The third is the lock.** Code running at the lower level can release the global interpreter lock while it does its work. Its counterpart in the scheduler is direct: the step that releases the lock does not want a CPU slot, so it can **overlap** with other tasks' steps. This is not new behavior; the shared definition's scheduler already treats every step that does not want a slot this way. The three effects can be turned on and off independently, and the table does exactly that: in some rows only the step count drops, in some boundary cost is added, in some the lock is released. What the model leaves out also has to be said, because a model cannot be read without saying what it does not cover. Boundary cost here is fixed per call; in reality it can depend on the size of the data transferred, and a fixed number misleads on a call that copies large data. Which side of the boundary owns the memory, how an error at the lower level is carried back into the language, and the extension's portability are not in the model either. These are not part of the gain question, they are part of the **cost question**; the table only measures gain and keeps this limit explicit. The measurement's assumptions: - **PE13** — In the model, the native extension is represented only by these three effects: step merging, boundary cost, releasing the lock. No other effect is modeled. - **PE14** — Only **consecutive and complete** groups merge. Steps that cannot fill a group keep being interpreted; an extension only takes over where it can take over the whole hot path. - **PE15** — Boundary cost is a fixed, whole number of steps per call. Argument conversion, data transfer, and building the return value are all rolled into this one number. - **PE16** — A step releasing the lock is not a CPU step for the scheduler: it does not want a slot and can overlap. The behavior of `tasks` and `run` is **not changed**; only a new step kind is added. - **PE17** — A real extension is not compiled, time is not measured. Gain is written only as **steps and ticks.** - **PE18** — The comparison baseline is the same task set's uncondensed run. The two loads are measured separately and nothing changes but the load; both runs are in the eight-worker, single-slot regime. #### Measurement ```python """Native extension model: when the hot path shortens, what happens to total tick.""" SEED = 20260817 CPU, IO, NATIVE = "cpu", "io", "native" def make_rng(seed): state = seed % 2147483646 + 1 def draw(n): nonlocal state state = (state * 48271) % 2147483647 return state % n return draw def tasks(count=8, steps=10, io_share=7, seed=SEED): draw, result = make_rng(seed), [] for i in range(count): result.append([IO if draw(10) < io_share else CPU for _ in range(steps)]) return result def run(jobs, workers, cpu_slots): remaining = [list(j) for j in jobs] tick = overlap = cpu_steps = io_steps = 0 while any(remaining): active = [i for i, j in enumerate(remaining) if j][:workers] if not active: break slots_left, advanced = cpu_slots, 0 for i in active: step = remaining[i][0] if step == CPU: if slots_left <= 0: continue slots_left -= 1 cpu_steps += 1 else: io_steps += 1 remaining[i].pop(0) advanced += 1 tick += 1 overlap += max(0, advanced - 1) return tick, overlap, cpu_steps, io_steps def condense(job, group_size, boundary=0, kind=CPU): """Reduces group_size consecutive CPU steps to one step; adds boundary steps.""" result, accumulated = [], 0 for step in job: if step == CPU: accumulated += 1 if accumulated == group_size: result.append(kind) result.extend([CPU] * boundary) accumulated = 0 else: result.extend([CPU] * accumulated) accumulated = 0 result.append(step) result.extend([CPU] * accumulated) return result REGIMES = ( ("baseline: every step interpreted", 1, 0, CPU), ("four steps in one call", 4, 0, CPU), ("four steps, boundary 1 step", 4, 1, CPU), ("four steps, lock released", 4, 0, NATIVE), ("four steps, lock + boundary 1", 4, 1, NATIVE), ("two steps, lock + boundary 1", 2, 1, NATIVE), ("ten steps, lock released", 10, 0, NATIVE), ) for load, share in (("CPU-bound", 1), ("I/O-bound", 7)): G = tasks(io_share=share) baseline = run(G, 8, 1)[0] print(f"{load:<30s} {'steps':>5s} {'tick':>4s} {'overlap':>8s} {'gain':>7s}") for name, group_size, boundary, kind in REGIMES: Y = [condense(g, group_size, boundary, kind) for g in G] tick, overlap, _, _ = run(Y, 8, 1) print(f" {name:<28s} {sum(len(g) for g in Y):5d} {tick:4d} {overlap:8d}" f" {baseline - tick:7d}") print() G = tasks(io_share=1) D = [condense(g, 4, 0, NATIVE) for g in G] print(f"under the CPU-bound load, steps before condensing {sum(len(g) for g in G)}, " f"after {sum(len(g) for g in D)}") print(f"groups of four reduced to one call {sum(g.count(NATIVE) for g in D)}, " f"cpu steps still interpreted {sum(g.count(CPU) for g in D)}") H = tasks(io_share=7) E = [condense(g, 4, 0, NATIVE) for g in H] print(f"under the I/O-bound load, groups of four {sum(g.count(NATIVE) for g in E)}, " f"steps {sum(len(g) for g in H)} -> {sum(len(g) for g in E)}") ``` ``` CPU-bound steps tick overlap gain baseline: every step interpreted 80 73 7 0 four steps in one call 44 37 7 36 four steps, boundary 1 step 56 49 7 24 four steps, lock released 44 25 19 48 four steps, lock + boundary 1 56 37 19 36 two steps, lock + boundary 1 80 44 36 29 ten steps, lock released 53 44 9 29 I/O-bound steps tick overlap gain baseline: every step interpreted 80 31 49 0 four steps in one call 80 31 49 0 four steps, boundary 1 step 80 31 49 0 four steps, lock released 80 31 49 0 four steps, lock + boundary 1 80 31 49 0 two steps, lock + boundary 1 80 24 56 7 ten steps, lock released 80 31 49 0 under the CPU-bound load, steps before condensing 80, after 44 groups of four reduced to one call 12, cpu steps still interpreted 25 under the I/O-bound load, groups of four 0, steps 80 -> 80 ``` #### The Drop in Steps Under the CPU-bound load, the baseline is **80 steps, 73 ticks**. Reducing groups of four to one call pulls steps to **44**, ticks to **37**; gain **36 ticks**. The block below shows where this comes from: **12** groups of four merged, 48 steps reduced to 12 steps, leaving **25** CPU steps still to be interpreted. What draws attention is that the overlap column stays at **7**. Step count dropped, ticks dropped, but overlap never rose. The reason is the single slot: a merged step is still a CPU step, and it still wants a slot. **Less work was done, but the work was still done in sequence.** When the lock is released, the table changes: the same 44 steps finish in **25 ticks**, and overlap rises from **7** to **19**. Gain from **36** to **48**. The 12 extra ticks come entirely from those 12 merged steps ceasing to ask for a slot. **The same step reduction gives two-thirds of the gain without releasing the lock, and the full gain with it.** #### What the Boundary Costs Boundary cost sits in the table as two rows and is easy to read. Without the lock, a one-step-per-call boundary pushes steps from **44** to **56** — 12 calls, 12 extra steps. Ticks from **37** to **49**, gain drops from **36** to **24**. The boundary took back **a third** of the gain. With the lock, the same boundary pulls gain from **48** down to **36**. This number is interesting: **36** is exactly equal to the gain of the row where the lock is never released but the boundary is never paid either. So boundary cost eats exactly the gain that releasing the lock brought. Two different mechanisms, the same total. If a measurement showed only the gain number for these two rows, no difference between them would show at all. This is the most fragile point of an extension decision: **gain grows with group size, boundary cost grows with call count.** Small groups mean many calls, and many calls mean many boundaries. #### Gain Without a Step Drop The sixth row shows this at the extreme. Reducing two steps to one call and paying a one-step boundary per call keeps total steps at **80** — two steps left, one call and one boundary step arrived; the arithmetic came back to where it started. Yet ticks drop from **73** to **44** and overlap rises from **7** to **36**. The work done never shrank at all; the gain is **29 ticks**. All of it comes from releasing the lock. This row is this lesson's share of the course's measurement axis: **an optimization's gain does not have to come from reducing the work done; it can also come from letting that work overlap.** #### The Limit of Growing the Group The last row is where intuition runs backward. When the group is made ten instead of four — handing more steps off to a single call, expecting a bigger gain — steps come out to **53** instead of **44**, ticks to **44** instead of **25**. The reason is in the shared definition's setup. In ten-step tasks, exactly ten consecutive CPU steps arriving in a row is rare; I/O steps cut in and break up the group. Steps that cannot fill the group do not merge, they are interpreted one by one. **When group size exceeds the hot path's real length, gain does not grow, it shrinks.** Placing this row next to the one above it teaches a second thing. The group-of-ten gives **44** ticks at **53** steps; the group-of-four, together with boundary cost, gives **37** ticks at **56** steps. So the regime with **fewer** steps spends **more** ticks. Total step count alone does not decide tick count; what decides is **which kind** those steps are. In the group-of-ten, only a few steps can release the lock, so overlap stays at **9**; in the group-of-four, it rises to **19**. **Counting steps is not enough; you have to count which steps can overlap too.** #### If the Hot Path Is Not There The second table is this lesson's harshest result. Under the I/O-bound load, five of the seven rows are **exactly identical** to the baseline: 80 steps, 31 ticks, 49 overlapping steps, gain **0**. The block below gives the reason in a single number: under this load, the count of groups of four consecutive CPU steps is **0**. There is nothing to merge. The extension was written, the boundary was built, the lock was released — and the table never moved. The one row with a gain is the pair-group: **31** to **24** ticks, gain **7**. Under this load, a run of two consecutive CPU steps can be found, and again the gain comes not from steps but from overlap; steps stay at **80**. The conclusion is this: **moving the hot path to a lower level only earns something if the hot path is actually there.** Both tables measure the same mechanism, the same group size, and the same boundary cost; one earns 48 ticks, the other none. The difference does not come from the code, it comes from **the composition of the load** — and the way to know that composition is to measure, as the previous two lessons did. The first lesson's call breakdown already showed this distinction. `is_cpu_step` was called **136** times under the I/O-bound load, **322** under the CPU-bound load; in the first, CPU steps did not pile up at the slot, in the second they did. The information needed for the extension decision was sitting there already. Moving to an extension without profiling first is the same thing as writing the second table's five rows: work done, measured gain zero. #### Summary - A native extension is represented in the model with three effects: the hot path's step count dropping, a fixed boundary cost per call, and a step ceasing to want a slot because the lock is released. - Under the CPU-bound load, **12** groups of four drop 80 steps to **44** and 73 ticks to **37**; if the lock is also released, ticks drop to **25** and overlap rises from **7** to **19**. - A one-step-per-call boundary cost pulls gain from **48** down to **36** — eating the entire gain that releasing the lock brought. Gain grows with group size, cost grows with call count. - Gain can happen without total steps ever dropping: at pair groups, steps stay at **80** while ticks drop from **73** to **44**, and all the gain comes from overlap. - Gain shrinks when group size exceeds the hot path's real length: the group-of-ten gives **44** ticks while the group-of-four gives **25**. - The same mechanism earns nothing under the I/O-bound load, because the count of consecutive groups of four is **0** under that load. #### Next Step This lesson did the same thing across a table: on every row, what was counted was written into the row's name — how many steps merged, whether a boundary was paid, whether the lock was released. If the names were stripped away, only numbers would remain, and **37** would sit next to **37** and claim to be measuring the same thing; yet one was the locked regime paying a boundary, the other the unlocked regime not paying one. The course's last lesson measures this danger on itself: same work, same regime, the only difference is what is included in the measurement — and how much difference that makes between two numbers. ### Measurement Discipline Source: https://academia.sh/en/courses/python-concurrency/measurement-discipline The same run gives six separate numbers between 26 and 80 across six separate declarations; excluding setup gives 31 ticks, including it gives 37, and this 19.35% difference flips the ranking of two regimes when it is counted in only one of them. The previous lesson wrote down what each row measured in the row's own name across a table, and two rows gave the same number: **37**. One was the regime that pays boundary cost but releases the lock, the other pays no boundary but does not release the lock either. Erase the names and two equal numbers remain, and it would be assumed the two measure the same thing. This course's last lesson measures exactly this danger. Its question is: **can a measurement be compared without saying what it counts?** The answer will both be shown and applied to the course's own method. #### What a Benchmark Requires A **benchmark** is a measurement that puts two options side by side on the same work. Even once the "same work" condition is met, a comparison is not automatically valid; three more declarations are required. **The unit's declaration:** what is being counted — ticks, steps, calls, objects? Every measurement in this course wrote its unit by name, and there was a reason for that. **The regime's declaration:** how many workers, how many slots, which load? The same setup gives 31 under eight workers and a single slot, 80 under a single flow; without declaring the regime, the number says nothing. **The boundary's declaration:** where does the measurement begin and where does it end? Which part of the work is included in the count? This third one is the most often skipped, and it is what this lesson measures. Above all three stands a fourth: the **oracle.** What says the measurement gives the correct result? In this course, the oracle is the setup itself — we produced the tasks' step sequence ourselves, so we know the total step count is 80 without measuring, and every table can be tested against that number. A measurement without an oracle can tell a big number from a small one, but it cannot tell right from wrong; when an optimization breaks the result, the table still looks fine. #### Is Setup Counted A piece of work has a price paid before it starts: setting up the execution context, distributing tasks, preparation that cannot be skipped on the first round. In the model this is not a duration, it is a **fixed number of steps**, and it is called setup. Whether setup is counted is not a true-or-false question. In a long-running service, setup is paid once and becomes negligible next to the total; in a short-lived job, it is paid again on every run and is a direct cost. **Both declarations are defensible.** What is not defensible is giving the number without writing down which one was done. The measurement's assumptions: - **PE19** — The measured work is the same in every row: the shared definition's eight tasks, total **80 steps**, the I/O-bound load. The only thing that changes is **the declaration of what is counted.** - **PE20** — Setup is not a duration, it is the **fixed step count** that cannot be skipped on the first round. - **PE21** — Setup is **declared** as **6** steps for the thread regime, **25** steps for the process regime. These numbers are not measured, they are the model's assumption; they rest on a separate execution context requiring more steps to set up, and the table reads them that way. - **PE22** — All six declarations are read from the same run, and **none of them is wrong.** What is wrong is writing the number without saying which one was read. - **PE23** — The oracle is the setup: total steps are **80**, and stay 80 no matter which regime is chosen. The regime does not change the step count, it changes how many ticks the steps spread across. - **PE24** — No row measures time. Even the measurement including setup is not a time measurement, it is a step count; the lesson's claim is not about time, it is about **counting boundary.** #### Measurement ```python """Measurement discipline: same run, different declarations; which number compares.""" SEED = 20260817 CPU, IO = "cpu", "io" def make_rng(seed): state = seed % 2147483646 + 1 def draw(n): nonlocal state state = (state * 48271) % 2147483647 return state % n return draw def tasks(count=8, steps=10, io_share=7, seed=SEED): draw, result = make_rng(seed), [] for i in range(count): result.append([IO if draw(10) < io_share else CPU for _ in range(steps)]) return result def run(jobs, workers, cpu_slots): remaining = [list(j) for j in jobs] tick = overlap = cpu_steps = io_steps = 0 while any(remaining): active = [i for i, j in enumerate(remaining) if j][:workers] if not active: break slots_left, advanced = cpu_slots, 0 for i in active: step = remaining[i][0] if step == CPU: if slots_left <= 0: continue slots_left -= 1 cpu_steps += 1 else: io_steps += 1 remaining[i].pop(0) advanced += 1 tick += 1 overlap += max(0, advanced - 1) return tick, overlap, cpu_steps, io_steps G = tasks(io_share=7) TICK, OVERLAP, CPU_STEPS, IO_STEPS = run(G, 8, 1) SETUP = {"thread (single slot)": 6, "process (four slots)": 25} print("same run, six declarations") for name, count in (("excluding setup, tick", TICK), ("including setup, tick", TICK + SETUP["thread (single slot)"]), ("excluding setup, cpu step", CPU_STEPS), ("excluding setup, io step", IO_STEPS), ("overlapping step", OVERLAP), ("total step", sum(len(g) for g in G))): print(f" {name:<34s} {count:4d}") k = SETUP["thread (single slot)"] print(f"\nwithout counting setup {TICK} ticks, counting it {TICK + k} ticks; " f"diff {k} ticks ({k / TICK:.4f})") THREAD = run(G, 8, 1)[0] PROCESS = run(G, 8, 4)[0] ti, pi = SETUP["thread (single slot)"], SETUP["process (four slots)"] print(f"\n{'setup declaration':<28s} {'thread':>13s} {'process':>12s} " f"{'winner':>14s}") for name, a, b in (("not counted in either", THREAD, PROCESS), ("counted in both", THREAD + ti, PROCESS + pi), ("counted only in process", THREAD, PROCESS + pi), ("counted only in thread", THREAD + ti, PROCESS)): winner = "thread" if a < b else ("process" if b < a else "equal") print(f" {name:<26s} {a:13d} {b:12d} {winner:>14s}") print(f"\nsetup declared in the model: thread {ti} steps, process {pi} steps") ``` ``` same run, six declarations excluding setup, tick 31 including setup, tick 37 excluding setup, cpu step 26 excluding setup, io step 54 overlapping step 49 total step 80 without counting setup 31 ticks, counting it 37 ticks; diff 6 ticks (0.1935) setup declaration thread process winner not counted in either 31 10 process counted in both 37 35 process counted only in process 31 35 thread counted only in thread 37 10 process setup declared in the model: thread 6 steps, process 25 steps ``` #### One Run, Six Numbers The top table's six rows are read from **a single run**. The tasks were produced once, the schedule ran once; the only thing that changes is which question is put to the result. The numbers that come out are **26, 31, 37, 49, 54, 80**. All six are correct. Whoever says "this work took 26 units" and whoever says "80 units" are both talking about the same run. The ratio between them is more than threefold, and all of that difference comes not from the measurement but from **the declaration**. From this comes the lesson's rule: **a bare number is not a measurement.** A measurement is the sum of the number and the declaration that says what the number counts. Without the declaration, what is left cannot be put into a comparison — because the other side's row is unknown. #### The Third Claim: 31 Against 37 The middle row carries the course's third claim on its own. The same task set, the same regime, the same scheduler: excluding setup gives **31 ticks**, including it gives **37**. Difference **6 ticks**, ratio **19.35%**. Nothing changed. The code is the same, the data is the same, the slot count is the same, the worker count is the same. Yet two numbers, with a difference between them close to a fifth. This is the most ordinary difference that could show up in a performance report. If an optimization were said to give a 19% gain, it would be taken seriously. Yet behind this 19.35% here there is no optimization at all — only two different counting boundaries. #### The Ranking Flipping The bottom table shows how the same flaw leaks into a comparison. Two regimes side by side: eight workers with a single slot, and eight workers with four slots. The first two rows are honest declarations. When setup is not counted on either side, it is **31** against **10**; when it is counted on both, **37** against **35**. In both, the winner is **process**. Counting setup drops the gap between them from 21 ticks to 2 ticks — the declaration does not change the result, but it changes **the size of the gap**. The third row is not honest, and it is the only row in the table where the winner changes. When setup is counted only for the process, it comes out **31** against **35** and **thread** wins. The regime that actually loses appears to win, because the measurement's boundary was widened on one side only. A table like this lies in none of its rows: 31 is correct, 35 is correct. What is wrong is putting the two **side by side**. Two numbers can be compared only if they were produced with the same boundary. The fourth row tries the reverse and the ranking does not flip: even if setup is counted only for the thread, the process still wins by **10**. So a declaration flaw does not always flip the result — **whether it does can only be known with the correct declaration.** The only way to notice the flaw is to have also done the correct measurement. #### The Course Applied to Itself This lesson has to justify a decision applied throughout the course: why was time never written anywhere? The answer is the same table above, one step further. If a number's declaration is missing, it cannot be compared; and if a number additionally **cannot be reproduced**, a declaration does not save it either. Time gives two separate values across two runs, even for the same code on the same machine. Even if you declare exactly which boundary that value was measured with, the second run changes the number. Step, tick, call, and object do not carry this flaw. In the profiling lesson, the same run was repeated three times and call count came out **322, 322, 322**. In the memory lesson, six shapes gave the same object count on every run. Overlapping step is computed from the setup, and the setup is fixed by the seed. This is not a claim of superiority, it is a scope decision. Time measurement has its place, and in a real system it eventually gets measured too; but **a number used to teach a concept has to be reproducible.** This course chose overlapping steps because overlap is the **cause** of a time difference; time is the result, and it is far noisier than the cause. #### Summary - A benchmark requires three declarations: the **unit** counted, the **regime** measured, and the measurement's **boundary**. The third is the most often skipped. - A single run gives **26, 31, 37, 49, 54, 80** across six separate declarations; all six are correct, and the difference between them comes not from the measurement but from the declaration. - Excluding setup gives **31**, including it gives **37** ticks: same work, same regime, **6 ticks** and **19.35%** difference, for zero change. - When setup is counted in only one regime, the ranking flips: **31** against **35** makes the losing regime appear to win. No row is wrong; what is wrong is putting them side by side. - A declaration flaw does not always flip the result, but whether it does can only be known with the correct declaration; noticing the flaw requires having also done the correct measurement. - This course counted steps instead of time because a teaching number has to be reproducible; overlapping steps are the cause of the difference, time is its noisy result. #### Course Wrap-Up The course turned a single setup across ten lessons: eight tasks, ten steps per task, 80 steps total. No lesson wrote down time; what was counted was steps, ticks, overlapping steps, calls, and objects. | Lesson | Measured regime | Tick / overlapping step or call | |---|---|---| | The Global Interpreter Lock | single flow, single slot | 80 ticks / 0 overlapping steps | | Threads | eight workers, single slot | 31 / 49 (I/O-bound), 73 / 7 (CPU-bound) | | Multiprocessing | eight workers, four and eight slots | 20 / 60 and 10 / 70 (CPU-bound) | | Asynchronous Programming | event loop, single flow, three yield-point regimes | 31 / 49, 36 / 44, and 80 / 0 | | Asynchronous Library Compatibility | event loop, blocking call and handoff to a pool | 43 / 37, 80 / 0, and 44 or 90 ticks on handoff | | Executor Pools | eight width, two backends, three loads | thread 31 / 49, 42 / 38, 73 / 7; process 10 / 70 | | Profiling | eight workers, single slot, two loads | 80 `advance` calls; hot path 136 and 322 calls | | Memory Usage | eight workers, single slot, six shapes | 31 / 49; objects held 1–172 | | Native Extensions | eight workers, single slot, condensed hot path | 25 / 19 (CPU-bound), 31 / 49 and gain 0 (I/O-bound) | | Measurement Discipline | eight workers, single slot | 31 ticks against 37 ticks | The table's reading gathers into one sentence: **the 80 steps never changed, what changed was only how many ticks those steps were spread across.** A regime does not reduce work, it overlaps it; and overlap is not possible on every step. This is why the same mechanism earns 49 ticks on one load and 7 on another, and why moving the hot path to a lower level gives 48 ticks on one load and **0** on the other. The last lesson put one more layer on top of this: none of these numbers can be set next to another's number without saying what it counts. A number that does not write down its boundary cannot be compared even if it is correct — and this rule applies to the course's own tables too; every row carries in its own name which regime, which load, and which unit it was measured in. The next course, Python Projects: Packaging and Testing, applies the same discipline to two other faces of the work. Writing down which environment, which dependencies, and which version rule a library was installed with is the same kind of work as writing down a measurement's boundary; knowing what a test checks and what a coverage number does not say is the same too. **Measuring is a discipline; packaging and testing are too.**