Measurement Discipline

The previous lesson wrote down what each row measured in the row’s own name across a table, and two rows gave the same number: 37. One was the regime that pays boundary cost but releases the lock, the other pays no boundary but does not release the lock either. Erase the names and two equal numbers remain, and it would be assumed the two measure the same thing.

This course’s last lesson measures exactly this danger. Its question is: can a measurement be compared without saying what it counts? The answer will both be shown and applied to the course’s own method.

What a Benchmark Requires

A benchmark is a measurement that puts two options side by side on the same work. Even once the “same work” condition is met, a comparison is not automatically valid; three more declarations are required.

The unit’s declaration: what is being counted — ticks, steps, calls, objects? Every measurement in this course wrote its unit by name, and there was a reason for that.

The regime’s declaration: how many workers, how many slots, which load? The same setup gives 31 under eight workers and a single slot, 80 under a single flow; without declaring the regime, the number says nothing.

The boundary’s declaration: where does the measurement begin and where does it end? Which part of the work is included in the count? This third one is the most often skipped, and it is what this lesson measures.

Above all three stands a fourth: the oracle. What says the measurement gives the correct result? In this course, the oracle is the setup itself — we produced the tasks’ step sequence ourselves, so we know the total step count is 80 without measuring, and every table can be tested against that number. A measurement without an oracle can tell a big number from a small one, but it cannot tell right from wrong; when an optimization breaks the result, the table still looks fine.

Is Setup Counted

A piece of work has a price paid before it starts: setting up the execution context, distributing tasks, preparation that cannot be skipped on the first round. In the model this is not a duration, it is a fixed number of steps, and it is called setup.

Whether setup is counted is not a true-or-false question. In a long-running service, setup is paid once and becomes negligible next to the total; in a short-lived job, it is paid again on every run and is a direct cost. Both declarations are defensible. What is not defensible is giving the number without writing down which one was done.

The measurement’s assumptions:

• PE19 — The measured work is the same in every row: the shared definition’s eight tasks, total 80 steps, the I/O-bound load. The only thing that changes is the declaration of what is counted.
• PE20 — Setup is not a duration, it is the fixed step count that cannot be skipped on the first round.
• PE21 — Setup is declared as 6 steps for the thread regime, 25 steps for the process regime. These numbers are not measured, they are the model’s assumption; they rest on a separate execution context requiring more steps to set up, and the table reads them that way.
• PE22 — All six declarations are read from the same run, and none of them is wrong. What is wrong is writing the number without saying which one was read.
• PE23 — The oracle is the setup: total steps are 80, and stay 80 no matter which regime is chosen. The regime does not change the step count, it changes how many ticks the steps spread across.
• PE24 — No row measures time. Even the measurement including setup is not a time measurement, it is a step count; the lesson’s claim is not about time, it is about counting boundary.

Measurement

"""Measurement discipline: same run, different declarations; which number compares."""

SEED = 20260817
CPU, IO = "cpu", "io"


def make_rng(seed):
    state = seed % 2147483646 + 1

    def draw(n):
        nonlocal state
        state = (state * 48271) % 2147483647
        return state % n
    return draw


def tasks(count=8, steps=10, io_share=7, seed=SEED):
    draw, result = make_rng(seed), []
    for i in range(count):
        result.append([IO if draw(10) < io_share else CPU for _ in range(steps)])
    return result


def run(jobs, workers, cpu_slots):
    remaining = [list(j) for j in jobs]
    tick = overlap = cpu_steps = io_steps = 0
    while any(remaining):
        active = [i for i, j in enumerate(remaining) if j][:workers]
        if not active:
            break
        slots_left, advanced = cpu_slots, 0
        for i in active:
            step = remaining[i][0]
            if step == CPU:
                if slots_left <= 0:
                    continue
                slots_left -= 1
                cpu_steps += 1
            else:
                io_steps += 1
            remaining[i].pop(0)
            advanced += 1
        tick += 1
        overlap += max(0, advanced - 1)
    return tick, overlap, cpu_steps, io_steps


G = tasks(io_share=7)
TICK, OVERLAP, CPU_STEPS, IO_STEPS = run(G, 8, 1)
SETUP = {"thread (single slot)": 6, "process (four slots)": 25}

print("same run, six declarations")
for name, count in (("excluding setup, tick", TICK),
                     ("including setup, tick", TICK + SETUP["thread (single slot)"]),
                     ("excluding setup, cpu step", CPU_STEPS),
                     ("excluding setup, io step", IO_STEPS),
                     ("overlapping step", OVERLAP),
                     ("total step", sum(len(g) for g in G))):
    print(f"  {name:<34s} {count:4d}")
k = SETUP["thread (single slot)"]
print(f"\nwithout counting setup {TICK} ticks, counting it {TICK + k} ticks; "
      f"diff {k} ticks ({k / TICK:.4f})")

THREAD = run(G, 8, 1)[0]
PROCESS = run(G, 8, 4)[0]
ti, pi = SETUP["thread (single slot)"], SETUP["process (four slots)"]
print(f"\n{'setup declaration':<28s} {'thread':>13s} {'process':>12s} "
      f"{'winner':>14s}")
for name, a, b in (("not counted in either", THREAD, PROCESS),
                    ("counted in both", THREAD + ti, PROCESS + pi),
                    ("counted only in process", THREAD, PROCESS + pi),
                    ("counted only in thread", THREAD + ti, PROCESS)):
    winner = "thread" if a < b else ("process" if b < a else "equal")
    print(f"  {name:<26s} {a:13d} {b:12d} {winner:>14s}")
print(f"\nsetup declared in the model: thread {ti} steps, process {pi} steps")

same run, six declarations
  excluding setup, tick                31
  including setup, tick                37
  excluding setup, cpu step            26
  excluding setup, io step             54
  overlapping step                     49
  total step                           80

without counting setup 31 ticks, counting it 37 ticks; diff 6 ticks (0.1935)

setup declaration                   thread      process         winner
  not counted in either                 31           10        process
  counted in both                       37           35        process
  counted only in process               31           35         thread
  counted only in thread                37           10        process

setup declared in the model: thread 6 steps, process 25 steps

One Run, Six Numbers

The top table’s six rows are read from a single run. The tasks were produced once, the schedule ran once; the only thing that changes is which question is put to the result. The numbers that come out are 26, 31, 37, 49, 54, 80.

All six are correct. Whoever says “this work took 26 units” and whoever says “80 units” are both talking about the same run. The ratio between them is more than threefold, and all of that difference comes not from the measurement but from the declaration.

From this comes the lesson’s rule: a bare number is not a measurement. A measurement is the sum of the number and the declaration that says what the number counts. Without the declaration, what is left cannot be put into a comparison — because the other side’s row is unknown.

The Third Claim: 31 Against 37

The middle row carries the course’s third claim on its own. The same task set, the same regime, the same scheduler: excluding setup gives 31 ticks, including it gives 37. Difference 6 ticks, ratio 19.35%.

Nothing changed. The code is the same, the data is the same, the slot count is the same, the worker count is the same. Yet two numbers, with a difference between them close to a fifth.

This is the most ordinary difference that could show up in a performance report. If an optimization were said to give a 19% gain, it would be taken seriously. Yet behind this 19.35% here there is no optimization at all — only two different counting boundaries.

The Ranking Flipping

The bottom table shows how the same flaw leaks into a comparison. Two regimes side by side: eight workers with a single slot, and eight workers with four slots.

The first two rows are honest declarations. When setup is not counted on either side, it is 31 against 10; when it is counted on both, 37 against 35. In both, the winner is process. Counting setup drops the gap between them from 21 ticks to 2 ticks — the declaration does not change the result, but it changes the size of the gap.

The third row is not honest, and it is the only row in the table where the winner changes. When setup is counted only for the process, it comes out 31 against 35 and thread wins. The regime that actually loses appears to win, because the measurement’s boundary was widened on one side only.

A table like this lies in none of its rows: 31 is correct, 35 is correct. What is wrong is putting the two side by side. Two numbers can be compared only if they were produced with the same boundary.

The fourth row tries the reverse and the ranking does not flip: even if setup is counted only for the thread, the process still wins by 10. So a declaration flaw does not always flip the result — whether it does can only be known with the correct declaration. The only way to notice the flaw is to have also done the correct measurement.

The Course Applied to Itself

This lesson has to justify a decision applied throughout the course: why was time never written anywhere?

The answer is the same table above, one step further. If a number’s declaration is missing, it cannot be compared; and if a number additionally cannot be reproduced, a declaration does not save it either. Time gives two separate values across two runs, even for the same code on the same machine. Even if you declare exactly which boundary that value was measured with, the second run changes the number.

Step, tick, call, and object do not carry this flaw. In the profiling lesson, the same run was repeated three times and call count came out 322, 322, 322. In the memory lesson, six shapes gave the same object count on every run. Overlapping step is computed from the setup, and the setup is fixed by the seed.

This is not a claim of superiority, it is a scope decision. Time measurement has its place, and in a real system it eventually gets measured too; but a number used to teach a concept has to be reproducible. This course chose overlapping steps because overlap is the cause of a time difference; time is the result, and it is far noisier than the cause.

Summary

• A benchmark requires three declarations: the unit counted, the regime measured, and the measurement’s boundary. The third is the most often skipped.
• A single run gives 26, 31, 37, 49, 54, 80 across six separate declarations; all six are correct, and the difference between them comes not from the measurement but from the declaration.
• Excluding setup gives 31, including it gives 37 ticks: same work, same regime, 6 ticks and 19.35% difference, for zero change.
• When setup is counted in only one regime, the ranking flips: 31 against 35 makes the losing regime appear to win. No row is wrong; what is wrong is putting them side by side.
• A declaration flaw does not always flip the result, but whether it does can only be known with the correct declaration; noticing the flaw requires having also done the correct measurement.
• This course counted steps instead of time because a teaching number has to be reproducible; overlapping steps are the cause of the difference, time is its noisy result.

Course Wrap-Up

The course turned a single setup across ten lessons: eight tasks, ten steps per task, 80 steps total. No lesson wrote down time; what was counted was steps, ticks, overlapping steps, calls, and objects.

Lesson	Measured regime	Tick / overlapping step or call
The Global Interpreter Lock	single flow, single slot	80 ticks / 0 overlapping steps
Threads	eight workers, single slot	31 / 49 (I/O-bound), 73 / 7 (CPU-bound)
Multiprocessing	eight workers, four and eight slots	20 / 60 and 10 / 70 (CPU-bound)
Asynchronous Programming	event loop, single flow, three yield-point regimes	31 / 49, 36 / 44, and 80 / 0
Asynchronous Library Compatibility	event loop, blocking call and handoff to a pool	43 / 37, 80 / 0, and 44 or 90 ticks on handoff
Executor Pools	eight width, two backends, three loads	thread 31 / 49, 42 / 38, 73 / 7; process 10 / 70
Profiling	eight workers, single slot, two loads	80 advance calls; hot path 136 and 322 calls
Memory Usage	eight workers, single slot, six shapes	31 / 49; objects held 1–172
Native Extensions	eight workers, single slot, condensed hot path	25 / 19 (CPU-bound), 31 / 49 and gain 0 (I/O-bound)
Measurement Discipline	eight workers, single slot	31 ticks against 37 ticks

The table’s reading gathers into one sentence: the 80 steps never changed, what changed was only how many ticks those steps were spread across. A regime does not reduce work, it overlaps it; and overlap is not possible on every step. This is why the same mechanism earns 49 ticks on one load and 7 on another, and why moving the hot path to a lower level gives 48 ticks on one load and 0 on the other.

The last lesson put one more layer on top of this: none of these numbers can be set next to another’s number without saying what it counts. A number that does not write down its boundary cannot be compared even if it is correct — and this rule applies to the course’s own tables too; every row carries in its own name which regime, which load, and which unit it was measured in.

The next course, Python Projects: Packaging and Testing, applies the same discipline to two other faces of the work. Writing down which environment, which dependencies, and which version rule a library was installed with is the same kind of work as writing down a measurement’s boundary; knowing what a test checks and what a coverage number does not say is the same too. Measuring is a discipline; packaging and testing are too.