---
title: 'Measurement Discipline'
source: 'https://academia.sh/en/courses/python-concurrency/measurement-discipline'
course: 'Concurrency and Performance'
language: en
updated: '2026-08-17T18:10:24+00:00'
license: 'CC BY-SA 4.0'
---

# Measurement Discipline

The same run gives six separate numbers between 26 and 80 across six separate declarations; excluding setup gives 31 ticks, including it gives 37, and this 19.35% difference flips the ranking of two regimes when it is counted in only one of them.

The previous lesson wrote down what each row measured in the row's own name across a table,
and two rows gave the same number: **37**. One was the regime that pays boundary cost but
releases the lock, the other pays no boundary but does not release the lock either. Erase the
names and two equal numbers remain, and it would be assumed the two measure the same thing.

This course's last lesson measures exactly this danger. Its question is: **can a measurement
be compared without saying what it counts?** The answer will both be shown and applied to the
course's own method.

## What a Benchmark Requires

A **benchmark** is a measurement that puts two options side by side on the same work. Even
once the "same work" condition is met, a comparison is not automatically valid; three more
declarations are required.

**The unit's declaration:** what is being counted — ticks, steps, calls, objects? Every
measurement in this course wrote its unit by name, and there was a reason for that.

**The regime's declaration:** how many workers, how many slots, which load? The same setup
gives 31 under eight workers and a single slot, 80 under a single flow; without declaring the
regime, the number says nothing.

**The boundary's declaration:** where does the measurement begin and where does it end? Which
part of the work is included in the count? This third one is the most often skipped, and it
is what this lesson measures.

Above all three stands a fourth: the **oracle.** What says the measurement gives the correct
result? In this course, the oracle is the setup itself — we produced the tasks' step
sequence ourselves, so we know the total step count is 80 without measuring, and every table
can be tested against that number. A measurement without an oracle can tell a big number from
a small one, but it cannot tell right from wrong; when an optimization breaks the result, the
table still looks fine.

## Is Setup Counted

A piece of work has a price paid before it starts: setting up the execution context,
distributing tasks, preparation that cannot be skipped on the first round. In the model this
is not a duration, it is a **fixed number of steps**, and it is called setup.

Whether setup is counted is not a true-or-false question. In a long-running service, setup is
paid once and becomes negligible next to the total; in a short-lived job, it is paid again on
every run and is a direct cost. **Both declarations are defensible.** What is not defensible
is giving the number without writing down which one was done.

The measurement's assumptions:

- **PE19** — The measured work is the same in every row: the shared definition's eight
  tasks, total **80 steps**, the I/O-bound load. The only thing that changes is **the
  declaration of what is counted.**
- **PE20** — Setup is not a duration, it is the **fixed step count** that cannot be skipped
  on the first round.
- **PE21** — Setup is **declared** as **6** steps for the thread regime, **25** steps for the
  process regime. These numbers are not measured, they are the model's assumption; they rest
  on a separate execution context requiring more steps to set up, and the table reads them
  that way.
- **PE22** — All six declarations are read from the same run, and **none of them is wrong.**
  What is wrong is writing the number without saying which one was read.
- **PE23** — The oracle is the setup: total steps are **80**, and stay 80 no matter which
  regime is chosen. The regime does not change the step count, it changes how many ticks the
  steps spread across.
- **PE24** — No row measures time. Even the measurement including setup is not a time
  measurement, it is a step count; the lesson's claim is not about time, it is about
  **counting boundary.**

## Measurement

```python
"""Measurement discipline: same run, different declarations; which number compares."""

SEED = 20260817
CPU, IO = "cpu", "io"


def make_rng(seed):
    state = seed % 2147483646 + 1

    def draw(n):
        nonlocal state
        state = (state * 48271) % 2147483647
        return state % n
    return draw


def tasks(count=8, steps=10, io_share=7, seed=SEED):
    draw, result = make_rng(seed), []
    for i in range(count):
        result.append([IO if draw(10) < io_share else CPU for _ in range(steps)])
    return result


def run(jobs, workers, cpu_slots):
    remaining = [list(j) for j in jobs]
    tick = overlap = cpu_steps = io_steps = 0
    while any(remaining):
        active = [i for i, j in enumerate(remaining) if j][:workers]
        if not active:
            break
        slots_left, advanced = cpu_slots, 0
        for i in active:
            step = remaining[i][0]
            if step == CPU:
                if slots_left <= 0:
                    continue
                slots_left -= 1
                cpu_steps += 1
            else:
                io_steps += 1
            remaining[i].pop(0)
            advanced += 1
        tick += 1
        overlap += max(0, advanced - 1)
    return tick, overlap, cpu_steps, io_steps


G = tasks(io_share=7)
TICK, OVERLAP, CPU_STEPS, IO_STEPS = run(G, 8, 1)
SETUP = {"thread (single slot)": 6, "process (four slots)": 25}

print("same run, six declarations")
for name, count in (("excluding setup, tick", TICK),
                     ("including setup, tick", TICK + SETUP["thread (single slot)"]),
                     ("excluding setup, cpu step", CPU_STEPS),
                     ("excluding setup, io step", IO_STEPS),
                     ("overlapping step", OVERLAP),
                     ("total step", sum(len(g) for g in G))):
    print(f"  {name:<34s} {count:4d}")
k = SETUP["thread (single slot)"]
print(f"\nwithout counting setup {TICK} ticks, counting it {TICK + k} ticks; "
      f"diff {k} ticks ({k / TICK:.4f})")

THREAD = run(G, 8, 1)[0]
PROCESS = run(G, 8, 4)[0]
ti, pi = SETUP["thread (single slot)"], SETUP["process (four slots)"]
print(f"\n{'setup declaration':<28s} {'thread':>13s} {'process':>12s} "
      f"{'winner':>14s}")
for name, a, b in (("not counted in either", THREAD, PROCESS),
                    ("counted in both", THREAD + ti, PROCESS + pi),
                    ("counted only in process", THREAD, PROCESS + pi),
                    ("counted only in thread", THREAD + ti, PROCESS)):
    winner = "thread" if a < b else ("process" if b < a else "equal")
    print(f"  {name:<26s} {a:13d} {b:12d} {winner:>14s}")
print(f"\nsetup declared in the model: thread {ti} steps, process {pi} steps")
```

```
same run, six declarations
  excluding setup, tick                31
  including setup, tick                37
  excluding setup, cpu step            26
  excluding setup, io step             54
  overlapping step                     49
  total step                           80

without counting setup 31 ticks, counting it 37 ticks; diff 6 ticks (0.1935)

setup declaration                   thread      process         winner
  not counted in either                 31           10        process
  counted in both                       37           35        process
  counted only in process               31           35         thread
  counted only in thread                37           10        process

setup declared in the model: thread 6 steps, process 25 steps
```

## One Run, Six Numbers

The top table's six rows are read from **a single run**. The tasks were produced once, the
schedule ran once; the only thing that changes is which question is put to the result. The
numbers that come out are **26, 31, 37, 49, 54, 80**.

All six are correct. Whoever says "this work took 26 units" and whoever says "80 units" are
both talking about the same run. The ratio between them is more than threefold, and all of
that difference comes not from the measurement but from **the declaration**.

From this comes the lesson's rule: **a bare number is not a measurement.** A measurement is
the sum of the number and the declaration that says what the number counts. Without the
declaration, what is left cannot be put into a comparison — because the other side's row is
unknown.

## The Third Claim: 31 Against 37

The middle row carries the course's third claim on its own. The same task set, the same
regime, the same scheduler: excluding setup gives **31 ticks**, including it gives **37**.
Difference **6 ticks**, ratio **19.35%**.

Nothing changed. The code is the same, the data is the same, the slot count is the same, the
worker count is the same. Yet two numbers, with a difference between them close to a fifth.

This is the most ordinary difference that could show up in a performance report. If an
optimization were said to give a 19% gain, it would be taken seriously. Yet behind this
19.35% here there is no optimization at all — only two different counting boundaries.

## The Ranking Flipping

The bottom table shows how the same flaw leaks into a comparison. Two regimes side by side:
eight workers with a single slot, and eight workers with four slots.

The first two rows are honest declarations. When setup is not counted on either side, it is
**31** against **10**; when it is counted on both, **37** against **35**. In both, the
winner is **process**. Counting setup drops the gap between them from 21 ticks to 2 ticks —
the declaration does not change the result, but it changes **the size of the gap**.

The third row is not honest, and it is the only row in the table where the winner changes.
When setup is counted only for the process, it comes out **31** against **35** and **thread**
wins. The regime that actually loses appears to win, because the measurement's boundary was
widened on one side only.

A table like this lies in none of its rows: 31 is correct, 35 is correct. What is wrong is
putting the two **side by side**. Two numbers can be compared only if they were produced with
the same boundary.

The fourth row tries the reverse and the ranking does not flip: even if setup is counted only
for the thread, the process still wins by **10**. So a declaration flaw does not always flip
the result — **whether it does can only be known with the correct declaration.** The only way
to notice the flaw is to have also done the correct measurement.

## The Course Applied to Itself

This lesson has to justify a decision applied throughout the course: why was time never
written anywhere?

The answer is the same table above, one step further. If a number's declaration is missing,
it cannot be compared; and if a number additionally **cannot be reproduced**, a declaration
does not save it either. Time gives two separate values across two runs, even for the same
code on the same machine. Even if you declare exactly which boundary that value was measured
with, the second run changes the number.

Step, tick, call, and object do not carry this flaw. In the profiling lesson, the same run
was repeated three times and call count came out **322, 322, 322**. In the memory lesson, six
shapes gave the same object count on every run. Overlapping step is computed from the setup,
and the setup is fixed by the seed.

This is not a claim of superiority, it is a scope decision. Time measurement has its place,
and in a real system it eventually gets measured too; but **a number used to teach a concept
has to be reproducible.** This course chose overlapping steps because overlap is the
**cause** of a time difference; time is the result, and it is far noisier than the cause.

## Summary

- A benchmark requires three declarations: the **unit** counted, the **regime** measured, and
  the measurement's **boundary**. The third is the most often skipped.
- A single run gives **26, 31, 37, 49, 54, 80** across six separate declarations; all six are
  correct, and the difference between them comes not from the measurement but from the
  declaration.
- Excluding setup gives **31**, including it gives **37** ticks: same work, same regime, **6
  ticks** and **19.35%** difference, for zero change.
- When setup is counted in only one regime, the ranking flips: **31** against **35** makes
  the losing regime appear to win. No row is wrong; what is wrong is putting them side by
  side.
- A declaration flaw does not always flip the result, but whether it does can only be known
  with the correct declaration; noticing the flaw requires having also done the correct
  measurement.
- This course counted steps instead of time because a teaching number has to be reproducible;
  overlapping steps are the cause of the difference, time is its noisy result.

## Course Wrap-Up

The course turned a single setup across ten lessons: eight tasks, ten steps per task, 80
steps total. No lesson wrote down time; what was counted was steps, ticks, overlapping
steps, calls, and objects.

| Lesson | Measured regime | Tick / overlapping step or call |
|---|---|---|
| The Global Interpreter Lock | single flow, single slot | 80 ticks / 0 overlapping steps |
| Threads | eight workers, single slot | 31 / 49 (I/O-bound), 73 / 7 (CPU-bound) |
| Multiprocessing | eight workers, four and eight slots | 20 / 60 and 10 / 70 (CPU-bound) |
| Asynchronous Programming | event loop, single flow, three yield-point regimes | 31 / 49, 36 / 44, and 80 / 0 |
| Asynchronous Library Compatibility | event loop, blocking call and handoff to a pool | 43 / 37, 80 / 0, and 44 or 90 ticks on handoff |
| Executor Pools | eight width, two backends, three loads | thread 31 / 49, 42 / 38, 73 / 7; process 10 / 70 |
| Profiling | eight workers, single slot, two loads | 80 `advance` calls; hot path 136 and 322 calls |
| Memory Usage | eight workers, single slot, six shapes | 31 / 49; objects held 1–172 |
| Native Extensions | eight workers, single slot, condensed hot path | 25 / 19 (CPU-bound), 31 / 49 and gain 0 (I/O-bound) |
| Measurement Discipline | eight workers, single slot | 31 ticks against 37 ticks |

The table's reading gathers into one sentence: **the 80 steps never changed, what changed was
only how many ticks those steps were spread across.** A regime does not reduce work, it
overlaps it; and overlap is not possible on every step. This is why the same mechanism earns
49 ticks on one load and 7 on another, and why moving the hot path to a lower level gives 48
ticks on one load and **0** on the other.

The last lesson put one more layer on top of this: none of these numbers can be set next to
another's number without saying what it counts. A number that does not write down its
boundary cannot be compared even if it is correct — and this rule applies to the course's own
tables too; every row carries in its own name which regime, which load, and which unit it was
measured in.

The next course, Python Projects: Packaging and Testing, applies the same discipline to two
other faces of the work. Writing down which environment, which dependencies, and which
version rule a library was installed with is the same kind of work as writing down a
measurement's boundary; knowing what a test checks and what a coverage number does not say is
the same too. **Measuring is a discipline; packaging and testing are too.**