--- title: 'Python Projects: Packaging and Testing' source: 'https://academia.sh/en/courses/python-projects' language: en updated: '2026-08-17T04:42:25+00:00' license: 'CC BY-SA 4.0' lessons: 14 outcomes: - 'Sets up isolated environments and dependency management reproducibly' - 'Writes a test suite and evaluates coverage with meaningful criteria' - 'Packages and publishes a library' --- # Python Projects: Packaging and Testing The number that describes a project is not whether it runs but how many separate results the same input produces; the course measures dependency resolution, test order, and coverage with that single question, and models the tools at the class level without naming them. ## Environment and Dependencies The rationale for isolation and the install lifecycle, project metadata and locking, resolution strategies producing separate sets from the same declaration, and switching between interpreter versions. ### Virtual Environments Source: https://academia.sh/en/courses/python-projects/virtual-environments The same resolution gives five distinct outcomes across six installs on a shared site, and the project sees the set it resolved itself on only two of the six orders; on an isolated site the number drops to one, and all six orders install the same set. The previous course counted steps instead of time across ten lessons, and left one sentence at its close: measuring is a discipline; so is packaging and testing. That course's question was **what** the measurement counted — whether setup counted toward the total, where the boundary was drawn, what name the unit was written under. This course's question steps back one level. There is a more basic question that has to be answered before a number's boundary can even be written: **does the same input give the same result every time?** If a measurement gives two separate values across two runs, no matter how well you write its boundary, it cannot be compared. This course asks that question in three separate places in a project: its dependencies, its tests, and the package it publishes. The first place is the one deciding which packages the code will run into. #### The Course's Measure: Distinct Outcome A project's number is not whether it runs. A project that runs might not run on a second machine, and this second outcome does not invalidate the first — both are true at once, just like the previous course's six statements. What can be counted is this: **how many distinct outcomes did the same input give?** The course's measure is named **distinct outcome**. An installation's outcome is the set of versions installed; if two installations give the same set, that counts as one outcome, if they give a different set, that counts as two. A number not changing across two runs is called **reproducibility**, and every measurement in this course searches for it. The rig is made of three packages: `metrics`, `report`, and `common`. Each has published versions in a registry, and the project asks for them with a **dependency declaration**: the smallest and largest accepted version, upper bound excluded. A declaration does not name a version, it names a range — this distinction is the next lesson's subject. No lesson in this course writes a tool's name. The virtual environment tool, **package manager**, test framework, and build system are referred to only by **class name**; their behavior is **modeled** in the lesson, and that it is modeled is stated. The standard library's `venv` and `sysconfig` modules belong to the language itself and can be named directly. In the measurement, **no real installation is performed, no real package is downloaded**; installation is modeled as an operation that writes to a dict. #### What Isolation Separates A **virtual environment** is the isolation boundary that makes the location the interpreter searches for packages specific to a project. This lesson calls the directory installed packages land in the **site**; the interpreter looks there during import. In the standard library, the `sysconfig` module gives this layout by scheme name, and the `venv` module produces a new layout. The real path is not written, because the path is machine-dependent and not reproducible. What isolation provides is not a privacy boundary. The only thing it provides is this: **the same-named package can exist at two separate versions in two projects.** With a single site, this is impossible, because one name corresponds to one version in one place. A direct consequence follows from this. When a single site is shared, the versions a project sees depend not only on its own declaration but also on **the other projects sharing the site, and the install order**. This is what is going to be measured. #### Where the Boundary Is Drawn What a virtual environment is not shapes the measurement as much as what it is, because everything left outside the boundary keeps producing distinct outcomes. A virtual environment **does not duplicate the interpreter**; it attaches to an existing interpreter and gives it a separate site. Setting up the environment amounts to nothing more than making the layout that interpreter looks at during import specific to the project. Activation is not a separate mechanism either, it is the same thing's shell-side counterpart: which interpreter gets invoked and which site it sees is decided there. In the standard library, this layout's names sit in `sysconfig` schemas; producing a new one is `venv`'s job. One consequence connects directly to this topic's last lesson: isolation separates **packages**, not **the interpreter**. An environment is meaningful only with the interpreter it is attached to; when the interpreter changes, the environment has to be rebuilt, since part of the site's packages may not have been built for that interpreter. A second thing left outside is the operating-system level. Shared system libraries, the compiler, environment variables, and the file system sit outside a virtual environment's boundary; if a package depends on one of these, two distinct outcomes are possible on two machines all over again. The tools drawing this wider boundary belong to a different class and are not this course's subject. The boundary measured here stops at one level: **the same machine, the same interpreter, two projects' package sets.** #### An Environment's Lifecycle The install flow is four steps: the environment is set up, the declaration is resolved, the chosen versions are installed to the site, and the project runs through that site. After the fourth step, the environment is now an **artifact** — a produced result, not a written source. This distinction's practical consequence is clear: the environment is not put under version control. The reason is not its size, it is that **its source lives somewhere else**. What will reproduce the environment is the declaration; the environment itself is one run of that declaration. If two runs give the same set, there is no payoff to keeping the environment. A test follows from this. If rebuilding the environment from the same declaration loses something, that thing was never written into the declaration — a hand-installed package, a hand-corrected version, a residue accumulated inside the environment. Deletion, for this reason, is not a maintenance chore, it is **a measurement**: nothing says more directly whether a declaration is complete. The measurement question's second half follows from this too. When the environment is deleted and rebuilt, does the **same set** come back? On an isolated site, with neighbors and order eliminated, one variable remains: the resolution itself. This lesson's measurement holds that variable fixed and counts only isolation's share. #### On a Shared Site, the Last Installer Wins Three projects use the same site. **A** is our project: it resolves its declaration and picks the newest eligible versions. **B** and **C** are neighbor projects; both arrive with their own locks, meaning the versions they will install are already written down and they do not resolve. The three projects' package sets partly overlap. The model's installation behavior is a single line: installing a package overwrites the version on the site. **The last installer wins.** This is the unavoidable behavior of a single-site layout; since one name can be bound to one version, the last project installing it turns the binding in its own favor. A's resolution is identical across all six orders, and the measurement prints this separately. What changes is not the set A resolved, it is the set A **sees at run time**. The measurement's assumptions: - **ED1** — The oracle is the rig itself: we wrote the registry's versions and the three projects' requests ourselves, so the correct result of every order is known without measuring. - **ED2** — The registry and the declaration are taken from the shared definition unchanged; the lesson does not modify either. - **ED3** — A resolves its declaration with the **newest eligible version** strategy. The strategy is fixed in this lesson; what happens when it changes is the next two lessons' subject. - **ED4** — B and C do not resolve; the versions they will install are written in their locks and are part of the rig. - **ED5** — Installation overwrites the version on the site. There is no version comparison or "already installed" check; the model's only rule is that the last installer wins. - **ED6** — In the shared regime there is a single site, and all three install to it. In the isolated regime, every project has its own site. - **ED7** — What is measured is the `metrics`, `common`, and `report` versions standing on A's site at run time; what the other projects see is not counted. - **ED8** — The six orders are all install orders of the three projects; none is privileged. On a real machine, what decides the order is the installs' schedule, and the project does not choose it. - **ED9** — Installation is an operation that writes to a dict. **No real installation is performed, no real package is downloaded, no file is left in a registry.** - **ED10** — Duration is not measured. What is counted is the **distinct outcome** and the number of orders producing each one. - **ED11** — The measurement is a single run and deterministic; there is no randomness in the rig. #### Measurement ```python """Isolation: same resolution, separate site; how many distinct outcomes does install order give.""" from itertools import permutations REGISTRY = { "metrics": [(1, 0), (1, 1), (1, 2), (2, 0)], "report": [(0, 9), (1, 0), (1, 1)], "common": [(3, 0), (3, 1), (3, 2), (4, 0)], } DECLARATION = {"metrics": ((1, 0), (2, 0)), "report": ((1, 0), (2, 0)), "common": ((3, 0), (4, 0))} # Neighbor projects come with their own locks; they use the same registry at separate versions. NEIGHBORS = {"B": {"common": (3, 0), "report": (1, 0)}, "C": {"metrics": (1, 1), "common": (3, 1)}} PACKAGES = ("metrics", "common", "report") def eligible(package): lo, hi = DECLARATION[package] return [s for s in REGISTRY[package] if lo <= s < hi] def resolve(strategy="newest"): chosen = {} for package in sorted(DECLARATION): candidates = eligible(package) chosen[package] = candidates[-1] if strategy == "newest" else candidates[0] return chosen def write(site, chosen): """Installation overwrites the version on the site: the last installer wins.""" site.update(chosen) return site def choice_of(name): return resolve() if name == "A" else NEIGHBORS[name] def shared(order): site = {} for name in order: write(site, choice_of(name)) return tuple(site[p] for p in PACKAGES) def isolated(order): sites = {"A": {}, "B": {}, "C": {}} for name in order: write(sites[name], choice_of(name)) return tuple(sites["A"][p] for p in PACKAGES) def format_versions(view): return " / ".join(".".join(map(str, s)) for s in view) ORDERS = list(permutations(("A", "B", "C"))) EXPECTED = tuple(resolve()[p] for p in PACKAGES) print(f"{'install order':<16s} {'A on shared site':>18s} {'A on isolated site':>19s}") p_view, y_view = [], [] for order in ORDERS: a, b = shared(order), isolated(order) p_view.append(a) y_view.append(b) print(f"{' '.join(order):<16s} {format_versions(a):>18s} {format_versions(b):>19s}") print(f"\nA's resolution is {format_versions(EXPECTED)} across all six orders") print(f"six install orders, {len(set(p_view))} distinct outcomes on the shared site, " f"{len(set(y_view))} on the isolated site") print(f"orders seeing its own resolved set: shared {p_view.count(EXPECTED)}/{len(ORDERS)}, " f"isolated {y_view.count(EXPECTED)}/{len(ORDERS)}") ``` ``` install order A on shared site A on isolated site A B C 1.1 / 3.1 / 1.0 1.2 / 3.2 / 1.1 A C B 1.1 / 3.0 / 1.0 1.2 / 3.2 / 1.1 B A C 1.1 / 3.1 / 1.1 1.2 / 3.2 / 1.1 B C A 1.2 / 3.2 / 1.1 1.2 / 3.2 / 1.1 C A B 1.2 / 3.0 / 1.0 1.2 / 3.2 / 1.1 C B A 1.2 / 3.2 / 1.1 1.2 / 3.2 / 1.1 A's resolution is 1.2 / 3.2 / 1.1 across all six orders six install orders, 5 distinct outcomes on the shared site, 1 on the isolated site orders seeing its own resolved set: shared 2/6, isolated 6/6 ``` #### Six Orders, Five Outcomes The left column gives five distinct sets across six rows. Only two orders produce the same result (`B C A` and `C B A`), and what they share is that A installed **last** in both. Each of the remaining four orders gives its own set. The first row gives the extreme case. In order `A B C`, A installs first and writes all three of its packages with its own chosen versions; then B writes over `common` and `report`, then C writes over `metrics` and `common`. In the end, **all three** of the versions A sees come from other projects — none of its own decisions survive. In order `B A C`, only `report` survives, because C never installs that package. In order `C A B`, the only surviving package is `metrics`; `common` and `report` come last from B. None of these five outcomes violates the declaration. A's declaration accepts all three versions for `common`, all three for `metrics`, and both for `report`. So no matter what order A installs in, it meets a version it accepts, and no warning is ever raised. This is where the defect's **silence** comes from: since the declaration is a range, every deviation inside that range is legitimate. Only one of the five sets is A's own choice. A never chose the remaining four, never wanted them, and never wrote them anywhere; and yet its code ran on top of them. This shows why distinct outcome is a separate measure from "worked" or "did not work": the project can run on all six of the six orders, none of them may raise an error, and five separate installations can still stand behind that. The right column shows the same six orders under the isolated regime and gives a single set: **1 distinct outcome**. Across all six orders, A sees the set it resolved itself. The bottom row writes the difference in its sharpest form. On the shared site, A sees the result of its own resolution on **two of six** orders; on the isolated site, on **all six**. The source of a three-quarters deviation is not in A's own decisions at all, it is entirely outside them. #### What Isolation Does Not Make Uniform Isolation solves one thing and does not solve another. This split is the frame for the rest of the course. **What it solves:** neighbor projects' decisions and install order no longer change the outcome. Five distinct outcomes dropped to one, and what dropped them was not a better resolution, it was **the boundary itself**. A's resolution was the same at the start of the measurement and at the end; what changed was whether the resolution's result was preserved. **What it does not solve:** A's resolution **itself**. Even on an isolated site, the set installed depends on which rule the resolution was made with. The measurement held this rule fixed — the newest eligible version — and that is why the right column gave a single set. Had the rule changed, the right column would have changed too, because the declaration states not a version but a range. A second boundary sits even further out: the same declaration can give a separate set again when the registry changes. Today's newest version may not be tomorrow's, and the declaration accepts the new one too. Isolation does nothing against either of these; the boundary ends at the site's wall. A virtual environment is therefore **not the whole of reproducibility, it is its first condition.** Drawing the boundary is not the same thing as deciding what happens inside it. #### Summary - The course's measure is **distinct outcome**: a project's number is not whether it runs, it is how many distinct outcomes the same input gives. - A **virtual environment** is an isolation boundary; the only thing it provides is that the same-named package can exist at two separate versions in two projects. - When a single site is shared, install order decides the outcome: six orders give **5 distinct outcomes**, and the project sees the set it resolved itself in only **2/6** orders. - On an isolated site, the same six orders give **1 distinct outcome**, and the project sees its set in **6/6** orders; A's resolution is the same in both regimes — what is preserved is the resolution's **result**. - None of the five outcomes violates the declaration; since the declaration is a range, a deviation within it is silent and raises no warning. - Isolation eliminates the neighbor and the order, but it does not eliminate the resolution's own rule or the registry changing over time. #### Next Step The single outcome on the isolated site was reached by holding one rule fixed in the measurement: the newest eligible version. That rule was a choice, and other choices are just as valid under the declaration. The next lesson looks inside the boundary and asks the same question once more: **on its own, how many distinct version sets does the same declaration specify?** The measurement puts two rules side by side, writes the resulting number to a file, and tests whether that file eliminates the resolution altogether. ### Dependency Declaration Source: https://academia.sh/en/courses/python-projects/dependency-declaration The same declaration gives 2 distinct version sets when the resolution strategy changes, and a lock file drops the number to 1; when the registry view also changes, four installs made from the declaration give 3 distinct sets while the locked four stay at one set. The previous lesson drew a boundary and measured its share: on an isolated site, install order and neighbor projects were eliminated, and five distinct outcomes dropped to one. That measurement was reached by holding one thing fixed — the resolution rule was always the same, always the newest eligible version. This lesson releases that constant. Its question is: **on its own, how many distinct version sets does the same declaration specify?** The declaration is the one thing the project writes, and it does not change; what changes is the rule of the side reading it. The answer will be measured, then written to a file, and the measurement repeated. #### A Declaration States a Range A **dependency declaration** is the list sitting in a project's metadata that says which packages are needed in which version range. The metadata itself carries more than this: the project's name, its own version, entry points, optional dependency groups. This lesson's concern is only the dependency list. How the declaration is written carries a critical distinction. What is written for a package is almost never a single version; it is a **range** — the smallest accepted version and the version above which it is no longer accepted. There is a reason for this: a declaration pinned to a single version cannot come together with another package that uses the same one. A range is the condition for coming together at all. Its cost comes from the same place. A range legitimizes every version that falls inside it. The side reading the declaration has to choose **one** version from the range, and the rule making that choice is not written in the declaration. The rule is the **package manager's** own **resolution strategy**, reading the range. This lesson compares two strategies: **newest** and **oldest compatible**. Both conform to the declaration; both are legitimate. The strategies and their behavior are **modeled** in the lesson; no product name is written and no real installation is performed. #### What a Lock Is, and Is Not A **lock file** is a document that writes a resolution's result line by line: not a range, the exact versions chosen. The declaration is the human-written request, the lock is the machine-produced decision. The two are separate documents, and both are put under version control — the opposite of the previous lesson's environment, because the environment is an artifact, while the lock is a decision. Saying what a lock is not matters more. A lock does not eliminate resolution. The lock file itself **is a resolution's output**; producing it required the ranges to be read once, a strategy applied, and a set chosen. What the lock does is make that work run once and **write its result down**. Later installs do not resolve, they read what is written. This distinction shows up in the measurement as two rows: two installs made with the lock file give the same set even when two separate strategies are requested. The strategy is no longer being read. The measurement's assumptions: - **ED12** — The registry, the declaration, and the resolve function are taken from the shared definition unchanged; the lesson does not change either's behavior. - **ED13** — The registry has been made a function parameter, and its default value is the shared definition's registry. The binding numbers are produced with the default registry. - **ED14** — The declaration is a range: the smallest version included, the largest excluded. - **ED15** — There are two strategies: **newest** picks the last of the eligible candidates; **oldest compatible** picks the first. Both conform to the declaration. - **ED16** — The oracle is the rig: we wrote which versions are published and what the declaration accepts, so every row's correct result is known without measuring. - **ED17** — A lock is a copy of a resolution's result. Resolution is not called while a lock exists; this is the model's one branch in the `install` function. - **ED18** — In the measurement, the lock is produced from the **newest** strategy's result. Had it been produced from a different strategy, the locked rows would show that set; what will not change is **how many distinct outcomes** locked installs give. - **ED19** — The second registry view is derived from the first by adding versions published afterward. The declaration does not change, the registry does; the added versions fall inside the declaration's range. - **ED20** — In this lesson, packages do not require each other; every package is chosen independently. What transitive requirements and conflict do is the next lesson's subject. - **ED21** — Version numbers are the rig packages' fictional versions; they are not a real package's or tool's version. - **ED22** — **No real installation is performed, no real package is downloaded.** Installation is returning the chosen dict. - **ED23** — Duration is not measured. What is counted is the **distinct version set**, that is, the distinct outcome. #### Measurement ```python """A declaration states a range: how many distinct sets does strategy and registry view give.""" REGISTRY = { "metrics": [(1, 0), (1, 1), (1, 2), (2, 0)], "report": [(0, 9), (1, 0), (1, 1)], "common": [(3, 0), (3, 1), (3, 2), (4, 0)], } DECLARATION = {"metrics": ((1, 0), (2, 0)), "report": ((1, 0), (2, 0)), "common": ((3, 0), (4, 0))} # Same registry, with versions published later: the declaration did not change, the registry did. NEWLY = {"metrics": [(1, 3)], "report": [(1, 2)], "common": [(3, 3)]} LATER = {p: sorted(REGISTRY[p] + NEWLY[p]) for p in REGISTRY} PACKAGES = ("metrics", "common", "report") def eligible(package, registry=REGISTRY): lo, hi = DECLARATION[package] return [s for s in registry[package] if lo <= s < hi] def resolve(strategy, registry=REGISTRY): """Two separate strategies resolve the same declaration.""" chosen = {} for package in sorted(DECLARATION): candidates = eligible(package, registry) chosen[package] = candidates[-1] if strategy == "newest" else candidates[0] return chosen def lock(chosen): """A lock fixes a resolution's result.""" return dict(chosen) def install(lock_file=None, strategy="newest", registry=REGISTRY): return lock_file if lock_file else resolve(strategy, registry) def format_selection(chosen): return " ".join(f"{'.'.join(map(str, chosen[p])):>7s}" for p in PACKAGES) newest, oldest = resolve("newest"), resolve("oldest") lock_file = lock(newest) print(f"{'install':<26s} {'metrics':>7s} {'common':>7s} {'report':>7s}") for name, chosen in (("strategy: newest", newest), ("strategy: oldest", oldest), ("with lock file", install(lock_file)), ("lock, other strategy", install(lock_file, "oldest"))): print(f"{name:<26s} " + format_selection(chosen)) distinct = len({tuple(sorted(s.items())) for s in (newest, oldest)}) distinct_locked = len({tuple(sorted(install(lock_file, st).items())) for st in ("newest", "oldest")}) print(f"same declaration, {distinct} distinct sets by strategy; with the lock {distinct_locked}") print() print(f"{'registry view / regime':<26s} {'metrics':>7s} {'common':>7s} {'report':>7s}") installs = [] for registry_name, registry in (("first registry", REGISTRY), ("later registry", LATER)): for st in ("newest", "oldest"): chosen = install(None, st, registry) installs.append(chosen) print(f"{registry_name + ', ' + st:<26s} " + format_selection(chosen)) locked = [install(lock_file, st, registry) for registry in (REGISTRY, LATER) for st in ("newest", "oldest")] print(f"{'lock file, all four':<26s} " + format_selection(locked[0])) print(f"two registry views and two strategies, four installs: with the declaration " f"{len({tuple(sorted(s.items())) for s in installs})} distinct sets, " f"with the lock {len({tuple(sorted(s.items())) for s in locked})}") print() produced = resolve("newest") print(f"the lock is produced from a resolution: lock == newest resolution -> " f"{lock(produced) == produced}") print("if the lock is regenerated on the later registry: " + format_selection(lock(resolve("newest", LATER)))) ``` ``` install metrics common report strategy: newest 1.2 3.2 1.1 strategy: oldest 1.0 3.0 1.0 with lock file 1.2 3.2 1.1 lock, other strategy 1.2 3.2 1.1 same declaration, 2 distinct sets by strategy; with the lock 1 registry view / regime metrics common report first registry, newest 1.2 3.2 1.1 first registry, oldest 1.0 3.0 1.0 later registry, newest 1.3 3.3 1.2 later registry, oldest 1.0 3.0 1.0 lock file, all four 1.2 3.2 1.1 two registry views and two strategies, four installs: with the declaration 3 distinct sets, with the lock 1 the lock is produced from a resolution: lock == newest resolution -> True if the lock is regenerated on the later registry: 1.3 3.3 1.2 ``` #### Two Sets, Then One The top table's first two rows carry the course's second claim. Same declaration, same registry, same three packages: with strategy **newest**, `1.2 / 3.2 / 1.1`; with **oldest compatible**, `1.0 / 3.0 / 1.0`. **2 distinct version sets.** There is no error anywhere between the two sets. The declaration accepts both; both conform to the request the project wrote. But these are not the same install. All three packages sit at separate versions; the behavior the code will meet, the fixed defects, and the added methods are separate. "The declaration is the same, so the install is the same" is the sentence the measurement disproves. There is no ranking of superiority between the two either, because they test separate things. **Newest** brings fixes, but since it brings the behavior closest to the declaration's upper bound, it picks the versions the project has tested the least. **Oldest compatible** does the opposite: it tests whether the declaration's lower end actually works. A range's lower bound is often copied in from somewhere and never tried; an install choosing it reveals whether the declaration is telling the truth. What the measurement says is not which one is good, it is **that the two are not the same install.** The bottom two rows bring the lock into play. The install made with the lock file gives `1.2 / 3.2 / 1.1`; the same set comes back even when the lock file **and** a different strategy are requested. **1 distinct outcome with the lock.** The fourth row says this on its own: the strategy is no longer being read, because there is no range left to read. #### The Registry Changes Over Time The second table adds one more variable. The declaration is still the same, but the registry is at its next view: three new versions falling into the range have been published. Four installs made with the declaration give **3 distinct sets**. The `newest` strategy produces two distinct sets across the two registry views — `1.2 / 3.2 / 1.1` and `1.3 / 3.3 / 1.2` — because "newest" is not a fixed version, it is **a function of the registry's current state**. `oldest compatible` gives the same set on both views; the range's lower end is not affected by the registry growing. This row shows that the uncertainty a declaration carries has two axes: **who resolves** and **when they resolve**. The same declaration, even with the same strategy, can give two distinct sets at two separate times. The install's date is part of its output. The four locked installs still give **1 distinct outcome**. The lock closes both axes at once, because what it writes is not a rule or a range, it is the versions themselves. This is a result that requires reading the two tables together: as the number of installs made with the declaration grows, the number of distinct outcomes grows with it — two with two installs, three with four — while on the locked side the number stays at **1** no matter how many installs are made. The last two lines show where the lock comes from. The lock equals the result of the `newest` resolution — producing it required the resolution to run. And when that same lock is regenerated on the later registry, `1.3 / 3.3 / 1.2` comes out. So a lock is a frozen resolution; when the resolution is run again, what it froze changes too. #### The Lock's Scope The declaration's and the lock's lengths are almost never the same, and the reason sits outside the measurement. The declaration writes what the project **directly** asks for. The installed set is often larger, because every requested package has its own declaration, and that declaration brings in other packages too. These are called **transitive dependencies**. The project's code never names them, but they sit on the site at run time and take part in its behavior. The lock file writes the **entirety** of the installed set; not what is directly requested, every package that lands on the site. This is why a lock is far longer than a declaration and is not a document meant to be hand-written. In this lesson's rig, packages do not require each other, which is why the two sets coincide and the tables can be read in three rows. Where the gap opens is the next lesson's measurement. The lock's second field looks under the version name. A version number is a **label**; it does not on its own say the content under that label has not changed over time. Lock files can therefore write a **content hash** next to the version and compare it against what is downloaded at install time. When the version name matches but the hash does not, what has surfaced is not a version problem, it is the same name carrying two separate contents — a distinct outcome a version number could never see. The third is scope itself. A project's declaration usually holds more than one group: what is needed to run, what is needed to test, what is needed to generate docs. These are separate sets and are installed separately. If a lock does not write which groups it covers, the same lock can produce two distinct sets across two separate installs — with both installs still conforming to the lock. A lock has to fix not only versions, but also **which question it answers**. #### A Lock Does Not Eliminate Resolution Every row of the measurement gathers into one sentence: **a lock does not eliminate resolution; it makes the decision once and writes it down.** Three direct consequences follow. First, a lock is not a guarantee, it is a **record**; it does not say how good the frozen set is, only which set was frozen. A bad resolution gets locked with the same fidelity. Second, renewing a lock is unavoidable work. If the range accepts new versions but the lock does not bring them in, the project is standing not somewhere the declaration permits, but at a point in the past. If that point was chosen deliberately, it is a decision; if it was stayed at unknowingly, it is an accumulation. The only thing separating the two is whether it is written down when and why the lock was renewed. Third, the declaration and the lock **can drift apart**. If the lock is hand-edited, or the declaration is changed and the lock is not renewed, the two documents say two separate things, and the install gives a distinct outcome depending on which one is read. The measurement's last line gives the test for this: regenerated from the same declaration with the same strategy, a lock should come out identical to itself. If it does not, either the registry has changed or one of the documents has been touched by hand — and which one is told apart by looking at the registry's view. #### Summary - A **dependency declaration** states not a version but a **range**; the range is the condition for coming together with other packages, and its cost is that the choice is made outside the declaration. - The same declaration gives **2 distinct version sets** when the resolution strategy changes: `1.2 / 3.2 / 1.1` versus `1.0 / 3.0 / 1.0`. Both conform to the declaration. - A **lock file** writes a resolution's result with exact versions; a locked install gives **1 distinct outcome** even when the strategy changes, because there is no range left to read. - When the registry view also changes, four installs made with the declaration give **3 distinct sets**; the uncertainty has two axes — who resolves and when. - A lock writes not what is directly requested but **the entirety of the installed set**; if it does not also write which groups it covers, the same lock can produce two distinct sets. - **A lock does not eliminate resolution; it makes the decision once and writes it down.** A lock is a record, not a guarantee, and the set it freezes changes when it is renewed. #### Next Step In this lesson's measurement, packages were chosen independently of each other: each was taken from its own range, and the choices never looked at one another. In a real set, packages carry their own declarations, and those can contradict each other. The next lesson measures that situation: what sets does the same declaration give once package requirements are also taken into account, where does it become visible when a strategy picks an inconsistent set, and what does resolution return when no consistent set exists at all? ### Package Managers Source: https://academia.sh/en/courses/python-projects/package-managers When packages require each other, the same declaration gives 4 distinct sets under four strategies, and only 3 of those are consistent; once constraints tighten, two backtracking strategies find no solution and the consistent-set count drops to 0, while a lock file brings all four down to 1. The previous lesson's measurement chose every package independently from its own range; the choices never looked at one another. That assumption made the table readable, but left out a fact: installed packages carry their own declarations too, and those can contradict each other. This lesson puts those constraints into the table. Its question has two layers: **what sets does the same declaration give once package requirements are also taken into account**, and where does it become visible when a strategy picks an inconsistent set? The second half of the answer will turn out to matter more than the first. #### What a Package Manager Does A **package manager** is the name of a tool class, and this course writes no product name; its behavior is **modeled** here. Its work splits into four steps: read the declaration, gather candidate versions from the registry, choose a set among the candidates, install the chosen set to the site. This lesson's concern is the third step. Where the fourth step writes was established in the first lesson: on an isolated site, installation changes only what that project sees; on a shared site, it also changes what neighbors see. This lesson assumes the site is isolated and does not return to that axis again; every distinct outcome here is born inside a single site. The third step alone has a name: **resolution**. Its input is a list of ranges and a pool of candidates; its output is one exact version per package. What stands in between is the package manager's **resolution strategy**, and as the previous lesson showed, that strategy is not written in the declaration. What a version number's parts mean, and meaningful versioning, were established in the Introduction to Version Control course's tagging lesson; not repeated here. The question here is not what a version number says, it is **what sets the ranges together make possible**. #### The Requirement Table and Consistency A single addition is made to the rig: a **requirement table**. The table writes which range each package version wants from which package. The registry and the project's declaration come from the shared definition unchanged; what is added is the packages' own requirements. The tension the table sets up in the rig reads in one sentence: `metrics`'s newer versions want `common`'s newer versions, while `report`'s newer version wants `common`'s older versions. An install requesting both at their newest ends up wanting two separate things for `common` at once. This is where the measurement's second concept is born. A set is **consistent** if and only if every chosen version's requirement is met by the other choices in the set. Consistency is not the same thing as installation succeeding: an inconsistent set can still be installed to the site, because installation is a file write and is not required to read the requirement table. #### Four Strategies The measurement compares four strategies. The first two are **independent** pickers: each takes a package from one end of its own range and never looks at the requirement table — these are the previous lesson's two strategies. The remaining two are **backtracking**: they choose a version, test the constraints, back off when it does not hold, and try another version. The two backtracking strategies differ only in **search order**: which package's version gets decided first. Same declaration, same requirement table, same registry — the only thing that changes is the order. This difference is going to produce the measurement's most important row. The measurement's assumptions: - **ED24** — The registry, declaration, and range rule are taken from the shared definition unchanged; the lesson does not change either's behavior. The only addition is the requirement table. - **ED25** — The requirement table is part of the rig and part of the oracle: we wrote which version wants what, so every set's consistency is known without measuring. - **ED26** — A package version carries at most one requirement, and a requirement is a range. The chain's depth is one level; deeper chains would grow the number, not change the rule. - **ED27** — Independent strategies do **not read** the requirement table. The set they build can be inconsistent, and installation still completes. - **ED28** — Backtracking tries candidates for every package in the search order, following the strategy's direction, and tests the partial set at every step. It returns the first consistent complete set. - **ED29** — Testing a partial set checks only requirements whose **both ends are already chosen**; a requirement whose target is not yet chosen does not count as violated at that step. - **ED30** — Two search orders are tried: `metrics, common, report` and `report, common, metrics`. Order is a configuration detail; the project does not write it in the declaration. - **ED31** — The tight requirement table is derived on the same registry by having all of `metrics`'s eligible versions want `common`'s newest version, and all of `report`'s eligible versions want an older one. It models a declaration set whose intersection is empty. - **ED32** — When no solution is found, backtracking returns `None`, and this is **not an error, it is a measurement result**: the consistent-set count is zero. - **ED33** — The lock is a copy of one of the consistent sets — the backtracking newest strategy's result. Resolution is not called while a lock exists. - **ED34** — Version numbers are the rig packages' fictional versions. - **ED35** — **No real installation is performed, no real package is downloaded;** resolution returns a dict. - **ED36** — Duration is not measured. What is counted is the **distinct set**, the **distinct consistent set**, and the **distinct outcome** the locked install gives. #### Measurement ```python """Resolution strategies: same declaration, separate set; what happens when conflict arises.""" REGISTRY = { "metrics": [(1, 0), (1, 1), (1, 2), (2, 0)], "report": [(0, 9), (1, 0), (1, 1)], "common": [(3, 0), (3, 1), (3, 2), (4, 0)], } DECLARATION = {"metrics": ((1, 0), (2, 0)), "report": ((1, 0), (2, 0)), "common": ((3, 0), (4, 0))} PACKAGES = ("metrics", "common", "report") # Packages' own dependencies: version -> (target package, lo, hi) REQUIREMENT = { ("metrics", (1, 0)): ("common", (3, 0), (4, 0)), ("metrics", (1, 1)): ("common", (3, 1), (4, 0)), ("metrics", (1, 2)): ("common", (3, 2), (4, 0)), ("report", (0, 9)): ("common", (3, 0), (3, 1)), ("report", (1, 0)): ("common", (3, 0), (4, 0)), ("report", (1, 1)): ("common", (3, 0), (3, 2)), } # Same registry, a tighter requirement table: all three metrics versions want a newer common. TIGHT = dict(REQUIREMENT) TIGHT[("metrics", (1, 0))] = ("common", (3, 2), (4, 0)) TIGHT[("metrics", (1, 1))] = ("common", (3, 2), (4, 0)) TIGHT[("report", (1, 0))] = ("common", (3, 0), (3, 2)) def eligible(package): lo, hi = DECLARATION[package] return [s for s in REGISTRY[package] if lo <= s < hi] def resolve(strategy="newest"): """Independent choice: each package taken from one end of its own range.""" chosen = {} for package in sorted(DECLARATION): candidates = eligible(package) chosen[package] = candidates[-1] if strategy == "newest" else candidates[0] return chosen def consistent(chosen, requirement): """Do the chosen versions' requirements get met by the other chosen packages.""" for package, version in chosen.items(): r = requirement.get((package, version)) if r is None: continue target, lo, hi = r if target in chosen and not lo <= chosen[target] < hi: return False return True def backtrack(order, requirement, strategy="newest"): """When a conflict arises, back off and try another version; order is the search order.""" def advance(i, chosen): if i == len(order): return dict(chosen) package = order[i] candidates = eligible(package) if strategy == "newest": candidates = candidates[::-1] for version in candidates: chosen[package] = version if consistent(chosen, requirement): result = advance(i + 1, chosen) if result is not None: return result del chosen[package] return None return advance(0, {}) def format_selection(chosen): if chosen is None: return f"{'no solution':>23s}" return " ".join(f"{'.'.join(map(str, chosen[p])):>7s}" for p in PACKAGES) def install(lock_file, func, requirement): """If a lock exists, no resolution runs; otherwise the strategy decides.""" return dict(lock_file) if lock_file else func(requirement) STRATEGIES = ( ("independent, newest", lambda g: resolve("newest")), ("independent, oldest", lambda g: resolve("oldest")), ("backtracking, newest", lambda g: backtrack(("metrics", "common", "report"), g)), ("backtracking, report first", lambda g: backtrack(("report", "common", "metrics"), g)), ) def table(heading, requirement): print(f"{heading:<26s} {'metrics':>7s} {'common':>7s} {'report':>7s} consistent") sets, sound = [], [] for name, func in STRATEGIES: chosen = func(requirement) status = chosen is not None and consistent(chosen, requirement) if chosen is not None: sets.append(tuple(sorted(chosen.items()))) if status: sound.append(tuple(sorted(chosen.items()))) print(f"{name:<26s} {format_selection(chosen)} {'yes' if status else 'no':>7s}") print(f"four strategies: {len(set(sets))} distinct sets, " f"{len(set(sound))} distinct consistent sets") table("requirement table", REQUIREMENT) print() table("tight requirement table", TIGHT) print() lock_file = backtrack(("metrics", "common", "report"), REQUIREMENT) locked = {tuple(sorted(install(lock_file, func, REQUIREMENT).items())) for _, func in STRATEGIES} print("set written to the lock: " + format_selection(lock_file)) print(f"with the lock file, four strategies: {len(locked)} distinct set") ``` ``` requirement table metrics common report consistent independent, newest 1.2 3.2 1.1 no independent, oldest 1.0 3.0 1.0 yes backtracking, newest 1.2 3.2 1.0 yes backtracking, report first 1.1 3.1 1.1 yes four strategies: 4 distinct sets, 3 distinct consistent sets tight requirement table metrics common report consistent independent, newest 1.2 3.2 1.1 no independent, oldest 1.0 3.0 1.0 no backtracking, newest no solution no backtracking, report first no solution no four strategies: 2 distinct sets, 0 distinct consistent sets set written to the lock: 1.2 3.2 1.0 with the lock file, four strategies: 1 distinct set ``` #### Four Strategies, Four Sets The top table's four rows give **4 distinct sets**, all from the same declaration. This adds two more to the previous lesson's two sets; the two added are sets independent choice could never reach. The first row is this lesson's real finding. The independent `newest` strategy chooses the set `1.2 / 3.2 / 1.1`, and this set is **inconsistent**: `report 1.1` wants an older version than `3.2` for `common`, but the set holds `common 3.2`. Even so, the install completes. No error message is raised, because the independent strategy never read the requirement table. The result is three packages sitting on the site that do not want each other. The second row shows independent `oldest` happening to build a consistent set. The word "happening" is exactly right here: the strategy still did not read the requirement table, it is only that the ranges' lower ends did not clash. The same strategy would build an inconsistent set with a different table — and indeed it does, in the table below. The last two rows give the backtracking strategies, and both are consistent, but **separate**: `1.2 / 3.2 / 1.0` versus `1.1 / 3.1 / 1.1`. The only difference between them is search order — which package's version gets decided first. Same declaration, same registry, same constraints; still two separate, legitimate installs. This row says **the resolver itself is a distinct-outcome axis**: changing the tool can change the install without changing the declaration. Total: four strategies, **4 distinct sets**, **3** of which are consistent. #### Where Does a Conflict Get Written The bottom table tightens the constraints: in this table, all of `metrics`'s eligible versions want `common`'s newest, and all of `report`'s eligible versions want an older one. The two requests' intersection is empty. Both backtracking strategies return **no solution**. The consistent-set count is **0**. This looks like a failure, and in one sense it is — but the measurement's real finding sits in the two rows above it: the independent strategies still return a set. The same contradiction produces either a rejection or a silent install, depending on which tool reads it. The difference between the two is **where the error surfaces**. A rejection surfaces at resolution time and leaves information behind: this set of declarations is contradictory, either a range has to be loosened or a package changed. A silent install leaves nothing behind; the contradiction gets installed to the site and only surfaces at run time, on the line where the two conflicting packages meet. And that line is not hit on every run. In the measure's own language: rejection produces **0 distinct outcomes**, and zero is a countable answer. A silent install produces **1 distinct outcome**, but does not measure whether that outcome is valid. A resolver saying it could not solve something is, information-wise, superior to one that does not say so. #### Is the Resolver Itself Reproducible The measurement compared four strategies, but it is also worth asking how many outcomes each strategy gives **on its own**. All four of the model's strategies are deterministic: the candidate list comes from the registry in order, the search order is fixed, there is no randomness anywhere. The same strategy called with the same input gives **1 distinct outcome**. This is the most basic property to ask of a resolver, and it does not come for free. If the candidate pool is gathered from more than one source, which source answers first can slip into the order; if a cache warms up, the same query can return a separate list; parallel downloads can change install order. None of these change the strategy's rule, but all of them change its **input** — and a rule whose input changes gives a separate set. A resolver's word is therefore two-layered. The first layer is the rule: which candidate gets chosen. The second layer is the rule's input: in what order, from what source, and at what moment the candidates were gathered. A reproducible install requires fixing both, and the document writing down the second layer is the lock file. #### The Lock's Place in This Table The last two lines put the previous lesson's result on top of this table. When one of the consistent sets — the one the backtracking newest strategy found, `1.2 / 3.2 / 1.0` — is written to the lock, all four strategies install the same set: **1 distinct outcome**. The axis the lock closes here is a new one. What was closed in the previous lesson was strategy direction and the registry's time; what is closed here is **the resolver itself**. Once the lock is written, which package manager is used no longer changes the outcome, because there is nothing left to resolve. This also bounds how much work the lock does. The lock carries the decision of whichever resolver produced it, with equal fidelity for a good decision or a bad one. Which of the three consistent sets in the table above got written to the lock cannot be told by looking at the lock — the lock writes only what was chosen, not why it was chosen. #### Summary - A **package manager** is a tool class; its work is reading the declaration, gathering candidates, choosing a set, and installing it. The step producing a distinct outcome is the choosing step. - Once packages' own requirements are taken into account, the same declaration gives **4 distinct sets** under four strategies, and **3** of them are consistent. - Independent choice does not read the requirement table: the set `1.2 / 3.2 / 1.1` installs despite being **inconsistent**, and raises no warning. - The two backtracking strategies give two separate consistent sets because of separate **search order**; the resolver itself is a distinct-outcome axis. - Under constraints with an empty intersection, backtracking returns **0 consistent sets**; rejection leaves information at resolution time, a silent install defers the contradiction to run time. - A lock file brings four strategies down to **1 distinct outcome**, but does not carry the reasoning behind the decision it writes. #### Next Step Every set in this lesson was comparable because a single interpreter was assumed. That assumption, like the previous two, was not written down either. The next lesson puts it into the measurement: same project, same declaration, separate interpreter versions. What is going to be asked is — how many distinct sets does resolution give depending on the interpreter, and what does a lock file produced on one interpreter do on another? ### Version Management Source: https://academia.sh/en/courses/python-projects/version-management The same declaration gives 2 distinct version sets across three rig interpreter versions; of nine installs made from lock files produced on one interpreter, 6 are accepted and 3 rejected, and the same lock gives 1 distinct outcome on a fixed interpreter but produces one rejection across three interpreters. The previous three lessons measured how many distinct version sets the same declaration gives across three separate axes: the isolation boundary, the resolution strategy, and the resolver itself. All three carried one unwritten assumption — a single interpreter. While package versions were being compared, it was accepted that all of them could be installed on the same interpreter. This lesson puts that assumption into the measurement. Its question has two parts: **how many distinct outcomes does the same project give across separate interpreter versions**, and what does a lock file produced on one interpreter do on another? The second part's answer will pin down exactly what kind of guarantee a lock is. #### The Interpreter Is a Dependency Too A project counts its packages when writing its own declaration; it often skips counting the interpreter. But the interpreter is something installed too, it has a version, and when that version changes, the behavior the project meets can change. This lesson **writes no real version number.** Interpreters are referred to with fictional labels: `Y1`, `Y2`, `Y3`. The labels' order is a time order — `Y1` the oldest, `Y3` the newest — and implies nothing beyond that. Package versions are fictional too, as in the previous three lessons. The measurement sets up a simple table: every package version declares which interpreter versions it can be installed on. This is not fiction, it is a real constraint; a package version is written against a particular language level, and on an interpreter not meeting that level, it either cannot be installed or does not run. In the rig, this relationship is written as a set and is **modeled**; no product name appears. A direct consequence follows: the interpreter **narrows** the candidate pool. The declaration has not changed, the registry has not changed, the strategy has not changed; but the list of versions selectable from the same range shortens depending on the interpreter. Resolution applies the same rule to a separate pool and returns a separate set. #### The Environment's Dependence on the Interpreter The first lesson said a virtual environment does not duplicate the interpreter, it attaches to it. This measurement shows the cost of that sentence. An environment is meaningful only with the interpreter it was built on. When the interpreter version changes, the environment does not carry over: part of the site's packages may not have been built for the new interpreter, and the import path is also named after the interpreter version. The switch is therefore not an upgrade, it is **setting up a new environment** — and what gets installed into the new one is resolved afresh. Does the same hold for a lock file? A lock writes exact versions and does not resolve; so it ought to install the same set on the new interpreter too. This is exactly the claim the measurement tests, and the answer cannot be given in one word. The measurement's assumptions: - **ED37** — The registry, declaration, and range rule are taken from the shared definition unchanged; the lesson does not change either's behavior. The only addition is the compatibility table. - **ED38** — Three interpreter versions are fictional and referred to by the labels `Y1`, `Y2`, `Y3`. **No real version number is written.** - **ED39** — The compatibility table is part of the rig and part of the oracle: we wrote which interpreters each package version can be installed on, so every row's correct result is known without measuring. - **ED40** — Compatibility is a binary decision: a version either can or cannot be installed on an interpreter. Partial working or install-with-warning is not modeled. - **ED41** — Resolution narrows the candidate pool first by the declaration, then by the compatibility table. Order does not change the result; both filters take an intersection. - **ED42** — Strategy is fixed in this lesson: **newest eligible version**. The strategy axis was measured in the previous two lessons and is not repeated here. - **ED43** — Packages do not require each other in this lesson; the requirement axis was measured in the previous lesson. The one new axis measured is the interpreter. - **ED44** — A lock file writes exact versions and does not resolve. Installing with a lock requires every written version to be installable on the target interpreter. - **ED45** — Install rejection is not an error, it is **a measurement result**, and it is counted as `reject` in the table. A rejected install produces no distinct outcome. - **ED46** — Every interpreter's lock is produced from its own resolution; nine installs are the three locks applied to the three interpreters. - **ED47** — Switching between interpreter versions is rebuilding the environment; the old site does not carry over, in the model either. - **ED48** — **No real installation is performed, no real package is downloaded, no real interpreter is invoked.** An interpreter is a label, an install is a dict. - **ED49** — Duration is not measured. What is counted is the **distinct version set**, the **accept**, and the **reject** count. - **ED50** — The measurement is a single run, deterministic, and there is no randomness in the rig. #### Measurement ```python """Interpreter versions: same project, same declaration, how many distinct outcomes.""" REGISTRY = { "metrics": [(1, 0), (1, 1), (1, 2), (2, 0)], "report": [(0, 9), (1, 0), (1, 1)], "common": [(3, 0), (3, 1), (3, 2), (4, 0)], } DECLARATION = {"metrics": ((1, 0), (2, 0)), "report": ((1, 0), (2, 0)), "common": ((3, 0), (4, 0))} PACKAGES = ("metrics", "common", "report") # Fictional interpreter versions; not real version numbers. INTERPRETERS = ("Y1", "Y2", "Y3") # Package version -> interpreters it supports COMPATIBILITY = { ("metrics", (1, 0)): {"Y1", "Y2"}, ("metrics", (1, 1)): {"Y1", "Y2"}, ("metrics", (1, 2)): {"Y2", "Y3"}, ("metrics", (2, 0)): {"Y3"}, ("common", (3, 0)): {"Y1", "Y2"}, ("common", (3, 1)): {"Y1", "Y2", "Y3"}, ("common", (3, 2)): {"Y2", "Y3"}, ("common", (4, 0)): {"Y3"}, ("report", (0, 9)): {"Y1"}, ("report", (1, 0)): {"Y1", "Y2"}, ("report", (1, 1)): {"Y2", "Y3"}, } def eligible(package, interpreter): lo, hi = DECLARATION[package] return [s for s in REGISTRY[package] if lo <= s < hi and interpreter in COMPATIBILITY[(package, s)]] def resolve(interpreter, strategy="newest"): chosen = {} for package in sorted(DECLARATION): candidates = eligible(package, interpreter) if not candidates: return None chosen[package] = candidates[-1] if strategy == "newest" else candidates[0] return chosen def lock(chosen): return dict(chosen) def install(lock_file, interpreter): """A lock writes every version exactly; if the interpreter does not support it, install is rejected.""" if all(interpreter in COMPATIBILITY[(p, s)] for p, s in lock_file.items()): return dict(lock_file) return None def format_selection(chosen): if chosen is None: return f"{'rejected':>23s}" return " ".join(f"{'.'.join(map(str, chosen[p])):>7s}" for p in PACKAGES) def format_short(chosen): if chosen is None: return f"{'reject':>12s}" return f"{'/'.join('.'.join(map(str, chosen[p])) for p in PACKAGES):>12s}" def key(chosen): return tuple(sorted(chosen.items())) print(f"{'resolution by declaration':<26s} {'metrics':>7s} {'common':>7s} {'report':>7s}") resolutions = {} for interpreter in INTERPRETERS: resolutions[interpreter] = resolve(interpreter) print(f"{'interpreter ' + interpreter:<26s} " + format_selection(resolutions[interpreter])) print(f"same declaration, three interpreters: {len({key(s) for s in resolutions.values()})} " f"distinct version sets") print() print(f"{'interpreter the lock was produced on':<30s} " + " ".join(f"{'install ' + interpreter:>12s}" for interpreter in INTERPRETERS)) accepted, sets = 0, [] for source in INTERPRETERS: current_lock = lock(resolutions[source]) cells = [] for target in INTERPRETERS: s = install(current_lock, target) accepted += s is not None if s is not None: sets.append(key(s)) cells.append(format_short(s)) print(f"{'lock produced on ' + source:<30s} " + " ".join(cells)) print(f"nine installs: {accepted} accepted, {9 - accepted} rejected; among the accepted " f"{len(set(sets))} distinct version sets") print() lock_y2 = lock(resolutions["Y2"]) fixed = [install(lock_y2, "Y2") for _ in range(3)] roaming = [install(lock_y2, interpreter) for interpreter in INTERPRETERS] for name, installs in (("same lock, three installs on Y2", fixed), ("same lock, three interpreters", roaming)): succeeded = [s for s in installs if s is not None] print(f"{name:<30s} {len({key(s) for s in succeeded})} distinct outcome, " f"{len(installs) - len(succeeded)} rejected") ``` ``` resolution by declaration metrics common report interpreter Y1 1.1 3.1 1.0 interpreter Y2 1.2 3.2 1.1 interpreter Y3 1.2 3.2 1.1 same declaration, three interpreters: 2 distinct version sets interpreter the lock was produced on install Y1 install Y2 install Y3 lock produced on Y1 1.1/3.1/1.0 1.1/3.1/1.0 reject lock produced on Y2 reject 1.2/3.2/1.1 1.2/3.2/1.1 lock produced on Y3 reject 1.2/3.2/1.1 1.2/3.2/1.1 nine installs: 6 accepted, 3 rejected; among the accepted 2 distinct version sets same lock, three installs on Y2 1 distinct outcome, 0 rejected same lock, three interpreters 1 distinct outcome, 1 rejected ``` #### Two Sets, Three Interpreters The top table resolves the same declaration on three interpreters and gives **2 distinct version sets**. The `Y1` row gives `1.1 / 3.1 / 1.0`. These are not the newest of all three; `metrics 1.2`, `common 3.2`, and `report 1.1` cannot be installed on this interpreter and drop out of the pool. The strategy is still "newest eligible version"; what changes is what counts as eligible. `Y2` and `Y3`, in turn, give the same set: `1.2 / 3.2 / 1.1`. Two observations follow from this. First, the interpreter axis **on its own** produces a distinct outcome: two installs were reached without touching the declaration, the strategy, or the registry. Second, the number is not the same as the interpreter count. Three interpreters gave two sets because `Y2` and `Y3` allow the same candidates. The distinct-outcome count is a function not of how many environments there are, but of **the compatibility table**. The `Y1` row has one more side effect. A project running on that interpreter never sees the packages carrying the defect fixes made in later versions, even if its declaration would accept them. Keeping an interpreter in place is also keeping package versions in place. #### A Lock Is Produced for an Interpreter The middle table gives nine installs: three locks, three target interpreters. **6 accepted, 3 rejected.** Every row has its own pattern. The lock produced on `Y1` installs on `Y1` and `Y2`, and is **rejected** on `Y3` — `metrics 1.1` and `report 1.0` cannot be installed on that interpreter. The locks produced on `Y2` and `Y3` write the same set, and both are rejected on `Y1`. Reading what the rejection is has to be done correctly. This is not an upgrade error; it is the lock's **correct** behavior. A lock writes exact versions and does not negotiate; if the version it writes cannot be installed on the target environment, its only option is to stop. The alternative would be silently bending it and installing a different version — at which point it would stop being a lock. The six accepted installs give **2 distinct version sets**. This number is the same as the top table's, and that is not a coincidence: the locks were also produced from those same resolutions. A lock does not create a new set, it carries an existing one. #### The Axis a Lock Guarantees The bottom two rows are the lesson's conclusion, and write exactly what kind of guarantee a lock is. The same lock, installed three times on a **fixed** interpreter, gives **1 distinct outcome, 0 rejected**. Installed on three separate interpreters, it gives **1 distinct outcome** again, but **1 rejected**. The lock never produced a second set anywhere — it kept its word. What it could not keep was something else: it could not keep its word **in every environment**. The lock's limit follows from this: **a lock guarantees reproducibility within one interpreter; it does not guarantee it across the interpreter axis.** On that axis, a lock is not a guarantee, it is a **constraint** — and being one is not a defect, it is its function. A lock's inability to travel is a declaration that it should not travel. The practical counterpart is a direct rule: a lock file should also write the interpreter version it was produced on. Without that, two documents are left on hand — one says which versions to install, the other does not say where it is valid — and the rejection only surfaces at install time. The declaration carrying a range for the interpreter too follows the same reasoning: if the project does not say which interpreters it runs on, the compatibility table can only be discovered by attempting the install. A third consequence sits outside the numbers. An interpreter switch looks like maintenance work, but the measurement shows it is a **resolution** job: on switching, the lock is regenerated, the new set can come out separate from the old one, and the project has not been tested against that new set. The switch's cost is not setting up the interpreter, it is **validating the new set**. #### Interpreters Standing Side by Side The measurement's three rows assume three interpreters can exist on one machine at once, and this assumption matches reality. Tools that keep multiple interpreter versions side by side form a class — a **version manager** — and their job is a choice: which interpreter gets invoked in which directory. Product names are not written here either. Where the choice is made matters. A virtual environment attaches to an interpreter the moment it is set up; from that point on, which interpreter a project uses is not a configuration setting, it is **the environment's identity**. A version manager chooses which interpreter is used **when a new environment is set up** — it does not hand an existing environment off to another interpreter. The switch flow is therefore five steps, and none can be skipped: install the new interpreter, set up a new environment on it, resolve the declaration again, produce a new lock, test the project against the new set. Only after this is the old environment discarded. The measurement's middle table shows exactly why the third and fourth steps cannot be skipped: installing the old lock on the new interpreter produces a rejection in one of three cases. A project supporting more than one interpreter at once is read from the same table too. In the measurement, every interpreter has its own lock: **three locks, 2 distinct sets.** Supporting two interpreters at once means keeping **two locks**, not fitting a single one to both. A single lock only covers interpreters sharing every version's compatibility set — in the table, `Y2` and `Y3` are such a pair, `Y1` and `Y3` are not. Standing side by side has a cost too, and it sits in the same table: a separate site, install, and lock for every supported interpreter. All three are paid separately for every promise the project makes; as supported versions grow, so does the number of sets to pin down, and each one waits to be validated on its own. #### Summary - The interpreter is a dependency too, and it narrows the candidate pool; **2 distinct version sets** are born without the declaration, registry, or strategy changing. - The distinct-outcome count depends not on how many environments there are but on the **compatibility table**: three interpreters give two sets because two of them allow the same candidates. - A lock produced on one interpreter can be rejected on another: **6** of nine installs are **accepted, 3 rejected**. Rejection is not an error, it is the lock's correct behavior. - **A lock guarantees reproducibility within one interpreter, not across the interpreter axis;** on that axis it is a constraint, not a guarantee. - Supporting two interpreters at once means not fitting one lock to both, it means **keeping two locks**; a single lock covers only interpreters sharing a common compatibility set. - An interpreter switch is rebuilding the environment, and the lock is regenerated; its cost is not installation, it is **validating the new set**. #### Next Step Across this topic, four axes were measured — the isolation boundary, the resolution strategy, the resolver itself, and the interpreter version — and all four were made uniform by the same method: make the decision once, write it down, have later runs read it. How many distinct outcomes an install gives can now be counted. What remains is the actual thing inside the install: **the code**. Once the environment has been brought down to a single set, does the code running in that set give the same decision on every run? The next topic asks that question, and to measure the answer, it takes up what a test is, what a suite declares, and how many distinct decisions the same suite can give. ## Testing Unit test writing styles, taking repetition out of a test with fixtures and parametrization, isolating external dependencies, making examples testable, and what coverage says and does not say. ### Test Frameworks Source: https://academia.sh/en/courses/python-projects/test-frameworks A framework does four jobs, and only one of them produces the verdict: the same three cases give 1 distinct outcome across two writing styles, while the same file gives 3 distinct outcomes in found-test terms across three discovery rules, and the report stays the same across all three. The previous lesson made the environment and dependencies singular: the interpreter version got pinned, the lock file wrote down the resolution's decision, and the same declaration now gives a single version set. No question is left to ask on the setup side. But the setup becoming singular does not mean the code running on top of it gives the same verdict every time. This lesson's question: does the code itself give the same result on every run, and who tells us that? What tells us is a **test framework**. A framework is not a product, it is a **class of tool**; no product name appears in this lesson. A framework's jobs get modeled within the lesson, while the framework the standard library provides gets used directly. #### The Framework's Four Jobs A test framework looks like it does one job — "runs the tests" — but it actually does four separate jobs, and only one of them decides the outcome. **Discovery.** Decides which callables count as tests. This decision rests on a naming convention, a directory pattern, or a marker, and **does not look at the test's content**. The rule's source can be the framework's default, or the project's setting; either way, the rule sits outside the test. **Run.** Calls every discovered test, decides the order, and inserts setup and teardown steps in between when needed. Order itself is a decision too, and it is what the next lesson measures. **Assertion.** Compares the expected against the actual inside the test. In Python, the plainest form of this is the `assert` statement; when the statement comes out false, it raises `AssertionError`, and the framework counts this exception as a failed test. Using an exception as flow control was built in the Python Fundamentals course; here the same mechanism gets used for reporting. **Report.** Writes how many tests ran, how many passed, and the names of the ones that failed; the reader takes the verdict from these lines. Of these four jobs, only **assertion** says anything about correctness. Discovery, run, and report are **configuration decisions** — and what this lesson measures is what configuration decisions do to the outcome. All four sit together, so they look like one job. Yet only assertion is written in the project's source files; the discovery rule and run order sit, in most setups, in a separate configuration file or in the run command's options. Part of the decision is inside the code, part is outside — and when the outside part changes, no trace stays in the code files. For this course's question, this split is exactly what is being sought: **same source, separate configuration, how many distinct outcomes.** #### The Assertion Needs an Oracle An assertion compares two things: the value the code produces, against the **expected** value. Where the expected value comes from is not visible inside the test; whoever wrote it got it from somewhere. This source is called an **oracle**, built as a concept in the Quality and Test Fundamentals course of the Software Quality and Testing curriculum. Theory is not repeated here; what the oracle **adds to the measurement** is. In our shared setup, the oracle is us: we wrote the correct discount behavior, the clean environment, and the correct version set. This lesson's test cases, though, are written not from the oracle, but from the function's **observed behavior**. That is, the expected values were obtained by writing down whatever the function gives today. A test written this way verifies that the function's behavior has **not changed**; it does not verify that the behavior is **correct**. What this difference costs gets measured in this topic's last lesson; for now, it is enough to note: **a test passing does not say the expected value is correct.** How the assertion is written is a choice too. When the `assert` statement's sentence comes out false, it raises `AssertionError`, and the report is left with only "the statement at this line came out false." The framework's comparison methods, though, capture both values and write them to the report. This difference shortens diagnosis time but does not change the verdict; the measurement's upper table shows exactly this. #### A Test Without an Assertion Passes Silently The modeled runner's rule is a simple one: call the test, count it as passed if `AssertionError` does not come. This rule has a direct consequence — **a test containing no assertion at all always passes.** The runner does not count the assertion, it counts the exception; if no assertion is ever written, there is no exception to raise either. This is not the framework's defect, it is its **limit**. The framework does not look at the test's body; if it did, it would have to know which call counts as an assertion, and that knowledge is application-specific. `helper_amount` in the measurement is a concrete example: it carries no assertion, so it would count as passed even if discovered. The same holds for a test whose assertion was accidentally commented out, or left inside a condition that never gets evaluated. The consequence: a passing test may have passed for at least **two separate reasons** — the assertion held, or there was no assertion at all. The report does not tell the two apart. To tell them apart, the place to look is not the report, it is the test itself; and a mechanism that measures this over the code gets built in this topic's last lesson. #### The Same Three Cases, Two Writing Styles A unit test has two common writing styles. In the first, every test is a function, and the assertion is written with the `assert` statement; the runner calls the function and counts the test as failed if it catches an exception. In the second, tests are a class's methods, and the assertion is written with the framework's comparison methods; the standard library's `unittest` framework uses this style. The difference between the two is a **writing** difference. The second style writes both values to the report on a failing assertion; the first only says the statement came out false. This is a report detail and does not change the verdict itself — the measurement shows this. The class-based style has a second contribution: hooks that run before and after every test can be defined. These hooks take repetition out of the test and are the next lesson's subject; here, only three plain tests are written, so no hook gets used, and the two styles genuinely do the same job. This was necessary for the comparison to be meaningful: the two styles can only be set side by side once they run the same three cases with the same setup. #### What the Discovery Rule Decides The discovery rule decides which callables in a file count as tests. When the rule changes, the file's content does not change, the **set of running tests** does. The file in the measurement carries five callables: three with a `t_` prefix, one with a `check_` prefix, and one a helper function. These three rules model a real situation. As a team grows, tests get written by more than one person, and not everyone carries the same naming habit; some use a `test_` prefix, some `t_`, some a `check_`-like prefix. If the discovery rule looks for only one of these, the others sit in the file, visible to the eye, tracked in version control — and **never run**. What matters here: a test that does not run does not count as failed. The framework never saw it, so it has no place in the report. The report says "no failures," and this sentence is the same whether five tests ran or none did. The discovery rule, for this reason, is not a detail, it is **part of the verdict**. The measurement's assumptions: #### The Measurement's Assumptions - **TE1** — The runner is modeled within the lesson: it calls every test, counts it as failed if it catches AssertionError, and does nothing else. No framework carrying a product name is used. - **TE2** — The class-based suite is run with the standard library's unittest framework; the report stream is swallowed, and only the numbers get read. - **TE3** — The three test cases are the shared setup's test set, and the expected values are written from the function's observed behavior, not from an oracle. - **TE4** — The discount function's behavior is not changed; the lesson only adds tests and a discovery rule. - **TE5** — The discovery rule is a name prefix and does not look at the test's body. Three rules get tried: the t_ prefix, the test_ prefix, and the combination of the t_ and check_ prefixes. - **TE6** — helper_amount in the file is not a test, it is a helper; no rule collects it, and if it did, it would count as passed because it carries no assertion. - **TE7** — Distinct outcome count is the number of mutually different result tuples observed across a measurement; duration is never measured in a run. - **TE8** — The report carries only the count of failed tests; a test that never runs has no name in the report, and the reader of the report cannot see this. #### The Measurement ```python """A framework does four jobs: discovery, run, assertion, and report. The runner is modeled within the lesson. Part 1 - the same three cases, two writing styles. Part 2 - the same file, three discovery rules. """ import io import unittest def discount(amount, member, coupon): """The shared setup's three-decision function; its behavior is not changed.""" rate = 0 if member: rate += 10 if coupon: rate += 15 if rate > 20: rate = 20 return amount - amount * rate // 100 CASES = (("member_only", (100, True, False), 90), ("coupon_only", (100, False, True), 85), ("both", (100, True, True), 80)) def runner(tests): """Modeled runner: calls every test, counts passed and failed.""" passed, failed = 0, [] for name, func in tests: try: func() except AssertionError: failed.append(name) else: passed += 1 return passed, failed def plain_suite(): def make(args, expected): def test(): assert discount(*args) == expected return test return [(name, make(args, exp)) for name, args, exp in CASES] class ClassSuite(unittest.TestCase): def test_member_only(self): self.assertEqual(discount(100, True, False), 90) def test_coupon_only(self): self.assertEqual(discount(100, False, True), 85) def test_both(self): self.assertEqual(discount(100, True, True), 80) def class_run(): suite = unittest.TestLoader().loadTestsFromTestCase(ClassSuite) result = unittest.TextTestRunner(stream=io.StringIO(), verbosity=0).run(suite) return result.testsRun - len(result.failures) - len(result.errors), len(result.failures) def t_member_only(): assert discount(100, True, False) == 90 def t_coupon_only(): assert discount(100, False, True) == 85 def t_both(): assert discount(100, True, True) == 80 def check_two_hundred(): assert discount(200, True, True) == 160 def helper_amount(amount): return amount FILE = {"t_member_only": t_member_only, "t_coupon_only": t_coupon_only, "t_both": t_both, "check_two_hundred": check_two_hundred, "helper_amount": helper_amount} RULES = {"t_ prefix": ("t_",), "test_ prefix": ("test_",), "t_ or check_": ("t_", "check_")} def discover(prefix): return [(name, f) for name, f in FILE.items() if name.startswith(prefix)] print(f"{'writing style':<22s} {'tests':>5s} {'passed':>6s} {'failed':>6s}") plain_passed, plain_failed = runner(plain_suite()) print(f"{'plain function suite':<22s} {len(CASES):5d} {plain_passed:6d} {len(plain_failed):6d}") class_passed, class_failed = class_run() print(f"{'class-based suite':<22s} {len(CASES):5d} {class_passed:6d} {class_failed:6d}") print(f"same three cases, two writing styles: " f"{len({(plain_passed, len(plain_failed)), (class_passed, class_failed)})} distinct outcomes") print() print(f"{'discovery rule':<18s} {'found':>8s} {'passed':>6s} {'failed':>6s} {'report':>16s}") found, report = [], [] for name, prefix in RULES.items(): tests = discover(prefix) passed, failed = runner(tests) found.append(len(tests)) report.append(len(failed)) print(f"{name:<18s} {len(tests):8d} {passed:6d} {len(failed):6d} " f"{'failed ' + str(len(failed)):>16s}") print(f"same file, three discovery rules: found tests {len(set(found))} distinct outcomes, " f"report {len(set(report))} distinct outcomes") print(f"tests that never run under the narrowest rule: {max(found) - min(found)}; " f"the report does not say this") ``` ``` writing style tests passed failed plain function suite 3 3 0 class-based suite 3 3 0 same three cases, two writing styles: 1 distinct outcomes discovery rule found passed failed report t_ prefix 3 3 0 failed 0 test_ prefix 0 0 0 failed 0 t_ or check_ 4 4 0 failed 0 same file, three discovery rules: found tests 3 distinct outcomes, report 1 distinct outcomes tests that never run under the narrowest rule: 4; the report does not say this ``` #### Reading the Numbers The upper table gives what the writing style costs: the same three cases, **1 distinct outcome** across two separate stylings. Both the plain function suite and the class-based suite pass 3 of 3 tests. Writing style is a preference, and it does not change the verdict. This does not mean choosing between frameworks is unimportant — but what the choice costs is not the outcome, it is the **report's detail**. The lower table gives what configuration costs, and the number changes. Same file, three discovery rules: found tests give **3 distinct outcomes** (3, 0, and 4), the report gives **1 distinct outcome**. In all three rules, the report says "failed 0." The second row is this measurement's real finding. The rule looking for the `test_` prefix finds **no test at all** in this file, zero tests run, and its report is "failed 0." This report is **indistinguishable** from the report where three tests passed. A suite being green is true under two separate conditions: the tests ran and passed, or no test ran at all. The third row shows the same fact from the other direction. A test named `check_two_hundred` sits in the file, its assertion is correct and it passes — but the default rule, looking only for the `t_` prefix, never collects it. The difference between the narrowest and widest rules is **4 tests**, and all of that difference stays outside the report. The rule that follows: **a test suite's verdict depends on the discovery rule as much as on the tests themselves.** The report counts the tests that ran, not the ones that did not; for this reason, the report's "failed 0" line is not proof on its own. Proof in full requires reading the **found test count** too — and, as the measurement shows, this number can take three separate values without the code itself ever changing. Read together, the two tables make the course's measurement axis visible for the first time here. Two separate questions got asked of the same input — same three cases, same function, same file. The first was "how did you write it," and it did not change the outcome: **1 distinct outcome**. The second was "what are you running," and it did change the outcome: **3 distinct outcomes**. What threatens reproducibility is not the first, it is the second, because the second is written not in a file, but in the setting that configures the run, and it does not get read alongside the code. This distinction's practical counterpart is clear too. A suite's green report only carries information once the count of running tests is known too; a green report read without that number looks the same in a setup where no test ran at all. For this reason, the running test count gets kept alongside the report, and its drop counts as an event as attention-worthy as a failing test. #### Summary - A test framework does four jobs: discovery, run, assertion, and report. Of these, only assertion says anything about correctness; the other three are configuration decisions. - An assertion takes its expected value from an oracle. A test passing does not show that expected value is correct. - The same three cases pass 3/3 with both the plain function suite and the class-based suite; the two writing styles give 1 distinct outcome, and the difference sits only in the report's detail. - The same file finds 3, 0, and 4 tests across three discovery rules — 3 distinct outcomes; the report says "failed 0" in all three and stays at 1 distinct outcome. - A test that never runs does not count as failed; under the narrowest rule, 4 tests stay outside the report, and a green report does not say this. #### Next Step The discovery rule decided which tests would run; but the running tests are not independent one by one. In the shared setup's five-test suite, one pollutes shared state, and another assumes an unpolluted state. In a suite like this, which order you call the tests in is a configuration detail — does that detail change the verdict? The next lesson runs the same suite in three separate orders, reads the number of distinct verdicts that come out, and measures two mechanisms that take repetition out of a test: the **fixture**, which rebuilds shared state fresh for every test, and **parametrization**, which runs the same body with more than one input. ### Fixtures and Parametrization Source: https://academia.sh/en/courses/python-projects/fixtures-and-parametrization A five-test suite passes 4/5 in writing order and 5/5 in the other two, giving 2 distinct verdicts by order; when a fixture rebuilds shared state fresh for every test, the count drops to 1, and when it is built at the session level, the count drops to 1 again, but this time at 4/5. The previous lesson measured that the discovery rule decides which tests run: the same file found 3, 0, and 4 tests across three rules, and the report stayed the same across all three. There, the tests were independent of each other; each one called a function, compared the returned value, and left no trace behind. Real suites are not like this. Most tests need a setup: an object gets built, a directory gets opened, a counter gets reset. When this setup gets shared, tests become each other's **neighbors** — and neighborhood gives rise to an order. This lesson's question: when run order changes, how many distinct verdicts does the same suite give, and what makes that number singular? #### Shared State Makes Order Matter The shared setup's five-test suite is the plainest model of this. Five tests operate on a single environment object. Three do not touch the environment and pass under every condition. One **pollutes** the environment: it increments a counter by one and still passes. One **assumes the environment is unpolluted**; it passes if the counter is zero, fails otherwise. None of the five tests is defective on its own. The polluting test does its job, and the dependent test does its job. The defect lies in the two running **together, in a particular order**. In writing order, the polluting test gets called before the dependent test, and the dependent test fails; in the order that moves the dependent test earlier, and in reverse order, the dependent test gets called first and passes. This was built as theory under "fast and independent tests" in the Unit Testing and Test-Driven Development course of the Software Quality and Testing curriculum. Theory is not repeated here; what gets measured is how many distinct verdicts the same suite gives. #### Who Decides the Order The order not being written into any test is this measurement's starting point. The order tests get called in is a decision the runner makes, and this decision comes from at least four places. The first is **discovery order**: whatever order the runner collected files in is the order tests run in, and the order of filenames changes when a file gets renamed. The second is **selection**: running only failing tests, or only a certain tag, takes the rest out and changes the remaining tests' neighborhood. The third is **distribution**: when tests get split across more than one worker, each worker runs its own subset in its own order, and how the split happens depends on worker count. The fourth is **deliberate shuffling**: some setups tie order to a value decided fresh on every run, because they want to catch order dependency early. What the four share: none of them is written in the project's source files. Order falls into the same class as the previous lesson's discovery rule — the part of the verdict that sits outside the code. This lesson's measurement changes exactly that part and reads the result: three orders, the same five tests. #### Fixture: Taking Setup Out of the Test A **fixture** is a unit, called by the framework, that takes the setup a test needs out of the test's body. The test itself does not build the environment; the fixture builds it and hands it to the test ready-made. Besides reducing repetition, this does a second and more important thing: it turns **when the setup happens** into an explicit decision. This decision is called the fixture's **scope**, and it takes three values in the measurement. **No fixture.** The environment gets built once, and the five tests share it. The trace the polluting test leaves is visible to every test that runs after it. **Each test.** The fixture rebuilds the environment before every test. The polluting test's trace never reaches the next test. **Session.** The fixture gets built once at the start of the run session, and every test, across every run, shares the same object. This scope gets chosen for expensive setups — so that a resource costly to build does not get rebuilt for every test. All three scopes have valid uses; the measurement gives not which one is "correct," but what each one does to distinct verdict count. A fixture's other half is **teardown**: taking back what got built. An in-memory object needs no teardown; the new object replaces the old one, and the old one drops away on its own. For a file, a directory, or a connection, though, teardown gets written explicitly and has to run even if the test fails; this is where the context manager protocol built in the Python Fundamentals course applies directly. A fixture with missing teardown thinks it removed the sharing while it leaves a residue, and the measurement becomes order-dependent again. #### Parametrization: Same Body, Separate Input Testing the same assertion with more than one input has two paths. In the first, the inputs get put into a loop inside a single test; in the second, every input becomes a **separate test**, and the body gets shared. The second is called **parametrization**. Parametrization shares something with the fixture: both take repetition out of the test's body. Where they part is **what** they take out. A fixture takes out **setup** and manages sharing between tests; parametrization takes out **input** and multiplies a single body across more than one case. For this reason, the two do not substitute for each other, they get used together: every parametrized case runs with its own fixture. The difference is invisible while tests pass. Both test the same assertion with the same inputs, and both give a green report. The difference shows up when an input fails: in the looped writing, the first deviation fails the test and the loop stops there, so the remaining inputs **never get tried at all**. The report carries a single line, and that line holds the test's name — not which input failed. In the parametrized writing, each input runs under its own name; the failing input shows up by name in the report, and the others keep running. The measurement's assumptions: #### The Measurement's Assumptions - **TE9** — The five-test suite and three orders are taken from the shared setup; the TESTS list and the tests' behavior are not changed. The lesson only adds fixture scope. - **TE10** — The oracle is the setup itself: we know all five tests pass in a clean environment because we wrote it that way. The correct verdict in every order is 5/5. - **TE11** — Three orders get tried: writing order, the order that moves the dependent test earlier, and reverse order. The order list is made of the tests' indices and is not written into any test. - **TE12** — In the no-fixture scope, the environment gets built once at the start of each run; it does not get rebuilt within the run. - **TE13** — In the each-test scope, the environment gets rebuilt right before every test in the order is called. - **TE14** — In the session scope, the environment gets built once for all three runs, and carries over between runs too. - **TE15** — Distinct verdict count is the number of mutually different passed-test counts the three orders give. - **TE16** — The assertion tested in the parametrization measurement is this: every order should give 5/5. This assertion comes from the oracle, and it comes out false in the no-fixture suite under writing order. - **TE17** — The looped writing returns at the first deviation; the parametrized writing runs all three orders. Duration is never measured; what gets counted is orders run and report line count. - **TE18** — Report line count is the number of records the framework writes to its result list; an input that never runs never enters the report. #### The Measurement ```python """Fixtures and parametrization: who sets up shared state, and when. Part 1 - three orders, three fixture regimes. Part 2 - the same three orders in a single test and in a parametrized test. """ class Environment: """State the tests share.""" def __init__(self): self.counter = 0 def t_clean_a(env): return True def t_polluter(env): env.counter += 1 return True def t_dependent(env): return env.counter == 0 def t_clean_b(env): return True def t_clean_c(env): return True TESTS = [("clean_a", t_clean_a), ("polluter", t_polluter), ("dependent", t_dependent), ("clean_b", t_clean_b), ("clean_c", t_clean_c)] ORDERS = {"writing order": [0, 1, 2, 3, 4], "dependent first": [0, 2, 1, 3, 4], "reverse": [4, 3, 2, 1, 0]} def run(order, regime="none", session=None): """Fixture scope: none, rebuilt each test, or built once for the session.""" env = session if regime == "session" else Environment() passed = 0 for i in order: if regime == "each test": env = Environment() _, func = TESTS[i] if func(env): passed += 1 return passed, len(order) print(f"{'fixture scope':<18s} " + " ".join(f"{label:>13s}" for label in ORDERS) + " distinct") for regime in ("none", "each test", "session"): session, passes = Environment(), [] for order in ORDERS.values(): passes.append(run(order, regime, session)[0]) print(f"{regime:<18s} " + " ".join(f"{p:>11d}/5" for p in passes) + f" {len(set(passes)):9d}") print() def single_test(): """Tries the three orders in one body, one loop; stops at the first deviation.""" ran = 0 for label, order in ORDERS.items(): ran += 1 passed, total = run(order) if passed != total: return [("three orders", "failed")], ran return [("three orders", "passed")], ran def parametrized_test(): """Each order is a separate test; all three run.""" report, ran = [], 0 for label, order in ORDERS.items(): ran += 1 passed, total = run(order) report.append((label, "passed" if passed == total else "failed")) return report, ran print(f"{'style':<16s} {'report lines':>12s} {'orders run':>11s} " f"{'named as failing':>26s}") lines, ran_counts = [], [] for label, func in (("single test", single_test), ("parametrized", parametrized_test)): report, ran = func() failing = [r[0] for r in report if r[1] == "failed"] lines.append(len(report)) ran_counts.append(ran) print(f"{label:<16s} {len(report):12d} {ran:11d} " f"{(failing[0] if failing else '-'):>26s}") print(f"same three orders: report lines {len(set(lines))}, orders run {len(set(ran_counts))} " f"distinct outcomes; failing verdict 1 in both") ``` ``` fixture scope writing order dependent first reverse distinct none 4/5 5/5 5/5 2 each test 5/5 5/5 5/5 1 session 4/5 4/5 4/5 1 style report lines orders run named as failing single test 1 1 three orders parametrized 3 3 writing order same three orders: report lines 2, orders run 2 distinct outcomes; failing verdict 1 in both ``` #### Two Distinct Verdicts and Two Paths to Singular The first row pays out the course's fourth reading. The no-fixture suite passes **4/5** in writing order, **5/5** in the order moving the dependent test earlier, **5/5** in reverse order — **2 distinct verdicts**. The five tests' code never changed; the only thing that changed was call order. Same input, same code, two distinct outcomes. This number's practical counterpart: a suite like this is green in some runs, red in others, and when red, the failing test is **not the test where the defect lives**. The report points to `dependent`, while the test polluting the environment is `polluter`. Order dependency carries diagnosis to the wrong place. The second row gives what the fixture pays off. When the environment gets rebuilt before every test, all three orders give **5/5**, and the count drops to **1 distinct verdict**. The polluting test still increments the counter, but the object it increments never reaches the next test. The fixture did not fix a defect — the defect was never in the tests — it **removed the sharing**. The third row is the measurement's real warning. The session-level fixture also drops the count to **1 distinct verdict**, but this time at **4/5**. Pollution accumulates on the first run and carries over into later runs too; the dependent test no longer passes in any order. The result became **stable**, and the value it stabilized at is **wrong**. The sentence that follows is this lesson's measurement axis at its sharpest: **distinct outcome count dropping to 1 does not mean the outcome it drops to is correct.** Reproducibility does not substitute for correctness; it is only the precondition for correctness being arguable at all. A suite that gives the same wrong answer on every run is reproducible, and still defective. What separates the second and third rows, for this reason, is not the scope's name, it is **which verdict the setup gives**. Both produce a single number; one cuts off pollution before it is ever born, the other makes pollution permanent. What shows a suite has escaped order dependency is not distinct verdict count being 1, it is that single verdict **matching the oracle** — and the oracle, in the shared setup, says all five of the five tests pass in a clean environment. A warning holds in the reverse direction too. **2 distinct verdicts** could be seen here because three orders got tried; had a single order run, the suite would have given either 4/5 always or 5/5 always, and order dependency would never have shown up. The condition for order dependency to be measurable at all is running order across more than one value. #### Report Lines and Running Inputs The lower table gives what parametrization pays off. When the three orders get tried inside a single test, the runner writes **1 report line**, and that line holds the test's name — "three orders." Which order failed is not in the report; and since it returns at the first deviation, **1 order** runs and the remaining two never get tried at all. This second point outweighs the first: the report is not only missing information, the **measurement itself gets done incompletely**. In the parametrized writing, the same three orders produce **3 report lines** and **3 orders** run. The failing line shows up by name: "writing order." Since the other two orders keep running, the reader of the report can see whether the defect sits in all three orders or in just one — and this distinction carries diagnosis straight to the `polluter` test. The two writings' failing-verdict count is the same: both turn the suite red. What differs is the information the report carries and the count of inputs that actually run. Parametrization does not change the test's verdict, it raises the verdict's **resolution**. This distinction has a cost too, and it should not be hidden. In a parametrized suite, test count grows with input count; ten inputs means ten tests, and those ten tests take up ten lines in the report. The looped writing gets by with one line. Parametrization, that is, **buys diagnostic information with report volume**. What the measurement says is not which one to pick, it is what the choice changes: **report lines 2 distinct outcomes, orders run 2 distinct outcomes, failing verdict 1.** #### Summary - Shared state ties a test's correctness to its neighbor; while none of the five tests is defective on its own, the suite gives 2 distinct verdicts by order: 4/5 in writing order, 5/5 in the other two. - A fixture takes setup out of the test and makes the real decision visible: when the setup happens. This decision is called the fixture's scope. - When the environment rebuilds fresh for every test, all three orders give 5/5, and the count drops to 1 distinct verdict; the fixture does not fix a defect, it removes the sharing. - A session-level fixture also drops the count to 1, but at 4/5; distinct outcome count becoming singular does not say the outcome it becomes singular at is correct. - Parametrization does not change the verdict, it raises its resolution: the same three orders give 1 report line and 1 order run in a single test, and 3 and 3 in the parametrized writing. #### Next Step The fixture made the count singular by rebuilding shared state fresh for every test — but it could do this because what it built was in its own hands: an in-memory object. Had what the fixture built been a package registry over a network, a clock, or a filesystem, "rebuilding" would not have been an option. The next lesson looks at this outward-reaching leg of testing: how many distinct outcomes does the same test give across three separate modes of an external resource, and what does the **mock object** that replaces the external resource drop that number to — and on what basis do we say the outcome it drops to is correct? ### Mock Objects Source: https://academia.sh/en/courses/python-projects/mock-objects The same test reaching an external source gives 3 distinct outcomes across three modes, dropping to 1 with a mock object; but a mock drifted from the registry also gives 1 distinct outcome and passes because the expectation is written alongside it, 2 distinct outcomes stand between it and the real source, and a contract test finds the drift in a single package. The previous lesson measured that a fixture dropped distinct verdict count from two to one by rebuilding shared state fresh for every test. It could do this because what it built was in its own hands: an in-memory object, a counter. Rebuilding was as cheap as an assignment, and it decided the outcome completely. Not everything a test touches is like this. A resolver asks a **package registry** to learn accepted versions; the registry sits at the other end of a network, its content is not under our control, and it does not give the same answer to the same question every time. This lesson's question: how many distinct outcomes does a test reaching such a source give, and what does that number drop to once a mock object takes its place? #### The External Source's Three Modes The external source in the measurement is a **registry client**, modeled within the lesson; no product name appears, no real request gets made. The client has a single method: it returns a package's published versions. The same call does three separate things across three separate modes. **Full.** The registry returns every known version. Resolution gives the shared setup's result. **Stale mirror.** A copy of the registry has not been updated yet and does not carry the latest version. The request succeeds, the response is valid, but **incomplete** — and the incompleteness is not visible inside the response. **Outage.** The registry cannot be reached, and the call raises an exception. Resolution produces no result at all. All three modes get seen in a real setup, and none is a defect; each is part of the registry's normal operation. From the test's point of view, though, the three are one thing: **the same test, three distinct outcomes.** The number three is the measurement's choice, not the source's limit. In a real registry, mode count is larger: a new version gets published and the list grows, a version gets pulled and the list shrinks, a request gets rate-limited and the response gets delayed. What the number means is not "the source does exactly three things," it means **whatever the source's mode count is, the test's outcome count equals it.** If a test depends on an external source, every distinct behavior of the source is a distinct outcome of the test. #### Where a Mock Can Be Put A mock object being able to substitute for the real thing depends on a design condition: the real thing being **supplied from outside**. The resolve function in the measurement does not produce its version list internally; it asks the client it is given. For this reason, putting a different object in the client's place amounts to nothing more than changing an argument. Had the same function asked the registry directly — built and used the client internally — there would be no place left for a mock to go. The test would then have two options: ask the real registry and stay exposed to the three modes, or rebind the name at runtime and change the object the function sees. The second works, but ties the test to the function's **internal implementation**: the moment the function starts building the client under a different name, the test fails without the behavior ever changing. Supplying a dependency from outside was built as "testable design" in the Unit Testing and Test-Driven Development course; not repeated here. The only thing worth noting: **a mock object's ability to drop distinct outcome count depends on the code being written to accept it.** #### What a Mock Object Changes A **mock object (mock / stub)** is an object put in place of a real dependency that answers the same calls. The test itself decides the answer; there is no network, no delay, no outage. This has a direct effect on the measurement: whatever the source's mode count is, the mock object gives a **single** answer, and the test produces **1 distinct outcome**. The types of mock objects and which one fits which situation were built in the Unit Testing and Test-Driven Development course of the Software Quality and Testing curriculum; not repeated here, only referenced. The only thing measured in this lesson is how many distinct outcomes isolation makes singular. The mock object in the measurement does two jobs. First, it **fixes** the answer: it returns the version dictionary it was given. Second, it **records** the calls it receives — how many times, and for which package it was called. This second job lets the test check not just the returned value, but the **call made** too; the measurement's last line writes this record. #### What Isolation Costs A mock object frees the test from the source's modes, but at a cost: the test no longer **touches the real source**. Nothing is left to say the fixed answer matches what the real registry actually gives. The mock object gets written once, while the registry keeps changing. This is called **contract drift**, and the measurement's second part counts it. When a version not present in the real registry gets added to the mock's version list, the test **still passes** — because the test's expectation is written alongside the mock, looking at the mock. The expectation and the mock verify each other, and the two drift away from reality together. Drift has two directions, and both are silent. The mock can keep giving an answer the real thing no longer gives; or the real thing can start giving an answer the mock never anticipated. In the first case, the test stays green and the setup **breaks**; in the second, the test stays green too, and the setup follows a path the mock **never tried**. The mechanism that makes this drift visible is called a **contract test**: the same calls get made to both the mock and the real source, and the answers get compared. A test like this breaks isolation — it touches the real source, and so is exposed to the source's three modes again. For this reason, it does not run alongside the unit tests, it gets kept as a separate, sparsely run set. The measurement's assumptions: #### The Measurement's Assumptions - **TE19** — The registry client is modeled within the lesson; no real network request is made, no real package is downloaded, no product name is written. - **TE20** — The oracle is the shared setup's registry: we know which package carries which versions because we wrote it. REGISTRY and MANIFEST are not changed. - **TE21** — The client has three modes: full, stale mirror, and outage. Stale mirror drops every package's latest version; outage raises an exception. - **TE22** — Resolution runs with the "newest" strategy and gets the version list only from the client; it looks at no other source. - **TE23** — The outage result gets written as "no result" and counts as a distinct outcome; an error is an outcome too. - **TE24** — The mock object fixes the answer and records calls. The concept of mode does not exist in the mock; it gives the same answer in all three runs. - **TE25** — The drifted mock adds to the metrics package a version not present in the real registry. The other two packages stay untouched. - **TE26** — The test's expectation is written alongside the mock in use, looking at the mock. This is the ordinary writing style in a suite that works with a mock object. - **TE27** — The contract test asks the same three calls of both sources and compares the answers package by package; it gives the count of packages where drift is found. - **TE28** — Duration is never measured. What gets counted is distinct outcomes, packages where drift is found, and calls made to the mock. #### The Measurement ```python """Mock object: an external source's three answers versus a single answer. The client is modeled within the lesson. Part 1 - the real client in three modes, the mock object in a single mode. Part 2 - what happens when the mock drifts from the registry. """ REGISTRY = {"metrics": [(1, 0), (1, 1), (1, 2), (2, 0)], "report": [(0, 9), (1, 0), (1, 1)], "common": [(3, 0), (3, 1), (3, 2), (4, 0)]} MANIFEST = {"metrics": ((1, 0), (2, 0)), "report": ((1, 0), (2, 0)), "common": ((3, 0), (4, 0))} class Outage(Exception): """Signals that the external source cannot be reached.""" class RealClient: """External source: the same call gives a separate answer per mode.""" def __init__(self, mode): self.mode = mode self.calls = 0 def versions(self, package): self.calls += 1 if self.mode == "outage": raise Outage(package) if self.mode == "stale mirror": return REGISTRY[package][:-1] return list(REGISTRY[package]) class MockClient: """Mock object that fixes the answer; also counts calls.""" def __init__(self, source=None): self.source = source if source is not None else REGISTRY self.calls = 0 self.requested = [] def versions(self, package): self.calls += 1 self.requested.append(package) return list(self.source[package]) def resolve(client, strategy="newest"): """Resolves the same manifest; gets the version list from the client.""" choice = {} for package in sorted(MANIFEST): low, high = MANIFEST[package] candidates = [v for v in client.versions(package) if low <= v < high] choice[package] = candidates[-1] if strategy == "newest" else candidates[0] return choice def render(choice): return " ".join(".".join(map(str, choice[p])) for p in ("metrics", "common", "report")) def attempt(client): try: return render(resolve(client)) except Outage: return "no result" MODES = ("full", "stale mirror", "outage") print(f"{'run':<22s} {'real client':>16s} {'mock object':>16s}") real, mock = [], [] for mode in MODES: r, m = attempt(RealClient(mode)), attempt(MockClient()) real.append(r) mock.append(m) print(f"{'mode: ' + mode:<22s} {r:>16s} {m:>16s}") print(f"same test, real source gives {len(set(real))} distinct outcomes; " f"mock object gives {len(set(mock))}") print() DRIFTED = dict(REGISTRY, metrics=REGISTRY["metrics"] + [(1, 3)]) drifted_mock = MockClient(DRIFTED) drifted = attempt(drifted_mock) # the expectation is written from the mock print(f"{'run':<31s} {'result':>12s} {'expected':>12s} {'verdict':>7s}") for label, result, expected in (("mock, matches registry", attempt(MockClient()), attempt(MockClient())), ("mock, drifted from registry", drifted, drifted), ("real source, drifted expectation", attempt(RealClient("full")), drifted)): print(f"{label:<31s} {result:>12s} {expected:>12s} " f"{('passes' if result == expected else 'fails'):>7s}") print(f"between the drifted mock and the real source there are " f"{len({drifted, attempt(RealClient('full'))})} distinct outcomes") contract_drift = [p for p in sorted(MANIFEST) if MockClient(DRIFTED).versions(p) != RealClient("full").versions(p)] print(f"the contract test asks the same three calls of both sources and finds " f"drift in {len(contract_drift)} package(s): {', '.join(contract_drift)}") print(f"the mock object received {drifted_mock.calls} calls, requested packages " f"{', '.join(drifted_mock.requested)}") ``` ``` run real client mock object mode: full 1.2 3.2 1.1 1.2 3.2 1.1 mode: stale mirror 1.2 3.2 1.0 1.2 3.2 1.1 mode: outage no result 1.2 3.2 1.1 same test, real source gives 3 distinct outcomes; mock object gives 1 run result expected verdict mock, matches registry 1.2 3.2 1.1 1.2 3.2 1.1 passes mock, drifted from registry 1.3 3.2 1.1 1.3 3.2 1.1 passes real source, drifted expectation 1.2 3.2 1.1 1.3 3.2 1.1 fails between the drifted mock and the real source there are 2 distinct outcomes the contract test asks the same three calls of both sources and finds drift in 1 package(s): metrics the mock object received 3 calls, requested packages common, metrics, report ``` #### From Three to One The upper table gives what isolation pays off. With the real client, the same test produces **3 distinct outcomes**: `1.2 3.2 1.1` in full mode, `1.2 3.2 1.0` in stale mirror, no result in outage. With the mock object, all three runs give `1.2 3.2 1.1` — **1 distinct outcome**. The middle row is the most dangerous of the three and deserves a separate look. Outage announces itself: it raises an exception, the run goes red, and the reason is plain. Stale mirror announces nothing. The request succeeds, the response is a valid list, resolution completes without a hitch, and **only the `report` package's version changes.** The only thing that catches a difference like this is the expectation being written **exactly** — the test saying "a result came back" is not enough. What the number dropping from three to one means: the mock object freed the test from the source's modes. The test is now insensitive to the registry's delay, its outages, and changes to its content — not a small thing; like order dependency, source mode is a factor that changes the verdict from outside the code. #### The Correctness of a Singular Outcome The lower table gives what the mock object **cannot** free the test from, and repeats the previous lesson's warning on a different surface. The first two rows give the same verdict: **passes**. Both mocks return a single answer, both produce **1 distinct outcome**. But one carries the registry's actual content, the other makes up a version that does not exist in the registry. The reason the test passes is the same in both: the **expectation was written looking at the mock**. Whatever the mock says, the expectation writes down, and the comparison holds on its own. The third row exists only if something calls the real source. The drifted expectation **fails** once it gets asked of the real source, because **2 distinct outcomes** stand between them. These two outcomes never meet in any run of the suite; the unit tests run with the mock, and the real source only enters during setup. A contract test closes exactly this gap: it asks the same three calls of both sources and finds drift in **1 package** — `metrics`. The drift being in a single package gives the measurement's scale too: in two of the three packages, the mock is still correct; in one, it has drifted. Without a contract test, this ratio is never known, because knowing it requires asking the real source. The last line shows the mock object's second job. The mock received **3 calls** and asked about all three packages. This record allows checking something beyond the returned value: did the resolver genuinely ask its registry for every package, or did it get some of them from a cache? The returned value does not carry the answer to this question, the call record does. The call record has its own cost too. A test that checks call count ties itself to **how** the resolver works: a cache gets added one day, call count drops from three to one, and the test fails — while resolution keeps giving the same result. A test on the returned value is unaffected by a change like this. The record, for this reason, only gets checked when the call itself is part of the contract — when asking each package separately is a requirement, say. Reading the three sections together: a mock object removes the source's variability and drops distinct outcome count from three to one; against that, it says nothing about **agreement with the source itself**, since it never asks the real source. What isolation makes singular is the outcome, not correctness — and the only mechanism tying the singular outcome back to reality is a test that measures that tie **separately**. #### Summary - The same test reaching an external source gives 3 distinct outcomes across the source's three modes: full answer, incomplete answer, and error. All three are part of the source's normal operation. - Stale mirror is the quietest of the three modes: the request succeeds, the response is valid, and only one package's selected version changes. - A mock object drops the count to 1 distinct outcome by fixing the answer and frees the test from the source's modes; it also lets a test check beyond the returned value by recording the calls it receives. - A mock drifted from the registry also gives 1 distinct outcome and passes the test, because the expectation is written looking at the mock; 2 distinct outcomes stand between the mock and the real source. - A contract test asks the same calls of both sources and finds the drift in 1 package; because it breaks isolation, it does not run alongside the unit tests, it gets kept as a separate set. #### Next Step Everything tested up to here was written in a test file: the assertion, the fixture, the mock object. But a project's correctness claims do not sit only there — they sit in documentation too. A function's docstring saying "this call returns that" is a claim, and no run tests it; it falsifies silently when the code changes. The next lesson makes these claims testable, and counts exactly one thing: after the same change, how many stale claims does the same document turn into sound, and how many does it leave silent? ### Doctests Source: https://academia.sh/en/courses/python-projects/doctests Seven of twelve documentation claims go stale from a single format change; in the prose-heavy writing, 4 of them turn audible and 3 stay silent, and once two claims get converted to examples, the audible count rises to 6 and only the rationale is left — the same staleness, 2 distinct outcomes. The previous lesson measured that a mock object dropped distinct outcome count from three to one, but said nothing about agreement with the real source. The claims there — the expected version set, the expected call count — were written in a test file and ran on every run. They could be wrong, but they could not stay silent. A project's correctness claims do not sit only in test files. A function's docstring saying "this call returns that value" is a claim too. The difference: **no run tests that sentence.** When the code changes, the sentence stays where it is, nothing breaks, and the claim falsifies silently. This lesson's question: after the same change, how many stale claims does the same document turn audible? #### Silent Staleness and Audible Staleness When the surface a documentation claim is tied to changes, the claim goes **stale**. The Technical Writing and Documentation course of the Software Development Practice curriculum measured this over fifty-six claims and split them into two classes: a claim embedded in a runnable example makes **noise** when it goes stale — the example fails to run; a claim that is not embedded produces nothing when it goes stale, and this is **silent staleness**. In that measurement, seventeen of the fifty-six claims were embedded, and that number was a **given**, not a choice. **In this lesson, embedding is a choice.** The measurement does not repeat those numbers; it asks a different question: if the same claim can be written as prose or as a runnable example, what does changing the writing do to the count of stale claims that go audible? Embedding was that measurement's **input**; here, it is this measurement's **variable**. The mechanism that makes this measurable is called a **doctest**, and it is provided by the standard library's `doctest` module. The mechanism is simple: lines in the document text starting with the interactive shell prompt count as expressions, and the lines beneath them count as expected output. The module runs the expressions, compares the output against expected, and counts mismatches as failing examples. The document, that is, becomes a test suite itself. It needs stating from the start that a doctest does not substitute for a unit test suite. What it tests is the example written in the document; an edge case, an error path, or a failure scenario not written there never enters a doctest at all. Multiplying examples to make them comprehensive also makes the document unreadable — a document is written to be **read**, not to test. A doctest's job is not to grow coverage, it is to **keep the claims already written standing**. #### Two Writings of a Claim The same fact can be written two ways. "The version text is made of numbers separated by periods" is a prose claim; it gets read, understood, and tested by nothing. The same fact, written as a call and its expected output, becomes testable — and its readability usually improves too, because an example puts a concrete value in place of an abstract description. The choice between the two looks like a style preference, yet it carries a measurable difference. A prose claim going stale only gets noticed if a human reads it; an example going stale turns the run red. The measurement counts exactly this difference. #### A Doctest Bounds What Can Be Written A doctest's comparison rule is strict: expected output and actual output have to be equal **as text**. This rule also decides which example can go into the document — an example whose output is not the same on every run cannot go into the document. In Python, this constraint has a few well-known sources. An object's default representation carries a number tied to its identity that changes from run to run; a floating-point number's representation is sensitive to operation order; the print order of a set or dictionary keys is a separate topic worth its own attention. Document examples, for this reason, either get written to return a deterministic value, or get made deterministic by sorting, formatting, or reducing the output to a comparison. The measurement's examples show this directly. The `diff` function's result is a list sorted with `sorted`; without the sort, the example could give a different order each run. The function producing the lock text also writes packages in name order. Neither decision was made to make the document testable, both were made to make the result **readable** — but the doctest rests on top of these decisions. The consequence looks backward at first glance: **adding a doctest affects the code, not the document.** If a function's output is not deterministic, that function's documentation cannot become testable; if that is wanted, what has to change is the function itself. #### The Claim That Cannot Convert Not every prose claim can convert to an example, and this is the doctest's limit. A doctest compares an expression's **value**; it cannot compare a sentence that has no value. Five of the measurement's six prose claims depend on a call's result and can convert to an example. The sixth is a **rationale**: "no prefix is written, because the text sorts like a number." This sentence's testable half — no prefix being written — is already tested in another example; its untestable half is what the sentence actually carries, the **reason** for the decision. A reason has no returned value. In practice: a doctest can protect sentences that state **what** is the case, it cannot protect sentences that state **why** it is the case. The second class is the most expensive one to go stale, because a wrong rationale leaves the reader alone with a wrong decision. When an example fails, the reader stops trusting the document; when a rationale goes stale, the reader keeps trusting it, and gets steered the wrong way. The measurement's assumptions: #### The Measurement's Assumptions - **TE29** — The four documented functions are this lesson's own: version text, lock text, and the function giving the diff between two dictionaries. The shared setup's REGISTRY, MANIFEST, and discount definitions are not touched. - **TE30** — Total claim count is 12. Six are runnable examples in the document text, six are prose sentences. The total is the same in both writings; only the distribution changes. - **TE31** — The change is a format change: a prefix gets added to the version text, and the lock text's field separator changes. The diff function stays untouched. - **TE32** — The oracle is the setup itself: we know which claim goes stale because we wrote both implementations. Prose claims' staleness gets flagged from this knowledge. - **TE33** — Runnable examples' staleness is not flagged, it is **measured**: the doctest gets run, and the count of failing examples gets read. - **TE34** — In the second writing, the convertible stale prose claims get converted to examples. The non-stale ones stay prose; the measurement counts only staleness's noise. - **TE35** — The one claim that cannot convert to an example is the rationale, and it stays prose in both writings. - **TE36** — The doctest takes the document text as a string separate from the source and runs it with two separate implementation sets; the run's output is swallowed, only the numbers get read. - **TE37** — An example fails if its expected output is not equal to the actual output; there is no partial match. - **TE38** — Before the change, both writings give 0 failures; the document is consistent with the implementation it was written against. - **TE39** — Duration is never measured, no file is left in the repository. What gets counted is claims, stale claims, failing examples, and claims that stay silent. - **TE40** — Distinct outcome count is the number of mutually different audible-stale-claim counts the two document writings give. #### The Measurement ```python """Doctests: same document, two writing styles, same staleness, two outcomes. Part 1 - the examples in the document get run; prose claims are flagged from the oracle. Part 2 - two prose claims get turned into examples and the measurement is repeated. """ import doctest CHOICE = {"metrics": (1, 2), "common": (3, 2), "report": (1, 1)} PREVIOUS = {"metrics": (1, 1), "common": (3, 2), "report": (1, 1)} def old_version(v): return ".".join(map(str, v)) def new_version(v): return "v" + ".".join(map(str, v)) def old_lock(choice): return " ".join(f"{p}={old_version(choice[p])}" for p in sorted(choice)) def new_lock(choice): return ", ".join(f"{p}={new_version(choice[p])}" for p in sorted(choice)) def diff(a, b): return sorted(p for p in a if a[p] != b[p]) OLD = {"version": old_version, "lock": old_lock, "diff": diff, "CHOICE": CHOICE, "PREVIOUS": PREVIOUS} NEW = {"version": new_version, "lock": new_lock, "diff": diff, "CHOICE": CHOICE, "PREVIOUS": PREVIOUS} EXAMPLES = """ >>> version((1, 2)) '1.2' >>> version((3, 0)) '3.0' >>> lock(CHOICE) 'common=3.2 metrics=1.2 report=1.1' >>> lock(PREVIOUS) 'common=3.2 metrics=1.1 report=1.1' >>> diff(CHOICE, PREVIOUS) ['metrics'] >>> diff(CHOICE, CHOICE) [] """ EXTRA_EXAMPLES = """ >>> set(version((1, 2))) <= set('0123456789.') True >>> lock(CHOICE).count(',') 0 """ # Prose claims: text, stale after the change?, convertible to an example? PROSE = (("version text carries only digits and periods", True, True), ("the lower bound is inclusive, the upper bound is exclusive", False, True), ("packages are written in name order", False, True), ("fields in the lock text are separated without commas", True, True), ("the diff list is given in name order", False, True), ("no prefix is written, because the text sorts like a number", True, False)) def run(text, namespace): test = doctest.DocTestParser().get_doctest(text, dict(namespace), "doc", None, 0) result = doctest.DocTestRunner(verbose=False).run(test, out=lambda s: None) return result.attempted, result.failed STALE_PROSE = sum(1 for _, s, _ in PROSE if s) CONVERTED = sum(1 for _, s, c in PROSE if s and c) STYLES = (("prose-heavy", EXAMPLES, 0), ("two claims as examples", EXAMPLES + EXTRA_EXAMPLES, CONVERTED)) print(f"{'doc style':<22s} {'examples':>8s} {'prose':>7s} {'audible stale':>13s} " f"{'silent stale':>13s}") audible, silents = [], [] for label, text, converted in STYLES: attempted, failed = run(text, NEW) silent = STALE_PROSE - converted audible.append(failed) silents.append(silent) print(f"{label:<22s} {attempted:8d} {len(PROSE) - converted:7d} {failed:13d} " f"{silent:13d}") print(f"claim total in both styles is {len(PROSE) + run(EXAMPLES, NEW)[0]}, " f"stale count {audible[0] + STALE_PROSE}") print(f"audible stale claims {audible[0]} and {audible[1]}: " f"{len(set(audible))} distinct outcomes") print(f"converting to examples turned {silents[0] - silents[1]} silent staleness " f"audible; {silents[1]} could not convert because it is a rationale") print(f"the same document gives {run(EXAMPLES, OLD)[1]} and " f"{run(EXAMPLES + EXTRA_EXAMPLES, OLD)[1]} failures before the change") ``` ``` doc style examples prose audible stale silent stale prose-heavy 6 6 4 3 two claims as examples 8 4 6 1 claim total in both styles is 12, stale count 7 audible stale claims 4 and 6: 2 distinct outcomes converting to examples turned 2 silent staleness audible; 1 could not convert because it is a rationale the same document gives 0 and 0 failures before the change ``` #### The Same Staleness, Two Distinct Outcomes The last line gives the measurement's starting point: before the change, both writings produce **0 failures**. The document is consistent with the implementation it was written against, and every claim is correct. Staleness gets born when the format changes. Without this line, the rest of the measurement would be unreadable: to say the failing examples come from staleness, it had to be shown that none of them failed before the change. After the change, **7** of twelve claims go stale: four runnable examples, three prose. Stale claim count is the same in both writings — how the document is written does not change **what** falsifies. The only thing it changes is whether the falsification gets **heard**. In the prose-heavy writing, the doctest gives **4** failures, and **3** stale claims stay silent. Once the two stale prose claims get converted to examples, the audible count rises to **6**, and what stays silent drops to **1**. The same staleness, **2 distinct outcomes** across the two document writings. The difference between these two numbers is exactly the writing itself. The code is the same, the change is the same, the set of claims that go stale is the same. Only two sentences moved from prose to example, and staleness's visibility spread across more than two. This is what a doctest adds to the measurement: **writing a claim so that it speaks when it falsifies.** It is worth noting where the audible count comes from. The four runnable examples failed because they went stale, and we did not flag their failure — the doctest ran, the count got read. Prose claims' staleness, though, could not be measured, it got flagged from the oracle: since we wrote both implementations, we know which sentence is no longer correct. The measurement's asymmetry is not an accident, it is the topic itself. **An audible claim can be counted; a silent claim can only be known** — and in a real project, that knowledge is not there either. Another reading: the difference between the two writings was **2**, while stale prose claim count was **3**. Converting to examples, that is, did not save all the stale prose claims, it saved two-thirds. This ratio is not the setup's choice, it is an example of claim classes' natural distribution: a document always contains sentences with no value. #### What Is Left The one claim staying silent is the rationale, and it staying there is not a gap, it is a **limit**. A doctest runs an expression and compares its value; a sentence starting with "because" has no expression to run. This claim can only be checked by a human reading it, and where that check belongs is the subject of the Software Development Practice curriculum. What the measurement says here is that the silent count **cannot be dropped to zero.** A doctest reduces silent staleness, it does not remove it; if it could, the whole document would become runnable and stop being a document at all. A document's value, in any case, is not only in its testable sentences; it is in the sentences that carry the reader to a decision. There is a limit in the reverse direction too. Every claim converted to an example makes the document **more fragile**. The two new examples in the second writing test the version text's character set and the comma count in the lock text; these are not the behavior itself, they are details of its **format**. When the format changes deliberately, these examples fail too, and what they say is not "the code broke," it is "the document needs updating." This is exactly a doctest's benefit: it gives both messages through the same channel, at the same time. One last thing needs writing: what a number does not mean. **6/12** does not mean half the document is tested; it means half the claims in the document make noise when they falsify. A claim not written in the document never enters this measurement, and a function's undocumented behavior sits entirely outside this number. #### Summary - A sentence in a docstring is a correctness claim too; its difference is that no run tests it. It falsifies silently when the surface it is tied to changes. - A doctest turns a document into a test suite by running the calls in the document text and comparing them against expected output; a claim being embedded is not a given, it is a writing choice. - 7 of 12 claims go stale from a single format change, and this number is independent of writing; writing only decides whether the staleness gets heard. - The prose-heavy writing turns 4 stale claims audible and leaves 3 silent; once two claims convert to examples, the audible count rises to 6 — 2 distinct outcomes. - The one claim left silent is a rationale and cannot convert to an example: a doctest compares an expression's value, and a reason has no value. #### Next Step Every measurement up to here counted what the test **sees**: which tests got discovered, in what order they ran, which source they looked at, which claim turned audible. What all of them share is that what got counted was the test's **point of view**. The next lesson turns the counter to the other side and looks at the code itself: when a three-test suite runs, which lines of the function **executed**, which branches got taken? This number is misleading to a degree, and the lesson's job is drawing exactly the boundary of that mistake — even when eight of eight lines execute and three of three tests pass, a call can still be found where the function does not honor its contract. ### Coverage Measurement Source: https://academia.sh/en/courses/python-projects/coverage-measurement A three-test suite covers 8 of 8 lines (1.0000), takes 6 of 6 branches, and passes 3/3; against that, in one of four input combinations the actual behavior deviates from the oracle and the test suite runs exactly that call — coverage counts where the code ran, not what the test looked at. The previous lesson made documentation claims testable and measured that four, then six, of the seven claims that went stale out of twelve turned audible. The question there was "does the claim get heard when it falsifies." Now it is time to look from the other direction: when tests run, **where in the code did execution go?** The number that answers this question is called **coverage**. Coverage does not say whether a test suite is good; it counts which parts of the code ran during a run. This lesson's job is to show exactly what that number measures and draw its limit — because drawing that limit wrong is the most expensive misreading in this entire topic. #### What a Coverage Counter Counts The coverage counter in the measurement is modeled within the lesson; no tool name appears. The mechanism the model rests on is Python's **trace hook**: the interpreter gets handed a function, and it calls that function at certain events during execution. One of these events is the `line` event, and it fires every time execution moves to a new source line. The counter is nothing more than this. When the hook gets called, the number of the running line gets recorded; once the run ends, the set of recorded numbers gives the covered lines. The only information the hook sees is "which line are we at" — which test is running, what the assertion expects, whether the result is correct: none of it reaches the hook. This fact decides the measurement's entire outcome in advance, and it is not a gap. The trace hook cannot see the test's expectation, because the expectation is a comparison inside the test, and the hook watches **execution**, not **meaning**. A coverage counter extracts the **geography of execution**. The function's eight lines correspond to the eight statements in its body, and their numbers get read from the code itself: the definition line and the docstring get left out, leaving eight executable lines. #### Two Metrics: Line and Branch **Line coverage** asks whether a line ran at least once. **Branch coverage** asks whether a condition's both true and false outcomes got taken. The difference between the two shows up when a conditional's body never runs: line coverage counts the condition line as covered, branch coverage records that only one branch got taken. These metrics and the hierarchy between them were built as theory in the Unit Testing and Test-Driven Development course of the Software Quality and Testing curriculum; not repeated here, only referenced. Both get measured together in this lesson, because the claim's strength depends on it: if only line coverage were full, one could object "you would have seen it with a stronger metric." The measurement closes off that objection — **both metrics are full, and the defect stays invisible anyway.** Branch coverage gets extracted from execution in the model too: for every condition, whether the line inside that condition's body ran on that call gets checked. If it ran, the true branch was taken; if not, the false branch was. Three conditions, six branches. Beyond these metrics, a third exists, not used here: **path coverage**, that is, how many of the conditions' mutually independent combinations got tried. This function's three conditions have four reachable combinations, and the measurement's second table lists exactly those four. This lesson does not write path coverage as a separate number, because it does not weaken the argument, it strengthens it: three of the four are in the test set, and the defective one is among them. #### The Oracle Is Written Separately The shared setup's discount function makes three decisions: membership ten points, coupon fifteen points, and a cap. The defect is in the third decision — the cap is written as **20**, while the contract says **25**. The oracle, the correct behavior, is written as a separate function and applies the cap of 25. The test set's expected values, though, are written not from the oracle, but from the function's **observed behavior**. This is the same situation built in the first lesson, and it is not a setup detail, it is a real writing habit: when writing a test for an existing function, expected values often get decided by running the function and looking at its output. A test written this way verifies that behavior has **not changed**; not that it is correct. The measurement's assumptions: #### The Measurement's Assumptions - **TE41** — The discount function and the oracle are taken from the shared setup; neither's behavior is changed. The lesson only adds the coverage counter and a fourth test. - **TE42** — The coverage counter is modeled with a trace hook. The hook counts only the line event and only the discount function's frame; it accesses no other information. - **TE43** — The eight lines counted are the executable statements in the function's body, and their numbers get read from the code itself; the definition line and the docstring are excluded. - **TE44** — Branch coverage is two outcomes for each of the three conditions, six branches total. That a branch was taken gets inferred from whether the line inside that condition's body ran. - **TE45** — The test set is the shared setup's three tests, and their expected values are written from the function's observed behavior. - **TE46** — The oracle comparison happens outside the measurement, in a separate table; the test set never calls the oracle. - **TE47** — The input space is held with amount fixed at 100, and the four combinations of two boolean flags get tried. Changing the amount does not change the defect, only its size. - **TE48** — A call being "in the set" means it equals one of the test set's three inputs. - **TE49** — The fourth test uses the same input but takes its expected value from the oracle; there is no other difference. - **TE50** — Passing test count is the count of tests where the expected value equals the actual value; there is no partial pass. - **TE51** — Ratios are written to four decimal places; the smallest measurable line difference in this set is 1/8, that is, 0.1250. - **TE52** — Duration is never measured, no file is left in the repository. What gets counted is covered lines, taken branches, passing tests, and calls deviating from the oracle. #### The Measurement ```python """Coverage counter: which line ran, which branch was taken, which call deviates from the oracle. The counter is modeled within the lesson with a trace hook; what gets measured is where the code ran. """ import sys def discount(amount, member, coupon): """Three-decision function; the defect is in the third decision.""" rate = 0 if member: rate += 10 if coupon: rate += 15 if rate > 20: rate = 20 return amount - amount * rate // 100 def correct_discount(amount, member, coupon): """Oracle: the correct behavior applies a cap of 25.""" rate = (10 if member else 0) + (15 if coupon else 0) if rate > 25: rate = 25 return amount - amount * rate // 100 NAMES = ("rate=0", "if member", "rate+=10", "if coupon", "rate+=15", "if rate>20", "rate=20", "return") BODY = discount.__code__.co_firstlineno + 1 # definition and docstring lines excluded NUMBERS = sorted({s for _, _, s in discount.__code__.co_lines() if s is not None and s > BODY}) LINES = dict(zip(NUMBERS, NAMES)) BRANCHES = {"if member": "rate+=10", "if coupon": "rate+=15", "if rate>20": "rate=20"} TEST_SET = (((100, True, False), 90), ((100, False, True), 85), ((100, True, True), 80)) ALL = [(100, m, c) for m in (False, True) for c in (False, True)] def trace_call(args): """Traces a single call and returns the names of the lines that ran.""" seen = set() def hook(frame, event, arg): if frame.f_code is discount.__code__: if event == "line": seen.add(LINES[frame.f_lineno]) return hook return None sys.settrace(hook) try: value = discount(*args) finally: sys.settrace(None) return value, seen def coverage(calls): """Line and branch coverage: what gets counted is whether the line ran.""" lines, branches = set(), set() for args in calls: _, seen = trace_call(args) lines |= seen for condition, body in BRANCHES.items(): branches.add((condition, body in seen)) return lines, branches def passing(cases): return sum(1 for args, expected in cases if trace_call(args)[0] == expected) calls = [a for a, _ in TEST_SET] lines, branches = coverage(calls) deviating = [a for a in ALL if discount(*a) != correct_discount(*a)] print(f"{'metric':<14s} {'covered':>9s} {'total':>7s} {'ratio':>7s}") print(f"{'line':<14s} {len(lines):9d} {len(NAMES):7d} {len(lines) / len(NAMES):7.4f}") print(f"{'branch':<14s} {len(branches):9d} {2 * len(BRANCHES):7d} " f"{len(branches) / (2 * len(BRANCHES)):7.4f}") print(f"passing tests {passing(TEST_SET)}/{len(TEST_SET)}") print() print(f"{'input':<26s} {'actual':>7s} {'oracle':>7s} {'deviation':>9s} {'in set':>7s}") for a in ALL: g, d = discount(*a), correct_discount(*a) print(f"{'amount=%d member=%d coupon=%d' % (a[0], a[1], a[2]):<26s} {g:7d} {d:7d} " f"{('yes' if g != d else 'no'):>9s} " f"{('yes' if a in calls else 'no'):>7s}") print(f"in {len(deviating)} of four combinations, actual behavior deviates from " f"the oracle; that call is in the set: " f"{'yes' if deviating[0] in calls else 'no'}") print() FOUR = TEST_SET + (((100, True, True), correct_discount(100, True, True)),) lines4, branches4 = coverage([a for a, _ in FOUR]) print(f"{'test set':<16s} {'tests':>5s} {'passing':>7s} {'line':>7s} {'branch':>7s}") print(f"{'three tests':<16s} {len(TEST_SET):5d} {passing(TEST_SET):7d} " f"{len(lines) / len(NAMES):7.4f} {len(branches) / (2 * len(BRANCHES)):7.4f}") print(f"{'oracle fourth':<16s} {len(FOUR):5d} {passing(FOUR):7d} " f"{len(lines4) / len(NAMES):7.4f} {len(branches4) / (2 * len(BRANCHES)):7.4f}") RATIOS = {(passing(TEST_SET), len(TEST_SET)), (passing(FOUR), len(FOUR))} print(f"the fourth test leaves coverage at {len({len(lines), len(lines4)})} distinct " f"outcome(s), and raises the passing ratio to {len(RATIOS)} distinct outcomes") ``` ``` metric covered total ratio line 8 8 1.0000 branch 6 6 1.0000 passing tests 3/3 input actual oracle deviation in set amount=100 member=0 coupon=0 100 100 no no amount=100 member=0 coupon=1 85 85 no yes amount=100 member=1 coupon=0 90 90 no yes amount=100 member=1 coupon=1 80 75 yes yes in 1 of four combinations, actual behavior deviates from the oracle; that call is in the set: yes test set tests passing line branch three tests 3 3 1.0000 1.0000 oracle fourth 4 3 1.0000 1.0000 the fourth test leaves coverage at 1 distinct outcome(s), and raises the passing ratio to 2 distinct outcomes ``` #### Full Coverage and Defective Behavior The upper table places three numbers side by side, and all three look positive. The three-test suite covers **eight of eight** lines: **1.0000**. It takes **six of six** branches: **1.0000**. And **3/3** tests pass. Anyone looking at these three numbers concludes the function is tested. The middle table gives the same function's entire input space, and deviation sits in the last row. When membership and coupon come together, the function gives **80**, the oracle gives **75** — the rate should be 25, but the cap pulls it down to 20. In **1 of four** input combinations, actual behavior deviates from the oracle. The link between the two tables is the measurement's real finding. The deviating call — membership and coupon together — is **inside** the test set. That is, the test set **runs** exactly that call. The line where the defect lives — the line applying the cap — runs; the coverage counter sees it and records it as covered. Coverage coming out at 1.0000 is, in fact, because this call is in the set. Why does the test pass, then? Because that test expects **80**. Since the expected value was written from the function's observed behavior, the function produces its defective value, and the comparison holds. **The reason it passes is not a coverage gap, it is that the expectation wrote down the defective value.** This course's third claim is exactly this: **coverage sees the line, it does not see the expectation.** And the reason was written in this lesson's first section: the only information reaching the trace hook is "which line are we at." The value the test expects never reaches that hook. #### Same Coverage, Separate Verdict The lower table settles this with an experiment. A fourth test gets added to the three-test suite; its input is the same — membership and coupon together — but its expected value comes from the **oracle**. The new test fails, and the suite drops to **3/4**. Coverage does not budge: line **1.0000**, branch **1.0000**. The same input was already in the set, so the set of lines that ran never changed. Coverage stays at **1 distinct outcome**, while the passing ratio rises to **2 distinct outcomes**. These two numbers coming apart is the plainest proof of what coverage is. The change that found the defect never changed coverage at all. Getting closer to the defect by growing coverage is impossible in this example, because coverage is already at its ceiling. What was missing was not a call, it was an **expectation** — and an expectation sits outside what coverage measures. It needs saying that the reverse holds too: low coverage is information, and real information. A line that never ran is a line no test ever tried, and anything can be true there. Coverage reliably says **which code was never tested**; it never says which code is **correct**. The number's direction runs one way only. This asymmetry has one more consequence. A line running and that line being **tested** are separate things, and the only thing that closes the gap between them is an assertion. By the coverage counter's reckoning, a line is covered even when a test with no assertion at all runs it — the assertion-free test seen in the first lesson does exactly this. Raising coverage, that is, only requires calling the code, never checking the result; the number cannot tell the two apart. #### How the Number Is Read Three reading rules follow from this. First: **coverage is a lower bound, not a certificate.** Low coverage is proof of a gap; high coverage is not proof of sufficiency. This asymmetry also explains why the number erodes once it gets set as a target — the cheapest way to raise it is not adding assertions, it is adding calls. Second: **a coverage percentage cannot be compared on its own.** In this measurement, the smallest measurable line difference is 1/8, that is, **0.1250**; in an eight-line function, there is no value between 1.0000 and 0.8750. A coverage number placed next to another codebase's percentage says nothing unless both sides state what they count. This is the previous course's measurement-discipline rule applied to this measurement. Third: **coverage is not a number that measures the quality of the expectation, and no number that does comes out of a coverage counter.** What decides the expectation's correctness is the **oracle** — a contract, a specification, or, as in this course, the setup itself. Without an oracle, every expectation written just re-records observed behavior, and the suite offers the code's current state as its own proof. #### Summary - A coverage counter counts the trace hook's line event; the only information it sees is which line execution is at. The test's expectation never reaches this hook. - A three-test suite covers 8 of 8 lines (1.0000), takes 6 of 6 branches (1.0000), and passes 3/3. - Against that, in 1 of four input combinations, actual behavior deviates from the oracle and the test set runs that exact call; the reason it passes is not a coverage gap, it is that the expectation wrote down the defective value. - A fourth test taking its expectation from the oracle drops the suite to 3/4 but does not change coverage: coverage stays at 1 distinct outcome, and the passing ratio rises to 2 distinct outcomes. - Coverage is one-directional information: it says which code was never tested, not which code is correct. Coverage counts where the code ran, not what the test looked at. #### Next Step Everything measured across this topic was about the code **running**: which tests got found, in what order they ran, which source they looked at, which claim turned audible, which line executed. Coverage was the last in this sequence, and it counted **where** the code ran. What remains is another face of the code: **how it is written.** A function producing the same behavior can be written in countless forms — different indentation, different line endings, different quoting and spacing choices — and none of this shows up in a coverage counter; the lines that run are the same lines. When these formatting decisions get left to a human, every review produces a fresh argument. The next lesson carries the question here: how many distinct writings of the same source exist, what does that number drop to, and who does the making-singular? ## Quality and Publishing Automated format and rule checking, verification across multiple environments, distribution formats and build systems, submission to a package registry with version marking, and generating documentation from source. ### Formatters and Linters Source: https://academia.sh/en/courses/python-projects/formatters-and-linters The same function's six separate forms give six separate source texts but a single parse tree; the formatter reduces the six texts to 1 distinct outcome, drops the linter's finding-set count from 5 to 1, but leaves the remaining 6 findings and the single call that deviates from the oracle untouched. The previous lesson measured coverage and showed what coverage does not count: eight of eight lines ran, three of three tests passed, and yet in one of four input combinations the real behavior deviated from the oracle. Coverage counts **where the code ran.** This lesson steps to the side: who unifies **how the code is written**? How many separate forms can the same function take on a team, and how many does that number drop to once it is handed off to a tool? For the question to be measurable, "form" has to have a precise definition; this lesson establishes that definition and reads the number. #### Two Separate Tool Classes Two tool classes are often named in the same sentence and confused. A **formatter** **rewrites** the source. Its input is a source text, its output is a source text too; the two share the same parse tree, they differ in layout. What a formatter produces is **single**: whichever text feeds the same tree, the same text comes out. A **linter** **reports on** the source. Its input is a source text, its output is a list of findings; it does not touch the source. Some findings concern layout, some do not — and this distinction sits at the center of what this lesson measures. The two classes' output is also of a different kind, and this is the most concrete difference between them. A formatter's output **replaces** the source; a linter's output **stands beside** it. The first is a transformation and it cannot be reversed, because the layout information that gets deleted cannot be brought back. The second is an observation and leaves the source as it is; it is still the writer who reads the finding and decides what to do. Handing a formatting debate off to a tool was measured in the Code Review and Team Process course, as one item of the review axis; the question asked there was how many of a review's comments were about formatting. Here the question is different, and it is not concerned with the debate itself: **how many separate forms of the same source are there, and how far does that number drop?** #### Setting Up the Model No lesson in this course writes a tool's name; tool behavior is modeled inside the lesson and it is said plainly that it is modeled. The model here has three parts. **Forms.** The shared setup's three-decision `discount` function is written with a single canonical text. Five more forms are **produced** by applying three layout transformations to this text: changing the indentation step, putting an `if` body on the same line, stripping the spaces around an operator. No transformation touches the function's behavior. **Formatter.** The model is the standard library function that parses the source and rewrites it from the tree. A real formatter preserves comments, applies a line-width rule, and breaks long expressions; this model does none of that. The only thing the model unifies is **layout**, and the measurement only asks about layout. **Linter.** There are two rule families. **Rules that look at text** scan the source's lines: is the indentation step four spaces, are there two statements on one line, is there a space on both sides of an operator. **Rules that look at the tree** walk the parse tree: is there an unnamed numeric constant, does the function's decision count exceed a threshold. The second family never sees how the source is written, because the tree does not carry layout. The measurement's assumptions: - **QP1** — Six forms are produced from a single canonical text; the only difference between them is the indentation step, where the body sits on the line, and the space around an operator. - **QP2** — The forms' identity is tested two separate ways: the parse tree's dump, and the return values across four input combinations. Both stand as separate columns in the table. - **QP3** — The formatter model does not keep comments or blank lines; the measurement is therefore done on a comment-free source, and this limit of the model is stated here. - **QP4** — Three rules that look at text and two rules that look at the tree are the linter's entirety. These are not a rule catalog, they are the smallest set that suffices to separate the two families. - **QP5** — The finding set is compared as a sorted tuple; if two forms' finding sets are identical, **one** distinct outcome is counted, even if their texts are separate. - **QP6** — The `discount` function's body is as in the shared setup and is not changed; the flawed ceiling stays in place. The oracle is written separately and applies a ceiling of 25. - **QP7** — No duration is measured, no file is written. What is counted is distinct text, distinct tree, distinct behavior, and distinct finding set. #### Measurement ```python """Reducing form to one outcome: six forms, how many distinct outcomes.""" import ast import re CANONICAL = ( "def discount(amount, member, coupon):\n" ' """A function with three decisions."""\n' " rate = 0\n" " if member:\n" " rate += 10\n" " if coupon:\n" " rate += 15\n" " if rate > 20:\n" " rate = 20\n" " return amount - amount * rate // 100\n" ) OPERATORS = ("+=", "//", ">", "-", "*") def reindent(k, step): return "\n".join(" " * (step * ((len(s) - len(s.lstrip())) // 4)) + s.lstrip() for s in k.split("\n")) def to_one_line(k): return re.sub(r"(?m)^(\s*)(if .+:)\n\s+(.+)$", r"\1\2 \3", k) def unspace_operators(k): for op in OPERATORS: k = k.replace(f" {op} ", op) return k FORMS = { "four spaces": CANONICAL, "two spaces": reindent(CANONICAL, 2), "eight spaces": reindent(CANONICAL, 8), "one-line body": to_one_line(CANONICAL), "unspaced operator": unspace_operators(CANONICAL), "both together": unspace_operators(to_one_line(reindent(CANONICAL, 2))), } def layout_findings(source): """Rules that look at text; the formatter silences these.""" findings = [] indents = [len(s) - len(s.lstrip()) for s in source.split("\n") if s.strip()] if min((g for g in indents if g > 0), default=4) != 4: findings.append("indentation step is not four spaces") for no, s in enumerate(source.split("\n"), 1): if re.match(r"\s*(if|for|while) .*:\s*\S", s): findings.append(f"line {no}: two statements on the same line") for op in OPERATORS: if op in s and f" {op} " not in s: findings.append(f"line {no}: no space around {op}") return findings def structure_findings(source): """Rules that look at the tree; the formatter does not silence these.""" tree = ast.parse(source) findings = [f"unnamed constant {d.value}" for d in ast.walk(tree) if isinstance(d, ast.Constant) and isinstance(d.value, int) and d.value not in (0, 1)] decisions = sum(1 for d in ast.walk(tree) if isinstance(d, ast.If)) return sorted(findings + ([f"decision count {decisions}, threshold 2"] if decisions > 2 else [])) def format_source(source): """Formatter model: source is rewritten from the tree by a single rule.""" return ast.unparse(ast.parse(source)) + "\n" def behavior(source): ns = {} exec(compile(source, "
", "exec"), ns) return tuple(ns["discount"](100, m, c) for m in (False, True) for c in (False, True)) def oracle(amount, member, coupon): """Oracle: correct behavior applies a 25 ceiling.""" rate = min((10 if member else 0) + (15 if coupon else 0), 25) return amount - amount * rate // 100 print(f"{'form':<20s} {'lines':>5s} {'layout':>6s} {'structure':>9s} " f"{'behavior':>22s}") for name, k in FORMS.items(): print(f"{name:<20s} {len(k.strip().split(chr(10))):5d} " f"{len(layout_findings(k)):6d} {len(structure_findings(k)):9d} " f"{str(behavior(k)):>22s}") K = list(FORMS.values()) B = [format_source(k) for k in K] before = {tuple(sorted(layout_findings(k) + structure_findings(k))) for k in K} after = {tuple(sorted(layout_findings(b) + structure_findings(b))) for b in B} print() print(f"distinct source text {len(set(K))} | distinct parse tree " f"{len({ast.dump(ast.parse(k)) for k in K})} | distinct behavior " f"{len({behavior(k) for k in K})} | distinct text after formatting " f"{len(set(B))}") print(f"linter: {len(before)} distinct finding sets before formatting, " f"{len(after)} after") print(f"remaining findings {len(sorted(after)[0])}: {', '.join(sorted(after)[0])}") print() deviating = [(m, c) for m in (False, True) for c in (False, True) if behavior(CANONICAL)[2 * m + c] != oracle(100, m, c)] print(f"in {len(deviating)} of four input combinations, behavior deviates " f"from the oracle: {deviating}") ``` ``` form lines layout structure behavior four spaces 10 0 6 (100, 85, 90, 80) two spaces 10 1 6 (100, 85, 90, 80) eight spaces 10 1 6 (100, 85, 90, 80) one-line body 7 3 6 (100, 85, 90, 80) unspaced operator 10 6 6 (100, 85, 90, 80) both together 7 10 6 (100, 85, 90, 80) distinct source text 6 | distinct parse tree 1 | distinct behavior 1 | distinct text after formatting 1 linter: 5 distinct finding sets before formatting, 1 after remaining findings 6: decision count 3, threshold 2, unnamed constant 10, unnamed constant 100, unnamed constant 15, unnamed constant 20, unnamed constant 20 in 1 of four input combinations, behavior deviates from the oracle: [(True, True)] ``` #### Six Forms, One Outcome The bottom line gives the lesson's central number: **6 distinct source texts, 1 distinct parse tree, 1 distinct behavior, 1 distinct text after formatting.** The three numbers have to be read together. The six texts really are separate — compared as files, all six come out different, and they appear as six separate lines in a change log. Yet the tree is **single**: what the interpreter sees is the same across all six. The behavior column confirms this independently; all six forms give `(100, 85, 90, 80)` across the four input combinations. What a formatter does is exactly close the gap between these three numbers. The tree was already single; the formatter unifies the text too. One rule is applied to the six and **1** text is left. This is the only guarantee the tool gives: the same text from the same tree. The top table's `lines` column shows where the cost sits. The one-line-body forms compress the function into **7** lines, the others hold at **10**. The formatted result preserves none of these choices; the writer's preference over line layout disappears. What the tool takes over is not the debate, it is the **decision**. This number's practical counterpart is that the source becomes **comparable**. If the difference between two forms does not reach the tree, that difference takes up space in a change log but says nothing. Once six forms are run through the formatter, all of the difference between them is zeroed out; every difference left over is a difference that **reaches the tree**. The measurement says this directly: the distinct-text count drops from 6 to 1 while the distinct-tree count was already 1 and did not change. #### The Rule Written in One Place The guarantee a formatter gives is a conditional one: two runs operating under **the same rule** produce the same text. If the rule changes, the output changes too. So the tool alone is not enough; where the rule is written is part of the measurement. The rule can live in three separate places. If it lives **with the writer**, the result is as many separate texts as there are people — the measurement's starting state is exactly this, six forms, six separate texts. If it lives **in the tool's default**, the result depends on the tool's configuration, and when two development environments carry separate defaults, the number is more than one again. If it lives **in the project's own file**, the rule travels with the source and every run reads it. The third is the same shape as the arrangement this course built on the dependency side. A lock file did not remove resolution; it made the decision **once** and put it in writing. A formatting rule does not remove the debate either; it makes the debate happen once, writes the result to a file, and binds every subsequent run to that file. In both, what becomes single is not the work itself, it is **the source of the decision.** The difference between the two is how reversible the decision is. When a lock file is changed, the versions that get installed change, and the program runs with different code. When a formatting rule is changed, the produced text changes but the tree does not; the measurement's `behavior` column shows this across all six forms. A formatting decision is therefore a **cheap** decision: if it is made wrong, its cost is readability, not correctness. #### Where the Linter Falls Silent The `layout` column gives **0, 1, 1, 3, 6, 10** findings across the six forms. This number changes from form to form because the rule looks at text. The `structure` column stays at **6** across all six forms: rules that look at the tree never see layout at all, so they are unaffected by the form. Where the numbers come from can be read one by one. In the one-line-body form, three `if` lines produce the "two statements on the same line" finding: **3**. In the unspaced-operator form, two increment lines, one comparison line, and the three operators in the `return` line give findings: **6**. When both are applied together these add up and the indentation-step finding is added: **10**. Finding count grows not with the source's quality but with **how far the source sits from the canonical form.** The second row carries an important detail. Six forms give **5** distinct finding sets, not 6 — because the two-space and eight-space forms produce the exact same single finding: "indentation step is not four spaces." **Being a distinct text does not require being a distinct finding set.** A finding set is a coarse summary of the source and can name two separate flaws with the same label. After formatting, the finding-set count drops to **1** and **6** findings remain in that set: five unnamed constants (`10`, `15`, `20`, `20`, `100`) and one decision-count finding (`3`, threshold `2`). The formatter silenced none of these, because none of them concerns layout. **A formatter is not a quality tool, it is a unification tool.** Why the remaining six findings cannot be silenced gathers into one sentence too: **none of them has a single correct fix.** Giving the `20` constant a name requires choosing the name, and what knows the right name is not the source itself, it is the domain's rule. Dropping the decision count from three to two requires splitting the body, and where to split cannot be read from the tree. For every layout finding, by contrast, the correct outcome is single and follows from the rule — this is why those can be handed off to a tool and the others cannot. The last line ties off the measurement. In **1** of four input combinations, real behavior deviates from the oracle: when both membership and coupon are given, the function returns **80**, while the oracle, applying its 25 ceiling, expects **75**. The number of findings the linter reports is **6**, the number of findings that report this deviation is **0**. This is the same shape as the previous lesson's result, and it sits on a different axis. Coverage counted **where the code ran** and did not see the flaw; the linter checks **how the code is written** and does not see the same flaw either. Neither knows the oracle. The only thing that knows the oracle is the test's **expectation**, and in this setup that expectation has written down the flawed value. #### Summary - A formatter rewrites the source and reduces separate forms to a single text; a linter does not touch the source, it reports findings. - The same function's **6** separate forms give **1** parse tree and **1** behavior; distinct-text count after formatting drops to **1**. - Rules that look at text produce between **0** and **10** findings across forms; rules that look at the tree stay fixed at **6** findings across all six forms. - Finding-set count is **5** before formatting, **1** after; being a distinct text does not require being a distinct finding set. - None of the **6** findings left after formatting reports the single call that deviates from the oracle; a formatting tool is not a quality tool. #### Next Step This lesson ran in a single environment: one interpreter, one dependency set, one run. What was measured was the source itself, and the source is independent of the environment. The same cannot be said for tests. The same team, as the shared setup showed, could give a separate decision just by order; order was a configuration detail. Interpreter version and the resolved dependency set are configuration details too. The next lesson opens this axis: running the same team across how many environments gives how many separate decisions, and which decisions bring that number back down? ### Multi-Environment Testing Source: https://academia.sh/en/courses/python-projects/multi-environment-testing The same seven-test team gives 4 distinct decisions across 18 runs on the axes of three interpreters, two resolutions, and three orders; the lock file drops runs to 9 and decisions to 3, the fixture to 3 and 2, and the remaining two decisions are the real environment difference itself. The previous lesson ran in a single environment. What it measured was the source itself: six forms, one tree, one behavior. Whichever interpreter reads the source text, the same tree comes out, which is why that measurement was independent of the environment. The same cannot be said for tests. A test runs not just the code but also **the rig the code sits inside**: which interpreter, which dependency set, which order. The shared setup's five-test team could give a separate decision just by order. This lesson's question widens that observation: **how many distinct decisions does the same team give across how many environments, and which decisions bring that number down?** #### How Many Axes Build an Environment "Environment" is not a single thing, it is a **product of axes.** This measurement has three axes. **The interpreter axis.** Three fictional interpreters — `I1`, `I2`, `I3` — are each modeled by a **capability set.** No real version number is written; the difference between interpreters is reduced to whether a single capability is present or not. `I1` does not carry the `union` capability, the other two do. **The resolution axis.** The same dependency declaration is resolved with two strategies: **newest** and **oldest compatible.** As the shared setup showed, this produces two separate version sets from the same declaration. A third option is a lock file, and it stops the strategy from changing the result. **The order axis.** The team is run in three separate orders. A test that pollutes shared state and a test that assumes a clean environment sit in the same team; which one runs first changes the result. The team is seven tests: the shared setup's five tests, plus two tests that read the environment. The `capability` test looks at the interpreter's capability set, the `version` test requires the resolved `report` version to be at least `1.1`. The five tests' behavior is not touched. How a pipeline defines this matrix, which job runs on which runner, and where the result gets reported was established on the operations side in the Continuous Integration and Delivery course and is not repeated here. This lesson has no pipeline; the only thing measured is **how many distinct decisions come out of how many environments.** #### The Matrix's Cost The axes multiply: three interpreters, two resolutions, three orders — **18** runs. Since the seven-test team runs from scratch on every run, a total of **126** test calls are made. As axis count rises, this number grows multiplicatively; adding one more axis multiplies run count by two or three. Against this, the matrix's **information yield** does not grow at the same rate. A matrix can include axis points that do not produce a separate decision; `I2` and `I3` carry the same capability set in this measurement and never separate from each other in any run. The measurement shows this directly. Two conditions are needed for an axis to belong in the matrix. **It has to be able to change the decision** — if it cannot, it only grows run count without adding information. **It has to be non-eliminable** — if it can be fixed by a configuration decision, fixing it is preferable to keeping it in the matrix. These two conditions are this lesson's definition of what it measures: the order and resolution axes satisfy the first condition but fail the second; the interpreter axis satisfies both and stays in the matrix. Duration is not measured here either. When a matrix's cost is written as run count and call count, it stays reproducible; the same matrix makes the same **126** calls on another machine. The rule of writing down what a measurement counts was established at the previous course's close, and it continues in this course too. The measurement's assumptions: - **QP8** — Interpreters are fictional and are separated only by capability sets; no real version number is written, no interpreter implementation is named. - **QP9** — The shared setup's five tests are taken as is; the `polluting` and `dependent` tests' behavior is not changed. The two added tests do not touch shared state, they only read the environment. - **QP10** — The `version` test redoes resolution on every call; when the lock is chosen, the locked set is read instead of resolution. - **QP11** — When the fixture is on, the shared environment is rebuilt **before every test.** This effect of the fixture was measured in this course's fixtures-and-parametrization lesson; here it is used as an **elimination tool**, not measured again. - **QP12** — A run's decision is its passed-test count. If two runs give the same number, **one** distinct decision is counted, even if the names of the failed tests differ. - **QP13** — No duration is measured. The matrix's cost is written as **run count** and **test calls.** - **QP14** — No real install is done, no real package is downloaded; the interpreter and the dependency set are entirely modeled inside the lesson. #### Measurement ```python """Multi-environment testing: same team, how many environments, how many distinct decisions.""" REGISTRY = {"metrics": [(1, 0), (1, 1), (1, 2), (2, 0)], "report": [(0, 9), (1, 0), (1, 1)], "common": [(3, 0), (3, 1), (3, 2), (4, 0)]} DECLARATION = {"metrics": ((1, 0), (2, 0)), "report": ((1, 0), (2, 0)), "common": ((3, 0), (4, 0))} def resolve(strategy): chosen = {} for package in sorted(DECLARATION): lo, hi = DECLARATION[package] candidates = [s for s in REGISTRY[package] if lo <= s < hi] chosen[package] = candidates[-1] if strategy == "newest" else candidates[0] return chosen LOCK = dict(resolve("newest")) class Environment: def __init__(self): self.counter = 0 def t_clean_a(e, i, s): return True def t_polluting(e, i, s): e.counter += 1 return True def t_dependent(e, i, s): return e.counter == 0 def t_clean_b(e, i, s): return True def t_clean_c(e, i, s): return True def t_capability(e, i, s): """Requires the interpreter's union capability.""" return "union" in INTERPRETER[i] def t_version(e, i, s): """Requires the resolved report version to be at least 1.1.""" return (LOCK if s == "lock" else resolve(s))["report"] >= (1, 1) TESTS = [("clean_a", t_clean_a), ("polluting", t_polluting), ("dependent", t_dependent), ("clean_b", t_clean_b), ("clean_c", t_clean_c), ("capability", t_capability), ("version", t_version)] ORDERS = {"source order": [0, 1, 2, 3, 4, 5, 6], "dependent first": [0, 2, 1, 3, 4, 5, 6], "reverse": [6, 5, 4, 3, 2, 1, 0]} INTERPRETER = {"I1": {"a"}, "I2": {"a", "union"}, "I3": {"a", "union"}} CALLS = 0 def run(i, s, order, fixture=False): """When the fixture is on, the shared environment is rebuilt before every test.""" global CALLS e, passed, failed = Environment(), 0, [] for idx in order: name, fn = TESTS[idx] if fixture: e = Environment() CALLS += 1 if fn(e, i, s): passed += 1 else: failed.append(name) return passed, failed print(f"{'interpreter':<12s} {'resolution':<10s} {'order':<16s} {'passed':>6s} failed") decisions = [] for i in INTERPRETER: for s in ("newest", "oldest"): for oname, order in ORDERS.items(): passed, failed = run(i, s, order) decisions.append(passed) print(f"{i:<12s} {s:<10s} {oname:<16s} {passed:6d} " f"{','.join(failed) or '-'}") print(f"\n{len(decisions)} runs, {len(set(decisions))} distinct decisions: " f"{sorted(set(decisions))}") print(f"test calls {CALLS}") print() print(f"{'phase':<34s} {'runs':>6s} {'distinct decision':>18s} {'calls':>6s}") for name, strategies, orders, fixture in ( ("raw matrix", ("newest", "oldest"), ORDERS, False), ("with lock file", ("lock",), ORDERS, False), ("lock and fixture", ("lock",), {"one": ORDERS["source order"]}, True)): CALLS, result = 0, [] for i in INTERPRETER: for s in strategies: for order in orders.values(): result.append(run(i, s, order, fixture)[0]) print(f"{name:<34s} {len(result):6d} {len(set(result)):18d} {CALLS:6d} " f"{sorted(set(result))}") print() for i in INTERPRETER: passed, failed = run(i, "lock", ORDERS["source order"], True) print(f"{i}: {passed}/7 passed, failed: {','.join(failed) or '-'}") ``` ``` interpreter resolution order passed failed I1 newest source order 5 dependent,capability I1 newest dependent first 6 capability I1 newest reverse 6 capability I1 oldest source order 4 dependent,capability,version I1 oldest dependent first 5 capability,version I1 oldest reverse 5 version,capability I2 newest source order 6 dependent I2 newest dependent first 7 - I2 newest reverse 7 - I2 oldest source order 5 dependent,version I2 oldest dependent first 6 version I2 oldest reverse 6 version I3 newest source order 6 dependent I3 newest dependent first 7 - I3 newest reverse 7 - I3 oldest source order 5 dependent,version I3 oldest dependent first 6 version I3 oldest reverse 6 version 18 runs, 4 distinct decisions: [4, 5, 6, 7] test calls 126 phase runs distinct decision calls raw matrix 18 4 126 [4, 5, 6, 7] with lock file 9 3 63 [5, 6, 7] lock and fixture 3 2 21 [6, 7] I1: 6/7 passed, failed: capability I2: 7/7 passed, failed: - I3: 7/7 passed, failed: - ``` #### Eighteen Runs, Four Decisions The top table lays the same team's eighteen runs side by side. Passed-test count ranges between **4** and **7**; there are **4 distinct decisions.** Reading the four one by one shows where the number comes from. The worst row is `I1` / `oldest` / `source order`: **4/7**, with the three failing tests `dependent`, `capability`, and `version`. The best rows are `I2` and `I3`'s `newest` / `dependent first` and `reverse` runs: **7/7**, nothing failing. What deserves attention is that the difference comes from **three separate sources.** The `dependent` test failing comes from the order axis, the `version` test failing comes from the resolution axis, the `capability` test failing comes from the interpreter axis. All three gather into the same column and blend into a single number — passed-test count. If a report showed only this number, four separate values would remain and none of them would say which axis was responsible. The `failed` column exists for exactly this reason: **decision count gives the difference's existence, the failing test's name gives the difference's source.** Growing the matrix by one more axis grows the first number, it does not give the second. #### Same Number, Separate Cause The value **5** appears in five separate rows in the top table, and the failing tests in those five rows form three separate sets: `dependent` with `capability`, `capability` with `version`, `dependent` with `version`. The same number, coming out for three separate reasons. This is the measurement's most commonly misread spot. Two runs giving the same number does not show they are in the same state; it only shows their **failing-test count** is equal. When counting decision count, we count these rows as one — the assumption block states this plainly — because what is measured is distinct-outcome count, not distinct-cause count. But when investigating a failure, the question reverses and the cause has to be separated. The two rows under `reverse` order carry a small example of the same distinction: in the `I1` / `oldest` / `reverse` run, the failing tests are written `version,capability` in that order; in the `dependent first` run, `capability,version`. The set is the same, the order is different — because the list fills in the order the tests run. It has no effect on the number, but a tool comparing two dumps as text would count them as separate. A practical result follows from this: **if a run report carries only the number, separate causes get gathered into the same row.** If the failing tests' names stand in the report, the three causes can be separated; if they do not, the matrix has made eighteen runs and only four numbers are left. The matrix's cost of widening is multiplicative; the information it yields depends on what the report carries. #### Eliminating Axes The middle table measures three stages of elimination. **Raw matrix**: **18** runs, **4** distinct decisions, **126** test calls. This is the state where no axis is fixed. **With lock file**: the resolution axis drops, because the lock stops the strategy from changing the result. Runs drop to **9**, calls to **63**, distinct decisions to **3**. The decision lost is `4`; the `version` test no longer fails in any run. The lock does not remove resolution — it makes the decision once and puts it in writing — but it **entirely** removes one axis from the environment matrix. **With lock and fixture**: the order axis drops too. Runs drop to **3**, calls to **21**, distinct decisions to **2**. The `dependent` test no longer fails in any order, because the shared environment is rebuilt before every test. The remaining **2** distinct decisions sit here and cannot be brought down further. The bottom rows show this: `I1` **6/7**, `I2` and `I3` **7/7**. The only test that fails is `capability`, and the reason it fails is not a configuration detail, it is that the interpreter genuinely does not carry a capability. This is exactly the difference multi-environment running is looking for. The order of elimination decides the intermediate numbers, not the final number. Had the fixture been applied before the lock, the middle row would show a different number than **3**, because the axis eliminated at that stage would be a different one. Each elimination step drops one axis entirely and takes that axis's contribution to the decision along with it; once both axes are dropped, the same **2** decisions remain regardless of the order they were dropped in. Intermediate numbers belong to the method, the final number belongs to the environment. The lesson's rule follows from this: **multi-environment running is for finding environment-sourced difference; if configuration-sourced difference is not eliminated first, which axis is responsible cannot be known.** The eighteen-run matrix gave four decisions, and two of them were not even about the environment. Elimination, while dropping run count to a sixth, loses only information that was never this measurement's subject to begin with. One last reading concerns the matrix's width. Three interpreters were run but distinct decisions came out to **2**; `I2` and `I3` never separated in any run, because their capability sets are the same. Adding one more point to the matrix grows run count and call count linearly but **may not** grow distinct-decision count. The matrix's width is a cost, distinct-decision count is a gain, and the two do not grow at the same rate. #### Summary - An environment is not a single thing, it is a product of axes; here three axes produce **18** runs and **126** test calls. - The same seven-test team gives **4 distinct decisions** in this matrix (**4**, **5**, **6**, **7**); the difference comes from three separate axes and blends into a single number. - The failing test's name gives the difference's source: `dependent` from order, `version` from resolution, `capability` from the interpreter. - The lock file drops the resolution axis — **9** runs, **3** distinct decisions; the fixture drops the order axis — **3** runs, **2** distinct decisions. - The remaining **2** decisions cannot be brought down and are the real environment difference: `I1` **6/7**, `I2` and `I3` **7/7**. Although three interpreters were run, distinct-decision count is two; a matrix widening does not have to grow distinct-outcome count. #### Next Step This lesson moved the team from environment to environment and assumed the same source ran in every environment. That assumption is silent and was not tested: was what ran genuinely the same source? When a project is handed off for someone else's use, the source does not go directly; it is packaged into a **distribution format**, and that package is unpacked and installed. The next lesson measures this step: how many distinct contents does a distribution produced from the same source give, which inputs grow that number past one, and what fixing brings it back to one? ### Packaging Source: https://academia.sh/en/courses/python-projects/packaging Nine builds made from the same source across three directory states and three inclusion rules give 5 distinct contents; the explicit list drops this number to 1, only using sorted order together with a source-derived stamp unifies the archive digest, and when the build step runs drops install from 3 distinct outcomes to 1. The previous lesson ran the same team on a three-axis matrix and showed that the remaining difference was the real environment difference. That measurement had a silent assumption: what ran in every environment was **the same source**. Tests were read straight from the working directory. Once a project is handed off for someone else's use, this assumption is not automatically satisfied. The source directory does not travel as is; it is packaged into a **distribution format**, that package is carried, and it is unpacked on the other side. This lesson's question is not how packaging is done: **how many distinct contents does a distribution produced from the same source give, and what fixing brings that number down to one?** #### Distribution Format and Build System Two distribution formats are separated, and the difference between them is this lesson's final measurement's subject. A **source distribution** carries source files and the project manifest. A **build step** runs on the installing side and produces the tree to be installed. A **built distribution** carries that step's result instead; only the unpacking work is left on the installing side. The tool class that produces both these formats is the **build system.** Its job is to produce the answer to three questions: which files go into the distribution, in what order the distribution is written, and when the build step runs. This course does not write tool names; the build system is **modeled** here with three functions — an inclusion rule, an archive writer, and a build step. The model produces no real archive, writes no file to disk, and downloads no package; the only thing it produces is content digests. #### Three Directory States What the inclusion rule does depends on **what stands next to the source.** A working directory is not just the source: build artifacts, cache files, and local settings files accumulate there too. The measurement defines three states. A **clean** directory carries only the source. A **cached** directory has one generated file. A **cluttered** directory also has a log file and a local settings file. Three inclusion rules are tried. **Collect everything** takes every file in the directory. **Exclude by pattern** leaves specific extensions out. **Explicit list** takes only the files named in the manifest. The measurement's assumptions: - **QP15** — The source tree is fixed and the same across all three states; states are separated only by the files standing next to the source. - **QP16** — The distribution's content is the digest of file names and bodies, sorted by name; directory listing order does not enter this digest. - **QP17** — The archive digest, by contrast, is sensitive to order and to the stamp; two runs' listing order and run stamp are modeled with **fictional** values, no real clock is read. - **QP18** — The build step produces a single file from the source and writes the name of the environment it runs in into it. A real build step does far more than this; the only property the model keeps is that **its output depends on the environment it runs in.** - **QP19** — Digests are truncated to eight hexadecimal digits; two digests being equal is read as their content being equal. - **QP20** — No file is written to disk, no archive is produced, no package is downloaded; the entire build is modeled in memory. - **QP21** — No duration is measured. What is counted is **distinct content**, **distinct digest**, and **distinct install** count. #### Measurement ```python """Packaging: how many distinct contents does a distribution from the same source give.""" import hashlib SOURCE = { "metrics/__init__.py": "from .discount import discount\n", "metrics/discount.py": "def discount(amount, member, coupon):\n return amount\n", "metrics/rate.json": '{"ceiling": 20}\n', "tests/test_discount.py": "def test_ceiling():\n pass\n", "README.md": "# metrics\n", "project.toml": '[project]\nname = "metrics"\n', } CLUTTER = { "clean": {}, "cached": {"metrics/__cache__/discount.chc": "compiled\n"}, "cluttered": {"metrics/__cache__/discount.chc": "compiled\n", "metrics/temp.log": "run log\n", "local_settings.txt": "path = /home/user\n"}, } MANIFEST = ("metrics/__init__.py", "metrics/discount.py", "metrics/rate.json", "project.toml") RULES = { "collect everything": lambda name: True, "exclude by pattern": lambda name: not name.endswith((".chc", ".log")), "explicit list": lambda name: name in MANIFEST, } def tree(state): return {**SOURCE, **CLUTTER[state]} def digest(text): return hashlib.sha256(text.encode()).hexdigest()[:8] def content_digest(files): """A distribution's content: name and body, sorted by name.""" return digest("".join(f"{a}\n{files[a]}" for a in sorted(files))) def select(state, rule): d = tree(state) return {a: g for a, g in d.items() if RULES[rule](a)} print(f"{'inclusion':<18s} " + " ".join(f"{d:>14s}" for d in CLUTTER) + " distinct content") total = [] for rule in RULES: row, digests = [], [] for state in CLUTTER: chosen = select(state, rule) d = content_digest(chosen) digests.append(d) total.append(d) row.append(f"{len(chosen)}·{d}") print(f"{rule:<18s} " + " ".join(f"{s:>14s}" for s in row) + f"{len(set(digests)):11d}") print(f"nine builds, {len(set(total))} distinct content") print() print(f"{'order rule':<18s} {'stamp rule':<22s} {'run A':>9s} " f"{'run B':>9s} distinct") def order(files, run): return sorted(files, reverse=(run == "run B")) def archive(files, sorted_order, from_source, run, stamp): names = sorted(files) if sorted_order else order(files, run) d = content_digest(files) if from_source else stamp return digest("".join(f"{a}\n{files[a]}" for a in names) + f"stamp={d}\n") CHOSEN = select("cluttered", "explicit list") STAMP = {"run A": "s-101", "run B": "s-207"} for s_name, sorted_order in (("listing order", False), ("sorted by name", True)): for d_name, from_source in (("run stamp", False), ("derived from source", True)): a = [archive(CHOSEN, sorted_order, from_source, k, STAMP[k]) for k in STAMP] print(f"{s_name:<18s} {d_name:<22s} {a[0]:>9s} {a[1]:>9s}" f"{len(set(a)):6d}") print() def build(files, environment): """Build step: produces one file from the source.""" return {**files, "metrics/_build.py": f'BUILT = "{environment}"\n'} sdist = CHOSEN built_dist = build(CHOSEN, "production environment") print(f"{'distribution format':<20s} " + " ".join(f"{d:>10s}" for d in CLUTTER) + " distinct install") for name, install in (("source distribution", lambda e: content_digest(build(sdist, e))), ("built distribution", lambda e: content_digest(built_dist))): o = [install(d) for d in CLUTTER] print(f"{name:<20s} " + " ".join(f"{x:>10s}" for x in o) + f"{len(set(o)):13d}") ``` ``` inclusion clean cached cluttered distinct content collect everything 6·7a83a9db 7·0bf44f2f 9·12b06330 3 exclude by pattern 6·7a83a9db 6·7a83a9db 7·bd9cba48 2 explicit list 4·c9929633 4·c9929633 4·c9929633 1 nine builds, 5 distinct content order rule stamp rule run A run B distinct listing order run stamp 6d80f316 547a38b3 2 listing order derived from source 5455d4ab ecb04028 2 sorted by name run stamp 6d80f316 1fd59d06 2 sorted by name derived from source 5455d4ab 5455d4ab 1 distribution format clean cached cluttered distinct install source distribution 6897a192 85226fbb fee3c544 3 built distribution 1440a878 1440a878 1440a878 1 ``` #### Nine Builds, Five Contents The top table gives the lesson's first number: **9** builds made from the same source give **5 distinct contents.** Reading the rows one by one shows where the number comes from. The **collect everything** rule takes **6**, **7**, and **9** files across the three directory states and gives **3 distinct contents.** This rule's result depends not on the source but on **what stands next to the source**; two working directories pulled from the same repository produce separate packages. The **exclude by pattern** rule gives the same result in two states — `clean` and `cached` carry the same digest, `7a83a9db` — because the extension it excludes catches the cache file. In the third state, though, the `local_settings.txt` file does not match the pattern and enters the content: **7** files, a separate digest. A rule can only exclude as much as it **knows** to. The **explicit list** gives **4** files and a single digest, `c9929633`, in all three states: **1 distinct content.** The difference is methodological. The first two rules say "what stays out," and every new unlisted file silently comes in. The third rule says "what comes in," and every unlisted file silently stays out. **The direction of silence changes, and unification comes from that change of direction.** Two digests appear in two of the nine builds at once: `7a83a9db` is in both the `collect everything`/`clean` and `exclude by pattern`/`clean` cells. Nine cells, five distinct contents; where rules coincide, the count drops. File counts make one more distinction visible. The explicit list takes **4** files, while the clean directory has **6** source files. The two left out are `tests/test_discount.py` and `README.md`. This is not a flaw, it is a decision: the distribution carries the **installed** tree, not everything in the repository. The test file stays in the repository and runs there; if it entered the distribution, it would take up disk space and import namespace on the installing side. The same decision could be made for the documentation file too, and the reverse is equally defensible — what is not defensible is the decision being **never made.** In the first two rules this decision is never made at all; whether a file enters the distribution is decided by its name's extension. #### The Manifest's Two Jobs All three inclusion rules are written somewhere, and that place is the project manifest. The manifest does two separate jobs, and because the two often sit in the same file, they get confused. **First job: the package's identity.** Name, version, dependency declaration, and the files that will enter the distribution. These define data; which tool reads them does not change the result. **Second job: choosing the build system.** Which build system will produce the distribution, and what interface that system will be called with. This is a directive; it directly decides the result. The distinction's practical counterpart is this: the first job is **declarative**, and two separate tools reading the same manifest choose the same file set. The second job is a **choice**, and the layout of the produced archive, its stamp, and the build step's behavior change depending on the system chosen. The measurement's top table tests the first job, its middle table tests the second. This is why a manifest being "complete" has two separate meanings. If the file list is complete, the distribution's content unifies. If the build system and its calling interface are also written, every run that produces the distribution calls the same system the same way. When the second is not written, production falls back to the tool's default and the shared setup's resolution measurement's situation repeats: **no decision was made, so the environment gives one.** #### Two Decisions That Fix the Archive Content unifying does not mean the archive unifies. The middle table archives the same file set — the four files the `explicit list` rule chose — in two runs and lays out four rule combinations side by side. In three combinations, the two runs give **2 distinct digests.** In one combination, **1**. When **listing order** is used, run A and run B write the files in separate order; the body is the same, the byte string is separate. This difference remains even when the stamp is derived from the source: the second row gives `5455d4ab` against `ecb04028`. When **run stamp** is used, the difference remains even when order is fixed: the third row gives `6d80f316` against `1fd59d06`. Only the fourth row gives **1**: order fixed by name **and** the stamp derived from the source. Both are needed, one is not enough. This is the definition of **reproducibility**: the same output from the same input, run to run. This also explains why the archive digest is measured separately. The installing side cannot compute the content itself; what it gets is an archive, and it can only verify it through its **bytes.** Even if the content is single, if the archive's bytes change run to run, it cannot be confirmed that two archives came from the same source. The reproducibility claim is therefore built not at the content level but at the **byte level.** Because the `0ef2d766`-equivalent digest in the third row matches the one in the first row, a detail becomes visible: run A's listing order is already sorted by name, so in that run the two order rules give the same result. The difference only shows up in run B. **A configuration flaw does not show up in every run; in the run where it does not show up, it is assumed not to exist.** #### When the Build Step Runs The bottom table gives the last measurement and separates the two distribution formats. The **source distribution** gives **3 distinct installs** across the three directory states. The reason does not show up in the table but is explicit in the model: the build step runs on the installing side and writes the name of the environment it runs in into the file it produces. The installed tree therefore changes from environment to environment. The distribution itself was single — four files, one digest — but the install outcome is not. The **built distribution** gives **1 distinct install** across all three states. The build step ran once, on the production side; the installing side only unpacks. The difference between the two is **where the build step runs**, and this is a distribution format decision. A source distribution carries the build to the installing side; in return, the installing side can produce output suited to its own environment. A built distribution does the build once; in return, it cannot reflect the environment difference into the distribution. This does not mean the built distribution is free. If the build step reads something from the environment, the built distribution **embeds** the value it read and now only fits that environment. In that case which environment the distribution fits has to be written on the package itself; if it is not, the installing side unpacks a package that does not fit, and the error shows up much later than install, during the run. The source distribution's **3** distinct installs are not a flaw, they are this adaptation itself; the flaw would be the three distinct outcomes being **unexpected.** The principle of a build output produced once and distributed everywhere was established on the operations side in the DevOps Culture and Fundamentals course. The measurement here gives that principle's counterpart on the language side, as a number: when the principle is applied, install outcome is **1**; when it is not, it is as many as the environment count. To gather the three measurements into one sentence: **for a distribution from the same source to give a single content, three things have to be fixed together** — the explicit list of files to include, the archive's write order and stamp, and which side the build step runs on. If any one of the three is left open, the number is more than one, and its size depends on how much the open spot is affected by the environment. #### Summary - Distribution format splits in two: a source distribution carries the build step to the installing side, a built distribution carries that step's result. - **9** builds from the same source give **5 distinct contents**; `collect everything` gives **3**, `exclude by pattern` **2**, `explicit list` **1** distinct content. - Excluding rules silently let an unlisted file in; an explicit list silently leaves it out. Unification comes from this change of direction. - For the archive digest to unify, order **and** stamp both have to be fixed together; only one of four combinations gives **1** distinct digest, three give **2**. - The source distribution gives **3 distinct installs** across three environments, the built distribution gives **1**; even when the distribution itself is single, install outcome may not be. #### Next Step This lesson produced the distribution and named its content with a digest. The digest is a number computed on the production side, and it stays there. Once a package is sent to a **package registry**, though, it is given a name and a version; the installing side no longer asks for the content, it asks for the **name.** The next lesson's question grows from here: can a published version's content change afterward, can the same name carry two distinct contents, and if it can, what does the version number the lock file writes actually guarantee? ### Publishing Source: https://academia.sh/en/courses/python-projects/publishing Nine publish attempts produce 9 accepted and 2 silent changes in an overwritable registry, 7 accepted and 2 rejected in an immutable one; the same lock file gives 2 distinct installs in the first registry, and a lock that also writes the digest catches both of those two mismatches and stops the install. The previous lesson produced the distribution and named its content with a digest. That digest was a number computed on the production side, and it stayed there; the installing side never knew about it. Once a package is sent to a **package registry**, this arrangement changes. The installing side no longer asks for the content, it asks for the **name**: "version `1.2` of package `metrics`." The registry gives back the name's counterpart. This lesson's question grows from here: **can the same name carry two distinct contents, and if it can, what does the version number the lock file writes actually guarantee?** #### What the Registry Stores A package registry is a mapping from an `(name, version)` pair to a distribution. This course does not write a registry's product name; the registry is **modeled** here with a single dictionary and a single operation is defined on it: publish. The model does not connect to a real registry, nothing is sent, and no package is downloaded. Version marking itself — tagging and semantic versioning — was established in the Introduction to Version Control course and is not repeated here. This lesson does not ask how the version number is chosen; it asks **how binding a chosen number is.** #### Two Policies What the publish operation does when the `(name, version)` pair has already been used is a policy question, and it has two answers. An **overwritable** registry puts the new body in the old one's place. The submission always succeeds. An **immutable** registry rejects the submission when a **distinct** body arrives for the same pair. It does not reject the same body being submitted again — that submission does not change the registry anyway. This is exactly where the distinction lies: immutability forbids not **resubmission**, it forbids **resubmission with distinct content.** Both can sit side by side in a publish pipeline, and the measurement includes both. The measurement defines nine publish attempts. Three publish the shared setup's lock-file versions for the first time, one publishes an older version, one resubmits the same body, two send distinct bodies for the same version, one sends the same body once more, one opens a new version. Two points in time are read: `t1` after the fifth attempt, `t2` after the ninth. The measurement's assumptions: - **QP22** — The lock file is the result of the shared setup's "newest" resolution: `metrics 1.2`, `common 3.2`, `report 1.1`. The lock's content does not change over the course of the measurement. - **QP23** — The distribution's content is represented by a single body string; its digest is a truncated eight-hex-digit value, and two digests being equal is read as content being equal. - **QP24** — Install is reading the registry's digests for the three versions in the lock file. No other side effect of install is modeled. - **QP25** — `t1` and `t2` are two separate installs done with the same lock file. The only difference between them is the publish attempts that reach the registry in that span. - **QP26** — A lock that also writes the digest compares the expected and the arriving digest at install time; if it finds a mismatch, install **stops.** It does not try to fix the mismatch. - **QP27** — Yanking **closes a version to resolution** but does not touch its content in the registry; because a lock file asks for the version directly, it never runs resolution at all. - **QP28** — No duration is measured, no real submission is sent to a registry. What is counted is **accepted**, **rejected**, **mismatch**, and **distinct install** count. #### Measurement ```python """Publishing: can the same name carry two distinct contents.""" import hashlib LOCK = {"metrics": (1, 2), "common": (3, 2), "report": (1, 1)} REGISTRY = {"report": [(0, 9), (1, 0), (1, 1)]} DECLARATION = {"report": ((1, 0), (2, 0))} YANKED = {("report", (1, 1))} ATTEMPTS = [ ("report", (1, 0), "report-body-0"), ("metrics", (1, 2), "metrics-body-1"), ("common", (3, 2), "common-body-1"), ("report", (1, 1), "report-body-1"), ("metrics", (1, 2), "metrics-body-1"), ("report", (1, 1), "report-body-2"), ("metrics", (1, 2), "metrics-body-2"), ("common", (3, 2), "common-body-1"), ("metrics", (2, 0), "metrics-body-3"), ] T1 = 5 POLICIES = (("overwritable", False), ("immutable", True)) def digest(text): return hashlib.sha256(text.encode()).hexdigest()[:8] def publish(attempts, immutable): """Package registry model: (name, version) -> distribution digest.""" registry, accepted, rejected, conflicts = {}, 0, 0, [] for package, version, body in attempts: key, d = (package, version), digest(body) existing = registry.get(key) if existing is not None and existing != d: conflicts.append(f"{package} {version[0]}.{version[1]}") if immutable: rejected += 1 continue registry[key] = d accepted += 1 return registry, accepted, rejected, conflicts def install(registry): """Versions in the lock file are read from the registry.""" return {p: registry[(p, v)] for p, v in LOCK.items()} print(f"{'policy':<20s} {'accepted':>8s} {'rejected':>8s} {'t1 install':>28s}" f" {'t2 install':>28s} {'distinct':>8s}") for name, immutable in POLICIES: b = install(publish(ATTEMPTS[:T1], immutable)[0]) registry, accepted, rejected, conflicts = publish(ATTEMPTS, immutable) g = install(registry) d1 = " ".join(b[p] for p in sorted(LOCK)) d2 = " ".join(g[p] for p in sorted(LOCK)) print(f"{name:<20s} {accepted:8d} {rejected:8d} {d1:>28s} {d2:>28s}" f" {len({d1, d2}):8d}") print(f"conflicting submission: {', '.join(publish(ATTEMPTS, True)[3])}") print() print(f"{'lock format':<16s} {'policy':<20s} {'mismatch':>9s} " f"{'caught':>7s} {'install':>9s} {'distinct content':>16s}") for kb, with_digest in (("version only", False), ("version and digest", True)): for name, immutable in POLICIES: b = install(publish(ATTEMPTS[:T1], immutable)[0]) g = install(publish(ATTEMPTS, immutable)[0]) mismatched = sum(1 for p in LOCK if b[p] != g[p]) caught = mismatched if with_digest else 0 stopped = caught > 0 installed = ({tuple(sorted(b.items()))} if stopped else {tuple(sorted(b.items())), tuple(sorted(g.items()))}) print(f"{kb:<16s} {name:<20s} {mismatched:9d} {caught:7d} " f"{'stops' if stopped else 'continues':>9s} {len(installed):16d}") print() def resolve(package, apply_yank): lo, hi = DECLARATION[package] return [s for s in REGISTRY[package] if lo <= s < hi and not (apply_yank and (package, s) in YANKED)][-1] registry = publish(ATTEMPTS, True)[0] print(f"{'install path':<18s} {'before yank':>20s} {'after':>20s} " f"{'distinct':>8s}") for name, chooser in (("with resolution", lambda y: resolve("report", y)), ("with lock file", lambda y: LOCK["report"])): o = [f"{'.'.join(map(str, chooser(y)))}·{registry[('report', chooser(y))]}" for y in (False, True)] print(f"{name:<18s} {o[0]:>20s} {o[1]:>20s} {len(set(o)):8d}") ``` ``` policy accepted rejected t1 install t2 install distinct overwritable 9 0 2600b1c5 03525db7 30aad9cf 2600b1c5 7fdd5554 9388981e 2 immutable 7 2 2600b1c5 03525db7 30aad9cf 2600b1c5 03525db7 30aad9cf 1 conflicting submission: report 1.1, metrics 1.2 lock format policy mismatch caught install distinct content version only overwritable 2 0 continues 2 version only immutable 0 0 continues 1 version and digest overwritable 2 2 stops 1 version and digest immutable 0 0 continues 1 install path before yank after distinct with resolution 1.1·30aad9cf 1.0·86e9c6b2 2 with lock file 1.1·30aad9cf 1.1·30aad9cf 1 ``` #### Nine Attempts, Two Rejections The top table compares the two policies on the same flow. The **overwritable** registry accepts all nine of the nine attempts: **9 accepted, 0 rejected.** No submission fails, no warning comes out. And yet the registry's content changes, and that change shows up in the `t1` and `t2` columns: `metrics 1.2`'s digest goes from `2600b1c5` to `7fdd5554`, `report 1.1`'s digest goes from `03525db7` to `9388981e`. The same lock file gives **2 distinct installs.** The **immutable** registry produces **7 accepted, 2 rejected** on the same flow. The two rejected are `report 1.1` and `metrics 1.2`; both are pairs already published earlier with a distinct body. The two attempts that resubmit the same body, by contrast, are accepted — the fifth and eighth attempts do not change the registry, so there is nothing to reject. The `t1` and `t2` columns are exactly identical: **1 distinct install.** The number reads like this: **9 accepted, 0 rejected** is not a health indicator. Both registries took the same nine attempts; one handled the two conflicts **loudly**, the other **silently.** The one that stayed quiet looks like it has the cleaner report, and precisely because of that it carries less information. The guarantee immutability gives can be written in one sentence: **a name-and-version pair names at most one content over its lifetime.** Without this guarantee, a version number is not an identifier, it is only a label; the same label can be stuck onto two separate boxes. The same body being accepted twice does not contradict this guarantee and is practically necessary. A submission can be interrupted, its result may never reach the sender; the sender then resubmits the same body. If the immutable registry rejected this, the sending side would first have to ask whether the submission had actually gone through. Because the rule is built by looking at content, the second submission does not change the registry and nothing arises to reject — the measurement's **7 accepted** count includes both of these submissions too. The forbidden operation is not "resubmitting," it is **changing the registry.** #### The Two Promises a Version Number Carries A version number makes two separate promises, and they are often mistaken for one. **The first promise concerns compatibility:** which part of the number increased tells whether the change is backward compatible. This promise was established in the Introduction to Version Control course's tagging lesson and is not repeated here. **The second promise concerns identity:** the number names **a single content.** This lesson measures only the second promise, and the measurement shows that this promise does not come from the number itself; it comes from the registry's policy. In an overwritable registry, the number can give the first promise but not the second. That the two are independent can be seen directly in the measurement's top table. Both rejected submissions are trying to send a distinct body to an existing pair; neither has anything to do with the compatibility promise. Even if the body were a fully compatible fix, the rejection would be the same, because what is checked is not compatibility, it is **identity.** The cost a rejection makes the publishing side pay is clear too: if a change is to be published, **a new version number has to be opened.** The measurement's ninth attempt does exactly this and is accepted as `metrics 2.0`. Immutability is not a prohibition, it is a redirection: it does not block the fix, it makes the fix's **naming** mandatory. In return, the installing side knows the meaning of the number it holds does not change over time. #### What the Lock Writes The middle table measures how much the installing side can protect itself against this situation. A **lock that writes only the version** catches **0** out of **2 mismatches** in the overwritable registry. Install continues and **2 distinct contents** get installed. The lock file has, as far as it is concerned, done its job — the requested versions were installed — but what it installed was two separate things across two separate runs. The lock unified resolution, it **did not unify content.** A **lock that also writes the version and the digest** catches **both of the 2 mismatches** in the same registry, and install **stops.** Installed distinct-content count drops to **1**, because the second install never completes. What is gained here is not one install's success, it is **the visibility of failure**: where the registry stayed silent, the lock speaks up. The bottom two rows show the situation where the policy is correctly built: in the immutable registry, mismatch **0**, caught **0**, install continues, distinct content **1**. Writing the digest has no cost here and shows no gain either — because there is nothing to catch. From this, two separate guarantees turn out to be independent. Immutability is a guarantee the **registry** gives, and the installing side cannot verify it. The digest is the **installing side's own** verification and works independently of the registry's policy. When both are built together, the installing side does not have to trust the registry; if only the first is present, it has to trust it; if only the second is present, it does not trust it but often stops. #### Yanking Is Not Deletion The bottom table measures a third operation. After a version is published, it can turn out that it should not be used. **Yanking** closes a version to resolution but does not touch its content in the registry. The measurement separates the two. The installing side **with resolution** gets `report` version `1.1` and digest `30aad9cf` before the yank; afterward it gets version `1.0` and digest `86e9c6b2`. **2 distinct versions.** The installing side **with a lock file**, by contrast, gets `1.1·30aad9cf` in both cases: **1.** The difference comes from the lock file never running resolution at all. Yanking changes resolution's candidate list; a lock file does not use a candidate list, it asks for the version directly and the registry keeps giving it. This is both the lock's strength and its limit: it **continues** using a yanked version, and it does so silently. Deletion is a separate operation and gives a separate result: on a deleted version, install with a lock file fails too. Three operations — overwriting, yanking, deletion — produce three separate outcomes, and only two of them notify the installing side. Placing the three side by side defines what the registry owes the installing side. **Overwriting** changes the content and says nothing; the installing side only notices through the digest it keeps itself. **Yanking** does not change the content and is visible only to new resolutions; a locked install is unaffected. **Deletion** removes the content and stops both paths. The ordering is not one of severity, it is one of **visibility**: the quietest operation is the most damaging one, because in the other two the installing side gets a response either way. Where yanking stays silent has a name too. A project that does not refresh its lock file keeps using the yanked version and does so without an error message; the yank decision only reaches that project once the lock is refreshed. A lock is a decision that freezes resolution's result — and it also leaves out any decisions made afterward about what it froze. #### Summary - A package registry is a mapping from an `(name, version)` pair to a distribution; it can be built with two policies, overwritable and immutable. - The same nine attempts give **9 accepted, 0 rejected** in the first registry, **7 accepted, 2 rejected** in the second; immutability forbids not resubmission but resubmission **with distinct content.** - The same lock file gives **2 distinct installs** in the overwritable registry, **1** in the immutable one; the version number alone does not determine content. - A lock that writes only the version catches **0 of 2 mismatches**, one that also writes the digest catches **2** and stops install; the registry's policy and the installing side's verification are independent guarantees. - Yanking affects resolution, it does not affect the lock: the installing side with resolution sees **2 distinct versions**, the one with a lock sees **1.** #### Next Step A published package carries code to the installing side. It does not carry **how to use** that code. That information comes from documentation, and documentation can be produced two ways: written by hand, or generated from the source. The difference between the two ways can be measured with the same question asked in every lesson of this course. The last lesson asks it: which outcome does generating documentation from source unify, which does it not unify, and where it does not unify, what cannot take its place? ### Documentation Source: https://academia.sh/en/courses/python-projects/documentation Documentation generated from four source versions gives 2 distinct outcomes and stale-signature count stays at 0 across all four, while in the handwritten document it sits between 2 and 3; generation unifies the signature, it does not unify the claim in the summary text, and only a runnable example catches that claim. The previous lesson bound a published version's content to its name: in an immutable registry, an `(name, version)` pair names a single distribution. The package carries code to the installing side. It does not carry **how to use** that code. That information comes from documentation, and documentation can be produced two ways: written by hand, or generated from the source. This course's last lesson measures both with the same question: **which outcome does generation unify, which does it not?** #### Generation's Input and Output Approaches to generating documentation from source were established in the Technical Writing and Documentation course; staleness and review were addressed there too. That discussion is not repeated here. The only thing measured is **how many distinct outcomes generation drops to one.** This course does not write a tool's name. The **documentation generator** is modeled here with a single function: it parses the source, selects functions whose name does not start with an underscore, reads each one's signature and its summary text's first line, and sorts the result by name. A real generator does far more — it builds links, processes type information, produces page layout; the only property the model keeps is that **its output is entirely derived from the source.** The measurement defines four source versions: the canonical form, a form with function definitions reordered, a form with a private helper function added, and a form with two signatures changed. Alongside these stands a **handwritten document**; it was written with the old signatures and has not been updated by hand since that day. The measurement's assumptions: - **QP29** — Three functions are taken from the shared setup; the `discount` function's body is not changed and the flawed ceiling stays in place. - **QP30** — The `discount` function's summary text **states** that the ceiling is 25 percent; the body applies 20. This gap between the claim and the behavior is the setup itself. - **QP31** — The documentation generator only reads from the source and never compares any claim against behavior; the number of checks it performs is **0.** - **QP32** — The handwritten document is the same across all four source versions; it does not track the source at all. - **QP33** — Runnable examples are run with the standard library's doctest fixture. What a doctest is was established in this course's doctests lesson; here it is used as a **verification tool**, not rebuilt. - **QP34** — No duration is measured, no file is written. What is counted is **distinct document**, **stale signature**, and **failing example** count. #### Measurement ```python """Documentation: what generation unifies, what it does not.""" import ast import doctest import hashlib import types BODY = { "discount": ( "(amount, member, coupon)", '"""Applies the member and coupon discount; the rate ceiling is 25 percent.\n\n' " >>> discount(100, True, True)\n 75\n \"\"\"\n" " rate = 0\n if member:\n rate += 10\n" " if coupon:\n rate += 15\n" " if rate > 20:\n rate = 20\n" " return amount - amount * rate // 100\n"), "eligible": ( "(package)", '"""Returns the versions falling within the declaration\'s range.\n\n' ' >>> eligible("report")\n [(1, 0), (1, 1)]\n """\n' " lo, hi = DECLARATION[package]\n" " return [s for s in REGISTRY[package] if lo <= s < hi]\n"), "lock": ( "(chosen)", '"""Fixes resolution\'s result.\n\n' " >>> lock({\"report\": (1, 1)})\n {'report': (1, 1)}\n \"\"\"\n" " return dict(chosen)\n"), } HEADER = ('REGISTRY = {"report": [(0, 9), (1, 0), (1, 1)]}\n' 'DECLARATION = {"report": ((1, 0), (2, 0))}\n\n\n') HELPER = 'def _normalize(s):\n """Internal use."""\n return tuple(s)\n\n\n' NEW_SIGNATURE = {"discount": "(amount, member, coupon, ceiling)", "lock": "(chosen, verify)"} def build(names, helper=False, signature_changed=False): p = HEADER + (HELPER if helper else "") for name in names: sig, body = BODY[name] if signature_changed and name in NEW_SIGNATURE: sig = NEW_SIGNATURE[name] p += f"def {name}{sig}:\n {body}\n\n" return p VERSIONS = { "canonical": build(["discount", "eligible", "lock"]), "order changed": build(["lock", "eligible", "discount"]), "helper added": build(["discount", "eligible", "lock"], helper=True), "signature changed": build(["discount", "eligible", "lock"], signature_changed=True), } HANDWRITTEN = {"discount": "discount(amount, member)", "eligible": "eligible(package, declaration)", "lock": "lock(chosen)"} def generate(source): """Documentation generator model: public names' signature and summary are read from source.""" record = {} for d in ast.parse(source).body: if isinstance(d, ast.FunctionDef) and not d.name.startswith("_"): sig = ", ".join(a.arg for a in d.args.args) first = (ast.get_docstring(d) or "").split("\n")[0] record[d.name] = f"{d.name}({sig}) — {first}" return "\n".join(record[a] for a in sorted(record)) def signature(source, name): for d in ast.parse(source).body: if isinstance(d, ast.FunctionDef) and d.name == name: return f"{name}({', '.join(a.arg for a in d.args.args)})" return "" def doc_signature(doc, name): for s in doc.splitlines(): if s.startswith(name + "("): return s.split(" — ")[0] return "" def digest(text): return hashlib.sha256(text.encode()).hexdigest()[:8] print(f"{'source version':<24s} {'public names':>12s} {'doc digest':>12s} " f"{'stale in generated':>19s} {'stale in handwritten':>21s}") docs = [] for name, k in VERSIONS.items(): b = generate(k) docs.append(digest(b)) stale_gen = sum(1 for a in HANDWRITTEN if doc_signature(b, a) != signature(k, a)) stale_hw = sum(1 for a in HANDWRITTEN if HANDWRITTEN[a] != signature(k, a)) print(f"{name:<24s} {len(b.splitlines()):12d} {digest(b):>12s} " f"{stale_gen:>16d}/{len(HANDWRITTEN)} {stale_hw:>18d}/{len(HANDWRITTEN)}") print(f"four source versions, {len(set(docs))} distinct generated docs; " f"handwritten doc is the same across all four") print() module = types.ModuleType("metrics") exec(compile(VERSIONS["canonical"], "", "exec"), module.__dict__) tried, failed = 0, [] for t in doctest.DocTestFinder().find(module): runner = doctest.DocTestRunner(verbose=False) runner.run(t, out=lambda s: None) tried += runner.tries if runner.failures: failed.append(t.name.split(".")[-1]) print(f"doc test tried {tried} examples, {len(failed)} failed: " f"{', '.join(failed)}") print("checks the documentation generator does for the claim: 0") ``` ``` source version public names doc digest stale in generated stale in handwritten canonical 3 644f3575 0/3 2/3 order changed 3 644f3575 0/3 2/3 helper added 3 644f3575 0/3 2/3 signature changed 3 ec5c99c8 0/3 3/3 four source versions, 2 distinct generated docs; handwritten doc is the same across all four doc test tried 3 examples, 1 failed: discount checks the documentation generator does for the claim: 0 ``` #### What Unifies: The Signature The table's first three rows give the same document digest: `644f3575`. Three source versions are distinct texts — one writes function definitions in a different order, one carries a private helper function — but the document they produce is single. Because the generator sorts its output by name, definition order drops out; because it skips a name starting with an underscore, the private helper drops out. The fourth row gives a separate digest: `ec5c99c8`. Two signatures changed, the document changed. This is not a flaw, it is generation's definition: **documentation tracks the source's public interface, and only that.** The stale-signature columns give the distinction as a number. In the generated document, stale signatures are **0/3** across all four versions. In the handwritten document, it is **2/3** in the first three versions, **3/3** in the fourth — even though the handwritten document never changed at all. Staleness grows on its own as the source moves forward, and nothing announces it. The difference between them is not a difference of effort, it is a difference of **binding.** In the generated document there is no copy between the signature and the source; the signature is reread on every generation. In the handwritten document there is a copy, and the copy staying equal to the original depends on a person remembering. This is exactly the outcome generation unifies: **a single document from the same source, and zero stale signatures in that document.** #### What Does Not Unify: The Claim The bottom rows give the measurement's second half and show where unification stops. The `discount` function's summary text states that the ceiling is 25 percent; the body applies 20. The generator carries this line into the document **as is.** The number of checks it performs is **0**, because the generator is a copier: whatever the source says, it writes that into the document. If the source itself says something wrong, generation **spreads that wrongness further.** The doc test, by contrast, tries **3** examples and **1** fails: `discount`. It fails not because the example is wrong, but because the example is **runnable.** Had the same claim been written as prose, nothing would have failed. From here comes the lesson's result. Generation unifies **two outcomes**: that the document shows the same interface as the source, and that the same document comes out of the same source. **It does not unify one outcome:** that the claim in the document matches the behavior. That match is tested only when the claim is written in a runnable form, and when it is tested, what checks it is not the documentation generator, it is the test. This is the same shape as the course's coverage measurement. Coverage counts where the code ran, not what the test looked at. Generation determines what the document shows, not whether it is correct. **Every tool looks at its own input; the only thing that has an oracle is the expectation.** #### Summary - A documentation generator derives its output entirely from the source; definition order and private names do not enter the result. - Four source versions give **2 distinct generated documents**; the first three merge into a single digest, the fourth separates because the signature changed. - Stale signatures in the generated document are **0/3** across all four versions; in the handwritten document they are **2/3** and **3/3**, and the document never changed at all. - Generation does not unify the claim: the 25-percent claim in the `discount` function's summary is carried into the document as is, and the checks the generator performs are **0.** - When the same claim is written as a runnable example, **1** of **3** examples fails; what tests the claim is not the generator, it is the test. #### Course Wrap-Up This course asked a single question across fourteen lessons: **how many distinct outcomes did the same input give?** The question never turned into "does it work" in any lesson, because how many separate ways a thing that works actually works is a separate question, and reproducibility is measured with that question. | Lesson | Measured | Distinct outcome count | |---|---|---| | Virtual Environments | six install orders, shared and isolated site | **5** in shared, **1** in isolated | | Dependency Declaration | same declaration, two strategies and two registry views | **2**, **1** with lock; **3** across four installs, **1** with lock | | Package Managers | four resolution strategies, loose and strict constraint | **4** sets (**3** consistent); **2** sets under strict (**0** consistent); **1** with lock | | Version Management | same declaration across three fictional interpreters | **2**; **6** accepted, **3** rejected across nine installs | | Test Frameworks | two forms of writing, three discovery rules | **1** in form; **3** in tests found, **1** in report | | Fixtures and Parametrization | same five-test team across three orders | **2**, **1** with fixture | | Mock Objects | same test in three modes, mock and real source | **3** with real, **1** with mock; **2** with a drifted mock | | Doctests | twelve claims growing stale with a form change | stale turning audible **4** and **6**: **2** | | Coverage Measurement | coverage of a three-test set and the call deviating from the oracle | line and branch **1.0000**, passed **3/3**; deviating call **1**; with a fourth test, coverage **1**, passed ratio **2** | | Formatters and Linters | the same function's six separate forms | text **6** → **1**; finding set **5** → **1** | | Multi-Environment Testing | same team in a three-axis matrix, 18 runs | **4**; **3** with lock, **2** with fixture | | Packaging | nine builds from the same source, three inclusion rules | **5**; **1** with explicit list; **3** and **1** at install | | Publishing | two installs with the same lock file, two registry policies | **2**, **1** in the immutable registry | | Documentation | documentation generated from four source versions | **2**; stale signature **0** against **2** and **3** | The table's reading gathers into one sentence: **in nearly every row, the first number is greater than one, the second is one.** The difference between them is not a tool difference, it is a decision difference. The lock file did not remove resolution; it made the decision once and put it in writing. Isolation did not make install easier; it stopped install order from changing the result. The fixture did not make testing faster; it stopped order from deciding the outcome. The explicit list did not choose files; it stopped the choice from depending on the directory's state. In every case, what unified was not the work itself, it was **the source of the decision.** A second pattern sits in the table too: **a unified number does not give correctness.** Coverage did not see the flawed expectation even where it dropped to one; the formatter dropped six forms to one but none of the remaining six findings reported the flaw; documentation generation unified the document but carried the wrong claim in it. Unification is the condition for comparability, not for correctness. The only thing that gives correctness is the oracle, and the oracle always comes from outside the measurement. | Course | Measurement axis | |---|---| | Python Fundamentals | the protocol syntax calls | | Data Structures and Functional Tools | the object created, held, and shared | | Object-Oriented Python and Types | who answered the call, who checked the claim | | Concurrency and Performance | overlapping step | | Python Projects: Packaging and Testing | how many distinct outcomes did the same input give | Five axes form a sequence. The first course looked beneath syntax and counted which special method each form calls. The second measured how an object is created, where it is held, and who it is shared with. The third separated which class answers a call and when a given claim gets checked. The fourth counted how many of a job's steps overlap and built the case for counting steps instead of time. The fifth put all of these into a project and asked how many distinct outcomes the same input gives. Together the five build **a language's own mechanism**: what the language calls, what it holds, what it checks, how it splits work, and how it becomes repeatable as a project. This mechanism is specific to the language and does not carry over to another language as is. What carries over is **the questions.** The language curricula starting with M08 ask the same five questions of another language, and the answers come out different: an answer given by a special method in one language is given by an interface in another; a claim checked at run time in one is checked at compile time in another; overlap bounded by an interpreter lock in one is met by another mechanism in another. **A language's own mechanism has been built; next is how another language answers the same questions.**