--- title: 'Multi-Environment Testing' source: 'https://academia.sh/en/courses/python-projects/multi-environment-testing' course: 'Python Projects: Packaging and Testing' language: en updated: '2026-08-17T18:10:32+00:00' license: 'CC BY-SA 4.0' --- # Multi-Environment Testing The same seven-test team gives 4 distinct decisions across 18 runs on the axes of three interpreters, two resolutions, and three orders; the lock file drops runs to 9 and decisions to 3, the fixture to 3 and 2, and the remaining two decisions are the real environment difference itself. The previous lesson ran in a single environment. What it measured was the source itself: six forms, one tree, one behavior. Whichever interpreter reads the source text, the same tree comes out, which is why that measurement was independent of the environment. The same cannot be said for tests. A test runs not just the code but also **the rig the code sits inside**: which interpreter, which dependency set, which order. The shared setup's five-test team could give a separate decision just by order. This lesson's question widens that observation: **how many distinct decisions does the same team give across how many environments, and which decisions bring that number down?** ## How Many Axes Build an Environment "Environment" is not a single thing, it is a **product of axes.** This measurement has three axes. **The interpreter axis.** Three fictional interpreters — `I1`, `I2`, `I3` — are each modeled by a **capability set.** No real version number is written; the difference between interpreters is reduced to whether a single capability is present or not. `I1` does not carry the `union` capability, the other two do. **The resolution axis.** The same dependency declaration is resolved with two strategies: **newest** and **oldest compatible.** As the shared setup showed, this produces two separate version sets from the same declaration. A third option is a lock file, and it stops the strategy from changing the result. **The order axis.** The team is run in three separate orders. A test that pollutes shared state and a test that assumes a clean environment sit in the same team; which one runs first changes the result. The team is seven tests: the shared setup's five tests, plus two tests that read the environment. The `capability` test looks at the interpreter's capability set, the `version` test requires the resolved `report` version to be at least `1.1`. The five tests' behavior is not touched. How a pipeline defines this matrix, which job runs on which runner, and where the result gets reported was established on the operations side in the Continuous Integration and Delivery course and is not repeated here. This lesson has no pipeline; the only thing measured is **how many distinct decisions come out of how many environments.** ## The Matrix's Cost The axes multiply: three interpreters, two resolutions, three orders — **18** runs. Since the seven-test team runs from scratch on every run, a total of **126** test calls are made. As axis count rises, this number grows multiplicatively; adding one more axis multiplies run count by two or three. Against this, the matrix's **information yield** does not grow at the same rate. A matrix can include axis points that do not produce a separate decision; `I2` and `I3` carry the same capability set in this measurement and never separate from each other in any run. The measurement shows this directly. Two conditions are needed for an axis to belong in the matrix. **It has to be able to change the decision** — if it cannot, it only grows run count without adding information. **It has to be non-eliminable** — if it can be fixed by a configuration decision, fixing it is preferable to keeping it in the matrix. These two conditions are this lesson's definition of what it measures: the order and resolution axes satisfy the first condition but fail the second; the interpreter axis satisfies both and stays in the matrix. Duration is not measured here either. When a matrix's cost is written as run count and call count, it stays reproducible; the same matrix makes the same **126** calls on another machine. The rule of writing down what a measurement counts was established at the previous course's close, and it continues in this course too. The measurement's assumptions: - **QP8** — Interpreters are fictional and are separated only by capability sets; no real version number is written, no interpreter implementation is named. - **QP9** — The shared setup's five tests are taken as is; the `polluting` and `dependent` tests' behavior is not changed. The two added tests do not touch shared state, they only read the environment. - **QP10** — The `version` test redoes resolution on every call; when the lock is chosen, the locked set is read instead of resolution. - **QP11** — When the fixture is on, the shared environment is rebuilt **before every test.** This effect of the fixture was measured in this course's fixtures-and-parametrization lesson; here it is used as an **elimination tool**, not measured again. - **QP12** — A run's decision is its passed-test count. If two runs give the same number, **one** distinct decision is counted, even if the names of the failed tests differ. - **QP13** — No duration is measured. The matrix's cost is written as **run count** and **test calls.** - **QP14** — No real install is done, no real package is downloaded; the interpreter and the dependency set are entirely modeled inside the lesson. ## Measurement ```python """Multi-environment testing: same team, how many environments, how many distinct decisions.""" REGISTRY = {"metrics": [(1, 0), (1, 1), (1, 2), (2, 0)], "report": [(0, 9), (1, 0), (1, 1)], "common": [(3, 0), (3, 1), (3, 2), (4, 0)]} DECLARATION = {"metrics": ((1, 0), (2, 0)), "report": ((1, 0), (2, 0)), "common": ((3, 0), (4, 0))} def resolve(strategy): chosen = {} for package in sorted(DECLARATION): lo, hi = DECLARATION[package] candidates = [s for s in REGISTRY[package] if lo <= s < hi] chosen[package] = candidates[-1] if strategy == "newest" else candidates[0] return chosen LOCK = dict(resolve("newest")) class Environment: def __init__(self): self.counter = 0 def t_clean_a(e, i, s): return True def t_polluting(e, i, s): e.counter += 1 return True def t_dependent(e, i, s): return e.counter == 0 def t_clean_b(e, i, s): return True def t_clean_c(e, i, s): return True def t_capability(e, i, s): """Requires the interpreter's union capability.""" return "union" in INTERPRETER[i] def t_version(e, i, s): """Requires the resolved report version to be at least 1.1.""" return (LOCK if s == "lock" else resolve(s))["report"] >= (1, 1) TESTS = [("clean_a", t_clean_a), ("polluting", t_polluting), ("dependent", t_dependent), ("clean_b", t_clean_b), ("clean_c", t_clean_c), ("capability", t_capability), ("version", t_version)] ORDERS = {"source order": [0, 1, 2, 3, 4, 5, 6], "dependent first": [0, 2, 1, 3, 4, 5, 6], "reverse": [6, 5, 4, 3, 2, 1, 0]} INTERPRETER = {"I1": {"a"}, "I2": {"a", "union"}, "I3": {"a", "union"}} CALLS = 0 def run(i, s, order, fixture=False): """When the fixture is on, the shared environment is rebuilt before every test.""" global CALLS e, passed, failed = Environment(), 0, [] for idx in order: name, fn = TESTS[idx] if fixture: e = Environment() CALLS += 1 if fn(e, i, s): passed += 1 else: failed.append(name) return passed, failed print(f"{'interpreter':<12s} {'resolution':<10s} {'order':<16s} {'passed':>6s} failed") decisions = [] for i in INTERPRETER: for s in ("newest", "oldest"): for oname, order in ORDERS.items(): passed, failed = run(i, s, order) decisions.append(passed) print(f"{i:<12s} {s:<10s} {oname:<16s} {passed:6d} " f"{','.join(failed) or '-'}") print(f"\n{len(decisions)} runs, {len(set(decisions))} distinct decisions: " f"{sorted(set(decisions))}") print(f"test calls {CALLS}") print() print(f"{'phase':<34s} {'runs':>6s} {'distinct decision':>18s} {'calls':>6s}") for name, strategies, orders, fixture in ( ("raw matrix", ("newest", "oldest"), ORDERS, False), ("with lock file", ("lock",), ORDERS, False), ("lock and fixture", ("lock",), {"one": ORDERS["source order"]}, True)): CALLS, result = 0, [] for i in INTERPRETER: for s in strategies: for order in orders.values(): result.append(run(i, s, order, fixture)[0]) print(f"{name:<34s} {len(result):6d} {len(set(result)):18d} {CALLS:6d} " f"{sorted(set(result))}") print() for i in INTERPRETER: passed, failed = run(i, "lock", ORDERS["source order"], True) print(f"{i}: {passed}/7 passed, failed: {','.join(failed) or '-'}") ``` ``` interpreter resolution order passed failed I1 newest source order 5 dependent,capability I1 newest dependent first 6 capability I1 newest reverse 6 capability I1 oldest source order 4 dependent,capability,version I1 oldest dependent first 5 capability,version I1 oldest reverse 5 version,capability I2 newest source order 6 dependent I2 newest dependent first 7 - I2 newest reverse 7 - I2 oldest source order 5 dependent,version I2 oldest dependent first 6 version I2 oldest reverse 6 version I3 newest source order 6 dependent I3 newest dependent first 7 - I3 newest reverse 7 - I3 oldest source order 5 dependent,version I3 oldest dependent first 6 version I3 oldest reverse 6 version 18 runs, 4 distinct decisions: [4, 5, 6, 7] test calls 126 phase runs distinct decision calls raw matrix 18 4 126 [4, 5, 6, 7] with lock file 9 3 63 [5, 6, 7] lock and fixture 3 2 21 [6, 7] I1: 6/7 passed, failed: capability I2: 7/7 passed, failed: - I3: 7/7 passed, failed: - ``` ## Eighteen Runs, Four Decisions The top table lays the same team's eighteen runs side by side. Passed-test count ranges between **4** and **7**; there are **4 distinct decisions.** Reading the four one by one shows where the number comes from. The worst row is `I1` / `oldest` / `source order`: **4/7**, with the three failing tests `dependent`, `capability`, and `version`. The best rows are `I2` and `I3`'s `newest` / `dependent first` and `reverse` runs: **7/7**, nothing failing. What deserves attention is that the difference comes from **three separate sources.** The `dependent` test failing comes from the order axis, the `version` test failing comes from the resolution axis, the `capability` test failing comes from the interpreter axis. All three gather into the same column and blend into a single number — passed-test count. If a report showed only this number, four separate values would remain and none of them would say which axis was responsible. The `failed` column exists for exactly this reason: **decision count gives the difference's existence, the failing test's name gives the difference's source.** Growing the matrix by one more axis grows the first number, it does not give the second. ## Same Number, Separate Cause The value **5** appears in five separate rows in the top table, and the failing tests in those five rows form three separate sets: `dependent` with `capability`, `capability` with `version`, `dependent` with `version`. The same number, coming out for three separate reasons. This is the measurement's most commonly misread spot. Two runs giving the same number does not show they are in the same state; it only shows their **failing-test count** is equal. When counting decision count, we count these rows as one — the assumption block states this plainly — because what is measured is distinct-outcome count, not distinct-cause count. But when investigating a failure, the question reverses and the cause has to be separated. The two rows under `reverse` order carry a small example of the same distinction: in the `I1` / `oldest` / `reverse` run, the failing tests are written `version,capability` in that order; in the `dependent first` run, `capability,version`. The set is the same, the order is different — because the list fills in the order the tests run. It has no effect on the number, but a tool comparing two dumps as text would count them as separate. A practical result follows from this: **if a run report carries only the number, separate causes get gathered into the same row.** If the failing tests' names stand in the report, the three causes can be separated; if they do not, the matrix has made eighteen runs and only four numbers are left. The matrix's cost of widening is multiplicative; the information it yields depends on what the report carries. ## Eliminating Axes The middle table measures three stages of elimination. **Raw matrix**: **18** runs, **4** distinct decisions, **126** test calls. This is the state where no axis is fixed. **With lock file**: the resolution axis drops, because the lock stops the strategy from changing the result. Runs drop to **9**, calls to **63**, distinct decisions to **3**. The decision lost is `4`; the `version` test no longer fails in any run. The lock does not remove resolution — it makes the decision once and puts it in writing — but it **entirely** removes one axis from the environment matrix. **With lock and fixture**: the order axis drops too. Runs drop to **3**, calls to **21**, distinct decisions to **2**. The `dependent` test no longer fails in any order, because the shared environment is rebuilt before every test. The remaining **2** distinct decisions sit here and cannot be brought down further. The bottom rows show this: `I1` **6/7**, `I2` and `I3` **7/7**. The only test that fails is `capability`, and the reason it fails is not a configuration detail, it is that the interpreter genuinely does not carry a capability. This is exactly the difference multi-environment running is looking for. The order of elimination decides the intermediate numbers, not the final number. Had the fixture been applied before the lock, the middle row would show a different number than **3**, because the axis eliminated at that stage would be a different one. Each elimination step drops one axis entirely and takes that axis's contribution to the decision along with it; once both axes are dropped, the same **2** decisions remain regardless of the order they were dropped in. Intermediate numbers belong to the method, the final number belongs to the environment. The lesson's rule follows from this: **multi-environment running is for finding environment-sourced difference; if configuration-sourced difference is not eliminated first, which axis is responsible cannot be known.** The eighteen-run matrix gave four decisions, and two of them were not even about the environment. Elimination, while dropping run count to a sixth, loses only information that was never this measurement's subject to begin with. One last reading concerns the matrix's width. Three interpreters were run but distinct decisions came out to **2**; `I2` and `I3` never separated in any run, because their capability sets are the same. Adding one more point to the matrix grows run count and call count linearly but **may not** grow distinct-decision count. The matrix's width is a cost, distinct-decision count is a gain, and the two do not grow at the same rate. ## Summary - An environment is not a single thing, it is a product of axes; here three axes produce **18** runs and **126** test calls. - The same seven-test team gives **4 distinct decisions** in this matrix (**4**, **5**, **6**, **7**); the difference comes from three separate axes and blends into a single number. - The failing test's name gives the difference's source: `dependent` from order, `version` from resolution, `capability` from the interpreter. - The lock file drops the resolution axis — **9** runs, **3** distinct decisions; the fixture drops the order axis — **3** runs, **2** distinct decisions. - The remaining **2** decisions cannot be brought down and are the real environment difference: `I1` **6/7**, `I2` and `I3` **7/7**. Although three interpreters were run, distinct-decision count is two; a matrix widening does not have to grow distinct-outcome count. ## Next Step This lesson moved the team from environment to environment and assumed the same source ran in every environment. That assumption is silent and was not tested: was what ran genuinely the same source? When a project is handed off for someone else's use, the source does not go directly; it is packaged into a **distribution format**, and that package is unpacked and installed. The next lesson measures this step: how many distinct contents does a distribution produced from the same source give, which inputs grow that number past one, and what fixing brings it back to one?