Multi-Environment Testing

The previous lesson ran in a single environment. What it measured was the source itself: six forms, one tree, one behavior. Whichever interpreter reads the source text, the same tree comes out, which is why that measurement was independent of the environment.

The same cannot be said for tests. A test runs not just the code but also the rig the code sits inside: which interpreter, which dependency set, which order. The shared setup’s five-test team could give a separate decision just by order. This lesson’s question widens that observation: how many distinct decisions does the same team give across how many environments, and which decisions bring that number down?

How Many Axes Build an Environment

“Environment” is not a single thing, it is a product of axes. This measurement has three axes.

The interpreter axis. Three fictional interpreters — I1, I2, I3 — are each modeled by a capability set. No real version number is written; the difference between interpreters is reduced to whether a single capability is present or not. I1 does not carry the union capability, the other two do.

The resolution axis. The same dependency declaration is resolved with two strategies: newest and oldest compatible. As the shared setup showed, this produces two separate version sets from the same declaration. A third option is a lock file, and it stops the strategy from changing the result.

The order axis. The team is run in three separate orders. A test that pollutes shared state and a test that assumes a clean environment sit in the same team; which one runs first changes the result.

The team is seven tests: the shared setup’s five tests, plus two tests that read the environment. The capability test looks at the interpreter’s capability set, the version test requires the resolved report version to be at least 1.1. The five tests’ behavior is not touched.

How a pipeline defines this matrix, which job runs on which runner, and where the result gets reported was established on the operations side in the Continuous Integration and Delivery course and is not repeated here. This lesson has no pipeline; the only thing measured is how many distinct decisions come out of how many environments.

The Matrix’s Cost

The axes multiply: three interpreters, two resolutions, three orders — 18 runs. Since the seven-test team runs from scratch on every run, a total of 126 test calls are made. As axis count rises, this number grows multiplicatively; adding one more axis multiplies run count by two or three.

Against this, the matrix’s information yield does not grow at the same rate. A matrix can include axis points that do not produce a separate decision; I2 and I3 carry the same capability set in this measurement and never separate from each other in any run. The measurement shows this directly.

Two conditions are needed for an axis to belong in the matrix. It has to be able to change the decision — if it cannot, it only grows run count without adding information. It has to be non-eliminable — if it can be fixed by a configuration decision, fixing it is preferable to keeping it in the matrix. These two conditions are this lesson’s definition of what it measures: the order and resolution axes satisfy the first condition but fail the second; the interpreter axis satisfies both and stays in the matrix.

Duration is not measured here either. When a matrix’s cost is written as run count and call count, it stays reproducible; the same matrix makes the same 126 calls on another machine. The rule of writing down what a measurement counts was established at the previous course’s close, and it continues in this course too.

The measurement’s assumptions:

• QP8 — Interpreters are fictional and are separated only by capability sets; no real version number is written, no interpreter implementation is named.
• QP9 — The shared setup’s five tests are taken as is; the polluting and dependent tests’ behavior is not changed. The two added tests do not touch shared state, they only read the environment.
• QP10 — The version test redoes resolution on every call; when the lock is chosen, the locked set is read instead of resolution.
• QP11 — When the fixture is on, the shared environment is rebuilt before every test. This effect of the fixture was measured in this course’s fixtures-and-parametrization lesson; here it is used as an elimination tool, not measured again.
• QP12 — A run’s decision is its passed-test count. If two runs give the same number, one distinct decision is counted, even if the names of the failed tests differ.
• QP13 — No duration is measured. The matrix’s cost is written as run count and test calls.
• QP14 — No real install is done, no real package is downloaded; the interpreter and the dependency set are entirely modeled inside the lesson.

Measurement

"""Multi-environment testing: same team, how many environments, how many distinct decisions."""

REGISTRY = {"metrics": [(1, 0), (1, 1), (1, 2), (2, 0)],
            "report": [(0, 9), (1, 0), (1, 1)],
            "common": [(3, 0), (3, 1), (3, 2), (4, 0)]}
DECLARATION = {"metrics": ((1, 0), (2, 0)), "report": ((1, 0), (2, 0)),
               "common": ((3, 0), (4, 0))}


def resolve(strategy):
    chosen = {}
    for package in sorted(DECLARATION):
        lo, hi = DECLARATION[package]
        candidates = [s for s in REGISTRY[package] if lo <= s < hi]
        chosen[package] = candidates[-1] if strategy == "newest" else candidates[0]
    return chosen


LOCK = dict(resolve("newest"))


class Environment:
    def __init__(self):
        self.counter = 0


def t_clean_a(e, i, s):
    return True


def t_polluting(e, i, s):
    e.counter += 1
    return True


def t_dependent(e, i, s):
    return e.counter == 0


def t_clean_b(e, i, s):
    return True


def t_clean_c(e, i, s):
    return True


def t_capability(e, i, s):
    """Requires the interpreter's union capability."""
    return "union" in INTERPRETER[i]


def t_version(e, i, s):
    """Requires the resolved report version to be at least 1.1."""
    return (LOCK if s == "lock" else resolve(s))["report"] >= (1, 1)


TESTS = [("clean_a", t_clean_a), ("polluting", t_polluting),
         ("dependent", t_dependent), ("clean_b", t_clean_b),
         ("clean_c", t_clean_c), ("capability", t_capability),
         ("version", t_version)]
ORDERS = {"source order": [0, 1, 2, 3, 4, 5, 6],
          "dependent first": [0, 2, 1, 3, 4, 5, 6],
          "reverse": [6, 5, 4, 3, 2, 1, 0]}
INTERPRETER = {"I1": {"a"}, "I2": {"a", "union"}, "I3": {"a", "union"}}
CALLS = 0


def run(i, s, order, fixture=False):
    """When the fixture is on, the shared environment is rebuilt before every test."""
    global CALLS
    e, passed, failed = Environment(), 0, []
    for idx in order:
        name, fn = TESTS[idx]
        if fixture:
            e = Environment()
        CALLS += 1
        if fn(e, i, s):
            passed += 1
        else:
            failed.append(name)
    return passed, failed


print(f"{'interpreter':<12s} {'resolution':<10s} {'order':<16s} {'passed':>6s}  failed")
decisions = []
for i in INTERPRETER:
    for s in ("newest", "oldest"):
        for oname, order in ORDERS.items():
            passed, failed = run(i, s, order)
            decisions.append(passed)
            print(f"{i:<12s} {s:<10s} {oname:<16s} {passed:6d}  "
                  f"{','.join(failed) or '-'}")
print(f"\n{len(decisions)} runs, {len(set(decisions))} distinct decisions: "
      f"{sorted(set(decisions))}")
print(f"test calls {CALLS}")

print()
print(f"{'phase':<34s} {'runs':>6s} {'distinct decision':>18s} {'calls':>6s}")
for name, strategies, orders, fixture in (
        ("raw matrix", ("newest", "oldest"), ORDERS, False),
        ("with lock file", ("lock",), ORDERS, False),
        ("lock and fixture", ("lock",), {"one": ORDERS["source order"]}, True)):
    CALLS, result = 0, []
    for i in INTERPRETER:
        for s in strategies:
            for order in orders.values():
                result.append(run(i, s, order, fixture)[0])
    print(f"{name:<34s} {len(result):6d} {len(set(result)):18d} {CALLS:6d}  "
          f"{sorted(set(result))}")

print()
for i in INTERPRETER:
    passed, failed = run(i, "lock", ORDERS["source order"], True)
    print(f"{i}: {passed}/7 passed, failed: {','.join(failed) or '-'}")

interpreter  resolution order            passed  failed
I1           newest     source order          5  dependent,capability
I1           newest     dependent first       6  capability
I1           newest     reverse               6  capability
I1           oldest     source order          4  dependent,capability,version
I1           oldest     dependent first       5  capability,version
I1           oldest     reverse               5  version,capability
I2           newest     source order          6  dependent
I2           newest     dependent first       7  -
I2           newest     reverse               7  -
I2           oldest     source order          5  dependent,version
I2           oldest     dependent first       6  version
I2           oldest     reverse               6  version
I3           newest     source order          6  dependent
I3           newest     dependent first       7  -
I3           newest     reverse               7  -
I3           oldest     source order          5  dependent,version
I3           oldest     dependent first       6  version
I3           oldest     reverse               6  version

18 runs, 4 distinct decisions: [4, 5, 6, 7]
test calls 126

phase                                runs  distinct decision  calls
raw matrix                             18                  4    126  [4, 5, 6, 7]
with lock file                          9                  3     63  [5, 6, 7]
lock and fixture                        3                  2     21  [6, 7]

I1: 6/7 passed, failed: capability
I2: 7/7 passed, failed: -
I3: 7/7 passed, failed: -

Eighteen Runs, Four Decisions

The top table lays the same team’s eighteen runs side by side. Passed-test count ranges between 4 and 7; there are 4 distinct decisions.

Reading the four one by one shows where the number comes from. The worst row is I1 / oldest / source order: 4/7, with the three failing tests dependent, capability, and version. The best rows are I2 and I3’s newest / dependent first and reverse runs: 7/7, nothing failing.

What deserves attention is that the difference comes from three separate sources. The dependent test failing comes from the order axis, the version test failing comes from the resolution axis, the capability test failing comes from the interpreter axis. All three gather into the same column and blend into a single number — passed-test count.

If a report showed only this number, four separate values would remain and none of them would say which axis was responsible. The failed column exists for exactly this reason: decision count gives the difference’s existence, the failing test’s name gives the difference’s source. Growing the matrix by one more axis grows the first number, it does not give the second.

Same Number, Separate Cause

The value 5 appears in five separate rows in the top table, and the failing tests in those five rows form three separate sets: dependent with capability, capability with version, dependent with version. The same number, coming out for three separate reasons.

This is the measurement’s most commonly misread spot. Two runs giving the same number does not show they are in the same state; it only shows their failing-test count is equal. When counting decision count, we count these rows as one — the assumption block states this plainly — because what is measured is distinct-outcome count, not distinct-cause count. But when investigating a failure, the question reverses and the cause has to be separated.

The two rows under reverse order carry a small example of the same distinction: in the I1 / oldest / reverse run, the failing tests are written version,capability in that order; in the dependent first run, capability,version. The set is the same, the order is different — because the list fills in the order the tests run. It has no effect on the number, but a tool comparing two dumps as text would count them as separate.

A practical result follows from this: if a run report carries only the number, separate causes get gathered into the same row. If the failing tests’ names stand in the report, the three causes can be separated; if they do not, the matrix has made eighteen runs and only four numbers are left. The matrix’s cost of widening is multiplicative; the information it yields depends on what the report carries.

Eliminating Axes

The middle table measures three stages of elimination.

Raw matrix: 18 runs, 4 distinct decisions, 126 test calls. This is the state where no axis is fixed.

With lock file: the resolution axis drops, because the lock stops the strategy from changing the result. Runs drop to 9, calls to 63, distinct decisions to 3. The decision lost is 4; the version test no longer fails in any run. The lock does not remove resolution — it makes the decision once and puts it in writing — but it entirely removes one axis from the environment matrix.

With lock and fixture: the order axis drops too. Runs drop to 3, calls to 21, distinct decisions to 2. The dependent test no longer fails in any order, because the shared environment is rebuilt before every test.

The remaining 2 distinct decisions sit here and cannot be brought down further. The bottom rows show this: I1 6/7, I2 and I3 7/7. The only test that fails is capability, and the reason it fails is not a configuration detail, it is that the interpreter genuinely does not carry a capability. This is exactly the difference multi-environment running is looking for.

The order of elimination decides the intermediate numbers, not the final number. Had the fixture been applied before the lock, the middle row would show a different number than 3, because the axis eliminated at that stage would be a different one. Each elimination step drops one axis entirely and takes that axis’s contribution to the decision along with it; once both axes are dropped, the same 2 decisions remain regardless of the order they were dropped in. Intermediate numbers belong to the method, the final number belongs to the environment.

The lesson’s rule follows from this: multi-environment running is for finding environment-sourced difference; if configuration-sourced difference is not eliminated first, which axis is responsible cannot be known. The eighteen-run matrix gave four decisions, and two of them were not even about the environment. Elimination, while dropping run count to a sixth, loses only information that was never this measurement’s subject to begin with.

One last reading concerns the matrix’s width. Three interpreters were run but distinct decisions came out to 2; I2 and I3 never separated in any run, because their capability sets are the same. Adding one more point to the matrix grows run count and call count linearly but may not grow distinct-decision count. The matrix’s width is a cost, distinct-decision count is a gain, and the two do not grow at the same rate.

Summary

• An environment is not a single thing, it is a product of axes; here three axes produce 18 runs and 126 test calls.
• The same seven-test team gives 4 distinct decisions in this matrix (4, 5, 6, 7); the difference comes from three separate axes and blends into a single number.
• The failing test’s name gives the difference’s source: dependent from order, version from resolution, capability from the interpreter.
• The lock file drops the resolution axis — 9 runs, 3 distinct decisions; the fixture drops the order axis — 3 runs, 2 distinct decisions.
• The remaining 2 decisions cannot be brought down and are the real environment difference: I1 6/7, I2 and I3 7/7. Although three interpreters were run, distinct-decision count is two; a matrix widening does not have to grow distinct-outcome count.

Next Step

This lesson moved the team from environment to environment and assumed the same source ran in every environment. That assumption is silent and was not tested: was what ran genuinely the same source? When a project is handed off for someone else’s use, the source does not go directly; it is packaged into a distribution format, and that package is unpacked and installed. The next lesson measures this step: how many distinct contents does a distribution produced from the same source give, which inputs grow that number past one, and what fixing brings it back to one?