Environment Pinning

The feature store brought one item down to a single place, and how many points of accuracy that item carried was measured. But the code that computes the feature is not the only thing that decides a run’s result. When the same job is run a second time, the data slice read, the periods set aside for testing, the weight grid searched, and the number of digits the intermediate numbers are rounded to may also have changed. None of them is on record, none of them counts as an error, and all of them together move the result.

Environment pinning is keeping each of these items unchanged from one run to the next. This lesson treats pinning not as a habit but as a measure, and asks three questions: pinning which item makes three runs exactly the same, what is the cost of pinning itself, and what does a pinning whose contribution cannot be shown actually say.

• RP49. The source, the first three stages of the pipeline, and the threshold-based model are fictional. The seed is 20260218, and the output does not depend on the run.
• RP50. Pinning is guaranteeing that an item stays the same from run to run. Five items are counted here: data version, feature code, test split, grid, and rounding. No hardware, runtime, or dependency name is written; the items are referred to by their kinds.
• RP51. A trial is a number that moves the unpinned items in a set direction, and three trials represent three separate runs of the same job. A trial is not randomness but a reproducible shift.
• RP52. The baseline is the unpinned setup: none of the five items is pinned, and there is no record saying what the run read.
• RP53. The label is imbalanced: {0: 1752, 1: 448}, the base class ratio is 0.7964, and in the test set it is 0.8067. Accuracy read alone misleads.
• RP54. Run variability and the tuning budget were measured in the Supervised Learning course and are not repeated; the only thing counted here is pinning which item makes the run the same.
• RP55. The effect of the feature window was measured in the previous lesson and becomes an input here: at window 1 the accuracy is 0.7800, at window 2 and above it is 0.8067.
• RP56. Resolution: the full test set is 600 rows, the step is 1/600 = 0.0017. The widest gap between runs in this lesson is 0.0123, that is, larger than seven times the step, and it has been measured.

The Three Numbers of the Unpinned Setup

A run’s result comes from three layers: which data it read, with which code it processed the data, and on which set it measured the result. When each of them is not recorded, it can drift silently, and the symptom of the drift is not an error message but another number. The block below builds the source, the first three stages of the pipeline that face the model, the fictional model, and the five pinning items; the run function moves every unpinned item according to the trial number.

# FICTIONAL. Meter readings, the first three stages of the pipeline, and the threshold-based
# model are defined within the lesson; no real learning library, runtime,
# or dependency manager is called.
SEED, M32 = 20260218, 0xFFFFFFFF


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


REGION = ["east", "west", "north", "south", "central"]
TYPE = ["residential", "commercial"]
PERIODS = [f"2026-{a:02d}" for a in range(1, 13)]
SUBSCRIBERS = {}
for i in range(200):
    r = rng(SEED + 37 * i)
    SUBSCRIBERS[f"A{i + 1:03d}"] = {"region": REGION[int(r() * 5)], "type": TYPE[int(r() * 2)],
                                    "start": 10000 + int(r() * 60000)}


def generate_source():
    records = []
    for i, (sub, a) in enumerate(SUBSCRIBERS.items()):
        e = a["start"]
        for p in PERIODS:
            r = rng(SEED + 101 * i + 7 * PERIODS.index(p))
            e += 4 + int(r() * 46)
            records.append({"subscriber": sub, "period": p, "region": a["region"], "type": a["type"],
                            "index": e, "unit": "m3"})
    return records


def s1_screen(records):
    REQUIRED = ("subscriber", "period", "region", "index", "unit")
    keep = [k for k in records if all(a in k for a in REQUIRED)]
    return keep, len(records) - len(keep)


def s2_dedupe(records):
    seen, keep = set(), []
    for k in records:
        key = (k["subscriber"], k["period"])
        if key in seen:
            continue
        seen.add(key)
        keep.append(k)
    return keep, len(records) - len(keep)


def s3_consumption(records):
    """Consumption from the period difference. Without a previous period the row cannot be computed."""
    table = {(k["subscriber"], k["period"]): k for k in records}
    out, dropped = [], 0
    for k in records:
        i = PERIODS.index(k["period"])
        if i == 0:
            continue
        prev = table.get((k["subscriber"], PERIODS[i - 1]))
        if prev is None:
            dropped += 1
            continue
        try:
            f = int(k["index"]) - int(prev["index"])
        except (TypeError, ValueError):
            dropped += 1
            continue
        out.append(dict(k, m3=f))
    return out, dropped


TRAIN = PERIODS[1:9]        # 2026-02 .. 2026-09
TEST = PERIODS[9:]          # 2026-10 .. 2026-12
THRESHOLD = 40              # consumption above this many m3 counts as "high"


def rows():
    v, _ = s1_screen(generate_source())
    v, _ = s2_dedupe(v)
    v, _ = s3_consumption(v)
    return v


def label(k):
    return 1 if k["m3"] > THRESHOLD else 0


def features(records, window=None, digits=4):
    """window=None -> full history; digits -> rounding of the intermediate means."""
    ordered = sorted(records, key=lambda k: PERIODS.index(k["period"]))
    sub, reg = {}, {}
    for k in ordered:
        sub.setdefault(k["subscriber"], []).append((PERIODS.index(k["period"]), k["m3"]))
        reg.setdefault(k["region"], []).append((PERIODS.index(k["period"]), k["m3"]))

    def mean(series, i):
        chosen = [v for d, v in series if (d < i and (window is None or d >= i - window))]
        return round(sum(chosen) / len(chosen), digits) if chosen else 0.0

    feats = {}
    for k in records:
        i = PERIODS.index(k["period"])
        feats[(k["subscriber"], k["period"])] = {
            "subscriber_mean": mean(sub[k["subscriber"]], i),
            "region_mean": mean(reg[k["region"]], i),
            "commercial": 1 if k["type"] == "commercial" else 0}
    return feats


GRID = [(a, b, c) for a in (0.6, 0.8, 1.0) for b in (0.0, 0.2, 0.4) for c in (0, 4, 8)]


def score(o, w):
    return w[0] * o["subscriber_mean"] + w[1] * o["region_mean"] + w[2] * o["commercial"]


def train(records, feats, grid=None, threshold=THRESHOLD, periods=None):
    grid = GRID if grid is None else grid
    periods = TRAIN if periods is None else periods
    subset = [k for k in records if k["period"] in periods]
    best, best_w = -1.0, None
    for w in grid:
        d = sum(1 for k in subset
                if (1 if score(feats[(k["subscriber"], k["period"])], w) > threshold else 0) == label(k))
        o = d / len(subset)
        if o > best:
            best, best_w = o, w
    return {"weights": best_w, "train_accuracy": round(best, 4),
            "candidates": len(grid), "train_rows": len(subset)}


def evaluate(records, model, feats, threshold=THRESHOLD, periods=None):
    periods = TEST if periods is None else periods
    subset = [k for k in records if k["period"] in periods]
    d = sum(1 for k in subset
            if (1 if score(feats[(k["subscriber"], k["period"])], model["weights"]) > threshold else 0)
            == label(k))
    return round(d / len(subset), 4)


# ---- Reproducibility: the five pinning items.
# Every item that is not pinned moves with the trial; a pinned item stays the same in every trial.
ITEMS = ("data_version", "feature_code", "split", "grid", "rounding")


def settings(pinned, trial):
    return {"row_trim": 0 if "data_version" in pinned else 20 * trial,
            "window": None if "feature_code" in pinned else (None if trial == 0 else 12 - trial),
            "digits": 4 if "rounding" in pinned else max(4 - trial, 0),
            "candidates": len(GRID) if "grid" in pinned else len(GRID) - 6 * trial,
            "periods": TEST if "split" in pinned else PERIODS[9 + (trial % 2):]}


def run(records, pinned, trial=0):
    rec = records if "data_version" in pinned else records[: len(records) - 20 * trial]
    window = None if "feature_code" in pinned else (None if trial == 0 else 12 - trial)
    digits = 4 if "rounding" in pinned else max(4 - trial, 0)
    grid = GRID if "grid" in pinned else GRID[: len(GRID) - 6 * trial]
    periods = TEST if "split" in pinned else PERIODS[9 + (trial % 2):]
    feats = features(rec, window, digits)
    m = train(rec, feats, grid)
    return {"weights": m["weights"], "train": m["train_accuracy"],
            "test": evaluate(rec, m, feats, periods=periods)}

The second block runs the baseline: none of the items is pinned, three trials, three results.

import collections

S = rows()
dist = collections.Counter(label(k) for k in S)
print("rows:", len(S), " label distribution:", dict(sorted(dist.items())),
      " base class ratio:", round(dist[0] / len(S), 4),
      " in test:", round(sum(1 for k in S if k["period"] in TEST and label(k) == 0)
                         / sum(1 for k in S if k["period"] in TEST), 4))
print("pinning items:", len(ITEMS), "->", ", ".join(ITEMS))

print(f"\n{'trial':<5}{'rows':>7}{'window':>9}{'digits':>9}{'candidates':>12}"
      f"{'test rows':>11}{'weights':>18}{'test':>9}")
BASELINE = []
for t in range(3):
    cfg, res = settings(set(), t), run(S, set(), t)
    remaining = len(S) - cfg["row_trim"]
    BASELINE.append(res["test"])
    print(f"{t:<5}{remaining:>7}{str(cfg['window'] or 'full'):>9}{cfg['digits']:>9}{cfg['candidates']:>12}"
          f"{sum(1 for k in S[:remaining] if k['period'] in cfg['periods']):>11}"
          f"{str(res['weights']):>18}{res['test']:>9.4f}")
print("three runs identical:", len(set(BASELINE)) == 1, " different results:", len(set(BASELINE)),
      " widest gap:", round(max(BASELINE) - min(BASELINE), 4))

rows: 2200  label distribution: {0: 1752, 1: 448}  base class ratio: 0.7964  in test: 0.8067
pinning items: 5 -> data_version, feature_code, split, grid, rounding

trial   rows   window   digits  candidates  test rows           weights     test
0       2200     full        4          27        600     (0.6, 0.4, 0)   0.8067
1       2180       11        3          21        396     (0.6, 0.4, 0)   0.7955
2       2160       10        2          15        588     (0.6, 0.4, 0)   0.8078
three runs identical: False  different results: 3  widest gap: 0.0123

The three runs give three different numbers, and the widest gap of 0.0123 is larger than seven times the resolution. What really has to be said is the weights column: all three runs find the triple (0.6, 0.4, 0). The model that is learned is the same; the only thing that changes is which number that model is reported with. The first run’s 0.8067 and the third run’s 0.8078 are two separate memories of the same model, and both look defensible, because the slice they were measured on is not written down.

The test rows column completes this. Because the split is not pinned, the second run is measured on only 396 rows; its step is 1/396 = 0.0025. The three numbers are not merely different, they are numbers produced with different denominators. Label imbalance enlarges the misreading: since the base class ratio in the test is 0.8067, all three numbers sit within the band of a setup that says the majority class.

How Many Items Are Enough

The third block pins the items one at a time in list order and restarts the three runs at every step. The question is a single one: does the added item make the three results exactly the same.

def three_runs(pinned):
    return [run(S, set(pinned), t)["test"] for t in range(3)]


print(f"{'pinned':<8}{'added item':<17}{'three runs':>28}{'same':>7}")
for n in range(len(ITEMS) + 1):
    res = three_runs(ITEMS[:n])
    print(f"{n:<8}{('-' if n == 0 else ITEMS[n - 1]):<17}"
          f"{'[' + ', '.join(f'{x:.4f}' for x in res) + ']':>28}"
          f"{('yes' if len(set(res)) == 1 else 'no'):>7}")

pinned  added item                         three runs   same
0       -                    [0.8067, 0.7955, 0.8078]     no
1       data_version         [0.8067, 0.7950, 0.8067]     no
2       feature_code         [0.8067, 0.7950, 0.8067]     no
3       split                [0.8067, 0.8067, 0.8067]    yes
4       grid                 [0.8067, 0.8067, 0.8067]    yes
5       rounding             [0.8067, 0.8067, 0.8067]    yes

The sameness arrives with the third item: when the data version, the feature code, and the split are pinned, the three runs give [0.8067, 0.8067, 0.8067]. When the fourth and fifth items are added the result does not change, because nothing is left that could change. Two readings follow: pinning the grid and the rounding adds nothing to this measurement, and what decides is the content of the list, not its length; a setup with two items pinned still gives three different numbers, and one with three items pinned does not.

Pinning Whose Contribution Cannot Be Measured

Adding in order does not show each item’s share separately, because later items are stacked on a result that has already become the same. The fourth block does the reverse: it pins all five items, then releases a single item. An item’s contribution counts as measured only if the result changes when it is released. The same block also writes the cost of pinning itself.

print(f"{'single unpinned item':<24}{'three runs':>28}{'contribution measured':>23}")
for item in ITEMS:
    res = three_runs([k for k in ITEMS if k != item])
    print(f"{item:<24}{'[' + ', '.join(f'{x:.4f}' for x in res) + ']':>28}"
          f"{('yes' if len(set(res)) > 1 else 'no'):>23}")

# The cost of pinning itself: item records written in each run , candidates evaluated , rows read.
none_pinned, all_pinned = [settings(set(), t) for t in range(3)], [settings(set(ITEMS), t) for t in range(3)]
print(f"\n{'cost':<26}{'unpinned':>10}{'pinned':>8}")
print(f"{'item records written':<26}{0:>10}{len(ITEMS) * 3:>8}")
print(f"{'candidates evaluated':<26}{sum(a['candidates'] for a in none_pinned):>10}"
      f"{sum(a['candidates'] for a in all_pinned):>8}")
print(f"{'rows read':<26}{sum(len(S) - a['row_trim'] for a in none_pinned):>10}"
      f"{sum(len(S) - a['row_trim'] for a in all_pinned):>8}")
print(f"{'different results':<26}{len(set(three_runs([]))):>10}"
      f"{len(set(three_runs(ITEMS))):>8}")

single unpinned item                      three runs  contribution measured
data_version                [0.8067, 0.8064, 0.8078]                    yes
feature_code                [0.8067, 0.8067, 0.8067]                     no
split                       [0.8067, 0.7950, 0.8067]                    yes
grid                        [0.8067, 0.8067, 0.8067]                     no
rounding                    [0.8067, 0.8067, 0.8067]                     no

cost                        unpinned  pinned
item records written               0      15
candidates evaluated              63      81
rows read                       6540    6600
different results                  3       1

Tested one at a time, only two items remain whose contribution can be measured: the data version and the split. When the other three are released the three runs are still exactly the same; this measurement cannot show anything gained from pinning them. That does not mean those three pinnings are unnecessary; it means exactly this: the assurance they give is an unmeasured assurance. A setup saying “we pin all five items” does not show that all five of those items touch the result; what has been shown is only two.

The feature code row explains this best. When this item is released, the window drops to 11 and 10, and the sweep known from the previous lesson says that from window 2 upward the result does not change. So the item’s contribution is not visible here because it was not perturbed enough; the same item had been measured to carry the accuracy from 0.8067 to 0.7800 when it dropped to window 1. “Contribution could not be measured” is not a property of an item but a property of that test.

The cost table is the other side of the trade. A pinned setup writes five item records per run, 15 records over three runs; in the unpinned setup this number is zero, and that is where the problem comes from. Pinning does not save work either: because the grid stays full-sized in every run the candidates evaluated rise from 63 to 81, and because the data version is not trimmed the rows read rise from 6540 to 6600. The only thing received in return is the last row: the three runs give a single result instead of three different results. The course’s rule defends the cost of that row: a number that cannot be reproduced is not a measurement but a memory.

Summary

• In the unpinned setup three runs find the same weight triple yet give 0.8067, 0.7955, and 0.8078; the model that is learned is the same, and what changes is only the slice on which the number was measured.
• When the split is not pinned the second run is measured on 396 rows; since the three numbers were produced with different denominators, even placing them side by side is wrong. Label imbalance (base ratio 0.7964, 0.8067 in the test) enlarges this misreading.
• When three of the five items are pinned the three runs come out exactly the same; the result does not change even if the grid and the rounding are not pinned, that is, what decides is the content of the pinning list, not its length.
• When the items are released one at a time, only the data version and the split remain with a measurable contribution; the other three give an unmeasured assurance, and that has to do with the perturbing power of the test.
• The cost of pinning is 15 item records over three runs, 81 candidate evaluations instead of 63, and 6600 row reads instead of 6540; in return three different results fall to a single result.

Next Step

Five items were pinned and the three runs now give the same number. But which of the runs produced this number is still written nowhere. What is in hand is a model that carries nothing but a weight triple, and since all three runs found the same triple, that model does not point to the run that produced it. Whether a model is being tried, being tested, or in service, and when it was moved to that stage, sit in the same gap. The next lesson counts this gap: it measures how many records are looked at to find which run a model at a given stage came from, how far that number falls with a registry, and how runs that give the same result are told apart from one another.