--- title: 'Environment Pinning' source: 'https://academia.sh/en/courses/mlops-fundamentals/environment-pinning' course: 'MLOps Fundamentals' language: en updated: '2026-10-21T05:03:01+00:00' license: 'CC BY-SA 4.0' objectives: - 'Counts the items that make a run reproducible and shows that in an unpinned setup three runs give three different numbers.' - 'Determines the smallest set that makes the three runs exactly the same by pinning the items one at a time and names the items whose contribution cannot be measured.' - 'Counts the cost of pinning itself as records written, candidates evaluated, and rows read.' --- # Environment Pinning Counting, one by one, the items that bring a run to the same result a second time: in an unpinned setup three runs find the same weights yet give three different test accuracies such as 0.8067, 0.7955, and 0.8078, because the rows read, the feature window, the rounding digits, the grid searched, and the test split all move in every run. When three of the five items are pinned the three runs come out exactly the same, the result does not change even if the grid and the rounding are not pinned, and when tested one at a time only two items remain whose contribution can be measured. The cost of pinning is fifteen item records, eighteen extra candidate evaluations, and sixty extra row reads. The feature store brought one item down to a single place, and how many points of accuracy that item carried was measured. But the code that computes the feature is not the only thing that decides a run's result. When the same job is run a second time, the data slice read, the periods set aside for testing, the weight grid searched, and the number of digits the intermediate numbers are rounded to may also have changed. None of them is on record, none of them counts as an error, and all of them together move the result. **Environment pinning** is keeping each of these items unchanged from one run to the next. This lesson treats **pinning** not as a habit but as a measure, and asks three questions: pinning which item makes three runs **exactly** the same, what is the cost of pinning itself, and what does a pinning whose contribution cannot be shown actually say. - **RP49.** The source, the first three stages of the pipeline, and the threshold-based model are **fictional**. The seed is **20260218**, and the output does not depend on the run. - **RP50.** **Pinning** is guaranteeing that an item stays the same from run to run. Five items are counted here: data version, feature code, test split, grid, and rounding. **No hardware, runtime, or dependency name is written**; the items are referred to by their kinds. - **RP51.** A **trial** is a number that moves the unpinned items in a set direction, and three trials represent three separate runs of the same job. A trial is not randomness but a reproducible shift. - **RP52.** The baseline is the **unpinned setup**: none of the five items is pinned, and there is no record saying what the run read. - **RP53.** The label is imbalanced: `{0: 1752, 1: 448}`, the **base class ratio is 0.7964**, and in the test set it is **0.8067**. Accuracy read alone misleads. - **RP54.** Run variability and the tuning budget were measured in the Supervised Learning course and are **not repeated**; the only thing counted here is **pinning which item makes the run the same**. - **RP55.** The effect of the feature window was measured in the previous lesson and becomes an **input** here: at window 1 the accuracy is 0.7800, at window 2 and above it is 0.8067. - **RP56.** Resolution: the full test set is **600 rows**, the step is **1/600 = 0.0017**. The widest gap between runs in this lesson is **0.0123**, that is, larger than seven times the step, and it has been measured. ## The Three Numbers of the Unpinned Setup A run's result comes from three layers: which data it read, with which code it processed the data, and on which set it measured the result. When each of them is not recorded, it can drift silently, and the symptom of the drift is not an error message but **another number**. The block below builds the source, the first three stages of the pipeline that face the model, the fictional model, and the five pinning items; the `run` function moves every unpinned item according to the trial number. ```python # FICTIONAL. Meter readings, the first three stages of the pipeline, and the threshold-based # model are defined within the lesson; no real learning library, runtime, # or dependency manager is called. SEED, M32 = 20260218, 0xFFFFFFFF def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value REGION = ["east", "west", "north", "south", "central"] TYPE = ["residential", "commercial"] PERIODS = [f"2026-{a:02d}" for a in range(1, 13)] SUBSCRIBERS = {} for i in range(200): r = rng(SEED + 37 * i) SUBSCRIBERS[f"A{i + 1:03d}"] = {"region": REGION[int(r() * 5)], "type": TYPE[int(r() * 2)], "start": 10000 + int(r() * 60000)} def generate_source(): records = [] for i, (sub, a) in enumerate(SUBSCRIBERS.items()): e = a["start"] for p in PERIODS: r = rng(SEED + 101 * i + 7 * PERIODS.index(p)) e += 4 + int(r() * 46) records.append({"subscriber": sub, "period": p, "region": a["region"], "type": a["type"], "index": e, "unit": "m3"}) return records def s1_screen(records): REQUIRED = ("subscriber", "period", "region", "index", "unit") keep = [k for k in records if all(a in k for a in REQUIRED)] return keep, len(records) - len(keep) def s2_dedupe(records): seen, keep = set(), [] for k in records: key = (k["subscriber"], k["period"]) if key in seen: continue seen.add(key) keep.append(k) return keep, len(records) - len(keep) def s3_consumption(records): """Consumption from the period difference. Without a previous period the row cannot be computed.""" table = {(k["subscriber"], k["period"]): k for k in records} out, dropped = [], 0 for k in records: i = PERIODS.index(k["period"]) if i == 0: continue prev = table.get((k["subscriber"], PERIODS[i - 1])) if prev is None: dropped += 1 continue try: f = int(k["index"]) - int(prev["index"]) except (TypeError, ValueError): dropped += 1 continue out.append(dict(k, m3=f)) return out, dropped TRAIN = PERIODS[1:9] # 2026-02 .. 2026-09 TEST = PERIODS[9:] # 2026-10 .. 2026-12 THRESHOLD = 40 # consumption above this many m3 counts as "high" def rows(): v, _ = s1_screen(generate_source()) v, _ = s2_dedupe(v) v, _ = s3_consumption(v) return v def label(k): return 1 if k["m3"] > THRESHOLD else 0 def features(records, window=None, digits=4): """window=None -> full history; digits -> rounding of the intermediate means.""" ordered = sorted(records, key=lambda k: PERIODS.index(k["period"])) sub, reg = {}, {} for k in ordered: sub.setdefault(k["subscriber"], []).append((PERIODS.index(k["period"]), k["m3"])) reg.setdefault(k["region"], []).append((PERIODS.index(k["period"]), k["m3"])) def mean(series, i): chosen = [v for d, v in series if (d < i and (window is None or d >= i - window))] return round(sum(chosen) / len(chosen), digits) if chosen else 0.0 feats = {} for k in records: i = PERIODS.index(k["period"]) feats[(k["subscriber"], k["period"])] = { "subscriber_mean": mean(sub[k["subscriber"]], i), "region_mean": mean(reg[k["region"]], i), "commercial": 1 if k["type"] == "commercial" else 0} return feats GRID = [(a, b, c) for a in (0.6, 0.8, 1.0) for b in (0.0, 0.2, 0.4) for c in (0, 4, 8)] def score(o, w): return w[0] * o["subscriber_mean"] + w[1] * o["region_mean"] + w[2] * o["commercial"] def train(records, feats, grid=None, threshold=THRESHOLD, periods=None): grid = GRID if grid is None else grid periods = TRAIN if periods is None else periods subset = [k for k in records if k["period"] in periods] best, best_w = -1.0, None for w in grid: d = sum(1 for k in subset if (1 if score(feats[(k["subscriber"], k["period"])], w) > threshold else 0) == label(k)) o = d / len(subset) if o > best: best, best_w = o, w return {"weights": best_w, "train_accuracy": round(best, 4), "candidates": len(grid), "train_rows": len(subset)} def evaluate(records, model, feats, threshold=THRESHOLD, periods=None): periods = TEST if periods is None else periods subset = [k for k in records if k["period"] in periods] d = sum(1 for k in subset if (1 if score(feats[(k["subscriber"], k["period"])], model["weights"]) > threshold else 0) == label(k)) return round(d / len(subset), 4) # ---- Reproducibility: the five pinning items. # Every item that is not pinned moves with the trial; a pinned item stays the same in every trial. ITEMS = ("data_version", "feature_code", "split", "grid", "rounding") def settings(pinned, trial): return {"row_trim": 0 if "data_version" in pinned else 20 * trial, "window": None if "feature_code" in pinned else (None if trial == 0 else 12 - trial), "digits": 4 if "rounding" in pinned else max(4 - trial, 0), "candidates": len(GRID) if "grid" in pinned else len(GRID) - 6 * trial, "periods": TEST if "split" in pinned else PERIODS[9 + (trial % 2):]} def run(records, pinned, trial=0): rec = records if "data_version" in pinned else records[: len(records) - 20 * trial] window = None if "feature_code" in pinned else (None if trial == 0 else 12 - trial) digits = 4 if "rounding" in pinned else max(4 - trial, 0) grid = GRID if "grid" in pinned else GRID[: len(GRID) - 6 * trial] periods = TEST if "split" in pinned else PERIODS[9 + (trial % 2):] feats = features(rec, window, digits) m = train(rec, feats, grid) return {"weights": m["weights"], "train": m["train_accuracy"], "test": evaluate(rec, m, feats, periods=periods)} ``` The second block runs the baseline: none of the items is pinned, three trials, three results. ```python import collections S = rows() dist = collections.Counter(label(k) for k in S) print("rows:", len(S), " label distribution:", dict(sorted(dist.items())), " base class ratio:", round(dist[0] / len(S), 4), " in test:", round(sum(1 for k in S if k["period"] in TEST and label(k) == 0) / sum(1 for k in S if k["period"] in TEST), 4)) print("pinning items:", len(ITEMS), "->", ", ".join(ITEMS)) print(f"\n{'trial':<5}{'rows':>7}{'window':>9}{'digits':>9}{'candidates':>12}" f"{'test rows':>11}{'weights':>18}{'test':>9}") BASELINE = [] for t in range(3): cfg, res = settings(set(), t), run(S, set(), t) remaining = len(S) - cfg["row_trim"] BASELINE.append(res["test"]) print(f"{t:<5}{remaining:>7}{str(cfg['window'] or 'full'):>9}{cfg['digits']:>9}{cfg['candidates']:>12}" f"{sum(1 for k in S[:remaining] if k['period'] in cfg['periods']):>11}" f"{str(res['weights']):>18}{res['test']:>9.4f}") print("three runs identical:", len(set(BASELINE)) == 1, " different results:", len(set(BASELINE)), " widest gap:", round(max(BASELINE) - min(BASELINE), 4)) ``` ``` rows: 2200 label distribution: {0: 1752, 1: 448} base class ratio: 0.7964 in test: 0.8067 pinning items: 5 -> data_version, feature_code, split, grid, rounding trial rows window digits candidates test rows weights test 0 2200 full 4 27 600 (0.6, 0.4, 0) 0.8067 1 2180 11 3 21 396 (0.6, 0.4, 0) 0.7955 2 2160 10 2 15 588 (0.6, 0.4, 0) 0.8078 three runs identical: False different results: 3 widest gap: 0.0123 ``` The three runs give **three different numbers**, and the widest gap of 0.0123 is larger than seven times the resolution. What really has to be said is **the weights column**: all three runs find the triple `(0.6, 0.4, 0)`. The model that is learned is the same; the only thing that changes is **which number that model is reported with**. The first run's 0.8067 and the third run's 0.8078 are two separate memories of the same model, and both look defensible, because the slice they were measured on is not written down. The `test rows` column completes this. Because the split is not pinned, the second run is measured on only **396** rows; its step is 1/396 = 0.0025. The three numbers are not merely different, they are numbers **produced with different denominators**. Label imbalance enlarges the misreading: since the base class ratio in the test is 0.8067, all three numbers sit within the band of a setup that says the majority class. ## How Many Items Are Enough The third block pins the items one at a time in list order and restarts the three runs at every step. The question is a single one: does the added item make the three results **exactly** the same. ```python def three_runs(pinned): return [run(S, set(pinned), t)["test"] for t in range(3)] print(f"{'pinned':<8}{'added item':<17}{'three runs':>28}{'same':>7}") for n in range(len(ITEMS) + 1): res = three_runs(ITEMS[:n]) print(f"{n:<8}{('-' if n == 0 else ITEMS[n - 1]):<17}" f"{'[' + ', '.join(f'{x:.4f}' for x in res) + ']':>28}" f"{('yes' if len(set(res)) == 1 else 'no'):>7}") ``` ``` pinned added item three runs same 0 - [0.8067, 0.7955, 0.8078] no 1 data_version [0.8067, 0.7950, 0.8067] no 2 feature_code [0.8067, 0.7950, 0.8067] no 3 split [0.8067, 0.8067, 0.8067] yes 4 grid [0.8067, 0.8067, 0.8067] yes 5 rounding [0.8067, 0.8067, 0.8067] yes ``` The sameness arrives **with the third item**: when the data version, the feature code, and the split are pinned, the three runs give `[0.8067, 0.8067, 0.8067]`. When the fourth and fifth items are added the result does not change, because nothing is left that could change. Two readings follow: **pinning the grid and the rounding adds nothing to this measurement**, and what decides is the **content** of the list, not its **length**; a setup with two items pinned still gives three different numbers, and one with three items pinned does not. ## Pinning Whose Contribution Cannot Be Measured Adding in order does not show each item's share separately, because later items are stacked on a result that has already become the same. The fourth block does the reverse: it pins all five items, then releases a single item. An item's contribution counts as **measured** only if the result changes when it is released. The same block also writes the cost of pinning itself. ```python print(f"{'single unpinned item':<24}{'three runs':>28}{'contribution measured':>23}") for item in ITEMS: res = three_runs([k for k in ITEMS if k != item]) print(f"{item:<24}{'[' + ', '.join(f'{x:.4f}' for x in res) + ']':>28}" f"{('yes' if len(set(res)) > 1 else 'no'):>23}") # The cost of pinning itself: item records written in each run , candidates evaluated , rows read. none_pinned, all_pinned = [settings(set(), t) for t in range(3)], [settings(set(ITEMS), t) for t in range(3)] print(f"\n{'cost':<26}{'unpinned':>10}{'pinned':>8}") print(f"{'item records written':<26}{0:>10}{len(ITEMS) * 3:>8}") print(f"{'candidates evaluated':<26}{sum(a['candidates'] for a in none_pinned):>10}" f"{sum(a['candidates'] for a in all_pinned):>8}") print(f"{'rows read':<26}{sum(len(S) - a['row_trim'] for a in none_pinned):>10}" f"{sum(len(S) - a['row_trim'] for a in all_pinned):>8}") print(f"{'different results':<26}{len(set(three_runs([]))):>10}" f"{len(set(three_runs(ITEMS))):>8}") ``` ``` single unpinned item three runs contribution measured data_version [0.8067, 0.8064, 0.8078] yes feature_code [0.8067, 0.8067, 0.8067] no split [0.8067, 0.7950, 0.8067] yes grid [0.8067, 0.8067, 0.8067] no rounding [0.8067, 0.8067, 0.8067] no cost unpinned pinned item records written 0 15 candidates evaluated 63 81 rows read 6540 6600 different results 3 1 ``` Tested one at a time, only **two items** remain whose contribution can be measured: the data version and the split. When the other three are released the three runs are still exactly the same; this measurement cannot show anything gained from pinning them. That does not mean those three pinnings are unnecessary; it means exactly this: the assurance they give is an **unmeasured** assurance. A setup saying "we pin all five items" does not show that all five of those items touch the result; what has been shown is only two. The feature code row explains this best. When this item is released, the window drops to 11 and 10, and the sweep known from the previous lesson says that from window 2 upward the result does not change. So the item's contribution is not visible here because it **was not perturbed enough**; the same item had been measured to carry the accuracy from 0.8067 to 0.7800 when it dropped to window 1. "Contribution could not be measured" is not a property of an item but a property of **that test**. The cost table is the other side of the trade. A pinned setup writes five item records per run, **15 records** over three runs; in the unpinned setup this number is zero, and that is where the problem comes from. Pinning does not save work either: because the grid stays full-sized in every run the candidates evaluated rise **from 63 to 81**, and because the data version is not trimmed the rows read rise **from 6540 to 6600**. The only thing received in return is the last row: the three runs give **a single result** instead of **three different results**. The course's rule defends the cost of that row: a number that cannot be reproduced is not a measurement but a memory. ## Summary - In the unpinned setup three runs find the same weight triple yet give 0.8067, 0.7955, and 0.8078; the model that is learned is the same, and what changes is only the slice on which the number was measured. - When the split is not pinned the second run is measured on 396 rows; since the three numbers were produced with different denominators, even placing them side by side is wrong. Label imbalance (base ratio 0.7964, 0.8067 in the test) enlarges this misreading. - When three of the five items are pinned the three runs come out exactly the same; the result does not change even if the grid and the rounding are not pinned, that is, what decides is the content of the pinning list, not its length. - When the items are released one at a time, only the data version and the split remain with a measurable contribution; the other three give an **unmeasured assurance**, and that has to do with the perturbing power of the test. - The cost of pinning is 15 item records over three runs, 81 candidate evaluations instead of 63, and 6600 row reads instead of 6540; in return three different results fall to a single result. ## Next Step Five items were pinned and the three runs now give the same number. But **which** of the runs produced this number is still written nowhere. What is in hand is a model that carries nothing but a weight triple, and since all three runs found the same triple, that model does not point to the run that produced it. Whether a model is being tried, being tested, or in service, and when it was moved to that stage, sit in the same gap. The next lesson counts this gap: it measures how many records are looked at to find which run a model at a given stage came from, how far that number falls with a registry, and how runs that give the same result are told apart from one another.