--- title: 'MLOps Fundamentals' source: 'https://academia.sh/en/courses/mlops-fundamentals' language: en updated: '2026-10-21T05:03:01+00:00' license: 'CC BY-SA 4.0' lessons: 12 outcomes: - 'Splits the machine learning lifecycle into versionable components and counts which items, once pinned, make a run reproducible' - 'Measures whether the training and serving paths produce the same feature and shows that the skew comes from the path, not the model' - 'Builds training and deployment pipelines and compares serving patterns by latency class and cost per prediction' --- # MLOps Fundamentals Counting whether the same model can be reproduced from the same data, and whether the path in training and the path in serving are the same: what an unversioned baseline gives when software engineering practice is applied to the model, which run parameter, metric and output logging in experiment tracking makes findable again, the change data and model versioning make traceable, the consistency a feature store keeps between training and inference, and accuracy with the same weights falling from 0.8067 to 0.7800 because of the feature path rather than the model, in environment pinning three of five items making three runs identical while the contribution of the other two cannot be measured, the lifecycle stages of a model registry, connecting data preparation, training and evaluation steps into a pipeline, code, data and model gates firing separately in continuous integration, packaging with a runtime and an interface contract, the latency class and cost of batch, online and streaming inference, the effect of request batching and caching on cost per prediction, and what shrinking a model to run at the edge takes from accuracy. ## Reproducibility Counting, item by item, whether the same model can be reproduced from the same data: an unversioned and unpinned baseline in which three runs give three different test accuracies, experiment tracking and what recording parameters, metrics and outputs does to make a run findable again along with the cost of the record itself, data and model versioning separating two different contents given the same name, the consistency a feature store keeps between training and inference and the same weights giving an accuracy of 0.8067 on the training path and 0.7800 on the serving path, the skew closing completely when the feature window is widened to two, environment pinning in which three of five items make three runs identical while the contributions of the grid and rounding items cannot be measured, and which run the stage transitions in a model registry are tied to. ### What Is MLOps Source: https://academia.sh/en/courses/mlops-fundamentals/what-is-mlops Setting up the application of software engineering practice to a model-producing pipeline and measuring the unpinned baseline: when the 2200 rows coming out of the pipeline are labeled with a 40 m3 threshold the distribution is 1752 to 448, the base class ratio is 0.7964, and the model searched on a 27-candidate grid gives 0.7931 in training and 0.8067 on the test set, but since the test set's own base ratio is also 0.8067 the model calls none of the 600 test rows high and its recall stays at 0.0000; when nothing is pinned and the same job is run three times, the weights stay the same in all three runs while the test accuracy comes out 0.8067, 0.7955, and 0.8078, and since all three numbers equal the base ratio of their own test set, the difference comes not from the model but from the set changing. Over forty-nine lessons, data was moved, stored, transformed and partitioned. The number of rows each stage of the pipeline drops was counted, how many stages later a corruption was caught was measured, and how rows are laid out and what is gained when they are split among four workers was written down. All of this work had a purpose that was never once opened up: the **model** this data feeds. This course asks two questions: whether **the same model can be reproduced** from the same data, and **how a trained model reaches production**. The first topic covers the first. This lesson sets up the course's measurement scheme, the label and the fictional model, and then measures a single thing: how many different numbers come out when the same job is run three times with no item pinned. - **RP1.** The source, the pipeline and the model are **fictional**. No real learning library, experiment tracking setup, feature store, model registry, or serving framework is called, and none is named. The seed is **20260218**. - **RP2.** The course's rule: a model's number is meaningful **together with the path that produced it**. **A number that cannot be reproduced is not a measurement but a memory.** - **RP3.** In every lesson the baseline is a **setup that is unversioned, unpinned and single-path**. - **RP4.** The label: a period's consumption is "high" (1) if it exceeds **40 m3** and 0 if it does not. The threshold is **a choice**, and every result sensitive to it is written together with the threshold. - **RP5.** The label distribution is **imbalanced**: `{0: 1752, 1: 448}`, that is, a **base class ratio of 0.7964**. An accuracy number is **not counted as measured** until it is written how far above this it sits. The baseline procedure was measured in the Supervised Learning course; here it is **inherited**, not repeated. - **RP6.** The fictional model is **a threshold with three weights**; training is **a search on a 27-candidate grid**. - **RP7.** The training periods are eight (`2026-02` through `2026-09`, **1600 rows**) and the test periods are three (`2026-10` through `2026-12`, **600 rows**). - **RP8.** Resolution: since the test set is 600 rows, the smallest step of the accuracy is **0.0017**. - **RP9.** **The training pipeline is handled on a single machine in this course.** Partitioning, shuffling, and skew were measured in the Distributed Data Processing course and are not repeated here. - **RP10.** The units counted are accuracy, candidate, training row, pinned item and identical run; **time is not written in raw milliseconds**. - **RP11.** Five pinning items: `data_version`, `feature_code`, `split`, `grid`, `rounding`. MLOps is **the name of a practice**, not a product name. #### Applying Software Practice to a Model A program's output is determined by its source, and all the tools of software practice were built on top of this assumption: version control stores the source, regression tests pin the behavior, continuous integration sends every change down the same path. These procedures were measured in the Continuous Integration and Delivery course and in the Data Pipelines and Orchestration course; they are not repeated. In a model the assumption fails. What determines the number a model gives is not only the source code. There are at least five items: the **data** the model sees, **how** that data is split into training and test, **which code** computes the feature, **which candidates** the search space consists of, and **how many digits** the numbers are rounded to. Of these five items, only one lives inside the source file; the remaining four sit either in a directory, in a command-line option, or nowhere. MLOps is the name of the practice of tying these items to the discipline source code already has: it brings not new mathematics but **record keeping**. #### The Fictional Model and the Label The block below rebuilds, verbatim, the fictional measurement network source from the Introduction to Data Engineering course and the first three stages of the pipeline, and adds this course's label, feature generation, and fictional model on top. ```python # FICTIONAL. Meter readings, the pipeline and the model are defined within the lesson; no real # learning library, experiment tracking setup, or model registry is called. SEED, M32 = 20260218, 0xFFFFFFFF def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value REGION = ["east", "west", "north", "south", "central"] TYPE = ["residential", "commercial"] PERIODS = [f"2026-{a:02d}" for a in range(1, 13)] SUBSCRIBERS = {} for i in range(200): r = rng(SEED + 37 * i) SUBSCRIBERS[f"A{i + 1:03d}"] = {"region": REGION[int(r() * 5)], "type": TYPE[int(r() * 2)], "start": 10000 + int(r() * 60000)} def generate_source(): records = [] for i, (sub, a) in enumerate(SUBSCRIBERS.items()): e = a["start"] for p in PERIODS: r = rng(SEED + 101 * i + 7 * PERIODS.index(p)) e += 4 + int(r() * 46) records.append({"subscriber": sub, "period": p, "region": a["region"], "type": a["type"], "index": e, "unit": "m3"}) return records def s1_screen(records): REQUIRED = ("subscriber", "period", "region", "index", "unit") keep = [k for k in records if all(a in k for a in REQUIRED)] return keep, len(records) - len(keep) def s2_dedupe(records): seen, keep = set(), [] for k in records: key = (k["subscriber"], k["period"]) if key in seen: continue seen.add(key) keep.append(k) return keep, len(records) - len(keep) def s3_consumption(records): table = {(k["subscriber"], k["period"]): k for k in records} out, dropped = [], 0 for k in records: i = PERIODS.index(k["period"]) if i == 0: continue prev = table.get((k["subscriber"], PERIODS[i - 1])) if prev is None: dropped += 1 continue try: f = int(k["index"]) - int(prev["index"]) except (TypeError, ValueError): dropped += 1 continue out.append(dict(k, m3=f)) return out, dropped TRAIN = PERIODS[1:9] # 2026-02 .. 2026-09 TEST = PERIODS[9:] # 2026-10 .. 2026-12 THRESHOLD = 40 # consumption above this many m3 counts as "high" def rows(): v, _ = s1_screen(generate_source()) v, _ = s2_dedupe(v) v, _ = s3_consumption(v) return v def label(k): return 1 if k["m3"] > THRESHOLD else 0 # ---- Feature generation. Independent of the weights , computed once. def features(records, window=None, digits=4): """window=None -> full history (training path).""" ordered = sorted(records, key=lambda k: PERIODS.index(k["period"])) sub, reg = {}, {} for k in ordered: sub.setdefault(k["subscriber"], []).append((PERIODS.index(k["period"]), k["m3"])) reg.setdefault(k["region"], []).append((PERIODS.index(k["period"]), k["m3"])) def mean(series, i): chosen = [v for d, v in series if (d < i and (window is None or d >= i - window))] return round(sum(chosen) / len(chosen), digits) if chosen else 0.0 feats = {} for k in records: i = PERIODS.index(k["period"]) feats[(k["subscriber"], k["period"])] = { "subscriber_mean": mean(sub[k["subscriber"]], i), "region_mean": mean(reg[k["region"]], i), "commercial": 1 if k["type"] == "commercial" else 0} return feats # ---- FICTIONAL model: a three-weight threshold. Training = searching the weights on a grid. GRID = [(a, b, c) for a in (0.6, 0.8, 1.0) for b in (0.0, 0.2, 0.4) for c in (0, 4, 8)] def score(o, w): return w[0] * o["subscriber_mean"] + w[1] * o["region_mean"] + w[2] * o["commercial"] def train(records, feats, grid=None, threshold=THRESHOLD, periods=None): grid = GRID if grid is None else grid periods = TRAIN if periods is None else periods subset = [k for k in records if k["period"] in periods] best, best_w = -1.0, None for w in grid: d = sum(1 for k in subset if (1 if score(feats[(k["subscriber"], k["period"])], w) > threshold else 0) == label(k)) o = d / len(subset) if o > best: best, best_w = o, w return {"weights": best_w, "train_accuracy": round(best, 4), "candidates": len(grid), "train_rows": len(subset)} def evaluate(records, model, feats, threshold=THRESHOLD, periods=None): periods = TEST if periods is None else periods subset = [k for k in records if k["period"] in periods] d = sum(1 for k in subset if (1 if score(feats[(k["subscriber"], k["period"])], model["weights"]) > threshold else 0) == label(k)) return round(d / len(subset), 4) # ---- Reproducibility: the pinned items ITEMS = ("data_version", "feature_code", "split", "grid", "rounding") def run(records, pinned, trial=0): """Every item that is not pinned changes with the trial.""" rec = records if "data_version" in pinned else records[: len(records) - 20 * trial] window = None if "feature_code" in pinned else (None if trial == 0 else 12 - trial) digits = 4 if "rounding" in pinned else max(4 - trial, 0) grid = GRID if "grid" in pinned else GRID[: len(GRID) - 6 * trial] periods = TEST if "split" in pinned else PERIODS[9 + (trial % 2):] feats = features(rec, window, digits) m = train(rec, feats, grid) return {"weights": m["weights"], "train": m["train_accuracy"], "test": evaluate(rec, m, feats, periods=periods)} ``` The model looks at a subscriber's past consumption average, the average of the subscriber's region, and whether the subscriber is commercial; it produces a score with three weights, and if the score exceeds the threshold it says "high". Training means trying the 27 candidates and choosing the one that gets the most right on the training set. Since the features are independent of the weights, `features` computes them **once**. ```python S = rows() distribution = {0: 0, 1: 0} for k in S: distribution[label(k)] += 1 print("rows out of the pipeline:", len(S), " threshold:", THRESHOLD) print("label distribution:", distribution) print("base class ratio:", round(distribution[0] / len(S), 4)) print() feats = features(S) model = train(S, feats) print("candidates:", model["candidates"], " train rows:", model["train_rows"]) print("weights:", model["weights"]) print("train accuracy:", model["train_accuracy"]) print("test accuracy:", evaluate(S, model, feats)) ``` ``` rows out of the pipeline: 2200 threshold: 40 label distribution: {0: 1752, 1: 448} base class ratio: 0.7964 candidates: 27 train rows: 1600 weights: (0.6, 0.4, 0) train accuracy: 0.7931 test accuracy: 0.8067 ``` Two numbers sit side by side here, and both appear throughout the rest of the course. The first is **the label distribution**: of the 2200 rows, 1752 are "not high" and 448 are "high", so the distribution is imbalanced. The second is **the base class ratio**: a rule that learns nothing and says "not high" for every row gets an accuracy of **0.7964**. The model got 0.8067; on paper it looks 0.0103 above the base. #### A Number Is Not Read Without Its Base Class Ratio That comparison is wrong, because 0.7964 is the base ratio of **all** the data, while the model was measured only on the test set. The correct one is made with the test set's own base ratio. ```python test_rows = [k for k in S if k["period"] in TEST] high = sum(1 for k in test_rows if label(k) == 1) predicted = {0: 0, 1: 0} hits = 0 for k in test_rows: t = 1 if score(feats[(k["subscriber"], k["period"])], model["weights"]) > THRESHOLD else 0 predicted[t] += 1 if t == 1 and label(k) == 1: hits += 1 print("test rows:", len(test_rows), " labeled high:", high) print("test base class ratio:", round((len(test_rows) - high) / len(test_rows), 4)) print("model test accuracy:", evaluate(S, model, feats)) print("model prediction distribution:", predicted) print(f"recall: {hits / high:.4f}") print(f"above the base: {evaluate(S, model, feats) - (len(test_rows) - high) / len(test_rows):.4f}") ``` ``` test rows: 600 labeled high: 116 test base class ratio: 0.8067 model test accuracy: 0.8067 model prediction distribution: {0: 600, 1: 0} recall: 0.0000 above the base: 0.0000 ``` The test set's base ratio is **0.8067**, and the model's accuracy is also **0.8067**. The difference is **0.0000**. The reason shows in the last two lines: the model does not say "high" to **any** of the 600 test rows, catches none of the 116 rows labeled high, and its **recall stays at 0.0000**. This result is written on purpose. The fictional model **does not rise above the baseline**, and that frees what this course measures from confusion: from here on, every wobble in the accuracy comes **not from the model's performance but from the path that produced the number**. The procedure for building a model, choosing a threshold, and coping with imbalance was measured in the Machine Learning and Data Science curriculum and is not repeated. The rule is this: **when 0.8067 is written alone it is a performance claim and it is wrong**; when it is written next to the base ratio it says the model measures nothing. #### The Same Job, Three Runs, Three Numbers Now the real question can be asked. The run above was done once and gave one number; if the same job is done again a week later, will the same number come out? Nothing is pinned: whatever is in the store that day is read, and the feature code may have changed, the grid may have shortened, the split may have shifted. `run` simulates this setup for three trials. ```python print("pinned items: 0 -- none of the five is pinned") print(f"{'trial':<5}{'weights':>16}{'train':>9}{'test':>9}") for trial in range(3): r = run(S, (), trial) print(f"{trial:<5}{str(r['weights']):>16}{r['train']:>9.4f}{r['test']:>9.4f}") print() print(f"{'trial':<5}{'rows':>7}{'candidates':>12}{'test periods':>14}{'test rows':>11}{'base ratio':>12}") for trial in range(3): rec = S[: len(S) - 20 * trial] periods = PERIODS[9 + (trial % 2):] test3 = [k for k in rec if k["period"] in periods] high3 = sum(1 for k in test3 if label(k) == 1) print(f"{trial:<5}{len(rec):>7}{len(GRID) - 6 * trial:>12}{len(periods):>14}" f"{len(test3):>11}{(len(test3) - high3) / len(test3):>12.4f}") ``` ``` pinned items: 0 -- none of the five is pinned trial weights train test 0 (0.6, 0.4, 0) 0.7931 0.8067 1 (0.6, 0.4, 0) 0.7932 0.7955 2 (0.6, 0.4, 0) 0.7926 0.8078 trial rows candidates test periods test rows base ratio 0 2200 27 3 600 0.8067 1 2180 21 2 396 0.7955 2 2160 15 3 588 0.8078 ``` The upper table is the course's baseline: **0.8067**, **0.7955**, **0.8078**. Between the largest and the smallest there is a **0.0123** difference; since the resolution is 0.0017, this is larger than seven steps, so it is not noise. The **weights of the three runs, though, are exactly the same**: `(0.6, 0.4, 0)`. The lower table says where the difference comes from. In each trial the number of rows read dropped, the number of candidates fell from 27 to 21 and to 15, and **the test periods changed**: three periods in the first trial, two in the second. The last column gives the definitive answer. The test base ratios of the three trials are **0.8067**, **0.7955**, and **0.8078** respectively; the **same** as the three numbers in the upper table. Since the model says "not high" to every row in all three trials, what is measured is not the model's performance but which rows fall into the test set in that trial. From this the course's rule can be read. The three numbers are not three answers to the same question; they are the answers to **three different questions**, and which question they answer is written nowhere. If one of these numbers went into a report a week later, the person reading it could not know how much data was read, how many candidates were tried, or which periods were set aside for testing. **A number that cannot be reproduced is not a measurement but a memory.** #### The Course's Three Columns For the rest of the course, every lesson puts three numbers side by side. The first is **the unpinned baseline**: the three runs above, three different numbers. The second is **the pinned setup**: the same job, with certain items pinned, counting how many of the runs come out identical. The third is **the cost**: the pinned item, the stored record, the added step. The third column cannot be left empty, because **a pinning whose contribution cannot be shown gives an unmeasured assurance**. #### Summary - MLOps is the name of the practice of tying the five items that determine a model's number to the discipline source code already has: data, split, feature code, search space and rounding. Of the five, only one lives inside the source file. - The fictional model is a threshold with three weights, and training is a search on a 27-candidate grid. The label is produced with a 40 m3 threshold, the distribution comes out `{0: 1752, 1: 448}`, and the **base class ratio** is **0.7964**. - The model's test accuracy is **0.8067**, and the test set's base ratio is also **0.8067**. The model says "high" to none of the 600 rows, its **recall is 0.0000**, and it sits **0.0000** above the base. - On the unpinned baseline, three runs give **0.8067**, **0.7955**, and **0.8078**; the **0.0123** difference between them is seven times the 0.0017 resolution. The weights of the three runs are exactly the same, and all three numbers equal the base ratio of their own test set. - The course's rule is this: a model's number is meaningful together with the path that produced it, and **a number that cannot be reproduced is not a measurement but a memory**. #### Next Step Three runs were made and three numbers came out, but none was written down anywhere. The only reason we can see the numbers side by side is that we ran all three in the same block; in a real study these three runs are spread over three separate days, and all that remains between them is a remembered number. The next lesson sets up that record and measures a single question: how many runs have to be repeated to find the weights that produced a result, how many with the record, and how much the record itself costs. ### Experiment Tracking Source: https://academia.sh/en/courses/mlops-fundamentals/experiment-tracking Measuring how recording parameters, metrics and outputs makes a run findable again: in a space of eighteen runs with no record, finding the weights that produced a reported number takes 3 re-runs for a target of 0.8078, 2 for 0.7955, 5 for 0.7950, 1 for 0.8067, and 18 for a target that was never produced, while with a full record all five need 0 repeats; the number alone does not identify the run, because fourteen of the eighteen runs give 0.8067 and the candidate found by re-running is only one of fourteen; the cost of the record is nine fields per run and 162 fields in total, while a 36-field ledger that records only metrics shortens only one of five searches. In the previous lesson three runs were made and three different numbers came out. The only reason we could see the three side by side was that all three had been run in the same block. In a real study these runs are spread over days, done by different people, and all that is left is a single number: an accuracy written on a slide, relayed in a message, or remembered by one person. This lesson goes after that number. Suppose we have a run space and someone passes along this information: "a run gave 0.8078". There is a single question: how much work it takes to find that run's weights and the items it used. The measure is **the number of re-run runs**. - **RP12.** The run space is **eighteen runs**: six pinning levels (from zero to five items) times three trials. The space is a choice, and every number sensitive to it is written together with the size of the space. - **RP13.** **Experiment tracking** is storing a run's parameters, metrics and output together with the run itself. It is not a product but **a record-keeping discipline**. - **RP14.** The baseline is **the setup with no record**: no run has been written down, and all that is at hand is the reported number. - **RP15.** The procedure for a search with no record is this: the space is re-run **from start to finish in order**, and it stops the moment the reported number is matched. The number of repeats depends on this order, and the order is **a choice**. - **RP16.** The cost of the record is counted as **stored fields**: the number of fields per record times the number of runs. Raw bytes are not written. - **RP17.** Training practice, run variability, and the tuning budget were measured in the Deep Learning course; they are **not repeated** here. This lesson only counts **how much the record shortens the search**. - **RP18.** The label distribution is `{0: 1752, 1: 448}` as set up in the previous lesson, the **base class ratio is 0.7964**, and the accuracy numbers in this lesson carry no performance claim above that ratio. The resolution is **0.0017**. #### The Run Space and the Ledger The first block rebuilds, verbatim, the fictional source, pipeline and model of the previous lesson. ```python # FICTIONAL. Meter readings, the pipeline and the model are defined within the lesson; no real # learning library or experiment tracking product is called. SEED, M32 = 20260218, 0xFFFFFFFF def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value REGION = ["east", "west", "north", "south", "central"] TYPE = ["residential", "commercial"] PERIODS = [f"2026-{a:02d}" for a in range(1, 13)] SUBSCRIBERS = {} for i in range(200): r = rng(SEED + 37 * i) SUBSCRIBERS[f"A{i + 1:03d}"] = {"region": REGION[int(r() * 5)], "type": TYPE[int(r() * 2)], "start": 10000 + int(r() * 60000)} def generate_source(): records = [] for i, (sub, a) in enumerate(SUBSCRIBERS.items()): e = a["start"] for p in PERIODS: r = rng(SEED + 101 * i + 7 * PERIODS.index(p)) e += 4 + int(r() * 46) records.append({"subscriber": sub, "period": p, "region": a["region"], "type": a["type"], "index": e, "unit": "m3"}) return records def s1_screen(records): REQUIRED = ("subscriber", "period", "region", "index", "unit") keep = [k for k in records if all(a in k for a in REQUIRED)] return keep, len(records) - len(keep) def s2_dedupe(records): seen, keep = set(), [] for k in records: key = (k["subscriber"], k["period"]) if key in seen: continue seen.add(key) keep.append(k) return keep, len(records) - len(keep) def s3_consumption(records): table = {(k["subscriber"], k["period"]): k for k in records} out, dropped = [], 0 for k in records: i = PERIODS.index(k["period"]) if i == 0: continue prev = table.get((k["subscriber"], PERIODS[i - 1])) if prev is None: dropped += 1 continue try: f = int(k["index"]) - int(prev["index"]) except (TypeError, ValueError): dropped += 1 continue out.append(dict(k, m3=f)) return out, dropped TRAIN = PERIODS[1:9] TEST = PERIODS[9:] THRESHOLD = 40 def rows(): v, _ = s1_screen(generate_source()) v, _ = s2_dedupe(v) v, _ = s3_consumption(v) return v def label(k): return 1 if k["m3"] > THRESHOLD else 0 def features(records, window=None, digits=4): """Independent of the weights , computed once.""" ordered = sorted(records, key=lambda k: PERIODS.index(k["period"])) sub, reg = {}, {} for k in ordered: sub.setdefault(k["subscriber"], []).append((PERIODS.index(k["period"]), k["m3"])) reg.setdefault(k["region"], []).append((PERIODS.index(k["period"]), k["m3"])) def mean(series, i): chosen = [v for d, v in series if (d < i and (window is None or d >= i - window))] return round(sum(chosen) / len(chosen), digits) if chosen else 0.0 feats = {} for k in records: i = PERIODS.index(k["period"]) feats[(k["subscriber"], k["period"])] = { "subscriber_mean": mean(sub[k["subscriber"]], i), "region_mean": mean(reg[k["region"]], i), "commercial": 1 if k["type"] == "commercial" else 0} return feats GRID = [(a, b, c) for a in (0.6, 0.8, 1.0) for b in (0.0, 0.2, 0.4) for c in (0, 4, 8)] def score(o, w): return w[0] * o["subscriber_mean"] + w[1] * o["region_mean"] + w[2] * o["commercial"] def train(records, feats, grid=None, threshold=THRESHOLD, periods=None): grid = GRID if grid is None else grid periods = TRAIN if periods is None else periods subset = [k for k in records if k["period"] in periods] best, best_w = -1.0, None for w in grid: d = sum(1 for k in subset if (1 if score(feats[(k["subscriber"], k["period"])], w) > threshold else 0) == label(k)) o = d / len(subset) if o > best: best, best_w = o, w return {"weights": best_w, "train_accuracy": round(best, 4), "candidates": len(grid), "train_rows": len(subset)} def evaluate(records, model, feats, threshold=THRESHOLD, periods=None): periods = TEST if periods is None else periods subset = [k for k in records if k["period"] in periods] d = sum(1 for k in subset if (1 if score(feats[(k["subscriber"], k["period"])], model["weights"]) > threshold else 0) == label(k)) return round(d / len(subset), 4) ITEMS = ("data_version", "feature_code", "split", "grid", "rounding") def run(records, pinned, trial=0): """Every item that is not pinned changes with the trial.""" rec = records if "data_version" in pinned else records[: len(records) - 20 * trial] window = None if "feature_code" in pinned else (None if trial == 0 else 12 - trial) digits = 4 if "rounding" in pinned else max(4 - trial, 0) grid = GRID if "grid" in pinned else GRID[: len(GRID) - 6 * trial] periods = TEST if "split" in pinned else PERIODS[9 + (trial % 2):] feats = features(rec, window, digits) m = train(rec, feats, grid) return {"weights": m["weights"], "train": m["train_accuracy"], "test": evaluate(rec, m, feats, periods=periods)} ``` An experiment record is not a line written after the run is over; it is produced **inside the run**, because only the run itself knows which items it used. The `make_record` below does this: it executes a run in the space and writes the number of rows that run read, the number of candidates it tried, the feature window, the number of test periods, the weights it found, and the accuracy it achieved into a single record. ```python S = rows() SPACE = [(ITEMS[:n], trial) for n in range(6) for trial in range(3)] def make_record(no): """Executes run number no in the space and records the items it used.""" pinned, trial = SPACE[no - 1] r = run(S, pinned, trial) return {"run": no, "pinned": len(pinned), "trial": trial, "rows": len(S) if "data_version" in pinned else len(S) - 20 * trial, "candidates": len(GRID) if "grid" in pinned else len(GRID) - 6 * trial, "window": None if "feature_code" in pinned else (None if trial == 0 else 12 - trial), "test_periods": len(TEST) if "split" in pinned else len(PERIODS) - 9 - (trial % 2), "weights": r["weights"], "test": r["test"]} LEDGER = [make_record(no) for no in range(1, len(SPACE) + 1)] print("runs:", len(LEDGER), " fields per record:", len(LEDGER[0]), " stored fields:", len(LEDGER) * len(LEDGER[0])) print() print(f"{'run':<7}{'pinned':>8}{'trial':>7}{'rows':>7}{'candidates':>12}{'window':>8}{'periods':>9}{'test':>9}") for k in LEDGER[:3] + LEDGER[-3:]: print(f"{k['run']:<7}{k['pinned']:>8}{k['trial']:>7}{k['rows']:>7}{k['candidates']:>12}" f"{str(k['window']):>8}{k['test_periods']:>9}{k['test']:>9.4f}") counts = {} for k in LEDGER: counts[k["test"]] = counts.get(k["test"], 0) + 1 print() print("different test numbers:", len(counts), " number -> how many runs:", dict(sorted(counts.items()))) ``` ``` runs: 18 fields per record: 9 stored fields: 162 run pinned trial rows candidates window periods test 1 0 0 2200 27 None 3 0.8067 2 0 1 2180 21 11 2 0.7955 3 0 2 2160 15 10 3 0.8078 16 5 0 2200 27 None 3 0.8067 17 5 1 2200 27 None 3 0.8067 18 5 2 2200 27 None 3 0.8067 different test numbers: 4 number -> how many runs: {0.795: 2, 0.7955: 1, 0.8067: 14, 0.8078: 1} ``` The first three lines of the ledger are the previous lesson's unpinned baseline; in the last three lines all five items are pinned, and all three trials use the same rows, the same candidates, the same window, and the same test periods. The last line gives the main finding: the eighteen runs produced only **four different numbers**, and of these **0.8067 came out in fourteen runs**. #### Searching for the Run That Produced a Number Now the place where the record pays off can be measured. In the setup with no record, all that is at hand is the reported number, and the only procedure is to re-run the space from the start. In the setup with a record, the ledger is read. In between there is a third case, and in practice it is the most common one: **only the metric was recorded**, that is, the run number and the accuracy were written but the parameters were not. ```python TARGETS = (0.8078, 0.7955, 0.7950, 0.8067, 0.8100) def search_without_record(target): """No record: the space is re-run from the start, and it stops when the number matches.""" repeats = 0 for pinned, trial in SPACE: repeats += 1 if run(S, pinned, trial)["test"] == target: return repeats return repeats METRICS_ONLY = [(k["run"], k["test"]) for k in LEDGER] print("metrics-only record -- fields:", len(METRICS_ONLY) * 2, " full record -- fields:", len(LEDGER) * len(LEDGER[0])) print() print(f"{'target':<9}{'no record':>10}{'metrics only':>14}{'full record':>13}{'same-number runs':>18}") for h in TARGETS: a = search_without_record(h) candidates = [k["run"] for k in LEDGER if k["test"] == h] o = 0 if not any(s == h for _, s in METRICS_ONLY) else a print(f"{h:<9.4f}{a:>10}{o:>14}{0:>13}{len(candidates):>18}") print() print("runs giving 0.8067:", [k["run"] for k in LEDGER if k["test"] == 0.8067]) ``` ``` metrics-only record -- fields: 36 full record -- fields: 162 target no record metrics only full record same-number runs 0.8078 3 3 0 1 0.7955 2 2 0 1 0.7950 5 5 0 2 0.8067 1 1 0 14 0.8100 18 0 0 0 runs giving 0.8067: [1, 4, 6, 7, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18] ``` The first column is the cost of the setup with no record, and depending on the target it varies between 1 and 18. The last row is the most expensive: the reported 0.8100 was never produced in this space, so the search repeats **all eighteen runs** and nothing is left in hand. This is the most common outcome of a search with no record, because the reported number often comes from the state the space was in that day, and the space has changed. The third column is **0** for every target. A search with a full record needs no re-run; the ledger is read, and that is all. #### A Number Does Not Identify a Run The real finding of the table sits in the last column, and it matters more than the repeat counts. At the target 0.8067, the search with no record "succeeds" in **one** run: the first run in the space gives this number and the search stops. But **fourteen runs** give the same number. The first run the search finds does not have to be the run that produced the reported number; it is only one of fourteen candidates. The consequence is this: re-running can reproduce **a number**, but it cannot diagnose **a run**. If the reported number is 0.8067, whether that run read 2200 rows or 2160 rows, whether it tried 27 candidates or 15, and whether it split the test into three periods or two cannot be deduced from the number. The ledger answers all of these questions in a single line and tells the fourteen candidates apart. What the record gives is **not savings on repeats but diagnosis**; savings on repeats are a by-product. #### The Cost of the Record The cost of the ledger has two items. The first is **stored fields**: each run writes nine fields, and eighteen runs hold **162 fields**. This grows in direct proportion to the number of runs; if hundreds of runs are made in a tuning sweep, the ledger grows by that same proportion and needs a retention policy. The second is **the added step**: `make_record` runs inside the run, which means a write step enters the training pipeline, and that step has to read the items the run used. A record that cannot read an item cannot write it. The second column shows how this cost can be wasted. A ledger that records only metrics holds 36 fields, a little more than a fifth of the full ledger, and it shortens **only one** of five searches: at the never-produced target 0.8100 it brings eighteen repeats down to zero. For the remaining four targets the number of repeats is **exactly the same** as in the setup with no record, because when the parameters are not written the run still has to be re-run. That a ledger is being kept is not an assurance; unless **which fields it holds** is measured, the ledger is nothing but a cost. #### Summary - Experiment tracking means writing a run's parameters, metrics and output inside the run itself; since only the run knows its items, the record cannot be completed afterward. - The eighteen-run space produces only **four different numbers**, and **0.8067 comes out in fourteen runs**. - In the setup with no record, finding the run that produced a reported number costs **1**, **2**, **3**, **5**, or **18** repeated runs; for a number that was never produced, the entire space is run in vain. With a full record, the number of repeats is **0** for all five targets. - Re-running reproduces a number but cannot diagnose the run: at the target 0.8067 the search stops after one repeat, and the run it finds is only one of **fourteen candidates**. - The cost of the record is **nine fields** per run, **162 fields** in total, and a write step added to the training pipeline. - A **36-field** ledger that records only metrics shortens only one of five searches; what is measured is not the existence of a ledger but **which fields it holds**. #### Next Step The ledger wrote down the number of rows the run used, but it did not write down **which** rows they were. All eighteen runs read their data from the same place, and the name of that place never changed; whether its content changed was never asked. When a run records that it read 2200 rows, the run a week later also records that it read 2200 rows, and the two numbers look the same. The next lesson measures the content under that name: in a setup with no versioning, how many runs point to the same name and give different results, how many with versioning, and how much the versioning itself costs. ### Data and Model Versioning Source: https://academia.sh/en/courses/mlops-fundamentals/data-and-model-versioning Measuring how well two different contents given the same name are told apart: when four deliveries come under the same `measurement-2026` name with 2200, 2180, 2160, and again 2200 rows, in an unversioned setup a single name leaves all four runs ambiguous and three different test numbers, 0.8067, 0.8064, and 0.8078, stand behind the same name, while naming by content signature produces three names and brings ambiguous runs to zero; on the model side, because the weights of the four runs are exactly the same, a model identity that looks only at the weights collects the four runs into a single version and three versions separate only when the data signature is added; as the cost, the date-named scheme stores 8740 rows while the content-addressed scheme stores 6540 rows and 8740 rows are read for the signature at every delivery; the output signature does not see a change the pipeline swallowed, and when sixty records arrive twice the source signature changes while the output signature and the accuracy stay exactly the same. The previous lesson's ledger wrote down how many rows each run read. All eighteen runs read their data from the same place, and the name of that place never changed. When the ledger says "2200 rows", we assumed that two runs read the same 2200 rows, but that assumption was never tested anywhere. This lesson tests that assumption. Data lineage, that is, the record of the link from a serving value back to the source, was measured in the Introduction to Data Engineering course and is not repeated here. The question here is narrower: when the content under a name changes and the name does not, how many runs become indistinguishable from one another. - **RP19.** The fictional producer brings the data in **four deliveries**. The deliveries are 2200, 2180, 2160, and again 2200 rows; all four are written under the same `measurement-2026` name. **The fourth delivery is exactly identical in content to the first.** - **RP20.** A **data version** is the pairing of a name with a content. Three schemes are compared: **unversioned** (a single name, overwritten), **date-named** (one name per delivery), and **content-addressed** (the name carries the signature produced from the content). - **RP21.** A **signature** is a short digest produced from the content and is written within the lesson; no external library is called. A signature is not a security tool but an **identity** tool. - **RP22.** The baseline is the **unversioned setup**: a single name, overwritten content, no retroactive reproduction. - **RP23.** The units measured are **the number of names**, **ambiguous runs**, and **stored rows**. An ambiguous run is a run that carries the same name yet gives a different result under that name. - **RP24.** The feature code is **fixed** in this lesson; all four runs see the full history and round to four digits. The only thing that changes is **the content of the data**. - **RP25.** The label distribution is `{0: 1752, 1: 448}`, the **base class ratio is 0.7964**, and the accuracy numbers in this lesson carry no performance claim. The resolution is **0.0017**. #### The Content Under the Same Name The first block rebuilds, verbatim, the fictional source, pipeline and model of the previous two lessons. ```python # FICTIONAL. Meter readings, the pipeline and the model are defined within the lesson; no real # learning library or model registry product is called. SEED, M32 = 20260218, 0xFFFFFFFF def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value REGION = ["east", "west", "north", "south", "central"] TYPE = ["residential", "commercial"] PERIODS = [f"2026-{a:02d}" for a in range(1, 13)] SUBSCRIBERS = {} for i in range(200): r = rng(SEED + 37 * i) SUBSCRIBERS[f"A{i + 1:03d}"] = {"region": REGION[int(r() * 5)], "type": TYPE[int(r() * 2)], "start": 10000 + int(r() * 60000)} def generate_source(): records = [] for i, (sub, a) in enumerate(SUBSCRIBERS.items()): e = a["start"] for p in PERIODS: r = rng(SEED + 101 * i + 7 * PERIODS.index(p)) e += 4 + int(r() * 46) records.append({"subscriber": sub, "period": p, "region": a["region"], "type": a["type"], "index": e, "unit": "m3"}) return records def s1_screen(records): REQUIRED = ("subscriber", "period", "region", "index", "unit") keep = [k for k in records if all(a in k for a in REQUIRED)] return keep, len(records) - len(keep) def s2_dedupe(records): seen, keep = set(), [] for k in records: key = (k["subscriber"], k["period"]) if key in seen: continue seen.add(key) keep.append(k) return keep, len(records) - len(keep) def s3_consumption(records): table = {(k["subscriber"], k["period"]): k for k in records} out, dropped = [], 0 for k in records: i = PERIODS.index(k["period"]) if i == 0: continue prev = table.get((k["subscriber"], PERIODS[i - 1])) if prev is None: dropped += 1 continue try: f = int(k["index"]) - int(prev["index"]) except (TypeError, ValueError): dropped += 1 continue out.append(dict(k, m3=f)) return out, dropped TRAIN = PERIODS[1:9] TEST = PERIODS[9:] THRESHOLD = 40 def rows(): v, _ = s1_screen(generate_source()) v, _ = s2_dedupe(v) v, _ = s3_consumption(v) return v def label(k): return 1 if k["m3"] > THRESHOLD else 0 def features(records, window=None, digits=4): """Independent of the weights , computed once.""" ordered = sorted(records, key=lambda k: PERIODS.index(k["period"])) sub, reg = {}, {} for k in ordered: sub.setdefault(k["subscriber"], []).append((PERIODS.index(k["period"]), k["m3"])) reg.setdefault(k["region"], []).append((PERIODS.index(k["period"]), k["m3"])) def mean(series, i): chosen = [v for d, v in series if (d < i and (window is None or d >= i - window))] return round(sum(chosen) / len(chosen), digits) if chosen else 0.0 feats = {} for k in records: i = PERIODS.index(k["period"]) feats[(k["subscriber"], k["period"])] = { "subscriber_mean": mean(sub[k["subscriber"]], i), "region_mean": mean(reg[k["region"]], i), "commercial": 1 if k["type"] == "commercial" else 0} return feats GRID = [(a, b, c) for a in (0.6, 0.8, 1.0) for b in (0.0, 0.2, 0.4) for c in (0, 4, 8)] def score(o, w): return w[0] * o["subscriber_mean"] + w[1] * o["region_mean"] + w[2] * o["commercial"] def train(records, feats, grid=None, threshold=THRESHOLD, periods=None): grid = GRID if grid is None else grid periods = TRAIN if periods is None else periods subset = [k for k in records if k["period"] in periods] best, best_w = -1.0, None for w in grid: d = sum(1 for k in subset if (1 if score(feats[(k["subscriber"], k["period"])], w) > threshold else 0) == label(k)) o = d / len(subset) if o > best: best, best_w = o, w return {"weights": best_w, "train_accuracy": round(best, 4), "candidates": len(grid), "train_rows": len(subset)} def evaluate(records, model, feats, threshold=THRESHOLD, periods=None): periods = TEST if periods is None else periods subset = [k for k in records if k["period"] in periods] d = sum(1 for k in subset if (1 if score(feats[(k["subscriber"], k["period"])], model["weights"]) > threshold else 0) == label(k)) return round(d / len(subset), 4) ``` The four deliveries run in the block below. In each delivery the same code reads the data under the same name, trains the model, and writes the result under the name `consumption-threshold`. `signature` produces a short digest from the content and is only reported for now; it enters no decision. ```python S = rows() def signature(text): """A short digest produced from the content. Not security , an identity tool.""" h = 2166136261 for c in text: h = ((h ^ ord(c)) * 16777619) & 0xFFFFFFFF return f"{h:08x}" def data_signature(records): return signature("|".join(f"{k['subscriber']}:{k['period']}:{k['m3']}" for k in records)) DELIVERIES = [2200, 2180, 2160, 2200] RUNS = [] for no, n in enumerate(DELIVERIES, start=1): rec = S[:n] feats = features(rec) m = train(rec, feats) RUNS.append({"run": no, "data_name": "measurement-2026", "rows": n, "data_signature": data_signature(rec), "model_name": "consumption-threshold", "weights": m["weights"], "test": evaluate(rec, m, feats)}) print(f"{'delivery':<10}{'data name':<18}{'rows':>7}{'data signature':>16}" f"{'model name':>24}{'weights':>17}{'test':>9}") for k in RUNS: print(f"{k['run']:<10}{k['data_name']:<18}{k['rows']:>7}{k['data_signature']:>16}" f"{k['model_name']:>24}{str(k['weights']):>17}{k['test']:>9.4f}") print() print("different test numbers:", len(set(k["test"] for k in RUNS)), " different weights:", len(set(k["weights"] for k in RUNS)), " different data contents:", len(set(k["data_signature"] for k in RUNS))) ``` ``` delivery data name rows data signature model name weights test 1 measurement-2026 2200 bd88a253 consumption-threshold (0.6, 0.4, 0) 0.8067 2 measurement-2026 2180 f008c71c consumption-threshold (0.6, 0.4, 0) 0.8064 3 measurement-2026 2160 cebfcc57 consumption-threshold (0.6, 0.4, 0) 0.8078 4 measurement-2026 2200 bd88a253 consumption-threshold (0.6, 0.4, 0) 0.8067 different test numbers: 3 different weights: 1 different data contents: 3 ``` In the table, two columns never change: the data name is `measurement-2026` in all four rows, and the model name is `consumption-threshold` in all four rows. One more column never changes: the weights are `(0.6, 0.4, 0)` in all four runs, that is, the model file that is produced is **exactly the same**. Two columns change: the content signature takes three different values, and the test accuracy gives three different numbers. #### How Many Contents a Name Has What is to be counted is now clear. If a naming scheme leaves runs that carry the same name yet give different results, those runs are **ambiguous**: which content they came from cannot be read from the name. ```python def measure(identity): """How many names there are , how many runs stay ambiguous , how many rows are stored.""" group = {} for k in RUNS: group.setdefault(identity(k), []).append(k) ambiguous = sum(len(v) for v in group.values() if len(set(x["test"] for x in v)) > 1) return len(group), ambiguous, sum(v[-1]["rows"] for v in group.values()) print(f"{'scheme':<18}{'data names':>12}{'ambiguous runs':>16}{'stored rows':>14}") for name, ident in (("unversioned", lambda k: k["data_name"]), ("date named", lambda k: f"{k['data_name']}@t{k['run']}"), ("content addressed", lambda k: f"{k['data_name']}@{k['data_signature']}")): a, b, s = measure(ident) print(f"{name:<18}{a:>12}{b:>16}{s:>14}") print() print(f"{'model identity':<26}{'versions':>10}{'ambiguous runs':>16}") for name, ident in (("weights only", lambda k: signature(str(k["weights"]))), ("data signature + weights", lambda k: signature(f"{k['data_signature']}|{k['weights']}"))): a, b, _ = measure(ident) print(f"{name:<26}{a:>10}{b:>16}") print() print("rows read for the signature:", sum(DELIVERIES)) ``` ``` scheme data names ambiguous runs stored rows unversioned 1 4 2200 date named 4 0 8740 content addressed 3 0 6540 model identity versions ambiguous runs weights only 1 4 data signature + weights 3 0 rows read for the signature: 8740 ``` In the unversioned setup there is a single name, and **all four runs are ambiguous**. This row is the data-side form of the course's rule: when the number 0.8078 goes into a report, the content that produced it no longer exists, because the fourth delivery overwrote the third. The stored rows column confirms it: the unversioned scheme takes up the least room, **2200 rows**, and in return loses the whole history. The date-named scheme brings the ambiguity to zero, but it has a flaw: it produces four names although there are three contents. The first and the fourth deliveries are **exactly the same content**, and since the date scheme cannot see this, it stores the same 2200 rows twice. The content-addressed scheme covers the same ambiguity with **three names** and **6540 rows**; the fourth delivery takes up no new room, because its signature is already in the ledger. #### What the Signature Signs `data_signature` signs the pipeline's **output**, that is, the triple of subscriber, period, and computed consumption. This is a choice and it has a cost: a producer-side change that the pipeline swallows never shows up in the output signature. The block below measures this. Suppose the producer sends sixty records twice in July; the source has changed, but the deduplication stage drops the repeated rows. ```python def source_signature(records): return signature("|".join(f"{k['subscriber']}:{k['period']}:{k['index']}" for k in records)) def feed_pipeline(source): v, _ = s1_screen(source) v, _ = s2_dedupe(v) v, _ = s3_consumption(v) return v raw = generate_source() # The producer sent 60 records twice: the source changed , the pipeline swallows it. duplicated = raw + [dict(k) for k in raw if k["subscriber"] <= "A060" and k["period"] == "2026-07"] print(f"{'delivery':<12}{'source rows':>13}{'source signature':>18}" f"{'output rows':>13}{'output signature':>18}{'test':>9}") for name, source in (("intact", raw), ("duplicated", duplicated)): c = feed_pipeline(source) feats2 = features(c) m2 = train(c, feats2) print(f"{name:<12}{len(source):>13}{source_signature(source):>18}{len(c):>13}" f"{data_signature(c):>18}{evaluate(c, m2, feats2):>9.4f}") print() print(f"{'delivery':<10}{'rows':>7}{'test rows':>11}{'resolution':>12}") for n in DELIVERIES[:3]: test_rows = [k for k in S[:n] if k["period"] in TEST] print(f"{n:<10}{n:>7}{len(test_rows):>11}{1 / len(test_rows):>12.4f}") ``` ``` delivery source rows source signature output rows output signature test intact 2400 d501f04f 2200 bd88a253 0.8067 duplicated 2460 93576832 2200 bd88a253 0.8067 delivery rows test rows resolution 2200 2200 600 0.0017 2180 2180 594 0.0017 2160 2160 588 0.0017 ``` The source signature changed, the output signature stayed **exactly the same**, and so did the accuracy. The version records of the two deliveries point to the same rows, although the content that came from the producer is different. This is the **correct** behavior of the pipeline, and it is a limit of versioning: the output signature does not see a change the pipeline swallowed. In the opposite direction it is also lacking; a scheme that signs the source flags a version difference here although the data going to the model has not changed at all. The two signatures answer two separate questions, and a data version stays incomplete **unless it is written which one is kept**. The lower table closes another detail. As the delivery gets smaller, the test set gets smaller too: 600, 594, and 588 rows. Across the three deliveries the resolution stays at 0.0017, but the numbers being compared were not measured **on the same set**. For this reason the difference between 0.8067 and 0.8064 cannot be read as a performance difference. #### A Model's Identity Is Not Only Its Weights The lower table carries the lesson's quietest finding. The model file is the same in all four runs, so a scheme that names the model **by its own content** collects the four runs into **a single version** and leaves all four ambiguous. This is the mirror image of the error on the data side: there a name concealed three contents, here a content conceals three numbers. The four model files look interchangeable, because they carry the same weights. They are not interchangeable: each was measured on a different data set, and the numbers they report are 0.8067, 0.8064, and 0.8078. The differences of 0.0014 and 0.0011 in between are **below** the resolution of 0.0017, and the numbers were moreover not measured on the same set; the reported number is not enough to tell these four files apart. When the data signature is added to the weights, the identity splits into **three versions** and ambiguous runs fall to **0**. A model version is not the signature of the model file but the joint signature of **the model file and the data version that produced it**. #### The Cost of Versioning and What It Does Not Say The cost has three items. The first is **stored rows**: the content-addressed scheme holds 6540 rows, **2.97 times** the unversioned scheme. The second is **rows read**: since the signature is computed from the content, all the rows are read once more at every delivery, **8740 rows** over four deliveries. The third is **the added step**: the write operation now has to compute a signature and append a ledger row. What is gained in return is narrow and exact: versioning says **which content produced which number**. What it does not say is which content is **correct**. Which of the three deliveries came in incomplete, which was whole, and whether the number 0.8078 can be trusted are things none of the three signatures tells; what tells them is the checks measured in the Introduction to Data Engineering course. Versioning is not a quality tool but an **identity tool**, and its contribution is measured only by the number of ambiguous runs. #### Summary - Four deliveries brought 2200, 2180, 2160, and again 2200 rows under the same `measurement-2026` name; the name never changed, the content received **three different signatures**, and the test accuracy came out **0.8067**, **0.8064**, and **0.8078**. - In the unversioned scheme there is a single name and **all four runs are ambiguous**; in the content-addressed scheme there are **three names** and ambiguous runs are **0**. - The date-named scheme closes the ambiguity in the same way but produces **four names** and stores **8740 rows**; the content-addressed scheme does the same job with **three names** and **6540 rows**, that is, **2.97 times** the unversioned scheme, because it sees that the fourth delivery is the same as the first. - The output signature does not see a change the pipeline swallowed: when sixty records arrive twice, the source signature changes and **the output signature and the accuracy stay exactly the same**. A data version stays incomplete unless it is written which signature is kept. - The model file is exactly the same in all four runs. A model identity that looks only at the weights collects the four runs into **a single version**; when the data signature is added, **three versions** separate and ambiguous runs become **0**. - Versioning says which content produced which number, but it does not say which is **correct**; that question belongs to the checks. #### Next Step In three lessons the run itself was recorded and versioned: which items it was run with was written down, which content it read was tied to a signature, and the three contents behind the same name were told apart. All of this work was done **on the training side**. It was never asked whether the feature that goes into the model could be computed by one piece of code during training and **by another piece of code** when called in production. The next lesson puts those two paths side by side and measures a single question: when the weights are exactly the same, why does the accuracy drop. ### Feature Stores Source: https://academia.sh/en/courses/mlops-fundamentals/feature-stores Measuring whether a feature is computed the same way between training and inference: with the same weights the test accuracy comes out 0.8067 through the training path and 0.7800 through the serving path, and the model never changes. The whole 0.0267 in between comes from which history the feature is computed with; it flips the prediction in 22 of the 600 test rows, breaking 19 and fixing three, and closes entirely when the feature window is raised to two. A feature store reduces the two computations to one and zeroes the skew; in return it brings the cost of a store of 6600 values, a fourth stage in the model branch, and rewriting everything stored when the feature code changes. In three lessons, what was versioned was **the run itself**: which parameters it ran with, which metric it produced, and which data version it read went into the record. All of these records look at the run **from the outside**, and none of them reached the number that goes into the model, that is, **how the feature is computed**. This lesson closes that gap. The code that trains the model runs in one place and the code that produces predictions runs in another; a feature with the same name is computed in both places, and whether the two computations give the same number is something nobody checks. What is to be measured is how that difference shows up in accuracy, and the number the lesson carries is this: **accuracy drops while the model does not change at all.** - **RP41.** The source, the first three stages of the pipeline, and the threshold-based model are **fictional**; no real learning library, feature store product, or serving framework is called. The seed is **20260218**, and the output does not depend on the run. - **RP42.** A **feature** is a number that goes into the model, **computed** from a raw row. There are three here: the subscriber's past consumption mean, the region's past consumption mean, and the commercial indicator. The feature generation procedure was measured in the Data Preparation and Feature Engineering course and is **not repeated**. - **RP43.** The **training path** computes the feature from the **full history**; the **serving path** sees only the **last period**. The difference between the accuracies the two paths give with the same weights is **training–serving skew**. - **RP44.** The label is imbalanced: `{0: 1752, 1: 448}`, the **base class ratio is 0.7964**, and in the test set it is **0.8067**. Accuracy read alone misleads, and in this lesson it will mislead twice. - **RP45.** The model is a threshold with three weights; training is a search on a **27-candidate grid**. Grid search was measured in the Supervised Learning course and is **not repeated**. - **RP46.** **Inference** is producing a prediction for a single row with the trained weights; in this course both training and inference are handled on a **single machine**. - **RP47.** The baseline is the **single-path setup**: the feature is computed separately in two different places, and there is no record saying that the two are the same. - **RP48.** Resolution: the test set is **600 rows** and the smallest step of accuracy is **1/600 = 0.0017**. This lesson's skew is **0.0267**, that is, **sixteen times** the step, and it has been measured. #### Two Computations of the Number That Goes Into the Model A feature is not the raw row itself; it is produced from the raw row by a **rule**. "The subscriber's past consumption mean" is such a rule, and written as a single sentence it looks complete. What is missing is what the sentence silently assumes: **where the past starts.** In training the answer comes by itself; all twelve periods are on the table and the mean is taken from the full history. In serving the situation is different: the prediction is produced as soon as the period arrives, **only that period** is often at hand, and the feature with the same name is computed with the value of the last period. Both rules are written correctly, both carry the same name, and both produce different numbers for the same row. The block below builds the source, the first three stages of the pipeline that face the model, the label, and the fictional model. The `window` parameter of the `features` function separates the two paths: `None` means the full history and `1` means only the last period. The features are independent of the weights and are computed **once**. ```python # FICTIONAL. Meter readings, the first three stages of the pipeline, and the threshold-based # model are defined within the lesson; no real learning library, feature store product, # or serving framework is called. SEED, M32 = 20260218, 0xFFFFFFFF def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value REGION = ["east", "west", "north", "south", "central"] TYPE = ["residential", "commercial"] PERIODS = [f"2026-{a:02d}" for a in range(1, 13)] SUBSCRIBERS = {} for i in range(200): r = rng(SEED + 37 * i) SUBSCRIBERS[f"A{i + 1:03d}"] = {"region": REGION[int(r() * 5)], "type": TYPE[int(r() * 2)], "start": 10000 + int(r() * 60000)} def generate_source(): records = [] for i, (sub, a) in enumerate(SUBSCRIBERS.items()): e = a["start"] for p in PERIODS: r = rng(SEED + 101 * i + 7 * PERIODS.index(p)) e += 4 + int(r() * 46) records.append({"subscriber": sub, "period": p, "region": a["region"], "type": a["type"], "index": e, "unit": "m3"}) return records def s1_screen(records): REQUIRED = ("subscriber", "period", "region", "index", "unit") keep = [k for k in records if all(a in k for a in REQUIRED)] return keep, len(records) - len(keep) def s2_dedupe(records): seen, keep = set(), [] for k in records: key = (k["subscriber"], k["period"]) if key in seen: continue seen.add(key) keep.append(k) return keep, len(records) - len(keep) def s3_consumption(records): """Consumption from the period difference. Without a previous period the row cannot be computed.""" table = {(k["subscriber"], k["period"]): k for k in records} out, dropped = [], 0 for k in records: i = PERIODS.index(k["period"]) if i == 0: continue prev = table.get((k["subscriber"], PERIODS[i - 1])) if prev is None: dropped += 1 continue try: f = int(k["index"]) - int(prev["index"]) except (TypeError, ValueError): dropped += 1 continue out.append(dict(k, m3=f)) return out, dropped TRAIN = PERIODS[1:9] # 2026-02 .. 2026-09 TEST = PERIODS[9:] # 2026-10 .. 2026-12 THRESHOLD = 40 # consumption above this many m3 counts as "high" def rows(): v, _ = s1_screen(generate_source()) v, _ = s2_dedupe(v) v, _ = s3_consumption(v) return v def label(k): return 1 if k["m3"] > THRESHOLD else 0 def features(records, window=None, digits=4): """window=None -> full history (training path); window=1 -> last period (serving path).""" ordered = sorted(records, key=lambda k: PERIODS.index(k["period"])) sub, reg = {}, {} for k in ordered: sub.setdefault(k["subscriber"], []).append((PERIODS.index(k["period"]), k["m3"])) reg.setdefault(k["region"], []).append((PERIODS.index(k["period"]), k["m3"])) def mean(series, i): chosen = [v for d, v in series if (d < i and (window is None or d >= i - window))] return round(sum(chosen) / len(chosen), digits) if chosen else 0.0 feats = {} for k in records: i = PERIODS.index(k["period"]) feats[(k["subscriber"], k["period"])] = { "subscriber_mean": mean(sub[k["subscriber"]], i), "region_mean": mean(reg[k["region"]], i), "commercial": 1 if k["type"] == "commercial" else 0} return feats GRID = [(a, b, c) for a in (0.6, 0.8, 1.0) for b in (0.0, 0.2, 0.4) for c in (0, 4, 8)] def score(o, w): return w[0] * o["subscriber_mean"] + w[1] * o["region_mean"] + w[2] * o["commercial"] def train(records, feats, grid=None, threshold=THRESHOLD, periods=None): """Training = searching the weight triple on the grid. The feature comes from outside.""" grid = GRID if grid is None else grid periods = TRAIN if periods is None else periods subset = [k for k in records if k["period"] in periods] best, best_w = -1.0, None for w in grid: d = sum(1 for k in subset if (1 if score(feats[(k["subscriber"], k["period"])], w) > threshold else 0) == label(k)) o = d / len(subset) if o > best: best, best_w = o, w return {"weights": best_w, "train_accuracy": round(best, 4), "candidates": len(grid), "train_rows": len(subset)} def evaluate(records, model, feats, threshold=THRESHOLD, periods=None): periods = TEST if periods is None else periods subset = [k for k in records if k["period"] in periods] d = sum(1 for k in subset if (1 if score(feats[(k["subscriber"], k["period"])], model["weights"]) > threshold else 0) == label(k)) return round(d / len(subset), 4) ``` #### The Same Weights, Two Accuracies The second block trains the model **once** and tests the resulting weight triple on two paths: the same three numbers, the same threshold, the same test periods. The only thing that changes is which history the feature is computed with. ```python import collections S = rows() TEST_ROWS = [k for k in S if k["period"] in TEST] KEYS = [(k["subscriber"], k["period"]) for k in TEST_ROWS] TRUTH = {(k["subscriber"], k["period"]): label(k) for k in TEST_ROWS} FEATS_TRAIN = features(S) # full history -> training path FEATS_SERVE = features(S, window=1) # last period -> serving path MODEL = train(S, FEATS_TRAIN) dist = collections.Counter(label(k) for k in S) base = collections.Counter(TRUTH.values()) print("rows:", len(S), " train:", MODEL["train_rows"], " test:", len(TEST_ROWS)) print("label distribution:", dict(sorted(dist.items())), " base class ratio:", round(dist[0] / len(S), 4), " in test:", round(base[0] / len(TEST_ROWS), 4)) print("model weights:", MODEL["weights"], " candidates:", MODEL["candidates"], " train accuracy:", MODEL["train_accuracy"]) def predictions(feats): return {a: (1 if score(feats[a], MODEL["weights"]) > THRESHOLD else 0) for a in KEYS} PT, PS = predictions(FEATS_TRAIN), predictions(FEATS_SERVE) print(f"\n{'path':<10}{'features differ':>17}{'said high':>11}{'true positive':>15}{'accuracy':>10}") for name, feats, p in (("training", FEATS_TRAIN, PT), ("serving", FEATS_SERVE, PS)): print(f"{name:<10}{sum(1 for a in KEYS if feats[a] != FEATS_TRAIN[a]):>17}" f"{sum(p.values()):>11}{sum(1 for a in KEYS if p[a] == 1 == TRUTH[a]):>15}" f"{evaluate(S, MODEL, feats):>10.4f}") broken = sum(1 for a in KEYS if PT[a] == TRUTH[a] != PS[a]) fixed = sum(1 for a in KEYS if PS[a] == TRUTH[a] != PT[a]) print("changed predictions:", broken + fixed, " broken:", broken, " fixed:", fixed, " net:", broken - fixed) print("skew:", round(evaluate(S, MODEL, FEATS_TRAIN) - evaluate(S, MODEL, FEATS_SERVE), 4), "= net / 600 ; accuracy step 1/600 =", round(1 / len(TEST_ROWS), 4)) ``` ``` rows: 2200 train: 1600 test: 600 label distribution: {0: 1752, 1: 448} base class ratio: 0.7964 in test: 0.8067 model weights: (0.6, 0.4, 0) candidates: 27 train accuracy: 0.7931 path features differ said high true positive accuracy training 0 0 0 0.8067 serving 600 22 3 0.7800 changed predictions: 22 broken: 19 fixed: 3 net: 16 skew: 0.0267 = net / 600 ; accuracy step 1/600 = 0.0017 ``` The weight triple `(0.6, 0.4, 0)` is the same in both rows. Accuracy drops **from 0.8067 to 0.7800**, and the whole drop comes from the feature path; not a single number in the model has changed. The breakdown shows how the difference arises: **the feature is different in 600 of the 600 rows**, but the prediction flips in only **22**, because a difference changes the prediction only when it can carry it to the other side of the threshold. Of the 22 flipped predictions **19 break**, **three get fixed**, a net of **16 rows** is lost, and 16/600 comes to exactly **0.0267**: sixteen times the resolution, that is, a measured quantity. This is also where accuracy misleads for the second time. The base class ratio of the test set is **0.8067**; the `said high` column is **zero** on the training path. That is, the model scoring 0.8067 says "high" to no row at all and produces exactly the number of a model that **says the majority class**. Neither number shows that a model is useful; what is measured is **how far the two paths diverge**. #### Where the Skew Comes From The difference is not a bug; as the history the computation sees gets shorter, the mean approaches the volatility of a single period. The third block sweeps the window from 1 to the full history and tests with the same weights at every step. Next to it is placed a model **trained** on the serving path. ```python FULL = evaluate(S, MODEL, FEATS_TRAIN) print(f"{'window':<9}{'features differ':>17}{'accuracy':>10}{'skew':>9}") for p in (1, 2, 3, 6, None): featsp = features(S, window=p) print(f"{('full' if p is None else p):<9}" f"{sum(1 for a in KEYS if featsp[a] != FEATS_TRAIN[a]):>17}" f"{evaluate(S, MODEL, featsp):>10.4f}{FULL - evaluate(S, MODEL, featsp):>9.4f}") SERVE_MODEL = train(S, FEATS_SERVE) print("\nmodel trained on the serving path:", SERVE_MODEL["weights"], " train:", SERVE_MODEL["train_accuracy"], " test via serving path:", f"{evaluate(S, SERVE_MODEL, FEATS_SERVE):.4f}") ``` ``` window features differ accuracy skew 1 600 0.7800 0.0267 2 600 0.8067 0.0000 3 600 0.8067 0.0000 6 600 0.8067 0.0000 full 0 0.8067 0.0000 model trained on the serving path: (0.6, 0.0, 0) train: 0.7925 test via serving path: 0.8067 ``` The skew **closes entirely at window 2** and does not open again at wider windows. This says the problem is not "the shortness of the history" but its **closeness to zero**. The second column sharpens it: at windows 2, 3, and 6 the feature is **still different in 600 of the 600 rows**, yet the accuracy is exactly the same. The size of the skew is measured not by the difference in feature values but by **how many rows that difference carries to the other side of the threshold**. The last line shows a second repair. When the model is trained with the short-window serving path itself, it finds a different weight `(0.6, 0.0, 0)` and, tested through the same short path, again gives **0.8067**. The moment the inconsistency is lifted, the number comes back; what decides is not which path is chosen but **training and serving using the same path**. #### The Feature Store and Its Cost A **feature store** closes this inconsistency with a **structure** instead of a rule: the feature is computed once, written under the `(subscriber, period)` key, and both training and inference **read** from the same key. Nobody has to check that the two computations agree, because there are not two computations. The fourth block writes the store's own items next to the closed skew. ```python # Feature store: the value is computed ONCE , written under the (subscriber , period) key , # and both training and serving read from the SAME key. STORE = features(S) FIELDS = ("subscriber_mean", "region_mean", "commercial") def from_store(key): return STORE[key] last = TEST[-1] print("store keys:", len(STORE), " fields:", len(FIELDS), " stored values:", len(STORE) * len(FIELDS)) print("read by serving in one period:", sum(1 for a in STORE if a[1] == last), "keys ,", sum(1 for a in STORE if a[1] == last) * len(FIELDS), "values") print("stages of the model branch:", 3, "->", 4, " places the feature rule lives:", 2, "->", 1) print("values rewritten when the feature code changes:", len(STORE) * len(FIELDS)) for name, feats in (("two paths , two computations", FEATS_SERVE), ("one store , one computation", STORE)): print(f"{name:<28} serving accuracy: {evaluate(S, MODEL, feats):.4f}" f" skew: {FULL - evaluate(S, MODEL, feats):.4f}") print("value read from the store the same as in training:", all(from_store(a) == FEATS_TRAIN[a] for a in KEYS)) ``` ``` store keys: 2200 fields: 3 stored values: 6600 read by serving in one period: 200 keys , 600 values stages of the model branch: 3 -> 4 places the feature rule lives: 2 -> 1 values rewritten when the feature code changes: 6600 two paths , two computations serving accuracy: 0.7800 skew: 0.0267 one store , one computation serving accuracy: 0.8067 skew: 0.0000 value read from the store the same as in training: True ``` The gain is a single line: the skew falls **from 0.0267 to 0.0000**, and every value read from the store is exactly the same as the one in training. The cost has four items. The model-facing branch of the pipeline goes **from three stages to four**, because writing the feature is a separate step. **6600 values** are stored, while one period of inference reads only **600** of them; the rest is there so that training can see the history. When the feature rule changes, **all 6600 values** have to be rewritten, otherwise the store carries the old rule and the new rule side by side. The fourth cannot be counted but is real: serving now depends on the store having been written. This trade is the counterpart of the course's rule. Without the store, 0.8067 was truly a number that had been produced, but because it was not written down **by which path** it was produced, it did not come back in serving. A number that cannot be reproduced is not a measurement but a memory. #### Summary - A feature is not a raw row but a number produced from a raw row by a rule; the rule silently answers the question "where does the past start", and training and serving can give it different answers. - With the same weight triple the accuracy is **0.8067** through the training path and **0.7800** through the serving path; the model has not changed at all, and the whole of the **0.0267** training–serving skew comes from the feature path. - Although the skew changes the feature in 600 of 600 rows, it flips the prediction in only **22** rows: 19 breaks, three fixes, a net of 16 rows, that is, 16/600. - The label is imbalanced (`{0: 1752, 1: 448}`, base ratio 0.7964, 0.8067 in the test) and the model scoring 0.8067 says "high" to no row; accuracy alone hides this. - The skew closes entirely at window 2; a feature store gives the same result with a single computation, and its cost is 6600 stored values, a stage that rises to four, and rewriting all of it when the rule changes. #### Next Step The feature store pinned one item: **by which code** the feature is computed now sits in a single place. But that was the only thing pinned. There is no record saying that the data slice read, the test split, the grid searched, and the number of digits the numbers are rounded to have not changed when the same job is run again, and a change in any one of them can change the result. The next lesson counts these five items one by one: it measures how many of them, when pinned, make three runs **exactly** the same, and what kind of assurance a pinning whose contribution cannot be measured gives. ### Environment Pinning Source: https://academia.sh/en/courses/mlops-fundamentals/environment-pinning Counting, one by one, the items that bring a run to the same result a second time: in an unpinned setup three runs find the same weights yet give three different test accuracies such as 0.8067, 0.7955, and 0.8078, because the rows read, the feature window, the rounding digits, the grid searched, and the test split all move in every run. When three of the five items are pinned the three runs come out exactly the same, the result does not change even if the grid and the rounding are not pinned, and when tested one at a time only two items remain whose contribution can be measured. The cost of pinning is fifteen item records, eighteen extra candidate evaluations, and sixty extra row reads. The feature store brought one item down to a single place, and how many points of accuracy that item carried was measured. But the code that computes the feature is not the only thing that decides a run's result. When the same job is run a second time, the data slice read, the periods set aside for testing, the weight grid searched, and the number of digits the intermediate numbers are rounded to may also have changed. None of them is on record, none of them counts as an error, and all of them together move the result. **Environment pinning** is keeping each of these items unchanged from one run to the next. This lesson treats **pinning** not as a habit but as a measure, and asks three questions: pinning which item makes three runs **exactly** the same, what is the cost of pinning itself, and what does a pinning whose contribution cannot be shown actually say. - **RP49.** The source, the first three stages of the pipeline, and the threshold-based model are **fictional**. The seed is **20260218**, and the output does not depend on the run. - **RP50.** **Pinning** is guaranteeing that an item stays the same from run to run. Five items are counted here: data version, feature code, test split, grid, and rounding. **No hardware, runtime, or dependency name is written**; the items are referred to by their kinds. - **RP51.** A **trial** is a number that moves the unpinned items in a set direction, and three trials represent three separate runs of the same job. A trial is not randomness but a reproducible shift. - **RP52.** The baseline is the **unpinned setup**: none of the five items is pinned, and there is no record saying what the run read. - **RP53.** The label is imbalanced: `{0: 1752, 1: 448}`, the **base class ratio is 0.7964**, and in the test set it is **0.8067**. Accuracy read alone misleads. - **RP54.** Run variability and the tuning budget were measured in the Supervised Learning course and are **not repeated**; the only thing counted here is **pinning which item makes the run the same**. - **RP55.** The effect of the feature window was measured in the previous lesson and becomes an **input** here: at window 1 the accuracy is 0.7800, at window 2 and above it is 0.8067. - **RP56.** Resolution: the full test set is **600 rows**, the step is **1/600 = 0.0017**. The widest gap between runs in this lesson is **0.0123**, that is, larger than seven times the step, and it has been measured. #### The Three Numbers of the Unpinned Setup A run's result comes from three layers: which data it read, with which code it processed the data, and on which set it measured the result. When each of them is not recorded, it can drift silently, and the symptom of the drift is not an error message but **another number**. The block below builds the source, the first three stages of the pipeline that face the model, the fictional model, and the five pinning items; the `run` function moves every unpinned item according to the trial number. ```python # FICTIONAL. Meter readings, the first three stages of the pipeline, and the threshold-based # model are defined within the lesson; no real learning library, runtime, # or dependency manager is called. SEED, M32 = 20260218, 0xFFFFFFFF def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value REGION = ["east", "west", "north", "south", "central"] TYPE = ["residential", "commercial"] PERIODS = [f"2026-{a:02d}" for a in range(1, 13)] SUBSCRIBERS = {} for i in range(200): r = rng(SEED + 37 * i) SUBSCRIBERS[f"A{i + 1:03d}"] = {"region": REGION[int(r() * 5)], "type": TYPE[int(r() * 2)], "start": 10000 + int(r() * 60000)} def generate_source(): records = [] for i, (sub, a) in enumerate(SUBSCRIBERS.items()): e = a["start"] for p in PERIODS: r = rng(SEED + 101 * i + 7 * PERIODS.index(p)) e += 4 + int(r() * 46) records.append({"subscriber": sub, "period": p, "region": a["region"], "type": a["type"], "index": e, "unit": "m3"}) return records def s1_screen(records): REQUIRED = ("subscriber", "period", "region", "index", "unit") keep = [k for k in records if all(a in k for a in REQUIRED)] return keep, len(records) - len(keep) def s2_dedupe(records): seen, keep = set(), [] for k in records: key = (k["subscriber"], k["period"]) if key in seen: continue seen.add(key) keep.append(k) return keep, len(records) - len(keep) def s3_consumption(records): """Consumption from the period difference. Without a previous period the row cannot be computed.""" table = {(k["subscriber"], k["period"]): k for k in records} out, dropped = [], 0 for k in records: i = PERIODS.index(k["period"]) if i == 0: continue prev = table.get((k["subscriber"], PERIODS[i - 1])) if prev is None: dropped += 1 continue try: f = int(k["index"]) - int(prev["index"]) except (TypeError, ValueError): dropped += 1 continue out.append(dict(k, m3=f)) return out, dropped TRAIN = PERIODS[1:9] # 2026-02 .. 2026-09 TEST = PERIODS[9:] # 2026-10 .. 2026-12 THRESHOLD = 40 # consumption above this many m3 counts as "high" def rows(): v, _ = s1_screen(generate_source()) v, _ = s2_dedupe(v) v, _ = s3_consumption(v) return v def label(k): return 1 if k["m3"] > THRESHOLD else 0 def features(records, window=None, digits=4): """window=None -> full history; digits -> rounding of the intermediate means.""" ordered = sorted(records, key=lambda k: PERIODS.index(k["period"])) sub, reg = {}, {} for k in ordered: sub.setdefault(k["subscriber"], []).append((PERIODS.index(k["period"]), k["m3"])) reg.setdefault(k["region"], []).append((PERIODS.index(k["period"]), k["m3"])) def mean(series, i): chosen = [v for d, v in series if (d < i and (window is None or d >= i - window))] return round(sum(chosen) / len(chosen), digits) if chosen else 0.0 feats = {} for k in records: i = PERIODS.index(k["period"]) feats[(k["subscriber"], k["period"])] = { "subscriber_mean": mean(sub[k["subscriber"]], i), "region_mean": mean(reg[k["region"]], i), "commercial": 1 if k["type"] == "commercial" else 0} return feats GRID = [(a, b, c) for a in (0.6, 0.8, 1.0) for b in (0.0, 0.2, 0.4) for c in (0, 4, 8)] def score(o, w): return w[0] * o["subscriber_mean"] + w[1] * o["region_mean"] + w[2] * o["commercial"] def train(records, feats, grid=None, threshold=THRESHOLD, periods=None): grid = GRID if grid is None else grid periods = TRAIN if periods is None else periods subset = [k for k in records if k["period"] in periods] best, best_w = -1.0, None for w in grid: d = sum(1 for k in subset if (1 if score(feats[(k["subscriber"], k["period"])], w) > threshold else 0) == label(k)) o = d / len(subset) if o > best: best, best_w = o, w return {"weights": best_w, "train_accuracy": round(best, 4), "candidates": len(grid), "train_rows": len(subset)} def evaluate(records, model, feats, threshold=THRESHOLD, periods=None): periods = TEST if periods is None else periods subset = [k for k in records if k["period"] in periods] d = sum(1 for k in subset if (1 if score(feats[(k["subscriber"], k["period"])], model["weights"]) > threshold else 0) == label(k)) return round(d / len(subset), 4) # ---- Reproducibility: the five pinning items. # Every item that is not pinned moves with the trial; a pinned item stays the same in every trial. ITEMS = ("data_version", "feature_code", "split", "grid", "rounding") def settings(pinned, trial): return {"row_trim": 0 if "data_version" in pinned else 20 * trial, "window": None if "feature_code" in pinned else (None if trial == 0 else 12 - trial), "digits": 4 if "rounding" in pinned else max(4 - trial, 0), "candidates": len(GRID) if "grid" in pinned else len(GRID) - 6 * trial, "periods": TEST if "split" in pinned else PERIODS[9 + (trial % 2):]} def run(records, pinned, trial=0): rec = records if "data_version" in pinned else records[: len(records) - 20 * trial] window = None if "feature_code" in pinned else (None if trial == 0 else 12 - trial) digits = 4 if "rounding" in pinned else max(4 - trial, 0) grid = GRID if "grid" in pinned else GRID[: len(GRID) - 6 * trial] periods = TEST if "split" in pinned else PERIODS[9 + (trial % 2):] feats = features(rec, window, digits) m = train(rec, feats, grid) return {"weights": m["weights"], "train": m["train_accuracy"], "test": evaluate(rec, m, feats, periods=periods)} ``` The second block runs the baseline: none of the items is pinned, three trials, three results. ```python import collections S = rows() dist = collections.Counter(label(k) for k in S) print("rows:", len(S), " label distribution:", dict(sorted(dist.items())), " base class ratio:", round(dist[0] / len(S), 4), " in test:", round(sum(1 for k in S if k["period"] in TEST and label(k) == 0) / sum(1 for k in S if k["period"] in TEST), 4)) print("pinning items:", len(ITEMS), "->", ", ".join(ITEMS)) print(f"\n{'trial':<5}{'rows':>7}{'window':>9}{'digits':>9}{'candidates':>12}" f"{'test rows':>11}{'weights':>18}{'test':>9}") BASELINE = [] for t in range(3): cfg, res = settings(set(), t), run(S, set(), t) remaining = len(S) - cfg["row_trim"] BASELINE.append(res["test"]) print(f"{t:<5}{remaining:>7}{str(cfg['window'] or 'full'):>9}{cfg['digits']:>9}{cfg['candidates']:>12}" f"{sum(1 for k in S[:remaining] if k['period'] in cfg['periods']):>11}" f"{str(res['weights']):>18}{res['test']:>9.4f}") print("three runs identical:", len(set(BASELINE)) == 1, " different results:", len(set(BASELINE)), " widest gap:", round(max(BASELINE) - min(BASELINE), 4)) ``` ``` rows: 2200 label distribution: {0: 1752, 1: 448} base class ratio: 0.7964 in test: 0.8067 pinning items: 5 -> data_version, feature_code, split, grid, rounding trial rows window digits candidates test rows weights test 0 2200 full 4 27 600 (0.6, 0.4, 0) 0.8067 1 2180 11 3 21 396 (0.6, 0.4, 0) 0.7955 2 2160 10 2 15 588 (0.6, 0.4, 0) 0.8078 three runs identical: False different results: 3 widest gap: 0.0123 ``` The three runs give **three different numbers**, and the widest gap of 0.0123 is larger than seven times the resolution. What really has to be said is **the weights column**: all three runs find the triple `(0.6, 0.4, 0)`. The model that is learned is the same; the only thing that changes is **which number that model is reported with**. The first run's 0.8067 and the third run's 0.8078 are two separate memories of the same model, and both look defensible, because the slice they were measured on is not written down. The `test rows` column completes this. Because the split is not pinned, the second run is measured on only **396** rows; its step is 1/396 = 0.0025. The three numbers are not merely different, they are numbers **produced with different denominators**. Label imbalance enlarges the misreading: since the base class ratio in the test is 0.8067, all three numbers sit within the band of a setup that says the majority class. #### How Many Items Are Enough The third block pins the items one at a time in list order and restarts the three runs at every step. The question is a single one: does the added item make the three results **exactly** the same. ```python def three_runs(pinned): return [run(S, set(pinned), t)["test"] for t in range(3)] print(f"{'pinned':<8}{'added item':<17}{'three runs':>28}{'same':>7}") for n in range(len(ITEMS) + 1): res = three_runs(ITEMS[:n]) print(f"{n:<8}{('-' if n == 0 else ITEMS[n - 1]):<17}" f"{'[' + ', '.join(f'{x:.4f}' for x in res) + ']':>28}" f"{('yes' if len(set(res)) == 1 else 'no'):>7}") ``` ``` pinned added item three runs same 0 - [0.8067, 0.7955, 0.8078] no 1 data_version [0.8067, 0.7950, 0.8067] no 2 feature_code [0.8067, 0.7950, 0.8067] no 3 split [0.8067, 0.8067, 0.8067] yes 4 grid [0.8067, 0.8067, 0.8067] yes 5 rounding [0.8067, 0.8067, 0.8067] yes ``` The sameness arrives **with the third item**: when the data version, the feature code, and the split are pinned, the three runs give `[0.8067, 0.8067, 0.8067]`. When the fourth and fifth items are added the result does not change, because nothing is left that could change. Two readings follow: **pinning the grid and the rounding adds nothing to this measurement**, and what decides is the **content** of the list, not its **length**; a setup with two items pinned still gives three different numbers, and one with three items pinned does not. #### Pinning Whose Contribution Cannot Be Measured Adding in order does not show each item's share separately, because later items are stacked on a result that has already become the same. The fourth block does the reverse: it pins all five items, then releases a single item. An item's contribution counts as **measured** only if the result changes when it is released. The same block also writes the cost of pinning itself. ```python print(f"{'single unpinned item':<24}{'three runs':>28}{'contribution measured':>23}") for item in ITEMS: res = three_runs([k for k in ITEMS if k != item]) print(f"{item:<24}{'[' + ', '.join(f'{x:.4f}' for x in res) + ']':>28}" f"{('yes' if len(set(res)) > 1 else 'no'):>23}") # The cost of pinning itself: item records written in each run , candidates evaluated , rows read. none_pinned, all_pinned = [settings(set(), t) for t in range(3)], [settings(set(ITEMS), t) for t in range(3)] print(f"\n{'cost':<26}{'unpinned':>10}{'pinned':>8}") print(f"{'item records written':<26}{0:>10}{len(ITEMS) * 3:>8}") print(f"{'candidates evaluated':<26}{sum(a['candidates'] for a in none_pinned):>10}" f"{sum(a['candidates'] for a in all_pinned):>8}") print(f"{'rows read':<26}{sum(len(S) - a['row_trim'] for a in none_pinned):>10}" f"{sum(len(S) - a['row_trim'] for a in all_pinned):>8}") print(f"{'different results':<26}{len(set(three_runs([]))):>10}" f"{len(set(three_runs(ITEMS))):>8}") ``` ``` single unpinned item three runs contribution measured data_version [0.8067, 0.8064, 0.8078] yes feature_code [0.8067, 0.8067, 0.8067] no split [0.8067, 0.7950, 0.8067] yes grid [0.8067, 0.8067, 0.8067] no rounding [0.8067, 0.8067, 0.8067] no cost unpinned pinned item records written 0 15 candidates evaluated 63 81 rows read 6540 6600 different results 3 1 ``` Tested one at a time, only **two items** remain whose contribution can be measured: the data version and the split. When the other three are released the three runs are still exactly the same; this measurement cannot show anything gained from pinning them. That does not mean those three pinnings are unnecessary; it means exactly this: the assurance they give is an **unmeasured** assurance. A setup saying "we pin all five items" does not show that all five of those items touch the result; what has been shown is only two. The feature code row explains this best. When this item is released, the window drops to 11 and 10, and the sweep known from the previous lesson says that from window 2 upward the result does not change. So the item's contribution is not visible here because it **was not perturbed enough**; the same item had been measured to carry the accuracy from 0.8067 to 0.7800 when it dropped to window 1. "Contribution could not be measured" is not a property of an item but a property of **that test**. The cost table is the other side of the trade. A pinned setup writes five item records per run, **15 records** over three runs; in the unpinned setup this number is zero, and that is where the problem comes from. Pinning does not save work either: because the grid stays full-sized in every run the candidates evaluated rise **from 63 to 81**, and because the data version is not trimmed the rows read rise **from 6540 to 6600**. The only thing received in return is the last row: the three runs give **a single result** instead of **three different results**. The course's rule defends the cost of that row: a number that cannot be reproduced is not a measurement but a memory. #### Summary - In the unpinned setup three runs find the same weight triple yet give 0.8067, 0.7955, and 0.8078; the model that is learned is the same, and what changes is only the slice on which the number was measured. - When the split is not pinned the second run is measured on 396 rows; since the three numbers were produced with different denominators, even placing them side by side is wrong. Label imbalance (base ratio 0.7964, 0.8067 in the test) enlarges this misreading. - When three of the five items are pinned the three runs come out exactly the same; the result does not change even if the grid and the rounding are not pinned, that is, what decides is the content of the pinning list, not its length. - When the items are released one at a time, only the data version and the split remain with a measurable contribution; the other three give an **unmeasured assurance**, and that has to do with the perturbing power of the test. - The cost of pinning is 15 item records over three runs, 81 candidate evaluations instead of 63, and 6600 row reads instead of 6540; in return three different results fall to a single result. #### Next Step Five items were pinned and the three runs now give the same number. But **which** of the runs produced this number is still written nowhere. What is in hand is a model that carries nothing but a weight triple, and since all three runs found the same triple, that model does not point to the run that produced it. Whether a model is being tried, being tested, or in service, and when it was moved to that stage, sit in the same gap. The next lesson counts this gap: it measures how many records are looked at to find which run a model at a given stage came from, how far that number falls with a registry, and how runs that give the same result are told apart from one another. ### Model Registry Source: https://academia.sh/en/courses/mlops-fundamentals/model-registry Counting the lifecycle stages of a model and which run each stage transition is tied to: in a record of sixteen runs only two different weight triples come out, the triple in production is carried exactly by twelve runs, and those twelve runs report their own result with two different numbers such as 0.7950 and 0.8067. Without a model registry, finding which run the production model came from means looking at all sixteen records and the search never goes below twelve candidates, while the previous production run to roll back to cannot be found at all; with a registry that ties four stages and five transitions to a run id, both questions are answered by reading a single record, and the cost of the registry is twenty-one records. Five items were pinned and the three runs now give the same number. That is the answer to the question of reproducibility, but it opens one more question: **which run** did the model in hand come from? The only thing a trained model carries is its weights; the weights do not carry within them the data version, the feature window, or the test split that produced them. This lesson measures that gap. Whether a model is being tried, being tested, or in service, and **with which run** it came to that state, lives not in the model's file but in a separate record. A **model registry** is exactly this record: it ties the stages and the transitions between stages to a run id. What is to be measured is how many records that tie makes you read when answering a question. - **RP57.** The source, the first three stages of the pipeline, the fictional model, and the registry are **fictional**; no real learning library or registry product is called. The seed is **20260218**. - **RP58.** A **run** is a single training job executed with particular settings, and it carries an id. A **model** is the weight triple that run produced, and it carries no id. - **RP59.** A **stage** is the state of the model in its lifecycle. Four are counted here: `candidate`, `staging`, `production`, `archived`. A **transition** is the record that moves a model from one stage to another, and it is **tied to a run id**. - **RP60.** The baseline is the **setup without a registry**: the runs are recorded somewhere, but no stage is tied to a run. The only way to get from a model at a stage to a run is **matching the weights**. - **RP61.** The label is imbalanced: `{0: 1752, 1: 448}`, the **base class ratio is 0.7964**, and in the test set it is **0.8067**. Accuracy read alone misleads; in this lesson two runs will carry the same model and report different numbers. - **RP62.** The run record itself and data versioning were handled in the earlier lessons of this topic; the only thing added here is **tying the stage transition to the run**. The lineage procedure was measured in the Introduction to Data Engineering course and is **not repeated**. - **RP63.** The registry is built on a record of **sixteen runs**. The runs are the combination of four feature windows, two grid sizes, and two test splits. - **RP64.** Resolution: the full test set is **600 rows**, the step is **1/600 = 0.0017**. In this lesson the difference between the two numbers reported by the same weights is **0.0117**, that is, larger than six times the step. #### The Model Has No Identity A trained model is the output of a procedure, and the output does not contain the procedure that produced it. Here the model consists of three numbers; in a real setup it would be far larger, but the situation does not change. Inside the model it is not written which data it was trained with, which feature window was chosen, or over which periods the test was done. The only thing that carries this information is **the run record**, and unless the tie between the model and the run is built separately, it does not exist. When the tie is not built, the only thing that can be done is **matching**: comparing the weights in hand with the weights in the run records and seeing which one fits. This method has a silent assumption: that every run produces a different model. The blocks below will test that assumption. The first block builds the source, the first three stages of the pipeline that face the model, and the fictional model. ```python # FICTIONAL. Meter readings, the first three stages of the pipeline, and the threshold-based # model are defined within the lesson; no real learning library or model registry # product is called. The registry is also built within the lesson as a list of records. SEED, M32 = 20260218, 0xFFFFFFFF def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value REGION = ["east", "west", "north", "south", "central"] TYPE = ["residential", "commercial"] PERIODS = [f"2026-{a:02d}" for a in range(1, 13)] SUBSCRIBERS = {} for i in range(200): r = rng(SEED + 37 * i) SUBSCRIBERS[f"A{i + 1:03d}"] = {"region": REGION[int(r() * 5)], "type": TYPE[int(r() * 2)], "start": 10000 + int(r() * 60000)} def generate_source(): records = [] for i, (sub, a) in enumerate(SUBSCRIBERS.items()): e = a["start"] for p in PERIODS: r = rng(SEED + 101 * i + 7 * PERIODS.index(p)) e += 4 + int(r() * 46) records.append({"subscriber": sub, "period": p, "region": a["region"], "type": a["type"], "index": e, "unit": "m3"}) return records def s1_screen(records): REQUIRED = ("subscriber", "period", "region", "index", "unit") keep = [k for k in records if all(a in k for a in REQUIRED)] return keep, len(records) - len(keep) def s2_dedupe(records): seen, keep = set(), [] for k in records: key = (k["subscriber"], k["period"]) if key in seen: continue seen.add(key) keep.append(k) return keep, len(records) - len(keep) def s3_consumption(records): """Consumption from the period difference. Without a previous period the row cannot be computed.""" table = {(k["subscriber"], k["period"]): k for k in records} out, dropped = [], 0 for k in records: i = PERIODS.index(k["period"]) if i == 0: continue prev = table.get((k["subscriber"], PERIODS[i - 1])) if prev is None: dropped += 1 continue try: f = int(k["index"]) - int(prev["index"]) except (TypeError, ValueError): dropped += 1 continue out.append(dict(k, m3=f)) return out, dropped TRAIN = PERIODS[1:9] # 2026-02 .. 2026-09 TEST = PERIODS[9:] # 2026-10 .. 2026-12 THRESHOLD = 40 # consumption above this many m3 counts as "high" def rows(): v, _ = s1_screen(generate_source()) v, _ = s2_dedupe(v) v, _ = s3_consumption(v) return v def label(k): return 1 if k["m3"] > THRESHOLD else 0 def features(records, window=None, digits=4): """window=None -> full history; window=n -> last n periods.""" ordered = sorted(records, key=lambda k: PERIODS.index(k["period"])) sub, reg = {}, {} for k in ordered: sub.setdefault(k["subscriber"], []).append((PERIODS.index(k["period"]), k["m3"])) reg.setdefault(k["region"], []).append((PERIODS.index(k["period"]), k["m3"])) def mean(series, i): chosen = [v for d, v in series if (d < i and (window is None or d >= i - window))] return round(sum(chosen) / len(chosen), digits) if chosen else 0.0 feats = {} for k in records: i = PERIODS.index(k["period"]) feats[(k["subscriber"], k["period"])] = { "subscriber_mean": mean(sub[k["subscriber"]], i), "region_mean": mean(reg[k["region"]], i), "commercial": 1 if k["type"] == "commercial" else 0} return feats GRID = [(a, b, c) for a in (0.6, 0.8, 1.0) for b in (0.0, 0.2, 0.4) for c in (0, 4, 8)] def score(o, w): return w[0] * o["subscriber_mean"] + w[1] * o["region_mean"] + w[2] * o["commercial"] def train(records, feats, grid=None, threshold=THRESHOLD, periods=None): grid = GRID if grid is None else grid periods = TRAIN if periods is None else periods subset = [k for k in records if k["period"] in periods] best, best_w = -1.0, None for w in grid: d = sum(1 for k in subset if (1 if score(feats[(k["subscriber"], k["period"])], w) > threshold else 0) == label(k)) o = d / len(subset) if o > best: best, best_w = o, w return {"weights": best_w, "train_accuracy": round(best, 4), "candidates": len(grid), "train_rows": len(subset)} def evaluate(records, model, feats, threshold=THRESHOLD, periods=None): periods = TEST if periods is None else periods subset = [k for k in records if k["period"] in periods] d = sum(1 for k in subset if (1 if score(feats[(k["subscriber"], k["period"])], model["weights"]) > threshold else 0) == label(k)) return round(d / len(subset), 4) ``` #### Twelve Runs That Produce the Same Weights The second block produces a record of sixteen runs. The runs really are different from one another: four different feature windows, two different grid sizes, and two different test splits have been tried. Then the record is grouped by the weight triple produced and the test accuracy reported. ```python import collections S = rows() dist = collections.Counter(label(k) for k in S) print("rows:", len(S), " label distribution:", dict(sorted(dist.items())), " base class ratio:", round(dist[0] / len(S), 4), " in test:", round(sum(1 for k in S if k["period"] in TEST and label(k) == 0) / sum(1 for k in S if k["period"] in TEST), 4)) # Run record: four feature windows x two grid sizes x two test splits. FEATS = {p: features(S, p) for p in (None, 1, 2, 3)} RUNS = [] for p in (None, 1, 2, 3): for cand in (27, 21): for per in (TEST, PERIODS[10:]): m = train(S, FEATS[p], GRID[:cand]) RUNS.append({"id": f"K{len(RUNS) + 1:02d}", "window": p, "candidates": cand, "periods": len(per), "weights": m["weights"], "test": evaluate(S, m, FEATS[p], periods=per)}) groups = {} for k in RUNS: groups.setdefault((k["weights"], k["test"]), []).append(k["id"]) print("\nruns:", len(RUNS), " different weight triples:", len(set(k["weights"] for k in RUNS)), " different (weights , test):", len(groups)) print(f"{'weights':<16}{'test':>8}{'runs':>7} ids") for (w, s), ks in groups.items(): print(f"{str(w):<16}{s:>8.4f}{len(ks):>7} {','.join(ks)}") ``` ``` rows: 2200 label distribution: {0: 1752, 1: 448} base class ratio: 0.7964 in test: 0.8067 runs: 16 different weight triples: 2 different (weights , test): 4 weights test runs ids (0.6, 0.4, 0) 0.8067 6 K01,K03,K09,K11,K13,K15 (0.6, 0.4, 0) 0.7950 6 K02,K04,K10,K12,K14,K16 (0.6, 0.0, 0) 0.8067 2 K05,K07 (0.6, 0.0, 0) 0.7950 2 K06,K08 ``` Sixteen separate runs produce only **two different weight triples**. The assumption of matching collapses here: someone holding `(0.6, 0.4, 0)` finds **twelve candidates** even if they read the whole run record, and the search goes no lower. This is not a flaw but the natural result of searching: every training job that wanders over a finite grid can arrive at the same candidate, and most of the time it does. The second number the table carries is heavier. Of the twelve runs carrying the same `(0.6, 0.4, 0)` triple, six report their result as **0.8067** and six as **0.7950**. Both are correct; what makes the difference is not the model itself but **over how many periods** the run did its test. So when the question "what was its accuracy" is asked about the model in production, the answer that can be given by going from the weights is a range 0.0117 wide, and that is larger than six times the resolution. Label imbalance makes this even more unfortunate: since the base class ratio in the test is 0.8067, both numbers sit within the band of the majority class, and taking the difference between them for a performance difference is misleading. #### The Run That Stages and Transitions Are Tied To The model registry does not touch the model's file; instead it keeps **a separate record**, and this record ties a stage to a run id. The four stages and the sequence of transitions between them are the contents of the registry; each transition row says which run was moved to which stage. The third block builds the registry and answers two questions in two setups. ```python # Model registry: four stages and the transitions between stages. Every transition is tied to ONE RUN. STAGES = ("candidate", "staging", "production", "archived") TRANSITIONS = [("K03", "candidate", "staging"), ("K03", "staging", "production"), ("K11", "candidate", "staging"), ("K11", "staging", "production"), ("K03", "production", "archived")] CURRENT = {} for run_id, _, target in TRANSITIONS: CURRENT[target] = run_id PRODUCTION_WEIGHTS = next(k for k in RUNS if k["id"] == CURRENT["production"])["weights"] matching = [k for k in RUNS if k["weights"] == PRODUCTION_WEIGHTS] scores = sorted(set(k["test"] for k in matching)) print("weights in production:", PRODUCTION_WEIGHTS, " transitions in the registry:", len(TRANSITIONS)) print(f"\n{'question':<38}{'looked at without registry':>28}{'answer without registry':>25}" f"{'looked at with registry':>25}{'answer':>8}") print(f"{'which run did this weight come from':<38}{len(RUNS):>28}" f"{str(len(matching)) + ' candidates':>25}{1:>25}{CURRENT['production']:>8}") print(f"{'to roll back: previous production':<38}{len(RUNS):>28}{'none':>25}" f"{1:>25}{CURRENT['archived']:>8}") print("\nruns carrying the same weights:", len(matching), " different tests they report:", len(scores), "->", " .. ".join(f"{x:.4f}" for x in scores), " gap:", round(scores[-1] - scores[0], 4)) print("cost of the registry: run records", len(RUNS), "+ transition records", len(TRANSITIONS), "=", len(RUNS) + len(TRANSITIONS), "records ;", "stages:", len(STAGES)) ``` ``` weights in production: (0.6, 0.4, 0) transitions in the registry: 5 question looked at without registry answer without registry looked at with registry answer which run did this weight come from 16 12 candidates 1 K11 to roll back: previous production 16 none 1 K03 runs carrying the same weights: 12 different tests they report: 2 -> 0.7950 .. 0.8067 gap: 0.0117 cost of the registry: run records 16 + transition records 5 = 21 records ; stages: 4 ``` The first row shows what the stage transition gains. In the setup without a registry all sixteen records are looked at and twelve candidates remain; the search does as much work as the number of records looked at and **never narrows**. In the setup with a registry, because the `production` stage is tied to a run id, a single record is read and the answer is `K11`. What is gained is not a saving of fifteen records; what is gained is **a unique answer**, and in the setup without a registry that answer cannot be obtained whatever the number of records. The second row is sharper. When a rollback is needed, the question asked becomes "which run was the previous production run", and in the setup without a registry the answer is not merely hard, it **does not exist**: the only thing in hand is the weights now in place, and the weights of the previous model have been written over. In the registry, on the other hand, it is recorded that `K03` passed from the `production` stage to the `archived` stage; the run to return to is read in a single record. The value of the stages lies less in their saying where the model is than in their saying **where it came from and where to go back to**. The cost of the registry is in the last row and it is modest: sixteen run records and five transition records, **21 records** in total. This cost does not touch the model itself, does not enlarge the weights, and does not slow inference; it only makes it mandatory that one row be written at every stage transition. What is lost when it is not written is exactly what this lesson measured: whom the number in production belongs to. #### Summary - A trained model carries only its weights; it carries within it neither the data version, nor the feature window, nor the test split that produced it, and therefore it has no identity. - Sixteen separate runs produce only two different weight triples; since **12 runs** carry the triple in production exactly, a search done by matching weights never reaches a unique answer. - The runs carrying the same triple report their result as **0.7950** and **0.8067**; the 0.0117 in between is larger than six times the resolution, and label imbalance (base ratio 0.7964, 0.8067 in the test) makes it easy to take this difference for a performance difference. - With a registry that ties four stages and five transitions to a run id, the question "which run did the production model come from" is answered in **a single record**; in the setup without a registry 16 records are read and 12 candidates remain. - The question of rollback has no answer in the setup without a registry; in the registry the archive transition gives the previous production run in a single record, and the registry's total cost is 21 records. #### Next Step In six lessons the model was made **reproducible**. The run itself was recorded and versioned, two different contents given the same name were told apart, the feature went from being computed twice in two places to being computed in a single place, it was counted which item's pinning made the three runs the same, and every stage transition was tied to a run id. Now it is written whom a number standing in production belongs to and how it can be obtained again. The only thing not written is **how this model will get there**. In none of the six lessons was training itself built as an end-to-end pipeline, no gate stopped a new data version or a falling metric from going to release, and the model was never packaged as an interface a caller could use. The next topic closes these three gaps in order: it counts the steps of the training pipeline and its end-to-end re-runnability, measures which breakage the code, data, and model gates stop, and writes how many callers break when the interface contract is not written. ## Training and Deployment Counting, step by step, the path by which a trained model reaches production: connecting data preparation, training and evaluation steps into a single pipeline and the rows each step drops, code, data and model gates firing separately in continuous integration and which gate stops which corruption, pinning the runtime and the interface contract in packaging and the caller breaking when no contract is written, the latency class, the cost per prediction and the freshness trade-off of batch, online and streaming inference, request batching and caching lowering the cost per prediction while adding to latency, and what shrinking a model to run at the edge takes from accuracy and on which task it takes it first. ### Training Pipelines Source: https://academia.sh/en/courses/mlops-fundamentals/training-pipelines Counting the training pipeline as the first step of a trained model's path to production: the rows dropped by each of six steps are measured separately, and while the consumption step loses 200 rows its own counter reads 0, the feature step leaves 200 rows history-less without dropping a single row, the split separates 1600 and 600 without dropping a single row; with a label distribution of 1752 to 448 and a base class ratio of 0.7964, the model says 0 to 600 of the 600 rows in the test set and its accuracy of 0.8067 is exactly the base class ratio; when the 40 late-arriving rows are backfilled, the only setup that raises the training rows from 1520 to 1600 is the pipeline that re-runs end to end, and all seven setups report the same 0.8067. The first six lessons of this course made the model **reproducible**: the experiment run was recorded, the data and the model were versioned, the feature path in training was compared with the path in serving, the environment items were pinned and the stage transitions were tied to a registry. Their shared assumption can be written in one sentence: the model has been trained once, and the problem is being able to train it **again**. None of them asked **how the trained model would reach production**. This lesson opens the first step of that question. In five stages, the pipeline of the Introduction to Data Engineering course went from 2400 rows to 2400, to 2200, to 55 summary cells and to 5 serving values. The model is not attached to the **end** of that chain; it is attached to the **output of the third stage**, that is, to the 2200 consumption rows. The question of this lesson is what becomes measurable once the steps that follow that point stop being a sequence run by hand and become an end-to-end runnable **training pipeline**. - **TD1.** The source, the pipeline and the model are **fictional**; no real machine learning library, experiment tracking setup, feature store or serving framework is called. The seed is **20260218**. - **TD2.** In this course the training pipeline is treated on **a single machine**; cluster sizing and distribution were measured in the Distributed Data Processing course and are not repeated. - **TD3.** Linking the steps to each other, solving the graph and the retry procedure were measured in the Data Pipelines and Orchestration course. The question asked here is not the order but **what each step does to the rows**. - **TD4.** The label: a row whose consumption exceeds **40 m3** counts as "high". The distribution is **imbalanced** and is written in every lesson. - **TD5.** The training periods are `2026-02`–`2026-09`, the test periods are `2026-10`–`2026-12`. - **TD6.** The model is a threshold with three weights; training is a search on a grid of 27 candidates. - **TD7.** The features are computed **once** and are independent of the weights. - **TD8.** The baseline is the steps run by hand: the intermediate outputs stay on hand, and a re-run is partial. - **TD9.** Resolution: the test set is **600 rows**, and the smallest step of accuracy is **0.0017**. - **TD10.** Time is not written as raw milliseconds; the measures are **rows**, **steps** and **periods**. #### The Six Steps of the Training Pipeline The first block sets up the source, the first three stages of the pipeline and the model side. None of it is specific to this lesson: the first three come from the data pipeline, and the model side is the shared definition of this course. ```python # FICTIONAL. Meter readings, pipeline and model are defined inside the lesson; no real # machine learning library, experiment tracking setup or serving framework is called. SEED, M32 = 20260218, 0xFFFFFFFF def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value REGION = ["east", "west", "north", "south", "central"] TYPE = ["residential", "commercial"] PERIODS = [f"2026-{a:02d}" for a in range(1, 13)] SUBSCRIBERS = {} for i in range(200): r = rng(SEED + 37 * i) SUBSCRIBERS[f"A{i + 1:03d}"] = {"region": REGION[int(r() * 5)], "type": TYPE[int(r() * 2)], "start": 10000 + int(r() * 60000)} def generate_source(): records = [] for i, (sub, a) in enumerate(SUBSCRIBERS.items()): e = a["start"] for p in PERIODS: r = rng(SEED + 101 * i + 7 * PERIODS.index(p)) e += 4 + int(r() * 46) records.append({"subscriber": sub, "period": p, "region": a["region"], "type": a["type"], "index": e, "unit": "m3"}) return records # ---- The first three stages of the data pipeline def s1_screen(records): REQUIRED = ("subscriber", "period", "region", "index", "unit") keep = [k for k in records if all(a in k for a in REQUIRED)] return keep, len(records) - len(keep) def s2_dedupe(records): seen, keep = set(), [] for k in records: key = (k["subscriber"], k["period"]) if key in seen: continue seen.add(key) keep.append(k) return keep, len(records) - len(keep) def s3_consumption(records): """Consumption from the period difference. Without a previous period the row cannot be computed.""" table = {(k["subscriber"], k["period"]): k for k in records} out, dropped = [], 0 for k in records: i = PERIODS.index(k["period"]) if i == 0: continue prev = table.get((k["subscriber"], PERIODS[i - 1])) if prev is None: dropped += 1 continue try: f = int(k["index"]) - int(prev["index"]) except (TypeError, ValueError): dropped += 1 continue out.append(dict(k, m3=f)) return out, dropped # ---- The model side TRAIN, TEST, THRESHOLD = PERIODS[1:9], PERIODS[9:], 40 def label(k): return 1 if k["m3"] > THRESHOLD else 0 def features(records, window=None, digits=4): """window=None -> full history. Independent of the weights , computed once.""" ordered = sorted(records, key=lambda k: PERIODS.index(k["period"])) sub, reg = {}, {} for k in ordered: sub.setdefault(k["subscriber"], []).append((PERIODS.index(k["period"]), k["m3"])) reg.setdefault(k["region"], []).append((PERIODS.index(k["period"]), k["m3"])) def mean(series, i): chosen = [v for d, v in series if (d < i and (window is None or d >= i - window))] return round(sum(chosen) / len(chosen), digits) if chosen else 0.0 return {(k["subscriber"], k["period"]): { "subscriber_mean": mean(sub[k["subscriber"]], PERIODS.index(k["period"])), "region_mean": mean(reg[k["region"]], PERIODS.index(k["period"])), "commercial": 1 if k["type"] == "commercial" else 0} for k in records} GRID = [(a, b, c) for a in (0.6, 0.8, 1.0) for b in (0.0, 0.2, 0.4) for c in (0, 4, 8)] def score(o, w): return w[0] * o["subscriber_mean"] + w[1] * o["region_mean"] + w[2] * o["commercial"] def train(records, feats): subset = [k for k in records if k["period"] in TRAIN] best, best_w = -1.0, None for w in GRID: d = sum(1 for k in subset if (1 if score(feats[(k["subscriber"], k["period"])], w) > THRESHOLD else 0) == label(k)) if d / len(subset) > best: best, best_w = d / len(subset), w return {"weights": best_w, "train_accuracy": round(best, 4), "candidates": len(GRID), "train_rows": len(subset)} def evaluate(records, model, feats): subset = [k for k in records if k["period"] in TEST] d = sum(1 for k in subset if (1 if score(feats[(k["subscriber"], k["period"])], model["weights"]) > THRESHOLD else 0) == label(k)) return round(d / len(subset), 4) ``` The second block sets up the pipeline. The first three of the six steps are the stages above; the remaining three are labeling, feature generation and the split. `run_pipeline` reports the input of each step, its output and **its own dropped-row counter** separately; when a `start` value is given, the first steps come from the stored output and only the rest re-runs. ```python def t4_label(data): return [dict(k, y=label(k)) for k in data], 0 def t5_features(data): feats = features(data) return [dict(k, **feats[(k["subscriber"], k["period"])]) for k in data], 0 def t6_split(data): return {"train": [k for k in data if k["period"] in TRAIN], "test": [k for k in data if k["period"] in TEST]}, 0 STEPS = [("screen", s1_screen), ("dedupe", s2_dedupe), ("consumption", s3_consumption), ("label", t4_label), ("features", t5_features), ("split", t6_split)] def size(data): return len(data) if isinstance(data, list) else sum(len(x) for x in data.values()) def run_pipeline(source, start=0, stored=None): """If start=p the first p steps come from the stored output , the rest re-run.""" data = source if start == 0 else stored[STEPS[start - 1][0]] inter, trace, processed = dict(stored or {}), [], 0 for i, (name, f) in enumerate(STEPS): if i < start: continue n_in = size(data) data, counter = f(data) inter[name], processed = data, processed + n_in trace.append((name, n_in, size(data), counter)) return data, trace, inter, processed def assess(split_data): subset = split_data["train"] + split_data["test"] feats = {(k["subscriber"], k["period"]): {a: k[a] for a in ("subscriber_mean", "region_mean", "commercial")} for k in subset} m = train(subset, feats) return m, evaluate(subset, m, feats), feats SOURCE = generate_source() SPLIT, TRACE, INTER, PROCESSED = run_pipeline(SOURCE) print(f"{'step':<11}{'input':>7}{'output':>7}{'lost':>10}{'step counter':>15}") for name, n_in, n_out, counter in TRACE: print(f"{name:<11}{n_in:>7}{n_out:>7}{n_in - n_out:>10}{counter:>15}") print("split:", {a: len(b) for a, b in SPLIT.items()}, " history-less rows:", sum(1 for k in INTER["features"] if k["subscriber_mean"] == 0.0)) DISTRIBUTION = {0: 0, 1: 0} for k in INTER["label"]: DISTRIBUTION[k["y"]] += 1 print("label distribution:", DISTRIBUTION, " base class ratio:", round(DISTRIBUTION[0] / len(INTER["label"]), 4), " processed rows:", PROCESSED) ``` ``` step input output lost step counter screen 2400 2400 0 0 dedupe 2400 2400 0 0 consumption 2400 2200 200 0 label 2200 2200 0 0 features 2200 2200 0 0 split 2200 2200 0 0 split: {'train': 1600, 'test': 600} history-less rows: 200 label distribution: {0: 1752, 1: 448} base class ratio: 0.7964 processed rows: 13800 ``` The last two columns of the table contradict each other, and the contradiction sits in a single step. The input of the consumption step is 2400 and its output is 2200; **200 rows** are lost in between. The step's own counter reads **0**. Both are correct: the counter counts only the **unexpected** drop, that is, the row whose previous period cannot be found. The 200 rows of the first period drop by definition, because consumption is the difference between two periods. Looking at the dropped-row counter of a step does not tell how many rows that step lost; **the difference between the input and the output has to be measured separately.** The second contradiction is in the feature step and runs the other way. This step **drops no rows at all**, but **200 rows in its output have a subscriber mean of 0.0**, because the `2026-02` rows have no period before themselves. A step whose dropped rows are zero can still corrupt the data; the counter of the pipeline does not see it. The split step really does drop no rows: 1600 training rows and 600 test rows add up to 2200. An end-to-end run processes **13800 rows** across six steps. #### The Number the Evaluation Step Produces The seventh and eighth steps of the pipeline are training and evaluation. The third block runs both and puts the resulting number next to the base class ratio. ```python MODEL, ACCURACY, FEAT_TABLE = assess(SPLIT) TEST_SET = SPLIT["test"] PREDICTIONS = [1 if score(FEAT_TABLE[(k["subscriber"], k["period"])], MODEL["weights"]) > THRESHOLD else 0 for k in TEST_SET] BASE_RATIO_TEST = sum(1 for k in TEST_SET if k["y"] == 0) / len(TEST_SET) print("weights:", MODEL["weights"], " candidates:", MODEL["candidates"], " training rows:", MODEL["train_rows"], " training accuracy:", MODEL["train_accuracy"]) print(f"test accuracy: {ACCURACY:.4f} test base class ratio: {BASE_RATIO_TEST:.4f}" f" gain over base class ratio: {ACCURACY - BASE_RATIO_TEST:.4f}") print("test prediction distribution:", {0: PREDICTIONS.count(0), 1: PREDICTIONS.count(1)}) ``` ``` weights: (0.6, 0.4, 0) candidates: 27 training rows: 1600 training accuracy: 0.7931 test accuracy: 0.8067 test base class ratio: 0.8067 gain over base class ratio: 0.0000 test prediction distribution: {0: 600, 1: 0} ``` The number the pipeline produces is **0.8067**, and read on its own it looks good. The two numbers next to it make it unreadable. The label distribution is **1752 to 448**, that is, the base class ratio is **0.7964**; in the test set this ratio is **0.8067**. The accuracy of the model is **exactly** the base class ratio, and the gain over the base class ratio is **0.0000**. The last line writes the reason: the model says 0 to **600** of the 600 test rows. It counts no row as "high", so its accuracy is nothing but the share of the majority class. The resolution of the test set is **0.0017**; the gain is not even below this smallest step, it is exactly **zero**. If the evaluation step of a training pipeline prints a single accuracy, that pipeline is not measuring. #### Re-Running End to End The source is not something fixed. In the period `2026-07` the readings of 40 subscribers **arrive late** and are backfilled afterwards. The fourth block first runs with the incomplete source, and then resumes the pipeline from different points with the backfilled source. The `stale step` count is the number of steps that come from the stored old output. ```python LATE = [k for k in SOURCE if not (k["period"] == "2026-07" and k["subscriber"] <= "A040")] splits_late, trace_late, inter_late, _ = run_pipeline(LATE) model_late, test_late, _ = assess(splits_late) print("late-arriving source:", len(LATE), " consumption output:", trace_late[2][2], " training rows:", model_late["train_rows"], f" test accuracy: {test_late:.4f}") print() print(f"{'stale step':>11}{'steps rerun':>14}{'processed rows':>15}" f"{'training rows':>14}{'training accuracy':>18}{'test accuracy':>15}") for p in range(7): if p == 6: parts, processed = inter_late["split"], 0 else: parts, _, _, processed = run_pipeline(SOURCE, start=p, stored=inter_late) m, s, _ = assess(parts) print(f"{p:>11}{6 - p:>14}{processed:>15}{m['train_rows']:>14}" f"{m['train_accuracy']:>18.4f}{s:>15.4f}") ``` ``` late-arriving source: 2360 consumption output: 2120 training rows: 1520 test accuracy: 0.8067 stale step steps rerun processed rows training rows training accuracy test accuracy 0 6 13800 1600 0.7931 0.8067 1 5 11080 1520 0.7914 0.8067 2 4 8720 1520 0.7914 0.8067 3 3 6360 1520 0.7914 0.8067 4 2 4240 1520 0.7914 0.8067 5 1 2120 1520 0.7914 0.8067 6 0 0 1520 0.7914 0.8067 ``` The 40 late-arriving rows take **80 rows** away from the output of the pipeline: they drop themselves, and 40 rows of the next period drop because they cannot find their previous period. The training set is built with **1520** rows instead of 1600. The table says a single thing. After the backfill, the only setup that raises the training rows to **1600** is the pipeline whose stale step is zero, that is, the pipeline that **re-runs end to end from the source**. It is enough for even a single step to come from the stored old output: even if the pipeline truly re-runs the remaining five steps, the training rows stay at **1520**, because the lost rows were already missing from the output of the first step. A partial run saves by bringing the processed rows down from 13800 to 11080, and in return it **never sees the corrected source**. The reported number is **0.8067** in all seven setups. No one who looks at the test accuracy can tell a model trained with 1600 rows from a model trained with 1520 rows. The only place that shows the difference is the training accuracy, and there the difference is **0.0017**. The rule of the course finds its full counterpart here: **a number that cannot be reproduced is not a measurement but a memory**, and a pipeline's being runnable end to end means that the source the number came from is **determinable**. The cost of an end-to-end run is also in the table: every run reprocesses **13800 rows**; in the fully stale setup this number is **0**. Pinning is not free. #### Summary - The training pipeline branches off not from the **fifth** stage of the data pipeline but from the **third**: the model is attached not to the 5 serving values but to the 2200 consumption rows. - When the input and the output of the six steps are counted separately, the consumption step loses **200 rows**, but the step's own counter reads **0**; the counter counts only the unexpected drop. - The feature step leaves **200 rows history-less** without dropping any row; a step whose dropped rows are zero can still corrupt the data. - The label distribution is **1752 to 448**, the base class ratio is **0.7964**, and in the test set it is **0.8067**. The model says 0 to 600 of the 600 rows; its accuracy is **0.8067** and its gain over the base class ratio is **0.0000**. - The 40 late-arriving rows take **80 rows** away from the training set; after the backfill, the only setup that returns to 1600 rows is the end-to-end run whose stale step is **zero**, and its cost is **13800 processed rows** per run. - All seven setups report the same **0.8067**; the test accuracy cannot tell training with 1520 rows from training with 1600 rows. #### Next Step Once the pipeline of this lesson was set up, it always ran the same way, and every time it ran it produced a model. Whether the produced model **would go to release** was never asked. The model of 1520 rows trained with the missing late-arriving rows and the backfilled model of 1600 rows could have gone to release with equal ease, because both reported the same number. The next lesson puts **gates** between the run and the release: a code gate, a data gate and a model gate. The measurement is how the six corruption classes defined in the Introduction to Data Engineering course are distributed over these three gates, and which class is the one **no gate stops**. ### Continuous Integration for Machine Learning Source: https://academia.sh/en/courses/mlops-fundamentals/continuous-integration-for-machine-learning Counting which corruption each of the code, data and model gates placed between the run and the release stops: of the six corruption classes the data gate stops four and the model gate stops three, the code gate stops none but catches a feature code change at the first step without looking at production data at all; the unit change stops only at the model gate, by dropping the training accuracy from 0.7931 to 0.5919; the late-arriving row stops at no gate because its deviation in training rows is exactly 0.0500 and the tolerance is 0.0500 with a strict comparison; two of the four runs the data gate stopped would have produced exactly the same model as the reference, and in eight of eight runs the test accuracy of 0.8067 cannot rise above the base class ratio. The pipeline of the previous lesson produced a model every time it ran, and whether the produced model **would go to release** was never asked. The model trained with 1520 rows because of the missing late-arriving rows and the model trained with the backfilled 1600 rows both reported the same **0.8067**; both could have gone to release with equal ease. This lesson places **gates** between the run and the release. A gate is a check that stops a run before it reaches release. There are three, and each looks at a different thing: the **code gate** looks at the code that was written, the **data gate** at the input of the pipeline, the **model gate** at the numbers of the trained model. The question itself is one sentence: how do the six corruption classes defined in the Introduction to Data Engineering course distribute over these three gates, and which class is the one **no gate stops**. - **TD11.** The source, the pipeline, the model and the gates are **fictional**; no real machine learning library or continuous integration product is called. The seed is **20260218**. - **TD12.** The procedure of continuous integration itself, its triggers, the ordering of its stages and its deployment forms were measured in the Continuous Integration and Delivery course of the DevOps and Platform Engineering curriculum, and pipeline orchestration in the Data Pipelines and Orchestration course. They are **not repeated**; this lesson adds only the **data and model gates**. - **TD13.** None of the six corruption classes is an attack; all are accidents on the producer side and each corrupts **a single period** (`2026-07`). - **TD14.** The gates run in order: code **1**, data **2**, model **3**. The number of the gate that first stops an event also says how much the corruption has dirtied the pipeline. - **TD15.** The code gate **never sees production data**; it runs on a fixed input of three rows. - **TD16.** The **reference** that the model gate compares against is the model run with the intact source; the tolerances are **0.0500** for training rows and **0.0100** for training accuracy. - **TD17.** The label distribution is **imbalanced**; the base class ratio is written in this lesson too. - **TD18.** The baseline is the **setup without gates**: there is no gate, every run goes to release. - **TD19.** Resolution: the test set is **600 rows**, the step of accuracy is **0.0017**; the source is **2400 rows**, the step of the row ratio is **0.0004**. - **TD20.** Time is not written as raw milliseconds; the measures are **runs**, **rows** and **steps**. #### The Three Gates The first block sets up the source, the six corruption classes, the first three stages of the pipeline and the model. None of it is specific to this lesson. ```python # FICTIONAL. Source, pipeline, model and gates are defined inside the lesson; no real # machine learning library or continuous integration product is called. SEED, M32 = 20260218, 0xFFFFFFFF def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value REGION = ["east", "west", "north", "south", "central"] TYPE = ["residential", "commercial"] PERIODS = [f"2026-{a:02d}" for a in range(1, 13)] SUBSCRIBERS = {} for i in range(200): r = rng(SEED + 37 * i) SUBSCRIBERS[f"A{i + 1:03d}"] = {"region": REGION[int(r() * 5)], "type": TYPE[int(r() * 2)], "start": 10000 + int(r() * 60000)} def generate_source(): records = [] for i, (sub, a) in enumerate(SUBSCRIBERS.items()): e = a["start"] for p in PERIODS: r = rng(SEED + 101 * i + 7 * PERIODS.index(p)) e += 4 + int(r() * 46) records.append({"subscriber": sub, "period": p, "region": a["region"], "type": a["type"], "index": e, "unit": "m3"}) return records # ---- Six corruption classes. None of them is an attack , all are producer-side accidents. CORRUPTIONS = ["late_arrival", "duplicated", "schema_changed", "unit_changed", "missing_partition", "type_drift"] def corrupt(records, cls, period="2026-07"): """Corrupts a single period , the remaining eleven periods stay intact.""" out = [] for k in records: if k["period"] != period: out.append(k) elif cls == "late_arrival": if k["subscriber"] > "A040": # 40 subscribers never arrived this period out.append(k) elif cls == "duplicated": out.append(k) if k["subscriber"] <= "A060": # 60 records arrived twice out.append(dict(k)) elif cls == "schema_changed": y = dict(k) y["index_value"] = y.pop("index") # field name changed out.append(y) elif cls == "unit_changed": out.append(dict(k, index=k["index"] * 1000, unit="m3")) # label did not change elif cls == "missing_partition": if k["region"] != "north": # one region's partition was never written out.append(k) elif cls == "type_drift": out.append(dict(k, index=str(k["index"]))) # number turned into text return out def s1_screen(records): REQUIRED = ("subscriber", "period", "region", "index", "unit") keep = [k for k in records if all(a in k for a in REQUIRED)] return keep, len(records) - len(keep) def s2_dedupe(records): seen, keep = set(), [] for k in records: key = (k["subscriber"], k["period"]) if key in seen: continue seen.add(key) keep.append(k) return keep, len(records) - len(keep) def s3_consumption(records): table = {(k["subscriber"], k["period"]): k for k in records} out, dropped = [], 0 for k in records: i = PERIODS.index(k["period"]) if i == 0: continue prev = table.get((k["subscriber"], PERIODS[i - 1])) if prev is None: dropped += 1 continue try: f = int(k["index"]) - int(prev["index"]) except (TypeError, ValueError): dropped += 1 continue out.append(dict(k, m3=f)) return out, dropped TRAIN, TEST, THRESHOLD = PERIODS[1:9], PERIODS[9:], 40 def label(k): return 1 if k["m3"] > THRESHOLD else 0 def features(records, window=None, digits=4): ordered = sorted(records, key=lambda k: PERIODS.index(k["period"])) sub, reg = {}, {} for k in ordered: sub.setdefault(k["subscriber"], []).append((PERIODS.index(k["period"]), k["m3"])) reg.setdefault(k["region"], []).append((PERIODS.index(k["period"]), k["m3"])) def mean(series, i): chosen = [v for d, v in series if (d < i and (window is None or d >= i - window))] return round(sum(chosen) / len(chosen), digits) if chosen else 0.0 return {(k["subscriber"], k["period"]): { "subscriber_mean": mean(sub[k["subscriber"]], PERIODS.index(k["period"])), "region_mean": mean(reg[k["region"]], PERIODS.index(k["period"])), "commercial": 1 if k["type"] == "commercial" else 0} for k in records} GRID = [(a, b, c) for a in (0.6, 0.8, 1.0) for b in (0.0, 0.2, 0.4) for c in (0, 4, 8)] def score(o, w): return w[0] * o["subscriber_mean"] + w[1] * o["region_mean"] + w[2] * o["commercial"] def train(records, feats): subset = [k for k in records if k["period"] in TRAIN] best, best_w = -1.0, None for w in GRID: d = sum(1 for k in subset if (1 if score(feats[(k["subscriber"], k["period"])], w) > THRESHOLD else 0) == label(k)) if d / len(subset) > best: best, best_w = d / len(subset), w return {"weights": best_w, "train_accuracy": round(best, 4), "train_rows": len(subset)} def evaluate(records, model, feats): subset = [k for k in records if k["period"] in TEST] d = sum(1 for k in subset if (1 if score(feats[(k["subscriber"], k["period"])], model["weights"]) > THRESHOLD else 0) == label(k)) return round(d / len(subset), 4), round(sum(1 for k in subset if label(k) == 0) / len(subset), 4) def run(records, window=None): v, _ = s1_screen(records) v, _ = s2_dedupe(v) v, _ = s3_consumption(v) feats = features(v, window) m = train(v, feats) accuracy, baseline = evaluate(v, m, feats) return m, accuracy, baseline, len(v) ``` The second block defines the three gates and passes eight events through the gates in order. Seven events are on the data side: the intact run and the six corruption classes. The eighth is on the code side: bringing the feature window down from the full history to a single period. ```python # ---- GATE 1: code. Fixed mini input , never sees production data. MINI = [{"subscriber": "A001", "period": d, "region": "east", "type": "commercial", "m3": v} for d, v in (("2026-02", 30), ("2026-03", 50), ("2026-04", 10))] EXPECTED = ({"subscriber_mean": 40.0, "region_mean": 40.0, "commercial": 1}, [0, 1, 0]) def code_gate(window=None): feats = features(MINI, window) return (feats[("A001", "2026-04")], [label(k) for k in MINI]) != EXPECTED # ---- GATE 2: data. Five checks on the input of the pipeline. def data_gate(records, expected=2400, tolerance=0.05): FIELDS = ("subscriber", "period", "region", "index", "unit") keys = [(k["subscriber"], k["period"]) for k in records] findings = [] if any(any(a not in k for a in FIELDS) for k in records): findings.append("schema") if any(not isinstance(k.get("index"), int) for k in records): findings.append("type") if len(keys) != len(set(keys)): findings.append("uniqueness") if abs(len(records) - expected) / expected > tolerance: findings.append("row_count") if set((k["region"], k["period"]) for k in records) != set((b, d) for b in REGION for d in PERIODS): findings.append("partition_period") return findings # ---- GATE 3: model. The trained model is compared with the reference in release. def model_gate(m, accuracy, baseline, reference, tolerance=0.05, drop=0.0100): findings = [] if abs(m["train_rows"] - reference["train_rows"]) / reference["train_rows"] > tolerance: findings.append("train_rows") if reference["train_accuracy"] - m["train_accuracy"] > drop: findings.append("train_drop") if m["weights"] != reference["weights"]: findings.append("weight_shift") return findings, accuracy <= baseline SOURCE = generate_source() REFERENCE = run(SOURCE)[0] EVENTS = [("intact", SOURCE, None)] + [(s, corrupt(SOURCE, s), None) for s in CORRUPTIONS] \ + [("feature_code", SOURCE, 1)] RESULTS = {} print(f"{'event':<18}{'code (1)':>9}{'data (2)':>26}{'model (3)':>18}{'first gate':>11}") for name, recs, window in EVENTS: code_stop = code_gate(window) data_findings = data_gate(recs) m, accuracy, baseline, n_rows = run(recs, window) model_findings, below_baseline = model_gate(m, accuracy, baseline, REFERENCE) first = 1 if code_stop else (2 if data_findings else (3 if model_findings else 0)) RESULTS[name] = (m, accuracy, baseline, n_rows, code_stop, data_findings, model_findings, below_baseline, first) print(f"{name:<18}{('stopped' if code_stop else '-'):>9}{(','.join(data_findings) or '-'):>26}" f"{(','.join(model_findings) or '-'):>18}{(first or '-'):>11}") print("events no gate stopped:", [a for a in RESULTS if RESULTS[a][8] == 0]) print("runs whose test accuracy cannot exceed the base class ratio:", sum(1 for a in RESULTS if RESULTS[a][7]), "/", len(EVENTS)) ``` ``` event code (1) data (2) model (3) first gate intact - - - - late_arrival - - - - duplicated - uniqueness - 2 schema_changed - schema,type train_rows 2 unit_changed - - train_drop 3 missing_partition - partition_period train_rows 2 type_drift - type - 2 feature_code stopped - weight_shift 1 events no gate stopped: ['intact', 'late_arrival'] runs whose test accuracy cannot exceed the base class ratio: 8 / 8 ``` #### Which Gate Stops Which Corruption The distribution is not even. The data gate stops **four** of the six classes: the duplicated record with the uniqueness check, the field name change with the schema and type checks, the missing partition with the period-aware completeness check, the number turned into text with the type check. The model gate stops **three**, and **the unit change stops only there**: while the input of the pipeline looks flawless, the training accuracy falls from 0.7931 to **0.5919**, and the drop of **0.2012** is twenty times the tolerance of 0.0100. The code gate stops **none** of the six classes, and this is not a defect. The code gate does not see production data; the only thing it sees is a fixed input of three rows. In return, it catches bringing the feature window down to a single period **at the first step**, without touching the source at all. The model gate catches the same change too, but at the third step: it fires a weight shift because the chosen weights come out as `(0.6, 0.0, 0)` instead of `(0.6, 0.4, 0)`. Both gates are right; the difference between them is that one stops the corruption before the pipeline runs at all, and the other after 2200 rows have been processed. The last line is the most uncomfortable number of the lesson. In **eight** of the eight runs the test accuracy **cannot exceed** the base class ratio. The label distribution is imbalanced, the base class ratio is **0.7964**, and in the test set it is **0.8067**; the model produces exactly that number. A model gate that makes the test accuracy the release criterion would let no model through in this setup. This is not a false alarm: it is the evidence, invisible when accuracy is read alone, that the model never rose above the base class ratio. #### The Class No Gate Stops The table leaves two events unstopped. One is the intact run, and it is expected not to stop. The other is the **late-arriving** row, and that is a gap. The third block measures where the gap comes from. ```python print(f"{'event':<18}{'source':>8}{'pipeline':>10}{'train rows':>12}{'train acc.':>12}{'test acc.':>11}") for name, recs, window in EVENTS: m, accuracy, baseline, n_rows, *_ = RESULTS[name] print(f"{name:<18}{len(recs):>8}{n_rows:>10}{m['train_rows']:>12}" f"{m['train_accuracy']:>12.4f}{accuracy:>11.4f}") print() late_model, late_accuracy, late_baseline, *_ = RESULTS["late_arrival"] print(f"late_arrival train rows deviation: {abs(late_model['train_rows'] - 1600) / 1600:.4f}" f" gate tolerance: 0.0500 comparison: strict") print("late_arrival if the tolerance were 0.0400:", model_gate(late_model, late_accuracy, late_baseline, REFERENCE, tolerance=0.04)[0], " intact run with the same tolerance:", model_gate(RESULTS["intact"][0], RESULTS["intact"][1], RESULTS["intact"][2], REFERENCE, tolerance=0.04)[0] or "did not stop") print("runs the data gate stopped but that gave exactly the same model as the reference:", [a for a in RESULTS if a != "intact" and RESULTS[a][5] and RESULTS[a][0] == REFERENCE]) ``` ``` event source pipeline train rows train acc. test acc. intact 2400 2200 1600 0.7931 0.8067 late_arrival 2360 2120 1520 0.7914 0.8067 duplicated 2460 2200 1600 0.7931 0.8067 schema_changed 2400 1800 1200 0.7892 0.8067 unit_changed 2400 2200 1600 0.5919 0.8067 missing_partition 2354 2108 1508 0.7924 0.8067 type_drift 2400 2200 1600 0.7931 0.8067 feature_code 2400 2200 1600 0.7925 0.8067 late_arrival train rows deviation: 0.0500 gate tolerance: 0.0500 comparison: strict late_arrival if the tolerance were 0.0400: ['train_rows'] intact run with the same tolerance: did not stop runs the data gate stopped but that gave exactly the same model as the reference: ['duplicated', 'type_drift'] ``` The late-arriving row squeaks past both gates. In the data gate the source is 2360 instead of 2400; the deviation is **0.0167** and the tolerance is 0.0500. In the model gate the training rows are 1520 instead of 1600; the deviation is **exactly 0.0500** and the tolerance is also 0.0500. What lets it through is not a measurement but a **comparison sign**: because a strict inequality is used, a deviation sitting on the boundary counts as inside. When the tolerance is lowered to 0.0400, the late-arriving row stops and the intact run does not. This is the fragility every threshold-based gate carries, and it cannot be seen until the number is written. That the measured false alarm comes out zero is a property of **this set**, not of the gate: in the fictional source every period carries exactly 200 rows, so the natural variation in the row count is zero. In a source with natural variation, the same tightening would stop intact runs too. The cost of the gates is on the last line. The data gate stops four runs, but **two** of them would have produced **exactly the same** model as the reference: the duplicated record is absorbed in the deduplication stage, the number turned into text in the type conversion, and in both the weights, the training rows and the training accuracy do not change. In the setup without gates, all eight of the eight runs go to release and six are corrupted; in the setup with gates, six runs stop, two pass, and one of those that pass is still corrupted. **One third** of the runs stopped were stopped without gaining anything. The rule of the course applies to the gate too: **a number that cannot be reproduced is not a measurement but a memory.** A gate being green is not a measurement either; the late-arriving row passes all three gates green and carries the model trained with 1520 rows to release in place of the model of 1600 rows. #### Summary - The gates run in order: code **1**, data **2**, model **3**. The number of the gate that first stops an event also says how much the corruption has dirtied the pipeline. - The data gate stops **four** of the six corruption classes and the model gate **three**; the **unit change stops only at the model gate**, by dropping the training accuracy from 0.7931 to **0.5919**. - The code gate stops **none** of the six classes, because it does not see production data; in return it catches the feature code change **at the first step**, before the pipeline runs at all. - **No gate** stops the late-arriving row: the source deviation is 0.0167, the deviation in training rows is **exactly 0.0500** and the tolerance is 0.0500 with a strict comparison. With a tolerance of 0.0400 it would stop, and the intact run would not. - The label distribution is imbalanced, the base class ratio is **0.7964**; in **eight** of the eight runs the test accuracy cannot exceed the base class ratio with 0.8067. - **Two** of the four runs the data gate stopped would have produced exactly the same model as the reference; the cost of adding a gate is that it also stops where it gains nothing. #### Next Step A run that passes the gates now counts as having produced a model, but what was produced is still not written down. Up to here the model was a dictionary of three weights and a threshold, and the calling side used it by **guessing** the names, types and order of the fields. The next lesson **packages** the model: next to the weights it puts an **interface contract** that writes which fields are expected with which type and which field comes back in return. The measure is the caller that breaks when the contract is not written: how many calls drop when a field name or type changes, how many give a wrong answer without dropping at all, and how many of these the contract catches **before release**. ### Model Packaging Source: https://academia.sh/en/courses/mlops-fundamentals/model-packaging Turning a trained model into a single unit with its weights, threshold and interface declaration, and counting the callers that break when no contract is written: when a field name changes in the 1800 calls of three callers the dropped calls rise from 600 to 1200, when the threshold field falls out of the package the number of dropped calls does not change but all 1200 of the 1200 calls that pass change their answer; a contract that declares only the input fields catches two of the three versions while a contract that also declares the output and the parameter catches all three before release, and for that it reads 3 sample records instead of 1800 calls; the label distribution is 1752 to 448, the base class ratio is 0.7964, and the test accuracy of the model, 0.8067, is exactly the base ratio in the test set. The run that passed the gates of the previous lesson now counted as having produced a model, but what was produced was still not written down. The model was a dictionary of three weights and a threshold; the calling side used it by **guessing** the names, types and order of the fields. This lesson turns that dictionary into a **package**. Packaging means putting next to the weights everything the caller has to know: which fields are expected with which type, which parameter is carried in the package, which field comes back in return. This declaration is called the **interface contract**. The question of the lesson is who breaks when no contract is written: when a field name or type changes, how many calls drop, how many give a wrong answer without dropping at all, and how many of these the contract catches **before release**. - **TD21.** The source, the pipeline, the model and the package are **fictional**. **No packaging format, container runtime, serving framework or machine learning library name is written.** The seed is **20260218**. - **TD22.** The **data** schema between producer and consumer and whom a schema change breaks were measured in the Introduction to Data Engineering course and are **not repeated**. The contract here is the contract not of the data but of the **interface of the model**. - **TD23.** The package carries four things: the **weights**, the **threshold**, the **declaration of the input fields** and the **name of the output field**. The runtime, the source separation and the form of deployment are outside this lesson. - **TD24.** A call counts as **dropped** if a field the package expects is missing from the record or if the type of the field does not enter arithmetic. Every call whose field is present and whose type matches **passes**. - **TD25.** There are three callers and all three send the same **600** test rows; **1800** calls in total. - **TD26.** The callers differ at three points: one **reads the field names from the declaration of the package**, one has **hard-coded the names in its own code**, one has hard-coded the names and carries one field as **text**. - **TD27.** The contract is the published interface; both the package and the caller conform to **it**. Two forms of it are compared: one that covers only the input and one that also covers the output and the parameter. - **TD28.** The label distribution is **imbalanced**; the base class ratio is written in this lesson too. - **TD29.** The baseline is **packaging without a contract**: no declaration, no pre-release validation. - **TD30.** Resolution: the test set is **600 rows**, the step of accuracy is **0.0017**; the step of the call count is **1 call**. - **TD31.** Time is not written as raw milliseconds; the measures are **calls**, **records** and **declarations**. #### What Is Inside the Package The first block sets up the source, the first three stages of the pipeline and the model; none of it is specific to this lesson. ```python # FICTIONAL. Source, pipeline, model and package are defined inside the lesson; no real # machine learning library, packaging format or runtime is called. SEED, M32 = 20260218, 0xFFFFFFFF def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value REGION = ["east", "west", "north", "south", "central"] TYPE = ["residential", "commercial"] PERIODS = [f"2026-{a:02d}" for a in range(1, 13)] SUBSCRIBERS = {} for i in range(200): r = rng(SEED + 37 * i) SUBSCRIBERS[f"A{i + 1:03d}"] = {"region": REGION[int(r() * 5)], "type": TYPE[int(r() * 2)], "start": 10000 + int(r() * 60000)} def generate_source(): records = [] for i, (sub, a) in enumerate(SUBSCRIBERS.items()): e = a["start"] for p in PERIODS: r = rng(SEED + 101 * i + 7 * PERIODS.index(p)) e += 4 + int(r() * 46) records.append({"subscriber": sub, "period": p, "region": a["region"], "type": a["type"], "index": e, "unit": "m3"}) return records def s1_screen(records): REQUIRED = ("subscriber", "period", "region", "index", "unit") keep = [k for k in records if all(a in k for a in REQUIRED)] return keep, len(records) - len(keep) def s2_dedupe(records): seen, keep = set(), [] for k in records: key = (k["subscriber"], k["period"]) if key in seen: continue seen.add(key) keep.append(k) return keep, len(records) - len(keep) def s3_consumption(records): table = {(k["subscriber"], k["period"]): k for k in records} out, dropped = [], 0 for k in records: i = PERIODS.index(k["period"]) if i == 0: continue prev = table.get((k["subscriber"], PERIODS[i - 1])) if prev is None: dropped += 1 continue try: f = int(k["index"]) - int(prev["index"]) except (TypeError, ValueError): dropped += 1 continue out.append(dict(k, m3=f)) return out, dropped TRAIN, TEST, THRESHOLD = PERIODS[1:9], PERIODS[9:], 40 def label(k): return 1 if k["m3"] > THRESHOLD else 0 def features(records, window=None, digits=4): ordered = sorted(records, key=lambda k: PERIODS.index(k["period"])) sub, reg = {}, {} for k in ordered: sub.setdefault(k["subscriber"], []).append((PERIODS.index(k["period"]), k["m3"])) reg.setdefault(k["region"], []).append((PERIODS.index(k["period"]), k["m3"])) def mean(series, i): chosen = [v for d, v in series if (d < i and (window is None or d >= i - window))] return round(sum(chosen) / len(chosen), digits) if chosen else 0.0 return {(k["subscriber"], k["period"]): { "subscriber_mean": mean(sub[k["subscriber"]], PERIODS.index(k["period"])), "region_mean": mean(reg[k["region"]], PERIODS.index(k["period"])), "commercial": 1 if k["type"] == "commercial" else 0} for k in records} GRID = [(a, b, c) for a in (0.6, 0.8, 1.0) for b in (0.0, 0.2, 0.4) for c in (0, 4, 8)] def score(o, w): return w[0] * o["subscriber_mean"] + w[1] * o["region_mean"] + w[2] * o["commercial"] def train(records, feats): subset = [k for k in records if k["period"] in TRAIN] best, best_w = -1.0, None for w in GRID: d = sum(1 for k in subset if (1 if score(feats[(k["subscriber"], k["period"])], w) > THRESHOLD else 0) == label(k)) if d / len(subset) > best: best, best_w = d / len(subset), w return {"weights": best_w, "train_accuracy": round(best, 4), "train_rows": len(subset)} def evaluate(records, model, feats): subset = [k for k in records if k["period"] in TEST] d = sum(1 for k in subset if (1 if score(feats[(k["subscriber"], k["period"])], model["weights"]) > THRESHOLD else 0) == label(k)) return round(d / len(subset), 4) ``` The second block packages the model and sets up its three versions. `v1` is the package in release, `v2` is a package that changed the name of one input field, and `v3` is a package that **does not carry** the threshold field at all. The three callers send the same 600 rows and the dropped calls are counted. ```python # ---- Packaging: weights , threshold and interface declaration in a single unit SOURCE = generate_source() V, _ = s1_screen(SOURCE) V, _ = s2_dedupe(V) ROWS, _ = s3_consumption(V) FEATS = features(ROWS) MODEL = train(ROWS, FEATS) TEST_SET = [k for k in ROWS if k["period"] in TEST] DISTRIBUTION = {e: sum(1 for k in ROWS if label(k) == e) for e in (0, 1)} TEST_BASE = round(sum(1 for k in TEST_SET if label(k) == 0) / len(TEST_SET), 4) print("label distribution:", DISTRIBUTION, " base class ratio:", round(DISTRIBUTION[0] / len(ROWS), 4)) print("weights:", MODEL["weights"], " training rows:", MODEL["train_rows"], " test rows:", len(TEST_SET), f" test accuracy: {evaluate(ROWS, MODEL, FEATS):.4f} test base class ratio: {TEST_BASE:.4f}") PACKAGE_V1 = {"weights": MODEL["weights"], "threshold": THRESHOLD, "output": "high", "input": ("subscriber_mean", "region_mean", "commercial")} PACKAGE_V2 = dict(PACKAGE_V1, input=("subscriber_average", "region_mean", "commercial")) PACKAGE_V3 = {a: v for a, v in PACKAGE_V1.items() if a != "threshold"} VERSIONS = [("v1 reference", PACKAGE_V1), ("v2 field name", PACKAGE_V2), ("v3 no threshold", PACKAGE_V3)] DEFAULT_THRESHOLD = 0 def call(package, record): w, g = package["weights"], package["input"] o = {a: record[a] for a in g} # missing field -> KeyError p = w[0] * o[g[0]] + w[1] * o[g[1]] + w[2] * o[g[2]] # type mismatch -> TypeError return {package["output"]: 1 if p > package.get("threshold", DEFAULT_THRESHOLD) else 0} def c_adaptive(package, o): """Reads the field names from the declaration of the package.""" return dict(zip(package["input"], (o["subscriber_mean"], o["region_mean"], o["commercial"]))) def c_fixed_name(package, o): """Has hard-coded the field names in its own code.""" return {"subscriber_mean": o["subscriber_mean"], "region_mean": o["region_mean"], "commercial": o["commercial"]} def c_fixed_type(package, o): """Has hard-coded the names and carries the commercial field as text.""" return dict(c_fixed_name(package, o), commercial=str(o["commercial"])) CALLERS = [("adaptive", c_adaptive), ("fixed_name", c_fixed_name), ("fixed_type", c_fixed_type)] print(f"{'version':<16}{'calls':>7}{'dropped':>9}{'passed':>7}{'high':>8}" + "".join(f"{a:>12}" for a, _ in CALLERS)) for vname, pkg in VERSIONS: dropped, passed, high, broken = 0, 0, 0, [] for cname, cfn in CALLERS: d = 0 for k in TEST_SET: try: high += call(pkg, cfn(pkg, FEATS[(k["subscriber"], k["period"])]))[pkg["output"]] passed += 1 except (KeyError, TypeError): d += 1 dropped += d broken.append(d) print(f"{vname:<16}{len(TEST_SET) * 3:>7}{dropped:>9}{passed:>7}{high:>8}" + "".join(f"{x:>12}" for x in broken)) ``` ``` label distribution: {0: 1752, 1: 448} base class ratio: 0.7964 weights: (0.6, 0.4, 0) training rows: 1600 test rows: 600 test accuracy: 0.8067 test base class ratio: 0.8067 version calls dropped passed high adaptive fixed_name fixed_type v1 reference 1800 600 1200 0 0 0 600 v2 field name 1800 1200 600 0 0 600 600 v3 no threshold 1800 600 1200 1200 0 0 600 ``` #### The Caller That Breaks When No Contract Is Written The first two lines write the model itself and carry the warning repeated in every lesson of this course. The label distribution is **1752 to 448**, that is, the base class ratio is **0.7964**; in the test set this ratio is **0.8067** and the test accuracy of the model is also **0.8067**. Accuracy misleads when it is read alone: the packaged model produces, in the test set, **exactly** the share of the majority class. All the call counts below are calls of this model. The baseline row is **v1**, and even there **600** of the 1800 calls drop. What breaks them is not a version change but a **type mismatch** that has been there from the start: the `fixed_type` caller carries the `commercial` field as text and the score computation cannot enter arithmetic. In a setup without a contract this mismatch comes out only in production, call by call. The **v2** row gives the cost of the field name change. The package starts to expect `subscriber_average` instead of `subscriber_mean` and the dropped calls rise from 600 to **1200**. Two callers that have hard-coded the names break at once, because the record they send does not have the field the package is looking for. The only caller left standing is the one that **reads** the names from the declaration of the package, and all 600 of its calls pass. A package's declaring its field names and the callers' reading that declaration are two separate decisions; without the second the first is of no use. The **v3** row is the most dangerous one, and no observation that looks at the number of dropped calls can see it. The threshold field has been taken out of the package; when the caller cannot find the field it falls back on its own default, that is, on **0**. The number of dropped calls does not change: in v3 as in v1, **600** calls drop and **1200** pass. The only column that changes is the last. **All 1200** of the 1200 calls that pass return the answer "high", whereas the number of highs returned in v1 was **0**. The model is the same, the weights are the same, the caller is the same; **every answer has flipped because of a single missing parameter**. There is not a single row that lands in the error log. #### What the Contract Catches Before Release The third block writes the contract. The contract is the **published interface**: both the package and the caller conform to it, and the two look not at each other but at the contract. Two forms of it are compared: a contract that declares only the input fields, and a contract that also declares the output and the parameter. ```python # ---- Interface contract: the published interface. Both package and caller conform to it. TYPES = {"float": float, "integer": int} CONTRACT_INPUT = {"input": {"subscriber_mean": "float", "region_mean": "float", "commercial": "integer"}} CONTRACT_FULL = dict(CONTRACT_INPUT, output="high", parameters=("threshold",)) def validate(contract, package, samples): findings = [] if set(package["input"]) != set(contract["input"]): findings.append("package: input field name") if "output" in contract and package["output"] != contract["output"]: findings.append("package: output field name") for p in contract.get("parameters", ()): if p not in package: findings.append(f"package: missing parameter {p}") for name, sample in samples: for a, t in contract["input"].items(): if a not in sample: findings.append(f"{name}: missing field {a}") elif type(sample[a]) is not TYPES[t]: findings.append(f"{name}: {a} type") return findings SAMPLE = FEATS[(TEST_SET[0]["subscriber"], TEST_SET[0]["period"])] print(f"{'version':<16}{'input contract':>18}{'full contract':>14} findings of the full contract") for vname, pkg in VERSIONS: samples = [(cname, cfn(pkg, SAMPLE)) for cname, cfn in CALLERS] g = validate(CONTRACT_INPUT, pkg, samples) t = validate(CONTRACT_FULL, pkg, samples) print(f"{vname:<16}{len(g):>18}{len(t):>14} {t}") print("records read before release:", len(CALLERS), " calls in the setup without a contract:", len(TEST_SET) * 3, " declarations in the contract:", len(CONTRACT_FULL["input"]) + 1 + len(CONTRACT_FULL["parameters"])) ``` ``` version input contract full contract findings of the full contract v1 reference 1 1 ['fixed_type: commercial type'] v2 field name 3 3 ['package: input field name', 'adaptive: missing field subscriber_mean', 'fixed_type: commercial type'] v3 no threshold 1 2 ['package: missing parameter threshold', 'fixed_type: commercial type'] records read before release: 3 calls in the setup without a contract: 1800 declarations in the contract: 5 ``` In all three versions the number of dropped calls falls to **0**, because the mismatch is found before release and the version is not published. The measured difference is where the mismatch is seen: in the setup without a contract **among the 1800 calls**, in the setup with a contract **in 3 sample records**. The validation reads a single record per caller. The v2 row shows what the contract is, too. Of the three findings, one belongs to the package and two to the callers. One of them is surprising: the `adaptive` caller, which reads the declaration of the package, **never dropped** in production, yet it produces a record that is incompatible according to the contract. Both are right. That caller conforms to the package and does not conform to the contract; because the package deviated from the contract and the caller followed the deviation silently. The job of the contract is not to compare the package with the caller, but to compare **both of them with the published interface**. The v3 row writes that **the contract itself can be defective**. The contract that declares only the input fields produces a single finding in v3, and that finding is irrelevant: the type mismatch that the `fixed_type` caller has had from the start. It does not see the missing threshold field, because there is no such thing as a parameter in its contract. The contract that also declares the output and the parameter produces **two** findings in the same version, and **the first** is exactly the missing parameter. The input contract catches two of the three versions as well; **the only thing that catches the silent break that flips 1200 answers is the declaration of the parameter.** The cost of the contract can be counted too. Five declarations are written: three input fields, one output field, one parameter. A validation step is added before every release, and that step asks for one sample record per caller. A contract with an item left out gives an **unmeasured assurance** where it does not cover: the input contract looked green on v3. The rule of the course applies to the interface too: **a number that cannot be reproduced is not a measurement but a memory.** The answer a call returns cannot be reproduced if the interface that produced it is not written down; the 1200 answers of v3 are the proof. #### Summary - Packaging means putting next to the weights everything the caller has to know: the **weights, the threshold, the input declaration and the output field**. This declaration is the interface contract. - The label distribution is **1752 to 448**, the base class ratio is **0.7964**; the test accuracy of the packaged model, **0.8067**, is exactly the base ratio in the test set. - In the setup without a contract, even in the baseline version **600** of the 1800 calls drop; when the field name changes the dropped calls rise to **1200**, and the only caller left standing is the one that reads the names from the declaration of the package. - When the threshold field falls out of the package, **the number of dropped calls does not change**: as in v1, 600 calls drop, 1200 calls pass and **all 1200 of them** change their answer; the number of highs, which was 0 in v1, becomes **1200** in v3. - In the setup with a contract, the dropped calls are **0** in all three versions; the mismatch is seen in **3 sample records** instead of 1800 calls, and its cost is **5 declarations** and one validation step. - The contract that covers only the input catches **two** of the three versions; the only item that catches the silent break is the **declaration of the parameter**. #### Next Step In these three lessons the model was attached to a pipeline, passed through gates and packaged with an interface. The assumption common to all three can be written in one sentence: the caller calls whenever it wants and the answer comes at once. The 1800 calls of this lesson were counted the same way, as if all of them ran at the same time and at the same cost. Yet running a packaged model **once at the end of the night for twelve periods** and running it **for a single row** in the middle of a request are not the same thing. The next lesson separates the serving patterns: batch, online and streaming inference. The measure is the **latency class**, the **staleness** and the **cost per prediction** of the three patterns. ### Serving Patterns Source: https://academia.sh/en/courses/mlops-fundamentals/serving-patterns Running the same package in three placements, batch, online and streaming, and counting the latency class, the freshness and the cost per prediction of the three separately: all three patterns give the same answer to the same 600 requests and the accuracy is 0.8067 in all three, so an observation that looks only at the output cannot tell which pattern ran. The difference is in three columns: the work done at request time is a lookup in the batch and streaming patterns and a scan of 352 to 550 rows in the online pattern; the age of the answer is one period in the batch pattern and zero in the others; the rows read per prediction are 11.00, 463.87 and 3.67. The staleness sweep gives 0 changed predictions up to six periods, and the reason is not the pattern but that the decision score stays at least 7.6468 away from the threshold. The cost is a denominator decision: the batch and online patterns are equal at 4.74 predictions used, and which task rests on which pattern is read from that point. The previous lesson packaged the model: the runtime was pinned, the interface contract was written, and the number of calls that dropped when no contract was written was counted. What the package accepts and what it returns is now written down. The only thing not written is **when** this package runs. The same package can run once at the start of a period and write two hundred predictions to a table, can run once whenever a request arrives, or can refresh itself whenever a new reading lands. The weights are the same in all three. These three placements are called a **serving pattern**. The difference between them is not in the model but in the moment and the scope at which **inference** is done. This lesson runs the three side by side on the same workload and, instead of a single accuracy number, writes three numbers separately: the work done at the moment the request arrives, the age of the newest reading the answer rests on, and the **cost per prediction**. - **TD41.** The source, the pipeline and the model are **fictional** and are generated inside the lesson. No serving framework, inference service, accelerator or cloud service is called; all three patterns are written here. The seed is 20260218. - **TD42.** The label is imbalanced: the distribution is `{0: 1752, 1: 448}` and the **base class ratio is 0.7964**. No accuracy is read without writing how far above this ratio it stands. The test set is 600 rows, so the smallest step of accuracy is 0.0017. - **TD43.** In this course the training pipeline is handled on a **single machine**; partitioning and shuffling were measured in the Distributed Data Processing course and are not repeated. - **TD44.** The cost is counted as **rows read**; no raw milliseconds are written. The latency is written as a **class**: `lookup` is the reading of a single stored record at request time; `scan` is the reading of the history of the subscriber and of its region at request time. - **TD45.** Freshness is counted in **periods**: the age of the answer is the number of periods that have to pass before the newest reading is reflected in the prediction. - **TD46.** The serving horizon is the test periods, `2026-10`, `2026-11`, `2026-12`, and in each period one request arrives for each of the two hundred subscribers. 600 requests in total. - **TD47.** The batch pattern runs once per period. This is a **choice** and is swept below. #### Three Patterns Side by Side The first block sets up the source, the three stages of the pipeline up to consumption, the feature path and the fictional model; then it runs the three patterns with the same weights. The features are computed **once** and are independent of the weights. ```python # FICTIONAL. The meter readings of the measurement network, the first three stages of the # pipeline, the feature path and the threshold model with three weights are generated inside # the lesson; no learning library, serving framework or inference service is called. # Seed 20260218. SEED, M32 = 20260218, 0xFFFFFFFF def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value REGION = ["east", "west", "north", "south", "central"] TYPE = ["residential", "commercial"] PERIODS = [f"2026-{a:02d}" for a in range(1, 13)] SUBSCRIBERS = {} for i in range(200): r = rng(SEED + 37 * i) SUBSCRIBERS[f"A{i + 1:03d}"] = {"region": REGION[int(r() * 5)], "type": TYPE[int(r() * 2)], "start": 10000 + int(r() * 60000)} def generate_source(): records = [] for i, (sub, a) in enumerate(SUBSCRIBERS.items()): e = a["start"] for p in PERIODS: r = rng(SEED + 101 * i + 7 * PERIODS.index(p)) e += 4 + int(r() * 46) records.append({"subscriber": sub, "period": p, "region": a["region"], "type": a["type"], "index": e, "unit": "m3"}) return records def s3_consumption(records): table = {(k["subscriber"], k["period"]): k for k in records} out = [] for k in records: i = PERIODS.index(k["period"]) prev = table.get((k["subscriber"], PERIODS[i - 1])) if i else None if prev is not None: out.append(dict(k, m3=int(k["index"]) - int(prev["index"]))) return out TRAIN, TEST, THRESHOLD = PERIODS[1:9], PERIODS[9:], 40 def label(k): return 1 if k["m3"] > THRESHOLD else 0 def features(records, window=None, digits=4): """The training path sees the full history. Independent of the weights , computed once.""" ordered = sorted(records, key=lambda k: PERIODS.index(k["period"])) sub, reg = {}, {} for k in ordered: sub.setdefault(k["subscriber"], []).append((PERIODS.index(k["period"]), k["m3"])) reg.setdefault(k["region"], []).append((PERIODS.index(k["period"]), k["m3"])) def mean(series, i): chosen = [v for d, v in series if (d < i and (window is None or d >= i - window))] return round(sum(chosen) / len(chosen), digits) if chosen else 0.0 return {(k["subscriber"], k["period"]): { "subscriber_mean": mean(sub[k["subscriber"]], PERIODS.index(k["period"])), "region_mean": mean(reg[k["region"]], PERIODS.index(k["period"])), "commercial": 1 if k["type"] == "commercial" else 0} for k in records} GRID = [(a, b, c) for a in (0.6, 0.8, 1.0) for b in (0.0, 0.2, 0.4) for c in (0, 4, 8)] def score(o, w): return w[0] * o["subscriber_mean"] + w[1] * o["region_mean"] + w[2] * o["commercial"] def train(records, feats): subset = [k for k in records if k["period"] in TRAIN] best, best_w = -1.0, None for w in GRID: d = sum(1 for k in subset if (1 if score(feats[(k["subscriber"], k["period"])], w) > THRESHOLD else 0) == label(k)) if d / len(subset) > best: best, best_w = d / len(subset), w return {"weights": best_w, "train_rows": len(subset), "candidates": len(GRID)} ALL = s3_consumption(generate_source()) FEATS = features(ALL) MODEL = train(ALL, FEATS) W = MODEL["weights"] SERVING, REQUESTS = TEST, sorted(SUBSCRIBERS) def batch_run(records, period, w): """The whole history is scanned once , all predictions are produced and stored.""" feats = features(records) return ({k["subscriber"]: (1 if score(feats[(k["subscriber"], k["period"])], w) > THRESHOLD else 0) for k in records if k["period"] == period}, len(records)) def online_request(idx, sub, period, w): """At request time the history of the subscriber and of its region is scanned.""" by_sub, by_reg = idx i, reg = PERIODS.index(period), by_sub[sub][0]["region"] sub_vals = [k["m3"] for k in by_sub[sub] if PERIODS.index(k["period"]) < i] reg_vals = [k["m3"] for k in by_reg[reg] if PERIODS.index(k["period"]) < i] o = {"subscriber_mean": round(sum(sub_vals) / len(sub_vals), 4) if sub_vals else 0.0, "region_mean": round(sum(reg_vals) / len(reg_vals), 4) if reg_vals else 0.0, "commercial": 1 if by_sub[sub][0]["type"] == "commercial" else 0} return (1 if score(o, w) > THRESHOLD else 0), len(by_sub[sub]) + len(by_reg[reg]) def stream_run(records, w): """As each reading arrives the state is updated incrementally and the prediction is refreshed.""" sub_state, reg_state, pred, read, refreshes = {}, {}, {}, 0, 0 for k in sorted(records, key=lambda k: (PERIODS.index(k["period"]), k["subscriber"])): read += 1 sub, reg = k["subscriber"], k["region"] o = {"subscriber_mean": round(sub_state[sub][0] / sub_state[sub][1], 4) if sub in sub_state else 0.0, "region_mean": round(reg_state[reg][0] / reg_state[reg][1], 4) if reg in reg_state else 0.0, "commercial": 1 if k["type"] == "commercial" else 0} pred[(sub, k["period"])] = 1 if score(o, w) > THRESHOLD else 0 sub_state.setdefault(sub, [0, 0]); reg_state.setdefault(reg, [0, 0]) sub_state[sub][0] += k["m3"]; sub_state[sub][1] += 1 reg_state[reg][0] += k["m3"]; reg_state[reg][1] += 1 refreshes += 1 return pred, read, refreshes, len(sub_state) + len(reg_state) by_sub, by_reg = {}, {} for k in ALL: by_sub.setdefault(k["subscriber"], []).append(k) by_reg.setdefault(k["region"], []).append(k) IDX = (by_sub, by_reg) TRUTH = {(k["subscriber"], k["period"]): label(k) for k in ALL if k["period"] in SERVING} N = len(REQUESTS) * len(SERVING) def accuracy(t): return round(sum(1 for c in TRUTH if t[c] == TRUTH[c]) / N, 4) BATCH, read_b = {}, 0 for d in SERVING: t, o = batch_run(ALL, d, W) read_b += o for a in REQUESTS: BATCH[(a, d)] = t[a] ONLINE, read_o, min_read, max_read = {}, 0, 10 ** 9, 0 for d in SERVING: for a in REQUESTS: p, o = online_request(IDX, a, d, W) ONLINE[(a, d)], read_o = p, read_o + o min_read, max_read = min(min_read, o), max(max_read, o) STREAM, read_s, refreshes, state = stream_run(ALL, W) print(f"training rows {MODEL['train_rows']} , weights {W} , candidates {MODEL['candidates']}") print(f"serving horizon {len(SERVING)} periods x {len(REQUESTS)} subscribers = {N} requests") print("pattern request time answer age rows read per prediction accuracy") for name, age, rd, acc in (("batch", 1, read_b, accuracy(BATCH)), ("online", 0, read_o, accuracy(ONLINE)), ("stream", 0, read_s, accuracy(STREAM))): print(f"{name:<9} {'scan' if name == 'online' else 'lookup':>12} {age:>10}" f" {rd:>12} {rd / N:>14.2f} {acc:>8}") print(f"rows read per online request: min {min_read} , max {max_read}") print(f"stream: refreshes {refreshes} , used {N} , state entries held {state}") ``` ``` training rows 1600 , weights (0.6, 0.4, 0) , candidates 27 serving horizon 3 periods x 200 subscribers = 600 requests pattern request time answer age rows read per prediction accuracy batch lookup 1 6600 11.00 0.8067 online scan 0 278322 463.87 0.8067 stream lookup 0 2200 3.67 0.8067 rows read per online request: min 352 , max 550 stream: refreshes 2200 , used 600 , state entries held 205 ``` The last column is **0.8067** in all three rows. The same weights, the same feature path, the same 600 requests; all three patterns give the same answers. The label is imbalanced and the base class ratio is 0.7964; the base ratio of the test set itself is also 0.8067, so this setup stands **0.0000** above the base, just as it was written in the first lesson of the course. The choice of pattern does not change that either, and an observation that looks only at the output cannot tell which pattern ran. The difference is in the other columns. The online pattern reads 352 to 550 rows per request, 463.87 on average. Almost all of this is the **history of the region**; the history of the subscriber itself is eleven rows. The `region_mean` feature cannot be computed for a single subscriber, it asks for all the subscribers of the region. **Online inference is expensive not because of the model but because of the scope of the feature**, and this is why each prediction reads forty-two times as many rows. **Batch inference** computes the same region mean once and shares it among all the subscribers of the region: 6600 rows, 600 predictions, 11.00 per prediction. In return the answer ages by one period, because no reading that arrives between two runs can enter the prediction. The streaming pattern reads each row once and updates the state incrementally: 2200 rows, 3.67 per prediction. The table does **not show** its cost; of the 2200 refreshes only 600 are used, 205 state entries are held continuously, and rebuilding the state when it is lost means reading the 2200 rows from the start. The rows read column is the column where streaming looks cheapest, and the cost of streaming is not written in exactly that column. #### What the Cost Is Divided By The answer aging by a period looks like a cost. The second block measures it first: it answers the same requests from an ever older run. Then it changes the denominator of the cost per prediction. ```python # This block continues the definitions of ALL , FEATS , W , SERVING , REQUESTS , # ONLINE , read_o , N , score , THRESHOLD , PERIODS and accuracy from the first block. print("answer age (periods) changed predictions accuracy") for g in range(0, 7): t = {} for d in SERVING: j = max(PERIODS.index(d) - g, 1) for a in REQUESTS: t[(a, d)] = 1 if score(FEATS[(a, PERIODS[j])], W) > THRESHOLD else 0 chg = sum(1 for c in t if t[c] != ONLINE[c]) print(f"{g:>20} {chg:>19} {accuracy(t):>8}") dist = sorted(abs(score(FEATS[c], W) - THRESHOLD) for c in ONLINE) print(f"threshold {THRESHOLD} , distance of the decision score from the threshold: min {dist[0]:.4f}" f" , median {dist[len(dist) // 2]:.4f}") print("\n predictions used batch online stream cheapest batch unused") mean_o = read_o / N for u in (1, 2, 5, 20, 50, 200): s = [len(ALL) / u, mean_o, len(REQUESTS) / u] print(f"{u:>17} {s[0]:>7.2f} {s[1]:>9.2f} {s[2]:>6.2f}" f" {('batch', 'online', 'stream')[s.index(min(s))]:>8} {len(REQUESTS) - u:>19}") print(f"batch and online cross at {len(ALL) / mean_o:.2f} predictions used") ``` ``` answer age (periods) changed predictions accuracy 0 0 0.8067 1 0 0.8067 2 0 0.8067 3 0 0.8067 4 0 0.8067 5 0 0.8067 6 0 0.8067 threshold 40 , distance of the decision score from the threshold: min 7.6468 , median 13.6826 predictions used batch online stream cheapest batch unused 1 2200.00 463.87 200.00 stream 199 2 1100.00 463.87 100.00 stream 198 5 440.00 463.87 40.00 stream 195 20 110.00 463.87 10.00 stream 180 50 44.00 463.87 4.00 stream 150 200 11.00 463.87 1.00 stream 0 batch and online cross at 4.74 predictions used ``` The result of the staleness sweep is a single number: while the age of the answer goes from zero to six periods, **the changed predictions are 0** and the accuracy stays at 0.8067. On this workload staleness has no measurable counterpart, and "the batch pattern is free" is **not concluded** from it. The reason is one line below: the smallest distance of the decision score to the threshold is 7.6468 and its median is 13.6826. The decisions stay away from the threshold, and one more period of history does not move the eleven-period cumulative mean enough to close that distance. When the decisions approach the threshold, the same staleness moves the prediction. **A cost that cannot be measured is not ignored; it is written that it could not be measured and under which condition it would show** — this is the second claim of the course read here from the side of the cost. The second table shows that the cost per prediction is a **denominator decision**. The batch pattern reads 2200 rows in one run and produces two hundred predictions; how many of those predictions are actually read is a question of the task, not of the pattern. If a single prediction is used, the cost per prediction of the batch pattern is **2200.00**, which is more than four times the 463.87 of the online pattern; the two patterns are equal at **4.74 predictions used**. The last column is the name of the cost of the batch pattern: when one prediction is used, 199 predictions are produced and never read. Which task rests on which pattern is read from these two tables. The job that lists all the subscribers at the start of the period rests on the batch pattern: the denominator is 200, the staleness cannot be measured, and the cost drops to 11.00. A single-subscriber query that is not known in advance rests on the online pattern: the denominator is 1, and the batch pattern is more than four times as expensive there, and moreover cannot answer a key that is not in the list at all. The job that has to be triggered **at the moment the reading arrives** is done only with streaming: streaming produces 2200 triggers, the online pattern produces 600 only when a request arrives and triggers nothing if no request arrives, while the batch pattern produces the 600 at the start of the period and one period late. #### Summary - The three serving patterns give the **same answers** to the same 600 requests and the accuracy is **0.8067** in all three; the base class ratio is 0.7964, the base ratio of the test set is 0.8067 and this setup stands **0.0000** above it. The choice of pattern is not an accuracy decision, and an observation that looks only at the output cannot tell which pattern ran. - The difference is in three columns: the latency class at request time is `lookup` in the batch and streaming patterns and a `scan` of 352 to 550 rows in the online pattern; the age of the answer is 1, 0 and 0 periods; the rows read per prediction are **11.00**, **463.87** and **3.67**. - The forty-two-fold difference of the online pattern comes not from the model but from the scope of the `region_mean` feature; the history of the subscriber itself is eleven rows, that of its region is between 341 and 539 rows. - When the age of the answer is raised to six periods, the changed predictions are **0**; the reason is not the pattern but that the decision score stays at least **7.6468** away from the threshold. A cost that cannot be measured is not a measured guarantee. - The cost per prediction depends on the denominator: the batch pattern gives **2200.00** when one prediction is used and **11.00** when two hundred are used, and it becomes equal to the online pattern at **4.74** predictions used. - The streaming pattern is the cheapest in rows read, but 1600 of its 2200 refreshes are not used and 205 state entries are held continuously; this cost does not show up in the rows read column at all. #### Next Step The pattern is chosen, but within each pattern one question is still unanswered: how many requests a call should carry **together**. In the batch pattern this is how many predictions a run produces; in the online pattern it means accumulating the arriving requests and processing them at once, and the cost of accumulating is that the first request that arrived **waits**. The next lesson runs the same requests at batch sizes between one and sixty-four, counts the fall of the cost per prediction side by side with the latency of the waiting request, and repeats the same sweep on two accelerator classes. ### Inference Scaling Source: https://academia.sh/en/courses/mlops-fundamentals/inference-scaling Counting how request batching, caching and the choice of accelerator class are traded off between the cost per prediction and the latency of the waiting request: as the batch size goes from one to two, four, eight, sixteen, thirty-two, sixty-four, one hundred twenty-eight and two hundred over the same 600 requests, the cost per prediction falls from 463.87 to 22.00 while the worst wait rises from 0 to 199 arrivals and the average wait to 99.50; the answers are identical in all nine rows and the accuracy stays at 0.8067, so the batch size is not a quality setting. The last group of each period stays at eight requests at sizes sixteen, thirty-two and sixty-four and waits for a group that will never fill; this wait does not show in the worst wait column at all. The crossover point of the two accelerator classes is the ratio of the setup difference to the unit difference and is 80 requests, but in the table the switch shows at one hundred twenty-eight because what is measured is not the requested size but the realized group size. On a stream of 600 requests the cache, with a hit rate of 0.6783, lowers the rows read per request from 464.86 to 148.70 and together with batching gives 26.38; the two gains cannot be multiplied. The previous lesson put three serving patterns side by side and showed that the cost per prediction is a denominator decision. The online pattern was the most expensive there: 463.87 rows per request on average. There was an assumption inside that number: every request was processed **on its own**. Yet two requests that belong to the same region do not have to read the region history twice; if the two are processed together, that history is read once. This lesson turns that togetherness into a setting. Arriving requests are accumulated until a group fills, then processed in a single call. The gain is measurable and large; the price is also measurable and sits in the same place: the first request waiting for the group receives no answer until the group fills. The lesson sweeps these two numbers, then repeats the same sweep on two **accelerator** classes and under a cache. - **TD48.** The source, the pipeline, the model and the two accelerator classes are **fictional** and are generated inside the lesson. No inference service, serving framework, accelerator hardware or cloud service name is written; the accelerators are named only by their **kinds**. The seed is 20260218. - **TD49.** The label is imbalanced: the distribution is `{0: 1752, 1: 448}` and the **base class ratio is 0.7964**. The test set is 600 rows and the smallest step of accuracy is 0.0017. - **TD50.** The batch call and cache procedure was measured in the Building Applications with Model APIs course of the AI Engineering curriculum — the upper limit set by the context window, the damage that a failure spreads and the staleness of the cache lifetime were counted there. It is **not repeated**; here only the **cost per prediction** and the **wait** are counted. - **TD51.** Latency is counted in **arrivals**: the number of arrivals a request waits until its group fills. No raw milliseconds are written. - **TD52.** The history of every subscriber and every region in a group is read **once**; the whole gain of request batching comes from this sharing. - **TD53.** The two accelerator classes are written in **work units**: the general-purpose processing unit is `(60, 4)` and the accelerator is `(300, 1)`; the first number is the setup per call and the second is the unit per request. This is a **choice** and its sensitivity is swept below. - **TD54.** The serving horizon is that of the previous lesson: `2026-10`, `2026-11`, `2026-12` and two hundred requests in each period. The groups form within a period. #### Batch Size Sweep The first block sets up the source, the pipeline, the feature path and the model; then it runs the same 600 requests at nine different batch sizes. The features are independent of the weights and are computed **once**. ```python # FICTIONAL. The meter readings of the measurement network, the first three stages of the # pipeline, the feature path, the threshold model with three weights and the two accelerator # classes are generated inside the lesson; no inference service or hardware is called. # Seed 20260218. SEED, M32 = 20260218, 0xFFFFFFFF def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value REGION = ["east", "west", "north", "south", "central"] TYPE = ["residential", "commercial"] PERIODS = [f"2026-{a:02d}" for a in range(1, 13)] SUBSCRIBERS = {} for i in range(200): r = rng(SEED + 37 * i) SUBSCRIBERS[f"A{i + 1:03d}"] = {"region": REGION[int(r() * 5)], "type": TYPE[int(r() * 2)], "start": 10000 + int(r() * 60000)} def generate_source(): records = [] for i, (sub, a) in enumerate(SUBSCRIBERS.items()): e = a["start"] for p in PERIODS: r = rng(SEED + 101 * i + 7 * PERIODS.index(p)) e += 4 + int(r() * 46) records.append({"subscriber": sub, "period": p, "region": a["region"], "type": a["type"], "index": e, "unit": "m3"}) return records def s3_consumption(records): table = {(k["subscriber"], k["period"]): k for k in records} out = [] for k in records: i = PERIODS.index(k["period"]) prev = table.get((k["subscriber"], PERIODS[i - 1])) if i else None if prev is not None: out.append(dict(k, m3=int(k["index"]) - int(prev["index"]))) return out TRAIN, TEST, THRESHOLD = PERIODS[1:9], PERIODS[9:], 40 def label(k): return 1 if k["m3"] > THRESHOLD else 0 def features(records): """The training path sees the full history. Independent of the weights , computed once.""" ordered = sorted(records, key=lambda k: PERIODS.index(k["period"])) sub, reg = {}, {} for k in ordered: sub.setdefault(k["subscriber"], []).append((PERIODS.index(k["period"]), k["m3"])) reg.setdefault(k["region"], []).append((PERIODS.index(k["period"]), k["m3"])) def mean(series, i): chosen = [v for d, v in series if d < i] return round(sum(chosen) / len(chosen), 4) if chosen else 0.0 return {(k["subscriber"], k["period"]): { "subscriber_mean": mean(sub[k["subscriber"]], PERIODS.index(k["period"])), "region_mean": mean(reg[k["region"]], PERIODS.index(k["period"])), "commercial": 1 if k["type"] == "commercial" else 0} for k in records} GRID = [(a, b, c) for a in (0.6, 0.8, 1.0) for b in (0.0, 0.2, 0.4) for c in (0, 4, 8)] def score(o, w): return w[0] * o["subscriber_mean"] + w[1] * o["region_mean"] + w[2] * o["commercial"] ALL = s3_consumption(generate_source()) FEATS = features(ALL) TRAIN_ROWS = [k for k in ALL if k["period"] in TRAIN] W = max(GRID, key=lambda w: sum( 1 for k in TRAIN_ROWS if (1 if score(FEATS[(k["subscriber"], k["period"])], w) > THRESHOLD else 0) == label(k))) SERVING, REQUESTS = TEST, sorted(SUBSCRIBERS) by_sub, by_reg = {}, {} for k in ALL: by_sub.setdefault(k["subscriber"], []).append(k) by_reg.setdefault(k["region"], []).append(k) TRUTH = {(k["subscriber"], k["period"]): label(k) for k in ALL if k["period"] in SERVING} N = len(REQUESTS) * len(SERVING) def process_group(requests, period): """The history of every subscriber and every region in a group is read ONCE.""" i = PERIODS.index(period) subs = sorted(set(requests)) regs = sorted(set(SUBSCRIBERS[a]["region"] for a in subs)) read = sum(len(by_sub[a]) for a in subs) + sum(len(by_reg[b]) for b in regs) reg_mean = {} for b in regs: s = [k["m3"] for k in by_reg[b] if PERIODS.index(k["period"]) < i] reg_mean[b] = round(sum(s) / len(s), 4) if s else 0.0 t = {} for a in subs: s = [k["m3"] for k in by_sub[a] if PERIODS.index(k["period"]) < i] t[(a, period)] = 1 if score({"subscriber_mean": round(sum(s) / len(s), 4) if s else 0.0, "region_mean": reg_mean[SUBSCRIBERS[a]["region"]], "commercial": 1 if SUBSCRIBERS[a]["type"] == "commercial" else 0}, W) > THRESHOLD else 0 return t, read GENERAL, ACCEL = (60, 4), (300, 1) SIZES = (1, 2, 4, 8, 16, 32, 64, 128, 200) def sweep(size): read, wait, calls, all_pred, work_g, work_a, leftover = 0, 0, 0, {}, 0, 0, 0 for d in SERVING: groups = [REQUESTS[i:i + size] for i in range(0, len(REQUESTS), size)] for g in groups: t, o = process_group(g, d) all_pred.update(t) read += o wait += len(g) * (len(g) - 1) // 2 calls += 1 work_g += GENERAL[0] + GENERAL[1] * len(g) work_a += ACCEL[0] + ACCEL[1] * len(g) leftover = len(groups[-1]) if len(groups[-1]) != size else 0 return all_pred, read, wait, calls, leftover, work_g, work_a SINGLE = sweep(1)[0] WORK = {} print(" size calls unfilled rows read rows/prediction worst wait avg wait" " same answer accuracy") for b in SIZES: t, rd, wt, cl, lo, wg, wa = sweep(b) WORK[b] = (wg, wa) acc = round(sum(1 for c in TRUTH if t[c] == TRUTH[c]) / N, 4) print(f"{b:>5} {cl:>5} {lo:>8} {rd:>12} {rd / N:>15.2f} {b - 1:>15}" f" {wt / N:>11.2f} {str(all(t[c] == SINGLE[c] for c in SINGLE)):>11} {acc:>8}") ``` ``` size calls unfilled rows read rows/prediction worst wait avg wait same answer accuracy 1 600 0 278322 463.87 0 0.00 True 0.8067 2 300 0 252450 420.75 1 0.50 True 0.8067 4 150 0 203610 339.35 3 1.50 True 0.8067 8 75 0 140316 233.86 7 3.50 True 0.8067 16 39 8 87318 145.53 15 7.34 True 0.8067 32 21 8 51183 85.31 31 15.02 True 0.8067 64 12 8 31383 52.30 63 30.38 True 0.8067 128 6 72 19800 33.00 127 53.42 True 0.8067 200 3 0 13200 22.00 199 99.50 True 0.8067 ``` Two columns go in opposite directions and never turn back. The rows read per prediction fall from 463.87 to **22.00**, a factor of twenty-one; the worst wait rises from 0 to **199 arrivals** and the average wait to 99.50. The last two columns close the **quality** side of the decision: at all nine sizes the answers are identical and the accuracy stays at 0.8067 — the base class ratio is 0.7964 and the base ratio of the test set is 0.8067, so this setup stands **0.0000** above the base. **The batch size is not a quality setting; it is a setting between cost and wait.** The gain is not linear. The first doubling lowers 463.87 only to 420.75, because the regions of two requests are mostly different among five regions and the two region histories are still read separately. The gain reaches its highest speed between sizes 16 and 32 and then slows down: at size 200 all the regions are already in a single group, so there is no new history left to share. **What request batching gains is not the number of requests but the rows that the requests read in common.** The third column carries the table's own flaw. At sizes 16, 32 and 64 the last group of each period stays at **eight** requests, and at size 128 at seventy-two. These requests wait for a group that will never fill and are released only by a timeout. Their wait does **not show** in the "worst wait" column, because that column assumes the group fills. The measured worst value stands below an unmeasured assumption. #### Setup Cost and Cache Rows read is not the only name of the cost. Every call pays a setup cost, and this cost changes with the unit the call runs on. The second block compares the two accelerator classes in the same sweep, computes the crossover point and adds a stream of repeated requests at the end. ```python # This block continues the definitions of ALL , REQUESTS , SERVING , N , SIZES , WORK , # GENERAL , ACCEL , SUBSCRIBERS , PERIODS , SEED , rng and process_group from the first block. print(" size actual group general-purpose accelerator cheaper") for b in SIZES: wg, wa = WORK[b] groups = [len(REQUESTS[i:i + b]) for i in range(0, len(REQUESTS), b)] print(f"{b:>5} {len(REQUESTS) / len(groups):>16.1f} {wg / N:>15.2f} {wa / N:>12.2f}" f" {'accelerator' if wa < wg else 'general-purpose':>15}") for setup in (150, 300, 600): print(f"setup {setup:>3} -> crossover point {(setup - GENERAL[0]) / (GENERAL[1] - ACCEL[1]):.1f} requests") def request_stream(n): r = rng(SEED + 911) return [f"A{int(r() * 200) + 1:03d}" for _ in range(n)] def cost(calls, size, period="2026-12"): return sum(process_group(calls[i:i + size], period)[1] for i in range(0, len(calls), size)) STREAM = request_stream(600) DISTINCT = list(dict.fromkeys(STREAM)) print(f"\nstream {len(STREAM)} requests , distinct {len(DISTINCT)} , cache hit rate" f" {(len(STREAM) - len(DISTINCT)) / len(STREAM):.4f}") base = cost(STREAM, 1) for name, calls, b in (("neither , size 1", STREAM, 1), ("batching only", STREAM, 32), ("cache only", DISTINCT, 1), ("both together", DISTINCT, 32)): rd = cost(calls, b) print(f"{name:<24} read {rd:>7} per request {rd / len(STREAM):>7.2f}" f" factor {rd / base:.4f}") t1, t2 = cost(STREAM, 32) / base, cost(DISTINCT, 1) / base print(f"predicted by the product {t1 * t2:.4f} , actual" f" {cost(DISTINCT, 32) / base:.4f}") ``` ``` size actual group general-purpose accelerator cheaper 1 1.0 64.00 301.00 general-purpose 2 2.0 34.00 151.00 general-purpose 4 4.0 19.00 76.00 general-purpose 8 8.0 11.50 38.50 general-purpose 16 15.4 7.90 20.50 general-purpose 32 28.6 6.10 11.50 general-purpose 64 50.0 5.20 7.00 general-purpose 128 100.0 4.60 4.00 accelerator 200 200.0 4.30 2.50 accelerator setup 150 -> crossover point 30.0 requests setup 300 -> crossover point 80.0 requests setup 600 -> crossover point 180.0 requests stream 600 requests , distinct 193 , cache hit rate 0.6783 neither , size 1 read 278916 per request 464.86 factor 1.0000 batching only read 47872 per request 79.79 factor 0.1716 cache only read 89221 per request 148.70 factor 0.3199 both together read 15829 per request 26.38 factor 0.0568 predicted by the product 0.0549 , actual 0.0568 ``` The accelerator class is more expensive up to size 128. The crossover point is not a measurement but a ratio: the setup difference divided by the unit difference, that is `(300 - 60) / (4 - 1) = 80` requests. The reason the switch shows at 128 in the table is the second column. When size 64 is requested, every period splits into groups of 64, 64, 64 and 8 requests, and the **realized average group is 50**, which is below 80; at size 128 the realized average is 100, which is above 80. **The case for an accelerator lies not in its own speed but in the group size that can be filled**; with a group that cannot be filled, the accelerator is only an expensive general-purpose unit. The setup cost sweep shows this with three numbers: the crossover points of setups 150, 300 and 600 are 30, 80 and 180 requests. The last four rows add the cache. In the stream of six hundred requests there are only **193** distinct requests; the cache hit rate is 0.6783 and the rows read per request fall from 464.86 to 148.70. Request batching alone is stronger: 79.79. The two together give **26.38**. But the two factors cannot be measured separately and multiplied: the product of 0.1716 and 0.3199 predicts 0.0549 and the actual is 0.0568. The difference is small and its cause is measurable — the cache **thins** the groups. When the remaining 193 requests are split into groups of thirty-two, a single request is left in the last group, and that single request reads the whole region history on its own. **The gain of one mechanism depends on which mechanism stands next to it.** #### Summary - When the batch size is swept between one and two hundred, the rows read per prediction fall **from 463.87 to 22.00**, the worst wait rises **from 0 to 199 arrivals** and the average wait to 99.50; the two columns go in opposite directions and a size cannot be chosen without writing which of them is paid. - At all nine sizes the answers are identical and the accuracy stays at 0.8067; the base class ratio is 0.7964, the base ratio of the test set is 0.8067 and this setup stands 0.0000 above it: the batch size is not a quality setting. - The gain comes not from the number of requests but from the **rows the requests read in common**; the first doubling lowers 463.87 only to 420.75, because the regions of two requests are mostly different. - At sizes 16, 32 and 64 the last group of each period stays at eight requests and waits for a group that will never fill; this wait does not show in the "worst wait" column at all, because that column assumes the group fills. - The crossover point of the accelerator class is `(300 - 60) / (4 - 1) = 80` requests, but in the table the switch shows at 128: what is measured is not the requested size but the **realized group size** (50 when 64 is requested, 100 when 128 is requested). - On a stream of 600 requests the cache lowers 464.86 to 148.70 with a hit rate of 0.6783; together with request batching it gives 26.38, while the product of the two factors predicts 25.52 — the cache thins the groups. #### Next Step In five lessons the model was packaged, placed in a pattern and scaled. In all five there was a silent assumption: the model sits at an endpoint that is **reachable over the network** from the place where the request arrives. When the network drops, the cost per prediction of this setup is undefined, because there is no prediction. The next lesson moves the model to the source of the request, to the device itself; it measures what shrinking takes from the accuracy and in which task it takes it first, and counts the independence gained by running at the edge as answered requests. ### Edge and On-Device Inference Source: https://academia.sh/en/courses/mlops-fundamentals/edge-and-on-device-inference Counting what moving the model to the source of the request takes from accuracy and from which task it takes first: the number written to the device is five, what is dropped is the history of 341 to 539 rows that produces the region feature, and when that history is dropped the two-class task stays at 0.8067 while the three-tier task falls from 0.4833 to 0.4733. The quantization sweep gives the same order: the two-class task does not move a single prediction down to level 4, the three-tier task moves two predictions at level 16 and five at level 8, and at level 2 the two collapse to 0.3833 and 0.2250. The independence gained by running at the edge is measured in correct answers: when the network outage rate rises to 0.20, the correct answers of the central setup fall from 484 to 391 and from 290 to 234, while the edge stays fixed at 484 and 284; the outage rate at which the two setups are equal is 0.0000 for the two-class task and 0.0207 for the three-tier task. The lesson closes the course and gathers the numbers of the twelve lessons into a single table. The previous lesson made the request cheaper on the server side: when the batch size rose to two hundred, the rows read per prediction fell to 22.00. Five lessons packaged, placed and scaled the model; beneath all five lay the same silent assumption: the model sits at an endpoint that is reachable **over the network** from the place where the request is born. When that network goes down, the cost per prediction is undefined, because there is no prediction. This lesson moves the model to the source of the request. What moves is small: three weights. What cannot move is the history that produces the feature those weights **multiply**. The lesson first counts that history, then measures what dropping it and **quantization** take from the two tasks, and finally writes the independence of the edge as answered requests. - **TD55.** The source, the pipeline, the model and the shrinking steps are **fictional**; no edge device, runtime, accelerator hardware or cloud service name is written. The seed is 20260218. - **TD56.** The label is imbalanced: `{0: 1752, 1: 448}`, the **base class ratio is 0.7964**; the test set is 600 rows and the step is **0.0017**. - **TD57.** The quantization and **distillation** procedure was measured in the Building Applications with Model APIs course; it is **not repeated** and its numbers are **inputs**. - **TD58.** Shrinking has two steps: because the region history is not on the device, `region_mean` is dropped and the model is retrained on a grid of nine candidates; then the features are reduced to `level` distinct values. - **TD59.** Two tasks are measured: **task A** has two classes (threshold 40), **task B** has three tiers (thresholds 20 and 40). - **TD60.** The network outage rate `p` means that a share `p` of the requests receive **no answer at all** from the center; the edge answers **all** of them. The one number that compares the two in a common unit is the **correct-answer count** (answered requests times accuracy). #### The History That Cannot Move to the Device The first block sets up the source, pipeline and tasks, trains the two models separately and sweeps the level. ```python # FICTIONAL. The meter readings of the measurement network, the first three stages of the # pipeline, the feature path, the threshold model with three weights and the shrinking steps # are generated inside the lesson; no edge device, runtime or hardware is called. # Seed 20260218. SEED, M32 = 20260218, 0xFFFFFFFF def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value REGION = ["east", "west", "north", "south", "central"] TYPE = ["residential", "commercial"] PERIODS = [f"2026-{a:02d}" for a in range(1, 13)] SUBSCRIBERS = {} for i in range(200): r = rng(SEED + 37 * i) SUBSCRIBERS[f"A{i + 1:03d}"] = {"region": REGION[int(r() * 5)], "type": TYPE[int(r() * 2)], "start": 10000 + int(r() * 60000)} def generate_source(): records = [] for i, (sub, a) in enumerate(SUBSCRIBERS.items()): e = a["start"] for p in PERIODS: r = rng(SEED + 101 * i + 7 * PERIODS.index(p)) e += 4 + int(r() * 46) records.append({"subscriber": sub, "period": p, "region": a["region"], "type": a["type"], "index": e, "unit": "m3"}) return records def s3_consumption(records): table = {(k["subscriber"], k["period"]): k for k in records} out = [] for k in records: i = PERIODS.index(k["period"]) prev = table.get((k["subscriber"], PERIODS[i - 1])) if i else None if prev is not None: out.append(dict(k, m3=int(k["index"]) - int(prev["index"]))) return out TRAIN, TEST, THRESHOLD = PERIODS[1:9], PERIODS[9:], 40 def features(records): """The training path sees the full history. Independent of the weights , computed once.""" ordered = sorted(records, key=lambda k: PERIODS.index(k["period"])) sub, reg = {}, {} for k in ordered: sub.setdefault(k["subscriber"], []).append((PERIODS.index(k["period"]), k["m3"])) reg.setdefault(k["region"], []).append((PERIODS.index(k["period"]), k["m3"])) def mean(series, i): chosen = [v for d, v in series if d < i] return round(sum(chosen) / len(chosen), 4) if chosen else 0.0 return {(k["subscriber"], k["period"]): { "subscriber_mean": mean(sub[k["subscriber"]], PERIODS.index(k["period"])), "region_mean": mean(reg[k["region"]], PERIODS.index(k["period"])), "commercial": 1 if k["type"] == "commercial" else 0} for k in records} def score(o, w): return w[0] * o["subscriber_mean"] + w[1] * o["region_mean"] + w[2] * o["commercial"] GRID = [(a, b, c) for a in (0.6, 0.8, 1.0) for b in (0.0, 0.2, 0.4) for c in (0, 4, 8)] EDGE_GRID = [w for w in GRID if w[1] == 0.0] # the region history is not on the device ALL = s3_consumption(generate_source()) FEATS = features(ALL) M3 = {(k["subscriber"], k["period"]): k["m3"] for k in ALL} TRAIN_KEYS = [c for c in M3 if c[1] in TRAIN] TEST_KEYS = [c for c in M3 if c[1] in TEST] N = len(TEST_KEYS) LOW_S, HIGH_S = min(o["subscriber_mean"] for o in FEATS.values()), max(o["subscriber_mean"] for o in FEATS.values()) LOW_R, HIGH_R = min(o["region_mean"] for o in FEATS.values()), max(o["region_mean"] for o in FEATS.values()) def task_a(v): return 1 if v > THRESHOLD else 0 def task_b(v): return 0 if v <= 20 else (1 if v <= THRESHOLD else 2) def train(task, grid, feats): return max(grid, key=lambda w: sum(1 for c in TRAIN_KEYS if task(score(feats[c], w)) == task(M3[c]))) def accuracy(task, w, feats): return round(sum(1 for c in TEST_KEYS if task(score(feats[c], w)) == task(M3[c])) / N, 4) def share(task): return round(max(sum(1 for c in TEST_KEYS if task(M3[c]) == s) for s in (0, 1, 2)) / N, 4) def quantize_features(level): def q(x, low, high): step = (high - low) / (level - 1) return low + round((x - low) / step) * step return {c: {"subscriber_mean": q(o["subscriber_mean"], LOW_S, HIGH_S), "region_mean": q(o["region_mean"], LOW_R, HIGH_R), "commercial": o["commercial"]} for c, o in FEATS.items()} WA, WB = train(task_a, GRID, FEATS), train(task_b, GRID, FEATS) WAK, WBK = train(task_a, EDGE_GRID, FEATS), train(task_b, EDGE_GRID, FEATS) region_sizes = [sum(1 for c in M3 if SUBSCRIBERS[c[0]]["region"] == b) for b in REGION] print(f"largest class share in the serving horizon: task A {share(task_a)} , task B {share(task_b)}") print(f"center model A {WA} accuracy {accuracy(task_a, WA, FEATS)}" f" | B {WB} accuracy {accuracy(task_b, WB, FEATS)}") print(f"edge model A {WAK} accuracy {accuracy(task_a, WAK, FEATS)}" f" | B {WBK} accuracy {accuracy(task_b, WBK, FEATS)}") print(f"number written to the device 5 (3 weights + 2 sums of the subscriber) ;" f" dropped region history {min(region_sizes)}-{max(region_sizes)} rows") base_a = {c: task_a(score(FEATS[c], WA)) for c in TEST_KEYS} base_b = {c: task_b(score(FEATS[c], WB)) for c in TEST_KEYS} print("\nlevel distinct values A changed A accuracy B changed B accuracy") for s in (256, 16, 12, 8, 4, 2): q_feats = quantize_features(s) ta = {c: task_a(score(q_feats[c], WA)) for c in TEST_KEYS} tb = {c: task_b(score(q_feats[c], WB)) for c in TEST_KEYS} fd = (len(set(o["subscriber_mean"] for o in q_feats.values())) + len(set(o["region_mean"] for o in q_feats.values()))) print(f"{s:>5} {fd:>15} {sum(1 for c in TEST_KEYS if ta[c] != base_a[c]):>9}" f" {accuracy(task_a, WA, q_feats):>10} {sum(1 for c in TEST_KEYS if tb[c] != base_b[c]):>9}" f" {accuracy(task_b, WB, q_feats):>10}") ``` ``` largest class share in the serving horizon: task A 0.8067 , task B 0.4833 center model A (0.6, 0.4, 0) accuracy 0.8067 | B (0.6, 0.4, 4) accuracy 0.4833 edge model A (0.6, 0.0, 0) accuracy 0.8067 | B (1.0, 0.0, 4) accuracy 0.4733 number written to the device 5 (3 weights + 2 sums of the subscriber) ; dropped region history 341-539 rows level distinct values A changed A accuracy B changed B accuracy 256 199 0 0.8067 0 0.4833 16 21 0 0.8067 2 0.48 12 16 0 0.8067 0 0.4833 8 11 0 0.8067 5 0.4783 4 7 0 0.8067 0 0.4833 2 4 402 0.3833 600 0.225 ``` The fourth line is the first sentence of the lesson: the number written to the device is **five**, and what is dropped is the **region history** of 341 to 539 rows. **What cannot move to the edge is not the model but the history that produces the feature.** The cost splits: in task A the accuracy stays at 0.8067, in task B it falls to **0.4733**. Placing the two accuracies next to the baseline changes the name of the cost. The label is imbalanced and the **base class ratio is 0.7964**; in the serving horizon the largest class of task A holds a share of 0.8067 and the largest tier of task B holds 0.4833, and the center gives exactly that share in both. The zero drop in task A comes from this: there is nothing there for shrinking to **take away**. The edge model, however, falls **0.0100 below** the baseline in task B, **six steps worse** than a procedure that says the largest tier for every row. The cost of shrinking is not the drop in accuracy but **which side of the baseline one stays on**. Quantization confirms the same order independently. Task A **does not move a single prediction** down to level 4; task B moves two predictions at level 16 and five at level 8. These movements are just above the 0.0017 resolution and the sweep is not monotonic, so the point where degradation begins is written as "between 16 and 2". At level 2 both fall below the baseline: task A **0.3833** (0.4234 below), task B **0.2250** (0.2583 below). **Shrinking erases the distinction, erases the finest task first, and once it goes far enough it carries the model below the baseline.** #### The Request Answered Under an Outage What shrinking takes away was written; what it gives was not. ```python # This block continues the definitions of N , FEATS , WA , WB , WAK , WBK , task_a , task_b , # accuracy , quantize_features , rng and SEED from the first block. mA, mB = accuracy(task_a, WA, FEATS), accuracy(task_b, WB, FEATS) kA, kB = accuracy(task_a, WAK, FEATS), accuracy(task_b, WBK, FEATS) r = rng(SEED + 733) outage = [r() for _ in range(N)] print("outage rate answered center A edge A center B edge B") for p in (0.00, 0.02, 0.05, 0.10, 0.20): y = sum(1 for x in outage if x >= p) print(f"{p:>11.2f} {y:>8} {round(y * mA):>8} {round(N * kA):>6}" f" {round(y * mB):>8} {round(N * kB):>6}") print(f"crossover point: task A {1 - kA / mA:.4f} , task B {1 - kB / mB:.4f}") print("when the edge model is quantized level 8: task B", accuracy(task_b, WBK, quantize_features(8)), ", level 4:", accuracy(task_b, WBK, quantize_features(4))) ``` ``` outage rate answered center A edge A center B edge B 0.00 600 484 484 290 284 0.02 589 475 484 285 284 0.05 566 457 484 274 284 0.10 536 432 484 259 284 0.20 485 391 484 234 284 crossover point: task A 0.0000 , task B 0.0207 when the edge model is quantized level 8: task B 0.4767 , level 4: 0.4533 ``` The edge columns are fixed, and this is the one real gain of the edge: whatever the outage rate, **all** 600 requests are answered, while at the center the answered requests fall to 485. The correct answers fall from 484 to **391** in task A and from 290 to **234** in task B. The two setups are equal at an outage rate of **0.0000** in task A and **0.0207** in task B: above this rate, the edge model that has fallen below the baseline still produces more correct answers. **Independence is not an accuracy number; it enters the same table only through the correct-answer count.** The last line combines both steps: 0.4767 at level 8, 0.4533 at level 4. #### Summary - The number written to the device is **five**; what cannot move is the **region history** of 341 to 539 rows. - Dropping the history leaves task A unchanged and lowers task B to **0.4733**. The base class ratio is 0.7964 and the largest tier of task B is 0.4833: the edge model is **0.0100 below the baseline**, and the zero in A is not robustness but an absence of measurement. - Quantization confirms the same order: task A does not move a single prediction down to level 4, task B moves two at level 16 and five at level 8; at level 2 the two collapse to **0.3833** and **0.2250**. - Whatever the outage rate, the edge answers all 600 requests; at a rate of 0.20 the correct answers of the center fall from 484 to **391** and from 290 to **234**. The two setups are equal at an outage rate of 0.0000 in task A and **0.0207** in task B, and the case for the edge is that rate. #### Course Wrap-Up The rows carry no common unit; the metric each is read by sits next to the lesson name, and the difference in parentheses is **pinned setup minus baseline**. | Lesson | Unpinned baseline | Pinned setup | Cost | |---|---|---|---| | What Is MLOps — test accuracy of three runs | no pinning: **0.8067 / 0.7955 / 0.8078** (spread **0.0123**) | the weights of the three runs are the same: `(0.6, 0.4, 0)` | the accuracy is the base ratio of the test set itself, recall **0.0000** | | Experiment Tracking — finding the run that produced a number | no record: **1, 2, 3, 5, 18** run repeats | with the full record **0** in all five (down by up to **−18**) | **162 fields**; a ledger that holds only the metric shortens one of five searches | | Data and Model Versioning — number of ambiguous runs | unversioned: **1 name**, ambiguous runs **4** | content-named: **3 names**, ambiguous runs **0** (**−4**) | **6540 rows** (date-named schema 8740); the output signature does not see a swallowed duplicate | | Feature Stores — test accuracy with the same weights | serving path, window 1: **0.7800** | feature store, window 2: **0.8067** (**+0.0267**) | 6600 stored values and a fourth stage; the skew changes 600 features and flips the prediction in **16** rows | | Environment Pinning — sameness of three runs | no pinning: **[0.8067 , 0.7955 , 0.8078]** | with three items: **[0.8067 x 3]** | 15 item records, **81** candidates, **6600** rows; the contribution of the grid and the rounding is **unmeasurable** | | Model Registry — finding the run of the model in production | no registry: **16 records**, **12 candidates** | with the registry: **1 record** (**−15**), **1 candidate** (**−11**) | **21 records**; runs with the same weights report 0.7950 and 0.8067 | | Training Pipelines — training rows | with 40 late rows: **1520** | end-to-end run: **1600** (**+80**), stale step **0** | **13800 processed rows**; all seven setups report the same **0.8067** | | Continuous Integration for Machine Learning — corruption class stopped | no gates: **0 / 6** | data gate **4**, model gate **3**, code gate **0** | **two** of the four stopped runs would have given the same model anyway; the **late** one stops at no gate | | Model Packaging — what drops out of 1800 calls | no contract: **600**, **1200** when the field name changes | with a contract: **0** (**−1200**) | 5 notices; when the threshold field drops, **the number of dropped calls does not change** and 1200 answers change silently | | Serving Patterns — rows read per prediction | call with no pattern written: **463.87** | batch: **11.00** (**−452.87**), stream: **3.67** | in the batch pattern the answer ages one period, and when a single prediction is used **199** of 200 are produced for nothing; in the stream **1600** refreshes go unused | | Inference Scaling — rows per prediction and wait | size 1: **463.87**, wait **0 arrivals** | size 200: **22.00** (**−441.87**), wait **199 arrivals** (**+199**) | **8** requests per period wait in a group that does not fill and do not show in the column; the crossover point of the accelerator is **80** requests | | Edge and On-Device Inference — correct answers, outage rate 0.20 | center: **391** (A), **234** (B) | edge: **484** (**+93**), **284** (**+50**) | the region history is dropped, task B falls **from 0.4833 to 0.4733**; the crossover point is **0.0207** | The table is a single rule written twelve times. **A model's number is meaningful together with the path that produced it; a number that cannot be reproduced is not a measurement but a memory.** In What Is MLOps the same job gives three different numbers, and the weights are the same in all three; in Data and Model Versioning the same name carries three different contents; in Model Registry runs with the same weights report 0.7950 and 0.8067. What changes is not the model but **by which path** the number was produced. **The second claim reads from the same table: pinning also has a cost, and a pinning whose contribution cannot be shown gives an unmeasured assurance.** In Environment Pinning three items are enough and the **contribution of the grid and the rounding is unmeasurable**; in Experiment Tracking a ledger that holds only the metric shortens only one of five searches; in Continuous Integration for Machine Learning two of the four runs stopped by the data gate would have given the same model anyway; in Serving Patterns, unmeasurable staleness is no assurance of the pattern but a result of the decision score standing 7.6468 from the threshold. In four rows, the pinning that was paid for has not been repaid, or cannot be shown to have been. **The third claim becomes sharpest in a single row.** In Feature Stores, with the same triple of weights, the accuracy is **0.8067** by the training path and **0.7800** by the serving path; **the model never changed**, and all of the 0.0267 comes from the history by which the feature was computed. Model Packaging is its silent twin: when the threshold field drops out of the package, the number of dropped calls does not change, and 1200 of the 1200 calls change their answer. **Testing the model is not enough; the feature path in serving and the declaration of the package must be tested too.** Over sixty-one lessons data was taken in, moved, stored, partitioned, tied to a model, made reproducible, packaged, served, scaled and moved to the device. None of them asked about what comes **after the model leaves**. The world changes and the input starts to **drift**; the label arrives **weeks later**, so today's accuracy cannot be measured today; and when the model will **retire** is written nowhere. The next course treats the model in production not as a delivery but as an object of observation, and measures these three questions.