--- title: 'Model Monitoring and Governance' source: 'https://academia.sh/en/courses/model-monitoring' language: en updated: '2026-10-21T05:03:02+00:00' license: 'CC BY-SA 4.0' lessons: 13 outcomes: - 'Measures data drift and concept drift separately and shows which one actually degrades performance' - 'Counts what proxy metrics see and miss under delayed labels and justifies the alert threshold' - 'Builds release staging, retraining policy and retirement decisions in an auditable way' --- # Model Monitoring and Governance Counting the life of a model after it reaches production: the prediction share rising from 0.4550 to 0.5450 as the input distribution drifts period by period while accuracy does not drop at all, accuracy collapsing from 0.9700 to 0.8150 when the rule changes although the input summary does not change at all, the only measure that sees it arriving two periods later because of delayed labels, proxy metrics used in the absence of labels firing on harmless drift and staying silent on harmful drift, false firing increasing as the alert threshold narrows, the share covered by collecting labels from production, how much the requests sent to the new model in shadow and canary deployment move the result, the smallest difference an online experiment can distinguish, a retraining trigger gaining nothing when the rule has not changed, decision logs making the reason for a prediction findable, the class-level counterpart of compliance obligations, tracking training and inference cost separately, and which measure decides that a model is retired. ## Monitoring Counting by which measure and after how many periods it is seen whether a model in production has degraded: the input summary going from 25.9220 to 27.0472 as the input distribution grows every period and not firing at any alert band, the prediction share in the same drift going from 0.4550 to 0.5450 and firing in three periods at the 0.05 band while accuracy goes from 0.9700 to 0.9800 and never falls, the input summary not changing at all when the rule changes and accuracy collapsing from 0.9700 to 0.8150 while the prediction share stays almost constant, the accuracy of a period being known only two periods later because of the delayed label and the two-period gap between the moment of degradation and the moment it is seen, the proxy metrics used when there is no label firing under a harmless drift and staying silent under a harmful drift, the number of firings growing as the alert band narrows and how many of them correspond to real degradation, the share covered by label collection in production and the bias of the collected label. ### Data Drift Source: https://academia.sh/en/courses/model-monitoring/data-drift Measuring the change of the input distribution in production and building the three conditions of a monitorable model: when the decision threshold is searched as a setting separate from the label threshold, when the training criterion is switched from accuracy to balanced accuracy, and when the label is tied to a rule the features see, recall rises from 0.0000 to 1.0000, because the same 135-candidate search stays at 0.5203 balanced accuracy with the old label, which is coin-flip level. In the corrected setup the input drifts period by period, the input summary rises from 25.9220 to 27.0472, the prediction share goes from 0.4550 to 0.5450 and the alert fires in three periods in the 0.05 band, yet accuracy never falls below the 0.9700 baseline; all three firings are false positives, and what the drift actually took away, compared with the no-drift run, is 0.0100 in 2026-12, that is two resolution steps. Sixty-one lessons measured the road up to the model **reaching production**: the rows the pipeline carries were counted, the place where a row would live was chosen, the work was divided, the run was pinned, the feature was read from a single place, and the model was packaged and served. The number at the end of that road belongs to **the moment of release** and says nothing once that moment has passed. What comes after release was never asked. The world changes under the model: the input distribution drifts, the rule between input and label changes, the true label arrives weeks later, and one day the model is retired. This course measures that time, but its first job is a correction. **An unmonitorable model cannot be monitored:** the model the MLOps Fundamentals course left behind cannot show any degradation in production, and there are three independent reasons for this. - **MO1.** The source, the first three stages of the pipeline that look at the model, the threshold-based model and the drift scenario are **fictional**; no real learning library, monitoring product or cloud service is called. The seed is **20260218**. - **MO2.** **Drift** is production data moving away from the training data: **data drift** is a change of the input distribution, **concept drift** is a change of the rule between input and label. The measurement procedure was built in the Model Evaluation, Interpretability and Ethics course and is **not repeated**; this course adds only the **life in production**. - **MO3.** The monitoring periods are **2026-07…2026-12**, **200 rows** per period. The training window is **2026-02…2026-09**; the first three monitoring periods overlap with it, while **the three periods in which the drift is measured lie outside it**. - **MO4.** This course's label is the rule `subscriber_mean + 6 x commercial > 30`, and **this rule is the CONCEPT**. Data drift does not change it. In the monitoring periods the **base class ratio is 0.5342**; the set is balanced, and accuracy can be read meaningfully here. - **MO5.** The **decision threshold** is the setting that turns a score into a prediction, and it is separate from the label threshold. **Balanced accuracy** is the mean of **recall** and **specificity**. - **MO6.** The baseline is the **unmonitored setup**: no indicator, no alert; the degradation goes **unseen**. The **alert threshold** is written as a two-sided **band**. - **MO7.** Resolution: the step of period accuracy is **1/200 = 0.0050**; in this lesson the total swing of accuracy is **0.0200**, that is, four steps. #### An Unmonitorable Model Cannot Be Monitored The first block builds the source, the three stages of the pipeline that look at the model, the features and the fictional model. The three fixes are marked by name inside the code: **the decision threshold is a separate setting** and is searched together with the weights in `DECISION_GRID`, the training criterion is not accuracy but **balanced accuracy**, and the label is tied to a rule the features **see**. ```python # FICTIONAL. The meter readings of the measurement network, the first three stages # of the pipeline that look at the model, the threshold-based model and the drift # scenario are defined inside the lesson; no real learning library, monitoring # product or cloud service is called. SEED, M32 = 20260218, 0xFFFFFFFF def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value REGION = ["east", "west", "north", "south", "central"] TYPE = ["residential", "commercial"] PERIODS = [f"2026-{a:02d}" for a in range(1, 13)] SUBSCRIBERS = {} for i in range(200): r = rng(SEED + 37 * i) SUBSCRIBERS[f"A{i + 1:03d}"] = {"region": REGION[int(r() * 5)], "type": TYPE[int(r() * 2)], "start": 10000 + int(r() * 60000)} def generate_source(): records = [] for i, (sub, a) in enumerate(SUBSCRIBERS.items()): e = a["start"] for p in PERIODS: r = rng(SEED + 101 * i + 7 * PERIODS.index(p)) e += 4 + int(r() * 46) records.append({"subscriber": sub, "period": p, "region": a["region"], "type": a["type"], "index": e, "unit": "m3"}) return records def s1_screen(records): REQUIRED = ("subscriber", "period", "region", "index", "unit") keep = [k for k in records if all(a in k for a in REQUIRED)] return keep, len(records) - len(keep) def s2_dedupe(records): seen, keep = set(), [] for k in records: key = (k["subscriber"], k["period"]) if key in seen: continue seen.add(key) keep.append(k) return keep, len(records) - len(keep) def s3_consumption(records): """Consumption from the period difference. Without a previous period the row cannot be computed.""" table = {(k["subscriber"], k["period"]): k for k in records} out, dropped = [], 0 for k in records: i = PERIODS.index(k["period"]) if i == 0: continue prev = table.get((k["subscriber"], PERIODS[i - 1])) if prev is None: dropped += 1 continue out.append(dict(k, m3=int(k["index"]) - int(prev["index"]))) return out, dropped def rows(): v, _ = s1_screen(generate_source()) v, _ = s2_dedupe(v) v, _ = s3_consumption(v) return v def features(records, digits=4): """Features are independent of the weights; computed ONCE from the full history.""" ordered = sorted(records, key=lambda k: PERIODS.index(k["period"])) sub, reg = {}, {} for k in ordered: sub.setdefault(k["subscriber"], []).append((PERIODS.index(k["period"]), k["m3"])) reg.setdefault(k["region"], []).append((PERIODS.index(k["period"]), k["m3"])) def mean(series, i): chosen = [v for d, v in series if d < i] return round(sum(chosen) / len(chosen), digits) if chosen else 0.0 return {(k["subscriber"], k["period"]): { "subscriber_mean": mean(sub[k["subscriber"]], PERIODS.index(k["period"])), "region_mean": mean(reg[k["region"]], PERIODS.index(k["period"])), "commercial": 1 if k["type"] == "commercial" else 0} for k in records} GRID = [(a, b, c) for a in (0.6, 0.8, 1.0) for b in (0.0, 0.2, 0.4) for c in (0, 4, 8)] def score(o, w): return w[0] * o["subscriber_mean"] + w[1] * o["region_mean"] + w[2] * o["commercial"] # ---- FIX 3: the label is tied to a rule the features SEE. The rule is the CONCEPT. RULE_THRESHOLD = 30 RULE_COMMERCIAL = 6 def label(feat_k, threshold=RULE_THRESHOLD, commercial_factor=RULE_COMMERCIAL): return 1 if feat_k["subscriber_mean"] + commercial_factor * feat_k["commercial"] > threshold else 0 MONITORED = PERIODS[6:] # 2026-07 .. 2026-12 : the periods monitored in production # ---- FIX 1: the decision threshold is a SEPARATE setting from the label threshold and is searched in training. DECISION_GRID = (18, 22, 26, 30, 34) def predict(feat_k, w, decision): return 1 if score(feat_k, w) > decision else 0 # ---- FIX 2: the training criterion is not ACCURACY but BALANCED ACCURACY. # rule=None -> this course's label; passing a rule from outside serves to measure the old setup. def balanced_accuracy(subset, feats, w, decision, rule=None): c = {(0, 0): 0, (0, 1): 0, (1, 0): 0, (1, 1): 0} for k in subset: f_k = feats[(k["subscriber"], k["period"])] c[(predict(f_k, w, decision), rule(k, f_k) if rule else label(f_k))] += 1 rec = c[(1, 1)] / max(c[(1, 1)] + c[(0, 1)], 1) # recall spec = c[(0, 0)] / max(c[(0, 0)] + c[(1, 0)], 1) # specificity return {"accuracy": (c[(1, 1)] + c[(0, 0)]) / len(subset), "recall": rec, "specificity": spec, "balanced": (rec + spec) / 2, "weights": w, "decision": decision} def train(records, feats, periods=None, rule=None): """Training = searching the weight TRIPLE together with the decision threshold: 27 x 5 = 135 candidates.""" subset = [k for k in records if k["period"] in (periods or PERIODS[1:9])] return max((balanced_accuracy(subset, feats, w, decision, rule) for w in GRID for decision in DECISION_GRID), key=lambda o: o["balanced"]) S = rows() FEATS = features(S) MON = [k for k in S if k["period"] in MONITORED] dist = {0: 0, 1: 0} for k in MON: dist[label(FEATS[(k["subscriber"], k["period"])])] += 1 print("rows:", len(S), " monitoring periods:", len(MONITORED), " monitoring rows:", len(MON), " per period:", len(MON) // len(MONITORED)) print("monitoring label distribution:", dist, " base class ratio:", round(dist[0] / len(MON), 4)) ``` ``` rows: 2200 monitoring periods: 6 monitoring rows: 1200 per period: 200 monitoring label distribution: {0: 641, 1: 559} base class ratio: 0.5342 ``` The base class ratio is **0.5342**; in the MLOps Fundamentals course it was 0.7964, and accuracy read on its own was misleading. The second block counts what the fixes bought: the upper table compares the two setups **on the same test periods**, the lower one runs **the same search** with two different label rules. ```python TEST_ROWS = [k for k in S if k["period"] in PERIODS[9:]] MODEL = train(S, FEATS) def old_rule(k, f_k): return 1 if k["m3"] > 40 else 0 HEADER = (f"{'setup':<46}{'weights / decision':>22}{'accuracy':>9}" f"{'recall':>12}{'specificity':>13}{'balanced':>9}") def show(name, o): print(f"{name:<46}{str(o['weights']) + ' / ' + str(o['decision']):>22}{o['accuracy']:>9.4f}" f"{o['recall']:>12.4f}{o['specificity']:>13.4f}{o['balanced']:>9.4f}") print("on the test periods (2026-10 .. 2026-12):\n" + HEADER) show("old : accuracy criterion", balanced_accuracy(TEST_ROWS, FEATS, (0.6, 0.4, 0), 40, old_rule)) show("this course : balanced accuracy criterion", balanced_accuracy(TEST_ROWS, FEATS, MODEL["weights"], MODEL["decision"])) print("\nthe best of the same 135-candidate search, on the training periods:\n" + HEADER) show("label m3 > 40", train(S, FEATS, rule=old_rule)) show("label subscriber_mean + 6 x commercial > 30", MODEL) ``` ``` on the test periods (2026-10 .. 2026-12): setup weights / decision accuracy recall specificity balanced old : accuracy criterion (0.6, 0.4, 0) / 40 0.8067 0.0000 1.0000 0.5000 this course : balanced accuracy criterion (0.6, 0.0, 4) / 18 0.9833 1.0000 0.9670 0.9835 the best of the same 135-candidate search, on the training periods: setup weights / decision accuracy recall specificity balanced label m3 > 40 (0.6, 0.2, 4) / 22 0.5075 0.5422 0.4984 0.5203 label subscriber_mean + 6 x commercial > 30 (0.6, 0.0, 4) / 18 0.9838 1.0000 0.9728 0.9864 ``` The 0.8067 in the upper table is not wrong; what it measures is something else. When the decision threshold is kept the same as the label threshold and the criterion is accuracy, the search stops at the model that **says high for no row at all** on the imbalanced set: **recall 0.0000**, balanced accuracy **0.5000**, that is, coin-flip level. The indicator of such a model does not move even if it degrades in production, because its output is already constant. In the corrected setup recall is **1.0000**, specificity **0.9670** and balanced accuracy **0.9835**. The lower table says why the third fix is needed. Even with the first two fixes in place, **the best of the 135 candidates stays at 0.5203** with the old label rule. The reason is structural: that rule looks at the consumption of the row in **its own period**, while the features see **only the previous periods**. There is no relation, so there is nothing to learn. Once the label is tied to a rule the features see, the same search gives **0.9864**. #### What Happens While the Input Drifts In data drift the input distribution changes and **the rule stays the same**. The fictional scenario is this: after `2026-09`, the consumption of commercial subscribers grows by a factor of 0.06 every period. The third block applies the drift and reads three indicators period by period: the **input summary** (the mean of the feature), the **prediction share** (the share of rows the model calls high) and **accuracy**. The last column is the accuracy the same model gives on **undrifted** data, that is, the no-drift run. ```python # ---- FICTIONAL data drift: after 2026-09 commercial consumption grows by 0.06 every period. # THE RULE STAYS THE SAME; only the input distribution changes. def drift(records, start="2026-09", slope=0.06): b = PERIODS.index(start) return [dict(k, m3=round(k["m3"] * (1.0 + slope * (PERIODS.index(k["period"]) - b + 1)), 4)) if PERIODS.index(k["period"]) >= b and k["type"] == "commercial" else dict(k) for k in records] def input_summary(records, period, feats): d = [feats[(k["subscriber"], k["period"])]["subscriber_mean"] for k in records if k["period"] == period] return round(sum(d) / len(d), 4) def prediction_share(records, model, feats, period): d = [predict(feats[(k["subscriber"], k["period"])], model["weights"], model["decision"]) for k in records if k["period"] == period] return round(sum(d) / len(d), 4) def accuracy(records, model, feats, period, labels=None): """labels=None -> the rule has not changed.""" subset = [k for k in records if k["period"] == period] d = 0 for k in subset: f_k = feats[(k["subscriber"], k["period"])] g = labels[(k["subscriber"], k["period"])] if labels else label(f_k) d += predict(f_k, model["weights"], model["decision"]) == g return round(d / len(subset), 4) DRIFTED = drift(S) FEATS_D = features(DRIFTED) MEASURES = {"input summary": [(d, input_summary(DRIFTED, d, FEATS_D)) for d in MONITORED], "prediction share": [(d, prediction_share(DRIFTED, MODEL, FEATS_D, d)) for d in MONITORED], "accuracy": [(d, accuracy(DRIFTED, MODEL, FEATS_D, d)) for d in MONITORED]} print(f"{'period':<10}{'input summary':>15}{'prediction share':>18}{'accuracy':>10}{'no-drift accuracy':>19}") for i, d in enumerate(MONITORED): print(f"{d:<10}{MEASURES['input summary'][i][1]:>15.4f}{MEASURES['prediction share'][i][1]:>18.4f}" f"{MEASURES['accuracy'][i][1]:>10.4f}{accuracy(S, MODEL, FEATS, d):>19.4f}") ``` ``` period input summary prediction share accuracy no-drift accuracy 2026-07 25.9220 0.4550 0.9700 0.9700 2026-08 26.0517 0.4700 0.9700 0.9700 2026-09 25.9800 0.4550 0.9900 0.9900 2026-10 26.4685 0.5150 0.9750 0.9750 2026-11 26.6053 0.5200 0.9800 0.9850 2026-12 27.0472 0.5450 0.9800 0.9900 ``` The input really does drift: the input summary rises **from 25.9220 to 27.0472** and the prediction share **from 0.4550 to 0.5450**, so the share of inferences that say high grows by about twenty percent. Accuracy, in contrast, **never falls below the 0.9700 baseline** in any period. What the drift actually took away is measured by the no-drift column: in `2026-12` the no-drift run gives 0.9900 and the drifted run 0.9800; the difference is **0.0100**, two resolution steps. As long as the rule does not change, a correct model stays correct on drifted input, because both the label and the prediction are computed from the same drifted features. #### A Firing Alert Is Not Evidence The fourth block attaches an alert band to the indicators. The band is the ratio of the deviation from the baseline; the baseline is `2026-08`, the last period before the one in which the drift starts. The same three bands are applied to all three indicators in the same way. ```python # ---- Alert: a measure fires if it deviates from the baseline by more than `share`. The band is two-sided. def alert(series, baseline, share): return [d for d, v in series if abs(v - baseline) / max(abs(baseline), 1e-9) > share] BASELINE_PERIOD = MONITORED[1] # 2026-08 : the last period before the one in which the drift starts ACCURACY = dict(MEASURES["accuracy"]) print(f"{'measure':<18}{'baseline':>10}{'largest deviation':>19}{'0.05':>7}{'0.10':>7}" f"{'0.20':>7}{'first firing':>16}") for name, series in MEASURES.items(): base = dict(series)[BASELINE_PERIOD] fires = [alert(series, base, p) for p in (0.05, 0.10, 0.20)] deviation = max(abs(v - base) / base for _, v in series) print(f"{name:<18}{base:>10.4f}{deviation:>19.4f}{len(fires[0]):>7}{len(fires[1]):>7}" f"{len(fires[2]):>7}{(fires[0][0] if fires[0] else '-'):>16}") FIRED = alert(MEASURES["prediction share"], dict(MEASURES["prediction share"])[BASELINE_PERIOD], 0.05) REAL = [d for d in FIRED if ACCURACY[d] < ACCURACY[BASELINE_PERIOD]] print("\nperiods firing in the 0.05 band:", FIRED) print("how many of them had accuracy fall below the baseline:", len(REAL), " false positive:", len(FIRED) - len(REAL)) print("total swing of accuracy:", f"{max(ACCURACY.values()) - min(ACCURACY.values()):.4f}", " step of period accuracy:", f"{1 / 200:.4f}", " what the drift cost in 2026-12:", f"{accuracy(S, MODEL, FEATS, MONITORED[-1]) - ACCURACY[MONITORED[-1]]:.4f}") ``` ``` measure baseline largest deviation 0.05 0.10 0.20 first firing input summary 26.0517 0.0382 0 0 0 - prediction share 0.4700 0.1596 3 2 0 2026-10 accuracy 0.9700 0.0206 0 0 0 - periods firing in the 0.05 band: ['2026-10', '2026-11', '2026-12'] how many of them had accuracy fall below the baseline: 0 false positive: 3 total swing of accuracy: 0.0200 step of period accuracy: 0.0050 what the drift cost in 2026-12: 0.0100 ``` The same drift, the same band, three different answers. The largest deviation of the input summary is **0.0382** and it fires in no band, because it is a mean and a mean smooths out the change in individual rows. The deviation of the prediction share is **0.1596**; it fires in **three periods** in the 0.05 band and in two in the 0.10 band, because it is sensitive to the rows **around** the decision threshold. The deviation of accuracy is **0.0206**, it does not fire, and moreover it is in the **upward** direction. The rule of the course can be read from here. The alert in `2026-10` did not work wrongly: the input really has drifted. But the only thing it says is **that the input has drifted**; it does not say that anything has degraded. In none of the three firing periods did accuracy fall below the baseline, and all three count as **false positives**. The firing of an indicator is not evidence of degradation; **an indicator whose firing and whose degradation are not shown separately counts as unmeasured**. The unmonitored baseline lands in a curious position here: the setup that attaches no indicator has never been wrong, because it has no alert that could be wrong. #### Summary - A monitorable model depends on three settings: the decision threshold is a setting **separate** from the label threshold and is searched in training, the training criterion is **balanced accuracy**, and the label is tied to a rule the features **see**. - In the old setup recall is **0.0000** and balanced accuracy **0.5000**; in the corrected one recall is **1.0000**, specificity **0.9670**, balanced accuracy **0.9835** and the base class ratio **0.5342**. - The first two fixes alone are not enough: with the old label rule the best of the 135 candidates stays at **0.5203**, coin-flip level, because that rule looks at a number the features do not see. - Under data drift the input summary rises **from 25.9220 to 27.0472** and the prediction share **from 0.4550 to 0.5450**, but accuracy goes **from 0.9700 to 0.9800**; what the drift took away is **0.0100** in `2026-12`, two resolution steps. - The prediction share fires in **three periods** in the 0.05 band and in none of the three did accuracy fall below the baseline: **a firing alert is not evidence of a problem**. #### Next Step In this lesson what drifted was **the input**, and because the rule stayed in place the model could stay correct; the firing indicator saw exactly this. But what if it were not the input that changed but **the rule itself**, that is, if the same features began to produce a different label: what would the same three indicators say. The input summary would not move, because the input has not moved; the prediction share would not move either, because the model keeps giving the same answer to the same inputs. The next lesson measures this case and finds the opposite result: while the indicators stay silent, accuracy falls **from 0.9700 to 0.8150**. ### Concept Drift Source: https://academia.sh/en/courses/model-monitoring/concept-drift Measuring a change of the rule between input and label while the input does not change at all: when the threshold for counting as high drops from 30 to 27, not a single value changes on the input side, yet the label of 132 of the 1200 monitoring rows changes and the accuracy of the model falls from 0.9700 to 0.8150, that is, thirty-one resolution steps. When the same three indicators are read side by side under the two kinds of drift, what the proxy metrics see and what they miss turns into a number: the input summary fires in no band in either scenario, the prediction share fires in three periods in the 0.05 band in both and so cannot tell the harmless from the harmful, and in the 0.10 band it fires in two periods under the harmless drift and in none under the harmful one, while accuracy stays silent under the harmless drift and fires in four periods under the harmful one, remaining the only measure that tells them apart. In the previous lesson what drifted was **the input**: the input summary rose from 25.9220 to 27.0472, the prediction share fired an alert in three periods, and accuracy never fell below the 0.9700 baseline. Because the rule stayed in place, the model could stay correct, and all three of the alerts that fired counted as false positives. This lesson builds the exact opposite. On the input side **not a single value changes**; what changes is **the rule itself** between input and label. The same history of the same subscriber now begins to produce a different label, because the criterion for "counting as high" has moved. The model stays where it is, untouched, and the question to measure is this: what do the three indicators of the previous lesson say in this case. - **MO8.** The source, the pipeline, the model and the drift scenarios are **fictional**; no real learning library, monitoring product or cloud service is called. The seed is **20260218**. - **MO9.** **Concept drift** is a change of the relationship between input and label. **Data drift** moves the input distribution, concept drift moves **the rule**. The measurement procedure for drift was built in the Model Evaluation, Interpretability and Ethics course and is **not repeated**; the addition here is only **the life in production**. - **MO10.** This course's label is the rule `subscriber_mean + 6 x commercial > 30`, and **the rule is the CONCEPT**. The fictional concept drift lowers the threshold **from 30 to 27** starting in `2026-09`; it does not touch the records, it only produces labels. - **MO11.** The monitoring periods are **2026-07…2026-12**, **200 rows** per period, **1200** in total. In the monitoring periods the **base class ratio is 0.5342**; the set is balanced. - **MO12.** Three indicators: the **input summary** (the period mean of the feature), the **prediction share** (the share of rows the model calls high) and **accuracy**. The first two do not ask for a label, the third does. - **MO13.** The baseline is the **unmonitored setup**: no indicator, no alert; the degradation goes **unseen**. - **MO14.** The **alert threshold** is a two-sided **band**: a measure fires if it deviates from the baseline by more than the band width. The baseline period is `2026-08`, that is, the last period before the one in which the drift starts. - **MO15.** Resolution: the step of period accuracy is **1/200 = 0.0050**. In this lesson the drop is **0.1550**, that is, **thirty-one steps** and far above the band; the swing in the previous lesson was four steps. #### Unchanged Input, Changed Rule The first block builds the setup of the previous lesson exactly: the source, the three stages of the pipeline that look at the model, the features, the model whose decision threshold is searched separately, and the three indicators. The model is **not retrained at all in this lesson**; what is measured is the rule being pulled out from under a fixed model. ```python # FICTIONAL. The meter readings of the measurement network, the first three stages # of the pipeline that look at the model, the threshold-based model and the drift # scenarios are defined inside the lesson; no real learning library, monitoring # product or cloud service is called. SEED, M32 = 20260218, 0xFFFFFFFF def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value REGION = ["east", "west", "north", "south", "central"] TYPE = ["residential", "commercial"] PERIODS = [f"2026-{a:02d}" for a in range(1, 13)] SUBSCRIBERS = {} for i in range(200): r = rng(SEED + 37 * i) SUBSCRIBERS[f"A{i + 1:03d}"] = {"region": REGION[int(r() * 5)], "type": TYPE[int(r() * 2)], "start": 10000 + int(r() * 60000)} def generate_source(): records = [] for i, (sub, a) in enumerate(SUBSCRIBERS.items()): e = a["start"] for p in PERIODS: r = rng(SEED + 101 * i + 7 * PERIODS.index(p)) e += 4 + int(r() * 46) records.append({"subscriber": sub, "period": p, "region": a["region"], "type": a["type"], "index": e, "unit": "m3"}) return records def s1_screen(records): REQUIRED = ("subscriber", "period", "region", "index", "unit") keep = [k for k in records if all(a in k for a in REQUIRED)] return keep, len(records) - len(keep) def s2_dedupe(records): seen, keep = set(), [] for k in records: key = (k["subscriber"], k["period"]) if key in seen: continue seen.add(key) keep.append(k) return keep, len(records) - len(keep) def s3_consumption(records): """Consumption from the period difference. Without a previous period the row cannot be computed.""" table = {(k["subscriber"], k["period"]): k for k in records} out, dropped = [], 0 for k in records: i = PERIODS.index(k["period"]) if i == 0: continue prev = table.get((k["subscriber"], PERIODS[i - 1])) if prev is None: dropped += 1 continue out.append(dict(k, m3=int(k["index"]) - int(prev["index"]))) return out, dropped def rows(): v, _ = s1_screen(generate_source()) v, _ = s2_dedupe(v) v, _ = s3_consumption(v) return v def features(records, digits=4): """Features are independent of the weights; computed ONCE from the full history.""" ordered = sorted(records, key=lambda k: PERIODS.index(k["period"])) sub, reg = {}, {} for k in ordered: sub.setdefault(k["subscriber"], []).append((PERIODS.index(k["period"]), k["m3"])) reg.setdefault(k["region"], []).append((PERIODS.index(k["period"]), k["m3"])) def mean(series, i): chosen = [v for d, v in series if d < i] return round(sum(chosen) / len(chosen), digits) if chosen else 0.0 return {(k["subscriber"], k["period"]): { "subscriber_mean": mean(sub[k["subscriber"]], PERIODS.index(k["period"])), "region_mean": mean(reg[k["region"]], PERIODS.index(k["period"])), "commercial": 1 if k["type"] == "commercial" else 0} for k in records} GRID = [(a, b, c) for a in (0.6, 0.8, 1.0) for b in (0.0, 0.2, 0.4) for c in (0, 4, 8)] def score(o, w): return w[0] * o["subscriber_mean"] + w[1] * o["region_mean"] + w[2] * o["commercial"] # The label depends on a rule the features SEE. RULE = CONCEPT. RULE_THRESHOLD = 30 RULE_COMMERCIAL = 6 def label(feat_k, threshold=RULE_THRESHOLD, commercial_factor=RULE_COMMERCIAL): return 1 if feat_k["subscriber_mean"] + commercial_factor * feat_k["commercial"] > threshold else 0 MONITORED = PERIODS[6:] # 2026-07 .. 2026-12 : the periods monitored in production DECISION_GRID = (18, 22, 26, 30, 34) # the decision threshold is a SEPARATE setting from the label threshold def predict(feat_k, w, decision): return 1 if score(feat_k, w) > decision else 0 def balanced_accuracy(subset, feats, w, decision, rule=None): c = {(0, 0): 0, (0, 1): 0, (1, 0): 0, (1, 1): 0} for k in subset: f_k = feats[(k["subscriber"], k["period"])] c[(predict(f_k, w, decision), rule(k, f_k) if rule else label(f_k))] += 1 rec = c[(1, 1)] / max(c[(1, 1)] + c[(0, 1)], 1) # recall spec = c[(0, 0)] / max(c[(0, 0)] + c[(1, 0)], 1) # specificity return {"accuracy": (c[(1, 1)] + c[(0, 0)]) / len(subset), "recall": rec, "specificity": spec, "balanced": (rec + spec) / 2, "weights": w, "decision": decision} def train(records, feats, periods=None, rule=None): """Training = searching the weight TRIPLE together with the decision threshold: 27 x 5 = 135 candidates.""" subset = [k for k in records if k["period"] in (periods or PERIODS[1:9])] return max((balanced_accuracy(subset, feats, w, decision, rule) for w in GRID for decision in DECISION_GRID), key=lambda o: o["balanced"]) def input_summary(records, period, feats): d = [feats[(k["subscriber"], k["period"])]["subscriber_mean"] for k in records if k["period"] == period] return round(sum(d) / len(d), 4) def prediction_share(records, model, feats, period): d = [predict(feats[(k["subscriber"], k["period"])], model["weights"], model["decision"]) for k in records if k["period"] == period] return round(sum(d) / len(d), 4) def accuracy(records, model, feats, period, labels=None): """labels=None -> the rule has not changed.""" subset = [k for k in records if k["period"] == period] d = 0 for k in subset: f_k = feats[(k["subscriber"], k["period"])] g = labels[(k["subscriber"], k["period"])] if labels else label(f_k) d += predict(f_k, model["weights"], model["decision"]) == g return round(d / len(subset), 4) S = rows() FEATS = features(S) MODEL = train(S, FEATS) MON = [k for k in S if k["period"] in MONITORED] dist = {0: 0, 1: 0} for k in MON: dist[label(FEATS[(k["subscriber"], k["period"])])] += 1 print("monitoring rows:", len(MON), " per period:", len(MON) // len(MONITORED), " base class ratio:", round(dist[0] / len(MON), 4)) print("model:", MODEL["weights"], "/ decision threshold", MODEL["decision"], " balanced accuracy:", round(MODEL["balanced"], 4)) ``` ``` monitoring rows: 1200 per period: 200 base class ratio: 0.5342 model: (0.6, 0.0, 4) / decision threshold 18 balanced accuracy: 0.9864 ``` #### What Shows Up When the Concept Drifts The second block applies the concept drift. The `concept_drift` function **does not touch the records**; it only produces labels in which, starting from `2026-09`, the threshold for counting as high has been lowered to 27. The last two columns of the table matter: the accuracy that would be taken in that period **had the rule not changed**, and the number of rows whose **label changed** in that period. ```python # ---- FICTIONAL concept drift: THE INPUT STAYS THE SAME , the rule changes. # After 2026-09 the threshold for "counting as high" drops from 30 to 27. def concept_drift(records, feats, start="2026-09", new_threshold=27): b = PERIODS.index(start) return {(k["subscriber"], k["period"]): label(feats[(k["subscriber"], k["period"])], new_threshold if PERIODS.index(k["period"]) >= b else RULE_THRESHOLD) for k in records} NEW = concept_drift(S, FEATS) print(f"{'period':<10}{'input summary':>15}{'prediction share':>18}{'accuracy':>10}" f"{'if rule unchanged':>19}{'label changed':>15}") for d in MONITORED: subset = [k for k in S if k["period"] == d] flipped = sum(1 for k in subset if NEW[(k["subscriber"], d)] != label(FEATS[(k["subscriber"], d)])) print(f"{d:<10}{input_summary(S, d, FEATS):>15.4f}{prediction_share(S, MODEL, FEATS, d):>18.4f}" f"{accuracy(S, MODEL, FEATS, d, NEW):>10.4f}{accuracy(S, MODEL, FEATS, d):>19.4f}" f"{flipped:>15}") ``` ``` period input summary prediction share accuracy if rule unchanged label changed 2026-07 25.9220 0.4550 0.9700 0.9700 0 2026-08 26.0517 0.4700 0.9700 0.9700 0 2026-09 25.9800 0.4550 0.8150 0.9900 39 2026-10 26.3525 0.5100 0.8700 0.9750 31 2026-11 26.3261 0.5100 0.8650 0.9850 30 2026-12 26.5325 0.5150 0.8500 0.9900 32 ``` Accuracy falls in `2026-09` **from 0.9700 to 0.8150**. The drop is **0.1550**, that is, **thirty-one steps** of resolution; the whole swing of the previous lesson was four steps. There is not a single number that changed in the model, and none on the input side either: the last column says where the drift comes from, and in `2026-09` the label of **39** rows has changed. Not all of the changed labels turn into errors, because the decision threshold of the model is already consistent with the new rule on some rows; in the following periods accuracy holds around 0.8700 but **never returns to its old level**. The first two columns behave exactly as expected. The input summary goes from 25.9220 to 26.5325 and the prediction share from 0.4550 to 0.5150. These two series would have been exactly the same had the concept drift **never happened**, because both look only at the input and the model; they have no knowledge of where the rule has moved. #### Firing Under the Harmless Drift, Silent Under the Harmful One The third block brings back the data drift scenario of the previous lesson and puts the two scenarios side by side **with the same three indicators and the same three bands**. What is to be read in the table is not the individual numbers but **the standing of the rows relative to each other**. ```python # ---- FICTIONAL data drift (previous lesson): the INPUT drifts , THE RULE STAYS THE SAME. def drift(records, start="2026-09", slope=0.06): b = PERIODS.index(start) return [dict(k, m3=round(k["m3"] * (1.0 + slope * (PERIODS.index(k["period"]) - b + 1)), 4)) if PERIODS.index(k["period"]) >= b and k["type"] == "commercial" else dict(k) for k in records] def alert(series, baseline, share): return [d for d, v in series if abs(v - baseline) / max(abs(baseline), 1e-9) > share] DRIFTED = drift(S) FEATS_D = features(DRIFTED) print(f"{'scenario':<16}{'measure':<18}{'baseline':>10}{'largest deviation':>19}" f"{'0.05':>7}{'0.10':>7}{'0.20':>7}") for name, (src, feats_s, lab) in (("data drift", (DRIFTED, FEATS_D, None)), ("concept drift", (S, FEATS, NEW))): for measure, series in (("input summary", [(d, input_summary(src, d, feats_s)) for d in MONITORED]), ("prediction share", [(d, prediction_share(src, MODEL, feats_s, d)) for d in MONITORED]), ("accuracy", [(d, accuracy(src, MODEL, feats_s, d, lab)) for d in MONITORED])): base = dict(series)[MONITORED[1]] print(f"{name:<16}{measure:<18}{base:>10.4f}" f"{max(abs(v - base) / base for _, v in series):>19.4f}" + "".join(f"{len(alert(series, base, p)):>7}" for p in (0.05, 0.10, 0.20))) print("\nrows changed under concept drift:", sum(1 for k in MON if NEW[(k["subscriber"], k["period"])] != label(FEATS[(k["subscriber"], k["period"])])), "/", len(MON), " values changed on the input side: 0") DROP = accuracy(S, MODEL, FEATS, MONITORED[0], NEW) - accuracy(S, MODEL, FEATS, MONITORED[2], NEW) print("drop in accuracy:", f"{DROP:.4f}", " step of period accuracy:", f"{1 / 200:.4f}", " that is", round(DROP * 200), "steps") ``` ``` scenario measure baseline largest deviation 0.05 0.10 0.20 data drift input summary 26.0517 0.0382 0 0 0 data drift prediction share 0.4700 0.1596 3 2 0 data drift accuracy 0.9700 0.0206 0 0 0 concept drift input summary 26.0517 0.0185 0 0 0 concept drift prediction share 0.4700 0.0957 3 0 0 concept drift accuracy 0.9700 0.1598 4 4 0 rows changed under concept drift: 132 / 1200 values changed on the input side: 0 drop in accuracy: 0.1550 step of period accuracy: 0.0050 that is 31 steps ``` The input summary fires in no band in either scenario. Under concept drift it could not be otherwise: there are **zero** values changed on the input side, and the indicator looks only at the input. The prediction share fires in both and misleads in both. In the 0.05 band it fires in **three periods in each of the two scenarios**, that is, it **cannot tell** the harmless drift from the harmful one. In the 0.10 band it works exactly the other way round: it fires in **two periods** under the harmless drift and in **none** under the harmful one. The deviation of 0.0957 under concept drift has nothing to do with the drift either; that series would have been the same had the rule never changed. When an indicator fires, on its own it shows neither the degradation nor its cause. Accuracy is the only measure that tells them apart. Under the harmless drift its deviation is **0.0206** and it fires in no band; under the harmful drift its deviation is **0.1598** and it fires in **four periods** in both the 0.05 and the 0.10 band. The rule of this course is read in both directions here: as the firing of an indicator is not evidence of degradation, **its silence is not evidence of soundness either**; an indicator whose firing and whose degradation are not shown separately counts as unmeasured. In the unmonitored baseline, the label of 132 rows has changed, accuracy has fallen thirty-one steps, and nothing has given notice. #### Summary - Concept drift is a change of the **rule** between input and label; unlike data drift, not a single value changes on the input side. - When the threshold drops from 30 to 27, the label of **132** of the 1200 monitoring rows changes and accuracy falls **from 0.9700 to 0.8150**: **0.1550**, that is, **thirty-one resolution steps**. The model has not changed at all. - The input summary fires in no band in either scenario; under concept drift its firing is impossible anyway, because nothing changes where it looks. - The prediction share fires in **three periods** in the 0.05 band in both scenarios and so cannot tell them apart; in the 0.10 band it fires in two periods under the harmless drift and in **none** under the harmful one. - Accuracy is the only measure that tells them apart: under the harmless drift its deviation is 0.0206 and it stays silent, under the harmful drift 0.1598 and it fires in four periods in the 0.05 and 0.10 bands. #### Next Step The conclusion that has emerged over two lessons is this: the only indicator that sees the degradation is **accuracy**. But accuracy asks for something different from the other two, the true label. The input summary and the prediction share can be computed the moment the prediction is produced; accuracy cannot be computed until it is known whether the row really is high, and that information does not arrive in production at the same time as the prediction. In this lesson the label was used as if it were at hand at calculation time. The next lesson removes this assumption: when the label arrives **two periods later**, it counts in which period the accuracy of a period can be known, the gap between the moment of degradation and the moment it is seen, and how many predictions are produced in that gap **unmeasured**. ### Performance Monitoring Source: https://academia.sh/en/courses/model-monitoring/performance-monitoring Measuring when performance can be known in a production setup where the true label does not arrive together with the prediction: because the accuracy of a period can only be computed two periods later, the drop in accuracy that concept drift caused in 2026-09 can be seen for the first time in 2026-11, and 400 predictions are produced unmeasured in this gap. In 2026-09 the actual accuracy is 0.8150 while the newest measure at hand is the 0.9700 of 2026-07, so the indicator is not wrong, it is old. The delay sweep gives a linear cost: at a delay of 0 periods the degradation is seen in the same period and the unmeasured predictions are 0, at 1 period 200, at 2 periods 400, at 3 periods 600, and at 4 periods the degradation is never seen inside the monitoring window; when no label is ever collected, the number of measurable periods is 0 and the degradation is never seen. The conclusion that emerged over two lessons was this: the only indicator that sees concept drift is **accuracy**. The input summary knows nothing about the rule having moved, the prediction share fires under the harmless drift and stays silent under the harmful one, and accuracy shows the degradation by name, falling from 0.9700 to 0.8150. This finding makes an assumption silently. To compute accuracy, it must be **known** whether the row really is high, and in the previous two lessons that information was treated as always at hand. In production it is not: the prediction is produced at the start of the period, and the true label arrives only after the meter has been read, the objection period has passed and the record has been closed. This lesson puts that delay inside the setup and measures a single question: **the accuracy of which period can be known in which period.** - **MO16.** The source, the pipeline, the model and the concept drift are **fictional**; no real learning library, monitoring product or cloud service is called. The seed is **20260218**. - **MO17.** A **delayed label** is a true label that arrives after the prediction. Here the delay is `DELAY = 2` **periods**: the accuracy of a period can be computed only two periods later. **The time is written in periods, not in a raw unit.** - **MO18.** The **detection lag** is the difference between the period in which the degradation happens and the period in which it can first be seen. The measure of **how many periods later an indicator shows** was built in the Data Pipelines and Orchestration course; it is **not repeated** here, it is taken as input. - **MO19.** The source of the degradation is the concept drift of the previous lesson: starting in `2026-09`, the threshold for counting as high drops from 30 to 27. There is no changed value on the input side. - **MO20.** The monitoring periods are **2026-07…2026-12**, **200 rows** per period, **1200** in total. In the monitoring periods the **base class ratio is 0.5342**; the set is balanced. - **MO21.** The alert band is **0.05** and the baseline is the accuracy of `2026-07`, **0.9700**. The band is two-sided. - **MO22.** The baseline is the **unmonitored setup**: no label is ever collected, the measurable periods are **0**, and the degradation is **never seen**. - **MO23.** Resolution: the step of period accuracy is **1/200 = 0.0050**; the drop caused by the concept drift is **0.1550**, that is, thirty-one steps and far above the band. The delay is counted in **periods**, and one period means **200 predictions**. #### The Label Arrives Two Periods Later The first block builds the setup of the previous two lessons and adds a single thing on top: `period_seen`. This function says whether the accuracy of a period **can be computed at a given moment**; the computation itself does not change, only **when it can be done** changes. ```python # FICTIONAL. The meter readings of the measurement network, the first three stages # of the pipeline that look at the model, the threshold-based model, the concept # drift and the label delay are defined inside the lesson; no real learning # library, monitoring product or cloud service is called. SEED, M32 = 20260218, 0xFFFFFFFF def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value REGION = ["east", "west", "north", "south", "central"] TYPE = ["residential", "commercial"] PERIODS = [f"2026-{a:02d}" for a in range(1, 13)] SUBSCRIBERS = {} for i in range(200): r = rng(SEED + 37 * i) SUBSCRIBERS[f"A{i + 1:03d}"] = {"region": REGION[int(r() * 5)], "type": TYPE[int(r() * 2)], "start": 10000 + int(r() * 60000)} def generate_source(): records = [] for i, (sub, a) in enumerate(SUBSCRIBERS.items()): e = a["start"] for p in PERIODS: r = rng(SEED + 101 * i + 7 * PERIODS.index(p)) e += 4 + int(r() * 46) records.append({"subscriber": sub, "period": p, "region": a["region"], "type": a["type"], "index": e, "unit": "m3"}) return records def s1_screen(records): REQUIRED = ("subscriber", "period", "region", "index", "unit") keep = [k for k in records if all(a in k for a in REQUIRED)] return keep, len(records) - len(keep) def s2_dedupe(records): seen, keep = set(), [] for k in records: key = (k["subscriber"], k["period"]) if key in seen: continue seen.add(key) keep.append(k) return keep, len(records) - len(keep) def s3_consumption(records): """Consumption from the period difference. Without a previous period the row cannot be computed.""" table = {(k["subscriber"], k["period"]): k for k in records} out, dropped = [], 0 for k in records: i = PERIODS.index(k["period"]) if i == 0: continue prev = table.get((k["subscriber"], PERIODS[i - 1])) if prev is None: dropped += 1 continue out.append(dict(k, m3=int(k["index"]) - int(prev["index"]))) return out, dropped def rows(): v, _ = s1_screen(generate_source()) v, _ = s2_dedupe(v) v, _ = s3_consumption(v) return v def features(records, digits=4): """Features are independent of the weights; computed ONCE from the full history.""" ordered = sorted(records, key=lambda k: PERIODS.index(k["period"])) sub, reg = {}, {} for k in ordered: sub.setdefault(k["subscriber"], []).append((PERIODS.index(k["period"]), k["m3"])) reg.setdefault(k["region"], []).append((PERIODS.index(k["period"]), k["m3"])) def mean(series, i): chosen = [v for d, v in series if d < i] return round(sum(chosen) / len(chosen), digits) if chosen else 0.0 return {(k["subscriber"], k["period"]): { "subscriber_mean": mean(sub[k["subscriber"]], PERIODS.index(k["period"])), "region_mean": mean(reg[k["region"]], PERIODS.index(k["period"])), "commercial": 1 if k["type"] == "commercial" else 0} for k in records} GRID = [(a, b, c) for a in (0.6, 0.8, 1.0) for b in (0.0, 0.2, 0.4) for c in (0, 4, 8)] def score(o, w): return w[0] * o["subscriber_mean"] + w[1] * o["region_mean"] + w[2] * o["commercial"] RULE_THRESHOLD = 30 RULE_COMMERCIAL = 6 def label(feat_k, threshold=RULE_THRESHOLD, commercial_factor=RULE_COMMERCIAL): return 1 if feat_k["subscriber_mean"] + commercial_factor * feat_k["commercial"] > threshold else 0 MONITORED = PERIODS[6:] # 2026-07 .. 2026-12 : the periods monitored in production DECISION_GRID = (18, 22, 26, 30, 34) # the decision threshold is a SEPARATE setting from the label threshold def predict(feat_k, w, decision): return 1 if score(feat_k, w) > decision else 0 def balanced_accuracy(subset, feats, w, decision, rule=None): c = {(0, 0): 0, (0, 1): 0, (1, 0): 0, (1, 1): 0} for k in subset: f_k = feats[(k["subscriber"], k["period"])] c[(predict(f_k, w, decision), rule(k, f_k) if rule else label(f_k))] += 1 rec = c[(1, 1)] / max(c[(1, 1)] + c[(0, 1)], 1) # recall spec = c[(0, 0)] / max(c[(0, 0)] + c[(1, 0)], 1) # specificity return {"accuracy": (c[(1, 1)] + c[(0, 0)]) / len(subset), "recall": rec, "specificity": spec, "balanced": (rec + spec) / 2, "weights": w, "decision": decision} def train(records, feats, periods=None, rule=None): """Training = searching the weight TRIPLE together with the decision threshold: 27 x 5 = 135 candidates.""" subset = [k for k in records if k["period"] in (periods or PERIODS[1:9])] return max((balanced_accuracy(subset, feats, w, decision, rule) for w in GRID for decision in DECISION_GRID), key=lambda o: o["balanced"]) def accuracy(records, model, feats, period, labels=None): """labels=None -> the rule has not changed.""" subset = [k for k in records if k["period"] == period] d = 0 for k in subset: f_k = feats[(k["subscriber"], k["period"])] g = labels[(k["subscriber"], k["period"])] if labels else label(f_k) d += predict(f_k, model["weights"], model["decision"]) == g return round(d / len(subset), 4) # ---- FICTIONAL concept drift (previous lesson): THE INPUT STAYS THE SAME , the rule changes. def concept_drift(records, feats, start="2026-09", new_threshold=27): b = PERIODS.index(start) return {(k["subscriber"], k["period"]): label(feats[(k["subscriber"], k["period"])], new_threshold if PERIODS.index(k["period"]) >= b else RULE_THRESHOLD) for k in records} DELAY = 2 # after how many periods the label of a period reaches us def period_seen(period, now, delay=DELAY): """The accuracy of a period can be computed only `delay` periods later.""" return PERIODS.index(period) + delay <= PERIODS.index(now) S = rows() FEATS = features(S) MODEL = train(S, FEATS) NEW = concept_drift(S, FEATS) MON = [k for k in S if k["period"] in MONITORED] dist = {0: 0, 1: 0} for k in MON: dist[label(FEATS[(k["subscriber"], k["period"])])] += 1 ACTUAL = {d: accuracy(S, MODEL, FEATS, d, NEW) for d in MONITORED} print("monitoring rows:", len(MON), " per period:", len(MON) // len(MONITORED), " base class ratio:", round(dist[0] / len(MON), 4), " delay:", DELAY, "periods") print("actual accuracy:", [f"{ACTUAL[d]:.4f}" for d in MONITORED]) ``` ``` monitoring rows: 1200 per period: 200 base class ratio: 0.5342 delay: 2 periods actual accuracy: ['0.9700', '0.9700', '0.8150', '0.8700', '0.8650', '0.8500'] ``` The `ACTUAL` dictionary is the accuracy series that someone looking back later could see. Someone standing in production never holds this series in full; in each period they see only a **prefix** of it. #### Which Period's Accuracy Is Known in Which Period The second block builds the table in production for each period: the newest **measurable** period at that moment, the accuracy of that period, the **actual** accuracy at that moment, and the number of predictions at hand that are not yet **measured**. The third column is the knowledge of someone looking back, not of an observer; the person standing in production looks at the second column. ```python BAND, BASE = 0.05, ACTUAL[MONITORED[0]] print(f"{'now':<10}{'latest measurable period':>26}{'accuracy of that period':>25}" f"{'current actual accuracy':>25}{'unmeasured predictions':>24}") for now in MONITORED: measured = [d for d in MONITORED if period_seen(d, now)] produced = MONITORED[: MONITORED.index(now) + 1] latest = measured[-1] if measured else None print(f"{now:<10}{(latest or '-'):>26}" f"{(f'{ACTUAL[latest]:.4f}' if latest else '-'):>25}{ACTUAL[now]:>25.4f}" f"{(len(produced) - len(measured)) * 200:>24}") DEGRADED = next(d for d in MONITORED if abs(ACTUAL[d] - BASE) / BASE > BAND) SEEN = next(s for s in MONITORED if any(period_seen(d, s) and abs(ACTUAL[d] - BASE) / BASE > BAND for d in MONITORED)) print("\nperiod of degradation:", DEGRADED, " period in which the alert can fire:", SEEN, " detection lag:", PERIODS.index(SEEN) - PERIODS.index(DEGRADED), "periods") GAP = MONITORED[MONITORED.index(DEGRADED):MONITORED.index(SEEN)] print("predictions produced unmeasured in this gap:", len(GAP) * 200, " periods in the gap:", GAP) print("number later found wrong among them:", sum(round((1 - ACTUAL[d]) * 200) for d in GAP), " number had the rule not changed:", sum(round((1 - accuracy(S, MODEL, FEATS, d)) * 200) for d in GAP)) ``` ``` now latest measurable period accuracy of that period current actual accuracy unmeasured predictions 2026-07 - - 0.9700 200 2026-08 - - 0.9700 400 2026-09 2026-07 0.9700 0.8150 400 2026-10 2026-08 0.9700 0.8700 400 2026-11 2026-09 0.8150 0.8650 400 2026-12 2026-10 0.8700 0.8500 400 period of degradation: 2026-09 period in which the alert can fire: 2026-11 detection lag: 2 periods predictions produced unmeasured in this gap: 400 periods in the gap: ['2026-09', '2026-10'] number later found wrong among them: 63 number had the rule not changed: 7 ``` The period of degradation is `2026-09`, and the number on the screen in that period is **0.9700**. This number is not wrong; the accuracy of `2026-07` really was 0.9700. The number is only **old**, and if its age is written nowhere, it cannot be told apart from a wrong one. The same happens in the next period: while the actual accuracy has fallen to 0.8700, the newest measure at hand is still 0.9700. The first period in which the alert can fire is **2026-11**, because the labels of `2026-09` are completed only then. **The detection lag is two periods**, and in this gap **400 predictions** were produced unmeasured. It becomes clear later that **63** of these 400 predictions were wrong; had the rule not changed, the number of wrong ones in the same two periods would have been **7**. The 56 predictions in between are the ones produced and used during the time the degradation went unseen, and they cannot be corrected retroactively, because the decision has already been made. The second thing to notice is the last column: even after the gap closes, **400 predictions** remain unmeasured in every period. The delay is not a temporary start-up problem, it is a **debt carried continuously**; the indicator in production never shows the present, it always shows two periods ago. #### The Cost of Delay The third block sweeps the delay. The rest of the setup is fixed; the only thing that changes is how many periods later the label arrives. The last row is the unmonitored baseline: no label is ever collected. ```python AFTER = len(MONITORED) - MONITORED.index(DEGRADED) # number of periods after the degradation print(f"{'delay':<12}{'degradation seen':>18}{'detection lag':>19}" f"{'measurable periods':>20}{'produced unmeasured':>21}") for delay in (0, 1, 2, 3, 4): seen_at = next((s for s in MONITORED if any(period_seen(d, s, delay) and abs(ACTUAL[d] - BASE) / BASE > BAND for d in MONITORED)), None) lag = PERIODS.index(seen_at) - PERIODS.index(DEGRADED) if seen_at else AFTER print(f"{delay:<12}{(seen_at or 'not seen'):>18}{(lag if seen_at else '-'):>19}" f"{sum(1 for d in MONITORED if period_seen(d, MONITORED[-1], delay)):>20}{lag * 200:>21}") print(f"{'no label':<12}{'not seen':>18}{'-':>19}{0:>20}{AFTER * 200:>21}") ``` ``` delay degradation seen detection lag measurable periods produced unmeasured 0 2026-09 0 6 0 1 2026-10 1 5 200 2 2026-11 2 4 400 3 2026-12 3 3 600 4 not seen - 2 800 no label not seen - 0 800 ``` The cost is linear and its unit is clear: **every period of delay means 200 unmeasured predictions**. In the fourth row the delay runs into the end of the monitoring window; the degradation happened in `2026-09`, but because its label will arrive in `2027-01`, it is **never seen** inside the window. This row matters, because when the delay grows large enough the measure **keeps existing but becomes useless**. The last row is the unmonitored baseline and gives the same result as a delay of 4: the degradation is not seen, and 800 predictions are produced unmeasured. The only difference is in the number of measurable periods, **2** against **0**. The rule of the course applies here too: a indicator counts as measured only when it **can fire**, and a indicator whose firing falls outside the window gives an unmeasured assurance. #### Summary - Unlike the input summary and the prediction share, accuracy asks for the **true label**; the label does not arrive in production together with the prediction, and in this setup it arrives **two periods** later. - The degradation happens in `2026-09`, but the first period in which the alert can fire is **2026-11**: the detection lag is **two periods**, and in this gap **400 predictions** were produced unmeasured; it becomes clear later that **63** of them were wrong, and had the rule not changed the number would have been **7**. - In `2026-09` the number on the screen is 0.9700 and is not wrong, only **two periods old**; a indicator whose age is not written cannot be told apart from a wrong indicator. - After the gap closes, 400 predictions remain unmeasured in every period: the delay is not temporary, it is a **debt carried continuously**. - The sweep gives a linear cost: 0 unmeasured predictions at a delay of 0, 200 at 1, 400 at 2, 600 at 3; at 4 the degradation is never seen inside the monitoring window, and in the unmonitored baseline the number of measurable periods is **0**. #### Next Step Over three lessons the **seeing** of the degradation was measured, and the last measure left a gap: after the degradation happens, the only measure that sees it stays silent for **two periods**, and in that gap 400 predictions are produced unmeasured. What is at hand in this gap was never asked. The input summary and the prediction share are there; they are computed without waiting for the label, and in the previous lesson both were seen to be able to mislead. The next lesson treats these as **proxy metrics** and counts what each sees one by one: which proxy can see the concept drift, which one fires under the harmless drift, and whether the table changes when a third proxy such as input completeness is added. ### Proxy Metrics Source: https://academia.sh/en/courses/model-monitoring/proxy-metrics Counting the measures that remain at hand in the two periods before the label arrives: when the input summary, the prediction share and the input completeness are put side by side in three scenarios, the three proxies of the concept-drift setup come out digit for digit the same as the three proxies of the sound setup, because the rule has changed and the input has not. The prediction share deviates by 0.0957 even in the sound setup and exceeds the 0.05 band, and it rises to 0.1596 in the data-drift setup, yet in that setup accuracy goes from 0.9700 to 0.9800; the input summary stays at 0.0382 in the only scenario where the input really drifts and does not fire, because the drift touches 102 of 200 subscribers and the mean dilutes it; the input completeness deviates by 0.0000 in all three scenarios and shows that it works by dropping two periods at once to 0.8000 when the reading of a single period never arrives. The only measure that separates the scenarios is accuracy, and it arrives two periods later. The previous lesson counted the delayed label: the accuracy of a period is known only after `DELAY = 2` periods, there is a gap of two periods between the moment of degradation and the moment it is seen, and predictions keep being produced in that gap. Over three lessons it was measured that the degradation is seen **two periods late**. The question not asked is **what is at hand in those two periods**. There is no accuracy, because the label has not arrived. Only the measures that need no label remain: those that can be read from the input itself and from the output of the model. These are called **proxy metrics**, because they are not the thing we want to measure itself, but something **assumed** to be related to it. This lesson puts three proxies side by side in three scenarios and asks a single question: which one sees concept drift, and which one fires for something it does not see. - **MO41.** The source, the five-stage pipeline, the fictional model and the two drift types are **fictional** and are produced inside the lesson. No monitoring product, indicator service, cloud service or learning library is called. The seed is 20260218. - **MO42.** The three proxies are: the **input summary**, the mean of the subscriber means in the period; the **prediction share**, the share of rows the model calls 1; and the **input completeness**, the ratio of the rows formed in the period to the expected 200 rows. All three **read no label**, so they are free of delay. - **MO43.** Three scenarios: **sound**; **data drift**, where after `2026-09` commercial consumption grows by 0.06 per period; and **concept drift**, where the input stays the same and the threshold of the rule `subscriber_mean + 6 x commercial > 30` drops to 27. - **MO44.** The **drift** procedure, its kinds, their measurement and their correction, was measured in the Model Evaluation, Interpretability and Ethics course of the Machine Learning and Data Science curriculum; it is **not repeated**. The addition here belongs only to the life in production: what the proxy sees when there is no label. - **MO45.** The model is inherited from the MLOps Fundamentals course with its flaw corrected: the decision threshold is a separate setting and is searched in training, the training criterion is **balanced accuracy**, and the label is tied to a rule the features see. In the monitoring periods the **base class ratio is 0.5342**. - **MO46.** The baseline is the **unmonitored setup**: no indicator, no alert, the degradation goes unseen. The step of period accuracy is **0.0050**, and a difference below this step is not read. #### What Is at Hand Before the Label Arrives The block below builds the shared definition, which is the source, the pipeline, the features, the fictional model, the two drift types and the three measures, and trains the model and prints the baseline. The features are computed **once** per scenario. ```python # FICTIONAL. The meter readings of the measurement network , the five-stage pipeline , the fictional model and the two drift # types are produced inside the lesson; no monitoring product , indicator service , cloud service or # learning library is called. Seed 20260218. SEED, M32 = 20260218, 0xFFFFFFFF def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value REGION = ["east", "west", "north", "south", "central"] TYPE = ["residential", "commercial"] PERIODS = [f"2026-{a:02d}" for a in range(1, 13)] SUBSCRIBERS = {} for i in range(200): r = rng(SEED + 37 * i) SUBSCRIBERS[f"A{i + 1:03d}"] = {"region": REGION[int(r() * 5)], "type": TYPE[int(r() * 2)], "start": 10000 + int(r() * 60000)} def generate_source(): records = [] for i, (sub, a) in enumerate(SUBSCRIBERS.items()): e = a["start"] for p in PERIODS: r = rng(SEED + 101 * i + 7 * PERIODS.index(p)) e += 4 + int(r() * 46) records.append({"subscriber": sub, "period": p, "region": a["region"], "type": a["type"], "index": e, "unit": "m3"}) return records def s1_screen(records): REQUIRED = ("subscriber", "period", "region", "index", "unit") keep = [k for k in records if all(a in k for a in REQUIRED)] return keep, len(records) - len(keep) def s2_dedupe(records): seen, keep = set(), [] for k in records: key = (k["subscriber"], k["period"]) if key not in seen: seen.add(key) keep.append(k) return keep, len(records) - len(keep) def s3_consumption(records): table = {(k["subscriber"], k["period"]): k for k in records} out, dropped = [], 0 for k in records: i = PERIODS.index(k["period"]) prev = table.get((k["subscriber"], PERIODS[i - 1])) if i else None if prev is None: dropped += i > 0 continue out.append(dict(k, m3=int(k["index"]) - int(prev["index"]))) return out, dropped def rows(source=None): v, _ = s1_screen(generate_source() if source is None else source) v, _ = s2_dedupe(v) v, _ = s3_consumption(v) return v def features(records, window=None, digits=4): ordered = sorted(records, key=lambda k: PERIODS.index(k["period"])) sub, reg = {}, {} for k in ordered: sub.setdefault(k["subscriber"], []).append((PERIODS.index(k["period"]), k["m3"])) reg.setdefault(k["region"], []).append((PERIODS.index(k["period"]), k["m3"])) def mean(series, i): chosen = [v for d, v in series if (d < i and (window is None or d >= i - window))] return round(sum(chosen) / len(chosen), digits) if chosen else 0.0 feats = {} for k in records: i = PERIODS.index(k["period"]) feats[(k["subscriber"], k["period"])] = {"subscriber_mean": mean(sub[k["subscriber"]], i), "region_mean": mean(reg[k["region"]], i), "commercial": 1 if k["type"] == "commercial" else 0} return feats GRID = [(a, b, c) for a in (0.6, 0.8, 1.0) for b in (0.0, 0.2, 0.4) for c in (0, 4, 8)] DECISION_GRID = (18, 22, 26, 30, 34) # the decision threshold is a SEPARATE setting , searched in training MONITORED = PERIODS[6:] # 2026-07 .. 2026-12 : the periods monitored in production DELAY = 2 # after how many periods the label arrives RULE_THRESHOLD, RULE_COMMERCIAL = 30, 6 # CONCEPT: subscriber_mean + 6 x commercial > 30 def score(o, w): return w[0] * o["subscriber_mean"] + w[1] * o["region_mean"] + w[2] * o["commercial"] def label(feat_k, threshold=RULE_THRESHOLD, commercial_factor=RULE_COMMERCIAL): return 1 if feat_k["subscriber_mean"] + commercial_factor * feat_k["commercial"] > threshold else 0 def predict(feat_k, w, decision): return 1 if score(feat_k, w) > decision else 0 def balanced_accuracy(subset, feats, w, decision): tp = tn = fp = fn = 0 for k in subset: f_k = feats[(k["subscriber"], k["period"])] t, g = predict(f_k, w, decision), label(f_k) tp, fp = tp + (t and g), fp + (t and not g) fn, tn = fn + (g and not t), tn + (not t and not g) recall = tp / (tp + fn) if tp + fn else 0.0 specificity = tn / (tn + fp) if tn + fp else 0.0 return (recall + specificity) / 2, round(recall, 4), round(specificity, 4) def train(records, feats, periods=None): """The training criterion is BALANCED ACCURACY: the mean of recall and specificity.""" subset = [k for k in records if k["period"] in (periods or PERIODS[1:9])] best = (-1.0, None, None, 0.0, 0.0) for w in GRID: for decision in DECISION_GRID: o, rec, spec = balanced_accuracy(subset, feats, w, decision) if o > best[0]: best = (o, w, decision, rec, spec) return {"weights": best[1], "decision": best[2], "balanced": round(best[0], 4), "recall": best[3], "specificity": best[4], "train_rows": len(subset)} def drift(records, start="2026-09", slope=0.06): """DATA DRIFT: after the start, commercial consumption grows by slope every period.""" b = PERIODS.index(start) return [dict(k, m3=round(k["m3"] * (1.0 + slope * (PERIODS.index(k["period"]) - b + 1)), 4)) if PERIODS.index(k["period"]) >= b and k["type"] == "commercial" else dict(k) for k in records] def concept_drift(records, feats, start="2026-09", new_threshold=27): """CONCEPT DRIFT: the input is the SAME , the rule changes. The threshold drops from 30 to new_threshold.""" b = PERIODS.index(start) return {(k["subscriber"], k["period"]): label( feats[(k["subscriber"], k["period"])], new_threshold if PERIODS.index(k["period"]) >= b else RULE_THRESHOLD) for k in records} def input_summary(records, period, feats): d = [feats[(k["subscriber"], k["period"])]["subscriber_mean"] for k in records if k["period"] == period] return round(sum(d) / len(d), 4) if d else 0.0 def prediction_share(records, model, feats, period): d = [predict(feats[(k["subscriber"], k["period"])], model["weights"], model["decision"]) for k in records if k["period"] == period] return round(sum(d) / len(d), 4) if d else 0.0 def accuracy(records, model, feats, period, labels=None): subset = [k for k in records if k["period"] == period] d = sum(1 for k in subset if predict(feats[(k["subscriber"], k["period"])], model["weights"], model["decision"]) == (labels[(k["subscriber"], k["period"])] if labels else label(feats[(k["subscriber"], k["period"])]))) return round(d / len(subset), 4) if subset else None ALL_ROWS = rows() FEATS = features(ALL_ROWS) # features are computed ONCE MODEL = train(ALL_ROWS, FEATS) MON = [k for k in ALL_ROWS if k["period"] in MONITORED] largest = max(sum(1 for k in MON if label(FEATS[(k["subscriber"], k["period"])]) == s) for s in (0, 1)) print(f"model {MODEL['weights']} decision threshold {MODEL['decision']} , balanced accuracy" f" {MODEL['balanced']} , recall {MODEL['recall']} , specificity {MODEL['specificity']}") print(f"monitoring {len(MONITORED)} periods , {len(MON)} rows , commercial subscribers" f" {sum(1 for a in SUBSCRIBERS.values() if a['type'] == 'commercial')} , base class ratio" f" {largest / len(MON):.4f} , step of period accuracy {1 / 200:.4f}") ``` ``` model (0.6, 0.0, 4) decision threshold 18 , balanced accuracy 0.9864 , recall 1.0 , specificity 0.9728 monitoring 6 periods , 1200 rows , commercial subscribers 102 , base class ratio 0.5342 , step of period accuracy 0.0050 ``` Because the base class ratio is **0.5342**, the set is balanced: a procedure that always says the same class gets this ratio, and the 0.9864 of the model is far above it. The model is in a monitorable state. The last two numbers of the second line bound the whole lesson: the subscribers the drift touches number **102**, the remaining 98 are untouched; and because the step of period accuracy is **0.0050**, no movement smaller than this is read. #### Three Proxies, Three Scenarios The block below computes the three measures over the six monitoring periods of the three scenarios and adds accuracy as the fourth row. The last column is **the largest relative deviation** seen in the following four periods against the `2026-08` baseline; an alert band is compared with exactly this number. ```python # This block continues the definitions of the first block. The features of each scenario are computed ONCE. def completeness(records, period, expected=200): """THIRD PROXY: how many rows formed in the period. It reads no value , it counts presence.""" return round(sum(1 for k in records if k["period"] == period) / expected, 4) DRIFTED = drift(ALL_ROWS) FEATS_D = features(DRIFTED) CONCEPT = concept_drift(ALL_ROWS, FEATS) SCENARIOS = (("sound", ALL_ROWS, FEATS, None), ("data drift", DRIFTED, FEATS_D, None), ("concept drift", ALL_ROWS, FEATS, CONCEPT)) MEASURES = (("input summary", lambda src, feats, lab, d: input_summary(src, d, feats)), ("prediction share", lambda src, feats, lab, d: prediction_share(src, MODEL, feats, d)), ("completeness", lambda src, feats, lab, d: completeness(src, d)), ("accuracy", lambda src, feats, lab, d: accuracy(src, MODEL, feats, d, lab))) print("measure scenario " + "".join(f"{d[-2:]:>8}" for d in MONITORED) + " deviation") for name, f in MEASURES: for sname, src, feats, lab in SCENARIOS: series = [f(src, feats, lab, d) for d in MONITORED] deviation = max(abs(v - series[1]) / series[1] for v in series[2:]) print(f"{name:<16} {sname:<14}" + "".join(f"{v:8.4f}" for v in series) + f"{deviation:>11.4f}") RAW = [k for k in generate_source() if not (k["subscriber"] <= "A040" and k["period"] == "2026-10")] LATE = rows(RAW) print("\ncheck , the 2026-10 reading of 40 subscribers never arrived:") print("completeness " + "".join(f"{completeness(LATE, d):8.4f}" for d in MONITORED)) ``` ``` measure scenario 07 08 09 10 11 12 deviation input summary sound 25.9220 26.0517 25.9800 26.3525 26.3261 26.5325 0.0185 input summary data drift 25.9220 26.0517 25.9800 26.4685 26.6053 27.0472 0.0382 input summary concept drift 25.9220 26.0517 25.9800 26.3525 26.3261 26.5325 0.0185 prediction share sound 0.4550 0.4700 0.4550 0.5100 0.5100 0.5150 0.0957 prediction share data drift 0.4550 0.4700 0.4550 0.5150 0.5200 0.5450 0.1596 prediction share concept drift 0.4550 0.4700 0.4550 0.5100 0.5100 0.5150 0.0957 completeness sound 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.0000 completeness data drift 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.0000 completeness concept drift 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.0000 accuracy sound 0.9700 0.9700 0.9900 0.9750 0.9850 0.9900 0.0206 accuracy data drift 0.9700 0.9700 0.9900 0.9750 0.9800 0.9800 0.0206 accuracy concept drift 0.9700 0.9700 0.8150 0.8700 0.8650 0.8500 0.1598 check , the 2026-10 reading of 40 subscribers never arrived: completeness 1.0000 1.0000 1.0000 0.8000 0.8000 1.0000 ``` The hardest part of the table is the comparison of the **sound** and **concept drift** rows. In these two scenarios all three proxies give the **digit-for-digit same** number: 26.5325 and 26.5325, 0.5150 and 0.5150, 1.0000 and 1.0000. This is not an approximation, it is an identity. By definition, concept drift does not touch the input, it only changes the rule; and no measure read from the input can carry that change. In the same periods accuracy goes down from 0.9900 to 0.8500, and in the last column the deviation rises from 0.0206 to 0.1598. **None of the three proxies sees concept drift.** #### The Proxy That Fires and the Proxy That Stays Silent The prediction share row gives the second reading. Its deviation is **0.0957** even in the **sound** setup; that is, in a production where nothing has degraded, a band of 0.05 fires. In the data-drift setup it rises to 0.1596 and exceeds the 0.10 band too, and in exactly that setup accuracy **rises** from 0.9700 to 0.9800. The proxy gives its loudest alarm in the scenario where performance improves. This is the barest form of the rule of the course: the firing of an indicator is not evidence of degradation. The cause is visible too: as long as the rule does not change, a correct model stays correct on a drifted input, it only says 1 to more rows. The input summary surprises in the opposite direction. In the only scenario where the input **really** drifts, its deviation stays at 0.0382 and does not exceed 0.05. The number is not wrong: the drift touches **102** of 200 subscribers, the mean divides this among the remaining 98 subscribers, and the movement from 25.9220 to 27.0472 stays under the band. What a proxy looks at is part of the measure as much as **which breakdown** it looks at; an unbroken-down mean dilutes the drift as the segment it touches gets smaller. The input completeness deviates by 0.0000 in all three scenarios. This does not mean the measure is broken; the check run separates the two. When the `2026-10` reading of 40 subscribers never arrives in the source, the completeness drops **two** periods at once to 0.8000, which is the `late_arrival` class of the Introduction to Data Engineering course, and the reason two periods drop is that consumption depends on the index of the previous period. The measure works; it only sees **another class of degradation**. No kind of drift drops rows, so in drift monitoring the completeness pays the cost of being collected and gives zero information in return. Reading the four rows together is the answer of the lesson. The only measure that separates the scenarios is **accuracy**: 0.1598 against 0.0206, that is, thirty-one resolution steps. But accuracy needs a label, and the label arrives two periods later. There are three measures at hand that are free of delay, and all three fail to separate; the one measure that separates is delayed. **A proxy metric does not remove the cost of delay.** #### Summary - The three proxies, the input summary, the prediction share and the input completeness, read no label and are free of delay; so they are the only class of measures left at hand in the two-period gap that the delayed label leaves. - In the concept-drift scenario all three proxies give the **digit-for-digit same** number as in the sound scenario (26.5325 / 0.5150 / 1.0000), because the rule changes and the input does not. The deviations are 0.0185, 0.0957 and 0.0000; **none of them sees concept drift**. - The prediction share deviates by 0.0957 in the sound setup and exceeds the 0.05 band; in the data-drift setup it rises to 0.1596, and in that setup accuracy goes from 0.9700 to 0.9800. **This is the proxy that fires under a harmless drift.** - The input summary stays at 0.0382 and does not fire in the only scenario where the input really drifts: the drift touches 102 of 200 subscribers and the unbroken-down mean dilutes it. - The input completeness deviates by 0.0000 in all three scenarios but is not broken: when the reading of 40 subscribers for one period does not arrive, it drops two periods at once to 0.8000. The drift does not drop rows, so the proxy gives no measurable contribution here. - The only measure that separates the scenarios is accuracy (0.1598 against 0.0206, thirty-one resolution steps), and it too arrives two periods later. #### Next Step The last column of this lesson was a deviation number, and that number does nothing by itself; compared with a **band** it turns into a decision. At 0.05 the prediction share speaks in all three scenarios, at 0.20 it speaks in none. The next lesson makes the band variable: at 0.05, 0.10 and 0.20 it counts in how many periods the three measures fire, how many of these firings correspond to **real degradation** and how many are false positives; in the end it is seen that the band that gives no false positive at all also misses the only real degradation. ### Alerts and Thresholds Source: https://academia.sh/en/courses/model-monitoring/alerts-and-thresholds Counting what a narrower alert band gains and what it costs: at the 0.05 band, across three measures and three scenarios, 13 periods fire and 6 of them are false positives; at 0.10 there are 6 firings and 2 false positives; at 0.20 there is no false positive at all but all four real degradations are missed, because a relative band of 0.20 on the 0.9700 baseline asks accuracy to fall below 0.7760 and the lowest value is 0.8150. The prediction share fires in exactly three periods at the 0.05 band both in the sound setup and in the concept-drift setup; the same firing counts as a pure false positive in one place and as a catch in the other, so it counts as unmeasured. Only accuracy never fires in the two harmless scenarios and fires in four periods in the harmful scenario, and the detection lag is two periods. The previous lesson counted the deviation of three proxies and left a single number in the last column: the largest relative deviation against the `2026-08` baseline. That number decides nothing by itself. The decision comes from comparing the deviation with a **band**; the name of the band is the **alert threshold**, and it is part not of a measure but of **a choice**. The choice looks cheap and is not. The narrower the band, the more periods fire; how many of the firing periods have really degraded is something the band does not know. This lesson fixes the band at 0.05, 0.10 and 0.20, runs three measures in three scenarios and writes two numbers separately: the **periods that fire** and **how many of these correspond to real degradation**. - **MO47.** The source, the pipeline, the fictional model and the two drift types are **fictional** and are produced inside the lesson. No monitoring product, indicator service, cloud service or learning library is called. The seed is 20260218. - **MO48. The alert threshold** is the share that the **relative** deviation of a measure from the `2026-08` baseline must not exceed. Three bands are tried: 0.05, 0.10 and 0.20. The baseline period is the last period in which none of the drifts has started. - **MO49. Real degradation** is defined independently of the band: the **real** accuracy of a period falls **more than 0.0500** below the `2026-08` baseline, that is, ten resolution steps. This measure is for evaluation; in production it is known two periods late. - **MO50. A false positive** is a period that fires but is not in the real degradation list. The measurement of the delay of an indicator was done in the pipeline observability lesson of the Data Pipelines and Orchestration course and is **not repeated**. The only addition here is the **delayed label**: `DELAY` is added to the delay of an alert that fires on accuracy. - **MO51.** The model is inherited from the MLOps Fundamentals course with its flaw corrected: the decision threshold is a separate setting and the training criterion is **balanced accuracy**. In the monitoring periods the **base class ratio is 0.5342**. - **MO52.** The baseline is the **unmonitored setup**: no band, 0 firing periods, the detection lag is undefined. The step of period accuracy is **0.0050**. #### An Alert Threshold Is a Comparison The block below builds the shared definition and adds two things to it: `alert`, which compares a series of a measure with the baseline, and `period_seen`, which says in which period the accuracy of a period can be known. ```python # FICTIONAL. The meter readings of the measurement network , the five-stage pipeline , the fictional model and the two drift # types are produced inside the lesson; no monitoring product , indicator service , cloud service or # learning library is called. Seed 20260218. SEED, M32 = 20260218, 0xFFFFFFFF def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value REGION = ["east", "west", "north", "south", "central"] TYPE = ["residential", "commercial"] PERIODS = [f"2026-{a:02d}" for a in range(1, 13)] SUBSCRIBERS = {} for i in range(200): r = rng(SEED + 37 * i) SUBSCRIBERS[f"A{i + 1:03d}"] = {"region": REGION[int(r() * 5)], "type": TYPE[int(r() * 2)], "start": 10000 + int(r() * 60000)} def generate_source(): records = [] for i, (sub, a) in enumerate(SUBSCRIBERS.items()): e = a["start"] for p in PERIODS: r = rng(SEED + 101 * i + 7 * PERIODS.index(p)) e += 4 + int(r() * 46) records.append({"subscriber": sub, "period": p, "region": a["region"], "type": a["type"], "index": e, "unit": "m3"}) return records def s1_screen(records): REQUIRED = ("subscriber", "period", "region", "index", "unit") keep = [k for k in records if all(a in k for a in REQUIRED)] return keep, len(records) - len(keep) def s2_dedupe(records): seen, keep = set(), [] for k in records: key = (k["subscriber"], k["period"]) if key not in seen: seen.add(key) keep.append(k) return keep, len(records) - len(keep) def s3_consumption(records): table = {(k["subscriber"], k["period"]): k for k in records} out, dropped = [], 0 for k in records: i = PERIODS.index(k["period"]) prev = table.get((k["subscriber"], PERIODS[i - 1])) if i else None if prev is None: dropped += i > 0 continue out.append(dict(k, m3=int(k["index"]) - int(prev["index"]))) return out, dropped def rows(source=None): v, _ = s1_screen(generate_source() if source is None else source) v, _ = s2_dedupe(v) v, _ = s3_consumption(v) return v def features(records, window=None, digits=4): ordered = sorted(records, key=lambda k: PERIODS.index(k["period"])) sub, reg = {}, {} for k in ordered: sub.setdefault(k["subscriber"], []).append((PERIODS.index(k["period"]), k["m3"])) reg.setdefault(k["region"], []).append((PERIODS.index(k["period"]), k["m3"])) def mean(series, i): chosen = [v for d, v in series if (d < i and (window is None or d >= i - window))] return round(sum(chosen) / len(chosen), digits) if chosen else 0.0 feats = {} for k in records: i = PERIODS.index(k["period"]) feats[(k["subscriber"], k["period"])] = {"subscriber_mean": mean(sub[k["subscriber"]], i), "region_mean": mean(reg[k["region"]], i), "commercial": 1 if k["type"] == "commercial" else 0} return feats GRID = [(a, b, c) for a in (0.6, 0.8, 1.0) for b in (0.0, 0.2, 0.4) for c in (0, 4, 8)] DECISION_GRID = (18, 22, 26, 30, 34) # the decision threshold is a SEPARATE setting , searched in training MONITORED = PERIODS[6:] # 2026-07 .. 2026-12 : the periods monitored in production DELAY = 2 # after how many periods the label arrives RULE_THRESHOLD, RULE_COMMERCIAL = 30, 6 # CONCEPT: subscriber_mean + 6 x commercial > 30 def score(o, w): return w[0] * o["subscriber_mean"] + w[1] * o["region_mean"] + w[2] * o["commercial"] def label(feat_k, threshold=RULE_THRESHOLD, commercial_factor=RULE_COMMERCIAL): return 1 if feat_k["subscriber_mean"] + commercial_factor * feat_k["commercial"] > threshold else 0 def predict(feat_k, w, decision): return 1 if score(feat_k, w) > decision else 0 def balanced_accuracy(subset, feats, w, decision): tp = tn = fp = fn = 0 for k in subset: f_k = feats[(k["subscriber"], k["period"])] t, g = predict(f_k, w, decision), label(f_k) tp, fp = tp + (t and g), fp + (t and not g) fn, tn = fn + (g and not t), tn + (not t and not g) recall = tp / (tp + fn) if tp + fn else 0.0 specificity = tn / (tn + fp) if tn + fp else 0.0 return (recall + specificity) / 2, round(recall, 4), round(specificity, 4) def train(records, feats, periods=None): """The training criterion is BALANCED ACCURACY: the mean of recall and specificity.""" subset = [k for k in records if k["period"] in (periods or PERIODS[1:9])] best = (-1.0, None, None, 0.0, 0.0) for w in GRID: for decision in DECISION_GRID: o, rec, spec = balanced_accuracy(subset, feats, w, decision) if o > best[0]: best = (o, w, decision, rec, spec) return {"weights": best[1], "decision": best[2], "balanced": round(best[0], 4), "recall": best[3], "specificity": best[4], "train_rows": len(subset)} def drift(records, start="2026-09", slope=0.06): """DATA DRIFT: after the start, commercial consumption grows by slope every period.""" b = PERIODS.index(start) return [dict(k, m3=round(k["m3"] * (1.0 + slope * (PERIODS.index(k["period"]) - b + 1)), 4)) if PERIODS.index(k["period"]) >= b and k["type"] == "commercial" else dict(k) for k in records] def concept_drift(records, feats, start="2026-09", new_threshold=27): """CONCEPT DRIFT: the input is the SAME , the rule changes. The threshold drops from 30 to new_threshold.""" b = PERIODS.index(start) return {(k["subscriber"], k["period"]): label( feats[(k["subscriber"], k["period"])], new_threshold if PERIODS.index(k["period"]) >= b else RULE_THRESHOLD) for k in records} def input_summary(records, period, feats): d = [feats[(k["subscriber"], k["period"])]["subscriber_mean"] for k in records if k["period"] == period] return round(sum(d) / len(d), 4) if d else 0.0 def prediction_share(records, model, feats, period): d = [predict(feats[(k["subscriber"], k["period"])], model["weights"], model["decision"]) for k in records if k["period"] == period] return round(sum(d) / len(d), 4) if d else 0.0 def accuracy(records, model, feats, period, labels=None): subset = [k for k in records if k["period"] == period] d = sum(1 for k in subset if predict(feats[(k["subscriber"], k["period"])], model["weights"], model["decision"]) == (labels[(k["subscriber"], k["period"])] if labels else label(feats[(k["subscriber"], k["period"])]))) return round(d / len(subset), 4) if subset else None def alert(series, baseline, share): """A measure fires in a period if it deviates from the baseline by more than `share`.""" return [d for d, v in series if abs(v - baseline) / max(abs(baseline), 1e-9) > share] def period_seen(period, now): """Delayed label: the accuracy of a period is known only DELAY periods later.""" return PERIODS.index(period) + DELAY <= PERIODS.index(now) ALL_ROWS = rows() FEATS = features(ALL_ROWS) # features are computed ONCE MODEL = train(ALL_ROWS, FEATS) MON = [k for k in ALL_ROWS if k["period"] in MONITORED] largest = max(sum(1 for k in MON if label(FEATS[(k["subscriber"], k["period"])]) == s) for s in (0, 1)) print(f"model {MODEL['weights']} decision threshold {MODEL['decision']} , balanced accuracy" f" {MODEL['balanced']} , {len(MON)} rows , base class ratio {largest / len(MON):.4f} ," f" step {1 / 200:.4f} , the accuracy of 2026-09" f" is known in period {[d for d in PERIODS if period_seen('2026-09', d)][0]}") ``` ``` model (0.6, 0.0, 4) decision threshold 18 , balanced accuracy 0.9864 , 1200 rows , base class ratio 0.5342 , step 0.0050 , the accuracy of 2026-09 is known in period 2026-11 ``` The last field says up front what the band cannot measure: even if something degrades in `2026-09`, the accuracy of that period is known in `2026-11`. An alert built on accuracy speaks **two periods later**, however narrow its band. The only thing the band can move is how many times the delay-free proxies will speak. #### As the Band Narrows The block below compares three measures in three scenarios with three bands. Each cell carries two numbers: the **periods that fire** and how many of them are **false positives**. The `degraded` column is independent of the band; it gives the number of periods that have really degraded in that scenario. ```python # This block continues the definitions of the first block. The features of each scenario are computed ONCE. BAND = (0.05, 0.10, 0.20) DROP = 0.0500 # REAL DEGRADATION: accuracy falls below the baseline by more than this DRIFTED = drift(ALL_ROWS) FEATS_D = features(DRIFTED) CONCEPT = concept_drift(ALL_ROWS, FEATS) SCENARIOS = (("sound", ALL_ROWS, FEATS, None), ("data drift", DRIFTED, FEATS_D, None), ("concept drift", ALL_ROWS, FEATS, CONCEPT)) MEASURES = (("input summary", lambda src, feats, lab, d: input_summary(src, d, feats)), ("prediction share", lambda src, feats, lab, d: prediction_share(src, MODEL, feats, d)), ("accuracy", lambda src, feats, lab, d: accuracy(src, MODEL, feats, d, lab))) FIRE, REAL = {}, {} for sname, src, feats, lab in SCENARIOS: acc_series = [(d, accuracy(src, MODEL, feats, d, lab)) for d in MONITORED] REAL[sname] = set(d for d, v in acc_series[2:] if dict(acc_series)["2026-08"] - v > DROP) for name, f in MEASURES: series = [(d, f(src, feats, lab, d)) for d in MONITORED] for b in BAND: FIRE[(sname, name, b)] = set(alert(series[2:], dict(series)["2026-08"], b)) print(f"real degradation: accuracy falling more than {DROP:.4f} below its 2026-08 baseline") print("cell: periods that fire / false positives\n") print("measure scenario degraded " + "".join(f"{b:>12.2f}" for b in BAND)) for name, _ in MEASURES: for sname, *_ in SCENARIOS: cells = [f"{len(FIRE[(sname, name, b)])} / {len(FIRE[(sname, name, b)] - REAL[sname])}" for b in BAND] print(f"{name:<16} {sname:<14} {len(REAL[sname]):>8} " + "".join(f"{x:>12}" for x in cells)) print("\nband total firings false positives caught delay: prediction share accuracy") for b in BAND: totals = [sum(len(FIRE[(s, a, b)]) for s, *_ in SCENARIOS for a, _ in MEASURES), sum(len(FIRE[(s, a, b)] - REAL[s]) for s, *_ in SCENARIOS for a, _ in MEASURES), sum(len(FIRE[(s, a, b)] & REAL[s]) for s, *_ in SCENARIOS for a, _ in MEASURES)] delays = [] for name, extra in (("prediction share", 0), ("accuracy", DELAY)): fired = sorted(FIRE[("concept drift", name, b)]) delays.append(str(PERIODS.index(fired[0]) - PERIODS.index("2026-09") + extra) if fired else "undefined") print(f"{b:<5.2f} {totals[0]:>13} {totals[1]:>15} {totals[2]:>8} {delays[0]:>23} {delays[1]:>9}") ``` ``` real degradation: accuracy falling more than 0.0500 below its 2026-08 baseline cell: periods that fire / false positives measure scenario degraded 0.05 0.10 0.20 input summary sound 0 0 / 0 0 / 0 0 / 0 input summary data drift 0 0 / 0 0 / 0 0 / 0 input summary concept drift 4 0 / 0 0 / 0 0 / 0 prediction share sound 0 3 / 3 0 / 0 0 / 0 prediction share data drift 0 3 / 3 2 / 2 0 / 0 prediction share concept drift 4 3 / 0 0 / 0 0 / 0 accuracy sound 0 0 / 0 0 / 0 0 / 0 accuracy data drift 0 0 / 0 0 / 0 0 / 0 accuracy concept drift 4 4 / 0 4 / 0 0 / 0 band total firings false positives caught delay: prediction share accuracy 0.05 13 6 7 1 2 0.10 6 2 4 undefined 2 0.20 0 0 0 undefined undefined ``` #### Where the Catch Count Is Invalid The first row of the lower table seems to defend the band: 13 firings at 0.05, 7 of them corresponding to real degradation. This number is invalid, and the reason is in the upper table. Look at the `prediction share` rows: in the **sound** setup it fires in 3 periods, and in the **concept drift** setup again in exactly 3 periods. The same measure, the same band, the same number of firings; the only difference is that in the second case those periods have degraded. The firing does not come from the degradation, it **coincides in time** with the degradation. The sound row proves it: when nothing is broken, the same three periods speak too. This is the exact statement of the rule of this course: an indicator whose firing and whose degradation are not shown separately counts as unmeasured. Without the sound row, the prediction share at the 0.05 band would read like a flawless indicator, 3 firings and 0 false positives. With the sound row, 3 of the 7 catches fall away and **4** remain. Only one measure provides those four: accuracy, the only row that never fires in the two harmless scenarios at the 0.05 and 0.10 bands and fires in four of four periods in the harmful scenario. Its cost is in the last column: the detection lag is **2 periods**, and it cannot be reduced by choosing a band. #### The Trade Between False Positives and Blindness Widening the band does not fix the count, it zeroes it. Going from 0.05 to 0.10, the false positives fall from 6 to 2 and the catches from 7 to 4, but the three catches that drop are the prediction share catches that were already invalid; that is, the 0.10 band throws away two thirds of the false positives **without losing any real information**. This is the only step in which widening the band is free. The 0.20 row, on the other hand, is three zeros: no false positive, no firing, no catch. The meaning of the band on accuracy can be computed: 0.80 times the 0.9700 baseline is **0.7760**, so a band of 0.20 expects accuracy to fall below this value. The lowest value of the concept drift is **0.8150**, and it stays 0.0390 above 0.7760, about eight resolution steps. The band is placed where the measure cannot reach; its silence is not a sign of health, it is **by definition a dead indicator**. Reading the three bands together gives the second claim of the course: monitoring breaks things too. The narrow band raises six alarms in a production where nothing has degraded; the wide band never speaks in a production where accuracy fell from 0.9700 to 0.8150. The band in between is a good choice only in this fiction, because we know which firing is real by **looking at the real accuracy**; in production that information arrives two periods later and the band is chosen without it. #### Summary - The alert threshold is not a measure but a choice: it is the share that the relative deviation from the `2026-08` baseline must not exceed, and it directly determines the number of firing periods. - At the 0.05 band, across three measures and three scenarios, 13 periods fire and 6 of them are false positives; at 0.10 there are 6 firings and 2 false positives; at 0.20 all three are zero. - 3 of the 7 catches at 0.05 are invalid: the prediction share fires in exactly 3 periods in the **sound** setup as well. An indicator whose firing and whose degradation are not shown separately counts as unmeasured; the valid catch is **4**, and all of it comes from accuracy. - Accuracy is the only measure that never fires in the two harmless scenarios and fires in four of four periods in the harmful scenario; its cost is a detection lag of **2 periods**, and it cannot be reduced by the choice of band. - The 0.20 band asks accuracy to fall below 0.7760; the lowest value is 0.8150, so the band sits where the measure cannot reach and its silence is not a sign of health. #### Next Step The only valid indicator of this lesson was accuracy, and accuracy needs a **label**. Until now the label was a gift of the fiction: the real class of every row was known, it only arrived two periods late. In production it is not so; the label is collected, and what is collected **does not arrive for all predictions**. The next lesson asks where the label comes from: when the collection procedure covers only a part of the predictions, it counts how far the measured accuracy deviates from the real accuracy, and which form of collection grows the bias while it grows the coverage. ### Feedback Collection Source: https://academia.sh/en/courses/model-monitoring/feedback-collection Counting the share covered by a label collected from production and the bias it carries: when five collection procedures are tried on the same 1200 rows, collecting labels for the positive predictions only gives a coverage of 0.4858 and shifts the measured accuracy +0.0894 above the real accuracy in the concept-drift production, because that procedure measures precision, not accuracy; period by period the measured accuracy rises from 0.9362 to 1.0000 while the real accuracy falls to 0.8500, and an alert built on the drop never fires. Negative predictions only deviate in the opposite direction by -0.0845, and the rows near the decision threshold deviate by -0.2909 with a coverage of 0.1850. The procedure that collects every fourth row deviates by +0.0033 and +0.0100 with half the coverage; coverage does not correct bias, the bias comes from the selection rule. The previous lesson counted three measures at three bands and left a single valid indicator: accuracy. It was the only measure that never fired in the two harmless scenarios and fired in four of four periods in the harmful scenario. But accuracy needs a **label**, and until now the label was a gift of the fiction: the real class of every row was known, it only arrived two periods late. In production the label is not a gift, it is **collected**. And what is collected rarely covers every prediction: a review goes only to flagged records, an objection comes only from the person who is bothered, a verification is requested only in the cases left undecided. This lesson runs five collection procedures on the same set and writes two numbers separately: the **coverage**, that is, the share of rows whose label arrives; and the **deviation**, that is, the difference between the accuracy measured with those labels and the real accuracy. - **MO53.** The source, the pipeline, the fictional model, the concept drift and the five collection procedures are **fictional** and are produced inside the lesson. No monitoring product, labeling service, cloud service or learning library is called. The seed is 20260218. - **MO54. The collection procedure** is the rule that determines the label of which prediction will arrive. Five are tried: `all`; `positives only`, the rows the model calls 1; `negatives only`, the rows it calls 0; `near threshold`, the rows whose score is within `MARGIN` of the decision threshold; `every fourth`, every fourth row in the sequence. - **MO55. The coverage** is the ratio of the rows whose label arrives to the 1200 rows of the monitoring periods. **The deviation** is the **real accuracy**, measured with all rows, subtracted from the accuracy measured with the collected labels. Its direction is meaningful: a plus sign says the model looks better than it is. - **MO56.** The return of the user signal to evaluation and the bias of that signal were measured in the feedback loop lesson of the AI Safety and Evaluation course in the AI Engineering curriculum; **the procedure is not repeated**. The measure there was for language model output; the addition here is **the delayed label of a numeric model**. - **MO57.** The model is inherited from the MLOps Fundamentals course with its flaw corrected: the decision threshold is a separate setting and the training criterion is **balanced accuracy**. In the monitoring periods the **base class ratio is 0.5342**. - **MO58.** The baseline is the **no-collection setup**: no label arrives, there is no measured accuracy, the deviation is undefined. The step of period accuracy is **0.0050**, and a deviation smaller than this is not read. #### Where the Label Comes From The block below builds the shared definition and trains the model. The last line gives the limits of the lesson: over how many rows we measure, what the resolution is below which we cannot go, and how many periods later the label arrives. ```python # FICTIONAL. The meter readings of the measurement network , the five-stage pipeline , the fictional model , the concept # drift and the five collection procedures are produced inside the lesson; no monitoring product , # labeling service , cloud service or learning library is called. Seed 20260218. SEED, M32 = 20260218, 0xFFFFFFFF def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value REGION = ["east", "west", "north", "south", "central"] TYPE = ["residential", "commercial"] PERIODS = [f"2026-{a:02d}" for a in range(1, 13)] SUBSCRIBERS = {} for i in range(200): r = rng(SEED + 37 * i) SUBSCRIBERS[f"A{i + 1:03d}"] = {"region": REGION[int(r() * 5)], "type": TYPE[int(r() * 2)], "start": 10000 + int(r() * 60000)} def generate_source(): records = [] for i, (sub, a) in enumerate(SUBSCRIBERS.items()): e = a["start"] for p in PERIODS: r = rng(SEED + 101 * i + 7 * PERIODS.index(p)) e += 4 + int(r() * 46) records.append({"subscriber": sub, "period": p, "region": a["region"], "type": a["type"], "index": e, "unit": "m3"}) return records def s1_screen(records): REQUIRED = ("subscriber", "period", "region", "index", "unit") keep = [k for k in records if all(a in k for a in REQUIRED)] return keep, len(records) - len(keep) def s2_dedupe(records): seen, keep = set(), [] for k in records: key = (k["subscriber"], k["period"]) if key not in seen: seen.add(key) keep.append(k) return keep, len(records) - len(keep) def s3_consumption(records): table = {(k["subscriber"], k["period"]): k for k in records} out, dropped = [], 0 for k in records: i = PERIODS.index(k["period"]) prev = table.get((k["subscriber"], PERIODS[i - 1])) if i else None if prev is None: dropped += i > 0 continue out.append(dict(k, m3=int(k["index"]) - int(prev["index"]))) return out, dropped def rows(source=None): v, _ = s1_screen(generate_source() if source is None else source) v, _ = s2_dedupe(v) v, _ = s3_consumption(v) return v def features(records, window=None, digits=4): ordered = sorted(records, key=lambda k: PERIODS.index(k["period"])) sub, reg = {}, {} for k in ordered: sub.setdefault(k["subscriber"], []).append((PERIODS.index(k["period"]), k["m3"])) reg.setdefault(k["region"], []).append((PERIODS.index(k["period"]), k["m3"])) def mean(series, i): chosen = [v for d, v in series if (d < i and (window is None or d >= i - window))] return round(sum(chosen) / len(chosen), digits) if chosen else 0.0 feats = {} for k in records: i = PERIODS.index(k["period"]) feats[(k["subscriber"], k["period"])] = {"subscriber_mean": mean(sub[k["subscriber"]], i), "region_mean": mean(reg[k["region"]], i), "commercial": 1 if k["type"] == "commercial" else 0} return feats GRID = [(a, b, c) for a in (0.6, 0.8, 1.0) for b in (0.0, 0.2, 0.4) for c in (0, 4, 8)] DECISION_GRID = (18, 22, 26, 30, 34) # the decision threshold is a SEPARATE setting , searched in training MONITORED = PERIODS[6:] # 2026-07 .. 2026-12 : the periods monitored in production DELAY = 2 # after how many periods the label arrives RULE_THRESHOLD, RULE_COMMERCIAL = 30, 6 # CONCEPT: subscriber_mean + 6 x commercial > 30 def score(o, w): return w[0] * o["subscriber_mean"] + w[1] * o["region_mean"] + w[2] * o["commercial"] def label(feat_k, threshold=RULE_THRESHOLD, commercial_factor=RULE_COMMERCIAL): return 1 if feat_k["subscriber_mean"] + commercial_factor * feat_k["commercial"] > threshold else 0 def predict(feat_k, w, decision): return 1 if score(feat_k, w) > decision else 0 def balanced_accuracy(subset, feats, w, decision): tp = tn = fp = fn = 0 for k in subset: f_k = feats[(k["subscriber"], k["period"])] t, g = predict(f_k, w, decision), label(f_k) tp, fp = tp + (t and g), fp + (t and not g) fn, tn = fn + (g and not t), tn + (not t and not g) recall = tp / (tp + fn) if tp + fn else 0.0 specificity = tn / (tn + fp) if tn + fp else 0.0 return (recall + specificity) / 2, round(recall, 4), round(specificity, 4) def train(records, feats, periods=None): """The training criterion is BALANCED ACCURACY: the mean of recall and specificity.""" subset = [k for k in records if k["period"] in (periods or PERIODS[1:9])] best = (-1.0, None, None, 0.0, 0.0) for w in GRID: for decision in DECISION_GRID: o, rec, spec = balanced_accuracy(subset, feats, w, decision) if o > best[0]: best = (o, w, decision, rec, spec) return {"weights": best[1], "decision": best[2], "balanced": round(best[0], 4), "recall": best[3], "specificity": best[4], "train_rows": len(subset)} def concept_drift(records, feats, start="2026-09", new_threshold=27): """CONCEPT DRIFT: the input is the SAME , the rule changes. The threshold drops from 30 to new_threshold.""" b = PERIODS.index(start) return {(k["subscriber"], k["period"]): label( feats[(k["subscriber"], k["period"])], new_threshold if PERIODS.index(k["period"]) >= b else RULE_THRESHOLD) for k in records} def alert(series, baseline, share): """A measure fires in a period if it deviates from the baseline by more than `share`.""" return [d for d, v in series if abs(v - baseline) / max(abs(baseline), 1e-9) > share] ALL_ROWS = rows() FEATS = features(ALL_ROWS) # features are computed ONCE MODEL = train(ALL_ROWS, FEATS) MON = [k for k in ALL_ROWS if k["period"] in MONITORED] largest = max(sum(1 for k in MON if label(FEATS[(k["subscriber"], k["period"])]) == s) for s in (0, 1)) print(f"model {MODEL['weights']} decision threshold {MODEL['decision']} , balanced accuracy" f" {MODEL['balanced']} , recall {MODEL['recall']} , monitoring {len(MON)} rows ," f" base class ratio {largest / len(MON):.4f} , step {1 / 200:.4f} , label delay" f" {DELAY} periods") ``` ``` model (0.6, 0.0, 4) decision threshold 18 , balanced accuracy 0.9864 , recall 1.0 , monitoring 1200 rows , base class ratio 0.5342 , step 0.0050 , label delay 2 periods ``` #### Coverage and Deviation The block below runs the five procedures in two scenarios. The upper table gives, for the whole monitoring window, the coverage, the accuracy measured with the collected labels, the real accuracy measured with all rows, and the difference between the two. The lower table counts, only in the concept-drift production, the accuracy measured period by period and how many periods an alert of 0.05 built on the **drop** fires. ```python # This block continues the definitions of the first block. The features are computed ONCE. MARGIN = 1.0 # scores this close to the decision threshold count as "near threshold" CONCEPT = concept_drift(ALL_ROWS, FEATS) W, DEC = MODEL["weights"], MODEL["decision"] PROCEDURES = (("all", lambda o, j: True), ("positives only", lambda o, j: predict(o, W, DEC) == 1), ("negatives only", lambda o, j: predict(o, W, DEC) == 0), ("near threshold", lambda o, j: abs(score(o, W) - DEC) <= MARGIN), ("every fourth", lambda o, j: j % 4 == 0)) def measure(labels, select, periods_set): """Only the label of the row that select picks arrives; accuracy is measured WITH THEM.""" total = correct = all_n = all_correct = 0 for j, k in enumerate(ALL_ROWS): if k["period"] not in periods_set: continue o = FEATS[(k["subscriber"], k["period"])] d = predict(o, W, DEC) == (labels[(k["subscriber"], k["period"])] if labels else label(o)) all_n, all_correct = all_n + 1, all_correct + d if select(o, j): total, correct = total + 1, correct + d return round(total / all_n, 4), round(correct / total, 4), round(all_correct / all_n, 4) print("procedure scenario coverage measured real deviation") for name, select in PROCEDURES: for sname, lab in (("sound", None), ("concept drift", CONCEPT)): cov, measured, real = measure(lab, select, MONITORED) print(f"{name:<16} {sname:<14} {cov:>8.4f} {measured:>8.4f} {real:>6.4f} {measured - real:>+9.4f}") print("\nconcept drift , accuracy measured period by period and firing on the DROP at the 0.05 band") for name, select in PROCEDURES: series = [(d, measure(CONCEPT, select, [d])[1]) for d in MONITORED] baseline = dict(series)["2026-08"] fired = [d for d, v in series[2:] if (baseline - v) / baseline > 0.05] print(f"{name:<16}" + "".join(f"{v:8.4f}" for _, v in series) + f" fires {len(fired)}") ``` ``` procedure scenario coverage measured real deviation all sound 1.0000 0.9800 0.9800 +0.0000 all concept drift 1.0000 0.8900 0.8900 +0.0000 positives only sound 0.4858 0.9588 0.9800 -0.0212 positives only concept drift 0.4858 0.9794 0.8900 +0.0894 negatives only sound 0.5142 1.0000 0.9800 +0.0200 negatives only concept drift 0.5142 0.8055 0.8900 -0.0845 near threshold sound 0.1850 0.8919 0.9800 -0.0881 near threshold concept drift 0.1850 0.5991 0.8900 -0.2909 every fourth sound 0.2500 0.9833 0.9800 +0.0033 every fourth concept drift 0.2500 0.9000 0.8900 +0.0100 concept drift , accuracy measured period by period and firing on the DROP at the 0.05 band all 0.9700 0.9700 0.8150 0.8700 0.8650 0.8500 fires 4 positives only 0.9341 0.9362 1.0000 1.0000 1.0000 1.0000 fires 0 negatives only 1.0000 1.0000 0.6606 0.7347 0.7245 0.6907 fires 4 near threshold 0.8286 0.8537 0.3824 0.5526 0.4857 0.4615 fires 4 every fourth 0.9800 0.9800 0.8600 0.8600 0.8200 0.9000 fires 4 ``` #### The Collection Procedure Chooses the Measure Too The `positives only` row is the hardest finding of the lesson. Its coverage is 0.4858, so it sees almost half of the predictions; this is not a small sample. In the concept-drift production the accuracy it measures is **0.9794**, the real accuracy is **0.8900**; the deviation is **+0.0894**, eighteen resolution steps, and **upward**. The lower table is worse still: the measured accuracy **rises** from 0.9362 to **1.0000** while the real accuracy falls to 0.8500. The alert built on the drop **never fires**. The mechanism is in the selection rule. When only the labels of the rows called 1 arrive, what is measured is not accuracy but **precision**: how many of the rows the model calls 1 are really 1. The concept drift **loosens** the label rule, the threshold drops from 30 to 27, and more rows really become 1. The old positives of the model are verified wholesale and the precision rises to 1.0000; the new positives it misses are never measured because they stay on the non-positive side. **The collection procedure ends up choosing the measure in which the degradation is least visible.** `negatives only` carries the same flaw with the opposite sign: in the sound production it measures 1.0000 and shows the model better than it is, in the concept-drift production it measures 0.8055 and is too pessimistic by **-0.0845**. The clearest example is `near threshold`: the rows near the decision threshold are by definition the rows where the model is least certain, and collecting only them makes a model of 0.8900 read as **0.5991**. The deviation is **-0.2909**, fifty-eight resolution steps. #### Coverage Does Not Correct Bias Putting two rows side by side gives the rule of the lesson. `positives only` deviates by +0.0894 with a coverage of **0.4858**; `every fourth` by +0.0100 with a coverage of **0.2500**. Half the coverage, one ninth of the deviation. In the sound production the deviation of `every fourth` is **+0.0033**, that is, below even the resolution step and **unreadable**. Increasing the number of collected labels does not reduce the bias; the only thing that determines the bias is **whether the selection rule is related to the prediction**. `every fourth` looks at the position in the sequence, not at the output of the model, and for this reason it is nine times more accurate with half the coverage. From here comes the counterpart of the course rule on the label side. As a firing indicator is not evidence of degradation, **a set of collected labels is not evidence of measurement either**. If a coverage number is reported alone, "we have labels for 49% of the predictions", it says nothing about what the measure is; what says it is **by which rule** that 49% was selected. A collection procedure whose deviation is not shown separately **counts as unmeasured**. #### Summary - The collection procedure is the rule that determines the label of which prediction will arrive; the coverage and the deviation are two separate numbers, and in every procedure other than `all` the deviation is different from zero. - `positives only` deviates by **+0.0894** in the concept-drift production with a coverage of 0.4858 and **raises** the measured accuracy from 0.9362 to 1.0000; the alert of 0.05 built on the drop never fires. What it measures is not accuracy but **precision**, and the precision rises when the rule loosens. - `negatives only` deviates in the opposite direction by -0.0845, and `near threshold` deviates by -0.2909 with a coverage of 0.1850; both show the degradation but read the level wrong by seventeen and fifty-eight resolution steps. - `every fourth` deviates by +0.0033 and +0.0100 with a coverage of 0.2500; half the coverage, one ninth of the deviation. **Coverage does not correct bias**, the bias comes from the selection rule being related to the prediction. - A collection procedure whose deviation is not shown separately counts as unmeasured; a coverage number alone is not evidence of the measurement. #### Next Step Six lessons divided a single question: **is the degradation seen**. The data drift was a change that fires but does not break, the concept drift a change that breaks but does not fire; the delayed label delayed the only measure that sees by two periods, the proxy metrics did not discriminate in that gap, the false positives grew as the band narrowed, and the collected label itself shifted the measure. What was measured was always the same: the **seeing** of a degradation. The question never asked is what is done **after** it is seen. A team that learns in `2026-11` that the rule changed in `2026-09` has a sequence of decisions in hand, and each carries its own cost: training a new model and putting it into service, giving it first to a share of the traffic, running two models at the same time and showing that the difference is real, recording the reason for the decision. The next topic counts these decisions, and its first measure is this: as the share of requests going to the new model grows, how much does the result move, and is it possible to take a measurement **without sending any request**. ## Release and Governance Counting the release, change and retirement of a model in an auditable way: in shadow deployment no request goes to the new model yet the measurement can still be taken, and in the canary the share sent to the new model moves the result by a measurable amount; the smallest difference an online experiment can distinguish with two hundred rows per period, and a comparison below that difference remaining inconclusive; a retraining trigger giving the same 0.9800 as the old model in all three windows when the rule has not changed, and the training cost being paid for nothing; decision records making findable which model and which feature produced a prediction; compliance obligations being named by their classes without writing a regulation name, and how many records the engineering counterpart of each class requires; tracking training and inference cost separately; by which measure the retirement of a model is decided, and for how long the retrospective auditability of the removed model is preserved. ### Shadow and Canary Deployment Source: https://academia.sh/en/courses/model-monitoring/shadow-and-canary-deployment Staging the release of a new model and counting how much the share sent to the new model moves both the result and the measure: in the shadow placement 0 requests go to the new model yet the candidate is measured on all two hundred rows and comes out at 0.9650, when the canary share is 0.10 the candidate is measured on only twenty rows, the resolution goes from 0.0050 to 0.0500 and the same candidate looks like 1.0000. The training search cannot separate two candidates at 0.9800 and 0.9741, while in production there is a 0.0150 difference between them, and as the share shrinks the trace of this difference falls below the measure. The previous topic measured how the degradation is **seen** in six lessons: an input drift produced an alert but did not lower accuracy, a concept drift lowered accuracy without showing in any proxy metric, the only measure that saw it arrived two periods late, and the label collected from production was wrong. The common flaw of the six lessons is this: none of them asked **what to do once it is seen**. The first thing done once it is seen is to change the model. Changing is itself a risk: the new model runs in production for the first time, and the only thing known about it is its behavior **on the training set**. This lesson puts the new model into release in three different placements and counts a single question: **how much the share sent to the new model moves the result, and how much it moves the measure.** Release staging itself was built in the Continuous Integration and Delivery course of the DevOps and Platform Engineering curriculum, and rollback and staged release were measured in the Resilience and Reliability course of the System Design and Distributed Systems curriculum; neither is repeated here. The question here is not about deployment but about the **measure**. - **RG1.** The source, the pipeline and the model are **fictional** and are generated inside the lesson. No deployment tool, router, experiment platform or monitoring product is called. Seed 20260218. - **RG2.** The label is tied to a rule that the features see: `subscriber_mean + 6 x commercial > 30`. The decision threshold is **a setting separate from** this rule and is searched during training; the training criterion is **balanced accuracy**, that is, the mean of **recall** and **specificity**. All three were established in the first lesson of this course. - **RG3.** In the monitored periods the **base class ratio is 0.5342**; no accuracy is read without this ratio being written next to it. There are 200 rows per period, so the smallest step of a period's accuracy is **0.0050**. - **RG4.** The production data has a **data drift**: after `2026-09` commercial consumption grows by 0.06 per period. The rule has not changed. - **RG5.** The release window is a single period: `2026-12`, 200 requests. In the canary, the share of requests counted from the start goes to the new model; this is a **fictional** split. - **RG6.** In the **shadow** placement the new model scores the same request in parallel with the old model, the result is only recorded, and **no response returns to the user**. - **RG7.** Cost is counted in **requests** and **rows**; raw time is not written. #### Two Candidates the Training Cannot Separate The first block sets up the source, the consumption stage, the feature path and the fictional model; then it retrains on the drifted data in production and takes the **first two ranks** of the search as release candidates. ```python # FICTIONAL. The meter readings of the measurement network, the consumption stage of the # pipeline, the feature path and the three-weight threshold model are generated inside # the lesson; no learning library, deployment tool, experiment platform or monitoring # product is called. # Seed 20260218. SEED, M32 = 20260218, 0xFFFFFFFF def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value REGION = ["east", "west", "north", "south", "central"] TYPE = ["residential", "commercial"] PERIODS = [f"2026-{a:02d}" for a in range(1, 13)] SUBSCRIBERS = {} for i in range(200): r = rng(SEED + 37 * i) SUBSCRIBERS[f"A{i + 1:03d}"] = {"region": REGION[int(r() * 5)], "type": TYPE[int(r() * 2)], "start": 10000 + int(r() * 60000)} def rows(): """Source readings and consumption from the period difference: the first three stages of the pipeline.""" records = [] for i, (sub, a) in enumerate(SUBSCRIBERS.items()): e = a["start"] for p in PERIODS: r = rng(SEED + 101 * i + 7 * PERIODS.index(p)) e += 4 + int(r() * 46) records.append({"subscriber": sub, "period": p, "region": a["region"], "type": a["type"], "index": e}) t = {(k["subscriber"], k["period"]): k["index"] for k in records} return [dict(k, m3=k["index"] - t[(k["subscriber"], PERIODS[PERIODS.index(k["period"]) - 1])]) for k in records if PERIODS.index(k["period"])] def features(records): """Independent of the weights and computed ONCE; it sees the full history.""" sub, reg = {}, {} for k in sorted(records, key=lambda k: PERIODS.index(k["period"])): sub.setdefault(k["subscriber"], []).append((PERIODS.index(k["period"]), k["m3"])) reg.setdefault(k["region"], []).append((PERIODS.index(k["period"]), k["m3"])) def mean(series, i): s = [v for d, v in series if d < i] return round(sum(s) / len(s), 4) if s else 0.0 return {(k["subscriber"], k["period"]): { "subscriber_mean": mean(sub[k["subscriber"]], PERIODS.index(k["period"])), "region_mean": mean(reg[k["region"]], PERIODS.index(k["period"])), "commercial": 1 if k["type"] == "commercial" else 0} for k in records} RULE_THRESHOLD, RULE_COMMERCIAL = 30, 6 # the RULE that generates the label , in other words the concept MONITORED = PERIODS[6:] # six periods monitored in production GRID = [(a, b, c) for a in (0.6, 0.8, 1.0) for b in (0.0, 0.2, 0.4) for c in (0, 4, 8)] DECISION_GRID = (18, 22, 26, 30, 34) # the decision threshold is A SEPARATE setting def label(o, threshold=RULE_THRESHOLD): return 1 if o["subscriber_mean"] + RULE_COMMERCIAL * o["commercial"] > threshold else 0 def score(o, w): return w[0] * o["subscriber_mean"] + w[1] * o["region_mean"] + w[2] * o["commercial"] def predict(o, w, decision): return 1 if score(o, w) > decision else 0 def balanced_accuracy(subset, feats, w, decision): """Training criterion: the mean of recall and specificity.""" tp = tn = fp = fn = 0 for k in subset: o = feats[(k["subscriber"], k["period"])] p, g = predict(o, w, decision), label(o) if p and g: tp += 1 elif p: fp += 1 elif g: fn += 1 else: tn += 1 recall = tp / (tp + fn) if tp + fn else 0.0 specificity = tn / (tn + fp) if tn + fp else 0.0 return (recall + specificity) / 2, round(recall, 4), round(specificity, 4) def train(records, feats, periods): """Weights and decision threshold are searched together: 27 x 5 = 135 candidates.""" subset = [k for k in records if k["period"] in periods] ranked = sorted(((balanced_accuracy(subset, feats, w, dc)[0], w, dc) for w in GRID for dc in DECISION_GRID), key=lambda a: -a[0]) best = ranked[0] return {"weights": best[1], "decision": best[2], "balanced": round(best[0], 4), "ranked": ranked, "rows": len(subset)} def drift(records, start="2026-09", slope=0.06): """FICTIONAL data drift: commercial consumption grows by `slope` every period.""" b = PERIODS.index(start) return [dict(k, m3=round(k["m3"] * (1.0 + slope * (PERIODS.index(k["period"]) - b + 1)), 4)) if PERIODS.index(k["period"]) >= b and k["type"] == "commercial" else dict(k) for k in records] ALL = rows() FEATS = features(ALL) BASELINE = train(ALL, FEATS, PERIODS[1:9]) PRODUCTION = drift(ALL) # the input has drifted in production FEATS_D = features(PRODUCTION) bal, recall, specificity = balanced_accuracy([k for k in ALL if k["period"] in PERIODS[1:9]], FEATS, BASELINE["weights"], BASELINE["decision"]) monitored_rows = [k for k in ALL if k["period"] in MONITORED] labels = [label(FEATS[(k["subscriber"], k["period"])]) for k in monitored_rows] print(f"baseline model {BASELINE['weights']} decision threshold {BASELINE['decision']}" f" balanced accuracy {BASELINE['balanced']} recall {recall:.4f} specificity {specificity:.4f}") print(f"monitoring {len(monitored_rows)} rows , base class ratio {round(1 - sum(labels) / len(labels), 4)}" f" , 200 rows per period , accuracy step 0.0050") # Two candidates: the retrained model and the second rank of the same search. RETRAINED = train(PRODUCTION, FEATS_D, PERIODS[5:11]) A = {"name": "A retrained", "weights": RETRAINED["weights"], "decision": RETRAINED["decision"], "train": RETRAINED["balanced"]} second = RETRAINED["ranked"][1] B = {"name": "B second of the search", "weights": second[1], "decision": second[2], "train": round(second[0], 4)} last = [k for k in PRODUCTION if k["period"] == "2026-12"] for m in (A, B): m["disagreement"] = sum(1 for k in last if predict(FEATS_D[(k["subscriber"], k["period"])], m["weights"], m["decision"]) != predict(FEATS_D[(k["subscriber"], k["period"])], BASELINE["weights"], BASELINE["decision"])) print(f"candidate {m['name']} {m['weights']}/{m['decision']}" f" training balanced accuracy {m['train']} , prediction disagreement {m['disagreement']} on 2026-12") ``` ``` baseline model (0.6, 0.0, 4) decision threshold 18 balanced accuracy 0.9864 recall 1.0000 specificity 0.9728 monitoring 1200 rows , base class ratio 0.5342 , 200 rows per period , accuracy step 0.0050 candidate A retrained (0.6, 0.0, 4)/18 training balanced accuracy 0.98 , prediction disagreement 0 on 2026-12 candidate B second of the search (0.8, 0.4, 4)/34 training balanced accuracy 0.9741 , prediction disagreement 11 on 2026-12 ``` The two candidates stand at **0.9800** and **0.9741** in training. The 0.0059 between them is a difference of about seven rows in the 1200 rows of the six-period training window; no training report can rank these two with confidence. Their situations in production, however, are different: A carries **exactly the same weights and the same decision threshold** as the old model, so its prediction disagreement on `2026-12` is **0** and releasing it cannot change anything. B, on the other hand, departs from the old model on **11 predictions** in the same period. The whole subject of the release decision is these 11 predictions. #### What the Share Does to the Measure The second block runs the three placements on the same 200 requests. The `sent` column is the number of requests sent to the new model, `served` is the accuracy of the responses returned to the user, `measured rows` is the number of rows on which a measure of the candidate can be collected, and `extra wrong` is the number of requests the old model got right but the new model served wrongly. ```python # This block continues the BASELINE, PRODUCTION, FEATS_D, A, B, last, predict, # label and score definitions of the first block. def release(old, new, feats, subset, share, shadow=False): """Shadow: no request goes to the new model , the measurement is still taken.""" n = 0 if shadow else int(len(subset) * share) measured = len(subset) if shadow else n served = cand = extra_wrong = 0 for j, k in enumerate(subset): o = feats[(k["subscriber"], k["period"])] g = label(o) p_old = predict(o, old["weights"], old["decision"]) p_new = predict(o, new["weights"], new["decision"]) served += ((p_new if j < n else p_old) == g) extra_wrong += (j < n and p_new != g and p_old == g) cand += ((shadow or j < n) and p_new == g) return (n, round(served / len(subset), 4), measured, round(cand / measured, 4), round(1 / measured, 4), extra_wrong) old_accuracy = sum(1 for k in last if predict(FEATS_D[(k["subscriber"], k["period"])], BASELINE["weights"], BASELINE["decision"]) == label(FEATS_D[(k["subscriber"], k["period"])])) print(f"2026-12 requests {len(last)} , old model accuracy {old_accuracy / len(last)}") print("cand placement sent served measured rows measure resolution extra wrong") for m in (A, B): for name, share, g in (("shadow", 0.0, True), ("canary 0.10", 0.10, False), ("canary 0.50", 0.50, False), ("full rollout", 1.00, False)): n, h, ms, cm, rs, ew = release(BASELINE, m, FEATS_D, last, share, g) print(f"{m['name'][0]} {name:<12} {n:>5} {h:>13} {ms:>13} {cm:>11}" f" {rs:>10} {ew:>11}") ``` ``` 2026-12 requests 200 , old model accuracy 0.98 cand placement sent served measured rows measure resolution extra wrong A shadow 0 0.98 200 0.98 0.005 0 A canary 0.10 20 0.98 20 1.0 0.05 0 A canary 0.50 100 0.98 100 0.99 0.01 0 A full rollout 200 0.98 200 0.98 0.005 0 B shadow 0 0.98 200 0.965 0.005 0 B canary 0.10 20 0.98 20 1.0 0.05 0 B canary 0.50 100 0.97 100 0.97 0.01 3 B full rollout 200 0.965 200 0.965 0.005 7 ``` The A rows are the same in all four placements: served accuracy **0.9800**, extra wrong **0**. This is not a success but a measure of an **emptiness**. For a candidate whose prediction disagreement is zero, a canary release means staging a change that can change nothing; the risk carried is zero, and the information gained is zero as well. Even the 1.0000 in A's `canary 0.10` row is not information: it is a number measured on twenty rows, and the old model's value on the same twenty rows is also 1.0000. The B rows carry the real reading. **In the shadow placement 0 requests go to the new model**, the served accuracy stays at the old model's 0.9800, no user sees the output of the new model, and the extra wrong is **0**. Even so, the candidate is measured on **all 200 rows** and its real value is read: **0.9650**, that is **0.0150** worse than the old model. This difference is **three steps** of the period resolution and lies above the band. When the canary share is 0.10 the table turns around. **20 requests** go to the new model, but now the candidate can be measured only on those 20 rows: the resolution of the measure goes **from 0.0050 to 0.0500**, ten times coarser. On those twenty rows B's measure is **1.0000**. A candidate whose real value is 0.9650 looks **flawless** at the most common canary share. The served accuracy also stays at **0.9800**, so someone looking at the release dashboard sees that nothing has changed. The trace of the bad candidate has dropped below the measure. When the share is raised to 0.50 the measure begins to recover — resolution 0.0100, the candidate's measure 0.9700 — but its cost becomes visible too: **3 requests** that the old model got right are answered wrongly. A share of 1.00 is not a canary but a **full rollout**; the measure returns to its full resolution and the extra wrong goes up to **7**. The **rollback** of a canary is pulling the share down to 0, and this does not bring back the requests that were served wrongly. The shadow has nothing to roll back, because no request ever went. From this comes a single rule with two sides. **In a canary the resolution of the measure and the risk carried depend on the same number, the share, and move in opposite directions**: reducing the share reduces the risk and also blinds the measure. **The shadow breaks this link** — its resolution is the same as that of the full rollout (0.0050) and the requests going to the new model are 0. The price the shadow pays is in another column and is not in this table: every request is scored twice, so 400 inferences are made for 200 requests. The only thing the shadow cannot measure is the behavior caused by **the response itself**; since nobody sees the score, nobody influences the result either. #### Summary - The same training search leaves two candidates at **0.9800** and **0.9741** and cannot separate them; in production one is a copy whose prediction disagreement with the old model is **0**, and the other is a real change that departs on **11 predictions** on `2026-12`. The base class ratio is 0.5342 and the step of a period's accuracy is 0.0050. - **The shadow placement sends 0 requests to the new model** and still measures the candidate on all 200 rows: its real value is **0.9650**, **0.0150** below the old model and as much as three steps of the resolution. - **The canary share divides the measure as well.** At a share of 0.10 the candidate is measured on only 20 rows, the resolution goes up to **0.0500**, and the candidate that is in reality bad looks like **1.0000**; since the served accuracy stays at 0.9800, someone looking at the dashboard sees nothing. - Raising the share fixes the measure but writes its cost: at 0.50 **3** requests and at 1.00 **7** requests that the old model got right are answered wrongly. A share of 1.00 is not a canary but a full rollout, and the rollback does not bring these requests back. - In a canary the resolution and the risk depend on **the same share** and go in opposite directions; the shadow breaks this link and pays its cost in **the number of inferences**: 400 scorings for 200 requests. #### Next Step The shadow read the candidate's real value in a single period as 0.9650 and found it 0.0150 below the old model's 0.9800. This number looks sufficient for a decision, but an unanswered question sits underneath it: **how small can** a difference measured with 200 rows become before it can no longer be read? The next lesson sweeps the comparison of two models over the number of periods, counts within which band the difference of the same candidate moves from period to period, and calculates the **smallest difference** this set size **can distinguish**. Every comparison that remains below the number found is inconclusive, and the lesson shows an example of this together with its mark. ### Online Experiments Source: https://academia.sh/en/courses/model-monitoring/online-experiments Counting how small a difference the comparison of two models in production can distinguish at this set size: with two hundred rows per period the resolution is 0.0050, with six periods it is 0.0008, yet single-period measurements of the same candidate move within a band 0.0400 wide and change sign. A sweep of one hundred thirty-five candidates gives the limit: the smallest difference of the 133 candidates that carry the same sign in all six of six periods is 0.0275, and the difference of the single candidate whose sign moves is 0.0050. No candidate was measured in the interval between them, and the minimum detectable effect lies between these two numbers; every comparison below it is inconclusive. The previous lesson showed that the shadow placement measured the candidate on all two hundred rows and gave **0.9650** on `2026-12`; the old model was **0.9800** in the same period. The **0.0150** between them looked sufficient for a decision. An unanswered question sits underneath this number: **how small can** the difference of the two models become before this set can no longer read it. The question is one of experiment design. Comparing two arms, sample size and confidence interval were built in the User Experience and Behavior Design course of the Interface Design and Design Systems curriculum, and distribution assumptions and the interval procedure were measured in the Statistical Analysis and Exploratory Data Analysis course of the Data Analytics curriculum; neither is repeated here. This lesson calculates a single number: **the minimum detectable effect of this set size.** Every comparison below the number found is **inconclusive**, whatever procedure it is made with, and the lesson shows the single example of this together with its mark. - **RG11.** The source, the pipeline and the model are **fictional**; no experiment platform, deployment tool or monitoring product is called. Seed 20260218. - **RG12.** The label is tied to the rule `subscriber_mean + 6 x commercial > 30`, the decision threshold is a separate setting and the training criterion is balanced accuracy. In the monitored periods the **base class ratio is 0.5342**. - **RG13.** The experiment horizon is the six monitored periods: **200 rows** per period, **1200** in total. The step of a period's accuracy is **0.0050**, the step of the six periods is **0.0008**. - **RG14.** The two arms being compared see **the same rows** and are measured against the same label. There is no selection bias coming from the split of the arms; this is an assumption that must be shown separately in a real experiment, and here it holds by the fiction. - **RG15.** The production data has a **data drift** and the rule has not changed. - **RG16.** The candidates are the first four ranks of the previous lesson's retraining search; the grid is **135 candidates** from 27 weights and 5 decision thresholds. #### The Same Candidate Moving From Period to Period The first block sets up the setup, takes the four candidates and measures the difference of each from the baseline model **period by period**. ```python # FICTIONAL. The meter readings of the measurement network, the consumption stage of the # pipeline, the feature path and the three-weight threshold model are generated inside # the lesson; no learning library, experiment platform or monitoring product is # called. Seed 20260218. SEED, M32 = 20260218, 0xFFFFFFFF def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value REGION = ["east", "west", "north", "south", "central"] TYPE = ["residential", "commercial"] PERIODS = [f"2026-{a:02d}" for a in range(1, 13)] SUBSCRIBERS = {} for i in range(200): r = rng(SEED + 37 * i) SUBSCRIBERS[f"A{i + 1:03d}"] = {"region": REGION[int(r() * 5)], "type": TYPE[int(r() * 2)], "start": 10000 + int(r() * 60000)} def rows(): """Source readings and consumption from the period difference: the first three stages of the pipeline.""" records = [] for i, (sub, a) in enumerate(SUBSCRIBERS.items()): e = a["start"] for p in PERIODS: r = rng(SEED + 101 * i + 7 * PERIODS.index(p)) e += 4 + int(r() * 46) records.append({"subscriber": sub, "period": p, "region": a["region"], "type": a["type"], "index": e}) t = {(k["subscriber"], k["period"]): k["index"] for k in records} return [dict(k, m3=k["index"] - t[(k["subscriber"], PERIODS[PERIODS.index(k["period"]) - 1])]) for k in records if PERIODS.index(k["period"])] def features(records): """Independent of the weights and computed ONCE; it sees the full history.""" sub, reg = {}, {} for k in sorted(records, key=lambda k: PERIODS.index(k["period"])): sub.setdefault(k["subscriber"], []).append((PERIODS.index(k["period"]), k["m3"])) reg.setdefault(k["region"], []).append((PERIODS.index(k["period"]), k["m3"])) def mean(series, i): s = [v for d, v in series if d < i] return round(sum(s) / len(s), 4) if s else 0.0 return {(k["subscriber"], k["period"]): { "subscriber_mean": mean(sub[k["subscriber"]], PERIODS.index(k["period"])), "region_mean": mean(reg[k["region"]], PERIODS.index(k["period"])), "commercial": 1 if k["type"] == "commercial" else 0} for k in records} RULE_THRESHOLD, RULE_COMMERCIAL = 30, 6 # the RULE that generates the label , in other words the concept MONITORED = PERIODS[6:] # six periods monitored in production GRID = [(a, b, c) for a in (0.6, 0.8, 1.0) for b in (0.0, 0.2, 0.4) for c in (0, 4, 8)] DECISION_GRID = (18, 22, 26, 30, 34) # the decision threshold is A SEPARATE setting def label(o, threshold=RULE_THRESHOLD): return 1 if o["subscriber_mean"] + RULE_COMMERCIAL * o["commercial"] > threshold else 0 def score(o, w): return w[0] * o["subscriber_mean"] + w[1] * o["region_mean"] + w[2] * o["commercial"] def predict(o, w, decision): return 1 if score(o, w) > decision else 0 def balanced_accuracy(subset, feats, w, decision): """Training criterion: the mean of recall and specificity.""" tp = tn = fp = fn = 0 for k in subset: o = feats[(k["subscriber"], k["period"])] p, g = predict(o, w, decision), label(o) if p and g: tp += 1 elif p: fp += 1 elif g: fn += 1 else: tn += 1 recall = tp / (tp + fn) if tp + fn else 0.0 specificity = tn / (tn + fp) if tn + fp else 0.0 return (recall + specificity) / 2, round(recall, 4), round(specificity, 4) def train(records, feats, periods): """Weights and decision threshold are searched together: 27 x 5 = 135 candidates.""" subset = [k for k in records if k["period"] in periods] ranked = sorted(((balanced_accuracy(subset, feats, w, dc)[0], w, dc) for w in GRID for dc in DECISION_GRID), key=lambda a: -a[0]) best = ranked[0] return {"weights": best[1], "decision": best[2], "balanced": round(best[0], 4), "ranked": ranked, "rows": len(subset)} def drift(records, start="2026-09", slope=0.06): """FICTIONAL data drift: commercial consumption grows by `slope` every period.""" b = PERIODS.index(start) return [dict(k, m3=round(k["m3"] * (1.0 + slope * (PERIODS.index(k["period"]) - b + 1)), 4)) if PERIODS.index(k["period"]) >= b and k["type"] == "commercial" else dict(k) for k in records] ALL = rows() FEATS = features(ALL) BASELINE = train(ALL, FEATS, PERIODS[1:9]) PRODUCTION = drift(ALL) # the input has drifted in production FEATS_D = features(PRODUCTION) monitored_rows = [k for k in ALL if k["period"] in MONITORED] labels = [label(FEATS[(k["subscriber"], k["period"])]) for k in monitored_rows] print(f"baseline model {BASELINE['weights']}/{BASELINE['decision']}" f" , monitoring {len(monitored_rows)} rows , base class ratio {round(1 - sum(labels) / len(labels), 4)}") RETRAINED = train(PRODUCTION, FEATS_D, PERIODS[5:11]) # the retraining search of the previous lesson CANDIDATES = [{"name": name, "weights": s[1], "decision": s[2], "train": round(s[0], 4)} for name, s in zip("ABCD", RETRAINED["ranked"][:4])] def diff(m, subset): """The accuracy difference of the candidate from the baseline model. Same rows , same label.""" def correct(x): return sum(1 for k in subset if predict(FEATS_D[(k["subscriber"], k["period"])], x["weights"], x["decision"]) == label(FEATS_D[(k["subscriber"], k["period"])])) return round((correct(m) - correct(BASELINE)) / len(subset), 4) PER_PERIOD = {m["name"]: [diff(m, [k for k in PRODUCTION if k["period"] == d]) for d in MONITORED] for m in CANDIDATES} print("cand weights/decision training balanced accuracy prediction disagreement") for m in CANDIDATES: disagree = sum(1 for k in PRODUCTION if k["period"] in MONITORED and predict(FEATS_D[(k["subscriber"], k["period"])], m["weights"], m["decision"]) != predict(FEATS_D[(k["subscriber"], k["period"])], BASELINE["weights"], BASELINE["decision"])) print(f"{m['name']} {str(m['weights']) + '/' + str(m['decision']):<20}" f" {m['train']:>26} {disagree:>23}") print("\nperiod A B C D") for i, d in enumerate(MONITORED): print(f"{d} " + " ".join(f"{PER_PERIOD[m['name']][i]:+.4f}" for m in CANDIDATES)) overall = {m["name"]: diff(m, [k for k in PRODUCTION if k["period"] in MONITORED]) for m in CANDIDATES} print("6 periods " + " ".join(f"{overall[m['name']]:+.4f}" for m in CANDIDATES)) print("band " + " ".join(f"{max(PER_PERIOD[m['name']]) - min(PER_PERIOD[m['name']]):.4f}" for m in CANDIDATES)) ``` ``` baseline model (0.6, 0.0, 4)/18 , monitoring 1200 rows , base class ratio 0.5342 cand weights/decision training balanced accuracy prediction disagreement A (0.6, 0.0, 4)/18 0.98 0 B (0.8, 0.4, 4)/34 0.9741 60 C (1.0, 0.0, 8)/30 0.9492 33 D (1.0, 0.2, 4)/34 0.9434 97 period A B C D 2026-07 +0.0000 +0.0250 -0.0200 -0.0150 2026-08 +0.0000 +0.0000 -0.0400 -0.0250 2026-09 +0.0000 -0.0150 -0.0350 -0.0500 2026-10 +0.0000 -0.0150 -0.0250 -0.0450 2026-11 +0.0000 -0.0100 -0.0250 -0.0400 2026-12 +0.0000 -0.0150 -0.0200 -0.0400 6 periods +0.0000 -0.0050 -0.0275 -0.0358 band 0.0000 0.0400 0.0200 0.0350 ``` Column A is entirely zero, and this is not a measurement result but a result of **definition**: A's prediction disagreement is 0, that is, the two models say the same thing on every row. The difference between them cannot be measured at any set size, because there is no difference to measure. This is the first question to ask before starting the experiment. Column B is the real finding. Over the six periods B is **0.0050** worse than the baseline model — exactly one step of a single period's resolution. But the period values move in a band **0.0400** wide, between **+0.0250** and **−0.0150**, and **change sign**. A single-period experiment run on `2026-07` declares B **five resolution steps better** than the baseline model. The number is not wrong; in that period it really is so. What is wrong is drawing a ranking out of that number. For C and D the band is wide too (0.0200 and 0.0350) but the **sign never turns in any period**. Both are below the baseline model in every period, and a single-period experiment ranks them correctly. #### The Minimum Detectable Effect The second block does two things. First it enlarges the experiment set and follows where the difference settles; then it looks for the limit by sweeping **all 135 candidates** in the grid. ```python # This block continues the CANDIDATES, diff, MONITORED, PRODUCTION, BASELINE, # GRID and DECISION_GRID definitions of the first block. print("period count rows resolution B C D") for n in range(1, 7): sub = [k for k in PRODUCTION if k["period"] in MONITORED[:n]] print(f"{n:>12} {len(sub):>5} {1 / len(sub):>10.4f} " + " ".join(f"{diff(m, sub):+.4f}" for m in CANDIDATES[1:])) # All 135 candidates: does a single-period experiment rank the 6-period result correctly SETS = {d: [k for k in PRODUCTION if k["period"] == d] for d in MONITORED} ALL_ROWS = [k for k in PRODUCTION if k["period"] in MONITORED] consistent, flipping = [], [] for w in GRID: for dc in DECISION_GRID: m = {"weights": w, "decision": dc} h = diff(m, ALL_ROWS) if h == 0: continue sign = (h > 0) - (h < 0) ps = [diff(m, SETS[d]) for d in MONITORED] (consistent if all((p > 0) - (p < 0) == sign for p in ps) else flipping).append(abs(h)) print(f"\ncandidates with nonzero 6-period diff: {len(consistent) + len(flipping)}") print(f"same sign in all six periods: candidates {len(consistent)}" f" , smallest diff {min(consistent):.4f}") print(f"sign flips from period to period: candidates {len(flipping)}" f" , largest diff {max(flipping):.4f}") ``` ``` period count rows resolution B C D 1 200 0.0050 +0.0250 -0.0200 -0.0150 2 400 0.0025 +0.0125 -0.0300 -0.0200 3 600 0.0017 +0.0033 -0.0317 -0.0300 4 800 0.0013 -0.0013 -0.0300 -0.0338 5 1000 0.0010 -0.0030 -0.0290 -0.0350 6 1200 0.0008 -0.0050 -0.0275 -0.0358 candidates with nonzero 6-period diff: 134 same sign in all six periods: candidates 133 , smallest diff 0.0275 sign flips from period to period: candidates 1 , largest diff 0.0050 ``` The table above shows the resolution improving with the set size: 0.0050 on 200 rows, **0.0008** on 1200 rows. No difference below this can even be **represented**; it is the smallest nonzero number a 1200-row experiment can give. B's sign turns in the **fourth period**, that is at 800 rows, and even there its value is −0.0013, exactly one step of that set. C and D carry the right sign from the first period on, and enlarging only settles the value. The sweep below gives the limit. Of the candidates in the grid, 134 have a nonzero 6-period difference. **133 of them** carry the same sign in all six of six periods, that is, they are ranked correctly whichever single period they are measured in; the **smallest difference of this group is 0.0275**. The **single candidate** that moves the sign is B, and its difference is **0.0050**. What comes out is an interval: **the minimum difference a single-period experiment can confidently distinguish on this set lies somewhere between 0.0050 and 0.0275.** It cannot be stated more narrowly because the grid contains no candidate in this interval, and it **should not be stated**; the uncertainty of the measure is part of the measure. The practical counterpart is single: **every comparison that declares a difference smaller than 0.0275 with a single-period experiment is inconclusive**, and B is the marked example of this. Why such a large threshold is needed is written in the prediction disagreement column. B and the baseline model say **the same thing on 1140 of the 1200 rows**; the real sample of the experiment is not 1200 but the **60 rows** on which they disagree. When two predictions disagree only one of them can be right, so the six-row total difference means that in 33 of these sixty the baseline is right and in 27 B is right. The denominator is 1200 but the information comes from 60 rows; the 1140 rows that do not disagree add nothing to the experiment and only make the difference look small. **The minimum difference an experiment can distinguish is limited not by the size of the set but by the number of rows on which the two models disagree.** #### Summary - The same candidate is measured at **−0.0050** over six periods and between **+0.0250** and **−0.0150** in single periods; its band is **0.0400** and its sign turns. The base class ratio is 0.5342, the period resolution is 0.0050, the resolution of the six periods is **0.0008**. - A candidate whose prediction disagreement is **0** cannot be distinguished by any experiment, because there is no difference; this is the question to ask before the experiment. - **133** of the grid's 134 candidates carry the same sign in every period and the smallest difference of this group is **0.0275**; the difference of the single candidate that moves the sign is **0.0050**. The minimum detectable effect lies between these two numbers and the grid contains no candidate that would narrow the interval. - **A comparison that declares a difference smaller than 0.0275 with a single period is inconclusive.** Enlarging the set lowers the resolution to 0.0008 but turns B's sign only in the fourth period, at 800 rows. - The real sample of the experiment is not 1200 rows but the **60 rows** on which the two models disagree; the total difference comes from the baseline being right in 33 of these sixty and the candidate in 27, and the remaining 1140 rows only enlarge the denominator. #### Next Step The experiment showed how expensive comparing two models is: for a distinguishable difference either the disagreement must be large or many periods must be waited for. This sharpens the question behind it even more: **why is the new model being trained.** Because an alert fired, because the calendar came, or because accuracy really dropped. The next lesson runs three kinds of triggers in two separate degradation scenarios, counts how many models each one causes to be trained and writes the accuracy gained next to it. On the data with a data drift, all three retraining windows give the same number, and that number is the old model's number. ### Retraining Policy Source: https://academia.sh/en/courses/model-monitoring/retraining-policy Counting the triggers that decide when a model is retrained and what that decision really gains: in production with a data drift all three retraining windows give 0.9800 in the period 2026-12, and this is exactly the same as the number of the old model that was never retrained. When the calendar, indicator and performance triggers are run in two degradation scenarios, the accuracy gained is 0.0000 in all six of six combinations, and in return 3,078,000 rows are scored. That what determines the gain is the window and not the trigger is shown by the retraining triggered at the same moment under a concept drift gaining 0.0000 with window 6 and 0.1100 with window 1. The previous lesson measured the cost of comparing two models in production: the minimum difference a single-period experiment can distinguish on this set lies between 0.0050 and 0.0275, and every comparison below it is inconclusive. That comparing is expensive sharpens the question before it: **why is the new model being trained.** In production the answer to this question is a **retraining trigger**: the calendar has come, an indicator has moved outside its band, or accuracy has really dropped. This lesson runs three kinds of triggers in two separate degradation scenarios and writes two numbers next to each: **how many models were trained** and **how much accuracy was gained**. The second number is the clearest example of the second claim of this course. - **RG21.** The source, the pipeline and the model are **fictional**; no learning library, orchestrator, deployment tool or monitoring product is called. Seed 20260218. - **RG22.** The label is tied to the **rule** `subscriber_mean + 6 x commercial > 30` and this rule **is the concept**. The decision threshold is a separate setting, the training criterion is **balanced accuracy**, that is, the mean of **recall** and **specificity**. In the monitored periods the **base class ratio is 0.5342**, there are 200 rows per period and the step of accuracy is **0.0050**. - **RG23.** There are two scenarios. In the **data drift** the input drifts and **the rule stays the same**. In the **concept drift** the input stays the same and the threshold of the rule falls from 30 to 27 starting from `2026-09`. - **RG24.** The label is **delayed**: the label of a period is known **two periods later**. Retraining can use only the labeled periods. - **RG25.** Training cost is counted as **rows scored**: every retraining scores the rows in the window for 135 candidates. Raw time is not written. - **RG26.** The gain is measured against the `2026-12` accuracy of the old model that was never retrained in that scenario. #### Window Sweep: The Best Case The first block sets up the setup and retrains with three separate windows in production with a data drift. The label delay is **ignored** in this block, that is, the best case that can be reached is measured. ```python # FICTIONAL. The meter readings of the measurement network, the consumption stage of the # pipeline, the feature path and the three-weight threshold model are generated inside # the lesson; no learning library, deployment tool, experiment platform or monitoring # product is called. # Seed 20260218. SEED, M32 = 20260218, 0xFFFFFFFF def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value REGION = ["east", "west", "north", "south", "central"] TYPE = ["residential", "commercial"] PERIODS = [f"2026-{a:02d}" for a in range(1, 13)] SUBSCRIBERS = {} for i in range(200): r = rng(SEED + 37 * i) SUBSCRIBERS[f"A{i + 1:03d}"] = {"region": REGION[int(r() * 5)], "type": TYPE[int(r() * 2)], "start": 10000 + int(r() * 60000)} def rows(): """Source readings and consumption from the period difference: the first three stages of the pipeline.""" records = [] for i, (sub, a) in enumerate(SUBSCRIBERS.items()): e = a["start"] for p in PERIODS: r = rng(SEED + 101 * i + 7 * PERIODS.index(p)) e += 4 + int(r() * 46) records.append({"subscriber": sub, "period": p, "region": a["region"], "type": a["type"], "index": e}) t = {(k["subscriber"], k["period"]): k["index"] for k in records} return [dict(k, m3=k["index"] - t[(k["subscriber"], PERIODS[PERIODS.index(k["period"]) - 1])]) for k in records if PERIODS.index(k["period"])] def features(records): """Independent of the weights and computed ONCE; it sees the full history.""" sub, reg = {}, {} for k in sorted(records, key=lambda k: PERIODS.index(k["period"])): sub.setdefault(k["subscriber"], []).append((PERIODS.index(k["period"]), k["m3"])) reg.setdefault(k["region"], []).append((PERIODS.index(k["period"]), k["m3"])) def mean(series, i): s = [v for d, v in series if d < i] return round(sum(s) / len(s), 4) if s else 0.0 return {(k["subscriber"], k["period"]): { "subscriber_mean": mean(sub[k["subscriber"]], PERIODS.index(k["period"])), "region_mean": mean(reg[k["region"]], PERIODS.index(k["period"])), "commercial": 1 if k["type"] == "commercial" else 0} for k in records} RULE_THRESHOLD, RULE_COMMERCIAL = 30, 6 # the RULE that generates the label , in other words the concept MONITORED = PERIODS[6:] # six periods monitored in production GRID = [(a, b, c) for a in (0.6, 0.8, 1.0) for b in (0.0, 0.2, 0.4) for c in (0, 4, 8)] DECISION_GRID = (18, 22, 26, 30, 34) # the decision threshold is A SEPARATE setting def label(o, threshold=RULE_THRESHOLD): return 1 if o["subscriber_mean"] + RULE_COMMERCIAL * o["commercial"] > threshold else 0 def score(o, w): return w[0] * o["subscriber_mean"] + w[1] * o["region_mean"] + w[2] * o["commercial"] def predict(o, w, decision): return 1 if score(o, w) > decision else 0 def train(records, feats, periods, labels): """Weights and decision threshold are searched TOGETHER (27 x 5 = 135 candidates); the criterion is the mean of recall and specificity, that is BALANCED ACCURACY.""" subset = [k for k in records if k["period"] in periods] best = (-1.0, None, None, 0.0, 0.0) for w in GRID: for dc in DECISION_GRID: tp = tn = fp = fn = 0 for k in subset: o = feats[(k["subscriber"], k["period"])] p, g = predict(o, w, dc), labels[(k["subscriber"], k["period"])] if p and g: tp += 1 elif p: fp += 1 elif g: fn += 1 else: tn += 1 recall = tp / (tp + fn) if tp + fn else 0.0 spec = tn / (tn + fp) if tn + fp else 0.0 if (recall + spec) / 2 > best[0]: best = ((recall + spec) / 2, w, dc, recall, spec) return {"weights": best[1], "decision": best[2], "balanced": round(best[0], 4), "recall": round(best[3], 4), "specificity": round(best[4], 4), "rows": len(subset)} def drift(records, start="2026-09", slope=0.06): """FICTIONAL data drift: commercial consumption grows by `slope` every period.""" b = PERIODS.index(start) return [dict(k, m3=round(k["m3"] * (1.0 + slope * (PERIODS.index(k["period"]) - b + 1)), 4)) if PERIODS.index(k["period"]) >= b and k["type"] == "commercial" else dict(k) for k in records] def concept_drift(feats, records, start="2026-09", new_threshold=27): """FICTIONAL concept drift: SAME INPUT , the rule changes (threshold 30 to 27).""" b = PERIODS.index(start) return {(k["subscriber"], k["period"]): label( feats[(k["subscriber"], k["period"])], new_threshold if PERIODS.index(k["period"]) >= b else RULE_THRESHOLD) for k in records} def accuracy(records, feats, m, period, labels): subset = [k for k in records if k["period"] == period] return round(sum(1 for k in subset if predict(feats[(k["subscriber"], k["period"])], m["weights"], m["decision"]) == labels[(k["subscriber"], k["period"])]) / len(subset), 4) def prediction_share(records, feats, m, period): subset = [k for k in records if k["period"] == period] return round(sum(predict(feats[(k["subscriber"], k["period"])], m["weights"], m["decision"]) for k in subset) / len(subset), 4) ALL = rows() FEATS = features(ALL) PRODUCTION = drift(ALL) # the input has drifted in production FEATS_D = features(PRODUCTION) LABELS_D = {c: label(o) for c, o in FEATS_D.items()} # data drift: SAME RULE LABELS_C = concept_drift(FEATS, ALL) # concept drift: the rule changed BASELINE = train(ALL, FEATS, PERIODS[1:9], {c: label(o) for c, o in FEATS.items()}) monitored_rows = [k for k in ALL if k["period"] in MONITORED] base_labels = [label(FEATS[(k["subscriber"], k["period"])]) for k in monitored_rows] print(f"baseline model {BASELINE['weights']}/{BASELINE['decision']} balanced accuracy {BASELINE['balanced']}" f" recall {BASELINE['recall']:.4f} specificity {BASELINE['specificity']:.4f}") print(f"monitoring {len(monitored_rows)} rows , base class ratio {round(1 - sum(base_labels) / len(base_labels), 4)}" f" , 200 rows per period , accuracy step 0.0050") # Retraining in production with a data drift: the BEST case that could be reached # if all labels were at hand , that is if there were no delay at all. print("\n window training periods rows balanced accuracy 2026-12 accuracy") for p in (3, 6, 11): span = [d for d in PERIODS if PERIODS.index("2026-11") - p < PERIODS.index(d) <= PERIODS.index("2026-11")] m = train(PRODUCTION, FEATS_D, span, LABELS_D) print(f"{p:>7} {span[0]}..{span[-1]} {m['rows']:>5} {m['balanced']:>17}" f" {accuracy(PRODUCTION, FEATS_D, m, '2026-12', LABELS_D):>16}") print(f" old model {BASELINE['balanced']}" f" {accuracy(PRODUCTION, FEATS_D, BASELINE, '2026-12', LABELS_D):>16}") ``` ``` baseline model (0.6, 0.0, 4)/18 balanced accuracy 0.9864 recall 1.0000 specificity 0.9728 monitoring 1200 rows , base class ratio 0.5342 , 200 rows per period , accuracy step 0.0050 window training periods rows balanced accuracy 2026-12 accuracy 3 2026-09..2026-11 600 0.9824 0.98 6 2026-06..2026-11 1200 0.98 0.98 11 2026-01..2026-11 2000 0.9849 0.98 old model 0.9864 0.98 ``` As the training window grows, the balanced accuracy on training moves: 0.9824, 0.9800, 0.9849. These three numbers give the impression that a choice can be made. The last column erases that impression. **All three windows give 0.9800 on `2026-12`**, and this is **exactly the same** as the number of the old model that was never retrained. All three have found the same weights and the same decision threshold, because the **rule** the search is trying to solve **has not changed**. The input has drifted, but as long as the relation that produces the label stays in place, a correct model stays correct on a drifted input as well. The accuracy gained is **0.0000** and the training cost has been paid. #### Trigger or Window The second block runs the three triggers in the two scenarios. The label delay is now applied: a retraining triggered in a period sees labeled data only up to **two periods earlier**. The `fired` column is the number of periods in which the trigger fired, that is, how many models were trained. ```python # This block continues the ALL, FEATS, PRODUCTION, FEATS_D, BASELINE, LABELS_D, LABELS_C, # MONITORED, GRID, DECISION_GRID, train, accuracy and prediction_share definitions # of the first block. DELAY, WINDOW, CACHE = 2, 6, {} def trigger(name, records, feats, labels, now, window=WINDOW): """The label is delayed: in period `now` only `now - DELAY` is labeled.""" i = PERIODS.index(now) - DELAY span = [d for d in PERIODS if i - window < PERIODS.index(d) <= i] key = (name, span[0], span[-1], window) if key not in CACHE: CACHE[key] = train(records, feats, span, labels) m = CACHE[key] return m, m["rows"] * len(GRID) * len(DECISION_GRID) def deviating(series, base, band=0.05): return [d for d, v in series if abs(v - base) / base > band] print("scenario trigger fired rows scored 2026-12 gain") for name, recs, feats, lab in (("data drift", PRODUCTION, FEATS_D, LABELS_D), ("concept drift", ALL, FEATS, LABELS_C)): base_12 = accuracy(recs, feats, BASELINE, "2026-12", lab) base_08 = accuracy(recs, feats, BASELINE, "2026-08", lab) for t, fired_periods in ( ("calendar", list(MONITORED)), ("indicator", deviating([(d, prediction_share(recs, feats, BASELINE, d)) for d in MONITORED], prediction_share(recs, feats, BASELINE, "2026-08"))), ("performance", deviating([(d, accuracy(recs, feats, BASELINE, PERIODS[PERIODS.index(d) - DELAY], lab)) for d in MONITORED[DELAY:]], base_08))): cost, last_model = 0, None for d in fired_periods: m, b = trigger(name, recs, feats, lab, d) cost += b last_model = m if PERIODS.index(d) < PERIODS.index("2026-12") else last_model acc_12 = accuracy(recs, feats, last_model, "2026-12", lab) if last_model else base_12 print(f"{name:<15} {t:<11} {len(fired_periods):>9} {cost:>15} {acc_12:>7} {acc_12 - base_12:+.4f}") print("\nconcept drift , retraining triggered in 2026-11: window choice") for p in (1, 3, 6): m, _ = trigger("cw", ALL, FEATS, LABELS_C, "2026-11", window=p) acc_12 = accuracy(ALL, FEATS, m, "2026-12", LABELS_C) print(f" window {p} {str(m['weights']) + '/' + str(m['decision']):<18}" f" training rows {m['rows']:>4} 2026-12 accuracy {acc_12}" f" gain {acc_12 - accuracy(ALL, FEATS, BASELINE, '2026-12', LABELS_C):+.4f}") ``` ``` scenario trigger fired rows scored 2026-12 gain data drift calendar 6 891000 0.98 +0.0000 data drift indicator 3 486000 0.98 +0.0000 data drift performance 0 0 0.98 +0.0000 concept drift calendar 6 891000 0.85 +0.0000 concept drift indicator 3 486000 0.85 +0.0000 concept drift performance 2 324000 0.85 +0.0000 concept drift , retraining triggered in 2026-11: window choice window 1 (0.6, 0.2, 4)/22 training rows 200 2026-12 accuracy 0.96 gain +0.1100 window 3 (0.8, 0.4, 4)/34 training rows 600 2026-12 accuracy 0.85 gain +0.0000 window 6 (0.6, 0.0, 4)/18 training rows 1200 2026-12 accuracy 0.85 gain +0.0000 ``` **In all six of six rows the gain is 0.0000.** A total of **3,078,000 rows** have been scored, retraining has run twenty times, and the number read on `2026-12` has come out every time as the number of the case without a trigger. Reading the rows one by one shows that there are three separate reasons for this. **The performance trigger never fires in the data drift** and this is the correct decision: there is nothing to gain, and its cost is 0. The same trigger fires in the concept drift but **two periods late**; the accuracy that dropped in `2026-09` is known only in `2026-11`. **The indicator trigger fires in the same three periods in both scenarios.** It does not tell the scenario with harmful degradation from the harmless one, because the prediction share it looks at drifts in the same way in both. The rule of the course is written on this row: **an indicator's firing is not evidence of degradation.** **The calendar trigger** looks at nothing, trains the most models and pays the largest cost. The last table gives the real reason. In the concept drift, the retraining triggered in `2026-11`, **at the same moment, with the same data and the same grid**, gives 0.8500 with window 6 and **0.9600** with window 1; the gain varies between **0.0000** and **0.1100**. Window 6 sees together the periods of which one belongs to the new rule and five to the old rule, and the search that maximizes balanced accuracy follows **the majority** and finds the old model again. Window 1 sees only the period of the new rule and lands on another model. **What retraining gains is not the trigger but which rule's periods the window contains.** The gain of 0.1100 is far above the minimum detectable effect of the previous lesson; 0.0000 is zero anyway. The measurable counterpart of the approval flow comes from here. When a retraining proposal comes up for approval, the sentence "the indicator fired" alone is not a justification; the three numbers that make the decision possible are **its window**, **the number of predictions on which the candidate disagrees with the old model** and **its gain measured in the shadow**. When the three are not written, the approval also approves a training whose result is 0.0000. #### Summary - In production with a data drift, three retraining windows give 0.9824, 0.9800 and 0.9849 in training, but on `2026-12` **all three are 0.9800** — **exactly the same** as the old model that was never retrained. The base class ratio is 0.5342, the step of accuracy is 0.0050. **If the rule has not changed, retraining gains nothing** and the training cost has been paid. - In **all six of six combinations** of three triggers and two scenarios the gain is 0.0000; in return **3,078,000 rows** are scored. The calendar trigger is the most expensive (891,000 rows), and the performance trigger never fires in the data drift and its cost is 0. - **The indicator trigger fires in the same three periods in both scenarios**; it does not tell harmful degradation from the harmless one. An indicator's firing is not evidence of degradation. - The performance trigger sees the concept drift but sees it **two periods late**; the accuracy that dropped in `2026-09` triggers in `2026-11`. - What determines the gain is not the trigger but the **window**: the retraining triggered at the same moment gives **+0.0000** with window 6 and **+0.1100** with window 1. A window containing the periods of the old rule adapts the search to the majority and finds the old model again. #### Next Step Over three lessons the model was measured in the shadow, given to the canary, compared by experiment and retrained. At every step a new model, a new window and a new decision threshold came out. The only thing not asked is to whom a **single prediction** given to a subscriber on `2026-12` will be put months later: which model produced it, with which feature values, with which decision threshold. The next lesson counts this question as a search — how many rows are read to find the reason for a prediction with and without a decision record, how many predictions in the canary period **cannot be explained by any reading**, and how many fields the record itself costs to store. ### Model Auditability Source: https://academia.sh/en/courses/model-monitoring/model-auditability Counting, with and without a decision record, whether the reason for a single produced prediction can still be reached months later: with the decision record a prediction's model, decision threshold, feature values and score are read on one row, and 1.00 row falls per prediction. Without the record 84,740 rows are read for the same two hundred predictions, which comes to 423.70 per prediction, and even so the model of 189 predictions cannot be determined, because in the canary period the two models say the same thing for 189 of the predictions. The cost of the record is 12,000 stored fields for six periods, and this is 0.8333 times the source readings. Over three lessons the model was measured in the shadow, given to the canary, compared by experiment and retrained. At every step a new weight, a new decision threshold and a new training window came out. The only thing not asked is **to whom** a single prediction given to a single subscriber in the period `2026-12` will be put months later: which model produced it, with which feature values, with which decision threshold. This question is separate from the model's **interpretability**. Which feature drives a model's decision, how it behaves on subgroups and how it will be explained was measured in the Model Evaluation, Interpretability and Ethics course of the Machine Learning and Data Science curriculum and is not repeated here. The question here is production's question: **which model was running.** Interpretability works when the model is at hand; the **audit trail** is the record itself that says which model it was. This lesson builds that record, counts the search made when it is missing and writes the cost of the record. - **RG31.** The source, the pipeline, the model and the release are **fictional**; no model registry, monitoring product or deployment tool is called. Seed 20260218. - **RG32.** The label is tied to the rule `subscriber_mean + 6 x commercial > 30`, and the decision threshold is a separate setting. In the monitored periods the **base class ratio is 0.5342**, there are 200 rows per period and the step of accuracy is **0.0050**. - **RG33.** The audited period is `2026-12`, and in that period **two models are open**: the old model and the candidate rolled out with a canary share of 0.10. - **RG34.** The **decision record** is a single row for every prediction and carries ten fields: period, subscriber, model, weights, decision threshold, three feature values, score and prediction. - **RG35.** The search cost is counted in **rows read**; raw time is not written. The search without a record reads 2 rows from the release record, then the raw rows to recompute the feature. - **RG36.** The storage cost is counted in **fields**: the number of records times the number of fields. #### What the Decision Record Carries The first block sets up the setup, runs the `2026-12` release with a canary share of 0.10 and produces a decision record for every prediction. ```python # FICTIONAL. The meter readings of the measurement network, the consumption stage of the # pipeline, the feature path and the three-weight threshold model are generated inside # the lesson; no learning library, deployment tool, experiment platform or monitoring # product is called. # Seed 20260218. SEED, M32 = 20260218, 0xFFFFFFFF def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value REGION = ["east", "west", "north", "south", "central"] TYPE = ["residential", "commercial"] PERIODS = [f"2026-{a:02d}" for a in range(1, 13)] SUBSCRIBERS = {} for i in range(200): r = rng(SEED + 37 * i) SUBSCRIBERS[f"A{i + 1:03d}"] = {"region": REGION[int(r() * 5)], "type": TYPE[int(r() * 2)], "start": 10000 + int(r() * 60000)} def rows(): """Source readings and consumption from the period difference: the first three stages of the pipeline.""" records = [] for i, (sub, a) in enumerate(SUBSCRIBERS.items()): e = a["start"] for p in PERIODS: r = rng(SEED + 101 * i + 7 * PERIODS.index(p)) e += 4 + int(r() * 46) records.append({"subscriber": sub, "period": p, "region": a["region"], "type": a["type"], "index": e}) t = {(k["subscriber"], k["period"]): k["index"] for k in records} return [dict(k, m3=k["index"] - t[(k["subscriber"], PERIODS[PERIODS.index(k["period"]) - 1])]) for k in records if PERIODS.index(k["period"])] def features(records): """Independent of the weights and computed ONCE; it sees the full history.""" sub, reg = {}, {} for k in sorted(records, key=lambda k: PERIODS.index(k["period"])): sub.setdefault(k["subscriber"], []).append((PERIODS.index(k["period"]), k["m3"])) reg.setdefault(k["region"], []).append((PERIODS.index(k["period"]), k["m3"])) def mean(series, i): s = [v for d, v in series if d < i] return round(sum(s) / len(s), 4) if s else 0.0 return {(k["subscriber"], k["period"]): { "subscriber_mean": mean(sub[k["subscriber"]], PERIODS.index(k["period"])), "region_mean": mean(reg[k["region"]], PERIODS.index(k["period"])), "commercial": 1 if k["type"] == "commercial" else 0} for k in records} RULE_THRESHOLD, RULE_COMMERCIAL = 30, 6 # the RULE that generates the label , in other words the concept MONITORED = PERIODS[6:] # six periods monitored in production GRID = [(a, b, c) for a in (0.6, 0.8, 1.0) for b in (0.0, 0.2, 0.4) for c in (0, 4, 8)] DECISION_GRID = (18, 22, 26, 30, 34) # the decision threshold is A SEPARATE setting def label(o, threshold=RULE_THRESHOLD): return 1 if o["subscriber_mean"] + RULE_COMMERCIAL * o["commercial"] > threshold else 0 def score(o, w): return w[0] * o["subscriber_mean"] + w[1] * o["region_mean"] + w[2] * o["commercial"] def predict(o, w, decision): return 1 if score(o, w) > decision else 0 def balanced_accuracy(subset, feats, w, decision): """Training criterion: the mean of recall and specificity.""" tp = tn = fp = fn = 0 for k in subset: o = feats[(k["subscriber"], k["period"])] p, g = predict(o, w, decision), label(o) if p and g: tp += 1 elif p: fp += 1 elif g: fn += 1 else: tn += 1 recall = tp / (tp + fn) if tp + fn else 0.0 spec = tn / (tn + fp) if tn + fp else 0.0 return (recall + spec) / 2, round(recall, 4), round(spec, 4) def train(records, feats, periods): """Weights and decision threshold are searched together: 27 x 5 = 135 candidates.""" subset = [k for k in records if k["period"] in periods] ranked = sorted(((balanced_accuracy(subset, feats, w, dc)[0], w, dc) for w in GRID for dc in DECISION_GRID), key=lambda a: -a[0]) best = ranked[0] return {"weights": best[1], "decision": best[2], "balanced": round(best[0], 4), "ranked": ranked, "rows": len(subset)} def drift(records, start="2026-09", slope=0.06): """FICTIONAL data drift: commercial consumption grows by `slope` every period.""" b = PERIODS.index(start) return [dict(k, m3=round(k["m3"] * (1.0 + slope * (PERIODS.index(k["period"]) - b + 1)), 4)) if PERIODS.index(k["period"]) >= b and k["type"] == "commercial" else dict(k) for k in records] ALL = rows() FEATS = features(ALL) BASELINE = train(ALL, FEATS, PERIODS[1:9]) PRODUCTION = drift(ALL) # the input has drifted in production FEATS_D = features(PRODUCTION) bal, recall, spec = balanced_accuracy([k for k in ALL if k["period"] in PERIODS[1:9]], FEATS, BASELINE["weights"], BASELINE["decision"]) monitored_rows = [k for k in ALL if k["period"] in MONITORED] labels = [label(FEATS[(k["subscriber"], k["period"])]) for k in monitored_rows] print(f"baseline model {BASELINE['weights']} decision threshold {BASELINE['decision']}" f" balanced accuracy {BASELINE['balanced']} recall {recall:.4f} specificity {spec:.4f}") print(f"monitoring {len(monitored_rows)} rows , base class ratio {round(1 - sum(labels) / len(labels), 4)}" f" , 200 rows per period , accuracy step 0.0050") # This block continues the ALL, FEATS, PRODUCTION, FEATS_D, BASELINE, MONITORED, PERIODS, # SUBSCRIBERS, label, score and predict definitions of the first block. CANDIDATE = {"name": "candidate", "weights": (0.8, 0.4, 4), "decision": 34} OLD = dict(BASELINE, name="old") SHARE, LAST = 0.10, "2026-12" last_rows = [k for k in PRODUCTION if k["period"] == LAST] N_CANARY = int(len(last_rows) * SHARE) FIELDS = ("period", "subscriber", "model", "weights", "decision_threshold", "subscriber_mean", "region_mean", "commercial", "score", "prediction") def decision_record(subset, feats, n_canary): """A single row for every prediction: which model , which feature , which threshold.""" record = {} for j, k in enumerate(subset): o = feats[(k["subscriber"], k["period"])] m = CANDIDATE if j < n_canary else OLD record[(k["subscriber"], k["period"])] = dict( zip(FIELDS, (k["period"], k["subscriber"], m["name"], m["weights"], m["decision"], o["subscriber_mean"], o["region_mean"], o["commercial"], round(score(o, m["weights"]), 4), predict(o, m["weights"], m["decision"])))) return record RECORD = decision_record(last_rows, FEATS_D, N_CANARY) disagree = [k["subscriber"] for k in last_rows if predict(FEATS_D[(k["subscriber"], LAST)], OLD["weights"], OLD["decision"]) != predict(FEATS_D[(k["subscriber"], LAST)], CANDIDATE["weights"], CANDIDATE["decision"])] print(f"{LAST} release: old {OLD['weights']}/{OLD['decision']} ," f" candidate {CANDIDATE['weights']}/{CANDIDATE['decision']} , canary share {SHARE}") print(f"requests {len(last_rows)} , sent to canary {N_CANARY} ," f" predictions where the two models disagree {len(disagree)}") print("decision record fields:", " ".join(FIELDS)) print("example:", " ".join(str(v) for v in RECORD[("A001", LAST)].values())) ``` ``` baseline model (0.6, 0.0, 4) decision threshold 18 balanced accuracy 0.9864 recall 1.0000 specificity 0.9728 monitoring 1200 rows , base class ratio 0.5342 , 200 rows per period , accuracy step 0.0050 2026-12 release: old (0.6, 0.0, 4)/18 , candidate (0.8, 0.4, 4)/34 , canary share 0.1 requests 200 , sent to canary 20 , predictions where the two models disagree 11 decision record fields: period subscriber model weights decision_threshold subscriber_mean region_mean commercial score prediction example: 2026-12 A001 candidate (0.8, 0.4, 4) 34 35.502 27.6233 1 43.4509 1 ``` The example row is an explanation on its own: the `2026-12` prediction of the subscriber `A001` was produced by the **candidate** model, with the weights `(0.8, 0.4, 4)` and the decision threshold **34**; the subscriber mean is 35.502, the region mean is 27.6233, the subscriber type is commercial and the score is 43.4509. Since the score exceeds the threshold, the prediction is 1. The record requires no recomputation and says **which model was running**. #### The Search Without a Record The second block searches the same two hundred predictions without a record: the two models open in that period are found from the release record, the features are recomputed from the raw rows, and **which models agree** with the observed prediction is counted. ```python # This block continues the RECORD, FIELDS, OLD, CANDIDATE, last_rows, LAST, N_CANARY, # PRODUCTION, FEATS_D, SUBSCRIBERS and PERIODS definitions of the second block. def search_without_record(subscriber, period, records, observed): """Without a record: the models from the release record , the features from the raw rows.""" i, r = PERIODS.index(period), SUBSCRIBERS[subscriber]["region"] sub = [k["m3"] for k in records if k["subscriber"] == subscriber and PERIODS.index(k["period"]) < i] reg = [k["m3"] for k in records if k["region"] == r and PERIODS.index(k["period"]) < i] o = {"subscriber_mean": round(sum(sub) / len(sub), 4), "region_mean": round(sum(reg) / len(reg), 4), "commercial": 1 if SUBSCRIBERS[subscriber]["type"] == "commercial" else 0} matching = [m["name"] for m in (OLD, CANDIDATE) if predict(o, m["weights"], m["decision"]) == observed] return 2 + len(sub) + len(reg), matching # 2 rows of release record + raw rows read_with = read_without = single_with = single_without = 0 for k in last_rows: rec = RECORD[(k["subscriber"], LAST)] read_with, single_with = read_with + 1, single_with + 1 # search with a record: one row , one model n_read, matching = search_without_record(k["subscriber"], LAST, PRODUCTION, rec["prediction"]) read_without, single_without = read_without + n_read, single_without + (len(matching) == 1) print("search rows read per prediction model identified unidentified") for name, rd, one in (("with decision record", read_with, single_with), ("without decision record", read_without, single_without)): print(f"{name:<23} {rd:>12} {rd / len(last_rows):>14.2f} {one:>17} {len(last_rows) - one:>14}") over_threshold = sum(1 for c, rec in RECORD.items() if rec["prediction"] == 1) commercial_decisive = sum(1 for c, rec in RECORD.items() if rec["prediction"] == 1 and rec["score"] - rec["weights"][2] * rec["commercial"] <= rec["decision_threshold"]) print(f"\nreason read from the decision record: predictions over the threshold {over_threshold} ," f" of which {commercial_decisive} have the subscriber type decisive on its own") record_count = len(MONITORED) * len(last_rows) source_fields = len(SUBSCRIBERS) * len(PERIODS) * 6 print(f"had the six monitored periods been recorded: {record_count} records x {len(FIELDS)} fields =" f" {record_count * len(FIELDS)} stored fields") print(f" source readings {len(SUBSCRIBERS) * len(PERIODS)} rows x 6 fields = {source_fields} fields") print(f" decision record to source ratio {record_count * len(FIELDS) / source_fields:.4f}") ``` ``` search rows read per prediction model identified unidentified with decision record 200 1.00 200 0 without decision record 84740 423.70 11 189 reason read from the decision record: predictions over the threshold 109 , of which 57 have the subscriber type decisive on its own had the six monitored periods been recorded: 1200 records x 10 fields = 12000 stored fields source readings 2400 rows x 6 fields = 14400 fields decision record to source ratio 0.8333 ``` The rows read column gives the expected difference: **1.00** rows per prediction with the decision record, **423.70** rows without it. The factor of four hundred between them comes from the scope of the feature path; `region_mean` cannot be computed for a single subscriber, it asks for the history of all subscribers in the region. This much is a difference in cost, and it closes if enough data is read. The last two columns show what does not close. With the decision record **200 of 200 predictions** are tied to a single model. In the search without a record this number falls to **11**, and **the model of 189 predictions cannot be determined**. The reason is written in the table: in the canary period the two models say **the same thing for 189 of the 200 predictions**, so the observed output agrees with both. Recomputation brings the feature back but **does not bring back the information of which model was running**; that information is the product not of a measurable trace but of **a written record**. The inverse here is worth noting. The only group the search without a record can resolve is the 11 predictions on which the two models **disagree** — that is, the place where the audit works by itself is the place where the candidate really departs from the old model. The small shares of a canary are chosen precisely to reduce this disagreement. **The same choice that shrinks the risk also shrinks retrospective auditability**; the trade-off experienced with the resolution of the measurement in the first lesson meets us again here with the audit. The explanations for these two predictions are not the same. Two models that give the same output carry different weights, different decision thresholds and different scores; **when it cannot be chosen which one was valid, there is not one explanation but two incompatible candidate explanations.** The answer to be given when a contest arrives cannot be either one of these two. The record itself does not come free. Had all six monitored periods been recorded, it would come to **1200 records** and **12,000 stored fields**; **0.8333** times the source readings, that is, a load nearly as large as the raw data the pipeline carries. In return the record gives not only the model but also the **reason** in a single pass: of the 109 predictions that exceed the threshold, **in 57 the subscriber type is decisive on its own**, that is, when the commercial contribution is removed from the score the threshold is not exceeded. This number is read from the `score` and `commercial` fields in the record; without the record, 84,740 rows must be read again for the same number, and which model the result belongs to for 189 predictions is still unknown. #### Summary - The decision record ties a prediction in ten fields: model, weights, decision threshold, three feature values, score and prediction. Rows read per prediction: **1.00**. - The search without a record reads **84,740 rows** for the same two hundred predictions, which comes to **423.70** per prediction, and even so **the model of 189 predictions cannot be determined**; the 11 predictions that can be determined are the ones on which the two models disagree. - Recomputation brings the feature back, **it does not bring back which model was running**. Interpretability works when the model is at hand; the audit trail is **the written record** that says which model it was, and it cannot be produced if it was not recorded. - Because the small share of a canary reduces disagreement, **the choice that shrinks the risk also shrinks retrospective auditability**; two models that give the same output carry different scores and different thresholds, so a prediction that cannot be determined has not one but **two incompatible** explanations. - The cost of the record is **12,000 stored fields** for six periods and is **0.8333** times the source readings. In return the reason is read in a single pass: in **57** of the 109 predictions that exceed the threshold the subscriber type is decisive on its own. The base class ratio is 0.5342 and the step of accuracy is 0.0050. #### Next Step Over four lessons the model was released, compared, retrained, and every prediction it produced was recorded. All of these measures face **inward**: they were kept so that the team could make its own decision. The thing not asked is **what the same setup owes to the outside** — a decision's justification being requested, a prediction being contested, a record being expected to be kept for a certain period. The next lesson counts these expectations by their classes and writes the engineering counterpart of each class: how many additional fields, how many stored records, and which of them are **already met** by the decision record built in this lesson. ### Compliance Requirements Source: https://academia.sh/en/courses/model-monitoring/compliance-requirements Counting how many additional fields and how many stored records each of the four obligation classes owed to the outside asks for: the baseline record keeps three fields per decision, and 36 of the 1200 decisions differ when recomputed from today's source, because the 80 late-arriving rows changed the features of 239 decisions. The record-keeping obligation adds four fields per decision and makes recomputation 1200 of 1200; the explanation obligation adds zero fields, because all five explanation values are derived from those four fields; the right to contest adds three fields only on the 108 negative decisions within one point of the threshold, and those 108 decisions cover 26 of the model's 26 wrong predictions; the retention policy adds no fields at all, it deletes records. With a retention of three periods 600 decision records are dropped, and the number of decisions deleted while the contest window is still open is 600. In four lessons the model was released, measured in the shadow and the canary, compared, and its decisions were recorded. The four share one trait: the addressee of the measure is **internal**. The team that sets the canary share, the minimum detectable effect and the room the decision record takes also decides whether they are enough. This lesson moves the addressee outside. A decision a model produces creates a debt toward its subject, and what that debt is is not left to the team's preference. The lesson takes it as **four obligation classes** — record-keeping, explanation, contest and retention period — and counts one engineering counterpart for each: **how many additional fields per decision and how many stored records.** - **RG41.** The source, the pipeline, the feature path, the model and the decision log are **fictional**. No name of a regulation, standard, institution, country, certificate or audit body is written; no product, service or library is called. Seed 20260218. - **RG42.** Obligations are referred to by their **classes**, and what is measured is not their sources but **what they ask of the system**: record-keeping asks that a decision be recomputable, explanation that the justification of a single decision be shown, the right to contest that the result of the review be written, and the retention period that the record be **deleted after a certain time**. - **RG43.** The baseline is **the setup without obligations**: the decision log keeps only the three fields needed to present, and it is assumed that nothing is owed to the outside. - **RG44.** The model and the label come from the shared definition: weights `(0.6, 0.0, 4)`, decision threshold 18, balanced accuracy 0.9864. Monitoring is six periods, 1200 decisions, **base class ratio 0.5342**, and the smallest step of period accuracy is 0.0050. - **RG45.** The record is tested with a **fictional late-arriving reading**: in the period `2026-04` the meters of 40 subscribers were not read on time, and the rows were added to the source **after** the decisions were made. The source is 2120 rows at decision time and 2200 rows today. - **RG46.** The contest window is **three periods**: a decision can be contested at the latest at the end of the third period. The retention period is also written in periods. Audit trail, ownership and retention policy were measured in the Introduction to Data Engineering course and the Governance, Risk and Compliance course; their procedures are **not repeated**. The previous lesson built the decision record. This lesson takes that record as **data** and asks one question: what does each of the four classes put on top of it. #### Translating an Obligation into a Record The first block sets up the source, the consumption stage of the pipeline, the feature path and the model; then it produces the same decisions twice — once from the source at decision time, once from today's source. ```python # FICTIONAL. The meter readings of the measurement network, the first three stages of the # pipeline, the feature path, the threshold model and the decision log are generated inside # the lesson; no regulation , audit body , product or service is called. Seed 20260218. SEED, M32 = 20260218, 0xFFFFFFFF def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value REGION = ["east", "west", "north", "south", "central"] TYPE = ["residential", "commercial"] PERIODS = [f"2026-{a:02d}" for a in range(1, 13)] SUBSCRIBERS = {} for i in range(200): r = rng(SEED + 37 * i) SUBSCRIBERS[f"A{i + 1:03d}"] = {"region": REGION[int(r() * 5)], "type": TYPE[int(r() * 2)], "start": 10000 + int(r() * 60000)} def generate_source(): records = [] for i, (sub, a) in enumerate(SUBSCRIBERS.items()): e = a["start"] for p in PERIODS: r = rng(SEED + 101 * i + 7 * PERIODS.index(p)) e += 4 + int(r() * 46) records.append({"subscriber": sub, "period": p, "region": a["region"], "type": a["type"], "index": e, "unit": "m3"}) return records def consumption(records): """Consumption from the period difference. If there is no previous period the row cannot be computed.""" table = {(k["subscriber"], k["period"]): k for k in records} out = [] for k in records: i = PERIODS.index(k["period"]) previous = table.get((k["subscriber"], PERIODS[i - 1])) if i else None if previous is not None: out.append(dict(k, m3=int(k["index"]) - int(previous["index"]))) return out def features(records): """Independent of the weights , computed once.""" ordered = sorted(records, key=lambda k: PERIODS.index(k["period"])) sub, reg = {}, {} for k in ordered: sub.setdefault(k["subscriber"], []).append((PERIODS.index(k["period"]), k["m3"])) reg.setdefault(k["region"], []).append((PERIODS.index(k["period"]), k["m3"])) def average(series, i): chosen = [v for d, v in series if d < i] return round(sum(chosen) / len(chosen), 4) if chosen else 0.0 return {(k["subscriber"], k["period"]): { "subscriber_mean": average(sub[k["subscriber"]], PERIODS.index(k["period"])), "region_mean": average(reg[k["region"]], PERIODS.index(k["period"])), "commercial": 1 if k["type"] == "commercial" else 0} for k in records} GRID = [(a, b, c) for a in (0.6, 0.8, 1.0) for b in (0.0, 0.2, 0.4) for c in (0, 4, 8)] DECISION_GRID = (18, 22, 26, 30, 34) MONITORED = PERIODS[6:] RULE_THRESHOLD, RULE_COMMERCIAL = 30, 6 def score(o, w): return w[0] * o["subscriber_mean"] + w[1] * o["region_mean"] + w[2] * o["commercial"] def predict(o, w, decision): return 1 if score(o, w) > decision else 0 def label(o, threshold=RULE_THRESHOLD): return 1 if o["subscriber_mean"] + RULE_COMMERCIAL * o["commercial"] > threshold else 0 def train(records, feats): """The training criterion is balanced accuracy; the decision threshold is A SEPARATE setting and is searched.""" subset = [k for k in records if k["period"] in PERIODS[1:9]] best = (-1.0, None, None) for w in GRID: for decision in DECISION_GRID: tp = tn = fp = fn = 0 for k in subset: o = feats[(k["subscriber"], k["period"])] p, g = predict(o, w, decision), label(o) tp, fp, fn, tn = (tp + (p and g), fp + (p and not g), fn + (not p and g), tn + (not p and not g)) total = (tp / (tp + fn) if tp + fn else 0.0) + (tn / (tn + fp) if tn + fp else 0.0) if total / 2 > best[0]: best = (total / 2, w, decision) return {"weights": best[1], "decision_threshold": best[2], "balanced": round(best[0], 4)} # FICTIONAL late-arriving reading: in 2026-04 the meters of 40 subscribers were not read on time # and the rows were added to the source AFTER the decisions were made. RAW = generate_source() AT_DECISION = consumption([k for k in RAW if not (k["period"] == "2026-04" and k["subscriber"] <= "A040")]) TODAY = consumption(RAW) FEATS_A, FEATS_B = features(AT_DECISION), features(TODAY) M = train(TODAY, FEATS_B) MONITORED_ROWS = [k for k in TODAY if k["period"] in MONITORED] share1 = sum(label(FEATS_B[(k["subscriber"], k["period"])]) for k in MONITORED_ROWS) / len(MONITORED_ROWS) print(f"model: weights {M['weights']} , decision threshold {M['decision_threshold']} ," f" balanced accuracy {M['balanced']}") print(f"monitoring periods {len(MONITORED)} , decisions {len(MONITORED_ROWS)} ," f" base class ratio {round(max(share1, 1 - share1), 4)}") # Baseline record: the least that is needed to present. BASE_FIELDS = ("subscriber", "period", "prediction") # The counterpart of the record-keeping obligation: EVERYTHING the decision READ is stored. S1_FIELDS = ("model_version", "subscriber_mean", "region_mean", "commercial") S1_MODEL_FIELDS = ("weights", "decision_threshold") VERSION = {"s1": {"weights": M["weights"], "decision_threshold": M["decision_threshold"]}} ledger = [] for k in [k for k in AT_DECISION if k["period"] in MONITORED]: o = FEATS_A[(k["subscriber"], k["period"])] ledger.append({"subscriber": k["subscriber"], "period": k["period"], "model_version": "s1", "prediction": predict(o, M["weights"], M["decision_threshold"]), "subscriber_mean": o["subscriber_mean"], "region_mean": o["region_mean"], "commercial": o["commercial"]}) ledger_same = sum(1 for r in ledger if predict(r, VERSION[r["model_version"]]["weights"], VERSION[r["model_version"]]["decision_threshold"]) == r["prediction"]) feature_diff = sum(1 for r in ledger if FEATS_B[(r["subscriber"], r["period"])]["subscriber_mean"] != r["subscriber_mean"]) decision_diff = sum(1 for r in ledger if predict(FEATS_B[(r["subscriber"], r["period"])], M["weights"], M["decision_threshold"]) != r["prediction"]) print(f"\nsource rows: at decision time {len(AT_DECISION)} , today {len(TODAY)}") print(f"recomputation from the ledger : exact {ledger_same}/{len(ledger)} , differing" f" {len(ledger) - ledger_same}") print(f"recomputation from today's source: exact {len(ledger) - decision_diff}/{len(ledger)} ," f" differing {decision_diff} , feature values changed {feature_diff}") ``` ``` model: weights (0.6, 0.0, 4) , decision threshold 18 , balanced accuracy 0.9864 monitoring periods 6 , decisions 1200 , base class ratio 0.5342 source rows: at decision time 2120 , today 2200 recomputation from the ledger : exact 1200/1200 , differing 0 recomputation from today's source: exact 1164/1200 , differing 36 , feature values changed 239 ``` The last two lines show why the record-keeping obligation is a question of **fields**. When the decision log carries the value of the features at the time of the decision, 1200 of 1200 decisions are recomputed exactly. When the same decisions are to be reproduced from today's source, the features of 239 decisions have changed and **36 decisions come out different**. Nobody touched the source; only the 80 late rows have settled into place. This is a measurement topic before it is a compliance topic. When the justification of a decision is asked, "we look at the source and recompute" is no answer unless the **state of the source at decision time** has been stored. The late-arriving reading is one of the six degradation classes of the pipeline and was measured as detection lag in the Introduction to Data Engineering course; what is new here is the **retrospective** face of the same degradation — the pipeline has recovered, the record has been corrupted. #### The Field Cost of the Four Classes The second block counts the four classes in a single table and sweeps the retention period. ```python # This block continues the PERIODS , MONITORED , FEATS_B , M , VERSION , ledger , score , # predict and label definitions of the first block. MARGIN, CONTEST_WINDOW, RETENTION = 1.0, 3, 3 def explanation(r): """DERIVED from the fields of class 1; asks for no new field.""" w = VERSION[r["model_version"]]["weights"] e = VERSION[r["model_version"]]["decision_threshold"] p = score(r, w) return {"subscriber_contribution": round(w[0] * r["subscriber_mean"], 4), "region_contribution": round(w[1] * r["region_mean"], 4), "type_contribution": round(w[2] * r["commercial"], 4), "score": round(p, 4), "distance_to_threshold": round(p - e, 4)} contested = [r for r in ledger if r["prediction"] == 1 and abs(explanation(r)["distance_to_threshold"]) <= MARGIN] example = contested[0] print(f"example of a negative decision closest to the threshold: {example['subscriber']} {example['period']}" f" prediction {example['prediction']} actual {label(example)}") print(f"explanation {explanation(example)}") wrong_near = sum(1 for r in contested if label(r) != r["prediction"]) wrong_all = sum(1 for r in ledger if label(r) != r["prediction"]) n, nb, ns = len(ledger), len(BASE_FIELDS), len(S1_FIELDS) s1 = n * (nb + ns) + len(S1_MODEL_FIELDS) print(f"\nclass fields per decision stored values recomputation") print(f"baseline (obligation-free) {nb:>19} {n * nb:>14} {n - decision_diff:>9}/{n}") print(f"record-keeping {nb + ns:>19} {s1:>14} {n:>9}/{n}") print(f"explanation {nb + ns:>19} {s1:>14} {n:>9}/{n}") print(f"contest {str(nb + ns) + ' (+3)':>19} {s1 + 3 * len(contested):>14}" f" {n:>9}/{n}") kept = len([d for d in MONITORED if PERIODS.index(d) > len(PERIODS) - 1 - RETENTION]) * 200 print(f"retention ({RETENTION} periods) {nb + ns:>19} {kept * (nb + ns) + 2:>14} {kept:>9}/{n}") print(f"\nnegative decisions within {MARGIN} point of the threshold: {len(contested)}" f" (share {round(len(contested) / n, 4)}) ; of these wrong {wrong_near} ," f" wrong in the whole log {wrong_all}") print(f"monitoring accuracy {round((n - wrong_all) / n, 4)} , step {round(1 / 200, 4)}") print(f"\nretention (periods) records kept records deleted deleted while contest open") for R in (1, 2, 3, 6, 12): kept_periods = [d for d in MONITORED if PERIODS.index(d) > len(PERIODS) - 1 - R] broken = 0 for t in range(len(PERIODS)): for d in MONITORED: i = PERIODS.index(d) if i <= t - R and i + CONTEST_WINDOW >= t: broken += 200 print(f"{R:>19} {len(kept_periods) * 200:>13} {n - len(kept_periods) * 200:>15} {broken:>26}") ``` ``` example of a negative decision closest to the threshold: A002 2026-08 prediction 1 actual 1 explanation {'subscriber_contribution': 18.6, 'region_contribution': 0.0, 'type_contribution': 0, 'score': 18.6, 'distance_to_threshold': 0.6} class fields per decision stored values recomputation baseline (obligation-free) 3 3600 1164/1200 record-keeping 7 8402 1200/1200 explanation 7 8402 1200/1200 contest 7 (+3) 8726 1200/1200 retention (3 periods) 7 4202 600/1200 negative decisions within 1.0 point of the threshold: 108 (share 0.09) ; of these wrong 26 , wrong in the whole log 26 monitoring accuracy 0.9783 , step 0.005 retention (periods) records kept records deleted deleted while contest open 1 200 1000 2400 2 400 800 1400 3 600 600 600 6 1200 0 0 12 1200 0 0 ``` The table shows that the four classes are **not equal**. Only **record-keeping** really asks for new fields per decision: four fields, from 3600 stored values to 8402. **The explanation obligation adds zero fields.** The example row shows this on one decision: for subscriber `A002`'s `2026-08` decision the three contributions, the score and the distance to the threshold are written, and all five are derived from the fields of the record-keeping class. As long as the weights and the threshold are stored, the explanation is a **computation**, not a record. The **right to contest** is met in part: record-keeping has already paid for recomputing the decision, what is new is only the result of the review, and that is written not for all decisions but for **108** of them. The number 108 is the sharpest reading of the lesson. The negative decisions within one point of the threshold make up 0.0900 of the 1200 decisions, but **26 of the model's 26 wrong predictions** are inside this band. With a monitoring accuracy of 0.9783 and a period step of 0.0050, 26 wrong predictions are a measurable size. Because recall is 1.0000, all of the wrong predictions are **false positives**: subscribers who fell immediately above the decision threshold and are in reality low. **The engineering counterpart of the right to contest is to open nine percent of the decisions to review and in return be able to reach all of the wrong ones** — nearness to the boundary is not a coincidence, it is a structural property of every model with a threshold. #### Retention Cuts Across the Other Three The counterpart of the fourth class is unlike the other three, because it **adds no field**: the period is already recorded and the duration is one policy number. What the retention policy does is **delete records**, and the records it deletes are the ones the first three classes rely on. With a retention of three periods the log falls from 1200 decisions to 600, 8402 stored values drop to 4202 and **the justification of 600 decisions can no longer be asked**. The last lesson of the Introduction to Data Engineering course measured this on the data side — the retention policy leaves periods that cannot be audited retrospectively; on the decision side the same thing happens by the same mechanism. The last column of the sweep turns the collision into a number. With a contest window and a retention period of three periods each, one period's set of decisions is deleted **while the right to contest is still open** in every period: 600 decisions in total. At two periods it becomes 1400, at one 2400; at six it drops to zero. This number is not an implementation error, it is **two obligation classes cutting each other**: one asks for the record, the other deletes it, and both are in force in the same system at the same time. The measure of the trade-off is exactly this column — when the retention period is not **longer than the contest window**, the number of unmet obligations is greater than zero; equality is not enough. The rule that follows is independent of the source of the classes: **an obligation class counts as unmeasured unless its counterpart is written into a field in the decision record or into a record count.** #### Summary - When the decision log holds the features at decision time, 1200 of 1200 decisions are recomputed exactly; when computed from today's source the features of 239 decisions change and **36 decisions come out different**. Nobody touched the source; the 80 late-arriving rows have settled into place. - **Record-keeping** adds four fields per decision and raises the stored values from 3600 to 8402. It is the only one among the four classes that really asks for new fields. - **Explanation** adds zero fields: the three contributions, the score and the distance to the threshold are derived from the fields of record-keeping. As long as the weights and the threshold are stored, the explanation is a computation. - **Contest** adds three fields only on the 108 negative decisions within one point of the threshold; those 108 decisions are 0.0900 of 1200 and cover **all 26 of the model's 26 wrong predictions**. - **Retention** adds no fields, it deletes records: with three periods the log falls to 600 and the number of decisions deleted while the contest window is still open is 600. Two obligation classes cut each other in the same system. #### Next Step The counterparts of the obligations were written as fields and records; all of them either added something to what is stored or took something away from it. But no line of this lesson asked what keeping the model running itself costs. A model does two separate jobs: it is **retrained** period by period and it produces **inference** in every period. The two are written to the same budget and read as a single total. The next lesson splits that total in two and counts which item grows as the retraining frequency increases and which decision reading it as a single number hides. ### Cost Management Source: https://academia.sh/en/courses/model-monitoring/cost-management Tracking training and inference separately and counting what a single total hides: the unit of work is one score evaluation, the inference of six monitoring periods costs 1200 score evaluations and stays constant whatever the frequency, while one retraining costs 162,000. When the retraining frequency goes from once in six periods to every period, the training item rises from 162,000 to 972,000, the training share rises from 0.9926 to 0.9988, and the 2026-12 accuracy stays at 0.9800 in all five setups; because the accuracy gained is 0.0000, the cost per gain is undefined. The cost per prediction read from the total is 811.00, read from inference it is 1.00: making inference completely free removes 0.0012 of the total, moving the frequency from one to six removes 0.8323 of it, and 0.0000 accuracy is paid in return. The crossover point lies between 27,000 and 162,000 requests per period. The previous lesson took the debt owed to the outside and wrote it as records: four obligation classes, seven fields per decision, 8402 stored values. Everything counted was written to **storage**. The running of the model itself, the work repeated every period, entered no line. That work is not one piece. A production model does two separate jobs: it is **retrained** period by period and it produces **inference** every period. The two are usually written to one budget item and read as a single total. This lesson splits that total in two, sweeps the retraining frequency and counts which decision the single number makes invisible. - **RG47.** The source, the pipeline, the feature path, the model and the cost counter are **fictional**. No cloud service, accelerator, product or price is called; no currency is written. Seed 20260218. - **RG48.** The unit of work is **one score evaluation**: multiplying a row's features by the weights and summing. Training and inference do only this, so both enter the same table **with the same unit**. The counter is inside the `score` function; cost is not assumed, it is **measured**. - **RG49.** No time is written. Cost is counted in **score evaluations**, delay in **periods**. - **RG50.** The baseline is **the setup without retraining**: the first model is trained once and never changes over the six monitoring periods. - **RG51.** In production the input drifts: after `2026-09` commercial consumption grows by 0.06 per period. The model is `(0.6, 0.0, 4)`, the decision threshold 18, the candidates 135, the training rows 1600. Monitoring is six periods, the **base class ratio 0.5342**, the step of period accuracy 0.0050. - **RG52.** The inference volume is 200 requests per period and is swept as a **free variable** in the last section. The cost per prediction was measured in the Building Applications with Model APIs course of the AI Engineering curriculum and in the scaling lesson of the MLOps Fundamentals course; their procedures are **not repeated**, and the measure here adds a single thing on top of them: **training and inference are counted separately.** #### Counting the Two Items Separately The first block sets up the source, the feature path and the model, places a counter in the `score` function and sweeps the retraining frequency. ```python # FICTIONAL. The source , the consumption stage of the pipeline , the feature path , the # threshold model and the cost counter are generated inside the lesson; no cloud service , # accelerator , product or price is called. Seed 20260218. SEED, M32 = 20260218, 0xFFFFFFFF def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value REGION = ["east", "west", "north", "south", "central"] TYPE = ["residential", "commercial"] PERIODS = [f"2026-{a:02d}" for a in range(1, 13)] SUBSCRIBERS = {} for i in range(200): r = rng(SEED + 37 * i) SUBSCRIBERS[f"A{i + 1:03d}"] = {"region": REGION[int(r() * 5)], "type": TYPE[int(r() * 2)], "start": 10000 + int(r() * 60000)} def generate_source(): records = [] for i, (sub, a) in enumerate(SUBSCRIBERS.items()): e = a["start"] for p in PERIODS: r = rng(SEED + 101 * i + 7 * PERIODS.index(p)) e += 4 + int(r() * 46) records.append({"subscriber": sub, "period": p, "region": a["region"], "type": a["type"], "index": e, "unit": "m3"}) return records def consumption(records): table = {(k["subscriber"], k["period"]): k for k in records} out = [] for k in records: i = PERIODS.index(k["period"]) previous = table.get((k["subscriber"], PERIODS[i - 1])) if i else None if previous is not None: out.append(dict(k, m3=int(k["index"]) - int(previous["index"]))) return out def features(records): """Independent of the weights , computed once.""" ordered = sorted(records, key=lambda k: PERIODS.index(k["period"])) sub, reg = {}, {} for k in ordered: sub.setdefault(k["subscriber"], []).append((PERIODS.index(k["period"]), k["m3"])) reg.setdefault(k["region"], []).append((PERIODS.index(k["period"]), k["m3"])) def average(series, i): chosen = [v for d, v in series if d < i] return round(sum(chosen) / len(chosen), 4) if chosen else 0.0 return {(k["subscriber"], k["period"]): { "subscriber_mean": average(sub[k["subscriber"]], PERIODS.index(k["period"])), "region_mean": average(reg[k["region"]], PERIODS.index(k["period"])), "commercial": 1 if k["type"] == "commercial" else 0} for k in records} GRID = [(a, b, c) for a in (0.6, 0.8, 1.0) for b in (0.0, 0.2, 0.4) for c in (0, 4, 8)] DECISION_GRID = (18, 22, 26, 30, 34) MONITORED = PERIODS[6:] RULE_THRESHOLD, RULE_COMMERCIAL = 30, 6 COUNTER = {"score": 0} # the single unit of work: one score evaluation def score(o, w): COUNTER["score"] += 1 return w[0] * o["subscriber_mean"] + w[1] * o["region_mean"] + w[2] * o["commercial"] def predict(o, w, decision): return 1 if score(o, w) > decision else 0 def label(o, threshold=RULE_THRESHOLD): return 1 if o["subscriber_mean"] + RULE_COMMERCIAL * o["commercial"] > threshold else 0 def train(records, feats, periods): """The training criterion is balanced accuracy; the decision threshold is A SEPARATE setting and is searched.""" subset = [k for k in records if k["period"] in periods] best = (-1.0, None, None) for w in GRID: for decision in DECISION_GRID: tp = tn = fp = fn = 0 for k in subset: o = feats[(k["subscriber"], k["period"])] p, g = predict(o, w, decision), label(o) tp, fp, fn, tn = (tp + (p and g), fp + (p and not g), fn + (not p and g), tn + (not p and not g)) total = (tp / (tp + fn) if tp + fn else 0.0) + (tn / (tn + fp) if tn + fp else 0.0) if total / 2 > best[0]: best = (total / 2, w, decision) return {"weights": best[1], "decision_threshold": best[2], "balanced": round(best[0], 4), "rows": len(subset)} def drift(records, start="2026-09", slope=0.06): """FICTIONAL data drift: commercial consumption grows by `slope` every period.""" b = PERIODS.index(start) return [dict(k, m3=round(k["m3"] * (1.0 + slope * (PERIODS.index(k["period"]) - b + 1)), 4)) if PERIODS.index(k["period"]) >= b and k["type"] == "commercial" else dict(k) for k in records] def accuracy(records, m, feats, period): subset = [k for k in records if k["period"] == period] d = sum(1 for k in subset if predict(feats[(k["subscriber"], k["period"])], m["weights"], m["decision_threshold"]) == label(feats[(k["subscriber"], k["period"])])) return round(d / len(subset), 4) CLEAN = consumption(generate_source()) FEATS_C = features(CLEAN) monitored_c = [k for k in CLEAN if k["period"] in MONITORED] p1 = sum(label(FEATS_C[(k["subscriber"], k["period"])]) for k in monitored_c) / len(monitored_c) DRIFTED = drift(CLEAN) # the input drifts in production FEATS = features(DRIFTED) COUNTER["score"] = 0 FIRST = train(DRIFTED, FEATS, PERIODS[1:9]) FIRST_COST = COUNTER["score"] print(f"base class ratio {round(max(p1, 1 - p1), 4)} , monitoring periods {len(MONITORED)} ," f" requests per period 200") print(f"first training: weights {FIRST['weights']} , decision threshold {FIRST['decision_threshold']} ," f" rows {FIRST['rows']} , candidates {len(GRID) * len(DECISION_GRID)}") print(f"measured training cost (score evaluations): {FIRST_COST}") COUNTER["score"] = 0 last = accuracy(DRIFTED, FIRST, FEATS, "2026-12") print(f"inference cost of one period (score evaluations): {COUNTER['score']} ," f" 2026-12 accuracy {last}") WINDOW = 6 print("\nfrequency retrainings training cost inference cost total training share" " 2026-12 accuracy") RESULT = [] for f in (0, 6, 3, 2, 1): COUNTER["score"] = 0 m = FIRST if f: for i, d in enumerate(MONITORED): if (i + 1) % f == 0: j = PERIODS.index(d) m = train(DRIFTED, FEATS, [x for x in PERIODS if j - WINDOW < PERIODS.index(x) <= j]) training = COUNTER["score"] COUNTER["score"] = 0 acc = accuracy(DRIFTED, m, FEATS, "2026-12") inference = len(MONITORED) * 200 total = training + inference RESULT.append((f, training, inference, total, acc)) print(f"{f if f else '-':>9} {len(MONITORED) // f if f else 0:>11} {training:>13}" f" {inference:>14} {total:>6} {round(training / total, 4):>14} {acc:>16}") ``` ``` base class ratio 0.5342 , monitoring periods 6 , requests per period 200 first training: weights (0.6, 0.0, 4) , decision threshold 18 , rows 1600 , candidates 135 measured training cost (score evaluations): 216000 inference cost of one period (score evaluations): 200 , 2026-12 accuracy 0.98 frequency retrainings training cost inference cost total training share 2026-12 accuracy - 0 0 1200 1200 0.0 0.98 6 1 162000 1200 163200 0.9926 0.98 3 2 324000 1200 325200 0.9963 0.98 2 3 486000 1200 487200 0.9975 0.98 1 6 972000 1200 973200 0.9988 0.98 ``` The scale difference between the two items comes directly from the counter. One period of inference is **200 score evaluations** and comes to 1200 over six periods; the first training scans 1600 rows with 135 candidates and costs **216,000**. A single retraining with a six-period window is **162,000**, that is **135 times** the inference of the whole monitoring horizon. Down the frequency column the inference item **never changes**; only the training item does, and it rises from 162,000 to **972,000** when going from once in six periods to every period. The last column writes what this increase buys. The `2026-12` accuracy is **0.9800** in all five setups — **exactly the same** as that of the model that is never retrained. This is the cost-side counterpart of the second claim of the course: the input drifted, the alert fired, six retrainings were run and **the accuracy did not move by a single step**. Since the period step is 0.0050, the 0.0000 is truly zero, not a small gain too small to measure. As long as the rule does not change, a correct model stays correct on a drifted input too, and retraining brings only the **cost**. #### What the Single Total Hides The second block reads the same results per prediction and sweeps the inference volume. ```python # This block continues the RESULT , MONITORED , FIRST_COST and WINDOW definitions of the first block. baseline_accuracy = RESULT[0][4] n_predictions = len(MONITORED) * 200 print("frequency per prediction from total per prediction from inference" " accuracy gained cost per gain") for f, training, inference, total, acc in RESULT: gain = round(acc - baseline_accuracy, 4) per_gain = "undefined" if gain <= 0 else str(round(training / gain)) print(f"{f if f else '-':>9} {round(total / n_predictions, 2):>25}" f" {round(inference / n_predictions, 2):>29} {gain:>15} {per_gain:>13}") print(f"\nshare removed from the total by making inference completely free:" f" {round(RESULT[-1][2] / RESULT[-1][3], 4)}") print(f"share removed from the total by moving from frequency 1 to frequency 6:" f" {round((RESULT[-1][3] - RESULT[1][3]) / RESULT[-1][3], 4)}" f" , accuracy paid {round(RESULT[1][4] - RESULT[-1][4], 4)}") print("\nrequests per period inference (6 periods) training frequency 6 training frequency 1" " which inference exceeded") for requests in (200, 2000, 20000, 50000, 200000): c = len(MONITORED) * requests passed = [str(f) for f, e, _, _, _ in RESULT[1:] if c > e] print(f"{requests:>19} {c:>21} {RESULT[1][1]:>20} {RESULT[-1][1]:>20}" f" {(' , '.join(passed) if passed else 'none'):>24}") print(f"\ncrossover point (requests per period): frequency 6 -> {RESULT[1][1] // len(MONITORED)}" f" , frequency 1 -> {RESULT[-1][1] // len(MONITORED)}") # Monitoring's own item: three indicators are computed every period. indicators = len(MONITORED) * 200 # prediction share , 0 if the inference predictions are reused print(f"monitoring item (score evaluations): at most {indicators} , when inference predictions" f" are reused 0 ; one retraining {RESULT[1][1]}") ``` ``` frequency per prediction from total per prediction from inference accuracy gained cost per gain - 1.0 1.0 0.0 undefined 6 136.0 1.0 0.0 undefined 3 271.0 1.0 0.0 undefined 2 406.0 1.0 0.0 undefined 1 811.0 1.0 0.0 undefined share removed from the total by making inference completely free: 0.0012 share removed from the total by moving from frequency 1 to frequency 6: 0.8323 , accuracy paid 0.0 requests per period inference (6 periods) training frequency 6 training frequency 1 which inference exceeded 200 1200 162000 972000 none 2000 12000 162000 972000 none 20000 120000 162000 972000 none 50000 300000 162000 972000 6 200000 1200000 162000 972000 6 , 3 , 2 , 1 crossover point (requests per period): frequency 6 -> 27000 , frequency 1 -> 162000 monitoring item (score evaluations): at most 1200 , when inference predictions are reused 0 ; one retraining 162000 ``` The first two columns answer the question of the lesson. When the total is read as one number and divided by the number of predictions, the cost per prediction is **811.00**; the same computation from the inference item alone gives **1.00**. The entire 810-fold difference is training and **belongs to no prediction**. A reader who sees only this number concludes that the prediction is expensive and turns to making inference cheaper: batching requests, a cache, shrinking the model. **The ceiling of that road is 0.0012** — even if inference were completely free, only 1.2 per thousand of the total would be removed. The decision that stays invisible lies the other way. Moving the frequency from every period to once in six periods removes **0.8323** of the total and costs **0.0000 accuracy**. The single total hides this decision, because in the total nothing separates training from inference; the training share lies between 0.9926 and 0.9988 and **the frequency at which one stands cannot be read from the total**. When the two items are counted separately the decision is a single line. The cost per gain column says the same thing once more: **undefined** in all five setups, because the divisor is zero. **The justification of a cost item can only be written together with a gain; spending without a shown gain counts as unmeasured.** The last sweep shows that the decision depends on volume. At 200 requests per period inference exceeds no training item; at 50,000 requests it exceeds only the sparsest frequency; at 200,000 requests it exceeds all four at once. The **crossover point** is **27,000** requests per period for frequency 6 and **162,000** for frequency 1. This means a conclusion such as "training always dominates" cannot be written: the dominant item depends on which side of the volume one stands, and **the single total does not show that side either**. The item of monitoring itself is the smallest row of this table. Computing three indicators over six periods is at most 1200 score evaluations, and it falls to zero when the predictions produced in inference are reused. But the work monitoring **triggers** is not small: the retraining started by a single alert that fires is 162,000. **The cost of monitoring is not in the indicator itself but in the response given to the indicator.** #### Summary - The unit of work is a **score evaluation** and the counter is inside the `score` function; training and inference enter the same table with the same unit. One period of inference is 200, the first training 216,000, a retraining with a six-period window **162,000** score evaluations. - In the frequency sweep the inference item **stays constant at 1200**, the training item rises from 162,000 to **972,000** and the training share rises from 0.9926 to 0.9988. - The `2026-12` accuracy is **0.9800** in all five setups. Since the period step is 0.0050, the accuracy gained is truly **0.0000** and the cost per gain is **undefined**. - The cost per prediction read from the total is **811.00**, read from inference **1.00**. Making inference free removes 0.0012 of the total, thinning out the frequency removes **0.8323** of it; the second costs 0.0000 accuracy and is **invisible in the single total**. - The crossover point lies between **27,000** and **162,000** requests per period; which item dominates depends on volume. The monitoring item is at most 1200; the retraining it triggers is 162,000. #### Next Step The two items were separated and a counterpart was written for both: retraining brings 0.0000 accuracy, while inference produces the whole value of the model for 200 requests per period. The only column missing from this table is the column of **the end**. Retraining can be thinned out, inference can be made cheaper, records can be shortened; but when a model will **end** is written nowhere. The next lesson builds the measure of the decision to retire a model and counts what removing the model leaves behind retrospectively: how many periods of decisions become **unexplainable** once it is removed. ### Model Retirement Source: https://academia.sh/en/courses/model-monitoring/model-retirement Counting by which measure the retirement decision is made and what removing the model leaves behind retrospectively: the indicator criterion fires at 2026-10 in both scenarios, and in one the retirement is unjustified because accuracy went from 0.9700 to 0.9800; the accuracy criterion never fires on the harmless drift and shows the 2026-09 degradation at 2026-11 on the harmful drift, two periods late; the challenger comparison criterion measures an advantage of 0.0000 on data drift and 0.0900 on concept drift and is the only criterion that separates the two scenarios. After removal without a decision record, six of six periods are unexplainable; when the model version record is deleted while the decision record exists, 1200 predictions are read but the ones whose justification is computed are 0. The lesson is the closing of the course and of the curriculum. The table of the previous lesson was missing the **end** column: when a model would end was written nowhere. This lesson asks two questions: which number justifies the removal, and what the removal leaves behind **retrospectively**. - **RG53.** The source, the pipeline, the model and the two drifts are **fictional**; no product, service or institution is called. Seed 20260218. - **RG54.** The model is `(0.6, 0.0, 4)`, the decision threshold is 18, balanced accuracy is 0.9864; monitoring runs six periods, the **base class ratio is 0.5342**, the step is 0.0050, the label is delayed by **two periods**. - **RG55.** In **data drift** the input moves and the rule stays the same; in **concept drift** the input stays the same and the rule changes (the rule threshold from 30 to 27). - **RG56.** Three criteria are tested: the **indicator** (the prediction share deviates from the baseline by 0.05), the **accuracy threshold** (the period accuracy falls below 0.95), the **challenger comparison** (the advantage is greater than zero). - **RG57.** Removal means the weight and threshold record becomes **unresolvable**; a period whose justification cannot be computed counts as **unexplainable**. The retirement is `2026-11`, the contest and the retention are three periods each. #### The Measure of the Retirement Decision The first block builds the model and the two drifts and fires the three criteria. ```python # FICTIONAL. The source , the consumption stage of the pipeline , the feature path , the threshold # model , the two drifts and the retirement criteria are generated inside the lesson; no product , # service or institution is called. Seed 20260218. SEED, M32 = 20260218, 0xFFFFFFFF def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value REGION, TYPE = ["east", "west", "north", "south", "central"], ["residential", "commercial"] PERIODS = [f"2026-{a:02d}" for a in range(1, 13)] SUBSCRIBERS = {} for i in range(200): r = rng(SEED + 37 * i) SUBSCRIBERS[f"A{i + 1:03d}"] = {"region": REGION[int(r() * 5)], "type": TYPE[int(r() * 2)], "start": 10000 + int(r() * 60000)} def rows(): raw = [] for i, (sub, a) in enumerate(SUBSCRIBERS.items()): e = a["start"] for p in PERIODS: r = rng(SEED + 101 * i + 7 * PERIODS.index(p)) e += 4 + int(r() * 46) raw.append({"subscriber": sub, "period": p, "region": a["region"], "type": a["type"], "index": e}) t = {(k["subscriber"], k["period"]): k for k in raw} return [dict(k, m3=k["index"] - t[(k["subscriber"], PERIODS[PERIODS.index(k["period"]) - 1])] ["index"]) for k in raw if PERIODS.index(k["period"])] def features(records): """Independent of the weights , computed once.""" sub, reg = {}, {} for k in sorted(records, key=lambda k: PERIODS.index(k["period"])): sub.setdefault(k["subscriber"], []).append((PERIODS.index(k["period"]), k["m3"])) reg.setdefault(k["region"], []).append((PERIODS.index(k["period"]), k["m3"])) def mean(series, i): s = [v for d, v in series if d < i] return round(sum(s) / len(s), 4) if s else 0.0 return {(k["subscriber"], k["period"]): { "subscriber_mean": mean(sub[k["subscriber"]], PERIODS.index(k["period"])), "region_mean": mean(reg[k["region"]], PERIODS.index(k["period"])), "commercial": 1 if k["type"] == "commercial" else 0} for k in records} GRID = [(a, b, c) for a in (0.6, 0.8, 1.0) for b in (0.0, 0.2, 0.4) for c in (0, 4, 8)] DECISION_GRID = (18, 22, 26, 30, 34) MONITORED, DELAY = PERIODS[6:], 2 RULE_THRESHOLD, RULE_COMMERCIAL, ACCURACY_THRESHOLD = 30, 6, 0.95 def score(o, w): return w[0] * o["subscriber_mean"] + w[1] * o["region_mean"] + w[2] * o["commercial"] def predict(o, w, decision): return 1 if score(o, w) > decision else 0 def label(o, threshold=RULE_THRESHOLD): return 1 if o["subscriber_mean"] + RULE_COMMERCIAL * o["commercial"] > threshold else 0 def train(records, feats, periods, labels=None): """The criterion is balanced accuracy; the decision threshold is A SEPARATE setting and is searched.""" subset = [k for k in records if k["period"] in periods] best = (-1.0, None, None) for w in GRID: for decision in DECISION_GRID: tp = tn = fp = fn = 0 for k in subset: c = (k["subscriber"], k["period"]) p = predict(feats[c], w, decision) g = labels[c] if labels else label(feats[c]) tp, fp, fn, tn = (tp + (p and g), fp + (p and not g), fn + (not p and g), tn + (not p and not g)) d = (tp / (tp + fn) if tp + fn else 0.0) + (tn / (tn + fp) if tn + fp else 0.0) if d / 2 > best[0]: best = (d / 2, w, decision) return {"weights": best[1], "decision_threshold": best[2], "balanced": round(best[0], 4)} def measure(records, m, feats, period, labels=None): subset = [(k["subscriber"], k["period"]) for k in records if k["period"] == period] p = [predict(feats[c], m["weights"], m["decision_threshold"]) for c in subset] g = [labels[c] if labels else label(feats[c]) for c in subset] return (round(sum(p) / len(p), 4), round(sum(1 for a, b in zip(p, g) if a == b) / len(p), 4)) CLEAN = rows() FEATS = features(CLEAN) M = train(CLEAN, FEATS, PERIODS[1:9]) # FICTIONAL data drift: the input drifts , the rule stays the same. b = PERIODS.index("2026-09") DRIFTED = [dict(k, m3=round(k["m3"] * (1 + 0.06 * (PERIODS.index(k["period"]) - b + 1)), 4)) if PERIODS.index(k["period"]) >= b and k["type"] == "commercial" else dict(k) for k in CLEAN] FEATS_D = features(DRIFTED) # FICTIONAL concept drift: the input stays THE SAME , the rule changes (the threshold goes down from 30 to 27). CONCEPT = {(k["subscriber"], k["period"]): label( FEATS[(k["subscriber"], k["period"])], 27 if PERIODS.index(k["period"]) >= b else RULE_THRESHOLD) for k in CLEAN} p1 = sum(label(FEATS[(k["subscriber"], k["period"])]) for k in CLEAN if k["period"] in MONITORED) / 1200 print(f"model {M['weights']} decision threshold {M['decision_threshold']} balanced {M['balanced']} ," f" base class ratio {round(max(p1, 1 - p1), 4)} , step {round(1 / 200, 4)}") print("\nperiod data: share accuracy | concept: share accuracy") SCENARIO = {"data": (DRIFTED, FEATS_D, None), "concept": (CLEAN, FEATS, CONCEPT)} MEASURED = {name: [(d,) + measure(k, M, o, d, e) for d in MONITORED] for name, (k, o, e) in SCENARIO.items()} for i, d in enumerate(MONITORED): print(f"{d} {MEASURED['data'][i][1]:>11} {MEASURED['data'][i][2]:>8} |" f" {MEASURED['concept'][i][1]:>14} {MEASURED['concept'][i][2]:>8}") print(f"\n{'criterion':<40}{'scenario':>8} {'fired':>7} {'drop':>7}" f" {'lag':>3} {'justified':>9}") for name, (rec, feats, lab) in SCENARIO.items(): baseline = MEASURED[name][1][1] indicator = [d for d, p, _ in MEASURED[name] if abs(p - baseline) / baseline > 0.05] drops = [d for d, _, i in MEASURED[name] if i < ACCURACY_THRESHOLD] seen = PERIODS[PERIODS.index(drops[0]) + DELAY] if drops else "-" challenger = train(rec, feats, MONITORED, lab) advantage = round(measure(rec, challenger, feats, "2026-12", lab)[1] - MEASURED[name][-1][2], 4) for criterion, fired, lag in (("indicator (share 0.05)", indicator[0] if indicator else "-", "-"), (f"accuracy threshold {ACCURACY_THRESHOLD}", seen, DELAY if drops else "-"), (f"challenger comparison (advantage {advantage})", seen if advantage > 0 else "-", DELAY if advantage > 0 else "-")): ok = "no" if (fired != "-" and not drops) else ("yes" if fired != "-" else "-") print(f"{criterion:<40}{name:>8} {str(fired):>7} {str(drops[0]) if drops else '-':>7}" f" {str(lag):>3} {ok:>9}") ``` ``` model (0.6, 0.0, 4) decision threshold 18 balanced 0.9864 , base class ratio 0.5342 , step 0.005 period data: share accuracy | concept: share accuracy 2026-07 0.455 0.97 | 0.455 0.97 2026-08 0.47 0.97 | 0.47 0.97 2026-09 0.455 0.99 | 0.455 0.815 2026-10 0.515 0.975 | 0.51 0.87 2026-11 0.52 0.98 | 0.51 0.865 2026-12 0.545 0.98 | 0.515 0.85 criterion scenario fired drop lag justified indicator (share 0.05) data 2026-10 - - no accuracy threshold 0.95 data - - - - challenger comparison (advantage 0.0) data - - - - indicator (share 0.05) concept 2026-10 2026-09 - yes accuracy threshold 0.95 concept 2026-11 2026-09 2 yes challenger comparison (advantage 0.09) concept 2026-11 2026-09 2 yes ``` The indicator criterion fires **at `2026-10` in both scenarios**: in one, accuracy has gone from 0.9700 to 0.9800 and the retirement is **unjustified**, in the other the degradation is real. The firing period is the same, the right decision is opposite. **An indicator whose firing and whose degradation are not shown separately cannot be the justification for removing a model.** The accuracy criterion is right in both: the degradation happens at `2026-09`, the criterion triggers at `2026-11` — the **two periods** in between are the delayed label itself. The challenger comparison measures the advantage as **0.0000** and **0.0900**, eighteen steps of the resolution. **The two criteria that separate also read labels** and arrive late; the only criterion with no delay is the one that cannot tell the difference. #### What Removal Leaves Behind Retrospectively The second block removes the model and counts what remains. ```python # This block continues the PERIODS , MONITORED , FEATS , M , CLEAN , score and predict # definitions of the first block. RETIREMENT, RETENTION, CONTEST = "2026-11", 3, 3 ledger = [dict(FEATS[(k["subscriber"], k["period"])], period=k["period"], model_version="s1", prediction=predict(FEATS[(k["subscriber"], k["period"])], M["weights"], M["decision_threshold"])) for k in CLEAN if k["period"] in MONITORED] FULL = {"s1": {"weights": M["weights"], "decision_threshold": M["decision_threshold"]}} remaining = [d for d in MONITORED if PERIODS.index(d) > len(PERIODS) - 1 - RETENTION] def justification(r, version): """The justification is computed only if the model version can be resolved.""" s = version.get(r["model_version"]) return None if s is None else round(score(r, s["weights"]) - s["decision_threshold"], 4) print("setup predictions read justification computed" " unexplainable periods") for name, rec, ver in (("no record , version deleted", [], {}), ("decision record , version deleted", ledger, {}), ("decision record + version record", ledger, FULL), (f"same , retention {RETENTION} periods", [r for r in ledger if r["period"] in remaining], FULL)): g = [r for r in rec if justification(r, ver) is not None] print(f"{name:<37}{len(rec):>16} {len(g):>22}" f" {len(MONITORED) - len(set(r['period'] for r in g)):>21}") e = PERIODS.index(RETIREMENT) shown = [d for d in MONITORED if PERIODS.index(d) <= e] open_ = [d for d in shown if PERIODS.index(d) + CONTEST >= e] late = [d for d in shown if PERIODS.index(d) + DELAY > e] print(f"\nretirement {RETIREMENT}: presented periods {len(shown)} , decisions {len(shown) * 200} ;" f" contest still open {len(open_)} periods / {len(open_) * 200} decisions ;" f" label never arriving {len(late)} periods / {len(late) * 200} decisions") ``` ``` setup predictions read justification computed unexplainable periods no record , version deleted 0 0 6 decision record , version deleted 1200 0 6 decision record + version record 1200 1200 0 same , retention 3 periods 600 600 3 retirement 2026-11: presented periods 5 , decisions 1000 ; contest still open 4 periods / 800 decisions ; label never arriving 2 periods / 400 decisions ``` The second row is the second finding of the lesson: the decision ledger stands complete, 1200 of 1200 predictions are read, but the ones whose justification is computed are **zero** and all six periods remain **unexplainable**. What is missing is two numbers: the weight triple and the decision threshold. **Reading the prediction is not explaining it.** When the version record is kept as well the number returns to 1200, and with a three-period retention it falls to **three**. The Introduction to Data Engineering course had measured this on the data side: a retention policy leaves periods that cannot be audited retrospectively. The last row is the cost of the **timing** of the retirement: of the 1000 decisions of the model removed at `2026-11`, 800 are still open to contest and **the labels of the 400 decisions of two periods will never arrive**, because removal stops the measurement. **Retiring leaves the last two periods permanently unmeasured.** #### Summary - The indicator criterion fires at `2026-10` in both scenarios and in one the retirement is **unjustified**; the accuracy criterion shows the `2026-09` degradation at `2026-11` and the challenger comparison measures an advantage of **0.0000** and **0.0900**. - When the version record is deleted, 1200 predictions are read, the ones whose justification is computed are **0** and six periods remain unexplainable. - Of the 1000 decisions of the model removed at `2026-11`, 800 are open to contest and **the labels of 400 will never arrive**. #### Course Wrap-Up The criterion sits next to the lesson name; the difference in parentheses is **the monitored setup minus the baseline**. | Lesson | Unmonitored baseline | Monitored setup | Cost | |---|---|---|---| | Data Drift — balanced accuracy | old setup: recall **0.0000**, balanced **0.5000** | corrected: recall 1.0000, balanced **0.9835** (**+0.4835**) | at share 0.05 three periods fire, and in all three accuracy does **not fall below** the baseline of 0.9700 | | Concept Drift — accuracy at `2026-09` | no drift seen: **0.9700** | under concept drift **0.8150** (**−0.1550**) | thirty-one steps; the input summary fires in no band, the label of 132 of the 1200 rows changes | | Performance Monitoring — measured period | no label awaited: **0** | delay 2: degradation `2026-09`, warning `2026-11` | in the gap **400 predictions** are produced unmeasured, **63** of them wrong; 400 stay unmeasured each period | | Proxy Metrics — deviation of the proxy | no proxy: **0** | three proxies: **0.0185 / 0.0957 / 0.0000** | all three give the same as the robust scenario under concept drift; the only measure that separates is accuracy (**0.1598**) | | Alerts and Thresholds — firing period | no band: **0** | **13** at the 0.05 band (**+13**) | **6 false alarms**; valid catches **4**, all from accuracy, detection lag 2 periods | | Feedback Collection — measured accuracy | no label arrives: **no** measurement | `positives only`, coverage 0.4858: **1.0000** | deviation **+0.0894**; measured accuracy rises from 0.9362 to **1.0000** while the truth falls to 0.8500 | | Shadow and Canary Deployment — measured accuracy of the candidate | no release: **no** measurement | shadow, 0 requests: **0.9650**; canary 0.10: **1.0000** | at the canary the resolution rises to **0.0500**; at full rollout **7** requests are served wrongly | | Online Experiments — measured difference | no experiment: **0** | six periods **−0.0050**, one period **+0.0250** with **−0.0150** | smallest distinguishable difference **0.0275**; real sample **60** rows | | Retraining Policy — `2026-12` accuracy | no retraining: **0.9800** | all three windows **0.9800** (**+0.0000**) | **3,078,000** rows scored; the indicator trigger fires in the same periods in both scenarios | | Model Auditability — rows read per prediction | no record: **423.70** | with the decision record **1.00** (**−422.70**) | 12,000 stored fields; in the search without a record the model of **189 predictions** cannot be identified | | Compliance Requirements — recomputed decisions | baseline record: **1164/1200** | with record-keeping **1200/1200** (**+36**) | **8402** stored values from 3600; with retention **600** decisions are deleted while the contest is open | | Cost Management — cost per prediction | no retraining: **1.00** | frequency 1: **811.00** (**+810.00**) | training share **0.9988**, accuracy gained **0.0000**, cost per gain undefined | | Model Retirement — unexplainable periods | removal without a record: **6/6** | decision and version record: **0/6** (**−6**) | at retirement the label of **400** decisions never arrives, **800** stay open to contest | The table is a single rule written thirteen times. **A firing indicator is not evidence of degradation; an indicator whose firing and whose degradation are not shown separately counts as unmeasured.** Alerts and Thresholds counts this: of the 13 firings in the 0.05 band, 6 are false alarms, the valid catches are 4, and in Data Drift the warning fires for three periods while accuracy rises. **The second claim is read from the same table: monitoring also breaks.** In Shadow and Canary Deployment the canary share divides the measure: at 0.10 the bad candidate looks like 1.0000, and at full rollout 7 requests are served wrongly. In Retraining Policy three triggers have 3,078,000 rows scored and the gain is 0.0000; in Cost Management the decision to retrain less often carries 0.8323 of the total, again for 0.0000. In Feedback Collection the collection procedure lifts the measured accuracy **above** the truth. **The third claim is written with the names of the proxy criteria: they fire on the harmless drift, they stay silent on the harmful drift.** In Proxy Metrics all three proxies give the same number as the robust scenario under concept drift, digit for digit; in Concept Drift the prediction share fires on the harmless drift in the 0.10 band and stays silent on the harmful drift. The first table of this lesson is the retirement form of that. Six courses read the same axis in six places: **degradation** was counted in the stages of the pipeline, **layout** in storage, **execution** in orchestration, **distribution** in distributed processing, **path** in reproducing the model and **visibility** in the model in production. The rule of the curriculum is written one last time: the number of a data pipeline is **not the rows it carries but how many steps later a wrong row is caught; a pipeline whose degradation-catching check is not written counts as unmeasured.** In seventy-four lessons data was moved, stored, transformed, distributed, given to a model and monitored. All of it was a system standing on the **server side**. How the number this system produces would appear on a **device in the user's pocket**, what that device could do with its own constraints and what would happen when it is **left without a connection** was never asked. The next curriculum moves the number from where it is produced to where it is received.