--- title: 'Model Retirement' source: 'https://academia.sh/en/courses/model-monitoring/model-retirement' course: 'Model Monitoring and Governance' language: en updated: '2026-10-21T05:03:03+00:00' license: 'CC BY-SA 4.0' objectives: - 'Measures three criteria usable for retirement side by side in two drift scenarios.' - 'Shows with numbers that an indicator firing cannot be the justification for a retirement.' - 'Computes how many periods of decisions removing the model leaves unexplainable.' - 'Determines the number of decisions whose label will never arrive at the moment of retirement.' --- # Model Retirement Counting by which measure the retirement decision is made and what removing the model leaves behind retrospectively: the indicator criterion fires at 2026-10 in both scenarios, and in one the retirement is unjustified because accuracy went from 0.9700 to 0.9800; the accuracy criterion never fires on the harmless drift and shows the 2026-09 degradation at 2026-11 on the harmful drift, two periods late; the challenger comparison criterion measures an advantage of 0.0000 on data drift and 0.0900 on concept drift and is the only criterion that separates the two scenarios. After removal without a decision record, six of six periods are unexplainable; when the model version record is deleted while the decision record exists, 1200 predictions are read but the ones whose justification is computed are 0. The lesson is the closing of the course and of the curriculum. The table of the previous lesson was missing the **end** column: when a model would end was written nowhere. This lesson asks two questions: which number justifies the removal, and what the removal leaves behind **retrospectively**. - **RG53.** The source, the pipeline, the model and the two drifts are **fictional**; no product, service or institution is called. Seed 20260218. - **RG54.** The model is `(0.6, 0.0, 4)`, the decision threshold is 18, balanced accuracy is 0.9864; monitoring runs six periods, the **base class ratio is 0.5342**, the step is 0.0050, the label is delayed by **two periods**. - **RG55.** In **data drift** the input moves and the rule stays the same; in **concept drift** the input stays the same and the rule changes (the rule threshold from 30 to 27). - **RG56.** Three criteria are tested: the **indicator** (the prediction share deviates from the baseline by 0.05), the **accuracy threshold** (the period accuracy falls below 0.95), the **challenger comparison** (the advantage is greater than zero). - **RG57.** Removal means the weight and threshold record becomes **unresolvable**; a period whose justification cannot be computed counts as **unexplainable**. The retirement is `2026-11`, the contest and the retention are three periods each. ## The Measure of the Retirement Decision The first block builds the model and the two drifts and fires the three criteria. ```python # FICTIONAL. The source , the consumption stage of the pipeline , the feature path , the threshold # model , the two drifts and the retirement criteria are generated inside the lesson; no product , # service or institution is called. Seed 20260218. SEED, M32 = 20260218, 0xFFFFFFFF def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value REGION, TYPE = ["east", "west", "north", "south", "central"], ["residential", "commercial"] PERIODS = [f"2026-{a:02d}" for a in range(1, 13)] SUBSCRIBERS = {} for i in range(200): r = rng(SEED + 37 * i) SUBSCRIBERS[f"A{i + 1:03d}"] = {"region": REGION[int(r() * 5)], "type": TYPE[int(r() * 2)], "start": 10000 + int(r() * 60000)} def rows(): raw = [] for i, (sub, a) in enumerate(SUBSCRIBERS.items()): e = a["start"] for p in PERIODS: r = rng(SEED + 101 * i + 7 * PERIODS.index(p)) e += 4 + int(r() * 46) raw.append({"subscriber": sub, "period": p, "region": a["region"], "type": a["type"], "index": e}) t = {(k["subscriber"], k["period"]): k for k in raw} return [dict(k, m3=k["index"] - t[(k["subscriber"], PERIODS[PERIODS.index(k["period"]) - 1])] ["index"]) for k in raw if PERIODS.index(k["period"])] def features(records): """Independent of the weights , computed once.""" sub, reg = {}, {} for k in sorted(records, key=lambda k: PERIODS.index(k["period"])): sub.setdefault(k["subscriber"], []).append((PERIODS.index(k["period"]), k["m3"])) reg.setdefault(k["region"], []).append((PERIODS.index(k["period"]), k["m3"])) def mean(series, i): s = [v for d, v in series if d < i] return round(sum(s) / len(s), 4) if s else 0.0 return {(k["subscriber"], k["period"]): { "subscriber_mean": mean(sub[k["subscriber"]], PERIODS.index(k["period"])), "region_mean": mean(reg[k["region"]], PERIODS.index(k["period"])), "commercial": 1 if k["type"] == "commercial" else 0} for k in records} GRID = [(a, b, c) for a in (0.6, 0.8, 1.0) for b in (0.0, 0.2, 0.4) for c in (0, 4, 8)] DECISION_GRID = (18, 22, 26, 30, 34) MONITORED, DELAY = PERIODS[6:], 2 RULE_THRESHOLD, RULE_COMMERCIAL, ACCURACY_THRESHOLD = 30, 6, 0.95 def score(o, w): return w[0] * o["subscriber_mean"] + w[1] * o["region_mean"] + w[2] * o["commercial"] def predict(o, w, decision): return 1 if score(o, w) > decision else 0 def label(o, threshold=RULE_THRESHOLD): return 1 if o["subscriber_mean"] + RULE_COMMERCIAL * o["commercial"] > threshold else 0 def train(records, feats, periods, labels=None): """The criterion is balanced accuracy; the decision threshold is A SEPARATE setting and is searched.""" subset = [k for k in records if k["period"] in periods] best = (-1.0, None, None) for w in GRID: for decision in DECISION_GRID: tp = tn = fp = fn = 0 for k in subset: c = (k["subscriber"], k["period"]) p = predict(feats[c], w, decision) g = labels[c] if labels else label(feats[c]) tp, fp, fn, tn = (tp + (p and g), fp + (p and not g), fn + (not p and g), tn + (not p and not g)) d = (tp / (tp + fn) if tp + fn else 0.0) + (tn / (tn + fp) if tn + fp else 0.0) if d / 2 > best[0]: best = (d / 2, w, decision) return {"weights": best[1], "decision_threshold": best[2], "balanced": round(best[0], 4)} def measure(records, m, feats, period, labels=None): subset = [(k["subscriber"], k["period"]) for k in records if k["period"] == period] p = [predict(feats[c], m["weights"], m["decision_threshold"]) for c in subset] g = [labels[c] if labels else label(feats[c]) for c in subset] return (round(sum(p) / len(p), 4), round(sum(1 for a, b in zip(p, g) if a == b) / len(p), 4)) CLEAN = rows() FEATS = features(CLEAN) M = train(CLEAN, FEATS, PERIODS[1:9]) # FICTIONAL data drift: the input drifts , the rule stays the same. b = PERIODS.index("2026-09") DRIFTED = [dict(k, m3=round(k["m3"] * (1 + 0.06 * (PERIODS.index(k["period"]) - b + 1)), 4)) if PERIODS.index(k["period"]) >= b and k["type"] == "commercial" else dict(k) for k in CLEAN] FEATS_D = features(DRIFTED) # FICTIONAL concept drift: the input stays THE SAME , the rule changes (the threshold goes down from 30 to 27). CONCEPT = {(k["subscriber"], k["period"]): label( FEATS[(k["subscriber"], k["period"])], 27 if PERIODS.index(k["period"]) >= b else RULE_THRESHOLD) for k in CLEAN} p1 = sum(label(FEATS[(k["subscriber"], k["period"])]) for k in CLEAN if k["period"] in MONITORED) / 1200 print(f"model {M['weights']} decision threshold {M['decision_threshold']} balanced {M['balanced']} ," f" base class ratio {round(max(p1, 1 - p1), 4)} , step {round(1 / 200, 4)}") print("\nperiod data: share accuracy | concept: share accuracy") SCENARIO = {"data": (DRIFTED, FEATS_D, None), "concept": (CLEAN, FEATS, CONCEPT)} MEASURED = {name: [(d,) + measure(k, M, o, d, e) for d in MONITORED] for name, (k, o, e) in SCENARIO.items()} for i, d in enumerate(MONITORED): print(f"{d} {MEASURED['data'][i][1]:>11} {MEASURED['data'][i][2]:>8} |" f" {MEASURED['concept'][i][1]:>14} {MEASURED['concept'][i][2]:>8}") print(f"\n{'criterion':<40}{'scenario':>8} {'fired':>7} {'drop':>7}" f" {'lag':>3} {'justified':>9}") for name, (rec, feats, lab) in SCENARIO.items(): baseline = MEASURED[name][1][1] indicator = [d for d, p, _ in MEASURED[name] if abs(p - baseline) / baseline > 0.05] drops = [d for d, _, i in MEASURED[name] if i < ACCURACY_THRESHOLD] seen = PERIODS[PERIODS.index(drops[0]) + DELAY] if drops else "-" challenger = train(rec, feats, MONITORED, lab) advantage = round(measure(rec, challenger, feats, "2026-12", lab)[1] - MEASURED[name][-1][2], 4) for criterion, fired, lag in (("indicator (share 0.05)", indicator[0] if indicator else "-", "-"), (f"accuracy threshold {ACCURACY_THRESHOLD}", seen, DELAY if drops else "-"), (f"challenger comparison (advantage {advantage})", seen if advantage > 0 else "-", DELAY if advantage > 0 else "-")): ok = "no" if (fired != "-" and not drops) else ("yes" if fired != "-" else "-") print(f"{criterion:<40}{name:>8} {str(fired):>7} {str(drops[0]) if drops else '-':>7}" f" {str(lag):>3} {ok:>9}") ``` ``` model (0.6, 0.0, 4) decision threshold 18 balanced 0.9864 , base class ratio 0.5342 , step 0.005 period data: share accuracy | concept: share accuracy 2026-07 0.455 0.97 | 0.455 0.97 2026-08 0.47 0.97 | 0.47 0.97 2026-09 0.455 0.99 | 0.455 0.815 2026-10 0.515 0.975 | 0.51 0.87 2026-11 0.52 0.98 | 0.51 0.865 2026-12 0.545 0.98 | 0.515 0.85 criterion scenario fired drop lag justified indicator (share 0.05) data 2026-10 - - no accuracy threshold 0.95 data - - - - challenger comparison (advantage 0.0) data - - - - indicator (share 0.05) concept 2026-10 2026-09 - yes accuracy threshold 0.95 concept 2026-11 2026-09 2 yes challenger comparison (advantage 0.09) concept 2026-11 2026-09 2 yes ``` The indicator criterion fires **at `2026-10` in both scenarios**: in one, accuracy has gone from 0.9700 to 0.9800 and the retirement is **unjustified**, in the other the degradation is real. The firing period is the same, the right decision is opposite. **An indicator whose firing and whose degradation are not shown separately cannot be the justification for removing a model.** The accuracy criterion is right in both: the degradation happens at `2026-09`, the criterion triggers at `2026-11` — the **two periods** in between are the delayed label itself. The challenger comparison measures the advantage as **0.0000** and **0.0900**, eighteen steps of the resolution. **The two criteria that separate also read labels** and arrive late; the only criterion with no delay is the one that cannot tell the difference. ## What Removal Leaves Behind Retrospectively The second block removes the model and counts what remains. ```python # This block continues the PERIODS , MONITORED , FEATS , M , CLEAN , score and predict # definitions of the first block. RETIREMENT, RETENTION, CONTEST = "2026-11", 3, 3 ledger = [dict(FEATS[(k["subscriber"], k["period"])], period=k["period"], model_version="s1", prediction=predict(FEATS[(k["subscriber"], k["period"])], M["weights"], M["decision_threshold"])) for k in CLEAN if k["period"] in MONITORED] FULL = {"s1": {"weights": M["weights"], "decision_threshold": M["decision_threshold"]}} remaining = [d for d in MONITORED if PERIODS.index(d) > len(PERIODS) - 1 - RETENTION] def justification(r, version): """The justification is computed only if the model version can be resolved.""" s = version.get(r["model_version"]) return None if s is None else round(score(r, s["weights"]) - s["decision_threshold"], 4) print("setup predictions read justification computed" " unexplainable periods") for name, rec, ver in (("no record , version deleted", [], {}), ("decision record , version deleted", ledger, {}), ("decision record + version record", ledger, FULL), (f"same , retention {RETENTION} periods", [r for r in ledger if r["period"] in remaining], FULL)): g = [r for r in rec if justification(r, ver) is not None] print(f"{name:<37}{len(rec):>16} {len(g):>22}" f" {len(MONITORED) - len(set(r['period'] for r in g)):>21}") e = PERIODS.index(RETIREMENT) shown = [d for d in MONITORED if PERIODS.index(d) <= e] open_ = [d for d in shown if PERIODS.index(d) + CONTEST >= e] late = [d for d in shown if PERIODS.index(d) + DELAY > e] print(f"\nretirement {RETIREMENT}: presented periods {len(shown)} , decisions {len(shown) * 200} ;" f" contest still open {len(open_)} periods / {len(open_) * 200} decisions ;" f" label never arriving {len(late)} periods / {len(late) * 200} decisions") ``` ``` setup predictions read justification computed unexplainable periods no record , version deleted 0 0 6 decision record , version deleted 1200 0 6 decision record + version record 1200 1200 0 same , retention 3 periods 600 600 3 retirement 2026-11: presented periods 5 , decisions 1000 ; contest still open 4 periods / 800 decisions ; label never arriving 2 periods / 400 decisions ``` The second row is the second finding of the lesson: the decision ledger stands complete, 1200 of 1200 predictions are read, but the ones whose justification is computed are **zero** and all six periods remain **unexplainable**. What is missing is two numbers: the weight triple and the decision threshold. **Reading the prediction is not explaining it.** When the version record is kept as well the number returns to 1200, and with a three-period retention it falls to **three**. The Introduction to Data Engineering course had measured this on the data side: a retention policy leaves periods that cannot be audited retrospectively. The last row is the cost of the **timing** of the retirement: of the 1000 decisions of the model removed at `2026-11`, 800 are still open to contest and **the labels of the 400 decisions of two periods will never arrive**, because removal stops the measurement. **Retiring leaves the last two periods permanently unmeasured.** ## Summary - The indicator criterion fires at `2026-10` in both scenarios and in one the retirement is **unjustified**; the accuracy criterion shows the `2026-09` degradation at `2026-11` and the challenger comparison measures an advantage of **0.0000** and **0.0900**. - When the version record is deleted, 1200 predictions are read, the ones whose justification is computed are **0** and six periods remain unexplainable. - Of the 1000 decisions of the model removed at `2026-11`, 800 are open to contest and **the labels of 400 will never arrive**. ## Course Wrap-Up The criterion sits next to the lesson name; the difference in parentheses is **the monitored setup minus the baseline**. | Lesson | Unmonitored baseline | Monitored setup | Cost | |---|---|---|---| | Data Drift — balanced accuracy | old setup: recall **0.0000**, balanced **0.5000** | corrected: recall 1.0000, balanced **0.9835** (**+0.4835**) | at share 0.05 three periods fire, and in all three accuracy does **not fall below** the baseline of 0.9700 | | Concept Drift — accuracy at `2026-09` | no drift seen: **0.9700** | under concept drift **0.8150** (**−0.1550**) | thirty-one steps; the input summary fires in no band, the label of 132 of the 1200 rows changes | | Performance Monitoring — measured period | no label awaited: **0** | delay 2: degradation `2026-09`, warning `2026-11` | in the gap **400 predictions** are produced unmeasured, **63** of them wrong; 400 stay unmeasured each period | | Proxy Metrics — deviation of the proxy | no proxy: **0** | three proxies: **0.0185 / 0.0957 / 0.0000** | all three give the same as the robust scenario under concept drift; the only measure that separates is accuracy (**0.1598**) | | Alerts and Thresholds — firing period | no band: **0** | **13** at the 0.05 band (**+13**) | **6 false alarms**; valid catches **4**, all from accuracy, detection lag 2 periods | | Feedback Collection — measured accuracy | no label arrives: **no** measurement | `positives only`, coverage 0.4858: **1.0000** | deviation **+0.0894**; measured accuracy rises from 0.9362 to **1.0000** while the truth falls to 0.8500 | | Shadow and Canary Deployment — measured accuracy of the candidate | no release: **no** measurement | shadow, 0 requests: **0.9650**; canary 0.10: **1.0000** | at the canary the resolution rises to **0.0500**; at full rollout **7** requests are served wrongly | | Online Experiments — measured difference | no experiment: **0** | six periods **−0.0050**, one period **+0.0250** with **−0.0150** | smallest distinguishable difference **0.0275**; real sample **60** rows | | Retraining Policy — `2026-12` accuracy | no retraining: **0.9800** | all three windows **0.9800** (**+0.0000**) | **3,078,000** rows scored; the indicator trigger fires in the same periods in both scenarios | | Model Auditability — rows read per prediction | no record: **423.70** | with the decision record **1.00** (**−422.70**) | 12,000 stored fields; in the search without a record the model of **189 predictions** cannot be identified | | Compliance Requirements — recomputed decisions | baseline record: **1164/1200** | with record-keeping **1200/1200** (**+36**) | **8402** stored values from 3600; with retention **600** decisions are deleted while the contest is open | | Cost Management — cost per prediction | no retraining: **1.00** | frequency 1: **811.00** (**+810.00**) | training share **0.9988**, accuracy gained **0.0000**, cost per gain undefined | | Model Retirement — unexplainable periods | removal without a record: **6/6** | decision and version record: **0/6** (**−6**) | at retirement the label of **400** decisions never arrives, **800** stay open to contest | The table is a single rule written thirteen times. **A firing indicator is not evidence of degradation; an indicator whose firing and whose degradation are not shown separately counts as unmeasured.** Alerts and Thresholds counts this: of the 13 firings in the 0.05 band, 6 are false alarms, the valid catches are 4, and in Data Drift the warning fires for three periods while accuracy rises. **The second claim is read from the same table: monitoring also breaks.** In Shadow and Canary Deployment the canary share divides the measure: at 0.10 the bad candidate looks like 1.0000, and at full rollout 7 requests are served wrongly. In Retraining Policy three triggers have 3,078,000 rows scored and the gain is 0.0000; in Cost Management the decision to retrain less often carries 0.8323 of the total, again for 0.0000. In Feedback Collection the collection procedure lifts the measured accuracy **above** the truth. **The third claim is written with the names of the proxy criteria: they fire on the harmless drift, they stay silent on the harmful drift.** In Proxy Metrics all three proxies give the same number as the robust scenario under concept drift, digit for digit; in Concept Drift the prediction share fires on the harmless drift in the 0.10 band and stays silent on the harmful drift. The first table of this lesson is the retirement form of that. Six courses read the same axis in six places: **degradation** was counted in the stages of the pipeline, **layout** in storage, **execution** in orchestration, **distribution** in distributed processing, **path** in reproducing the model and **visibility** in the model in production. The rule of the curriculum is written one last time: the number of a data pipeline is **not the rows it carries but how many steps later a wrong row is caught; a pipeline whose degradation-catching check is not written counts as unmeasured.** In seventy-four lessons data was moved, stored, transformed, distributed, given to a model and monitored. All of it was a system standing on the **server side**. How the number this system produces would appear on a **device in the user's pocket**, what that device could do with its own constraints and what would happen when it is **left without a connection** was never asked. The next curriculum moves the number from where it is produced to where it is received.