Model Retirement

The table of the previous lesson was missing the end column: when a model would end was written nowhere. This lesson asks two questions: which number justifies the removal, and what the removal leaves behind retrospectively.

• RG53. The source, the pipeline, the model and the two drifts are fictional; no product, service or institution is called. Seed 20260218.
• RG54. The model is (0.6, 0.0, 4), the decision threshold is 18, balanced accuracy is 0.9864; monitoring runs six periods, the base class ratio is 0.5342, the step is 0.0050, the label is delayed by two periods.
• RG55. In data drift the input moves and the rule stays the same; in concept drift the input stays the same and the rule changes (the rule threshold from 30 to 27).
• RG56. Three criteria are tested: the indicator (the prediction share deviates from the baseline by 0.05), the accuracy threshold (the period accuracy falls below 0.95), the challenger comparison (the advantage is greater than zero).
• RG57. Removal means the weight and threshold record becomes unresolvable; a period whose justification cannot be computed counts as unexplainable. The retirement is 2026-11, the contest and the retention are three periods each.

The Measure of the Retirement Decision

The first block builds the model and the two drifts and fires the three criteria.

# FICTIONAL. The source , the consumption stage of the pipeline , the feature path , the threshold
# model , the two drifts and the retirement criteria are generated inside the lesson; no product ,
# service or institution is called. Seed 20260218.
SEED, M32 = 20260218, 0xFFFFFFFF


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


REGION, TYPE = ["east", "west", "north", "south", "central"], ["residential", "commercial"]
PERIODS = [f"2026-{a:02d}" for a in range(1, 13)]
SUBSCRIBERS = {}
for i in range(200):
    r = rng(SEED + 37 * i)
    SUBSCRIBERS[f"A{i + 1:03d}"] = {"region": REGION[int(r() * 5)], "type": TYPE[int(r() * 2)],
                                    "start": 10000 + int(r() * 60000)}


def rows():
    raw = []
    for i, (sub, a) in enumerate(SUBSCRIBERS.items()):
        e = a["start"]
        for p in PERIODS:
            r = rng(SEED + 101 * i + 7 * PERIODS.index(p))
            e += 4 + int(r() * 46)
            raw.append({"subscriber": sub, "period": p, "region": a["region"], "type": a["type"],
                        "index": e})
    t = {(k["subscriber"], k["period"]): k for k in raw}
    return [dict(k, m3=k["index"] - t[(k["subscriber"], PERIODS[PERIODS.index(k["period"]) - 1])]
                 ["index"]) for k in raw if PERIODS.index(k["period"])]


def features(records):
    """Independent of the weights , computed once."""
    sub, reg = {}, {}
    for k in sorted(records, key=lambda k: PERIODS.index(k["period"])):
        sub.setdefault(k["subscriber"], []).append((PERIODS.index(k["period"]), k["m3"]))
        reg.setdefault(k["region"], []).append((PERIODS.index(k["period"]), k["m3"]))

    def mean(series, i):
        s = [v for d, v in series if d < i]
        return round(sum(s) / len(s), 4) if s else 0.0

    return {(k["subscriber"], k["period"]): {
        "subscriber_mean": mean(sub[k["subscriber"]], PERIODS.index(k["period"])),
        "region_mean": mean(reg[k["region"]], PERIODS.index(k["period"])),
        "commercial": 1 if k["type"] == "commercial" else 0} for k in records}


GRID = [(a, b, c) for a in (0.6, 0.8, 1.0) for b in (0.0, 0.2, 0.4) for c in (0, 4, 8)]
DECISION_GRID = (18, 22, 26, 30, 34)
MONITORED, DELAY = PERIODS[6:], 2
RULE_THRESHOLD, RULE_COMMERCIAL, ACCURACY_THRESHOLD = 30, 6, 0.95


def score(o, w):
    return w[0] * o["subscriber_mean"] + w[1] * o["region_mean"] + w[2] * o["commercial"]


def predict(o, w, decision):
    return 1 if score(o, w) > decision else 0


def label(o, threshold=RULE_THRESHOLD):
    return 1 if o["subscriber_mean"] + RULE_COMMERCIAL * o["commercial"] > threshold else 0


def train(records, feats, periods, labels=None):
    """The criterion is balanced accuracy; the decision threshold is A SEPARATE setting and is searched."""
    subset = [k for k in records if k["period"] in periods]
    best = (-1.0, None, None)
    for w in GRID:
        for decision in DECISION_GRID:
            tp = tn = fp = fn = 0
            for k in subset:
                c = (k["subscriber"], k["period"])
                p = predict(feats[c], w, decision)
                g = labels[c] if labels else label(feats[c])
                tp, fp, fn, tn = (tp + (p and g), fp + (p and not g),
                                  fn + (not p and g), tn + (not p and not g))
            d = (tp / (tp + fn) if tp + fn else 0.0) + (tn / (tn + fp) if tn + fp else 0.0)
            if d / 2 > best[0]:
                best = (d / 2, w, decision)
    return {"weights": best[1], "decision_threshold": best[2], "balanced": round(best[0], 4)}


def measure(records, m, feats, period, labels=None):
    subset = [(k["subscriber"], k["period"]) for k in records if k["period"] == period]
    p = [predict(feats[c], m["weights"], m["decision_threshold"]) for c in subset]
    g = [labels[c] if labels else label(feats[c]) for c in subset]
    return (round(sum(p) / len(p), 4),
            round(sum(1 for a, b in zip(p, g) if a == b) / len(p), 4))


CLEAN = rows()
FEATS = features(CLEAN)
M = train(CLEAN, FEATS, PERIODS[1:9])
# FICTIONAL data drift: the input drifts , the rule stays the same.
b = PERIODS.index("2026-09")
DRIFTED = [dict(k, m3=round(k["m3"] * (1 + 0.06 * (PERIODS.index(k["period"]) - b + 1)), 4))
           if PERIODS.index(k["period"]) >= b and k["type"] == "commercial" else dict(k) for k in CLEAN]
FEATS_D = features(DRIFTED)
# FICTIONAL concept drift: the input stays THE SAME , the rule changes (the threshold goes down from 30 to 27).
CONCEPT = {(k["subscriber"], k["period"]): label(
    FEATS[(k["subscriber"], k["period"])], 27 if PERIODS.index(k["period"]) >= b else RULE_THRESHOLD)
    for k in CLEAN}

p1 = sum(label(FEATS[(k["subscriber"], k["period"])]) for k in CLEAN if k["period"] in MONITORED) / 1200
print(f"model {M['weights']} decision threshold {M['decision_threshold']} balanced {M['balanced']} ,"
      f" base class ratio {round(max(p1, 1 - p1), 4)} , step {round(1 / 200, 4)}")
print("\nperiod     data: share  accuracy | concept: share  accuracy")
SCENARIO = {"data": (DRIFTED, FEATS_D, None), "concept": (CLEAN, FEATS, CONCEPT)}
MEASURED = {name: [(d,) + measure(k, M, o, d, e) for d in MONITORED]
            for name, (k, o, e) in SCENARIO.items()}
for i, d in enumerate(MONITORED):
    print(f"{d}    {MEASURED['data'][i][1]:>11}  {MEASURED['data'][i][2]:>8} |"
          f" {MEASURED['concept'][i][1]:>14}  {MEASURED['concept'][i][2]:>8}")

print(f"\n{'criterion':<40}{'scenario':>8}  {'fired':>7}  {'drop':>7}"
      f"  {'lag':>3}  {'justified':>9}")
for name, (rec, feats, lab) in SCENARIO.items():
    baseline = MEASURED[name][1][1]
    indicator = [d for d, p, _ in MEASURED[name] if abs(p - baseline) / baseline > 0.05]
    drops = [d for d, _, i in MEASURED[name] if i < ACCURACY_THRESHOLD]
    seen = PERIODS[PERIODS.index(drops[0]) + DELAY] if drops else "-"
    challenger = train(rec, feats, MONITORED, lab)
    advantage = round(measure(rec, challenger, feats, "2026-12", lab)[1] - MEASURED[name][-1][2], 4)
    for criterion, fired, lag in (("indicator (share 0.05)", indicator[0] if indicator else "-", "-"),
                                  (f"accuracy threshold {ACCURACY_THRESHOLD}", seen, DELAY if drops else "-"),
                                  (f"challenger comparison (advantage {advantage})",
                                   seen if advantage > 0 else "-", DELAY if advantage > 0 else "-")):
        ok = "no" if (fired != "-" and not drops) else ("yes" if fired != "-" else "-")
        print(f"{criterion:<40}{name:>8}  {str(fired):>7}  {str(drops[0]) if drops else '-':>7}"
              f"  {str(lag):>3}  {ok:>9}")

model (0.6, 0.0, 4) decision threshold 18 balanced 0.9864 , base class ratio 0.5342 , step 0.005

period     data: share  accuracy | concept: share  accuracy
2026-07          0.455      0.97 |          0.455      0.97
2026-08           0.47      0.97 |           0.47      0.97
2026-09          0.455      0.99 |          0.455     0.815
2026-10          0.515     0.975 |           0.51      0.87
2026-11           0.52      0.98 |           0.51     0.865
2026-12          0.545      0.98 |          0.515      0.85

criterion                               scenario    fired     drop  lag  justified
indicator (share 0.05)                      data  2026-10        -    -         no
accuracy threshold 0.95                     data        -        -    -          -
challenger comparison (advantage 0.0)       data        -        -    -          -
indicator (share 0.05)                   concept  2026-10  2026-09    -        yes
accuracy threshold 0.95                  concept  2026-11  2026-09    2        yes
challenger comparison (advantage 0.09)   concept  2026-11  2026-09    2        yes

The indicator criterion fires at 2026-10 in both scenarios: in one, accuracy has gone from 0.9700 to 0.9800 and the retirement is unjustified, in the other the degradation is real. The firing period is the same, the right decision is opposite. An indicator whose firing and whose degradation are not shown separately cannot be the justification for removing a model.

The accuracy criterion is right in both: the degradation happens at 2026-09, the criterion triggers at 2026-11 — the two periods in between are the delayed label itself. The challenger comparison measures the advantage as 0.0000 and 0.0900, eighteen steps of the resolution. The two criteria that separate also read labels and arrive late; the only criterion with no delay is the one that cannot tell the difference.

What Removal Leaves Behind Retrospectively

The second block removes the model and counts what remains.

# This block continues the PERIODS , MONITORED , FEATS , M , CLEAN , score and predict
# definitions of the first block.
RETIREMENT, RETENTION, CONTEST = "2026-11", 3, 3
ledger = [dict(FEATS[(k["subscriber"], k["period"])], period=k["period"], model_version="s1",
               prediction=predict(FEATS[(k["subscriber"], k["period"])], M["weights"], M["decision_threshold"]))
          for k in CLEAN if k["period"] in MONITORED]
FULL = {"s1": {"weights": M["weights"], "decision_threshold": M["decision_threshold"]}}
remaining = [d for d in MONITORED if PERIODS.index(d) > len(PERIODS) - 1 - RETENTION]


def justification(r, version):
    """The justification is computed only if the model version can be resolved."""
    s = version.get(r["model_version"])
    return None if s is None else round(score(r, s["weights"]) - s["decision_threshold"], 4)


print("setup                                predictions read  justification computed"
      "  unexplainable periods")
for name, rec, ver in (("no record , version deleted", [], {}),
                       ("decision record , version deleted", ledger, {}),
                       ("decision record + version record", ledger, FULL),
                       (f"same , retention {RETENTION} periods",
                        [r for r in ledger if r["period"] in remaining], FULL)):
    g = [r for r in rec if justification(r, ver) is not None]
    print(f"{name:<37}{len(rec):>16}  {len(g):>22}"
          f"  {len(MONITORED) - len(set(r['period'] for r in g)):>21}")

e = PERIODS.index(RETIREMENT)
shown = [d for d in MONITORED if PERIODS.index(d) <= e]
open_ = [d for d in shown if PERIODS.index(d) + CONTEST >= e]
late = [d for d in shown if PERIODS.index(d) + DELAY > e]
print(f"\nretirement {RETIREMENT}: presented periods {len(shown)} , decisions {len(shown) * 200} ;"
      f" contest still open {len(open_)} periods / {len(open_) * 200} decisions ;"
      f" label never arriving {len(late)} periods / {len(late) * 200} decisions")

setup                                predictions read  justification computed  unexplainable periods
no record , version deleted                         0                       0                      6
decision record , version deleted                1200                       0                      6
decision record + version record                 1200                    1200                      0
same , retention 3 periods                        600                     600                      3

retirement 2026-11: presented periods 5 , decisions 1000 ; contest still open 4 periods / 800 decisions ; label never arriving 2 periods / 400 decisions

The second row is the second finding of the lesson: the decision ledger stands complete, 1200 of 1200 predictions are read, but the ones whose justification is computed are zero and all six periods remain unexplainable. What is missing is two numbers: the weight triple and the decision threshold. Reading the prediction is not explaining it. When the version record is kept as well the number returns to 1200, and with a three-period retention it falls to three. The Introduction to Data Engineering course had measured this on the data side: a retention policy leaves periods that cannot be audited retrospectively.

The last row is the cost of the timing of the retirement: of the 1000 decisions of the model removed at 2026-11, 800 are still open to contest and the labels of the 400 decisions of two periods will never arrive, because removal stops the measurement. Retiring leaves the last two periods permanently unmeasured.

Summary

• The indicator criterion fires at 2026-10 in both scenarios and in one the retirement is unjustified; the accuracy criterion shows the 2026-09 degradation at 2026-11 and the challenger comparison measures an advantage of 0.0000 and 0.0900.
• When the version record is deleted, 1200 predictions are read, the ones whose justification is computed are 0 and six periods remain unexplainable.
• Of the 1000 decisions of the model removed at 2026-11, 800 are open to contest and the labels of 400 will never arrive.

Course Wrap-Up

The criterion sits next to the lesson name; the difference in parentheses is the monitored setup minus the baseline.

The table is a single rule written thirteen times. A firing indicator is not evidence of degradation; an indicator whose firing and whose degradation are not shown separately counts as unmeasured. Alerts and Thresholds counts this: of the 13 firings in the 0.05 band, 6 are false alarms, the valid catches are 4, and in Data Drift the warning fires for three periods while accuracy rises.

The second claim is read from the same table: monitoring also breaks. In Shadow and Canary Deployment the canary share divides the measure: at 0.10 the bad candidate looks like 1.0000, and at full rollout 7 requests are served wrongly. In Retraining Policy three triggers have 3,078,000 rows scored and the gain is 0.0000; in Cost Management the decision to retrain less often carries 0.8323 of the total, again for 0.0000. In Feedback Collection the collection procedure lifts the measured accuracy above the truth.

The third claim is written with the names of the proxy criteria: they fire on the harmless drift, they stay silent on the harmful drift. In Proxy Metrics all three proxies give the same number as the robust scenario under concept drift, digit for digit; in Concept Drift the prediction share fires on the harmless drift in the 0.10 band and stays silent on the harmful drift. The first table of this lesson is the retirement form of that.

Six courses read the same axis in six places: degradation was counted in the stages of the pipeline, layout in storage, execution in orchestration, distribution in distributed processing, path in reproducing the model and visibility in the model in production. The rule of the curriculum is written one last time: the number of a data pipeline is not the rows it carries but how many steps later a wrong row is caught; a pipeline whose degradation-catching check is not written counts as unmeasured.

In seventy-four lessons data was moved, stored, transformed, distributed, given to a model and monitored. All of it was a system standing on the server side. How the number this system produces would appear on a device in the user’s pocket, what that device could do with its own constraints and what would happen when it is left without a connection was never asked. The next curriculum moves the number from where it is produced to where it is received.