Retraining Policy

The previous lesson measured the cost of comparing two models in production: the minimum difference a single-period experiment can distinguish on this set lies between 0.0050 and 0.0275, and every comparison below it is inconclusive. That comparing is expensive sharpens the question before it: why is the new model being trained.

In production the answer to this question is a retraining trigger: the calendar has come, an indicator has moved outside its band, or accuracy has really dropped. This lesson runs three kinds of triggers in two separate degradation scenarios and writes two numbers next to each: how many models were trained and how much accuracy was gained. The second number is the clearest example of the second claim of this course.

• RG21. The source, the pipeline and the model are fictional; no learning library, orchestrator, deployment tool or monitoring product is called. Seed 20260218.
• RG22. The label is tied to the rule subscriber_mean + 6 x commercial > 30 and this rule is the concept. The decision threshold is a separate setting, the training criterion is balanced accuracy, that is, the mean of recall and specificity. In the monitored periods the base class ratio is 0.5342, there are 200 rows per period and the step of accuracy is 0.0050.
• RG23. There are two scenarios. In the data drift the input drifts and the rule stays the same. In the concept drift the input stays the same and the threshold of the rule falls from 30 to 27 starting from 2026-09.
• RG24. The label is delayed: the label of a period is known two periods later. Retraining can use only the labeled periods.
• RG25. Training cost is counted as rows scored: every retraining scores the rows in the window for 135 candidates. Raw time is not written.
• RG26. The gain is measured against the 2026-12 accuracy of the old model that was never retrained in that scenario.

Window Sweep: The Best Case

The first block sets up the setup and retrains with three separate windows in production with a data drift. The label delay is ignored in this block, that is, the best case that can be reached is measured.

# FICTIONAL. The meter readings of the measurement network, the consumption stage of the
# pipeline, the feature path and the three-weight threshold model are generated inside
# the lesson; no learning library, deployment tool, experiment platform or monitoring
# product is called.
# Seed 20260218.
SEED, M32 = 20260218, 0xFFFFFFFF


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


REGION = ["east", "west", "north", "south", "central"]
TYPE = ["residential", "commercial"]
PERIODS = [f"2026-{a:02d}" for a in range(1, 13)]
SUBSCRIBERS = {}
for i in range(200):
    r = rng(SEED + 37 * i)
    SUBSCRIBERS[f"A{i + 1:03d}"] = {"region": REGION[int(r() * 5)], "type": TYPE[int(r() * 2)],
                                    "start": 10000 + int(r() * 60000)}


def rows():
    """Source readings and consumption from the period difference: the first three stages of the pipeline."""
    records = []
    for i, (sub, a) in enumerate(SUBSCRIBERS.items()):
        e = a["start"]
        for p in PERIODS:
            r = rng(SEED + 101 * i + 7 * PERIODS.index(p))
            e += 4 + int(r() * 46)
            records.append({"subscriber": sub, "period": p, "region": a["region"], "type": a["type"],
                            "index": e})
    t = {(k["subscriber"], k["period"]): k["index"] for k in records}
    return [dict(k, m3=k["index"] - t[(k["subscriber"], PERIODS[PERIODS.index(k["period"]) - 1])])
            for k in records if PERIODS.index(k["period"])]


def features(records):
    """Independent of the weights and computed ONCE; it sees the full history."""
    sub, reg = {}, {}
    for k in sorted(records, key=lambda k: PERIODS.index(k["period"])):
        sub.setdefault(k["subscriber"], []).append((PERIODS.index(k["period"]), k["m3"]))
        reg.setdefault(k["region"], []).append((PERIODS.index(k["period"]), k["m3"]))

    def mean(series, i):
        s = [v for d, v in series if d < i]
        return round(sum(s) / len(s), 4) if s else 0.0

    return {(k["subscriber"], k["period"]): {
        "subscriber_mean": mean(sub[k["subscriber"]], PERIODS.index(k["period"])),
        "region_mean": mean(reg[k["region"]], PERIODS.index(k["period"])),
        "commercial": 1 if k["type"] == "commercial" else 0} for k in records}


RULE_THRESHOLD, RULE_COMMERCIAL = 30, 6   # the RULE that generates the label , in other words the concept
MONITORED = PERIODS[6:]                   # six periods monitored in production
GRID = [(a, b, c) for a in (0.6, 0.8, 1.0) for b in (0.0, 0.2, 0.4) for c in (0, 4, 8)]
DECISION_GRID = (18, 22, 26, 30, 34)      # the decision threshold is A SEPARATE setting


def label(o, threshold=RULE_THRESHOLD):
    return 1 if o["subscriber_mean"] + RULE_COMMERCIAL * o["commercial"] > threshold else 0


def score(o, w):
    return w[0] * o["subscriber_mean"] + w[1] * o["region_mean"] + w[2] * o["commercial"]


def predict(o, w, decision):
    return 1 if score(o, w) > decision else 0


def train(records, feats, periods, labels):
    """Weights and decision threshold are searched TOGETHER (27 x 5 = 135 candidates); the
    criterion is the mean of recall and specificity, that is BALANCED ACCURACY."""
    subset = [k for k in records if k["period"] in periods]
    best = (-1.0, None, None, 0.0, 0.0)
    for w in GRID:
        for dc in DECISION_GRID:
            tp = tn = fp = fn = 0
            for k in subset:
                o = feats[(k["subscriber"], k["period"])]
                p, g = predict(o, w, dc), labels[(k["subscriber"], k["period"])]
                if p and g:
                    tp += 1
                elif p:
                    fp += 1
                elif g:
                    fn += 1
                else:
                    tn += 1
            recall = tp / (tp + fn) if tp + fn else 0.0
            spec = tn / (tn + fp) if tn + fp else 0.0
            if (recall + spec) / 2 > best[0]:
                best = ((recall + spec) / 2, w, dc, recall, spec)
    return {"weights": best[1], "decision": best[2], "balanced": round(best[0], 4),
            "recall": round(best[3], 4), "specificity": round(best[4], 4), "rows": len(subset)}


def drift(records, start="2026-09", slope=0.06):
    """FICTIONAL data drift: commercial consumption grows by `slope` every period."""
    b = PERIODS.index(start)
    return [dict(k, m3=round(k["m3"] * (1.0 + slope * (PERIODS.index(k["period"]) - b + 1)), 4))
            if PERIODS.index(k["period"]) >= b and k["type"] == "commercial" else dict(k)
            for k in records]



def concept_drift(feats, records, start="2026-09", new_threshold=27):
    """FICTIONAL concept drift: SAME INPUT , the rule changes (threshold 30 to 27)."""
    b = PERIODS.index(start)
    return {(k["subscriber"], k["period"]): label(
        feats[(k["subscriber"], k["period"])],
        new_threshold if PERIODS.index(k["period"]) >= b else RULE_THRESHOLD) for k in records}


def accuracy(records, feats, m, period, labels):
    subset = [k for k in records if k["period"] == period]
    return round(sum(1 for k in subset
                     if predict(feats[(k["subscriber"], k["period"])], m["weights"], m["decision"])
                     == labels[(k["subscriber"], k["period"])]) / len(subset), 4)


def prediction_share(records, feats, m, period):
    subset = [k for k in records if k["period"] == period]
    return round(sum(predict(feats[(k["subscriber"], k["period"])], m["weights"], m["decision"])
                     for k in subset) / len(subset), 4)


ALL = rows()
FEATS = features(ALL)
PRODUCTION = drift(ALL)                   # the input has drifted in production
FEATS_D = features(PRODUCTION)
LABELS_D = {c: label(o) for c, o in FEATS_D.items()}      # data drift: SAME RULE
LABELS_C = concept_drift(FEATS, ALL)                      # concept drift: the rule changed
BASELINE = train(ALL, FEATS, PERIODS[1:9], {c: label(o) for c, o in FEATS.items()})
monitored_rows = [k for k in ALL if k["period"] in MONITORED]
base_labels = [label(FEATS[(k["subscriber"], k["period"])]) for k in monitored_rows]
print(f"baseline model {BASELINE['weights']}/{BASELINE['decision']} balanced accuracy {BASELINE['balanced']}"
      f" recall {BASELINE['recall']:.4f} specificity {BASELINE['specificity']:.4f}")
print(f"monitoring {len(monitored_rows)} rows , base class ratio {round(1 - sum(base_labels) / len(base_labels), 4)}"
      f" , 200 rows per period , accuracy step 0.0050")

# Retraining in production with a data drift: the BEST case that could be reached
# if all labels were at hand , that is if there were no delay at all.
print("\n window  training periods   rows  balanced accuracy  2026-12 accuracy")
for p in (3, 6, 11):
    span = [d for d in PERIODS if PERIODS.index("2026-11") - p < PERIODS.index(d)
            <= PERIODS.index("2026-11")]
    m = train(PRODUCTION, FEATS_D, span, LABELS_D)
    print(f"{p:>7}  {span[0]}..{span[-1]}  {m['rows']:>5}  {m['balanced']:>17}"
          f"  {accuracy(PRODUCTION, FEATS_D, m, '2026-12', LABELS_D):>16}")
print(f"   old model                                 {BASELINE['balanced']}"
      f"  {accuracy(PRODUCTION, FEATS_D, BASELINE, '2026-12', LABELS_D):>16}")

baseline model (0.6, 0.0, 4)/18 balanced accuracy 0.9864 recall 1.0000 specificity 0.9728
monitoring 1200 rows , base class ratio 0.5342 , 200 rows per period , accuracy step 0.0050

 window  training periods   rows  balanced accuracy  2026-12 accuracy
      3  2026-09..2026-11    600             0.9824              0.98
      6  2026-06..2026-11   1200               0.98              0.98
     11  2026-01..2026-11   2000             0.9849              0.98
   old model                                 0.9864              0.98

As the training window grows, the balanced accuracy on training moves: 0.9824, 0.9800, 0.9849. These three numbers give the impression that a choice can be made. The last column erases that impression. All three windows give 0.9800 on 2026-12, and this is exactly the same as the number of the old model that was never retrained. All three have found the same weights and the same decision threshold, because the rule the search is trying to solve has not changed. The input has drifted, but as long as the relation that produces the label stays in place, a correct model stays correct on a drifted input as well. The accuracy gained is 0.0000 and the training cost has been paid.

Trigger or Window

The second block runs the three triggers in the two scenarios. The label delay is now applied: a retraining triggered in a period sees labeled data only up to two periods earlier. The fired column is the number of periods in which the trigger fired, that is, how many models were trained.

# This block continues the ALL, FEATS, PRODUCTION, FEATS_D, BASELINE, LABELS_D, LABELS_C,
# MONITORED, GRID, DECISION_GRID, train, accuracy and prediction_share definitions
# of the first block.
DELAY, WINDOW, CACHE = 2, 6, {}


def trigger(name, records, feats, labels, now, window=WINDOW):
    """The label is delayed: in period `now` only `now - DELAY` is labeled."""
    i = PERIODS.index(now) - DELAY
    span = [d for d in PERIODS if i - window < PERIODS.index(d) <= i]
    key = (name, span[0], span[-1], window)
    if key not in CACHE:
        CACHE[key] = train(records, feats, span, labels)
    m = CACHE[key]
    return m, m["rows"] * len(GRID) * len(DECISION_GRID)


def deviating(series, base, band=0.05):
    return [d for d, v in series if abs(v - base) / base > band]


print("scenario        trigger          fired      rows scored  2026-12  gain")
for name, recs, feats, lab in (("data drift", PRODUCTION, FEATS_D, LABELS_D),
                               ("concept drift", ALL, FEATS, LABELS_C)):
    base_12 = accuracy(recs, feats, BASELINE, "2026-12", lab)
    base_08 = accuracy(recs, feats, BASELINE, "2026-08", lab)
    for t, fired_periods in (
            ("calendar", list(MONITORED)),
            ("indicator", deviating([(d, prediction_share(recs, feats, BASELINE, d)) for d in MONITORED],
                                    prediction_share(recs, feats, BASELINE, "2026-08"))),
            ("performance", deviating([(d, accuracy(recs, feats, BASELINE,
                                                    PERIODS[PERIODS.index(d) - DELAY], lab))
                                       for d in MONITORED[DELAY:]], base_08))):
        cost, last_model = 0, None
        for d in fired_periods:
            m, b = trigger(name, recs, feats, lab, d)
            cost += b
            last_model = m if PERIODS.index(d) < PERIODS.index("2026-12") else last_model
        acc_12 = accuracy(recs, feats, last_model, "2026-12", lab) if last_model else base_12
        print(f"{name:<15} {t:<11}  {len(fired_periods):>9}  {cost:>15}  {acc_12:>7}  {acc_12 - base_12:+.4f}")

print("\nconcept drift , retraining triggered in 2026-11: window choice")
for p in (1, 3, 6):
    m, _ = trigger("cw", ALL, FEATS, LABELS_C, "2026-11", window=p)
    acc_12 = accuracy(ALL, FEATS, m, "2026-12", LABELS_C)
    print(f"  window {p}  {str(m['weights']) + '/' + str(m['decision']):<18}"
          f" training rows {m['rows']:>4}  2026-12 accuracy {acc_12}"
          f"  gain {acc_12 - accuracy(ALL, FEATS, BASELINE, '2026-12', LABELS_C):+.4f}")

scenario        trigger          fired      rows scored  2026-12  gain
data drift      calendar             6           891000     0.98  +0.0000
data drift      indicator            3           486000     0.98  +0.0000
data drift      performance          0                0     0.98  +0.0000
concept drift   calendar             6           891000     0.85  +0.0000
concept drift   indicator            3           486000     0.85  +0.0000
concept drift   performance          2           324000     0.85  +0.0000

concept drift , retraining triggered in 2026-11: window choice
  window 1  (0.6, 0.2, 4)/22   training rows  200  2026-12 accuracy 0.96  gain +0.1100
  window 3  (0.8, 0.4, 4)/34   training rows  600  2026-12 accuracy 0.85  gain +0.0000
  window 6  (0.6, 0.0, 4)/18   training rows 1200  2026-12 accuracy 0.85  gain +0.0000

In all six of six rows the gain is 0.0000. A total of 3,078,000 rows have been scored, retraining has run twenty times, and the number read on 2026-12 has come out every time as the number of the case without a trigger. Reading the rows one by one shows that there are three separate reasons for this.

The performance trigger never fires in the data drift and this is the correct decision: there is nothing to gain, and its cost is 0. The same trigger fires in the concept drift but two periods late; the accuracy that dropped in 2026-09 is known only in 2026-11. The indicator trigger fires in the same three periods in both scenarios. It does not tell the scenario with harmful degradation from the harmless one, because the prediction share it looks at drifts in the same way in both. The rule of the course is written on this row: an indicator’s firing is not evidence of degradation. The calendar trigger looks at nothing, trains the most models and pays the largest cost.

The last table gives the real reason. In the concept drift, the retraining triggered in 2026-11, at the same moment, with the same data and the same grid, gives 0.8500 with window 6 and 0.9600 with window 1; the gain varies between 0.0000 and 0.1100. Window 6 sees together the periods of which one belongs to the new rule and five to the old rule, and the search that maximizes balanced accuracy follows the majority and finds the old model again. Window 1 sees only the period of the new rule and lands on another model. What retraining gains is not the trigger but which rule’s periods the window contains. The gain of 0.1100 is far above the minimum detectable effect of the previous lesson; 0.0000 is zero anyway.

The measurable counterpart of the approval flow comes from here. When a retraining proposal comes up for approval, the sentence “the indicator fired” alone is not a justification; the three numbers that make the decision possible are its window, the number of predictions on which the candidate disagrees with the old model and its gain measured in the shadow. When the three are not written, the approval also approves a training whose result is 0.0000.

Summary

• In production with a data drift, three retraining windows give 0.9824, 0.9800 and 0.9849 in training, but on 2026-12 all three are 0.9800 — exactly the same as the old model that was never retrained. The base class ratio is 0.5342, the step of accuracy is 0.0050. If the rule has not changed, retraining gains nothing and the training cost has been paid.
• In all six of six combinations of three triggers and two scenarios the gain is 0.0000; in return 3,078,000 rows are scored. The calendar trigger is the most expensive (891,000 rows), and the performance trigger never fires in the data drift and its cost is 0.
• The indicator trigger fires in the same three periods in both scenarios; it does not tell harmful degradation from the harmless one. An indicator’s firing is not evidence of degradation.
• The performance trigger sees the concept drift but sees it two periods late; the accuracy that dropped in 2026-09 triggers in 2026-11.
• What determines the gain is not the trigger but the window: the retraining triggered at the same moment gives +0.0000 with window 6 and +0.1100 with window 1. A window containing the periods of the old rule adapts the search to the majority and finds the old model again.

Next Step

Over three lessons the model was measured in the shadow, given to the canary, compared by experiment and retrained. At every step a new model, a new window and a new decision threshold came out. The only thing not asked is to whom a single prediction given to a subscriber on 2026-12 will be put months later: which model produced it, with which feature values, with which decision threshold. The next lesson counts this question as a search — how many rows are read to find the reason for a prediction with and without a decision record, how many predictions in the canary period cannot be explained by any reading, and how many fields the record itself costs to store.