Data Drift

Sixty-one lessons measured the road up to the model reaching production: the rows the pipeline carries were counted, the place where a row would live was chosen, the work was divided, the run was pinned, the feature was read from a single place, and the model was packaged and served. The number at the end of that road belongs to the moment of release and says nothing once that moment has passed.

What comes after release was never asked. The world changes under the model: the input distribution drifts, the rule between input and label changes, the true label arrives weeks later, and one day the model is retired. This course measures that time, but its first job is a correction. An unmonitorable model cannot be monitored: the model the MLOps Fundamentals course left behind cannot show any degradation in production, and there are three independent reasons for this.

• MO1. The source, the first three stages of the pipeline that look at the model, the threshold-based model and the drift scenario are fictional; no real learning library, monitoring product or cloud service is called. The seed is 20260218.
• MO2. Drift is production data moving away from the training data: data drift is a change of the input distribution, concept drift is a change of the rule between input and label. The measurement procedure was built in the Model Evaluation, Interpretability and Ethics course and is not repeated; this course adds only the life in production.
• MO3. The monitoring periods are 2026-07…2026-12, 200 rows per period. The training window is 2026-02…2026-09; the first three monitoring periods overlap with it, while the three periods in which the drift is measured lie outside it.
• MO4. This course’s label is the rule subscriber_mean + 6 x commercial > 30, and this rule is the CONCEPT. Data drift does not change it. In the monitoring periods the base class ratio is 0.5342; the set is balanced, and accuracy can be read meaningfully here.
• MO5. The decision threshold is the setting that turns a score into a prediction, and it is separate from the label threshold. Balanced accuracy is the mean of recall and specificity.
• MO6. The baseline is the unmonitored setup: no indicator, no alert; the degradation goes unseen. The alert threshold is written as a two-sided band.
• MO7. Resolution: the step of period accuracy is 1/200 = 0.0050; in this lesson the total swing of accuracy is 0.0200, that is, four steps.

An Unmonitorable Model Cannot Be Monitored

The first block builds the source, the three stages of the pipeline that look at the model, the features and the fictional model. The three fixes are marked by name inside the code: the decision threshold is a separate setting and is searched together with the weights in DECISION_GRID, the training criterion is not accuracy but balanced accuracy, and the label is tied to a rule the features see.

# FICTIONAL. The meter readings of the measurement network, the first three stages
# of the pipeline that look at the model, the threshold-based model and the drift
# scenario are defined inside the lesson; no real learning library, monitoring
# product or cloud service is called.
SEED, M32 = 20260218, 0xFFFFFFFF


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


REGION = ["east", "west", "north", "south", "central"]
TYPE = ["residential", "commercial"]
PERIODS = [f"2026-{a:02d}" for a in range(1, 13)]
SUBSCRIBERS = {}
for i in range(200):
    r = rng(SEED + 37 * i)
    SUBSCRIBERS[f"A{i + 1:03d}"] = {"region": REGION[int(r() * 5)], "type": TYPE[int(r() * 2)],
                                    "start": 10000 + int(r() * 60000)}


def generate_source():
    records = []
    for i, (sub, a) in enumerate(SUBSCRIBERS.items()):
        e = a["start"]
        for p in PERIODS:
            r = rng(SEED + 101 * i + 7 * PERIODS.index(p))
            e += 4 + int(r() * 46)
            records.append({"subscriber": sub, "period": p, "region": a["region"], "type": a["type"],
                            "index": e, "unit": "m3"})
    return records


def s1_screen(records):
    REQUIRED = ("subscriber", "period", "region", "index", "unit")
    keep = [k for k in records if all(a in k for a in REQUIRED)]
    return keep, len(records) - len(keep)


def s2_dedupe(records):
    seen, keep = set(), []
    for k in records:
        key = (k["subscriber"], k["period"])
        if key in seen:
            continue
        seen.add(key)
        keep.append(k)
    return keep, len(records) - len(keep)


def s3_consumption(records):
    """Consumption from the period difference. Without a previous period the row cannot be computed."""
    table = {(k["subscriber"], k["period"]): k for k in records}
    out, dropped = [], 0
    for k in records:
        i = PERIODS.index(k["period"])
        if i == 0:
            continue
        prev = table.get((k["subscriber"], PERIODS[i - 1]))
        if prev is None:
            dropped += 1
            continue
        out.append(dict(k, m3=int(k["index"]) - int(prev["index"])))
    return out, dropped


def rows():
    v, _ = s1_screen(generate_source())
    v, _ = s2_dedupe(v)
    v, _ = s3_consumption(v)
    return v


def features(records, digits=4):
    """Features are independent of the weights; computed ONCE from the full history."""
    ordered = sorted(records, key=lambda k: PERIODS.index(k["period"]))
    sub, reg = {}, {}
    for k in ordered:
        sub.setdefault(k["subscriber"], []).append((PERIODS.index(k["period"]), k["m3"]))
        reg.setdefault(k["region"], []).append((PERIODS.index(k["period"]), k["m3"]))

    def mean(series, i):
        chosen = [v for d, v in series if d < i]
        return round(sum(chosen) / len(chosen), digits) if chosen else 0.0

    return {(k["subscriber"], k["period"]): {
        "subscriber_mean": mean(sub[k["subscriber"]], PERIODS.index(k["period"])),
        "region_mean": mean(reg[k["region"]], PERIODS.index(k["period"])),
        "commercial": 1 if k["type"] == "commercial" else 0} for k in records}


GRID = [(a, b, c) for a in (0.6, 0.8, 1.0) for b in (0.0, 0.2, 0.4) for c in (0, 4, 8)]


def score(o, w):
    return w[0] * o["subscriber_mean"] + w[1] * o["region_mean"] + w[2] * o["commercial"]


# ---- FIX 3: the label is tied to a rule the features SEE. The rule is the CONCEPT.
RULE_THRESHOLD = 30
RULE_COMMERCIAL = 6


def label(feat_k, threshold=RULE_THRESHOLD, commercial_factor=RULE_COMMERCIAL):
    return 1 if feat_k["subscriber_mean"] + commercial_factor * feat_k["commercial"] > threshold else 0


MONITORED = PERIODS[6:]               # 2026-07 .. 2026-12 : the periods monitored in production
# ---- FIX 1: the decision threshold is a SEPARATE setting from the label threshold and is searched in training.
DECISION_GRID = (18, 22, 26, 30, 34)


def predict(feat_k, w, decision):
    return 1 if score(feat_k, w) > decision else 0


# ---- FIX 2: the training criterion is not ACCURACY but BALANCED ACCURACY.
# rule=None -> this course's label; passing a rule from outside serves to measure the old setup.
def balanced_accuracy(subset, feats, w, decision, rule=None):
    c = {(0, 0): 0, (0, 1): 0, (1, 0): 0, (1, 1): 0}
    for k in subset:
        f_k = feats[(k["subscriber"], k["period"])]
        c[(predict(f_k, w, decision), rule(k, f_k) if rule else label(f_k))] += 1
    rec = c[(1, 1)] / max(c[(1, 1)] + c[(0, 1)], 1)      # recall
    spec = c[(0, 0)] / max(c[(0, 0)] + c[(1, 0)], 1)     # specificity
    return {"accuracy": (c[(1, 1)] + c[(0, 0)]) / len(subset), "recall": rec, "specificity": spec,
            "balanced": (rec + spec) / 2, "weights": w, "decision": decision}


def train(records, feats, periods=None, rule=None):
    """Training = searching the weight TRIPLE together with the decision threshold: 27 x 5 = 135 candidates."""
    subset = [k for k in records if k["period"] in (periods or PERIODS[1:9])]
    return max((balanced_accuracy(subset, feats, w, decision, rule)
                for w in GRID for decision in DECISION_GRID), key=lambda o: o["balanced"])


S = rows()
FEATS = features(S)
MON = [k for k in S if k["period"] in MONITORED]
dist = {0: 0, 1: 0}
for k in MON:
    dist[label(FEATS[(k["subscriber"], k["period"])])] += 1
print("rows:", len(S), " monitoring periods:", len(MONITORED), " monitoring rows:", len(MON),
      " per period:", len(MON) // len(MONITORED))
print("monitoring label distribution:", dist, " base class ratio:", round(dist[0] / len(MON), 4))

rows: 2200  monitoring periods: 6  monitoring rows: 1200  per period: 200
monitoring label distribution: {0: 641, 1: 559}  base class ratio: 0.5342

The base class ratio is 0.5342; in the MLOps Fundamentals course it was 0.7964, and accuracy read on its own was misleading. The second block counts what the fixes bought: the upper table compares the two setups on the same test periods, the lower one runs the same search with two different label rules.

TEST_ROWS = [k for k in S if k["period"] in PERIODS[9:]]
MODEL = train(S, FEATS)


def old_rule(k, f_k):
    return 1 if k["m3"] > 40 else 0


HEADER = (f"{'setup':<46}{'weights / decision':>22}{'accuracy':>9}"
          f"{'recall':>12}{'specificity':>13}{'balanced':>9}")


def show(name, o):
    print(f"{name:<46}{str(o['weights']) + ' / ' + str(o['decision']):>22}{o['accuracy']:>9.4f}"
          f"{o['recall']:>12.4f}{o['specificity']:>13.4f}{o['balanced']:>9.4f}")


print("on the test periods (2026-10 .. 2026-12):\n" + HEADER)
show("old         : accuracy criterion", balanced_accuracy(TEST_ROWS, FEATS, (0.6, 0.4, 0), 40, old_rule))
show("this course : balanced accuracy criterion", balanced_accuracy(TEST_ROWS, FEATS, MODEL["weights"], MODEL["decision"]))
print("\nthe best of the same 135-candidate search, on the training periods:\n" + HEADER)
show("label m3 > 40", train(S, FEATS, rule=old_rule))
show("label subscriber_mean + 6 x commercial > 30", MODEL)

on the test periods (2026-10 .. 2026-12):
setup                                             weights / decision accuracy      recall  specificity balanced
old         : accuracy criterion                  (0.6, 0.4, 0) / 40   0.8067      0.0000       1.0000   0.5000
this course : balanced accuracy criterion         (0.6, 0.0, 4) / 18   0.9833      1.0000       0.9670   0.9835

the best of the same 135-candidate search, on the training periods:
setup                                             weights / decision accuracy      recall  specificity balanced
label m3 > 40                                     (0.6, 0.2, 4) / 22   0.5075      0.5422       0.4984   0.5203
label subscriber_mean + 6 x commercial > 30       (0.6, 0.0, 4) / 18   0.9838      1.0000       0.9728   0.9864

The 0.8067 in the upper table is not wrong; what it measures is something else. When the decision threshold is kept the same as the label threshold and the criterion is accuracy, the search stops at the model that says high for no row at all on the imbalanced set: recall 0.0000, balanced accuracy 0.5000, that is, coin-flip level. The indicator of such a model does not move even if it degrades in production, because its output is already constant. In the corrected setup recall is 1.0000, specificity 0.9670 and balanced accuracy 0.9835.

The lower table says why the third fix is needed. Even with the first two fixes in place, the best of the 135 candidates stays at 0.5203 with the old label rule. The reason is structural: that rule looks at the consumption of the row in its own period, while the features see only the previous periods. There is no relation, so there is nothing to learn. Once the label is tied to a rule the features see, the same search gives 0.9864.

What Happens While the Input Drifts

In data drift the input distribution changes and the rule stays the same. The fictional scenario is this: after 2026-09, the consumption of commercial subscribers grows by a factor of 0.06 every period. The third block applies the drift and reads three indicators period by period: the input summary (the mean of the feature), the prediction share (the share of rows the model calls high) and accuracy. The last column is the accuracy the same model gives on undrifted data, that is, the no-drift run.

# ---- FICTIONAL data drift: after 2026-09 commercial consumption grows by 0.06 every period.
# THE RULE STAYS THE SAME; only the input distribution changes.
def drift(records, start="2026-09", slope=0.06):
    b = PERIODS.index(start)
    return [dict(k, m3=round(k["m3"] * (1.0 + slope * (PERIODS.index(k["period"]) - b + 1)), 4))
            if PERIODS.index(k["period"]) >= b and k["type"] == "commercial" else dict(k)
            for k in records]


def input_summary(records, period, feats):
    d = [feats[(k["subscriber"], k["period"])]["subscriber_mean"] for k in records if k["period"] == period]
    return round(sum(d) / len(d), 4)


def prediction_share(records, model, feats, period):
    d = [predict(feats[(k["subscriber"], k["period"])], model["weights"], model["decision"])
         for k in records if k["period"] == period]
    return round(sum(d) / len(d), 4)


def accuracy(records, model, feats, period, labels=None):
    """labels=None -> the rule has not changed."""
    subset = [k for k in records if k["period"] == period]
    d = 0
    for k in subset:
        f_k = feats[(k["subscriber"], k["period"])]
        g = labels[(k["subscriber"], k["period"])] if labels else label(f_k)
        d += predict(f_k, model["weights"], model["decision"]) == g
    return round(d / len(subset), 4)


DRIFTED = drift(S)
FEATS_D = features(DRIFTED)
MEASURES = {"input summary": [(d, input_summary(DRIFTED, d, FEATS_D)) for d in MONITORED],
            "prediction share": [(d, prediction_share(DRIFTED, MODEL, FEATS_D, d)) for d in MONITORED],
            "accuracy": [(d, accuracy(DRIFTED, MODEL, FEATS_D, d)) for d in MONITORED]}
print(f"{'period':<10}{'input summary':>15}{'prediction share':>18}{'accuracy':>10}{'no-drift accuracy':>19}")
for i, d in enumerate(MONITORED):
    print(f"{d:<10}{MEASURES['input summary'][i][1]:>15.4f}{MEASURES['prediction share'][i][1]:>18.4f}"
          f"{MEASURES['accuracy'][i][1]:>10.4f}{accuracy(S, MODEL, FEATS, d):>19.4f}")

period      input summary  prediction share  accuracy  no-drift accuracy
2026-07           25.9220            0.4550    0.9700             0.9700
2026-08           26.0517            0.4700    0.9700             0.9700
2026-09           25.9800            0.4550    0.9900             0.9900
2026-10           26.4685            0.5150    0.9750             0.9750
2026-11           26.6053            0.5200    0.9800             0.9850
2026-12           27.0472            0.5450    0.9800             0.9900

The input really does drift: the input summary rises from 25.9220 to 27.0472 and the prediction share from 0.4550 to 0.5450, so the share of inferences that say high grows by about twenty percent. Accuracy, in contrast, never falls below the 0.9700 baseline in any period. What the drift actually took away is measured by the no-drift column: in 2026-12 the no-drift run gives 0.9900 and the drifted run 0.9800; the difference is 0.0100, two resolution steps. As long as the rule does not change, a correct model stays correct on drifted input, because both the label and the prediction are computed from the same drifted features.

A Firing Alert Is Not Evidence

The fourth block attaches an alert band to the indicators. The band is the ratio of the deviation from the baseline; the baseline is 2026-08, the last period before the one in which the drift starts. The same three bands are applied to all three indicators in the same way.

# ---- Alert: a measure fires if it deviates from the baseline by more than `share`. The band is two-sided.
def alert(series, baseline, share):
    return [d for d, v in series if abs(v - baseline) / max(abs(baseline), 1e-9) > share]


BASELINE_PERIOD = MONITORED[1]   # 2026-08 : the last period before the one in which the drift starts
ACCURACY = dict(MEASURES["accuracy"])
print(f"{'measure':<18}{'baseline':>10}{'largest deviation':>19}{'0.05':>7}{'0.10':>7}"
      f"{'0.20':>7}{'first firing':>16}")
for name, series in MEASURES.items():
    base = dict(series)[BASELINE_PERIOD]
    fires = [alert(series, base, p) for p in (0.05, 0.10, 0.20)]
    deviation = max(abs(v - base) / base for _, v in series)
    print(f"{name:<18}{base:>10.4f}{deviation:>19.4f}{len(fires[0]):>7}{len(fires[1]):>7}"
          f"{len(fires[2]):>7}{(fires[0][0] if fires[0] else '-'):>16}")

FIRED = alert(MEASURES["prediction share"], dict(MEASURES["prediction share"])[BASELINE_PERIOD], 0.05)
REAL = [d for d in FIRED if ACCURACY[d] < ACCURACY[BASELINE_PERIOD]]
print("\nperiods firing in the 0.05 band:", FIRED)
print("how many of them had accuracy fall below the baseline:", len(REAL),
      " false positive:", len(FIRED) - len(REAL))
print("total swing of accuracy:", f"{max(ACCURACY.values()) - min(ACCURACY.values()):.4f}",
      " step of period accuracy:", f"{1 / 200:.4f}",
      " what the drift cost in 2026-12:",
      f"{accuracy(S, MODEL, FEATS, MONITORED[-1]) - ACCURACY[MONITORED[-1]]:.4f}")

measure             baseline  largest deviation   0.05   0.10   0.20    first firing
input summary        26.0517             0.0382      0      0      0               -
prediction share      0.4700             0.1596      3      2      0         2026-10
accuracy              0.9700             0.0206      0      0      0               -

periods firing in the 0.05 band: ['2026-10', '2026-11', '2026-12']
how many of them had accuracy fall below the baseline: 0  false positive: 3
total swing of accuracy: 0.0200  step of period accuracy: 0.0050  what the drift cost in 2026-12: 0.0100

The same drift, the same band, three different answers. The largest deviation of the input summary is 0.0382 and it fires in no band, because it is a mean and a mean smooths out the change in individual rows. The deviation of the prediction share is 0.1596; it fires in three periods in the 0.05 band and in two in the 0.10 band, because it is sensitive to the rows around the decision threshold. The deviation of accuracy is 0.0206, it does not fire, and moreover it is in the upward direction.

The rule of the course can be read from here. The alert in 2026-10 did not work wrongly: the input really has drifted. But the only thing it says is that the input has drifted; it does not say that anything has degraded. In none of the three firing periods did accuracy fall below the baseline, and all three count as false positives. The firing of an indicator is not evidence of degradation; an indicator whose firing and whose degradation are not shown separately counts as unmeasured. The unmonitored baseline lands in a curious position here: the setup that attaches no indicator has never been wrong, because it has no alert that could be wrong.

Summary

• A monitorable model depends on three settings: the decision threshold is a setting separate from the label threshold and is searched in training, the training criterion is balanced accuracy, and the label is tied to a rule the features see.
• In the old setup recall is 0.0000 and balanced accuracy 0.5000; in the corrected one recall is 1.0000, specificity 0.9670, balanced accuracy 0.9835 and the base class ratio 0.5342.
• The first two fixes alone are not enough: with the old label rule the best of the 135 candidates stays at 0.5203, coin-flip level, because that rule looks at a number the features do not see.
• Under data drift the input summary rises from 25.9220 to 27.0472 and the prediction share from 0.4550 to 0.5450, but accuracy goes from 0.9700 to 0.9800; what the drift took away is 0.0100 in 2026-12, two resolution steps.
• The prediction share fires in three periods in the 0.05 band and in none of the three did accuracy fall below the baseline: a firing alert is not evidence of a problem.

Next Step

In this lesson what drifted was the input, and because the rule stayed in place the model could stay correct; the firing indicator saw exactly this. But what if it were not the input that changed but the rule itself, that is, if the same features began to produce a different label: what would the same three indicators say. The input summary would not move, because the input has not moved; the prediction share would not move either, because the model keeps giving the same answer to the same inputs. The next lesson measures this case and finds the opposite result: while the indicators stay silent, accuracy falls from 0.9700 to 0.8150.