--- title: 'Data Drift' source: 'https://academia.sh/en/courses/model-monitoring/data-drift' course: 'Model Monitoring and Governance' language: en updated: '2026-10-21T05:03:02+00:00' license: 'CC BY-SA 4.0' objectives: - 'Separates data drift from concept drift and measures the input summary, the prediction share and accuracy side by side in the same production run.' - 'Lists the three conditions of a monitorable model: the decision threshold being a separate setting, the training criterion being balanced accuracy, and the label being tied to a rule the features see.' - 'Shows that a firing alert is not evidence of degradation, and compares the number of firing periods with the number of periods that correspond to real degradation.' --- # Data Drift Measuring the change of the input distribution in production and building the three conditions of a monitorable model: when the decision threshold is searched as a setting separate from the label threshold, when the training criterion is switched from accuracy to balanced accuracy, and when the label is tied to a rule the features see, recall rises from 0.0000 to 1.0000, because the same 135-candidate search stays at 0.5203 balanced accuracy with the old label, which is coin-flip level. In the corrected setup the input drifts period by period, the input summary rises from 25.9220 to 27.0472, the prediction share goes from 0.4550 to 0.5450 and the alert fires in three periods in the 0.05 band, yet accuracy never falls below the 0.9700 baseline; all three firings are false positives, and what the drift actually took away, compared with the no-drift run, is 0.0100 in 2026-12, that is two resolution steps. Sixty-one lessons measured the road up to the model **reaching production**: the rows the pipeline carries were counted, the place where a row would live was chosen, the work was divided, the run was pinned, the feature was read from a single place, and the model was packaged and served. The number at the end of that road belongs to **the moment of release** and says nothing once that moment has passed. What comes after release was never asked. The world changes under the model: the input distribution drifts, the rule between input and label changes, the true label arrives weeks later, and one day the model is retired. This course measures that time, but its first job is a correction. **An unmonitorable model cannot be monitored:** the model the MLOps Fundamentals course left behind cannot show any degradation in production, and there are three independent reasons for this. - **MO1.** The source, the first three stages of the pipeline that look at the model, the threshold-based model and the drift scenario are **fictional**; no real learning library, monitoring product or cloud service is called. The seed is **20260218**. - **MO2.** **Drift** is production data moving away from the training data: **data drift** is a change of the input distribution, **concept drift** is a change of the rule between input and label. The measurement procedure was built in the Model Evaluation, Interpretability and Ethics course and is **not repeated**; this course adds only the **life in production**. - **MO3.** The monitoring periods are **2026-07…2026-12**, **200 rows** per period. The training window is **2026-02…2026-09**; the first three monitoring periods overlap with it, while **the three periods in which the drift is measured lie outside it**. - **MO4.** This course's label is the rule `subscriber_mean + 6 x commercial > 30`, and **this rule is the CONCEPT**. Data drift does not change it. In the monitoring periods the **base class ratio is 0.5342**; the set is balanced, and accuracy can be read meaningfully here. - **MO5.** The **decision threshold** is the setting that turns a score into a prediction, and it is separate from the label threshold. **Balanced accuracy** is the mean of **recall** and **specificity**. - **MO6.** The baseline is the **unmonitored setup**: no indicator, no alert; the degradation goes **unseen**. The **alert threshold** is written as a two-sided **band**. - **MO7.** Resolution: the step of period accuracy is **1/200 = 0.0050**; in this lesson the total swing of accuracy is **0.0200**, that is, four steps. ## An Unmonitorable Model Cannot Be Monitored The first block builds the source, the three stages of the pipeline that look at the model, the features and the fictional model. The three fixes are marked by name inside the code: **the decision threshold is a separate setting** and is searched together with the weights in `DECISION_GRID`, the training criterion is not accuracy but **balanced accuracy**, and the label is tied to a rule the features **see**. ```python # FICTIONAL. The meter readings of the measurement network, the first three stages # of the pipeline that look at the model, the threshold-based model and the drift # scenario are defined inside the lesson; no real learning library, monitoring # product or cloud service is called. SEED, M32 = 20260218, 0xFFFFFFFF def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value REGION = ["east", "west", "north", "south", "central"] TYPE = ["residential", "commercial"] PERIODS = [f"2026-{a:02d}" for a in range(1, 13)] SUBSCRIBERS = {} for i in range(200): r = rng(SEED + 37 * i) SUBSCRIBERS[f"A{i + 1:03d}"] = {"region": REGION[int(r() * 5)], "type": TYPE[int(r() * 2)], "start": 10000 + int(r() * 60000)} def generate_source(): records = [] for i, (sub, a) in enumerate(SUBSCRIBERS.items()): e = a["start"] for p in PERIODS: r = rng(SEED + 101 * i + 7 * PERIODS.index(p)) e += 4 + int(r() * 46) records.append({"subscriber": sub, "period": p, "region": a["region"], "type": a["type"], "index": e, "unit": "m3"}) return records def s1_screen(records): REQUIRED = ("subscriber", "period", "region", "index", "unit") keep = [k for k in records if all(a in k for a in REQUIRED)] return keep, len(records) - len(keep) def s2_dedupe(records): seen, keep = set(), [] for k in records: key = (k["subscriber"], k["period"]) if key in seen: continue seen.add(key) keep.append(k) return keep, len(records) - len(keep) def s3_consumption(records): """Consumption from the period difference. Without a previous period the row cannot be computed.""" table = {(k["subscriber"], k["period"]): k for k in records} out, dropped = [], 0 for k in records: i = PERIODS.index(k["period"]) if i == 0: continue prev = table.get((k["subscriber"], PERIODS[i - 1])) if prev is None: dropped += 1 continue out.append(dict(k, m3=int(k["index"]) - int(prev["index"]))) return out, dropped def rows(): v, _ = s1_screen(generate_source()) v, _ = s2_dedupe(v) v, _ = s3_consumption(v) return v def features(records, digits=4): """Features are independent of the weights; computed ONCE from the full history.""" ordered = sorted(records, key=lambda k: PERIODS.index(k["period"])) sub, reg = {}, {} for k in ordered: sub.setdefault(k["subscriber"], []).append((PERIODS.index(k["period"]), k["m3"])) reg.setdefault(k["region"], []).append((PERIODS.index(k["period"]), k["m3"])) def mean(series, i): chosen = [v for d, v in series if d < i] return round(sum(chosen) / len(chosen), digits) if chosen else 0.0 return {(k["subscriber"], k["period"]): { "subscriber_mean": mean(sub[k["subscriber"]], PERIODS.index(k["period"])), "region_mean": mean(reg[k["region"]], PERIODS.index(k["period"])), "commercial": 1 if k["type"] == "commercial" else 0} for k in records} GRID = [(a, b, c) for a in (0.6, 0.8, 1.0) for b in (0.0, 0.2, 0.4) for c in (0, 4, 8)] def score(o, w): return w[0] * o["subscriber_mean"] + w[1] * o["region_mean"] + w[2] * o["commercial"] # ---- FIX 3: the label is tied to a rule the features SEE. The rule is the CONCEPT. RULE_THRESHOLD = 30 RULE_COMMERCIAL = 6 def label(feat_k, threshold=RULE_THRESHOLD, commercial_factor=RULE_COMMERCIAL): return 1 if feat_k["subscriber_mean"] + commercial_factor * feat_k["commercial"] > threshold else 0 MONITORED = PERIODS[6:] # 2026-07 .. 2026-12 : the periods monitored in production # ---- FIX 1: the decision threshold is a SEPARATE setting from the label threshold and is searched in training. DECISION_GRID = (18, 22, 26, 30, 34) def predict(feat_k, w, decision): return 1 if score(feat_k, w) > decision else 0 # ---- FIX 2: the training criterion is not ACCURACY but BALANCED ACCURACY. # rule=None -> this course's label; passing a rule from outside serves to measure the old setup. def balanced_accuracy(subset, feats, w, decision, rule=None): c = {(0, 0): 0, (0, 1): 0, (1, 0): 0, (1, 1): 0} for k in subset: f_k = feats[(k["subscriber"], k["period"])] c[(predict(f_k, w, decision), rule(k, f_k) if rule else label(f_k))] += 1 rec = c[(1, 1)] / max(c[(1, 1)] + c[(0, 1)], 1) # recall spec = c[(0, 0)] / max(c[(0, 0)] + c[(1, 0)], 1) # specificity return {"accuracy": (c[(1, 1)] + c[(0, 0)]) / len(subset), "recall": rec, "specificity": spec, "balanced": (rec + spec) / 2, "weights": w, "decision": decision} def train(records, feats, periods=None, rule=None): """Training = searching the weight TRIPLE together with the decision threshold: 27 x 5 = 135 candidates.""" subset = [k for k in records if k["period"] in (periods or PERIODS[1:9])] return max((balanced_accuracy(subset, feats, w, decision, rule) for w in GRID for decision in DECISION_GRID), key=lambda o: o["balanced"]) S = rows() FEATS = features(S) MON = [k for k in S if k["period"] in MONITORED] dist = {0: 0, 1: 0} for k in MON: dist[label(FEATS[(k["subscriber"], k["period"])])] += 1 print("rows:", len(S), " monitoring periods:", len(MONITORED), " monitoring rows:", len(MON), " per period:", len(MON) // len(MONITORED)) print("monitoring label distribution:", dist, " base class ratio:", round(dist[0] / len(MON), 4)) ``` ``` rows: 2200 monitoring periods: 6 monitoring rows: 1200 per period: 200 monitoring label distribution: {0: 641, 1: 559} base class ratio: 0.5342 ``` The base class ratio is **0.5342**; in the MLOps Fundamentals course it was 0.7964, and accuracy read on its own was misleading. The second block counts what the fixes bought: the upper table compares the two setups **on the same test periods**, the lower one runs **the same search** with two different label rules. ```python TEST_ROWS = [k for k in S if k["period"] in PERIODS[9:]] MODEL = train(S, FEATS) def old_rule(k, f_k): return 1 if k["m3"] > 40 else 0 HEADER = (f"{'setup':<46}{'weights / decision':>22}{'accuracy':>9}" f"{'recall':>12}{'specificity':>13}{'balanced':>9}") def show(name, o): print(f"{name:<46}{str(o['weights']) + ' / ' + str(o['decision']):>22}{o['accuracy']:>9.4f}" f"{o['recall']:>12.4f}{o['specificity']:>13.4f}{o['balanced']:>9.4f}") print("on the test periods (2026-10 .. 2026-12):\n" + HEADER) show("old : accuracy criterion", balanced_accuracy(TEST_ROWS, FEATS, (0.6, 0.4, 0), 40, old_rule)) show("this course : balanced accuracy criterion", balanced_accuracy(TEST_ROWS, FEATS, MODEL["weights"], MODEL["decision"])) print("\nthe best of the same 135-candidate search, on the training periods:\n" + HEADER) show("label m3 > 40", train(S, FEATS, rule=old_rule)) show("label subscriber_mean + 6 x commercial > 30", MODEL) ``` ``` on the test periods (2026-10 .. 2026-12): setup weights / decision accuracy recall specificity balanced old : accuracy criterion (0.6, 0.4, 0) / 40 0.8067 0.0000 1.0000 0.5000 this course : balanced accuracy criterion (0.6, 0.0, 4) / 18 0.9833 1.0000 0.9670 0.9835 the best of the same 135-candidate search, on the training periods: setup weights / decision accuracy recall specificity balanced label m3 > 40 (0.6, 0.2, 4) / 22 0.5075 0.5422 0.4984 0.5203 label subscriber_mean + 6 x commercial > 30 (0.6, 0.0, 4) / 18 0.9838 1.0000 0.9728 0.9864 ``` The 0.8067 in the upper table is not wrong; what it measures is something else. When the decision threshold is kept the same as the label threshold and the criterion is accuracy, the search stops at the model that **says high for no row at all** on the imbalanced set: **recall 0.0000**, balanced accuracy **0.5000**, that is, coin-flip level. The indicator of such a model does not move even if it degrades in production, because its output is already constant. In the corrected setup recall is **1.0000**, specificity **0.9670** and balanced accuracy **0.9835**. The lower table says why the third fix is needed. Even with the first two fixes in place, **the best of the 135 candidates stays at 0.5203** with the old label rule. The reason is structural: that rule looks at the consumption of the row in **its own period**, while the features see **only the previous periods**. There is no relation, so there is nothing to learn. Once the label is tied to a rule the features see, the same search gives **0.9864**. ## What Happens While the Input Drifts In data drift the input distribution changes and **the rule stays the same**. The fictional scenario is this: after `2026-09`, the consumption of commercial subscribers grows by a factor of 0.06 every period. The third block applies the drift and reads three indicators period by period: the **input summary** (the mean of the feature), the **prediction share** (the share of rows the model calls high) and **accuracy**. The last column is the accuracy the same model gives on **undrifted** data, that is, the no-drift run. ```python # ---- FICTIONAL data drift: after 2026-09 commercial consumption grows by 0.06 every period. # THE RULE STAYS THE SAME; only the input distribution changes. def drift(records, start="2026-09", slope=0.06): b = PERIODS.index(start) return [dict(k, m3=round(k["m3"] * (1.0 + slope * (PERIODS.index(k["period"]) - b + 1)), 4)) if PERIODS.index(k["period"]) >= b and k["type"] == "commercial" else dict(k) for k in records] def input_summary(records, period, feats): d = [feats[(k["subscriber"], k["period"])]["subscriber_mean"] for k in records if k["period"] == period] return round(sum(d) / len(d), 4) def prediction_share(records, model, feats, period): d = [predict(feats[(k["subscriber"], k["period"])], model["weights"], model["decision"]) for k in records if k["period"] == period] return round(sum(d) / len(d), 4) def accuracy(records, model, feats, period, labels=None): """labels=None -> the rule has not changed.""" subset = [k for k in records if k["period"] == period] d = 0 for k in subset: f_k = feats[(k["subscriber"], k["period"])] g = labels[(k["subscriber"], k["period"])] if labels else label(f_k) d += predict(f_k, model["weights"], model["decision"]) == g return round(d / len(subset), 4) DRIFTED = drift(S) FEATS_D = features(DRIFTED) MEASURES = {"input summary": [(d, input_summary(DRIFTED, d, FEATS_D)) for d in MONITORED], "prediction share": [(d, prediction_share(DRIFTED, MODEL, FEATS_D, d)) for d in MONITORED], "accuracy": [(d, accuracy(DRIFTED, MODEL, FEATS_D, d)) for d in MONITORED]} print(f"{'period':<10}{'input summary':>15}{'prediction share':>18}{'accuracy':>10}{'no-drift accuracy':>19}") for i, d in enumerate(MONITORED): print(f"{d:<10}{MEASURES['input summary'][i][1]:>15.4f}{MEASURES['prediction share'][i][1]:>18.4f}" f"{MEASURES['accuracy'][i][1]:>10.4f}{accuracy(S, MODEL, FEATS, d):>19.4f}") ``` ``` period input summary prediction share accuracy no-drift accuracy 2026-07 25.9220 0.4550 0.9700 0.9700 2026-08 26.0517 0.4700 0.9700 0.9700 2026-09 25.9800 0.4550 0.9900 0.9900 2026-10 26.4685 0.5150 0.9750 0.9750 2026-11 26.6053 0.5200 0.9800 0.9850 2026-12 27.0472 0.5450 0.9800 0.9900 ``` The input really does drift: the input summary rises **from 25.9220 to 27.0472** and the prediction share **from 0.4550 to 0.5450**, so the share of inferences that say high grows by about twenty percent. Accuracy, in contrast, **never falls below the 0.9700 baseline** in any period. What the drift actually took away is measured by the no-drift column: in `2026-12` the no-drift run gives 0.9900 and the drifted run 0.9800; the difference is **0.0100**, two resolution steps. As long as the rule does not change, a correct model stays correct on drifted input, because both the label and the prediction are computed from the same drifted features. ## A Firing Alert Is Not Evidence The fourth block attaches an alert band to the indicators. The band is the ratio of the deviation from the baseline; the baseline is `2026-08`, the last period before the one in which the drift starts. The same three bands are applied to all three indicators in the same way. ```python # ---- Alert: a measure fires if it deviates from the baseline by more than `share`. The band is two-sided. def alert(series, baseline, share): return [d for d, v in series if abs(v - baseline) / max(abs(baseline), 1e-9) > share] BASELINE_PERIOD = MONITORED[1] # 2026-08 : the last period before the one in which the drift starts ACCURACY = dict(MEASURES["accuracy"]) print(f"{'measure':<18}{'baseline':>10}{'largest deviation':>19}{'0.05':>7}{'0.10':>7}" f"{'0.20':>7}{'first firing':>16}") for name, series in MEASURES.items(): base = dict(series)[BASELINE_PERIOD] fires = [alert(series, base, p) for p in (0.05, 0.10, 0.20)] deviation = max(abs(v - base) / base for _, v in series) print(f"{name:<18}{base:>10.4f}{deviation:>19.4f}{len(fires[0]):>7}{len(fires[1]):>7}" f"{len(fires[2]):>7}{(fires[0][0] if fires[0] else '-'):>16}") FIRED = alert(MEASURES["prediction share"], dict(MEASURES["prediction share"])[BASELINE_PERIOD], 0.05) REAL = [d for d in FIRED if ACCURACY[d] < ACCURACY[BASELINE_PERIOD]] print("\nperiods firing in the 0.05 band:", FIRED) print("how many of them had accuracy fall below the baseline:", len(REAL), " false positive:", len(FIRED) - len(REAL)) print("total swing of accuracy:", f"{max(ACCURACY.values()) - min(ACCURACY.values()):.4f}", " step of period accuracy:", f"{1 / 200:.4f}", " what the drift cost in 2026-12:", f"{accuracy(S, MODEL, FEATS, MONITORED[-1]) - ACCURACY[MONITORED[-1]]:.4f}") ``` ``` measure baseline largest deviation 0.05 0.10 0.20 first firing input summary 26.0517 0.0382 0 0 0 - prediction share 0.4700 0.1596 3 2 0 2026-10 accuracy 0.9700 0.0206 0 0 0 - periods firing in the 0.05 band: ['2026-10', '2026-11', '2026-12'] how many of them had accuracy fall below the baseline: 0 false positive: 3 total swing of accuracy: 0.0200 step of period accuracy: 0.0050 what the drift cost in 2026-12: 0.0100 ``` The same drift, the same band, three different answers. The largest deviation of the input summary is **0.0382** and it fires in no band, because it is a mean and a mean smooths out the change in individual rows. The deviation of the prediction share is **0.1596**; it fires in **three periods** in the 0.05 band and in two in the 0.10 band, because it is sensitive to the rows **around** the decision threshold. The deviation of accuracy is **0.0206**, it does not fire, and moreover it is in the **upward** direction. The rule of the course can be read from here. The alert in `2026-10` did not work wrongly: the input really has drifted. But the only thing it says is **that the input has drifted**; it does not say that anything has degraded. In none of the three firing periods did accuracy fall below the baseline, and all three count as **false positives**. The firing of an indicator is not evidence of degradation; **an indicator whose firing and whose degradation are not shown separately counts as unmeasured**. The unmonitored baseline lands in a curious position here: the setup that attaches no indicator has never been wrong, because it has no alert that could be wrong. ## Summary - A monitorable model depends on three settings: the decision threshold is a setting **separate** from the label threshold and is searched in training, the training criterion is **balanced accuracy**, and the label is tied to a rule the features **see**. - In the old setup recall is **0.0000** and balanced accuracy **0.5000**; in the corrected one recall is **1.0000**, specificity **0.9670**, balanced accuracy **0.9835** and the base class ratio **0.5342**. - The first two fixes alone are not enough: with the old label rule the best of the 135 candidates stays at **0.5203**, coin-flip level, because that rule looks at a number the features do not see. - Under data drift the input summary rises **from 25.9220 to 27.0472** and the prediction share **from 0.4550 to 0.5450**, but accuracy goes **from 0.9700 to 0.9800**; what the drift took away is **0.0100** in `2026-12`, two resolution steps. - The prediction share fires in **three periods** in the 0.05 band and in none of the three did accuracy fall below the baseline: **a firing alert is not evidence of a problem**. ## Next Step In this lesson what drifted was **the input**, and because the rule stayed in place the model could stay correct; the firing indicator saw exactly this. But what if it were not the input that changed but **the rule itself**, that is, if the same features began to produce a different label: what would the same three indicators say. The input summary would not move, because the input has not moved; the prediction share would not move either, because the model keeps giving the same answer to the same inputs. The next lesson measures this case and finds the opposite result: while the indicators stay silent, accuracy falls **from 0.9700 to 0.8150**.