--- title: 'Concept Drift' source: 'https://academia.sh/en/courses/model-monitoring/concept-drift' course: 'Model Monitoring and Governance' language: en updated: '2026-10-21T05:03:02+00:00' license: 'CC BY-SA 4.0' objectives: - 'Separates concept drift from data drift and measures the number of rows whose label changes in a run where accuracy falls while no value on the input side changes.' - 'Puts the same three indicators side by side under the two kinds of drift and counts in which band each indicator fires in how many periods.' - 'Shows that proxy indicators can fire under a harmless drift and stay silent under a harmful one, and writes down that the cause of a firing may not be the drift itself.' --- # Concept Drift Measuring a change of the rule between input and label while the input does not change at all: when the threshold for counting as high drops from 30 to 27, not a single value changes on the input side, yet the label of 132 of the 1200 monitoring rows changes and the accuracy of the model falls from 0.9700 to 0.8150, that is, thirty-one resolution steps. When the same three indicators are read side by side under the two kinds of drift, what the proxy metrics see and what they miss turns into a number: the input summary fires in no band in either scenario, the prediction share fires in three periods in the 0.05 band in both and so cannot tell the harmless from the harmful, and in the 0.10 band it fires in two periods under the harmless drift and in none under the harmful one, while accuracy stays silent under the harmless drift and fires in four periods under the harmful one, remaining the only measure that tells them apart. In the previous lesson what drifted was **the input**: the input summary rose from 25.9220 to 27.0472, the prediction share fired an alert in three periods, and accuracy never fell below the 0.9700 baseline. Because the rule stayed in place, the model could stay correct, and all three of the alerts that fired counted as false positives. This lesson builds the exact opposite. On the input side **not a single value changes**; what changes is **the rule itself** between input and label. The same history of the same subscriber now begins to produce a different label, because the criterion for "counting as high" has moved. The model stays where it is, untouched, and the question to measure is this: what do the three indicators of the previous lesson say in this case. - **MO8.** The source, the pipeline, the model and the drift scenarios are **fictional**; no real learning library, monitoring product or cloud service is called. The seed is **20260218**. - **MO9.** **Concept drift** is a change of the relationship between input and label. **Data drift** moves the input distribution, concept drift moves **the rule**. The measurement procedure for drift was built in the Model Evaluation, Interpretability and Ethics course and is **not repeated**; the addition here is only **the life in production**. - **MO10.** This course's label is the rule `subscriber_mean + 6 x commercial > 30`, and **the rule is the CONCEPT**. The fictional concept drift lowers the threshold **from 30 to 27** starting in `2026-09`; it does not touch the records, it only produces labels. - **MO11.** The monitoring periods are **2026-07…2026-12**, **200 rows** per period, **1200** in total. In the monitoring periods the **base class ratio is 0.5342**; the set is balanced. - **MO12.** Three indicators: the **input summary** (the period mean of the feature), the **prediction share** (the share of rows the model calls high) and **accuracy**. The first two do not ask for a label, the third does. - **MO13.** The baseline is the **unmonitored setup**: no indicator, no alert; the degradation goes **unseen**. - **MO14.** The **alert threshold** is a two-sided **band**: a measure fires if it deviates from the baseline by more than the band width. The baseline period is `2026-08`, that is, the last period before the one in which the drift starts. - **MO15.** Resolution: the step of period accuracy is **1/200 = 0.0050**. In this lesson the drop is **0.1550**, that is, **thirty-one steps** and far above the band; the swing in the previous lesson was four steps. ## Unchanged Input, Changed Rule The first block builds the setup of the previous lesson exactly: the source, the three stages of the pipeline that look at the model, the features, the model whose decision threshold is searched separately, and the three indicators. The model is **not retrained at all in this lesson**; what is measured is the rule being pulled out from under a fixed model. ```python # FICTIONAL. The meter readings of the measurement network, the first three stages # of the pipeline that look at the model, the threshold-based model and the drift # scenarios are defined inside the lesson; no real learning library, monitoring # product or cloud service is called. SEED, M32 = 20260218, 0xFFFFFFFF def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value REGION = ["east", "west", "north", "south", "central"] TYPE = ["residential", "commercial"] PERIODS = [f"2026-{a:02d}" for a in range(1, 13)] SUBSCRIBERS = {} for i in range(200): r = rng(SEED + 37 * i) SUBSCRIBERS[f"A{i + 1:03d}"] = {"region": REGION[int(r() * 5)], "type": TYPE[int(r() * 2)], "start": 10000 + int(r() * 60000)} def generate_source(): records = [] for i, (sub, a) in enumerate(SUBSCRIBERS.items()): e = a["start"] for p in PERIODS: r = rng(SEED + 101 * i + 7 * PERIODS.index(p)) e += 4 + int(r() * 46) records.append({"subscriber": sub, "period": p, "region": a["region"], "type": a["type"], "index": e, "unit": "m3"}) return records def s1_screen(records): REQUIRED = ("subscriber", "period", "region", "index", "unit") keep = [k for k in records if all(a in k for a in REQUIRED)] return keep, len(records) - len(keep) def s2_dedupe(records): seen, keep = set(), [] for k in records: key = (k["subscriber"], k["period"]) if key in seen: continue seen.add(key) keep.append(k) return keep, len(records) - len(keep) def s3_consumption(records): """Consumption from the period difference. Without a previous period the row cannot be computed.""" table = {(k["subscriber"], k["period"]): k for k in records} out, dropped = [], 0 for k in records: i = PERIODS.index(k["period"]) if i == 0: continue prev = table.get((k["subscriber"], PERIODS[i - 1])) if prev is None: dropped += 1 continue out.append(dict(k, m3=int(k["index"]) - int(prev["index"]))) return out, dropped def rows(): v, _ = s1_screen(generate_source()) v, _ = s2_dedupe(v) v, _ = s3_consumption(v) return v def features(records, digits=4): """Features are independent of the weights; computed ONCE from the full history.""" ordered = sorted(records, key=lambda k: PERIODS.index(k["period"])) sub, reg = {}, {} for k in ordered: sub.setdefault(k["subscriber"], []).append((PERIODS.index(k["period"]), k["m3"])) reg.setdefault(k["region"], []).append((PERIODS.index(k["period"]), k["m3"])) def mean(series, i): chosen = [v for d, v in series if d < i] return round(sum(chosen) / len(chosen), digits) if chosen else 0.0 return {(k["subscriber"], k["period"]): { "subscriber_mean": mean(sub[k["subscriber"]], PERIODS.index(k["period"])), "region_mean": mean(reg[k["region"]], PERIODS.index(k["period"])), "commercial": 1 if k["type"] == "commercial" else 0} for k in records} GRID = [(a, b, c) for a in (0.6, 0.8, 1.0) for b in (0.0, 0.2, 0.4) for c in (0, 4, 8)] def score(o, w): return w[0] * o["subscriber_mean"] + w[1] * o["region_mean"] + w[2] * o["commercial"] # The label depends on a rule the features SEE. RULE = CONCEPT. RULE_THRESHOLD = 30 RULE_COMMERCIAL = 6 def label(feat_k, threshold=RULE_THRESHOLD, commercial_factor=RULE_COMMERCIAL): return 1 if feat_k["subscriber_mean"] + commercial_factor * feat_k["commercial"] > threshold else 0 MONITORED = PERIODS[6:] # 2026-07 .. 2026-12 : the periods monitored in production DECISION_GRID = (18, 22, 26, 30, 34) # the decision threshold is a SEPARATE setting from the label threshold def predict(feat_k, w, decision): return 1 if score(feat_k, w) > decision else 0 def balanced_accuracy(subset, feats, w, decision, rule=None): c = {(0, 0): 0, (0, 1): 0, (1, 0): 0, (1, 1): 0} for k in subset: f_k = feats[(k["subscriber"], k["period"])] c[(predict(f_k, w, decision), rule(k, f_k) if rule else label(f_k))] += 1 rec = c[(1, 1)] / max(c[(1, 1)] + c[(0, 1)], 1) # recall spec = c[(0, 0)] / max(c[(0, 0)] + c[(1, 0)], 1) # specificity return {"accuracy": (c[(1, 1)] + c[(0, 0)]) / len(subset), "recall": rec, "specificity": spec, "balanced": (rec + spec) / 2, "weights": w, "decision": decision} def train(records, feats, periods=None, rule=None): """Training = searching the weight TRIPLE together with the decision threshold: 27 x 5 = 135 candidates.""" subset = [k for k in records if k["period"] in (periods or PERIODS[1:9])] return max((balanced_accuracy(subset, feats, w, decision, rule) for w in GRID for decision in DECISION_GRID), key=lambda o: o["balanced"]) def input_summary(records, period, feats): d = [feats[(k["subscriber"], k["period"])]["subscriber_mean"] for k in records if k["period"] == period] return round(sum(d) / len(d), 4) def prediction_share(records, model, feats, period): d = [predict(feats[(k["subscriber"], k["period"])], model["weights"], model["decision"]) for k in records if k["period"] == period] return round(sum(d) / len(d), 4) def accuracy(records, model, feats, period, labels=None): """labels=None -> the rule has not changed.""" subset = [k for k in records if k["period"] == period] d = 0 for k in subset: f_k = feats[(k["subscriber"], k["period"])] g = labels[(k["subscriber"], k["period"])] if labels else label(f_k) d += predict(f_k, model["weights"], model["decision"]) == g return round(d / len(subset), 4) S = rows() FEATS = features(S) MODEL = train(S, FEATS) MON = [k for k in S if k["period"] in MONITORED] dist = {0: 0, 1: 0} for k in MON: dist[label(FEATS[(k["subscriber"], k["period"])])] += 1 print("monitoring rows:", len(MON), " per period:", len(MON) // len(MONITORED), " base class ratio:", round(dist[0] / len(MON), 4)) print("model:", MODEL["weights"], "/ decision threshold", MODEL["decision"], " balanced accuracy:", round(MODEL["balanced"], 4)) ``` ``` monitoring rows: 1200 per period: 200 base class ratio: 0.5342 model: (0.6, 0.0, 4) / decision threshold 18 balanced accuracy: 0.9864 ``` ## What Shows Up When the Concept Drifts The second block applies the concept drift. The `concept_drift` function **does not touch the records**; it only produces labels in which, starting from `2026-09`, the threshold for counting as high has been lowered to 27. The last two columns of the table matter: the accuracy that would be taken in that period **had the rule not changed**, and the number of rows whose **label changed** in that period. ```python # ---- FICTIONAL concept drift: THE INPUT STAYS THE SAME , the rule changes. # After 2026-09 the threshold for "counting as high" drops from 30 to 27. def concept_drift(records, feats, start="2026-09", new_threshold=27): b = PERIODS.index(start) return {(k["subscriber"], k["period"]): label(feats[(k["subscriber"], k["period"])], new_threshold if PERIODS.index(k["period"]) >= b else RULE_THRESHOLD) for k in records} NEW = concept_drift(S, FEATS) print(f"{'period':<10}{'input summary':>15}{'prediction share':>18}{'accuracy':>10}" f"{'if rule unchanged':>19}{'label changed':>15}") for d in MONITORED: subset = [k for k in S if k["period"] == d] flipped = sum(1 for k in subset if NEW[(k["subscriber"], d)] != label(FEATS[(k["subscriber"], d)])) print(f"{d:<10}{input_summary(S, d, FEATS):>15.4f}{prediction_share(S, MODEL, FEATS, d):>18.4f}" f"{accuracy(S, MODEL, FEATS, d, NEW):>10.4f}{accuracy(S, MODEL, FEATS, d):>19.4f}" f"{flipped:>15}") ``` ``` period input summary prediction share accuracy if rule unchanged label changed 2026-07 25.9220 0.4550 0.9700 0.9700 0 2026-08 26.0517 0.4700 0.9700 0.9700 0 2026-09 25.9800 0.4550 0.8150 0.9900 39 2026-10 26.3525 0.5100 0.8700 0.9750 31 2026-11 26.3261 0.5100 0.8650 0.9850 30 2026-12 26.5325 0.5150 0.8500 0.9900 32 ``` Accuracy falls in `2026-09` **from 0.9700 to 0.8150**. The drop is **0.1550**, that is, **thirty-one steps** of resolution; the whole swing of the previous lesson was four steps. There is not a single number that changed in the model, and none on the input side either: the last column says where the drift comes from, and in `2026-09` the label of **39** rows has changed. Not all of the changed labels turn into errors, because the decision threshold of the model is already consistent with the new rule on some rows; in the following periods accuracy holds around 0.8700 but **never returns to its old level**. The first two columns behave exactly as expected. The input summary goes from 25.9220 to 26.5325 and the prediction share from 0.4550 to 0.5150. These two series would have been exactly the same had the concept drift **never happened**, because both look only at the input and the model; they have no knowledge of where the rule has moved. ## Firing Under the Harmless Drift, Silent Under the Harmful One The third block brings back the data drift scenario of the previous lesson and puts the two scenarios side by side **with the same three indicators and the same three bands**. What is to be read in the table is not the individual numbers but **the standing of the rows relative to each other**. ```python # ---- FICTIONAL data drift (previous lesson): the INPUT drifts , THE RULE STAYS THE SAME. def drift(records, start="2026-09", slope=0.06): b = PERIODS.index(start) return [dict(k, m3=round(k["m3"] * (1.0 + slope * (PERIODS.index(k["period"]) - b + 1)), 4)) if PERIODS.index(k["period"]) >= b and k["type"] == "commercial" else dict(k) for k in records] def alert(series, baseline, share): return [d for d, v in series if abs(v - baseline) / max(abs(baseline), 1e-9) > share] DRIFTED = drift(S) FEATS_D = features(DRIFTED) print(f"{'scenario':<16}{'measure':<18}{'baseline':>10}{'largest deviation':>19}" f"{'0.05':>7}{'0.10':>7}{'0.20':>7}") for name, (src, feats_s, lab) in (("data drift", (DRIFTED, FEATS_D, None)), ("concept drift", (S, FEATS, NEW))): for measure, series in (("input summary", [(d, input_summary(src, d, feats_s)) for d in MONITORED]), ("prediction share", [(d, prediction_share(src, MODEL, feats_s, d)) for d in MONITORED]), ("accuracy", [(d, accuracy(src, MODEL, feats_s, d, lab)) for d in MONITORED])): base = dict(series)[MONITORED[1]] print(f"{name:<16}{measure:<18}{base:>10.4f}" f"{max(abs(v - base) / base for _, v in series):>19.4f}" + "".join(f"{len(alert(series, base, p)):>7}" for p in (0.05, 0.10, 0.20))) print("\nrows changed under concept drift:", sum(1 for k in MON if NEW[(k["subscriber"], k["period"])] != label(FEATS[(k["subscriber"], k["period"])])), "/", len(MON), " values changed on the input side: 0") DROP = accuracy(S, MODEL, FEATS, MONITORED[0], NEW) - accuracy(S, MODEL, FEATS, MONITORED[2], NEW) print("drop in accuracy:", f"{DROP:.4f}", " step of period accuracy:", f"{1 / 200:.4f}", " that is", round(DROP * 200), "steps") ``` ``` scenario measure baseline largest deviation 0.05 0.10 0.20 data drift input summary 26.0517 0.0382 0 0 0 data drift prediction share 0.4700 0.1596 3 2 0 data drift accuracy 0.9700 0.0206 0 0 0 concept drift input summary 26.0517 0.0185 0 0 0 concept drift prediction share 0.4700 0.0957 3 0 0 concept drift accuracy 0.9700 0.1598 4 4 0 rows changed under concept drift: 132 / 1200 values changed on the input side: 0 drop in accuracy: 0.1550 step of period accuracy: 0.0050 that is 31 steps ``` The input summary fires in no band in either scenario. Under concept drift it could not be otherwise: there are **zero** values changed on the input side, and the indicator looks only at the input. The prediction share fires in both and misleads in both. In the 0.05 band it fires in **three periods in each of the two scenarios**, that is, it **cannot tell** the harmless drift from the harmful one. In the 0.10 band it works exactly the other way round: it fires in **two periods** under the harmless drift and in **none** under the harmful one. The deviation of 0.0957 under concept drift has nothing to do with the drift either; that series would have been the same had the rule never changed. When an indicator fires, on its own it shows neither the degradation nor its cause. Accuracy is the only measure that tells them apart. Under the harmless drift its deviation is **0.0206** and it fires in no band; under the harmful drift its deviation is **0.1598** and it fires in **four periods** in both the 0.05 and the 0.10 band. The rule of this course is read in both directions here: as the firing of an indicator is not evidence of degradation, **its silence is not evidence of soundness either**; an indicator whose firing and whose degradation are not shown separately counts as unmeasured. In the unmonitored baseline, the label of 132 rows has changed, accuracy has fallen thirty-one steps, and nothing has given notice. ## Summary - Concept drift is a change of the **rule** between input and label; unlike data drift, not a single value changes on the input side. - When the threshold drops from 30 to 27, the label of **132** of the 1200 monitoring rows changes and accuracy falls **from 0.9700 to 0.8150**: **0.1550**, that is, **thirty-one resolution steps**. The model has not changed at all. - The input summary fires in no band in either scenario; under concept drift its firing is impossible anyway, because nothing changes where it looks. - The prediction share fires in **three periods** in the 0.05 band in both scenarios and so cannot tell them apart; in the 0.10 band it fires in two periods under the harmless drift and in **none** under the harmful one. - Accuracy is the only measure that tells them apart: under the harmless drift its deviation is 0.0206 and it stays silent, under the harmful drift 0.1598 and it fires in four periods in the 0.05 and 0.10 bands. ## Next Step The conclusion that has emerged over two lessons is this: the only indicator that sees the degradation is **accuracy**. But accuracy asks for something different from the other two, the true label. The input summary and the prediction share can be computed the moment the prediction is produced; accuracy cannot be computed until it is known whether the row really is high, and that information does not arrive in production at the same time as the prediction. In this lesson the label was used as if it were at hand at calculation time. The next lesson removes this assumption: when the label arrives **two periods later**, it counts in which period the accuracy of a period can be known, the gap between the moment of degradation and the moment it is seen, and how many predictions are produced in that gap **unmeasured**.