Concept Drift

In the previous lesson what drifted was the input: the input summary rose from 25.9220 to 27.0472, the prediction share fired an alert in three periods, and accuracy never fell below the 0.9700 baseline. Because the rule stayed in place, the model could stay correct, and all three of the alerts that fired counted as false positives.

This lesson builds the exact opposite. On the input side not a single value changes; what changes is the rule itself between input and label. The same history of the same subscriber now begins to produce a different label, because the criterion for “counting as high” has moved. The model stays where it is, untouched, and the question to measure is this: what do the three indicators of the previous lesson say in this case.

• MO8. The source, the pipeline, the model and the drift scenarios are fictional; no real learning library, monitoring product or cloud service is called. The seed is 20260218.
• MO9. Concept drift is a change of the relationship between input and label. Data drift moves the input distribution, concept drift moves the rule. The measurement procedure for drift was built in the Model Evaluation, Interpretability and Ethics course and is not repeated; the addition here is only the life in production.
• MO10. This course’s label is the rule subscriber_mean + 6 x commercial > 30, and the rule is the CONCEPT. The fictional concept drift lowers the threshold from 30 to 27 starting in 2026-09; it does not touch the records, it only produces labels.
• MO11. The monitoring periods are 2026-07…2026-12, 200 rows per period, 1200 in total. In the monitoring periods the base class ratio is 0.5342; the set is balanced.
• MO12. Three indicators: the input summary (the period mean of the feature), the prediction share (the share of rows the model calls high) and accuracy. The first two do not ask for a label, the third does.
• MO13. The baseline is the unmonitored setup: no indicator, no alert; the degradation goes unseen.
• MO14. The alert threshold is a two-sided band: a measure fires if it deviates from the baseline by more than the band width. The baseline period is 2026-08, that is, the last period before the one in which the drift starts.
• MO15. Resolution: the step of period accuracy is 1/200 = 0.0050. In this lesson the drop is 0.1550, that is, thirty-one steps and far above the band; the swing in the previous lesson was four steps.

Unchanged Input, Changed Rule

The first block builds the setup of the previous lesson exactly: the source, the three stages of the pipeline that look at the model, the features, the model whose decision threshold is searched separately, and the three indicators. The model is not retrained at all in this lesson; what is measured is the rule being pulled out from under a fixed model.

# FICTIONAL. The meter readings of the measurement network, the first three stages
# of the pipeline that look at the model, the threshold-based model and the drift
# scenarios are defined inside the lesson; no real learning library, monitoring
# product or cloud service is called.
SEED, M32 = 20260218, 0xFFFFFFFF


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


REGION = ["east", "west", "north", "south", "central"]
TYPE = ["residential", "commercial"]
PERIODS = [f"2026-{a:02d}" for a in range(1, 13)]
SUBSCRIBERS = {}
for i in range(200):
    r = rng(SEED + 37 * i)
    SUBSCRIBERS[f"A{i + 1:03d}"] = {"region": REGION[int(r() * 5)], "type": TYPE[int(r() * 2)],
                                    "start": 10000 + int(r() * 60000)}


def generate_source():
    records = []
    for i, (sub, a) in enumerate(SUBSCRIBERS.items()):
        e = a["start"]
        for p in PERIODS:
            r = rng(SEED + 101 * i + 7 * PERIODS.index(p))
            e += 4 + int(r() * 46)
            records.append({"subscriber": sub, "period": p, "region": a["region"], "type": a["type"],
                            "index": e, "unit": "m3"})
    return records


def s1_screen(records):
    REQUIRED = ("subscriber", "period", "region", "index", "unit")
    keep = [k for k in records if all(a in k for a in REQUIRED)]
    return keep, len(records) - len(keep)


def s2_dedupe(records):
    seen, keep = set(), []
    for k in records:
        key = (k["subscriber"], k["period"])
        if key in seen:
            continue
        seen.add(key)
        keep.append(k)
    return keep, len(records) - len(keep)


def s3_consumption(records):
    """Consumption from the period difference. Without a previous period the row cannot be computed."""
    table = {(k["subscriber"], k["period"]): k for k in records}
    out, dropped = [], 0
    for k in records:
        i = PERIODS.index(k["period"])
        if i == 0:
            continue
        prev = table.get((k["subscriber"], PERIODS[i - 1]))
        if prev is None:
            dropped += 1
            continue
        out.append(dict(k, m3=int(k["index"]) - int(prev["index"])))
    return out, dropped


def rows():
    v, _ = s1_screen(generate_source())
    v, _ = s2_dedupe(v)
    v, _ = s3_consumption(v)
    return v


def features(records, digits=4):
    """Features are independent of the weights; computed ONCE from the full history."""
    ordered = sorted(records, key=lambda k: PERIODS.index(k["period"]))
    sub, reg = {}, {}
    for k in ordered:
        sub.setdefault(k["subscriber"], []).append((PERIODS.index(k["period"]), k["m3"]))
        reg.setdefault(k["region"], []).append((PERIODS.index(k["period"]), k["m3"]))

    def mean(series, i):
        chosen = [v for d, v in series if d < i]
        return round(sum(chosen) / len(chosen), digits) if chosen else 0.0

    return {(k["subscriber"], k["period"]): {
        "subscriber_mean": mean(sub[k["subscriber"]], PERIODS.index(k["period"])),
        "region_mean": mean(reg[k["region"]], PERIODS.index(k["period"])),
        "commercial": 1 if k["type"] == "commercial" else 0} for k in records}


GRID = [(a, b, c) for a in (0.6, 0.8, 1.0) for b in (0.0, 0.2, 0.4) for c in (0, 4, 8)]


def score(o, w):
    return w[0] * o["subscriber_mean"] + w[1] * o["region_mean"] + w[2] * o["commercial"]


# The label depends on a rule the features SEE. RULE = CONCEPT.
RULE_THRESHOLD = 30
RULE_COMMERCIAL = 6


def label(feat_k, threshold=RULE_THRESHOLD, commercial_factor=RULE_COMMERCIAL):
    return 1 if feat_k["subscriber_mean"] + commercial_factor * feat_k["commercial"] > threshold else 0


MONITORED = PERIODS[6:]               # 2026-07 .. 2026-12 : the periods monitored in production
DECISION_GRID = (18, 22, 26, 30, 34)  # the decision threshold is a SEPARATE setting from the label threshold


def predict(feat_k, w, decision):
    return 1 if score(feat_k, w) > decision else 0


def balanced_accuracy(subset, feats, w, decision, rule=None):
    c = {(0, 0): 0, (0, 1): 0, (1, 0): 0, (1, 1): 0}
    for k in subset:
        f_k = feats[(k["subscriber"], k["period"])]
        c[(predict(f_k, w, decision), rule(k, f_k) if rule else label(f_k))] += 1
    rec = c[(1, 1)] / max(c[(1, 1)] + c[(0, 1)], 1)      # recall
    spec = c[(0, 0)] / max(c[(0, 0)] + c[(1, 0)], 1)     # specificity
    return {"accuracy": (c[(1, 1)] + c[(0, 0)]) / len(subset), "recall": rec, "specificity": spec,
            "balanced": (rec + spec) / 2, "weights": w, "decision": decision}


def train(records, feats, periods=None, rule=None):
    """Training = searching the weight TRIPLE together with the decision threshold: 27 x 5 = 135 candidates."""
    subset = [k for k in records if k["period"] in (periods or PERIODS[1:9])]
    return max((balanced_accuracy(subset, feats, w, decision, rule)
                for w in GRID for decision in DECISION_GRID), key=lambda o: o["balanced"])


def input_summary(records, period, feats):
    d = [feats[(k["subscriber"], k["period"])]["subscriber_mean"] for k in records if k["period"] == period]
    return round(sum(d) / len(d), 4)


def prediction_share(records, model, feats, period):
    d = [predict(feats[(k["subscriber"], k["period"])], model["weights"], model["decision"])
         for k in records if k["period"] == period]
    return round(sum(d) / len(d), 4)


def accuracy(records, model, feats, period, labels=None):
    """labels=None -> the rule has not changed."""
    subset = [k for k in records if k["period"] == period]
    d = 0
    for k in subset:
        f_k = feats[(k["subscriber"], k["period"])]
        g = labels[(k["subscriber"], k["period"])] if labels else label(f_k)
        d += predict(f_k, model["weights"], model["decision"]) == g
    return round(d / len(subset), 4)


S = rows()
FEATS = features(S)
MODEL = train(S, FEATS)
MON = [k for k in S if k["period"] in MONITORED]
dist = {0: 0, 1: 0}
for k in MON:
    dist[label(FEATS[(k["subscriber"], k["period"])])] += 1
print("monitoring rows:", len(MON), " per period:", len(MON) // len(MONITORED),
      " base class ratio:", round(dist[0] / len(MON), 4))
print("model:", MODEL["weights"], "/ decision threshold", MODEL["decision"],
      " balanced accuracy:", round(MODEL["balanced"], 4))

monitoring rows: 1200  per period: 200  base class ratio: 0.5342
model: (0.6, 0.0, 4) / decision threshold 18  balanced accuracy: 0.9864

What Shows Up When the Concept Drifts

The second block applies the concept drift. The concept_drift function does not touch the records; it only produces labels in which, starting from 2026-09, the threshold for counting as high has been lowered to 27. The last two columns of the table matter: the accuracy that would be taken in that period had the rule not changed, and the number of rows whose label changed in that period.

# ---- FICTIONAL concept drift: THE INPUT STAYS THE SAME , the rule changes.
# After 2026-09 the threshold for "counting as high" drops from 30 to 27.
def concept_drift(records, feats, start="2026-09", new_threshold=27):
    b = PERIODS.index(start)
    return {(k["subscriber"], k["period"]):
            label(feats[(k["subscriber"], k["period"])],
                  new_threshold if PERIODS.index(k["period"]) >= b else RULE_THRESHOLD)
            for k in records}


NEW = concept_drift(S, FEATS)
print(f"{'period':<10}{'input summary':>15}{'prediction share':>18}{'accuracy':>10}"
      f"{'if rule unchanged':>19}{'label changed':>15}")
for d in MONITORED:
    subset = [k for k in S if k["period"] == d]
    flipped = sum(1 for k in subset
                  if NEW[(k["subscriber"], d)] != label(FEATS[(k["subscriber"], d)]))
    print(f"{d:<10}{input_summary(S, d, FEATS):>15.4f}{prediction_share(S, MODEL, FEATS, d):>18.4f}"
          f"{accuracy(S, MODEL, FEATS, d, NEW):>10.4f}{accuracy(S, MODEL, FEATS, d):>19.4f}"
          f"{flipped:>15}")

period      input summary  prediction share  accuracy  if rule unchanged  label changed
2026-07           25.9220            0.4550    0.9700             0.9700              0
2026-08           26.0517            0.4700    0.9700             0.9700              0
2026-09           25.9800            0.4550    0.8150             0.9900             39
2026-10           26.3525            0.5100    0.8700             0.9750             31
2026-11           26.3261            0.5100    0.8650             0.9850             30
2026-12           26.5325            0.5150    0.8500             0.9900             32

Accuracy falls in 2026-09 from 0.9700 to 0.8150. The drop is 0.1550, that is, thirty-one steps of resolution; the whole swing of the previous lesson was four steps. There is not a single number that changed in the model, and none on the input side either: the last column says where the drift comes from, and in 2026-09 the label of 39 rows has changed. Not all of the changed labels turn into errors, because the decision threshold of the model is already consistent with the new rule on some rows; in the following periods accuracy holds around 0.8700 but never returns to its old level.

The first two columns behave exactly as expected. The input summary goes from 25.9220 to 26.5325 and the prediction share from 0.4550 to 0.5150. These two series would have been exactly the same had the concept drift never happened, because both look only at the input and the model; they have no knowledge of where the rule has moved.

Firing Under the Harmless Drift, Silent Under the Harmful One

The third block brings back the data drift scenario of the previous lesson and puts the two scenarios side by side with the same three indicators and the same three bands. What is to be read in the table is not the individual numbers but the standing of the rows relative to each other.

# ---- FICTIONAL data drift (previous lesson): the INPUT drifts , THE RULE STAYS THE SAME.
def drift(records, start="2026-09", slope=0.06):
    b = PERIODS.index(start)
    return [dict(k, m3=round(k["m3"] * (1.0 + slope * (PERIODS.index(k["period"]) - b + 1)), 4))
            if PERIODS.index(k["period"]) >= b and k["type"] == "commercial" else dict(k)
            for k in records]


def alert(series, baseline, share):
    return [d for d, v in series if abs(v - baseline) / max(abs(baseline), 1e-9) > share]


DRIFTED = drift(S)
FEATS_D = features(DRIFTED)
print(f"{'scenario':<16}{'measure':<18}{'baseline':>10}{'largest deviation':>19}"
      f"{'0.05':>7}{'0.10':>7}{'0.20':>7}")
for name, (src, feats_s, lab) in (("data drift", (DRIFTED, FEATS_D, None)),
                                  ("concept drift", (S, FEATS, NEW))):
    for measure, series in (("input summary", [(d, input_summary(src, d, feats_s)) for d in MONITORED]),
                            ("prediction share", [(d, prediction_share(src, MODEL, feats_s, d)) for d in MONITORED]),
                            ("accuracy", [(d, accuracy(src, MODEL, feats_s, d, lab)) for d in MONITORED])):
        base = dict(series)[MONITORED[1]]
        print(f"{name:<16}{measure:<18}{base:>10.4f}"
              f"{max(abs(v - base) / base for _, v in series):>19.4f}"
              + "".join(f"{len(alert(series, base, p)):>7}" for p in (0.05, 0.10, 0.20)))

print("\nrows changed under concept drift:",
      sum(1 for k in MON if NEW[(k["subscriber"], k["period"])] != label(FEATS[(k["subscriber"], k["period"])])),
      "/", len(MON), " values changed on the input side: 0")
DROP = accuracy(S, MODEL, FEATS, MONITORED[0], NEW) - accuracy(S, MODEL, FEATS, MONITORED[2], NEW)
print("drop in accuracy:", f"{DROP:.4f}", " step of period accuracy:", f"{1 / 200:.4f}",
      " that is", round(DROP * 200), "steps")

scenario        measure             baseline  largest deviation   0.05   0.10   0.20
data drift      input summary        26.0517             0.0382      0      0      0
data drift      prediction share      0.4700             0.1596      3      2      0
data drift      accuracy              0.9700             0.0206      0      0      0
concept drift   input summary        26.0517             0.0185      0      0      0
concept drift   prediction share      0.4700             0.0957      3      0      0
concept drift   accuracy              0.9700             0.1598      4      4      0

rows changed under concept drift: 132 / 1200  values changed on the input side: 0
drop in accuracy: 0.1550  step of period accuracy: 0.0050  that is 31 steps

The input summary fires in no band in either scenario. Under concept drift it could not be otherwise: there are zero values changed on the input side, and the indicator looks only at the input.

The prediction share fires in both and misleads in both. In the 0.05 band it fires in three periods in each of the two scenarios, that is, it cannot tell the harmless drift from the harmful one. In the 0.10 band it works exactly the other way round: it fires in two periods under the harmless drift and in none under the harmful one. The deviation of 0.0957 under concept drift has nothing to do with the drift either; that series would have been the same had the rule never changed. When an indicator fires, on its own it shows neither the degradation nor its cause.

Accuracy is the only measure that tells them apart. Under the harmless drift its deviation is 0.0206 and it fires in no band; under the harmful drift its deviation is 0.1598 and it fires in four periods in both the 0.05 and the 0.10 band. The rule of this course is read in both directions here: as the firing of an indicator is not evidence of degradation, its silence is not evidence of soundness either; an indicator whose firing and whose degradation are not shown separately counts as unmeasured. In the unmonitored baseline, the label of 132 rows has changed, accuracy has fallen thirty-one steps, and nothing has given notice.

Summary

• Concept drift is a change of the rule between input and label; unlike data drift, not a single value changes on the input side.
• When the threshold drops from 30 to 27, the label of 132 of the 1200 monitoring rows changes and accuracy falls from 0.9700 to 0.8150: 0.1550, that is, thirty-one resolution steps. The model has not changed at all.
• The input summary fires in no band in either scenario; under concept drift its firing is impossible anyway, because nothing changes where it looks.
• The prediction share fires in three periods in the 0.05 band in both scenarios and so cannot tell them apart; in the 0.10 band it fires in two periods under the harmless drift and in none under the harmful one.
• Accuracy is the only measure that tells them apart: under the harmless drift its deviation is 0.0206 and it stays silent, under the harmful drift 0.1598 and it fires in four periods in the 0.05 and 0.10 bands.

Next Step

The conclusion that has emerged over two lessons is this: the only indicator that sees the degradation is accuracy. But accuracy asks for something different from the other two, the true label. The input summary and the prediction share can be computed the moment the prediction is produced; accuracy cannot be computed until it is known whether the row really is high, and that information does not arrive in production at the same time as the prediction. In this lesson the label was used as if it were at hand at calculation time. The next lesson removes this assumption: when the label arrives two periods later, it counts in which period the accuracy of a period can be known, the gap between the moment of degradation and the moment it is seen, and how many predictions are produced in that gap unmeasured.