Performance Monitoring

The conclusion that emerged over two lessons was this: the only indicator that sees concept drift is accuracy. The input summary knows nothing about the rule having moved, the prediction share fires under the harmless drift and stays silent under the harmful one, and accuracy shows the degradation by name, falling from 0.9700 to 0.8150.

This finding makes an assumption silently. To compute accuracy, it must be known whether the row really is high, and in the previous two lessons that information was treated as always at hand. In production it is not: the prediction is produced at the start of the period, and the true label arrives only after the meter has been read, the objection period has passed and the record has been closed. This lesson puts that delay inside the setup and measures a single question: the accuracy of which period can be known in which period.

• MO16. The source, the pipeline, the model and the concept drift are fictional; no real learning library, monitoring product or cloud service is called. The seed is 20260218.
• MO17. A delayed label is a true label that arrives after the prediction. Here the delay is DELAY = 2 periods: the accuracy of a period can be computed only two periods later. The time is written in periods, not in a raw unit.
• MO18. The detection lag is the difference between the period in which the degradation happens and the period in which it can first be seen. The measure of how many periods later an indicator shows was built in the Data Pipelines and Orchestration course; it is not repeated here, it is taken as input.
• MO19. The source of the degradation is the concept drift of the previous lesson: starting in 2026-09, the threshold for counting as high drops from 30 to 27. There is no changed value on the input side.
• MO20. The monitoring periods are 2026-07…2026-12, 200 rows per period, 1200 in total. In the monitoring periods the base class ratio is 0.5342; the set is balanced.
• MO21. The alert band is 0.05 and the baseline is the accuracy of 2026-07, 0.9700. The band is two-sided.
• MO22. The baseline is the unmonitored setup: no label is ever collected, the measurable periods are 0, and the degradation is never seen.
• MO23. Resolution: the step of period accuracy is 1/200 = 0.0050; the drop caused by the concept drift is 0.1550, that is, thirty-one steps and far above the band. The delay is counted in periods, and one period means 200 predictions.

The Label Arrives Two Periods Later

The first block builds the setup of the previous two lessons and adds a single thing on top: period_seen. This function says whether the accuracy of a period can be computed at a given moment; the computation itself does not change, only when it can be done changes.

# FICTIONAL. The meter readings of the measurement network, the first three stages
# of the pipeline that look at the model, the threshold-based model, the concept
# drift and the label delay are defined inside the lesson; no real learning
# library, monitoring product or cloud service is called.
SEED, M32 = 20260218, 0xFFFFFFFF


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


REGION = ["east", "west", "north", "south", "central"]
TYPE = ["residential", "commercial"]
PERIODS = [f"2026-{a:02d}" for a in range(1, 13)]
SUBSCRIBERS = {}
for i in range(200):
    r = rng(SEED + 37 * i)
    SUBSCRIBERS[f"A{i + 1:03d}"] = {"region": REGION[int(r() * 5)], "type": TYPE[int(r() * 2)],
                                    "start": 10000 + int(r() * 60000)}


def generate_source():
    records = []
    for i, (sub, a) in enumerate(SUBSCRIBERS.items()):
        e = a["start"]
        for p in PERIODS:
            r = rng(SEED + 101 * i + 7 * PERIODS.index(p))
            e += 4 + int(r() * 46)
            records.append({"subscriber": sub, "period": p, "region": a["region"], "type": a["type"],
                            "index": e, "unit": "m3"})
    return records


def s1_screen(records):
    REQUIRED = ("subscriber", "period", "region", "index", "unit")
    keep = [k for k in records if all(a in k for a in REQUIRED)]
    return keep, len(records) - len(keep)


def s2_dedupe(records):
    seen, keep = set(), []
    for k in records:
        key = (k["subscriber"], k["period"])
        if key in seen:
            continue
        seen.add(key)
        keep.append(k)
    return keep, len(records) - len(keep)


def s3_consumption(records):
    """Consumption from the period difference. Without a previous period the row cannot be computed."""
    table = {(k["subscriber"], k["period"]): k for k in records}
    out, dropped = [], 0
    for k in records:
        i = PERIODS.index(k["period"])
        if i == 0:
            continue
        prev = table.get((k["subscriber"], PERIODS[i - 1]))
        if prev is None:
            dropped += 1
            continue
        out.append(dict(k, m3=int(k["index"]) - int(prev["index"])))
    return out, dropped


def rows():
    v, _ = s1_screen(generate_source())
    v, _ = s2_dedupe(v)
    v, _ = s3_consumption(v)
    return v


def features(records, digits=4):
    """Features are independent of the weights; computed ONCE from the full history."""
    ordered = sorted(records, key=lambda k: PERIODS.index(k["period"]))
    sub, reg = {}, {}
    for k in ordered:
        sub.setdefault(k["subscriber"], []).append((PERIODS.index(k["period"]), k["m3"]))
        reg.setdefault(k["region"], []).append((PERIODS.index(k["period"]), k["m3"]))

    def mean(series, i):
        chosen = [v for d, v in series if d < i]
        return round(sum(chosen) / len(chosen), digits) if chosen else 0.0

    return {(k["subscriber"], k["period"]): {
        "subscriber_mean": mean(sub[k["subscriber"]], PERIODS.index(k["period"])),
        "region_mean": mean(reg[k["region"]], PERIODS.index(k["period"])),
        "commercial": 1 if k["type"] == "commercial" else 0} for k in records}


GRID = [(a, b, c) for a in (0.6, 0.8, 1.0) for b in (0.0, 0.2, 0.4) for c in (0, 4, 8)]


def score(o, w):
    return w[0] * o["subscriber_mean"] + w[1] * o["region_mean"] + w[2] * o["commercial"]


RULE_THRESHOLD = 30
RULE_COMMERCIAL = 6


def label(feat_k, threshold=RULE_THRESHOLD, commercial_factor=RULE_COMMERCIAL):
    return 1 if feat_k["subscriber_mean"] + commercial_factor * feat_k["commercial"] > threshold else 0


MONITORED = PERIODS[6:]               # 2026-07 .. 2026-12 : the periods monitored in production
DECISION_GRID = (18, 22, 26, 30, 34)  # the decision threshold is a SEPARATE setting from the label threshold


def predict(feat_k, w, decision):
    return 1 if score(feat_k, w) > decision else 0


def balanced_accuracy(subset, feats, w, decision, rule=None):
    c = {(0, 0): 0, (0, 1): 0, (1, 0): 0, (1, 1): 0}
    for k in subset:
        f_k = feats[(k["subscriber"], k["period"])]
        c[(predict(f_k, w, decision), rule(k, f_k) if rule else label(f_k))] += 1
    rec = c[(1, 1)] / max(c[(1, 1)] + c[(0, 1)], 1)      # recall
    spec = c[(0, 0)] / max(c[(0, 0)] + c[(1, 0)], 1)     # specificity
    return {"accuracy": (c[(1, 1)] + c[(0, 0)]) / len(subset), "recall": rec, "specificity": spec,
            "balanced": (rec + spec) / 2, "weights": w, "decision": decision}


def train(records, feats, periods=None, rule=None):
    """Training = searching the weight TRIPLE together with the decision threshold: 27 x 5 = 135 candidates."""
    subset = [k for k in records if k["period"] in (periods or PERIODS[1:9])]
    return max((balanced_accuracy(subset, feats, w, decision, rule)
                for w in GRID for decision in DECISION_GRID), key=lambda o: o["balanced"])


def accuracy(records, model, feats, period, labels=None):
    """labels=None -> the rule has not changed."""
    subset = [k for k in records if k["period"] == period]
    d = 0
    for k in subset:
        f_k = feats[(k["subscriber"], k["period"])]
        g = labels[(k["subscriber"], k["period"])] if labels else label(f_k)
        d += predict(f_k, model["weights"], model["decision"]) == g
    return round(d / len(subset), 4)


# ---- FICTIONAL concept drift (previous lesson): THE INPUT STAYS THE SAME , the rule changes.
def concept_drift(records, feats, start="2026-09", new_threshold=27):
    b = PERIODS.index(start)
    return {(k["subscriber"], k["period"]):
            label(feats[(k["subscriber"], k["period"])],
                  new_threshold if PERIODS.index(k["period"]) >= b else RULE_THRESHOLD)
            for k in records}


DELAY = 2                  # after how many periods the label of a period reaches us


def period_seen(period, now, delay=DELAY):
    """The accuracy of a period can be computed only `delay` periods later."""
    return PERIODS.index(period) + delay <= PERIODS.index(now)


S = rows()
FEATS = features(S)
MODEL = train(S, FEATS)
NEW = concept_drift(S, FEATS)
MON = [k for k in S if k["period"] in MONITORED]
dist = {0: 0, 1: 0}
for k in MON:
    dist[label(FEATS[(k["subscriber"], k["period"])])] += 1
ACTUAL = {d: accuracy(S, MODEL, FEATS, d, NEW) for d in MONITORED}
print("monitoring rows:", len(MON), " per period:", len(MON) // len(MONITORED),
      " base class ratio:", round(dist[0] / len(MON), 4), " delay:", DELAY, "periods")
print("actual accuracy:", [f"{ACTUAL[d]:.4f}" for d in MONITORED])

monitoring rows: 1200  per period: 200  base class ratio: 0.5342  delay: 2 periods
actual accuracy: ['0.9700', '0.9700', '0.8150', '0.8700', '0.8650', '0.8500']

The ACTUAL dictionary is the accuracy series that someone looking back later could see. Someone standing in production never holds this series in full; in each period they see only a prefix of it.

Which Period’s Accuracy Is Known in Which Period

The second block builds the table in production for each period: the newest measurable period at that moment, the accuracy of that period, the actual accuracy at that moment, and the number of predictions at hand that are not yet measured. The third column is the knowledge of someone looking back, not of an observer; the person standing in production looks at the second column.

BAND, BASE = 0.05, ACTUAL[MONITORED[0]]
print(f"{'now':<10}{'latest measurable period':>26}{'accuracy of that period':>25}"
      f"{'current actual accuracy':>25}{'unmeasured predictions':>24}")
for now in MONITORED:
    measured = [d for d in MONITORED if period_seen(d, now)]
    produced = MONITORED[: MONITORED.index(now) + 1]
    latest = measured[-1] if measured else None
    print(f"{now:<10}{(latest or '-'):>26}"
          f"{(f'{ACTUAL[latest]:.4f}' if latest else '-'):>25}{ACTUAL[now]:>25.4f}"
          f"{(len(produced) - len(measured)) * 200:>24}")

DEGRADED = next(d for d in MONITORED if abs(ACTUAL[d] - BASE) / BASE > BAND)
SEEN = next(s for s in MONITORED
            if any(period_seen(d, s) and abs(ACTUAL[d] - BASE) / BASE > BAND
                   for d in MONITORED))
print("\nperiod of degradation:", DEGRADED, " period in which the alert can fire:", SEEN,
      " detection lag:", PERIODS.index(SEEN) - PERIODS.index(DEGRADED), "periods")
GAP = MONITORED[MONITORED.index(DEGRADED):MONITORED.index(SEEN)]
print("predictions produced unmeasured in this gap:", len(GAP) * 200,
      " periods in the gap:", GAP)
print("number later found wrong among them:",
      sum(round((1 - ACTUAL[d]) * 200) for d in GAP),
      " number had the rule not changed:",
      sum(round((1 - accuracy(S, MODEL, FEATS, d)) * 200) for d in GAP))

now         latest measurable period  accuracy of that period  current actual accuracy  unmeasured predictions
2026-07                            -                        -                   0.9700                     200
2026-08                            -                        -                   0.9700                     400
2026-09                      2026-07                   0.9700                   0.8150                     400
2026-10                      2026-08                   0.9700                   0.8700                     400
2026-11                      2026-09                   0.8150                   0.8650                     400
2026-12                      2026-10                   0.8700                   0.8500                     400

period of degradation: 2026-09  period in which the alert can fire: 2026-11  detection lag: 2 periods
predictions produced unmeasured in this gap: 400  periods in the gap: ['2026-09', '2026-10']
number later found wrong among them: 63  number had the rule not changed: 7

The period of degradation is 2026-09, and the number on the screen in that period is 0.9700. This number is not wrong; the accuracy of 2026-07 really was 0.9700. The number is only old, and if its age is written nowhere, it cannot be told apart from a wrong one. The same happens in the next period: while the actual accuracy has fallen to 0.8700, the newest measure at hand is still 0.9700.

The first period in which the alert can fire is 2026-11, because the labels of 2026-09 are completed only then. The detection lag is two periods, and in this gap 400 predictions were produced unmeasured. It becomes clear later that 63 of these 400 predictions were wrong; had the rule not changed, the number of wrong ones in the same two periods would have been 7. The 56 predictions in between are the ones produced and used during the time the degradation went unseen, and they cannot be corrected retroactively, because the decision has already been made.

The second thing to notice is the last column: even after the gap closes, 400 predictions remain unmeasured in every period. The delay is not a temporary start-up problem, it is a debt carried continuously; the indicator in production never shows the present, it always shows two periods ago.

The Cost of Delay

The third block sweeps the delay. The rest of the setup is fixed; the only thing that changes is how many periods later the label arrives. The last row is the unmonitored baseline: no label is ever collected.

AFTER = len(MONITORED) - MONITORED.index(DEGRADED)     # number of periods after the degradation
print(f"{'delay':<12}{'degradation seen':>18}{'detection lag':>19}"
      f"{'measurable periods':>20}{'produced unmeasured':>21}")
for delay in (0, 1, 2, 3, 4):
    seen_at = next((s for s in MONITORED
                    if any(period_seen(d, s, delay) and abs(ACTUAL[d] - BASE) / BASE > BAND
                           for d in MONITORED)), None)
    lag = PERIODS.index(seen_at) - PERIODS.index(DEGRADED) if seen_at else AFTER
    print(f"{delay:<12}{(seen_at or 'not seen'):>18}{(lag if seen_at else '-'):>19}"
          f"{sum(1 for d in MONITORED if period_seen(d, MONITORED[-1], delay)):>20}{lag * 200:>21}")
print(f"{'no label':<12}{'not seen':>18}{'-':>19}{0:>20}{AFTER * 200:>21}")

delay         degradation seen      detection lag  measurable periods  produced unmeasured
0                      2026-09                  0                   6                    0
1                      2026-10                  1                   5                  200
2                      2026-11                  2                   4                  400
3                      2026-12                  3                   3                  600
4                     not seen                  -                   2                  800
no label              not seen                  -                   0                  800

The cost is linear and its unit is clear: every period of delay means 200 unmeasured predictions. In the fourth row the delay runs into the end of the monitoring window; the degradation happened in 2026-09, but because its label will arrive in 2027-01, it is never seen inside the window. This row matters, because when the delay grows large enough the measure keeps existing but becomes useless.

The last row is the unmonitored baseline and gives the same result as a delay of 4: the degradation is not seen, and 800 predictions are produced unmeasured. The only difference is in the number of measurable periods, 2 against 0. The rule of the course applies here too: a indicator counts as measured only when it can fire, and a indicator whose firing falls outside the window gives an unmeasured assurance.

Summary

• Unlike the input summary and the prediction share, accuracy asks for the true label; the label does not arrive in production together with the prediction, and in this setup it arrives two periods later.
• The degradation happens in 2026-09, but the first period in which the alert can fire is 2026-11: the detection lag is two periods, and in this gap 400 predictions were produced unmeasured; it becomes clear later that 63 of them were wrong, and had the rule not changed the number would have been 7.
• In 2026-09 the number on the screen is 0.9700 and is not wrong, only two periods old; a indicator whose age is not written cannot be told apart from a wrong indicator.
• After the gap closes, 400 predictions remain unmeasured in every period: the delay is not temporary, it is a debt carried continuously.
• The sweep gives a linear cost: 0 unmeasured predictions at a delay of 0, 200 at 1, 400 at 2, 600 at 3; at 4 the degradation is never seen inside the monitoring window, and in the unmonitored baseline the number of measurable periods is 0.

Next Step

Over three lessons the seeing of the degradation was measured, and the last measure left a gap: after the degradation happens, the only measure that sees it stays silent for two periods, and in that gap 400 predictions are produced unmeasured. What is at hand in this gap was never asked. The input summary and the prediction share are there; they are computed without waiting for the label, and in the previous lesson both were seen to be able to mislead. The next lesson treats these as proxy metrics and counts what each sees one by one: which proxy can see the concept drift, which one fires under the harmless drift, and whether the table changes when a third proxy such as input completeness is added.