--- title: 'Performance Monitoring' source: 'https://academia.sh/en/courses/model-monitoring/performance-monitoring' course: 'Model Monitoring and Governance' language: en updated: '2026-10-21T05:03:02+00:00' license: 'CC BY-SA 4.0' objectives: - 'Computes in which period the accuracy of a period can be known under a delayed label and measures the difference between the moment of degradation and the moment it is seen, in periods.' - 'Counts the predictions produced unmeasured during the detection lag and compares this with the unmonitored baseline.' - 'Shows by a sweep how the moment of seeing and the number of unmeasured predictions change as the delay changes.' --- # Performance Monitoring Measuring when performance can be known in a production setup where the true label does not arrive together with the prediction: because the accuracy of a period can only be computed two periods later, the drop in accuracy that concept drift caused in 2026-09 can be seen for the first time in 2026-11, and 400 predictions are produced unmeasured in this gap. In 2026-09 the actual accuracy is 0.8150 while the newest measure at hand is the 0.9700 of 2026-07, so the indicator is not wrong, it is old. The delay sweep gives a linear cost: at a delay of 0 periods the degradation is seen in the same period and the unmeasured predictions are 0, at 1 period 200, at 2 periods 400, at 3 periods 600, and at 4 periods the degradation is never seen inside the monitoring window; when no label is ever collected, the number of measurable periods is 0 and the degradation is never seen. The conclusion that emerged over two lessons was this: the only indicator that sees concept drift is **accuracy**. The input summary knows nothing about the rule having moved, the prediction share fires under the harmless drift and stays silent under the harmful one, and accuracy shows the degradation by name, falling from 0.9700 to 0.8150. This finding makes an assumption silently. To compute accuracy, it must be **known** whether the row really is high, and in the previous two lessons that information was treated as always at hand. In production it is not: the prediction is produced at the start of the period, and the true label arrives only after the meter has been read, the objection period has passed and the record has been closed. This lesson puts that delay inside the setup and measures a single question: **the accuracy of which period can be known in which period.** - **MO16.** The source, the pipeline, the model and the concept drift are **fictional**; no real learning library, monitoring product or cloud service is called. The seed is **20260218**. - **MO17.** A **delayed label** is a true label that arrives after the prediction. Here the delay is `DELAY = 2` **periods**: the accuracy of a period can be computed only two periods later. **The time is written in periods, not in a raw unit.** - **MO18.** The **detection lag** is the difference between the period in which the degradation happens and the period in which it can first be seen. The measure of **how many periods later an indicator shows** was built in the Data Pipelines and Orchestration course; it is **not repeated** here, it is taken as input. - **MO19.** The source of the degradation is the concept drift of the previous lesson: starting in `2026-09`, the threshold for counting as high drops from 30 to 27. There is no changed value on the input side. - **MO20.** The monitoring periods are **2026-07…2026-12**, **200 rows** per period, **1200** in total. In the monitoring periods the **base class ratio is 0.5342**; the set is balanced. - **MO21.** The alert band is **0.05** and the baseline is the accuracy of `2026-07`, **0.9700**. The band is two-sided. - **MO22.** The baseline is the **unmonitored setup**: no label is ever collected, the measurable periods are **0**, and the degradation is **never seen**. - **MO23.** Resolution: the step of period accuracy is **1/200 = 0.0050**; the drop caused by the concept drift is **0.1550**, that is, thirty-one steps and far above the band. The delay is counted in **periods**, and one period means **200 predictions**. ## The Label Arrives Two Periods Later The first block builds the setup of the previous two lessons and adds a single thing on top: `period_seen`. This function says whether the accuracy of a period **can be computed at a given moment**; the computation itself does not change, only **when it can be done** changes. ```python # FICTIONAL. The meter readings of the measurement network, the first three stages # of the pipeline that look at the model, the threshold-based model, the concept # drift and the label delay are defined inside the lesson; no real learning # library, monitoring product or cloud service is called. SEED, M32 = 20260218, 0xFFFFFFFF def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value REGION = ["east", "west", "north", "south", "central"] TYPE = ["residential", "commercial"] PERIODS = [f"2026-{a:02d}" for a in range(1, 13)] SUBSCRIBERS = {} for i in range(200): r = rng(SEED + 37 * i) SUBSCRIBERS[f"A{i + 1:03d}"] = {"region": REGION[int(r() * 5)], "type": TYPE[int(r() * 2)], "start": 10000 + int(r() * 60000)} def generate_source(): records = [] for i, (sub, a) in enumerate(SUBSCRIBERS.items()): e = a["start"] for p in PERIODS: r = rng(SEED + 101 * i + 7 * PERIODS.index(p)) e += 4 + int(r() * 46) records.append({"subscriber": sub, "period": p, "region": a["region"], "type": a["type"], "index": e, "unit": "m3"}) return records def s1_screen(records): REQUIRED = ("subscriber", "period", "region", "index", "unit") keep = [k for k in records if all(a in k for a in REQUIRED)] return keep, len(records) - len(keep) def s2_dedupe(records): seen, keep = set(), [] for k in records: key = (k["subscriber"], k["period"]) if key in seen: continue seen.add(key) keep.append(k) return keep, len(records) - len(keep) def s3_consumption(records): """Consumption from the period difference. Without a previous period the row cannot be computed.""" table = {(k["subscriber"], k["period"]): k for k in records} out, dropped = [], 0 for k in records: i = PERIODS.index(k["period"]) if i == 0: continue prev = table.get((k["subscriber"], PERIODS[i - 1])) if prev is None: dropped += 1 continue out.append(dict(k, m3=int(k["index"]) - int(prev["index"]))) return out, dropped def rows(): v, _ = s1_screen(generate_source()) v, _ = s2_dedupe(v) v, _ = s3_consumption(v) return v def features(records, digits=4): """Features are independent of the weights; computed ONCE from the full history.""" ordered = sorted(records, key=lambda k: PERIODS.index(k["period"])) sub, reg = {}, {} for k in ordered: sub.setdefault(k["subscriber"], []).append((PERIODS.index(k["period"]), k["m3"])) reg.setdefault(k["region"], []).append((PERIODS.index(k["period"]), k["m3"])) def mean(series, i): chosen = [v for d, v in series if d < i] return round(sum(chosen) / len(chosen), digits) if chosen else 0.0 return {(k["subscriber"], k["period"]): { "subscriber_mean": mean(sub[k["subscriber"]], PERIODS.index(k["period"])), "region_mean": mean(reg[k["region"]], PERIODS.index(k["period"])), "commercial": 1 if k["type"] == "commercial" else 0} for k in records} GRID = [(a, b, c) for a in (0.6, 0.8, 1.0) for b in (0.0, 0.2, 0.4) for c in (0, 4, 8)] def score(o, w): return w[0] * o["subscriber_mean"] + w[1] * o["region_mean"] + w[2] * o["commercial"] RULE_THRESHOLD = 30 RULE_COMMERCIAL = 6 def label(feat_k, threshold=RULE_THRESHOLD, commercial_factor=RULE_COMMERCIAL): return 1 if feat_k["subscriber_mean"] + commercial_factor * feat_k["commercial"] > threshold else 0 MONITORED = PERIODS[6:] # 2026-07 .. 2026-12 : the periods monitored in production DECISION_GRID = (18, 22, 26, 30, 34) # the decision threshold is a SEPARATE setting from the label threshold def predict(feat_k, w, decision): return 1 if score(feat_k, w) > decision else 0 def balanced_accuracy(subset, feats, w, decision, rule=None): c = {(0, 0): 0, (0, 1): 0, (1, 0): 0, (1, 1): 0} for k in subset: f_k = feats[(k["subscriber"], k["period"])] c[(predict(f_k, w, decision), rule(k, f_k) if rule else label(f_k))] += 1 rec = c[(1, 1)] / max(c[(1, 1)] + c[(0, 1)], 1) # recall spec = c[(0, 0)] / max(c[(0, 0)] + c[(1, 0)], 1) # specificity return {"accuracy": (c[(1, 1)] + c[(0, 0)]) / len(subset), "recall": rec, "specificity": spec, "balanced": (rec + spec) / 2, "weights": w, "decision": decision} def train(records, feats, periods=None, rule=None): """Training = searching the weight TRIPLE together with the decision threshold: 27 x 5 = 135 candidates.""" subset = [k for k in records if k["period"] in (periods or PERIODS[1:9])] return max((balanced_accuracy(subset, feats, w, decision, rule) for w in GRID for decision in DECISION_GRID), key=lambda o: o["balanced"]) def accuracy(records, model, feats, period, labels=None): """labels=None -> the rule has not changed.""" subset = [k for k in records if k["period"] == period] d = 0 for k in subset: f_k = feats[(k["subscriber"], k["period"])] g = labels[(k["subscriber"], k["period"])] if labels else label(f_k) d += predict(f_k, model["weights"], model["decision"]) == g return round(d / len(subset), 4) # ---- FICTIONAL concept drift (previous lesson): THE INPUT STAYS THE SAME , the rule changes. def concept_drift(records, feats, start="2026-09", new_threshold=27): b = PERIODS.index(start) return {(k["subscriber"], k["period"]): label(feats[(k["subscriber"], k["period"])], new_threshold if PERIODS.index(k["period"]) >= b else RULE_THRESHOLD) for k in records} DELAY = 2 # after how many periods the label of a period reaches us def period_seen(period, now, delay=DELAY): """The accuracy of a period can be computed only `delay` periods later.""" return PERIODS.index(period) + delay <= PERIODS.index(now) S = rows() FEATS = features(S) MODEL = train(S, FEATS) NEW = concept_drift(S, FEATS) MON = [k for k in S if k["period"] in MONITORED] dist = {0: 0, 1: 0} for k in MON: dist[label(FEATS[(k["subscriber"], k["period"])])] += 1 ACTUAL = {d: accuracy(S, MODEL, FEATS, d, NEW) for d in MONITORED} print("monitoring rows:", len(MON), " per period:", len(MON) // len(MONITORED), " base class ratio:", round(dist[0] / len(MON), 4), " delay:", DELAY, "periods") print("actual accuracy:", [f"{ACTUAL[d]:.4f}" for d in MONITORED]) ``` ``` monitoring rows: 1200 per period: 200 base class ratio: 0.5342 delay: 2 periods actual accuracy: ['0.9700', '0.9700', '0.8150', '0.8700', '0.8650', '0.8500'] ``` The `ACTUAL` dictionary is the accuracy series that someone looking back later could see. Someone standing in production never holds this series in full; in each period they see only a **prefix** of it. ## Which Period's Accuracy Is Known in Which Period The second block builds the table in production for each period: the newest **measurable** period at that moment, the accuracy of that period, the **actual** accuracy at that moment, and the number of predictions at hand that are not yet **measured**. The third column is the knowledge of someone looking back, not of an observer; the person standing in production looks at the second column. ```python BAND, BASE = 0.05, ACTUAL[MONITORED[0]] print(f"{'now':<10}{'latest measurable period':>26}{'accuracy of that period':>25}" f"{'current actual accuracy':>25}{'unmeasured predictions':>24}") for now in MONITORED: measured = [d for d in MONITORED if period_seen(d, now)] produced = MONITORED[: MONITORED.index(now) + 1] latest = measured[-1] if measured else None print(f"{now:<10}{(latest or '-'):>26}" f"{(f'{ACTUAL[latest]:.4f}' if latest else '-'):>25}{ACTUAL[now]:>25.4f}" f"{(len(produced) - len(measured)) * 200:>24}") DEGRADED = next(d for d in MONITORED if abs(ACTUAL[d] - BASE) / BASE > BAND) SEEN = next(s for s in MONITORED if any(period_seen(d, s) and abs(ACTUAL[d] - BASE) / BASE > BAND for d in MONITORED)) print("\nperiod of degradation:", DEGRADED, " period in which the alert can fire:", SEEN, " detection lag:", PERIODS.index(SEEN) - PERIODS.index(DEGRADED), "periods") GAP = MONITORED[MONITORED.index(DEGRADED):MONITORED.index(SEEN)] print("predictions produced unmeasured in this gap:", len(GAP) * 200, " periods in the gap:", GAP) print("number later found wrong among them:", sum(round((1 - ACTUAL[d]) * 200) for d in GAP), " number had the rule not changed:", sum(round((1 - accuracy(S, MODEL, FEATS, d)) * 200) for d in GAP)) ``` ``` now latest measurable period accuracy of that period current actual accuracy unmeasured predictions 2026-07 - - 0.9700 200 2026-08 - - 0.9700 400 2026-09 2026-07 0.9700 0.8150 400 2026-10 2026-08 0.9700 0.8700 400 2026-11 2026-09 0.8150 0.8650 400 2026-12 2026-10 0.8700 0.8500 400 period of degradation: 2026-09 period in which the alert can fire: 2026-11 detection lag: 2 periods predictions produced unmeasured in this gap: 400 periods in the gap: ['2026-09', '2026-10'] number later found wrong among them: 63 number had the rule not changed: 7 ``` The period of degradation is `2026-09`, and the number on the screen in that period is **0.9700**. This number is not wrong; the accuracy of `2026-07` really was 0.9700. The number is only **old**, and if its age is written nowhere, it cannot be told apart from a wrong one. The same happens in the next period: while the actual accuracy has fallen to 0.8700, the newest measure at hand is still 0.9700. The first period in which the alert can fire is **2026-11**, because the labels of `2026-09` are completed only then. **The detection lag is two periods**, and in this gap **400 predictions** were produced unmeasured. It becomes clear later that **63** of these 400 predictions were wrong; had the rule not changed, the number of wrong ones in the same two periods would have been **7**. The 56 predictions in between are the ones produced and used during the time the degradation went unseen, and they cannot be corrected retroactively, because the decision has already been made. The second thing to notice is the last column: even after the gap closes, **400 predictions** remain unmeasured in every period. The delay is not a temporary start-up problem, it is a **debt carried continuously**; the indicator in production never shows the present, it always shows two periods ago. ## The Cost of Delay The third block sweeps the delay. The rest of the setup is fixed; the only thing that changes is how many periods later the label arrives. The last row is the unmonitored baseline: no label is ever collected. ```python AFTER = len(MONITORED) - MONITORED.index(DEGRADED) # number of periods after the degradation print(f"{'delay':<12}{'degradation seen':>18}{'detection lag':>19}" f"{'measurable periods':>20}{'produced unmeasured':>21}") for delay in (0, 1, 2, 3, 4): seen_at = next((s for s in MONITORED if any(period_seen(d, s, delay) and abs(ACTUAL[d] - BASE) / BASE > BAND for d in MONITORED)), None) lag = PERIODS.index(seen_at) - PERIODS.index(DEGRADED) if seen_at else AFTER print(f"{delay:<12}{(seen_at or 'not seen'):>18}{(lag if seen_at else '-'):>19}" f"{sum(1 for d in MONITORED if period_seen(d, MONITORED[-1], delay)):>20}{lag * 200:>21}") print(f"{'no label':<12}{'not seen':>18}{'-':>19}{0:>20}{AFTER * 200:>21}") ``` ``` delay degradation seen detection lag measurable periods produced unmeasured 0 2026-09 0 6 0 1 2026-10 1 5 200 2 2026-11 2 4 400 3 2026-12 3 3 600 4 not seen - 2 800 no label not seen - 0 800 ``` The cost is linear and its unit is clear: **every period of delay means 200 unmeasured predictions**. In the fourth row the delay runs into the end of the monitoring window; the degradation happened in `2026-09`, but because its label will arrive in `2027-01`, it is **never seen** inside the window. This row matters, because when the delay grows large enough the measure **keeps existing but becomes useless**. The last row is the unmonitored baseline and gives the same result as a delay of 4: the degradation is not seen, and 800 predictions are produced unmeasured. The only difference is in the number of measurable periods, **2** against **0**. The rule of the course applies here too: a indicator counts as measured only when it **can fire**, and a indicator whose firing falls outside the window gives an unmeasured assurance. ## Summary - Unlike the input summary and the prediction share, accuracy asks for the **true label**; the label does not arrive in production together with the prediction, and in this setup it arrives **two periods** later. - The degradation happens in `2026-09`, but the first period in which the alert can fire is **2026-11**: the detection lag is **two periods**, and in this gap **400 predictions** were produced unmeasured; it becomes clear later that **63** of them were wrong, and had the rule not changed the number would have been **7**. - In `2026-09` the number on the screen is 0.9700 and is not wrong, only **two periods old**; a indicator whose age is not written cannot be told apart from a wrong indicator. - After the gap closes, 400 predictions remain unmeasured in every period: the delay is not temporary, it is a **debt carried continuously**. - The sweep gives a linear cost: 0 unmeasured predictions at a delay of 0, 200 at 1, 400 at 2, 600 at 3; at 4 the degradation is never seen inside the monitoring window, and in the unmonitored baseline the number of measurable periods is **0**. ## Next Step Over three lessons the **seeing** of the degradation was measured, and the last measure left a gap: after the degradation happens, the only measure that sees it stays silent for **two periods**, and in that gap 400 predictions are produced unmeasured. What is at hand in this gap was never asked. The input summary and the prediction share are there; they are computed without waiting for the label, and in the previous lesson both were seen to be able to mislead. The next lesson treats these as **proxy metrics** and counts what each sees one by one: which proxy can see the concept drift, which one fires under the harmless drift, and whether the table changes when a third proxy such as input completeness is added.