Alerts and Thresholds

The previous lesson counted the deviation of three proxies and left a single number in the last column: the largest relative deviation against the 2026-08 baseline. That number decides nothing by itself. The decision comes from comparing the deviation with a band; the name of the band is the alert threshold, and it is part not of a measure but of a choice.

The choice looks cheap and is not. The narrower the band, the more periods fire; how many of the firing periods have really degraded is something the band does not know. This lesson fixes the band at 0.05, 0.10 and 0.20, runs three measures in three scenarios and writes two numbers separately: the periods that fire and how many of these correspond to real degradation.

• MO47. The source, the pipeline, the fictional model and the two drift types are fictional and are produced inside the lesson. No monitoring product, indicator service, cloud service or learning library is called. The seed is 20260218.
• MO48. The alert threshold is the share that the relative deviation of a measure from the 2026-08 baseline must not exceed. Three bands are tried: 0.05, 0.10 and 0.20. The baseline period is the last period in which none of the drifts has started.
• MO49. Real degradation is defined independently of the band: the real accuracy of a period falls more than 0.0500 below the 2026-08 baseline, that is, ten resolution steps. This measure is for evaluation; in production it is known two periods late.
• MO50. A false positive is a period that fires but is not in the real degradation list. The measurement of the delay of an indicator was done in the pipeline observability lesson of the Data Pipelines and Orchestration course and is not repeated. The only addition here is the delayed label: DELAY is added to the delay of an alert that fires on accuracy.
• MO51. The model is inherited from the MLOps Fundamentals course with its flaw corrected: the decision threshold is a separate setting and the training criterion is balanced accuracy. In the monitoring periods the base class ratio is 0.5342.
• MO52. The baseline is the unmonitored setup: no band, 0 firing periods, the detection lag is undefined. The step of period accuracy is 0.0050.

An Alert Threshold Is a Comparison

The block below builds the shared definition and adds two things to it: alert, which compares a series of a measure with the baseline, and period_seen, which says in which period the accuracy of a period can be known.

# FICTIONAL. The meter readings of the measurement network , the five-stage pipeline , the fictional model and the two drift
# types are produced inside the lesson; no monitoring product , indicator service , cloud service or
# learning library is called. Seed 20260218.
SEED, M32 = 20260218, 0xFFFFFFFF


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


REGION = ["east", "west", "north", "south", "central"]
TYPE = ["residential", "commercial"]
PERIODS = [f"2026-{a:02d}" for a in range(1, 13)]
SUBSCRIBERS = {}
for i in range(200):
    r = rng(SEED + 37 * i)
    SUBSCRIBERS[f"A{i + 1:03d}"] = {"region": REGION[int(r() * 5)], "type": TYPE[int(r() * 2)],
                                    "start": 10000 + int(r() * 60000)}


def generate_source():
    records = []
    for i, (sub, a) in enumerate(SUBSCRIBERS.items()):
        e = a["start"]
        for p in PERIODS:
            r = rng(SEED + 101 * i + 7 * PERIODS.index(p))
            e += 4 + int(r() * 46)
            records.append({"subscriber": sub, "period": p, "region": a["region"], "type": a["type"],
                            "index": e, "unit": "m3"})
    return records


def s1_screen(records):
    REQUIRED = ("subscriber", "period", "region", "index", "unit")
    keep = [k for k in records if all(a in k for a in REQUIRED)]
    return keep, len(records) - len(keep)


def s2_dedupe(records):
    seen, keep = set(), []
    for k in records:
        key = (k["subscriber"], k["period"])
        if key not in seen:
            seen.add(key)
            keep.append(k)
    return keep, len(records) - len(keep)


def s3_consumption(records):
    table = {(k["subscriber"], k["period"]): k for k in records}
    out, dropped = [], 0
    for k in records:
        i = PERIODS.index(k["period"])
        prev = table.get((k["subscriber"], PERIODS[i - 1])) if i else None
        if prev is None:
            dropped += i > 0
            continue
        out.append(dict(k, m3=int(k["index"]) - int(prev["index"])))
    return out, dropped


def rows(source=None):
    v, _ = s1_screen(generate_source() if source is None else source)
    v, _ = s2_dedupe(v)
    v, _ = s3_consumption(v)
    return v


def features(records, window=None, digits=4):
    ordered = sorted(records, key=lambda k: PERIODS.index(k["period"]))
    sub, reg = {}, {}
    for k in ordered:
        sub.setdefault(k["subscriber"], []).append((PERIODS.index(k["period"]), k["m3"]))
        reg.setdefault(k["region"], []).append((PERIODS.index(k["period"]), k["m3"]))

    def mean(series, i):
        chosen = [v for d, v in series if (d < i and (window is None or d >= i - window))]
        return round(sum(chosen) / len(chosen), digits) if chosen else 0.0

    feats = {}
    for k in records:
        i = PERIODS.index(k["period"])
        feats[(k["subscriber"], k["period"])] = {"subscriber_mean": mean(sub[k["subscriber"]], i),
                                                 "region_mean": mean(reg[k["region"]], i),
                                                 "commercial": 1 if k["type"] == "commercial" else 0}
    return feats


GRID = [(a, b, c) for a in (0.6, 0.8, 1.0) for b in (0.0, 0.2, 0.4) for c in (0, 4, 8)]
DECISION_GRID = (18, 22, 26, 30, 34)    # the decision threshold is a SEPARATE setting , searched in training
MONITORED = PERIODS[6:]                 # 2026-07 .. 2026-12 : the periods monitored in production
DELAY = 2                               # after how many periods the label arrives
RULE_THRESHOLD, RULE_COMMERCIAL = 30, 6 # CONCEPT: subscriber_mean + 6 x commercial > 30


def score(o, w):
    return w[0] * o["subscriber_mean"] + w[1] * o["region_mean"] + w[2] * o["commercial"]


def label(feat_k, threshold=RULE_THRESHOLD, commercial_factor=RULE_COMMERCIAL):
    return 1 if feat_k["subscriber_mean"] + commercial_factor * feat_k["commercial"] > threshold else 0


def predict(feat_k, w, decision):
    return 1 if score(feat_k, w) > decision else 0


def balanced_accuracy(subset, feats, w, decision):
    tp = tn = fp = fn = 0
    for k in subset:
        f_k = feats[(k["subscriber"], k["period"])]
        t, g = predict(f_k, w, decision), label(f_k)
        tp, fp = tp + (t and g), fp + (t and not g)
        fn, tn = fn + (g and not t), tn + (not t and not g)
    recall = tp / (tp + fn) if tp + fn else 0.0
    specificity = tn / (tn + fp) if tn + fp else 0.0
    return (recall + specificity) / 2, round(recall, 4), round(specificity, 4)


def train(records, feats, periods=None):
    """The training criterion is BALANCED ACCURACY: the mean of recall and specificity."""
    subset = [k for k in records if k["period"] in (periods or PERIODS[1:9])]
    best = (-1.0, None, None, 0.0, 0.0)
    for w in GRID:
        for decision in DECISION_GRID:
            o, rec, spec = balanced_accuracy(subset, feats, w, decision)
            if o > best[0]:
                best = (o, w, decision, rec, spec)
    return {"weights": best[1], "decision": best[2], "balanced": round(best[0], 4),
            "recall": best[3], "specificity": best[4], "train_rows": len(subset)}


def drift(records, start="2026-09", slope=0.06):
    """DATA DRIFT: after the start, commercial consumption grows by slope every period."""
    b = PERIODS.index(start)
    return [dict(k, m3=round(k["m3"] * (1.0 + slope * (PERIODS.index(k["period"]) - b + 1)), 4))
            if PERIODS.index(k["period"]) >= b and k["type"] == "commercial" else dict(k)
            for k in records]


def concept_drift(records, feats, start="2026-09", new_threshold=27):
    """CONCEPT DRIFT: the input is the SAME , the rule changes. The threshold drops from 30 to new_threshold."""
    b = PERIODS.index(start)
    return {(k["subscriber"], k["period"]): label(
        feats[(k["subscriber"], k["period"])],
        new_threshold if PERIODS.index(k["period"]) >= b else RULE_THRESHOLD) for k in records}


def input_summary(records, period, feats):
    d = [feats[(k["subscriber"], k["period"])]["subscriber_mean"] for k in records if k["period"] == period]
    return round(sum(d) / len(d), 4) if d else 0.0


def prediction_share(records, model, feats, period):
    d = [predict(feats[(k["subscriber"], k["period"])], model["weights"], model["decision"])
         for k in records if k["period"] == period]
    return round(sum(d) / len(d), 4) if d else 0.0


def accuracy(records, model, feats, period, labels=None):
    subset = [k for k in records if k["period"] == period]
    d = sum(1 for k in subset
            if predict(feats[(k["subscriber"], k["period"])], model["weights"], model["decision"])
            == (labels[(k["subscriber"], k["period"])] if labels
                else label(feats[(k["subscriber"], k["period"])])))
    return round(d / len(subset), 4) if subset else None


def alert(series, baseline, share):
    """A measure fires in a period if it deviates from the baseline by more than `share`."""
    return [d for d, v in series if abs(v - baseline) / max(abs(baseline), 1e-9) > share]


def period_seen(period, now):
    """Delayed label: the accuracy of a period is known only DELAY periods later."""
    return PERIODS.index(period) + DELAY <= PERIODS.index(now)


ALL_ROWS = rows()
FEATS = features(ALL_ROWS)                  # features are computed ONCE
MODEL = train(ALL_ROWS, FEATS)
MON = [k for k in ALL_ROWS if k["period"] in MONITORED]
largest = max(sum(1 for k in MON if label(FEATS[(k["subscriber"], k["period"])]) == s) for s in (0, 1))
print(f"model {MODEL['weights']} decision threshold {MODEL['decision']} , balanced accuracy"
      f" {MODEL['balanced']} , {len(MON)} rows , base class ratio {largest / len(MON):.4f} ,"
      f" step {1 / 200:.4f} , the accuracy of 2026-09"
      f" is known in period {[d for d in PERIODS if period_seen('2026-09', d)][0]}")

model (0.6, 0.0, 4) decision threshold 18 , balanced accuracy 0.9864 , 1200 rows , base class ratio 0.5342 , step 0.0050 , the accuracy of 2026-09 is known in period 2026-11

The last field says up front what the band cannot measure: even if something degrades in 2026-09, the accuracy of that period is known in 2026-11. An alert built on accuracy speaks two periods later, however narrow its band. The only thing the band can move is how many times the delay-free proxies will speak.

As the Band Narrows

The block below compares three measures in three scenarios with three bands. Each cell carries two numbers: the periods that fire and how many of them are false positives. The degraded column is independent of the band; it gives the number of periods that have really degraded in that scenario.

# This block continues the definitions of the first block. The features of each scenario are computed ONCE.
BAND = (0.05, 0.10, 0.20)
DROP = 0.0500         # REAL DEGRADATION: accuracy falls below the baseline by more than this
DRIFTED = drift(ALL_ROWS)
FEATS_D = features(DRIFTED)
CONCEPT = concept_drift(ALL_ROWS, FEATS)
SCENARIOS = (("sound", ALL_ROWS, FEATS, None), ("data drift", DRIFTED, FEATS_D, None),
             ("concept drift", ALL_ROWS, FEATS, CONCEPT))
MEASURES = (("input summary", lambda src, feats, lab, d: input_summary(src, d, feats)),
            ("prediction share", lambda src, feats, lab, d: prediction_share(src, MODEL, feats, d)),
            ("accuracy", lambda src, feats, lab, d: accuracy(src, MODEL, feats, d, lab)))
FIRE, REAL = {}, {}
for sname, src, feats, lab in SCENARIOS:
    acc_series = [(d, accuracy(src, MODEL, feats, d, lab)) for d in MONITORED]
    REAL[sname] = set(d for d, v in acc_series[2:] if dict(acc_series)["2026-08"] - v > DROP)
    for name, f in MEASURES:
        series = [(d, f(src, feats, lab, d)) for d in MONITORED]
        for b in BAND:
            FIRE[(sname, name, b)] = set(alert(series[2:], dict(series)["2026-08"], b))

print(f"real degradation: accuracy falling more than {DROP:.4f} below its 2026-08 baseline")
print("cell: periods that fire / false positives\n")
print("measure          scenario        degraded " + "".join(f"{b:>12.2f}" for b in BAND))
for name, _ in MEASURES:
    for sname, *_ in SCENARIOS:
        cells = [f"{len(FIRE[(sname, name, b)])} / {len(FIRE[(sname, name, b)] - REAL[sname])}"
                 for b in BAND]
        print(f"{name:<16} {sname:<14} {len(REAL[sname]):>8} " + "".join(f"{x:>12}" for x in cells))

print("\nband  total firings false positives   caught delay: prediction share  accuracy")
for b in BAND:
    totals = [sum(len(FIRE[(s, a, b)]) for s, *_ in SCENARIOS for a, _ in MEASURES),
              sum(len(FIRE[(s, a, b)] - REAL[s]) for s, *_ in SCENARIOS for a, _ in MEASURES),
              sum(len(FIRE[(s, a, b)] & REAL[s]) for s, *_ in SCENARIOS for a, _ in MEASURES)]
    delays = []
    for name, extra in (("prediction share", 0), ("accuracy", DELAY)):
        fired = sorted(FIRE[("concept drift", name, b)])
        delays.append(str(PERIODS.index(fired[0]) - PERIODS.index("2026-09") + extra) if fired else "undefined")
    print(f"{b:<5.2f} {totals[0]:>13} {totals[1]:>15} {totals[2]:>8} {delays[0]:>23} {delays[1]:>9}")

real degradation: accuracy falling more than 0.0500 below its 2026-08 baseline
cell: periods that fire / false positives

measure          scenario        degraded         0.05        0.10        0.20
input summary    sound                 0        0 / 0       0 / 0       0 / 0
input summary    data drift            0        0 / 0       0 / 0       0 / 0
input summary    concept drift         4        0 / 0       0 / 0       0 / 0
prediction share sound                 0        3 / 3       0 / 0       0 / 0
prediction share data drift            0        3 / 3       2 / 2       0 / 0
prediction share concept drift         4        3 / 0       0 / 0       0 / 0
accuracy         sound                 0        0 / 0       0 / 0       0 / 0
accuracy         data drift            0        0 / 0       0 / 0       0 / 0
accuracy         concept drift         4        4 / 0       4 / 0       0 / 0

band  total firings false positives   caught delay: prediction share  accuracy
0.05             13               6        7                       1         2
0.10              6               2        4               undefined         2
0.20              0               0        0               undefined undefined

Where the Catch Count Is Invalid

The first row of the lower table seems to defend the band: 13 firings at 0.05, 7 of them corresponding to real degradation. This number is invalid, and the reason is in the upper table. Look at the prediction share rows: in the sound setup it fires in 3 periods, and in the concept drift setup again in exactly 3 periods. The same measure, the same band, the same number of firings; the only difference is that in the second case those periods have degraded. The firing does not come from the degradation, it coincides in time with the degradation. The sound row proves it: when nothing is broken, the same three periods speak too.

This is the exact statement of the rule of this course: an indicator whose firing and whose degradation are not shown separately counts as unmeasured. Without the sound row, the prediction share at the 0.05 band would read like a flawless indicator, 3 firings and 0 false positives. With the sound row, 3 of the 7 catches fall away and 4 remain. Only one measure provides those four: accuracy, the only row that never fires in the two harmless scenarios at the 0.05 and 0.10 bands and fires in four of four periods in the harmful scenario. Its cost is in the last column: the detection lag is 2 periods, and it cannot be reduced by choosing a band.

The Trade Between False Positives and Blindness

Widening the band does not fix the count, it zeroes it. Going from 0.05 to 0.10, the false positives fall from 6 to 2 and the catches from 7 to 4, but the three catches that drop are the prediction share catches that were already invalid; that is, the 0.10 band throws away two thirds of the false positives without losing any real information. This is the only step in which widening the band is free.

The 0.20 row, on the other hand, is three zeros: no false positive, no firing, no catch. The meaning of the band on accuracy can be computed: 0.80 times the 0.9700 baseline is 0.7760, so a band of 0.20 expects accuracy to fall below this value. The lowest value of the concept drift is 0.8150, and it stays 0.0390 above 0.7760, about eight resolution steps. The band is placed where the measure cannot reach; its silence is not a sign of health, it is by definition a dead indicator.

Reading the three bands together gives the second claim of the course: monitoring breaks things too. The narrow band raises six alarms in a production where nothing has degraded; the wide band never speaks in a production where accuracy fell from 0.9700 to 0.8150. The band in between is a good choice only in this fiction, because we know which firing is real by looking at the real accuracy; in production that information arrives two periods later and the band is chosen without it.

Summary

• The alert threshold is not a measure but a choice: it is the share that the relative deviation from the 2026-08 baseline must not exceed, and it directly determines the number of firing periods.
• At the 0.05 band, across three measures and three scenarios, 13 periods fire and 6 of them are false positives; at 0.10 there are 6 firings and 2 false positives; at 0.20 all three are zero.
• 3 of the 7 catches at 0.05 are invalid: the prediction share fires in exactly 3 periods in the sound setup as well. An indicator whose firing and whose degradation are not shown separately counts as unmeasured; the valid catch is 4, and all of it comes from accuracy.
• Accuracy is the only measure that never fires in the two harmless scenarios and fires in four of four periods in the harmful scenario; its cost is a detection lag of 2 periods, and it cannot be reduced by the choice of band.
• The 0.20 band asks accuracy to fall below 0.7760; the lowest value is 0.8150, so the band sits where the measure cannot reach and its silence is not a sign of health.

Next Step

The only valid indicator of this lesson was accuracy, and accuracy needs a label. Until now the label was a gift of the fiction: the real class of every row was known, it only arrived two periods late. In production it is not so; the label is collected, and what is collected does not arrive for all predictions. The next lesson asks where the label comes from: when the collection procedure covers only a part of the predictions, it counts how far the measured accuracy deviates from the real accuracy, and which form of collection grows the bias while it grows the coverage.