Proxy Metrics

The previous lesson counted the delayed label: the accuracy of a period is known only after DELAY = 2 periods, there is a gap of two periods between the moment of degradation and the moment it is seen, and predictions keep being produced in that gap. Over three lessons it was measured that the degradation is seen two periods late.

The question not asked is what is at hand in those two periods. There is no accuracy, because the label has not arrived. Only the measures that need no label remain: those that can be read from the input itself and from the output of the model. These are called proxy metrics, because they are not the thing we want to measure itself, but something assumed to be related to it. This lesson puts three proxies side by side in three scenarios and asks a single question: which one sees concept drift, and which one fires for something it does not see.

• MO41. The source, the five-stage pipeline, the fictional model and the two drift types are fictional and are produced inside the lesson. No monitoring product, indicator service, cloud service or learning library is called. The seed is 20260218.
• MO42. The three proxies are: the input summary, the mean of the subscriber means in the period; the prediction share, the share of rows the model calls 1; and the input completeness, the ratio of the rows formed in the period to the expected 200 rows. All three read no label, so they are free of delay.
• MO43. Three scenarios: sound; data drift, where after 2026-09 commercial consumption grows by 0.06 per period; and concept drift, where the input stays the same and the threshold of the rule subscriber_mean + 6 x commercial > 30 drops to 27.
• MO44. The drift procedure, its kinds, their measurement and their correction, was measured in the Model Evaluation, Interpretability and Ethics course of the Machine Learning and Data Science curriculum; it is not repeated. The addition here belongs only to the life in production: what the proxy sees when there is no label.
• MO45. The model is inherited from the MLOps Fundamentals course with its flaw corrected: the decision threshold is a separate setting and is searched in training, the training criterion is balanced accuracy, and the label is tied to a rule the features see. In the monitoring periods the base class ratio is 0.5342.
• MO46. The baseline is the unmonitored setup: no indicator, no alert, the degradation goes unseen. The step of period accuracy is 0.0050, and a difference below this step is not read.

What Is at Hand Before the Label Arrives

The block below builds the shared definition, which is the source, the pipeline, the features, the fictional model, the two drift types and the three measures, and trains the model and prints the baseline. The features are computed once per scenario.

# FICTIONAL. The meter readings of the measurement network , the five-stage pipeline , the fictional model and the two drift
# types are produced inside the lesson; no monitoring product , indicator service , cloud service or
# learning library is called. Seed 20260218.
SEED, M32 = 20260218, 0xFFFFFFFF


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


REGION = ["east", "west", "north", "south", "central"]
TYPE = ["residential", "commercial"]
PERIODS = [f"2026-{a:02d}" for a in range(1, 13)]
SUBSCRIBERS = {}
for i in range(200):
    r = rng(SEED + 37 * i)
    SUBSCRIBERS[f"A{i + 1:03d}"] = {"region": REGION[int(r() * 5)], "type": TYPE[int(r() * 2)],
                                    "start": 10000 + int(r() * 60000)}


def generate_source():
    records = []
    for i, (sub, a) in enumerate(SUBSCRIBERS.items()):
        e = a["start"]
        for p in PERIODS:
            r = rng(SEED + 101 * i + 7 * PERIODS.index(p))
            e += 4 + int(r() * 46)
            records.append({"subscriber": sub, "period": p, "region": a["region"], "type": a["type"],
                            "index": e, "unit": "m3"})
    return records


def s1_screen(records):
    REQUIRED = ("subscriber", "period", "region", "index", "unit")
    keep = [k for k in records if all(a in k for a in REQUIRED)]
    return keep, len(records) - len(keep)


def s2_dedupe(records):
    seen, keep = set(), []
    for k in records:
        key = (k["subscriber"], k["period"])
        if key not in seen:
            seen.add(key)
            keep.append(k)
    return keep, len(records) - len(keep)


def s3_consumption(records):
    table = {(k["subscriber"], k["period"]): k for k in records}
    out, dropped = [], 0
    for k in records:
        i = PERIODS.index(k["period"])
        prev = table.get((k["subscriber"], PERIODS[i - 1])) if i else None
        if prev is None:
            dropped += i > 0
            continue
        out.append(dict(k, m3=int(k["index"]) - int(prev["index"])))
    return out, dropped


def rows(source=None):
    v, _ = s1_screen(generate_source() if source is None else source)
    v, _ = s2_dedupe(v)
    v, _ = s3_consumption(v)
    return v


def features(records, window=None, digits=4):
    ordered = sorted(records, key=lambda k: PERIODS.index(k["period"]))
    sub, reg = {}, {}
    for k in ordered:
        sub.setdefault(k["subscriber"], []).append((PERIODS.index(k["period"]), k["m3"]))
        reg.setdefault(k["region"], []).append((PERIODS.index(k["period"]), k["m3"]))

    def mean(series, i):
        chosen = [v for d, v in series if (d < i and (window is None or d >= i - window))]
        return round(sum(chosen) / len(chosen), digits) if chosen else 0.0

    feats = {}
    for k in records:
        i = PERIODS.index(k["period"])
        feats[(k["subscriber"], k["period"])] = {"subscriber_mean": mean(sub[k["subscriber"]], i),
                                                 "region_mean": mean(reg[k["region"]], i),
                                                 "commercial": 1 if k["type"] == "commercial" else 0}
    return feats


GRID = [(a, b, c) for a in (0.6, 0.8, 1.0) for b in (0.0, 0.2, 0.4) for c in (0, 4, 8)]
DECISION_GRID = (18, 22, 26, 30, 34)    # the decision threshold is a SEPARATE setting , searched in training
MONITORED = PERIODS[6:]                 # 2026-07 .. 2026-12 : the periods monitored in production
DELAY = 2                               # after how many periods the label arrives
RULE_THRESHOLD, RULE_COMMERCIAL = 30, 6 # CONCEPT: subscriber_mean + 6 x commercial > 30


def score(o, w):
    return w[0] * o["subscriber_mean"] + w[1] * o["region_mean"] + w[2] * o["commercial"]


def label(feat_k, threshold=RULE_THRESHOLD, commercial_factor=RULE_COMMERCIAL):
    return 1 if feat_k["subscriber_mean"] + commercial_factor * feat_k["commercial"] > threshold else 0


def predict(feat_k, w, decision):
    return 1 if score(feat_k, w) > decision else 0


def balanced_accuracy(subset, feats, w, decision):
    tp = tn = fp = fn = 0
    for k in subset:
        f_k = feats[(k["subscriber"], k["period"])]
        t, g = predict(f_k, w, decision), label(f_k)
        tp, fp = tp + (t and g), fp + (t and not g)
        fn, tn = fn + (g and not t), tn + (not t and not g)
    recall = tp / (tp + fn) if tp + fn else 0.0
    specificity = tn / (tn + fp) if tn + fp else 0.0
    return (recall + specificity) / 2, round(recall, 4), round(specificity, 4)


def train(records, feats, periods=None):
    """The training criterion is BALANCED ACCURACY: the mean of recall and specificity."""
    subset = [k for k in records if k["period"] in (periods or PERIODS[1:9])]
    best = (-1.0, None, None, 0.0, 0.0)
    for w in GRID:
        for decision in DECISION_GRID:
            o, rec, spec = balanced_accuracy(subset, feats, w, decision)
            if o > best[0]:
                best = (o, w, decision, rec, spec)
    return {"weights": best[1], "decision": best[2], "balanced": round(best[0], 4),
            "recall": best[3], "specificity": best[4], "train_rows": len(subset)}


def drift(records, start="2026-09", slope=0.06):
    """DATA DRIFT: after the start, commercial consumption grows by slope every period."""
    b = PERIODS.index(start)
    return [dict(k, m3=round(k["m3"] * (1.0 + slope * (PERIODS.index(k["period"]) - b + 1)), 4))
            if PERIODS.index(k["period"]) >= b and k["type"] == "commercial" else dict(k)
            for k in records]


def concept_drift(records, feats, start="2026-09", new_threshold=27):
    """CONCEPT DRIFT: the input is the SAME , the rule changes. The threshold drops from 30 to new_threshold."""
    b = PERIODS.index(start)
    return {(k["subscriber"], k["period"]): label(
        feats[(k["subscriber"], k["period"])],
        new_threshold if PERIODS.index(k["period"]) >= b else RULE_THRESHOLD) for k in records}


def input_summary(records, period, feats):
    d = [feats[(k["subscriber"], k["period"])]["subscriber_mean"] for k in records if k["period"] == period]
    return round(sum(d) / len(d), 4) if d else 0.0


def prediction_share(records, model, feats, period):
    d = [predict(feats[(k["subscriber"], k["period"])], model["weights"], model["decision"])
         for k in records if k["period"] == period]
    return round(sum(d) / len(d), 4) if d else 0.0


def accuracy(records, model, feats, period, labels=None):
    subset = [k for k in records if k["period"] == period]
    d = sum(1 for k in subset
            if predict(feats[(k["subscriber"], k["period"])], model["weights"], model["decision"])
            == (labels[(k["subscriber"], k["period"])] if labels
                else label(feats[(k["subscriber"], k["period"])])))
    return round(d / len(subset), 4) if subset else None


ALL_ROWS = rows()
FEATS = features(ALL_ROWS)                  # features are computed ONCE
MODEL = train(ALL_ROWS, FEATS)
MON = [k for k in ALL_ROWS if k["period"] in MONITORED]
largest = max(sum(1 for k in MON if label(FEATS[(k["subscriber"], k["period"])]) == s) for s in (0, 1))
print(f"model {MODEL['weights']} decision threshold {MODEL['decision']} , balanced accuracy"
      f" {MODEL['balanced']} , recall {MODEL['recall']} , specificity {MODEL['specificity']}")
print(f"monitoring {len(MONITORED)} periods , {len(MON)} rows , commercial subscribers"
      f" {sum(1 for a in SUBSCRIBERS.values() if a['type'] == 'commercial')} , base class ratio"
      f" {largest / len(MON):.4f} , step of period accuracy {1 / 200:.4f}")

model (0.6, 0.0, 4) decision threshold 18 , balanced accuracy 0.9864 , recall 1.0 , specificity 0.9728
monitoring 6 periods , 1200 rows , commercial subscribers 102 , base class ratio 0.5342 , step of period accuracy 0.0050

Because the base class ratio is 0.5342, the set is balanced: a procedure that always says the same class gets this ratio, and the 0.9864 of the model is far above it. The model is in a monitorable state. The last two numbers of the second line bound the whole lesson: the subscribers the drift touches number 102, the remaining 98 are untouched; and because the step of period accuracy is 0.0050, no movement smaller than this is read.

Three Proxies, Three Scenarios

The block below computes the three measures over the six monitoring periods of the three scenarios and adds accuracy as the fourth row. The last column is the largest relative deviation seen in the following four periods against the 2026-08 baseline; an alert band is compared with exactly this number.

# This block continues the definitions of the first block. The features of each scenario are computed ONCE.
def completeness(records, period, expected=200):
    """THIRD PROXY: how many rows formed in the period. It reads no value , it counts presence."""
    return round(sum(1 for k in records if k["period"] == period) / expected, 4)


DRIFTED = drift(ALL_ROWS)
FEATS_D = features(DRIFTED)
CONCEPT = concept_drift(ALL_ROWS, FEATS)
SCENARIOS = (("sound", ALL_ROWS, FEATS, None), ("data drift", DRIFTED, FEATS_D, None),
             ("concept drift", ALL_ROWS, FEATS, CONCEPT))
MEASURES = (("input summary", lambda src, feats, lab, d: input_summary(src, d, feats)),
            ("prediction share", lambda src, feats, lab, d: prediction_share(src, MODEL, feats, d)),
            ("completeness", lambda src, feats, lab, d: completeness(src, d)),
            ("accuracy", lambda src, feats, lab, d: accuracy(src, MODEL, feats, d, lab)))

print("measure          scenario      " + "".join(f"{d[-2:]:>8}" for d in MONITORED) + "  deviation")
for name, f in MEASURES:
    for sname, src, feats, lab in SCENARIOS:
        series = [f(src, feats, lab, d) for d in MONITORED]
        deviation = max(abs(v - series[1]) / series[1] for v in series[2:])
        print(f"{name:<16} {sname:<14}" + "".join(f"{v:8.4f}" for v in series) + f"{deviation:>11.4f}")

RAW = [k for k in generate_source() if not (k["subscriber"] <= "A040" and k["period"] == "2026-10")]
LATE = rows(RAW)
print("\ncheck , the 2026-10 reading of 40 subscribers never arrived:")
print("completeness " + "".join(f"{completeness(LATE, d):8.4f}" for d in MONITORED))

measure          scenario            07      08      09      10      11      12  deviation
input summary    sound          25.9220 26.0517 25.9800 26.3525 26.3261 26.5325     0.0185
input summary    data drift     25.9220 26.0517 25.9800 26.4685 26.6053 27.0472     0.0382
input summary    concept drift  25.9220 26.0517 25.9800 26.3525 26.3261 26.5325     0.0185
prediction share sound           0.4550  0.4700  0.4550  0.5100  0.5100  0.5150     0.0957
prediction share data drift      0.4550  0.4700  0.4550  0.5150  0.5200  0.5450     0.1596
prediction share concept drift   0.4550  0.4700  0.4550  0.5100  0.5100  0.5150     0.0957
completeness     sound           1.0000  1.0000  1.0000  1.0000  1.0000  1.0000     0.0000
completeness     data drift      1.0000  1.0000  1.0000  1.0000  1.0000  1.0000     0.0000
completeness     concept drift   1.0000  1.0000  1.0000  1.0000  1.0000  1.0000     0.0000
accuracy         sound           0.9700  0.9700  0.9900  0.9750  0.9850  0.9900     0.0206
accuracy         data drift      0.9700  0.9700  0.9900  0.9750  0.9800  0.9800     0.0206
accuracy         concept drift   0.9700  0.9700  0.8150  0.8700  0.8650  0.8500     0.1598

check , the 2026-10 reading of 40 subscribers never arrived:
completeness   1.0000  1.0000  1.0000  0.8000  0.8000  1.0000

The hardest part of the table is the comparison of the sound and concept drift rows. In these two scenarios all three proxies give the digit-for-digit same number: 26.5325 and 26.5325, 0.5150 and 0.5150, 1.0000 and 1.0000. This is not an approximation, it is an identity. By definition, concept drift does not touch the input, it only changes the rule; and no measure read from the input can carry that change. In the same periods accuracy goes down from 0.9900 to 0.8500, and in the last column the deviation rises from 0.0206 to 0.1598. None of the three proxies sees concept drift.

The Proxy That Fires and the Proxy That Stays Silent

The prediction share row gives the second reading. Its deviation is 0.0957 even in the sound setup; that is, in a production where nothing has degraded, a band of 0.05 fires. In the data-drift setup it rises to 0.1596 and exceeds the 0.10 band too, and in exactly that setup accuracy rises from 0.9700 to 0.9800. The proxy gives its loudest alarm in the scenario where performance improves. This is the barest form of the rule of the course: the firing of an indicator is not evidence of degradation. The cause is visible too: as long as the rule does not change, a correct model stays correct on a drifted input, it only says 1 to more rows.

The input summary surprises in the opposite direction. In the only scenario where the input really drifts, its deviation stays at 0.0382 and does not exceed 0.05. The number is not wrong: the drift touches 102 of 200 subscribers, the mean divides this among the remaining 98 subscribers, and the movement from 25.9220 to 27.0472 stays under the band. What a proxy looks at is part of the measure as much as which breakdown it looks at; an unbroken-down mean dilutes the drift as the segment it touches gets smaller.

The input completeness deviates by 0.0000 in all three scenarios. This does not mean the measure is broken; the check run separates the two. When the 2026-10 reading of 40 subscribers never arrives in the source, the completeness drops two periods at once to 0.8000, which is the late_arrival class of the Introduction to Data Engineering course, and the reason two periods drop is that consumption depends on the index of the previous period. The measure works; it only sees another class of degradation. No kind of drift drops rows, so in drift monitoring the completeness pays the cost of being collected and gives zero information in return.

Reading the four rows together is the answer of the lesson. The only measure that separates the scenarios is accuracy: 0.1598 against 0.0206, that is, thirty-one resolution steps. But accuracy needs a label, and the label arrives two periods later. There are three measures at hand that are free of delay, and all three fail to separate; the one measure that separates is delayed. A proxy metric does not remove the cost of delay.

Summary

• The three proxies, the input summary, the prediction share and the input completeness, read no label and are free of delay; so they are the only class of measures left at hand in the two-period gap that the delayed label leaves.
• In the concept-drift scenario all three proxies give the digit-for-digit same number as in the sound scenario (26.5325 / 0.5150 / 1.0000), because the rule changes and the input does not. The deviations are 0.0185, 0.0957 and 0.0000; none of them sees concept drift.
• The prediction share deviates by 0.0957 in the sound setup and exceeds the 0.05 band; in the data-drift setup it rises to 0.1596, and in that setup accuracy goes from 0.9700 to 0.9800. This is the proxy that fires under a harmless drift.
• The input summary stays at 0.0382 and does not fire in the only scenario where the input really drifts: the drift touches 102 of 200 subscribers and the unbroken-down mean dilutes it.
• The input completeness deviates by 0.0000 in all three scenarios but is not broken: when the reading of 40 subscribers for one period does not arrive, it drops two periods at once to 0.8000. The drift does not drop rows, so the proxy gives no measurable contribution here.
• The only measure that separates the scenarios is accuracy (0.1598 against 0.0206, thirty-one resolution steps), and it too arrives two periods later.

Next Step

The last column of this lesson was a deviation number, and that number does nothing by itself; compared with a band it turns into a decision. At 0.05 the prediction share speaks in all three scenarios, at 0.20 it speaks in none. The next lesson makes the band variable: at 0.05, 0.10 and 0.20 it counts in how many periods the three measures fire, how many of these firings correspond to real degradation and how many are false positives; in the end it is seen that the band that gives no false positive at all also misses the only real degradation.