Feedback Collection

The previous lesson counted three measures at three bands and left a single valid indicator: accuracy. It was the only measure that never fired in the two harmless scenarios and fired in four of four periods in the harmful scenario. But accuracy needs a label, and until now the label was a gift of the fiction: the real class of every row was known, it only arrived two periods late.

In production the label is not a gift, it is collected. And what is collected rarely covers every prediction: a review goes only to flagged records, an objection comes only from the person who is bothered, a verification is requested only in the cases left undecided. This lesson runs five collection procedures on the same set and writes two numbers separately: the coverage, that is, the share of rows whose label arrives; and the deviation, that is, the difference between the accuracy measured with those labels and the real accuracy.

• MO53. The source, the pipeline, the fictional model, the concept drift and the five collection procedures are fictional and are produced inside the lesson. No monitoring product, labeling service, cloud service or learning library is called. The seed is 20260218.
• MO54. The collection procedure is the rule that determines the label of which prediction will arrive. Five are tried: all; positives only, the rows the model calls 1; negatives only, the rows it calls 0; near threshold, the rows whose score is within MARGIN of the decision threshold; every fourth, every fourth row in the sequence.
• MO55. The coverage is the ratio of the rows whose label arrives to the 1200 rows of the monitoring periods. The deviation is the real accuracy, measured with all rows, subtracted from the accuracy measured with the collected labels. Its direction is meaningful: a plus sign says the model looks better than it is.
• MO56. The return of the user signal to evaluation and the bias of that signal were measured in the feedback loop lesson of the AI Safety and Evaluation course in the AI Engineering curriculum; the procedure is not repeated. The measure there was for language model output; the addition here is the delayed label of a numeric model.
• MO57. The model is inherited from the MLOps Fundamentals course with its flaw corrected: the decision threshold is a separate setting and the training criterion is balanced accuracy. In the monitoring periods the base class ratio is 0.5342.
• MO58. The baseline is the no-collection setup: no label arrives, there is no measured accuracy, the deviation is undefined. The step of period accuracy is 0.0050, and a deviation smaller than this is not read.

Where the Label Comes From

The block below builds the shared definition and trains the model. The last line gives the limits of the lesson: over how many rows we measure, what the resolution is below which we cannot go, and how many periods later the label arrives.

# FICTIONAL. The meter readings of the measurement network , the five-stage pipeline , the fictional model , the concept
# drift and the five collection procedures are produced inside the lesson; no monitoring product ,
# labeling service , cloud service or learning library is called. Seed 20260218.
SEED, M32 = 20260218, 0xFFFFFFFF


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


REGION = ["east", "west", "north", "south", "central"]
TYPE = ["residential", "commercial"]
PERIODS = [f"2026-{a:02d}" for a in range(1, 13)]
SUBSCRIBERS = {}
for i in range(200):
    r = rng(SEED + 37 * i)
    SUBSCRIBERS[f"A{i + 1:03d}"] = {"region": REGION[int(r() * 5)], "type": TYPE[int(r() * 2)],
                                    "start": 10000 + int(r() * 60000)}


def generate_source():
    records = []
    for i, (sub, a) in enumerate(SUBSCRIBERS.items()):
        e = a["start"]
        for p in PERIODS:
            r = rng(SEED + 101 * i + 7 * PERIODS.index(p))
            e += 4 + int(r() * 46)
            records.append({"subscriber": sub, "period": p, "region": a["region"], "type": a["type"],
                            "index": e, "unit": "m3"})
    return records


def s1_screen(records):
    REQUIRED = ("subscriber", "period", "region", "index", "unit")
    keep = [k for k in records if all(a in k for a in REQUIRED)]
    return keep, len(records) - len(keep)


def s2_dedupe(records):
    seen, keep = set(), []
    for k in records:
        key = (k["subscriber"], k["period"])
        if key not in seen:
            seen.add(key)
            keep.append(k)
    return keep, len(records) - len(keep)


def s3_consumption(records):
    table = {(k["subscriber"], k["period"]): k for k in records}
    out, dropped = [], 0
    for k in records:
        i = PERIODS.index(k["period"])
        prev = table.get((k["subscriber"], PERIODS[i - 1])) if i else None
        if prev is None:
            dropped += i > 0
            continue
        out.append(dict(k, m3=int(k["index"]) - int(prev["index"])))
    return out, dropped


def rows(source=None):
    v, _ = s1_screen(generate_source() if source is None else source)
    v, _ = s2_dedupe(v)
    v, _ = s3_consumption(v)
    return v


def features(records, window=None, digits=4):
    ordered = sorted(records, key=lambda k: PERIODS.index(k["period"]))
    sub, reg = {}, {}
    for k in ordered:
        sub.setdefault(k["subscriber"], []).append((PERIODS.index(k["period"]), k["m3"]))
        reg.setdefault(k["region"], []).append((PERIODS.index(k["period"]), k["m3"]))

    def mean(series, i):
        chosen = [v for d, v in series if (d < i and (window is None or d >= i - window))]
        return round(sum(chosen) / len(chosen), digits) if chosen else 0.0

    feats = {}
    for k in records:
        i = PERIODS.index(k["period"])
        feats[(k["subscriber"], k["period"])] = {"subscriber_mean": mean(sub[k["subscriber"]], i),
                                                 "region_mean": mean(reg[k["region"]], i),
                                                 "commercial": 1 if k["type"] == "commercial" else 0}
    return feats


GRID = [(a, b, c) for a in (0.6, 0.8, 1.0) for b in (0.0, 0.2, 0.4) for c in (0, 4, 8)]
DECISION_GRID = (18, 22, 26, 30, 34)    # the decision threshold is a SEPARATE setting , searched in training
MONITORED = PERIODS[6:]                 # 2026-07 .. 2026-12 : the periods monitored in production
DELAY = 2                               # after how many periods the label arrives
RULE_THRESHOLD, RULE_COMMERCIAL = 30, 6 # CONCEPT: subscriber_mean + 6 x commercial > 30


def score(o, w):
    return w[0] * o["subscriber_mean"] + w[1] * o["region_mean"] + w[2] * o["commercial"]


def label(feat_k, threshold=RULE_THRESHOLD, commercial_factor=RULE_COMMERCIAL):
    return 1 if feat_k["subscriber_mean"] + commercial_factor * feat_k["commercial"] > threshold else 0


def predict(feat_k, w, decision):
    return 1 if score(feat_k, w) > decision else 0


def balanced_accuracy(subset, feats, w, decision):
    tp = tn = fp = fn = 0
    for k in subset:
        f_k = feats[(k["subscriber"], k["period"])]
        t, g = predict(f_k, w, decision), label(f_k)
        tp, fp = tp + (t and g), fp + (t and not g)
        fn, tn = fn + (g and not t), tn + (not t and not g)
    recall = tp / (tp + fn) if tp + fn else 0.0
    specificity = tn / (tn + fp) if tn + fp else 0.0
    return (recall + specificity) / 2, round(recall, 4), round(specificity, 4)


def train(records, feats, periods=None):
    """The training criterion is BALANCED ACCURACY: the mean of recall and specificity."""
    subset = [k for k in records if k["period"] in (periods or PERIODS[1:9])]
    best = (-1.0, None, None, 0.0, 0.0)
    for w in GRID:
        for decision in DECISION_GRID:
            o, rec, spec = balanced_accuracy(subset, feats, w, decision)
            if o > best[0]:
                best = (o, w, decision, rec, spec)
    return {"weights": best[1], "decision": best[2], "balanced": round(best[0], 4),
            "recall": best[3], "specificity": best[4], "train_rows": len(subset)}


def concept_drift(records, feats, start="2026-09", new_threshold=27):
    """CONCEPT DRIFT: the input is the SAME , the rule changes. The threshold drops from 30 to new_threshold."""
    b = PERIODS.index(start)
    return {(k["subscriber"], k["period"]): label(
        feats[(k["subscriber"], k["period"])],
        new_threshold if PERIODS.index(k["period"]) >= b else RULE_THRESHOLD) for k in records}


def alert(series, baseline, share):
    """A measure fires in a period if it deviates from the baseline by more than `share`."""
    return [d for d, v in series if abs(v - baseline) / max(abs(baseline), 1e-9) > share]


ALL_ROWS = rows()
FEATS = features(ALL_ROWS)                  # features are computed ONCE
MODEL = train(ALL_ROWS, FEATS)
MON = [k for k in ALL_ROWS if k["period"] in MONITORED]
largest = max(sum(1 for k in MON if label(FEATS[(k["subscriber"], k["period"])]) == s) for s in (0, 1))
print(f"model {MODEL['weights']} decision threshold {MODEL['decision']} , balanced accuracy"
      f" {MODEL['balanced']} , recall {MODEL['recall']} , monitoring {len(MON)} rows ,"
      f" base class ratio {largest / len(MON):.4f} , step {1 / 200:.4f} , label delay"
      f" {DELAY} periods")

model (0.6, 0.0, 4) decision threshold 18 , balanced accuracy 0.9864 , recall 1.0 , monitoring 1200 rows , base class ratio 0.5342 , step 0.0050 , label delay 2 periods

Coverage and Deviation

The block below runs the five procedures in two scenarios. The upper table gives, for the whole monitoring window, the coverage, the accuracy measured with the collected labels, the real accuracy measured with all rows, and the difference between the two. The lower table counts, only in the concept-drift production, the accuracy measured period by period and how many periods an alert of 0.05 built on the drop fires.

# This block continues the definitions of the first block. The features are computed ONCE.
MARGIN = 1.0          # scores this close to the decision threshold count as "near threshold"
CONCEPT = concept_drift(ALL_ROWS, FEATS)
W, DEC = MODEL["weights"], MODEL["decision"]
PROCEDURES = (("all", lambda o, j: True),
              ("positives only", lambda o, j: predict(o, W, DEC) == 1),
              ("negatives only", lambda o, j: predict(o, W, DEC) == 0),
              ("near threshold", lambda o, j: abs(score(o, W) - DEC) <= MARGIN),
              ("every fourth", lambda o, j: j % 4 == 0))


def measure(labels, select, periods_set):
    """Only the label of the row that select picks arrives; accuracy is measured WITH THEM."""
    total = correct = all_n = all_correct = 0
    for j, k in enumerate(ALL_ROWS):
        if k["period"] not in periods_set:
            continue
        o = FEATS[(k["subscriber"], k["period"])]
        d = predict(o, W, DEC) == (labels[(k["subscriber"], k["period"])] if labels
                                   else label(o))
        all_n, all_correct = all_n + 1, all_correct + d
        if select(o, j):
            total, correct = total + 1, correct + d
    return round(total / all_n, 4), round(correct / total, 4), round(all_correct / all_n, 4)


print("procedure        scenario       coverage  measured    real  deviation")
for name, select in PROCEDURES:
    for sname, lab in (("sound", None), ("concept drift", CONCEPT)):
        cov, measured, real = measure(lab, select, MONITORED)
        print(f"{name:<16} {sname:<14} {cov:>8.4f}  {measured:>8.4f}  {real:>6.4f}  {measured - real:>+9.4f}")

print("\nconcept drift , accuracy measured period by period and firing on the DROP at the 0.05 band")
for name, select in PROCEDURES:
    series = [(d, measure(CONCEPT, select, [d])[1]) for d in MONITORED]
    baseline = dict(series)["2026-08"]
    fired = [d for d, v in series[2:] if (baseline - v) / baseline > 0.05]
    print(f"{name:<16}" + "".join(f"{v:8.4f}" for _, v in series) + f"   fires {len(fired)}")

procedure        scenario       coverage  measured    real  deviation
all              sound            1.0000    0.9800  0.9800    +0.0000
all              concept drift    1.0000    0.8900  0.8900    +0.0000
positives only   sound            0.4858    0.9588  0.9800    -0.0212
positives only   concept drift    0.4858    0.9794  0.8900    +0.0894
negatives only   sound            0.5142    1.0000  0.9800    +0.0200
negatives only   concept drift    0.5142    0.8055  0.8900    -0.0845
near threshold   sound            0.1850    0.8919  0.9800    -0.0881
near threshold   concept drift    0.1850    0.5991  0.8900    -0.2909
every fourth     sound            0.2500    0.9833  0.9800    +0.0033
every fourth     concept drift    0.2500    0.9000  0.8900    +0.0100

concept drift , accuracy measured period by period and firing on the DROP at the 0.05 band
all               0.9700  0.9700  0.8150  0.8700  0.8650  0.8500   fires 4
positives only    0.9341  0.9362  1.0000  1.0000  1.0000  1.0000   fires 0
negatives only    1.0000  1.0000  0.6606  0.7347  0.7245  0.6907   fires 4
near threshold    0.8286  0.8537  0.3824  0.5526  0.4857  0.4615   fires 4
every fourth      0.9800  0.9800  0.8600  0.8600  0.8200  0.9000   fires 4

The Collection Procedure Chooses the Measure Too

The positives only row is the hardest finding of the lesson. Its coverage is 0.4858, so it sees almost half of the predictions; this is not a small sample. In the concept-drift production the accuracy it measures is 0.9794, the real accuracy is 0.8900; the deviation is +0.0894, eighteen resolution steps, and upward. The lower table is worse still: the measured accuracy rises from 0.9362 to 1.0000 while the real accuracy falls to 0.8500. The alert built on the drop never fires.

The mechanism is in the selection rule. When only the labels of the rows called 1 arrive, what is measured is not accuracy but precision: how many of the rows the model calls 1 are really 1. The concept drift loosens the label rule, the threshold drops from 30 to 27, and more rows really become 1. The old positives of the model are verified wholesale and the precision rises to 1.0000; the new positives it misses are never measured because they stay on the non-positive side. The collection procedure ends up choosing the measure in which the degradation is least visible.

negatives only carries the same flaw with the opposite sign: in the sound production it measures 1.0000 and shows the model better than it is, in the concept-drift production it measures 0.8055 and is too pessimistic by -0.0845. The clearest example is near threshold: the rows near the decision threshold are by definition the rows where the model is least certain, and collecting only them makes a model of 0.8900 read as 0.5991. The deviation is -0.2909, fifty-eight resolution steps.

Coverage Does Not Correct Bias

Putting two rows side by side gives the rule of the lesson. positives only deviates by +0.0894 with a coverage of 0.4858; every fourth by +0.0100 with a coverage of 0.2500. Half the coverage, one ninth of the deviation. In the sound production the deviation of every fourth is +0.0033, that is, below even the resolution step and unreadable. Increasing the number of collected labels does not reduce the bias; the only thing that determines the bias is whether the selection rule is related to the prediction. every fourth looks at the position in the sequence, not at the output of the model, and for this reason it is nine times more accurate with half the coverage.

From here comes the counterpart of the course rule on the label side. As a firing indicator is not evidence of degradation, a set of collected labels is not evidence of measurement either. If a coverage number is reported alone, “we have labels for 49% of the predictions”, it says nothing about what the measure is; what says it is by which rule that 49% was selected. A collection procedure whose deviation is not shown separately counts as unmeasured.

Summary

• The collection procedure is the rule that determines the label of which prediction will arrive; the coverage and the deviation are two separate numbers, and in every procedure other than all the deviation is different from zero.
• positives only deviates by +0.0894 in the concept-drift production with a coverage of 0.4858 and raises the measured accuracy from 0.9362 to 1.0000; the alert of 0.05 built on the drop never fires. What it measures is not accuracy but precision, and the precision rises when the rule loosens.
• negatives only deviates in the opposite direction by -0.0845, and near threshold deviates by -0.2909 with a coverage of 0.1850; both show the degradation but read the level wrong by seventeen and fifty-eight resolution steps.
• every fourth deviates by +0.0033 and +0.0100 with a coverage of 0.2500; half the coverage, one ninth of the deviation. Coverage does not correct bias, the bias comes from the selection rule being related to the prediction.
• A collection procedure whose deviation is not shown separately counts as unmeasured; a coverage number alone is not evidence of the measurement.

Next Step

Six lessons divided a single question: is the degradation seen. The data drift was a change that fires but does not break, the concept drift a change that breaks but does not fire; the delayed label delayed the only measure that sees by two periods, the proxy metrics did not discriminate in that gap, the false positives grew as the band narrowed, and the collected label itself shifted the measure. What was measured was always the same: the seeing of a degradation.

The question never asked is what is done after it is seen. A team that learns in 2026-11 that the rule changed in 2026-09 has a sequence of decisions in hand, and each carries its own cost: training a new model and putting it into service, giving it first to a share of the traffic, running two models at the same time and showing that the difference is real, recording the reason for the decision. The next topic counts these decisions, and its first measure is this: as the share of requests going to the new model grows, how much does the result move, and is it possible to take a measurement without sending any request.