--- title: 'Feedback Collection' source: 'https://academia.sh/en/courses/model-monitoring/feedback-collection' course: 'Model Monitoring and Governance' language: en updated: '2026-10-21T05:03:02+00:00' license: 'CC BY-SA 4.0' objectives: - 'Defines the collection procedure as a selection rule and computes the coverage and the deviation separately on the same set.' - 'Compares how far five collection procedures shift the measured accuracy from the real accuracy in the sound and in the concept-drift production.' - 'Shows why the measured accuracy rises when labels are collected for the positive predictions only, and why the alert built on the drop never fires.' - 'Argues that the size of the coverage and the bias are independent, by showing that a procedure with a smaller coverage deviates less.' --- # Feedback Collection Counting the share covered by a label collected from production and the bias it carries: when five collection procedures are tried on the same 1200 rows, collecting labels for the positive predictions only gives a coverage of 0.4858 and shifts the measured accuracy +0.0894 above the real accuracy in the concept-drift production, because that procedure measures precision, not accuracy; period by period the measured accuracy rises from 0.9362 to 1.0000 while the real accuracy falls to 0.8500, and an alert built on the drop never fires. Negative predictions only deviate in the opposite direction by -0.0845, and the rows near the decision threshold deviate by -0.2909 with a coverage of 0.1850. The procedure that collects every fourth row deviates by +0.0033 and +0.0100 with half the coverage; coverage does not correct bias, the bias comes from the selection rule. The previous lesson counted three measures at three bands and left a single valid indicator: accuracy. It was the only measure that never fired in the two harmless scenarios and fired in four of four periods in the harmful scenario. But accuracy needs a **label**, and until now the label was a gift of the fiction: the real class of every row was known, it only arrived two periods late. In production the label is not a gift, it is **collected**. And what is collected rarely covers every prediction: a review goes only to flagged records, an objection comes only from the person who is bothered, a verification is requested only in the cases left undecided. This lesson runs five collection procedures on the same set and writes two numbers separately: the **coverage**, that is, the share of rows whose label arrives; and the **deviation**, that is, the difference between the accuracy measured with those labels and the real accuracy. - **MO53.** The source, the pipeline, the fictional model, the concept drift and the five collection procedures are **fictional** and are produced inside the lesson. No monitoring product, labeling service, cloud service or learning library is called. The seed is 20260218. - **MO54. The collection procedure** is the rule that determines the label of which prediction will arrive. Five are tried: `all`; `positives only`, the rows the model calls 1; `negatives only`, the rows it calls 0; `near threshold`, the rows whose score is within `MARGIN` of the decision threshold; `every fourth`, every fourth row in the sequence. - **MO55. The coverage** is the ratio of the rows whose label arrives to the 1200 rows of the monitoring periods. **The deviation** is the **real accuracy**, measured with all rows, subtracted from the accuracy measured with the collected labels. Its direction is meaningful: a plus sign says the model looks better than it is. - **MO56.** The return of the user signal to evaluation and the bias of that signal were measured in the feedback loop lesson of the AI Safety and Evaluation course in the AI Engineering curriculum; **the procedure is not repeated**. The measure there was for language model output; the addition here is **the delayed label of a numeric model**. - **MO57.** The model is inherited from the MLOps Fundamentals course with its flaw corrected: the decision threshold is a separate setting and the training criterion is **balanced accuracy**. In the monitoring periods the **base class ratio is 0.5342**. - **MO58.** The baseline is the **no-collection setup**: no label arrives, there is no measured accuracy, the deviation is undefined. The step of period accuracy is **0.0050**, and a deviation smaller than this is not read. ## Where the Label Comes From The block below builds the shared definition and trains the model. The last line gives the limits of the lesson: over how many rows we measure, what the resolution is below which we cannot go, and how many periods later the label arrives. ```python # FICTIONAL. The meter readings of the measurement network , the five-stage pipeline , the fictional model , the concept # drift and the five collection procedures are produced inside the lesson; no monitoring product , # labeling service , cloud service or learning library is called. Seed 20260218. SEED, M32 = 20260218, 0xFFFFFFFF def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value REGION = ["east", "west", "north", "south", "central"] TYPE = ["residential", "commercial"] PERIODS = [f"2026-{a:02d}" for a in range(1, 13)] SUBSCRIBERS = {} for i in range(200): r = rng(SEED + 37 * i) SUBSCRIBERS[f"A{i + 1:03d}"] = {"region": REGION[int(r() * 5)], "type": TYPE[int(r() * 2)], "start": 10000 + int(r() * 60000)} def generate_source(): records = [] for i, (sub, a) in enumerate(SUBSCRIBERS.items()): e = a["start"] for p in PERIODS: r = rng(SEED + 101 * i + 7 * PERIODS.index(p)) e += 4 + int(r() * 46) records.append({"subscriber": sub, "period": p, "region": a["region"], "type": a["type"], "index": e, "unit": "m3"}) return records def s1_screen(records): REQUIRED = ("subscriber", "period", "region", "index", "unit") keep = [k for k in records if all(a in k for a in REQUIRED)] return keep, len(records) - len(keep) def s2_dedupe(records): seen, keep = set(), [] for k in records: key = (k["subscriber"], k["period"]) if key not in seen: seen.add(key) keep.append(k) return keep, len(records) - len(keep) def s3_consumption(records): table = {(k["subscriber"], k["period"]): k for k in records} out, dropped = [], 0 for k in records: i = PERIODS.index(k["period"]) prev = table.get((k["subscriber"], PERIODS[i - 1])) if i else None if prev is None: dropped += i > 0 continue out.append(dict(k, m3=int(k["index"]) - int(prev["index"]))) return out, dropped def rows(source=None): v, _ = s1_screen(generate_source() if source is None else source) v, _ = s2_dedupe(v) v, _ = s3_consumption(v) return v def features(records, window=None, digits=4): ordered = sorted(records, key=lambda k: PERIODS.index(k["period"])) sub, reg = {}, {} for k in ordered: sub.setdefault(k["subscriber"], []).append((PERIODS.index(k["period"]), k["m3"])) reg.setdefault(k["region"], []).append((PERIODS.index(k["period"]), k["m3"])) def mean(series, i): chosen = [v for d, v in series if (d < i and (window is None or d >= i - window))] return round(sum(chosen) / len(chosen), digits) if chosen else 0.0 feats = {} for k in records: i = PERIODS.index(k["period"]) feats[(k["subscriber"], k["period"])] = {"subscriber_mean": mean(sub[k["subscriber"]], i), "region_mean": mean(reg[k["region"]], i), "commercial": 1 if k["type"] == "commercial" else 0} return feats GRID = [(a, b, c) for a in (0.6, 0.8, 1.0) for b in (0.0, 0.2, 0.4) for c in (0, 4, 8)] DECISION_GRID = (18, 22, 26, 30, 34) # the decision threshold is a SEPARATE setting , searched in training MONITORED = PERIODS[6:] # 2026-07 .. 2026-12 : the periods monitored in production DELAY = 2 # after how many periods the label arrives RULE_THRESHOLD, RULE_COMMERCIAL = 30, 6 # CONCEPT: subscriber_mean + 6 x commercial > 30 def score(o, w): return w[0] * o["subscriber_mean"] + w[1] * o["region_mean"] + w[2] * o["commercial"] def label(feat_k, threshold=RULE_THRESHOLD, commercial_factor=RULE_COMMERCIAL): return 1 if feat_k["subscriber_mean"] + commercial_factor * feat_k["commercial"] > threshold else 0 def predict(feat_k, w, decision): return 1 if score(feat_k, w) > decision else 0 def balanced_accuracy(subset, feats, w, decision): tp = tn = fp = fn = 0 for k in subset: f_k = feats[(k["subscriber"], k["period"])] t, g = predict(f_k, w, decision), label(f_k) tp, fp = tp + (t and g), fp + (t and not g) fn, tn = fn + (g and not t), tn + (not t and not g) recall = tp / (tp + fn) if tp + fn else 0.0 specificity = tn / (tn + fp) if tn + fp else 0.0 return (recall + specificity) / 2, round(recall, 4), round(specificity, 4) def train(records, feats, periods=None): """The training criterion is BALANCED ACCURACY: the mean of recall and specificity.""" subset = [k for k in records if k["period"] in (periods or PERIODS[1:9])] best = (-1.0, None, None, 0.0, 0.0) for w in GRID: for decision in DECISION_GRID: o, rec, spec = balanced_accuracy(subset, feats, w, decision) if o > best[0]: best = (o, w, decision, rec, spec) return {"weights": best[1], "decision": best[2], "balanced": round(best[0], 4), "recall": best[3], "specificity": best[4], "train_rows": len(subset)} def concept_drift(records, feats, start="2026-09", new_threshold=27): """CONCEPT DRIFT: the input is the SAME , the rule changes. The threshold drops from 30 to new_threshold.""" b = PERIODS.index(start) return {(k["subscriber"], k["period"]): label( feats[(k["subscriber"], k["period"])], new_threshold if PERIODS.index(k["period"]) >= b else RULE_THRESHOLD) for k in records} def alert(series, baseline, share): """A measure fires in a period if it deviates from the baseline by more than `share`.""" return [d for d, v in series if abs(v - baseline) / max(abs(baseline), 1e-9) > share] ALL_ROWS = rows() FEATS = features(ALL_ROWS) # features are computed ONCE MODEL = train(ALL_ROWS, FEATS) MON = [k for k in ALL_ROWS if k["period"] in MONITORED] largest = max(sum(1 for k in MON if label(FEATS[(k["subscriber"], k["period"])]) == s) for s in (0, 1)) print(f"model {MODEL['weights']} decision threshold {MODEL['decision']} , balanced accuracy" f" {MODEL['balanced']} , recall {MODEL['recall']} , monitoring {len(MON)} rows ," f" base class ratio {largest / len(MON):.4f} , step {1 / 200:.4f} , label delay" f" {DELAY} periods") ``` ``` model (0.6, 0.0, 4) decision threshold 18 , balanced accuracy 0.9864 , recall 1.0 , monitoring 1200 rows , base class ratio 0.5342 , step 0.0050 , label delay 2 periods ``` ## Coverage and Deviation The block below runs the five procedures in two scenarios. The upper table gives, for the whole monitoring window, the coverage, the accuracy measured with the collected labels, the real accuracy measured with all rows, and the difference between the two. The lower table counts, only in the concept-drift production, the accuracy measured period by period and how many periods an alert of 0.05 built on the **drop** fires. ```python # This block continues the definitions of the first block. The features are computed ONCE. MARGIN = 1.0 # scores this close to the decision threshold count as "near threshold" CONCEPT = concept_drift(ALL_ROWS, FEATS) W, DEC = MODEL["weights"], MODEL["decision"] PROCEDURES = (("all", lambda o, j: True), ("positives only", lambda o, j: predict(o, W, DEC) == 1), ("negatives only", lambda o, j: predict(o, W, DEC) == 0), ("near threshold", lambda o, j: abs(score(o, W) - DEC) <= MARGIN), ("every fourth", lambda o, j: j % 4 == 0)) def measure(labels, select, periods_set): """Only the label of the row that select picks arrives; accuracy is measured WITH THEM.""" total = correct = all_n = all_correct = 0 for j, k in enumerate(ALL_ROWS): if k["period"] not in periods_set: continue o = FEATS[(k["subscriber"], k["period"])] d = predict(o, W, DEC) == (labels[(k["subscriber"], k["period"])] if labels else label(o)) all_n, all_correct = all_n + 1, all_correct + d if select(o, j): total, correct = total + 1, correct + d return round(total / all_n, 4), round(correct / total, 4), round(all_correct / all_n, 4) print("procedure scenario coverage measured real deviation") for name, select in PROCEDURES: for sname, lab in (("sound", None), ("concept drift", CONCEPT)): cov, measured, real = measure(lab, select, MONITORED) print(f"{name:<16} {sname:<14} {cov:>8.4f} {measured:>8.4f} {real:>6.4f} {measured - real:>+9.4f}") print("\nconcept drift , accuracy measured period by period and firing on the DROP at the 0.05 band") for name, select in PROCEDURES: series = [(d, measure(CONCEPT, select, [d])[1]) for d in MONITORED] baseline = dict(series)["2026-08"] fired = [d for d, v in series[2:] if (baseline - v) / baseline > 0.05] print(f"{name:<16}" + "".join(f"{v:8.4f}" for _, v in series) + f" fires {len(fired)}") ``` ``` procedure scenario coverage measured real deviation all sound 1.0000 0.9800 0.9800 +0.0000 all concept drift 1.0000 0.8900 0.8900 +0.0000 positives only sound 0.4858 0.9588 0.9800 -0.0212 positives only concept drift 0.4858 0.9794 0.8900 +0.0894 negatives only sound 0.5142 1.0000 0.9800 +0.0200 negatives only concept drift 0.5142 0.8055 0.8900 -0.0845 near threshold sound 0.1850 0.8919 0.9800 -0.0881 near threshold concept drift 0.1850 0.5991 0.8900 -0.2909 every fourth sound 0.2500 0.9833 0.9800 +0.0033 every fourth concept drift 0.2500 0.9000 0.8900 +0.0100 concept drift , accuracy measured period by period and firing on the DROP at the 0.05 band all 0.9700 0.9700 0.8150 0.8700 0.8650 0.8500 fires 4 positives only 0.9341 0.9362 1.0000 1.0000 1.0000 1.0000 fires 0 negatives only 1.0000 1.0000 0.6606 0.7347 0.7245 0.6907 fires 4 near threshold 0.8286 0.8537 0.3824 0.5526 0.4857 0.4615 fires 4 every fourth 0.9800 0.9800 0.8600 0.8600 0.8200 0.9000 fires 4 ``` ## The Collection Procedure Chooses the Measure Too The `positives only` row is the hardest finding of the lesson. Its coverage is 0.4858, so it sees almost half of the predictions; this is not a small sample. In the concept-drift production the accuracy it measures is **0.9794**, the real accuracy is **0.8900**; the deviation is **+0.0894**, eighteen resolution steps, and **upward**. The lower table is worse still: the measured accuracy **rises** from 0.9362 to **1.0000** while the real accuracy falls to 0.8500. The alert built on the drop **never fires**. The mechanism is in the selection rule. When only the labels of the rows called 1 arrive, what is measured is not accuracy but **precision**: how many of the rows the model calls 1 are really 1. The concept drift **loosens** the label rule, the threshold drops from 30 to 27, and more rows really become 1. The old positives of the model are verified wholesale and the precision rises to 1.0000; the new positives it misses are never measured because they stay on the non-positive side. **The collection procedure ends up choosing the measure in which the degradation is least visible.** `negatives only` carries the same flaw with the opposite sign: in the sound production it measures 1.0000 and shows the model better than it is, in the concept-drift production it measures 0.8055 and is too pessimistic by **-0.0845**. The clearest example is `near threshold`: the rows near the decision threshold are by definition the rows where the model is least certain, and collecting only them makes a model of 0.8900 read as **0.5991**. The deviation is **-0.2909**, fifty-eight resolution steps. ## Coverage Does Not Correct Bias Putting two rows side by side gives the rule of the lesson. `positives only` deviates by +0.0894 with a coverage of **0.4858**; `every fourth` by +0.0100 with a coverage of **0.2500**. Half the coverage, one ninth of the deviation. In the sound production the deviation of `every fourth` is **+0.0033**, that is, below even the resolution step and **unreadable**. Increasing the number of collected labels does not reduce the bias; the only thing that determines the bias is **whether the selection rule is related to the prediction**. `every fourth` looks at the position in the sequence, not at the output of the model, and for this reason it is nine times more accurate with half the coverage. From here comes the counterpart of the course rule on the label side. As a firing indicator is not evidence of degradation, **a set of collected labels is not evidence of measurement either**. If a coverage number is reported alone, "we have labels for 49% of the predictions", it says nothing about what the measure is; what says it is **by which rule** that 49% was selected. A collection procedure whose deviation is not shown separately **counts as unmeasured**. ## Summary - The collection procedure is the rule that determines the label of which prediction will arrive; the coverage and the deviation are two separate numbers, and in every procedure other than `all` the deviation is different from zero. - `positives only` deviates by **+0.0894** in the concept-drift production with a coverage of 0.4858 and **raises** the measured accuracy from 0.9362 to 1.0000; the alert of 0.05 built on the drop never fires. What it measures is not accuracy but **precision**, and the precision rises when the rule loosens. - `negatives only` deviates in the opposite direction by -0.0845, and `near threshold` deviates by -0.2909 with a coverage of 0.1850; both show the degradation but read the level wrong by seventeen and fifty-eight resolution steps. - `every fourth` deviates by +0.0033 and +0.0100 with a coverage of 0.2500; half the coverage, one ninth of the deviation. **Coverage does not correct bias**, the bias comes from the selection rule being related to the prediction. - A collection procedure whose deviation is not shown separately counts as unmeasured; a coverage number alone is not evidence of the measurement. ## Next Step Six lessons divided a single question: **is the degradation seen**. The data drift was a change that fires but does not break, the concept drift a change that breaks but does not fire; the delayed label delayed the only measure that sees by two periods, the proxy metrics did not discriminate in that gap, the false positives grew as the band narrowed, and the collected label itself shifted the measure. What was measured was always the same: the **seeing** of a degradation. The question never asked is what is done **after** it is seen. A team that learns in `2026-11` that the rule changed in `2026-09` has a sequence of decisions in hand, and each carries its own cost: training a new model and putting it into service, giving it first to a share of the traffic, running two models at the same time and showing that the difference is real, recording the reason for the decision. The next topic counts these decisions, and its first measure is this: as the share of requests going to the new model grows, how much does the result move, and is it possible to take a measurement **without sending any request**.