--- title: 'The Feedback Loop' source: 'https://academia.sh/en/courses/ai-safety-and-evaluation/feedback-loop' course: 'AI Safety and Evaluation' language: en updated: '2026-10-21T05:02:28+00:00' license: 'CC BY-SA 4.0' objectives: - 'Measures the share covered by the feedback signal separately for blocked and passing records.' - 'Writes down the direction of the bias with numbers: the covered share of the false alarm side is far below that of the escaped harm side.' - 'Shows by how many times a loop tuned from the signal multiplies the false alarms and why it cannot see it.' --- # The Feedback Loop Measuring why the user signal comes back biased when it returns to evaluation: the threshold 0.50 setup blocks 74 records, 8 of them false alarms, and 27 harmful ones escape; 0.6296 of the escaped harm produces a complaint, but the covered share of the false alarms is 0.0000 with the appeal channel closed and 0.3750 with a narrow one, and only three of the six appeals are real false alarms. A threshold tuned for five rounds by the two numbers read from the signal drops from 0.50 to 0.30: false alarms rise from 8 to 27, the false block rate from 0.0544 to 0.1837, recall from 0.7097 to 0.9032, and the loop sees only the second. Total errors go from 35 to 36: the loop does not reduce errors, it changes who pays for them. In the previous lesson the indicator was computed from two sources: the trace record kept for every request, and the human review of a third of the blocked records. The second is expensive: thirty-two reviews in three hundred records. There is a third source that looks free: the user. The user says they did not like the reply, sends a correction, asks the same question again. This signal can return to the evaluation set and can tune the defense of the next round. The question not asked is this: **who can send this signal.** A blocked user never sees the reply; the only thing they see is a block message, and they have nowhere to speak. Only the voice of those who pass reaches the loop. This lesson measures the share the signal covers and the direction of the bias, then counts what a defense tuned for five rounds with that signal becomes. - **EM57.** The records, the classes, the defense and the signal are **fictional**. The text of the records is entirely harmless support requests and has no relation to the class; the class lives only in the structural feature profile. The set is the course's **240 records**, **93 harmful** and **147 clean**. - **EM58.** The defense deployed in production is the **model filter, threshold 0.50**. The undefended baseline contains no filter, marking or gate. - **EM59.** On a passing record the user sees the reply: **0.55** of the harmful records and **0.04** of the clean ones produce a complaint. The complaint share is high on harmful records because the bad reply is seen. - **EM60.** A blocked user **cannot produce a complaint**. The appeal channel is **closed** or **narrow**; in the narrow channel **0.25** of the clean records blocked by mistake and **0.05** of the harmful records blocked correctly appeal. - **EM61.** The rule of the loop: the threshold is moved by **0.05** in the direction of the larger of the two numbers read from the signal. The loop **does not see** the true labels; the labeled columns are only for measurement. - **EM62.** The resolution of the set: the step of recall is **1/93 = 0.0108**, the step of the false block rate is **1/147 = 0.0068**. The two runs use seeds **20260218** and **20260219**. ## The Share the Signal Covers The first block builds the common definition of the course and adds the signal model on top. Every function that runs is on the defense side. ```python # FICTIONAL. The text of the records is entirely harmless support requests and has # no relation to the class; the class lives only in the structural feature profile. SEED, M32 = 20260218, 0xFFFFFFFF def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def trace_id(m): h = 2166136261 for c in str(m): h = ((h ^ ord(c)) * 16777619) & M32 return h CLASSES = ["clean", "indirect_instruction", "hidden_field_request", "privilege_escalation", "source_spoofing", "corrupted_input"] SOURCES = ["user", "retrieved_document", "tool_output"] FEATURES = ["instruction_mode", "role_change_request", "hidden_field_name", "side_effect_tool_name", "source_stamp_mismatch", "unusual_stamp_distribution", "excessive_length"] PROFILE = { "clean": [0.06, 0.02, 0.05, 0.03, 0.02, 0.04, 0.05], "indirect_instruction": [0.88, 0.42, 0.10, 0.30, 0.35, 0.08, 0.12], "hidden_field_request": [0.35, 0.15, 0.86, 0.12, 0.10, 0.06, 0.08], "privilege_escalation": [0.30, 0.25, 0.12, 0.84, 0.14, 0.05, 0.07], "source_spoofing": [0.22, 0.55, 0.08, 0.10, 0.82, 0.07, 0.06], "corrupted_input": [0.05, 0.03, 0.04, 0.03, 0.06, 0.80, 0.62], } SOURCE_SHARE = { "clean": [0.72, 0.20, 0.08], "indirect_instruction": [0.05, 0.70, 0.25], "hidden_field_request": [0.55, 0.30, 0.15], "privilege_escalation": [0.40, 0.35, 0.25], "source_spoofing": [0.10, 0.60, 0.30], "corrupted_input": [0.65, 0.20, 0.15], } TEXT = ["bu ayki fatura tutari gecen aya gore yuksek geldi", "sayac endeksi yanlis girilmis olabilir", "mahallede kesinti ne zaman bitecek", "tarife basamagimi ogrenmek istiyorum", "ariza kaydim icin ekip ne zaman gelir", "adres degisikligi nasil yapilir"] RECORDS = [] for i in range(240): r = rng(SEED + 6151 * i) c = CLASSES[pick(r(), [0.62, 0.10, 0.07, 0.07, 0.07, 0.07])] source = SOURCES[pick(r(), SOURCE_SHARE[c])] feat = {o: (r() < p) for o, p in zip(FEATURES, PROFILE[c])} RECORDS.append({"no": i + 1, "class": c, "source": source, "features": feat, "text": TEXT[int(r() * len(TEXT))], "length": 40 + int(r() * 260)}) def harmful(k): return k["class"] != "clean" WEIGHT = {"instruction_mode": 0.42, "role_change_request": 0.26, "hidden_field_name": 0.44, "side_effect_tool_name": 0.40, "source_stamp_mismatch": 0.38, "unusual_stamp_distribution": 0.30, "excessive_length": 0.12} SOURCE_BONUS = {"user": 0.00, "retrieved_document": 0.18, "tool_output": 0.12} def score(k): p = sum(WEIGHT[o] for o in FEATURES if k["features"][o]) + SOURCE_BONUS[k["source"]] r = rng(SEED ^ trace_id(k["no"])) return min(max(p + (r() - 0.5) * 0.20, 0.0), 1.6) def model_filter(k, threshold=0.40): return score(k) >= threshold def measure(records, decision): tp = fp = tn = fn = 0 for k in records: flagged, actual = decision(k), harmful(k) if flagged and actual: tp += 1 elif flagged and not actual: fp += 1 elif not flagged and actual: fn += 1 else: tn += 1 precision = tp / (tp + fp) if tp + fp else 0.0 recall = tp / (tp + fn) if tp + fn else 0.0 return {"tp": tp, "fp": fp, "fn": fn, "tn": tn, "precision": round(precision, 4), "recall": round(recall, 4), "false_block_rate": round(fp / (fp + tn), 4) if fp + tn else 0.0} # ---- USER SIGNAL (fictional). A blocked user never sees the reply: # only a narrow appeal channel is open to them, and only if it has been opened. THRESHOLD = 0.50 # defense deployed in production COMPLAINT_H, COMPLAINT_C = 0.55, 0.04 # complaint share of a passing record APPEAL_C, APPEAL_H = 0.25, 0.05 # appeal share of a blocked record def signal(k, blocked, seed, channel): r = rng(seed ^ trace_id(("sinyal", k["no"]))) if blocked: if not channel: return None return "appeal" if r() < (APPEAL_C if not harmful(k) else APPEAL_H) else None return "complaint" if r() < (COMPLAINT_H if harmful(k) else COMPLAINT_C) else None def loop_round(threshold, seed, channel): """One round: the defense runs, the signal is collected. The class label is only for measurement.""" kr = {k["no"]: model_filter(k, threshold) for k in RECORDS} m = measure(RECORDS, lambda k: kr[k["no"]]) s = [signal(k, kr[k["no"]], seed, channel) for k in RECORDS] count = {"appeal": sum(x == "appeal" for x in s), "complaint": sum(x == "complaint" for x in s), "true_appeal": sum(1 for k, x in zip(RECORDS, s) if x == "appeal" and not harmful(k)), "true_complaint": sum(1 for k, x in zip(RECORDS, s) if x == "complaint" and harmful(k)), "coverage": sum(x is not None for x in s)} return m, count t = measure(RECORDS, lambda k: model_filter(k, THRESHOLD)) print(f"undefended baseline: blocked 0 , false alarms 0 , escaped harm {t['tp'] + t['fn']}") print(f"threshold {THRESHOLD:.2f}: blocked {t['tp'] + t['fp']} , false alarms {t['fp']} ," f" escaped harm {t['fn']}") print("\nseed channel complaint(true) appeal(true) covered fn covered fp coverage") for seed in (SEED, 20260219): for name, channel in (("closed", False), ("narrow", True)): m, s = loop_round(THRESHOLD, seed, channel) print(f"{seed} {name:<7} {s['complaint']:>7} ({s['true_complaint']:>2})" f" {s['appeal']:>9} ({s['true_appeal']}) {s['true_complaint'] / m['fn']:>12.4f}" f" {s['true_appeal'] / m['fp']:>12.4f} {s['coverage'] / len(RECORDS):>12.4f}") ``` ``` undefended baseline: blocked 0 , false alarms 0 , escaped harm 93 threshold 0.50: blocked 74 , false alarms 8 , escaped harm 27 seed channel complaint(true) appeal(true) covered fn covered fp coverage 20260218 closed 20 (17) 0 (0) 0.6296 0.0000 0.0833 20260218 narrow 20 (17) 6 (3) 0.6296 0.3750 0.1083 20260219 closed 25 (18) 0 (0) 0.6667 0.0000 0.1042 20260219 narrow 25 (18) 8 (3) 0.6667 0.3750 0.1375 ``` On the undefended baseline nobody is silenced: since no record is blocked, the signal channel is open to everyone and the false alarm is zero anyway. **The bias arises from the defense itself.** In the threshold 0.50 setup 74 records are blocked, 8 of them false alarms, and those eight users have never seen the reply. The covered share is entirely different on the two sides. **0.6296** of the 27 escaped harmful records produce a complaint. On the false alarm side the share is **0.0000** with the channel closed and **0.3750** with the narrow one. The total coverage of the signal is between 0.0833 and 0.1375: only about one record in ten says anything to the loop. The narrow channel is not a clean measure either: of the 6 appeals that arrive, only **3** are real false alarms, because the 66 correctly blocked records outnumber the 8 wrongly blocked ones. **The precision of the signal that looks for false alarms is 0.50.** ## Five Rounds of Tuning The second block moves the threshold with the two numbers read from the signal and writes the true numbers beside each round. ```python # The rule of the loop: a step is taken in the direction of the larger of the two numbers read from the signal. # The loop does NOT see the true labels; the actual columns below are only for measurement. STEP, ROUNDS, START_THRESHOLD = 0.05, 5, 0.50 def loop(seed, channel, round_count=ROUNDS): threshold, rows = START_THRESHOLD, [] for _ in range(round_count): m, s = loop_round(threshold, seed, channel) rows.append((threshold, m, s)) if s["complaint"] > s["appeal"]: threshold = round(threshold - STEP, 2) # block more elif s["appeal"] > s["complaint"]: threshold = round(threshold + STEP, 2) # block less return rows print("seed 20260218") print(" channel rnd thr. actual fp actual fn false block recall compl. appeal") for name, channel in (("closed", False), ("narrow", True)): for i, (e, m, s) in enumerate(loop(SEED, channel)): print(f" {name:<7} {i + 1:>3} {e:>5.2f} {m['fp']:>10} {m['fn']:>10}" f" {m['false_block_rate']:>12.4f} {m['recall']:>11.4f}" f" {s['complaint']:>8} {s['appeal']:>7}") print("\nseed 20260219: threshold path the same , signal counts different") for name, channel in (("closed", False), ("narrow", True)): d = loop(20260219, channel) print(f" {name:<7} complaint {[s['complaint'] for _, _, s in d]}" f" appeal {[s['appeal'] for _, _, s in d]}") print("\nthr. fp+fn (labeled reading , closed to the loop)") for e in (0.50, 0.45, 0.40, 0.35, 0.30): m = measure(RECORDS, lambda k, e=e: model_filter(k, e)) print(f"{e:.2f} {m['fp'] + m['fn']:>7}") ``` ``` seed 20260218 channel rnd thr. actual fp actual fn false block recall compl. appeal closed 1 0.50 8 27 0.0544 0.7097 20 0 closed 2 0.45 15 20 0.1020 0.7849 16 0 closed 3 0.40 20 16 0.1361 0.8280 14 0 closed 4 0.35 23 12 0.1565 0.8710 11 0 closed 5 0.30 27 9 0.1837 0.9032 10 0 narrow 1 0.50 8 27 0.0544 0.7097 20 6 narrow 2 0.45 15 20 0.1020 0.7849 16 6 narrow 3 0.40 20 16 0.1361 0.8280 14 6 narrow 4 0.35 23 12 0.1565 0.8710 11 10 narrow 5 0.30 27 9 0.1837 0.9032 10 11 seed 20260219: threshold path the same , signal counts different closed complaint [25, 20, 18, 16, 14] appeal [0, 0, 0, 0, 0] narrow complaint [25, 20, 18, 16, 14] appeal [8, 8, 9, 10, 11] thr. fp+fn (labeled reading , closed to the loop) 0.50 35 0.45 35 0.40 36 0.35 35 0.30 36 ``` ## The Direction of the Bias With the channel closed, the appeal column is zero in all five rounds. The loop reads this as "no false alarms" and takes one more step in the same direction every round. At the end of five rounds the threshold has dropped from 0.50 to 0.30, false alarms have risen from 8 to **27**, and the false block rate from 0.0544 to **0.1837**. Both are far above the resolution of the set: the step of the false block rate is 0.0068, and the observed change is nineteen times that. The complaint count, the only number the loop sees, has dropped from 20 to 10, so the loop has reported an **improvement** by its own measure in every round. In the same table recall rises from 0.7097 to 0.9032, and this is a real gain. The rule of the course applies exactly here: **a defense whose blocked share and wrongly blocked share are not written together counts as unmeasured.** The loop cannot write the second share, because the users who pay for it have no channel. The last table summarizes it in one line: total errors at the five thresholds are 35, 35, 36, 35, 36. The loop did not reduce errors; it **changed who pays for them**, and because it could not see who pays, it recorded this as an improvement. The narrow channel is not a measure but a brake. In the fifth round the appeals are 11 and the complaints 10, and the loop will turn the other way for the first time. But by then the false alarms have already risen **3.4 times**: the brake holds three rounds late. The second run does not disturb this reading: the threshold path is exactly the same over the two seeds, only the signal counts change (complaints 25 instead of 20, appeals 8 instead of 6). **The decision is reproducible, the numbers are not.** The resolution must be written when reading the covered share too: the denominator of the false alarm side is eight records, its step is 0.1250, and 0.3750 is only three steps. The source of the bias is not the quality of the signal but **the geometry of the channel**. A passing record shows the user a reply, and if the reply is bad there is something to say; a blocked record shows no reply, and nothing to say ever forms. For this reason the covered share of the escaped harm side is 1.68 times that of the false alarm side in the narrow channel and infinitely many times in the closed one. What corrects this is not collecting more signal: the only unbiased reading of the false alarm side is the **sampled human review** of the previous lesson, and its cost is thirty-two reviews in three hundred records. A free signal does not replace a paid measure. This lesson is an example of the second claim of the course. What was added is not a filter but an **audit apparatus**: a loop that tunes the defense with data from the user. It caught no new class of harm, it multiplied the false alarms by 3.4, and it could not see this in its own indicator. ## Summary - The bias arises from the defense: on the undefended baseline nobody is blocked and the channel is open to everyone; at threshold 0.50 eight users never see the reply and have nowhere to speak. - 0.6296 of the escaped harm produces a signal; the covered share of the false alarms is 0.0000 with the channel closed and 0.3750 with a narrow one; total coverage is one record in ten. - The precision of the narrow channel is low too: three of six appeals are real false alarms, because the 66 correctly blocked records outnumber the 8 wrongly blocked ones. - With the signal, the threshold tuned for five rounds drops from 0.50 to 0.30; false alarms rise from 8 to 27 and the false block rate from 0.0544 to 0.1837, while the loop reports an improvement every round. - Total errors stay between 35 and 36: the loop does not reduce errors, it changes who pays for them. ## Next Step Over seven lessons we measured, monitored and tuned. All of these numbers so far stayed **inside**: the team that built the set, read the indicator and moved the threshold is the same team. The party that uses the system, gives it data and has nowhere to appeal when blocked has seen none of it. What a system says to the outside, which measure it publishes, which it does not, and what the unpublished measure means has never been asked. The last lesson measures this: the numbers a system card carries, the numbers it does not, and every number it does not carry remaining a claim.