--- title: 'Content Moderation' source: 'https://academia.sh/en/courses/ai-safety-and-evaluation/content-moderation' course: 'AI Safety and Evaluation' language: en updated: '2026-10-21T05:02:28+00:00' license: 'CC BY-SA 4.0' objectives: - 'Compute the precision and recall of a content moderation step that produces a class name separately for each class.' - 'Show which class a single average rate hides by putting the binary reading and the per-class reading side by side.' - 'State with numbers that in at least one class the defense produces more wrong labels than right ones and falls below the baseline in that class.' --- # Content Moderation Counting that blocking harmful content and giving it a class name are two separate jobs: a moderation step that produces a class name gives 0.9355 recall and 0.7311 precision in the binary reading, but per class the precision spreads between 0.2917 and 0.6364 and the recall of the class label between 0.3333 and 0.8235; in the indirect instruction class the moderation step puts the label on 21 records and knows 8 of them correctly, and it marks 9 entirely clean records with the same label, so when the class label is tied to an action the action fires wrongly far more often than rightly and the defense falls below the baseline in that class; the recall of the corrupted input class rises from 0.2632 to 0.8421 while the false block rate rises from 0.1565 to 0.2177. All the numbers of the previous lesson were read from a single decision: block or let through. The rule-based filter blocked 74 of the 93 harmful records, blocked 23 clean records, and its average recall came out 0.7957. The last column of the same table, however, showed that 14 of the 19 missed records came from **a single class**. The average number did not carry that concentration. When a defense does not stop at blocking but also tells the record **which kind of harmful it is**, two things change at once. First, the filter can gain a rule that looks at the class it was missing. Second, and more important, the measure stops being a single pair of numbers: each class gets its own precision and recall. This lesson builds that distinction and counts what the average hides. - **DF8.** The record set, the classes, the features and the content moderation step are **fictional**; no real model endpoint is called. The seed is **20260218**, the second run uses **20260219**. - **DF9.** **This is not an attack lesson.** All of the code that runs sits on the defense side: classification, filter and count. No working attack, payload or evasion routine is written. - **DF10.** **An attack is not a text but a feature profile.** The `text` field of the records equals one of six ordinary support requests and **has no relation to the class**; the moderation step does not read that text either, it reads only the structural features. - **DF11.** The baseline is the **undefended setup**: no filter or moderation step, harmful records blocked 0, false positives 0, class labels produced 0. - **DF12.** The content moderation step returns a single class name; the priority order is a fiction, and if a record has more than one feature only the class at the front is written. - **DF13.** The general routine of model evaluation and subgroup analysis was measured in the Supervised Learning and Model Evaluation, Interpretability and Ethics courses and is not repeated here; this lesson adds only the reading **per defense class**. - **DF14.** The step of a per-class rate is the size of that class and is larger than the general step of the set: **0.1000** in the smallest class, **0.0417** in the largest harmful class. Over all harmful records the step of recall is **0.0108**, the step of the false block rate is **0.0068**. ## A Defense That Produces a Class Name The first block sets up the course's shared definition: the record set, the classes, the rule-based filter, the model-based filter and the measure. Which class the six support sentences fall into is random; the class information lives only in the `PROFILE` dictionary. ```python # FICTIONAL. The record set, classes, features and defense layers are defined inside # the lesson; no real model endpoint is called. SEED, M32 = 20260218, 0xFFFFFFFF def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def trace_id(m): h = 2166136261 for c in str(m): h = ((h ^ ord(c)) * 16777619) & M32 return h # ---- FICTIONAL records: no attack TEXT is written, an attack is a FEATURE PROFILE CLASSES = ["clean", "indirect_instruction", "hidden_field_request", "privilege_escalation", "source_spoofing", "corrupted_input"] SOURCES = ["user", "retrieved_document", "tool_output"] # Features are structural; none of them is a working payload. FEATURES = ["instruction_mode", "role_change_request", "hidden_field_name", "side_effect_tool_name", "source_stamp_mismatch", "unusual_stamp_distribution", "excessive_length"] # Feature probability per class (fictional). Clean records carry small shares too: # in the real world, well-meant text can carry these features as well. PROFILE = { "clean": [0.06, 0.02, 0.05, 0.03, 0.02, 0.04, 0.05], "indirect_instruction": [0.88, 0.42, 0.10, 0.30, 0.35, 0.08, 0.12], "hidden_field_request": [0.35, 0.15, 0.86, 0.12, 0.10, 0.06, 0.08], "privilege_escalation": [0.30, 0.25, 0.12, 0.84, 0.14, 0.05, 0.07], "source_spoofing": [0.22, 0.55, 0.08, 0.10, 0.82, 0.07, 0.06], "corrupted_input": [0.05, 0.03, 0.04, 0.03, 0.06, 0.80, 0.62], } SOURCE_SHARE = { "clean": [0.72, 0.20, 0.08], "indirect_instruction": [0.05, 0.70, 0.25], "hidden_field_request": [0.55, 0.30, 0.15], "privilege_escalation": [0.40, 0.35, 0.25], "source_spoofing": [0.10, 0.60, 0.30], "corrupted_input": [0.65, 0.20, 0.15], } # Fictional, entirely harmless support texts. They have NO relation to the class; # the class information lives only in the feature profile. TEXT = ["bu ayki fatura tutari gecen aya gore yuksek geldi", "sayac endeksi yanlis girilmis olabilir", "mahallede kesinti ne zaman bitecek", "tarife basamagimi ogrenmek istiyorum", "ariza kaydim icin ekip ne zaman gelir", "adres degisikligi nasil yapilir"] RECORDS = [] for i in range(240): r = rng(SEED + 6151 * i) c = CLASSES[pick(r(), [0.62, 0.10, 0.07, 0.07, 0.07, 0.07])] source = SOURCES[pick(r(), SOURCE_SHARE[c])] feat = {o: (r() < p) for o, p in zip(FEATURES, PROFILE[c])} RECORDS.append({"no": i + 1, "class": c, "source": source, "features": feat, "text": TEXT[int(r() * len(TEXT))], "length": 40 + int(r() * 260)}) def harmful(k): return k["class"] != "clean" # ---- DEFENSE 1: rule filter (structural, explainable) def rule_filter(k, rules=("instruction_mode", "hidden_field_name", "side_effect_tool_name", "source_stamp_mismatch")): return any(k["features"][o] for o in rules) # ---- DEFENSE 2: model-based filter (score + threshold) WEIGHT = {"instruction_mode": 0.42, "role_change_request": 0.26, "hidden_field_name": 0.44, "side_effect_tool_name": 0.40, "source_stamp_mismatch": 0.38, "unusual_stamp_distribution": 0.30, "excessive_length": 0.12} SOURCE_BONUS = {"user": 0.00, "retrieved_document": 0.18, "tool_output": 0.12} def score(k): p = sum(WEIGHT[o] for o in FEATURES if k["features"][o]) + SOURCE_BONUS[k["source"]] r = rng(SEED ^ trace_id(k["no"])) return min(max(p + (r() - 0.5) * 0.20, 0.0), 1.6) def model_filter(k, threshold=0.40): return score(k) >= threshold # ---- MEASURE: what was blocked and what was blocked by mistake are written together def measure(records, decision): tp = fp = tn = fn = 0 for k in records: flagged, actual = decision(k), harmful(k) if flagged and actual: tp += 1 elif flagged and not actual: fp += 1 elif not flagged and actual: fn += 1 else: tn += 1 precision = tp / (tp + fp) if tp + fp else 0.0 recall = tp / (tp + fn) if tp + fn else 0.0 return {"tp": tp, "fp": fp, "fn": fn, "tn": tn, "precision": round(precision, 4), "recall": round(recall, 4), "false_block_rate": round(fp / (fp + tn), 4) if fp + tn else 0.0} ``` ## Precision and Recall per Class The content moderation step differs from the rule-based filter in two points: it returns a class name, and it reads one more feature, the stamp distribution. The priority order puts each class's own distinguishing feature at the front. The second block first prints the **binary reading** — which number comes out if saying "not clean" counts as blocking — and then opens the same decision class by class. ```python def content_moderation(k): """Defense: assigns a class name to the record. No attack text is read; only structural features are evaluated in priority order.""" o = k["features"] if o["hidden_field_name"]: return "hidden_field_request" if o["side_effect_tool_name"]: return "privilege_escalation" if o["source_stamp_mismatch"]: return "source_spoofing" if o["unusual_stamp_distribution"]: return "corrupted_input" if o["instruction_mode"] or o["role_change_request"]: return "indirect_instruction" return "clean" binary = measure(RECORDS, lambda k: content_moderation(k) != "clean") print("binary reading, if saying 'not clean' counts as blocking:") print(f" tp {binary['tp']} fp {binary['fp']} fn {binary['fn']}" f" precision {binary['precision']} recall {binary['recall']}" f" false block rate {binary['false_block_rate']}") print() print(f"{'class':<20}{'n':>4}{'assigned':>9}{'correct':>8}{'wrong class':>13}" f"{'clean':>6}{'precision':>10}{'recall':>12}{'step':>8}") for s in CLASSES[1:]: g = [k for k in RECORDS if k["class"] == s] a = [k for k in RECORDS if content_moderation(k) == s] d = sum(1 for k in a if k["class"] == s) ws = sum(1 for k in a if harmful(k) and k["class"] != s) cl = sum(1 for k in a if not harmful(k)) print(f"{s:<20}{len(g):>4}{len(a):>9}{d:>8}{ws:>13}{cl:>6}" f"{d / len(a):>10.4f}{d / len(g):>12.4f}{1 / len(g):>8.4f}") passed = [k for k in RECORDS if content_moderation(k) == "clean"] pc = sum(1 for k in passed if not harmful(k)) print(f"{'clean':<20}{147:>4}{len(passed):>9}{pc:>8}{len(passed) - pc:>13}{'-':>6}" f"{pc / len(passed):>10.4f}{pc / 147:>12.4f}{1 / 147:>8.4f}") print() print("classes of the harmful records passed as clean:", {s: sum(1 for k in passed if k["class"] == s) for s in CLASSES[1:] if sum(1 for k in passed if k["class"] == s)}) ``` ``` binary reading, if saying 'not clean' counts as blocking: tp 87 fp 32 fn 6 precision 0.7311 recall 0.9355 false block rate 0.2177 class n assigned correct wrong class clean precision recall step indirect_instruction 24 21 8 4 9 0.3810 0.3333 0.0417 hidden_field_request 17 32 14 9 9 0.4375 0.8235 0.0588 privilege_escalation 10 24 7 14 3 0.2917 0.7000 0.1000 source_spoofing 23 22 14 4 4 0.6364 0.6087 0.0435 corrupted_input 19 20 12 1 7 0.6000 0.6316 0.0526 clean 147 121 115 6 - 0.9504 0.7823 0.0068 classes of the harmful records passed as clean: {'hidden_field_request': 1, 'privilege_escalation': 1, 'source_spoofing': 1, 'corrupted_input': 3} ``` ## What the Average Hides The first rows show the best recall so far: **0.9355**. Where the undefended baseline blocked 0 harmful records, the moderation step flags 87 of the 93 harmful records and misses only 6. This is where a defense would be reported with a single line, and this line is **wrong** — more precisely, what it says is true but it does not answer the question asked. What is expected of a content moderation step is not that the record is blocked but that **it is told what it is**; blocking was already the job of the rule-based filter. The class table opens the same decision by name and no single rate remains. The recall of the class label spreads between 0.3333 and 0.8235, and its precision between 0.2917 and 0.6364. Two classes stand out. The recall of the `hidden_field_request` class is the highest at **0.8235**, but its precision is **0.4375**: the moderation step puts this label on 32 records and only 14 of them really belong to that class. Because the label stands at the front of the priority order, it pulls in every record that carries a hidden field name — 9 of them from another harmful class, 9 of them entirely clean. `privilege_escalation` is sharper: the label is put on 24 records, 7 are correct, **14 are from another harmful class**. This row carries a warning — the decision to block is **right** for these 14 records, because all of them really are harmful; only the decision given to the name is wrong. The class precision of a defense can be 0.2917 and the same defense can look good in the binary reading. **Blocking and naming are two separate jobs and are measured with two separate numbers.** The last row writes the cost: the moderation step counts 115 of the 147 clean records as clean and puts a harmful class name on **the other 32**. The false block rate is 0.2177, nine resolution steps higher than the 0.1565 of the rule-based filter. Half of the 6 missed harmful records again belong to the corrupted input class. ## Recall per Class, Three Defenses The same class split applies to the two filters of the previous lesson. The third block prints four columns: the rule-based filter, the model-based filter, the second run of the same model filter, and the binary decision of the moderation step. What is measured here is not the class label but **whether the record is flagged**. ```python def score_second(k): """Same model, second run: only the noise seed changes.""" p = sum(WEIGHT[o] for o in FEATURES if k["features"][o]) + SOURCE_BONUS[k["source"]] r = rng(20260219 ^ trace_id(k["no"])) return min(max(p + (r() - 0.5) * 0.20, 0.0), 1.6) print(f"{'class':<20}{'n':>4}{'rule':>10}{'model 0.50':>12}" f"{'run 2':>10}{'moderation':>12}{'step':>8}") harmful_records = [k for k in RECORDS if harmful(k)] for s in CLASSES[1:]: g = [k for k in RECORDS if k["class"] == s] rates = [sum(1 for k in g if f(k)) / len(g) for f in (rule_filter, lambda k: model_filter(k, 0.50), lambda k: score_second(k) >= 0.50, lambda k: content_moderation(k) != "clean")] print(f"{s:<20}{len(g):>4}{rates[0]:>10.4f}{rates[1]:>12.4f}" f"{rates[2]:>10.4f}{rates[3]:>12.4f}{1 / len(g):>8.4f}") rates = [sum(1 for k in harmful_records if f(k)) / len(harmful_records) for f in (rule_filter, lambda k: model_filter(k, 0.50), lambda k: score_second(k) >= 0.50, lambda k: content_moderation(k) != "clean")] print(f"{'average':<20}{len(harmful_records):>4}{rates[0]:>10.4f}{rates[1]:>12.4f}" f"{rates[2]:>10.4f}{rates[3]:>12.4f}{1 / len(harmful_records):>8.4f}") print() print("false block rate:", {"rule": measure(RECORDS, rule_filter)["false_block_rate"], "model 0.50": measure(RECORDS, lambda k: model_filter(k, 0.50))["false_block_rate"], "moderation": binary["false_block_rate"]}) ``` ``` class n rule model 0.50 run 2 moderation step indirect_instruction 24 1.0000 0.9583 0.9583 1.0000 0.0417 hidden_field_request 17 0.9412 0.7059 0.7059 0.9412 0.0588 privilege_escalation 10 0.9000 0.8000 0.8000 0.9000 0.1000 source_spoofing 23 0.8696 0.8261 0.8261 0.9565 0.0435 corrupted_input 19 0.2632 0.2105 0.3684 0.8421 0.0526 average 93 0.7957 0.7097 0.7419 0.9355 0.0108 false block rate: {'rule': 0.1565, 'model 0.50': 0.0544, 'moderation': 0.2177} ``` ## The Class Below the Baseline The average of 0.7957 of the rule-based filter lies between 0.8696 and 1.0000 in four classes and is **0.2632** in the fifth. The average is four good numbers carrying one bad number; this row does not tell the reader which class is unprotected. The moderation step raises that class to 0.8421, because it reads the stamp distribution that the rule set does not look at — but the cost is in the same row: the false block rate rises from 0.1565 to 0.2177, and the blocked clean records from 23 to 32. The corrupted input row has to be read together with the second run. The recall of the model-based filter in this class moves between 0.2105 and 0.3684 across the two runs; the difference is 0.1579, that is, **three steps** of the resolution of that class. In the other four classes the difference is exactly zero. The weakest class is also the most unstable class, and the number reported from a single run for that class is not a number but a band. Now the harshest application of the course's rule. In the `indirect_instruction` class the moderation step puts this label on **21 records**; **8 are correct**, 4 are from another harmful class, and **9 are entirely clean**. If the class label is tied to an action — a warning specific to that class, a rewrite, a human review — the action fires correctly 8 times and **wrongly 13 times** in this set, and in 9 of the wrong firings there is a real subscriber on the other side. On the undefended baseline this action never fires: neither rightly nor wrongly. If the cost of a wrong firing is not smaller than the gain of a right firing, **in this class the defense is below the baseline.** The difference between 8 correct labels and 9 labels put on clean records is a single record and could change sign in another draw; what is readable is that the ratio is **close to one**. The fact that the same class reads 1.0000 in the binary column completes this table. The moderation step blocks **all** of the indirect instruction class and puts the right name on **a third** of it. The choice of which to write, if a single rate were reported, depends on what the defense was built for — and that is why both are written. ## Summary - The content moderation step that produces a class name blocks 87 harmful records and blocks 32 clean records in the binary reading: precision 0.7311, recall 0.9355, false block rate 0.2177. - The class label of the same moderation step is much weaker: class precision spreads between 0.2917 and 0.6364, class recall between 0.3333 and 0.8235. Blocking and naming are two separate jobs. - The average of 0.7957 of the rule-based filter is above 0.8696 in four classes and 0.2632 in the corrupted input class; the moderation step raises that class to 0.8421, and the false block rate rises from 0.1565 to 0.2177. - The corrupted input class moves between 0.2105 and 0.3684 across two runs, that is, three steps of the class resolution; in the other four classes the difference is zero. - In the indirect instruction class the label is put on 21 records, 8 are correct and 9 land on entirely clean records; if the class label is tied to an action, the action fires wrongly far more often than rightly and the defense falls below the baseline in that class. ## Next Step In two lessons the defense decided on the **whole** of the record: let through, block, or put a class name. In all three the record was an indivisible whole. Yet the part of a record that should go to the model can be separated from the part that should not: the subscriber identity, the contact information, the address and the access string can be removed and the rest sent. This is a defense that blocks no record, so it has no false positive — its measure has to be read from somewhere else. The next lesson counts how many records masking cleans and **how many tasks lose the field they require**.