--- title: 'Training Data Poisoning' source: 'https://academia.sh/en/courses/ai-safety-and-evaluation/training-data-poisoning' course: 'AI Safety and Evaluation' language: en updated: '2026-10-21T05:02:27+00:00' license: 'CC BY-SA 4.0' objectives: - 'Computes the harmful share per source class and reads the share apart from the volume.' - 'Writes what filtering by source adds to the harmful share of the corpus together with the clean records it loses.' - 'Compares the gain of a source gate placed on top of the rule gate with the resolution of the set.' --- # Training Data Poisoning Poisoning counted as the harmful share per source class, not as a procedure: 93 of 240 records in the unfiltered corpus are harmful, a share of 0.3875. The share is 0.2701 for the user, 0.5417 for the retrieved document, 0.5484 for the tool output, yet volume reverses the order: the lowest-share channel adds 37 harmful records and the highest adds 17. The source gate lowers the corpus share to 0.2701 and loses 47 clean records; the rule gate lowers it to 0.1329 and loses 23. Using both gives 0.1383, a difference below the 0.0106 step of a 94-record corpus, so unmeasurable, while lost clean records rise from 23 to 66. Rule filter recall is 0.6486 on the user channel and 0.9487 on retrieved documents. Shares move across two runs; the ordering does not. What the previous lesson counted belonged to the **request**: a record arrived, carried the side-effect tool name flag, and the approval gate routed it to a human or did not. The decision rested on a single record, and once processed the record left no trace in the system. A system fed from its own traffic does not work this way. Answered support requests, retrieved documents, and tool outputs are collected in a **corpus**, that corpus becomes the input of the next adaptation, and from that point the harmful attribute profile stops being an event and becomes a **feature**. This is exactly what training data poisoning is, and this lesson treats it not as a procedure but as **a class and a count**. The question is: what share of the records entering the corpus carries a harmful attribute profile, how does that share change by source class, and what does keeping a source out of the corpus gain and lose? No poisoning procedure, sample generation, or label corruption method is written. The distinction decides the unit of measurement. When a defense at run time decides wrongly, a single record is affected and the decision can be reversed; a record that enters the corpus by mistake spreads to every response fed by that corpus and is not repaired by undoing one record. So on the corpus side the measured unit is not the record but the **share**. - **AS49.** The record set, the source classes, and the gates are **fictional**; there is no real corpus or training run. Seed **20260218**. - **AS50.** An attack is not a text but an **attribute profile**. The text of the records is entirely harmless support requests and has **no relation to the class**; the class lives only in the attribute profile. - **AS51.** Poisoning here is **the harmful share of the corpus**. What a record does to the model after entering the corpus is neither measured nor claimed in this lesson; the only thing measured is the composition of the corpus. - **AS52.** The baseline is the **unfiltered corpus**: no gate, all 240 records are taken, stopped harmful is **0**, lost clean is **0**. - **AS53.** The source stamp is read from the record itself and assumed **correct**. Spoofing the stamp is a separate class (`source_spoofing`) and this lesson counts that class's stamp as correct too; so the count for filtering by source here is the **most optimistic** count. - **AS54.** The source gate is a **corpus gate**, not a block at run time. Its cost is not blocked traffic but the number of records not taken into the corpus. - **AS55.** Resolution: the step of the share is **0.0042** at 240 records, **0.0070** if the remaining corpus is 143 records, and **0.0106** at 94 records. The step per source is **0.0073** for the user, **0.0139** for the retrieved document, and **0.0323** for the tool output. - **AS56.** Data quality, data leakage, and the training data preparation procedure were measured in earlier curricula and are not repeated; this lesson adds only the share per source. ## Poisoning Is a Share, Not a Procedure The first block builds the set, the classes, and the rule filter. The source of a record can be `user`, `retrieved_document`, or `tool_output`, and every class has a different source distribution: the indirect injection class comes mostly from retrieved documents, the corrupted input class mostly from the user. This distribution is fictional, and the poisoning count comes from it. ```python # FICTIONAL. Records, source classes and gates are defined inside the lesson; there is no # real corpus or training run. No attack TEXT is written: it is a FEATURE PROFILE. SEED, M32 = 20260218, 0xFFFFFFFF def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 CLASSES = ["clean", "indirect_instruction", "hidden_field_request", "privilege_escalation", "source_spoofing", "corrupted_input"] SOURCES = ["user", "retrieved_document", "tool_output"] # Features are structural; none of them is a working payload. FEATURES = ["instruction_mode", "role_change_request", "hidden_field_name", "side_effect_tool_name", "source_stamp_mismatch", "unusual_stamp_distribution", "excessive_length"] PROFILE = { "clean": [0.06, 0.02, 0.05, 0.03, 0.02, 0.04, 0.05], "indirect_instruction": [0.88, 0.42, 0.10, 0.30, 0.35, 0.08, 0.12], "hidden_field_request": [0.35, 0.15, 0.86, 0.12, 0.10, 0.06, 0.08], "privilege_escalation": [0.30, 0.25, 0.12, 0.84, 0.14, 0.05, 0.07], "source_spoofing": [0.22, 0.55, 0.08, 0.10, 0.82, 0.07, 0.06], "corrupted_input": [0.05, 0.03, 0.04, 0.03, 0.06, 0.80, 0.62], } SOURCE_SHARE = { "clean": [0.72, 0.20, 0.08], "indirect_instruction": [0.05, 0.70, 0.25], "hidden_field_request": [0.55, 0.30, 0.15], "privilege_escalation": [0.40, 0.35, 0.25], "source_spoofing": [0.10, 0.60, 0.30], "corrupted_input": [0.65, 0.20, 0.15], } # Entirely harmless support texts; they have NO relation to the class. TEXT = ["bu ayki fatura tutari gecen aya gore yuksek geldi", "sayac endeksi yanlis girilmis olabilir", "mahallede kesinti ne zaman bitecek", "tarife basamagimi ogrenmek istiyorum", "ariza kaydim icin ekip ne zaman gelir", "adres degisikligi nasil yapilir"] def build_set(seed=SEED): ks = [] for i in range(240): r = rng(seed + 6151 * i) c = CLASSES[pick(r(), [0.62, 0.10, 0.07, 0.07, 0.07, 0.07])] source = SOURCES[pick(r(), SOURCE_SHARE[c])] feat = {o: (r() < p) for o, p in zip(FEATURES, PROFILE[c])} ks.append({"no": i + 1, "class": c, "source": source, "features": feat, "text": TEXT[int(r() * len(TEXT))], "length": 40 + int(r() * 260)}) return ks RECORDS = build_set() def harmful(k): return k["class"] != "clean" def rule_filter(k, rules=("instruction_mode", "hidden_field_name", "side_effect_tool_name", "source_stamp_mismatch")): return any(k["features"][o] for o in rules) def measure(records, decision): """What was stopped and what was stopped by mistake are written TOGETHER.""" tp = fp = tn = fn = 0 for k in records: flagged, actual = decision(k), harmful(k) if flagged and actual: tp += 1 elif flagged and not actual: fp += 1 elif not flagged and actual: fn += 1 else: tn += 1 precision = tp / (tp + fp) if tp + fp else 0.0 recall = tp / (tp + fn) if tp + fn else 0.0 return {"tp": tp, "fp": fp, "fn": fn, "tn": tn, "precision": round(precision, 4), "recall": round(recall, 4), "false_block_rate": round(fp / (fp + tn), 4) if fp + tn else 0.0} ``` ## Counting per Source The second block first puts the harmful share per source, then four corpus gates side by side. A record caught by a gate is not blocked, only left out of the corpus; so the cost column is **lost clean records**. ```python def assemble(records, gate): """Corpus gate: a record caught by the gate is NOT taken into the corpus.""" kept = [k for k in records if not gate(k)] dropped = [k for k in records if gate(k)] h = sum(1 for k in kept if harmful(k)) return (len(kept), h, h / len(kept) if kept else 0.0, len(dropped), sum(1 for k in dropped if not harmful(k))) def non_user(k): return k["source"] != "user" print("source records harmful harmful share") for src in SOURCES: group = [k for k in RECORDS if k["source"] == src] h = sum(1 for k in group if harmful(k)) print(f"{src:<19} {len(group):>7} {h:>8} {h / len(group):>14.4f}") total = sum(1 for k in RECORDS if harmful(k)) print(f"{'whole set':<19} {len(RECORDS):>7} {total:>8} {total / len(RECORDS):>14.4f}") print("\ncorpus gate kept harmful harmful share dropped dropped clean") for name, gate in (("no filter", lambda k: False), ("source gate", non_user), ("rule gate", rule_filter), ("source + rule", lambda k: non_user(k) or rule_filter(k))): n, h, share, nd, dc = assemble(RECORDS, gate) print(f"{name:<16} {n:>5} {h:>8} {share:>14.4f} {nd:>8} {dc:>14}") print("\nrule filter per source") for src in SOURCES: s = measure([k for k in RECORDS if k["source"] == src], rule_filter) print(f" {src:<19} tp {s['tp']:>3} fp {s['fp']:>3}" f" precision {s['precision']:.4f} recall {s['recall']:.4f}") print("\nclasses of the 37 harmful records of the user channel") uh = [k for k in RECORDS if k["source"] == "user" and harmful(k)] print(" " + " ".join(f"{c} {sum(1 for k in uh if k['class'] == c)}" for c in CLASSES[1:] if any(k["class"] == c for k in uh))) print("\nharmful share per source, two runs") for t in (20260218, 20260219): ks = build_set(t) row = [] for src in SOURCES: group = [k for k in ks if k["source"] == src] row.append(f"{src} {sum(1 for k in group if harmful(k)) / len(group):.4f}") print(f" {t}: " + " ".join(row)) ``` ``` source records harmful harmful share user 137 37 0.2701 retrieved_document 72 39 0.5417 tool_output 31 17 0.5484 whole set 240 93 0.3875 corpus gate kept harmful harmful share dropped dropped clean no filter 240 93 0.3875 0 0 source gate 137 37 0.2701 103 47 rule gate 143 19 0.1329 97 23 source + rule 94 13 0.1383 146 66 rule filter per source user tp 24 fp 19 precision 0.5581 recall 0.6486 retrieved_document tp 37 fp 4 precision 0.9024 recall 0.9487 tool_output tp 13 fp 0 precision 1.0000 recall 0.7647 classes of the 37 harmful records of the user channel indirect_instruction 2 hidden_field_request 13 privilege_escalation 3 source_spoofing 3 corrupted_input 16 harmful share per source, two runs 20260218: user 0.2701 retrieved_document 0.5417 tool_output 0.5484 20260219: user 0.2323 retrieved_document 0.5484 tool_output 0.5652 ``` ## The Rate and the Volume of a Channel The first table builds the case for filtering by source: the harmful share is **0.5417** for the retrieved document, **0.5484** for the tool output, and **0.2701** for the user. The two non-user channels carry a rate close to twice that of the user channel. The difference between the user and the retrieved document is **0.2716**, more than nineteen times the 0.0139 step of the retrieved document channel; this is not a fluctuation but a measurement. Anyone reading up to here reaches a clear conclusion: the corpus should take only records coming from the user. The second column of the same table breaks that conclusion. The user channel adds **37** harmful records, the retrieved document **39**, and the tool output **17**. The lowest-rate channel brings only two fewer harmful records than the retrieved document channel, because **137** of the records come from there. **The rate is a property of a channel, the volume is its weight**; when the two are not written separately, the sentence "what comes from the user is cleaner" is both true and misleading at once. Any decision that leaves the user channel open keeps about two fifths of the harmful records of the corpus in place. The second table puts two gates side by side. The source gate lowers the corpus share from 0.3875 to **0.2701**, drops 103 records, and **47 of them are clean**. The rule gate lowers the same share to **0.1329**, drops 97 records, and **23 of them are clean**. The rule gate is superior on both axes: it leaves less harmful in the corpus and loses fewer clean records. The reason is structural. The source is a **proxy** for the class, and a weak one; the attribute profile stands much closer to the class itself. The farther a proxy criterion is from what it wants to measure, the more clean records it burns. The difference between the third and the fourth row is the place in this lesson that needs the most careful reading. Placing the source gate on top of the rule gate moves the share from 0.1329 to **0.1383**, that is, it does not improve it. But writing this difference as a finding would be wrong: in a 94-record corpus the step of the share is **0.0106** and the difference is **0.0054**, below the step. The two shares are **indistinguishable**; the gain of the layer is unmeasurable. What can be measured is the cost: lost clean records rise from 23 to **66**, forty-three more, and the corpus shrinks from 143 to 94. The second claim of the course appears here in its cleanest form — **the added defense charged a measurable cost for an unmeasurable gain**. Saying that a setup has two gates and counting two layers means, in this table, one layer and forty-three losses. The third table shows why the rule filter behaves differently across channels. Recall is **0.9487** on the retrieved document, 0.7647 on the tool output, and only **0.6486** on the user. The filter is weakest in the channel from which most records come. The reason is written in the fourth table: **16** of the 37 harmful records of the user channel fall in the corrupted input class, and the two flags that class carries, `unusual_stamp_distribution` and `excessive_length`, are **not** in the rule list. The weakness is a property not of the channel but of **the list**, and it cannot be closed by closing a channel. The precision of 1.0000 in the same table also needs careful reading: there are only 14 clean records in the tool output channel, the step of the false block rate there is 0.0714, so that perfect number is a number read from fourteen records. The same table also shows the weakest place of a source-based defense. The source spoofing class is, by its name, the class where the source stamp does not agree with the rest of the record; **3** of the 23 records of that class carry a user stamp and pass the source gate clean. Because this lesson counts the stamp as correct, the 0.2701 of the source gate is its **most optimistic** number. In a setup where the stamp is decided by the party sending the record, the gate reads its decision from a field written by the very party it wants to filter. The rule filter carries no such dependency: the flags it looks at come from the structure of the record, not from its stamp. Part of the difference between 0.2701 and 0.1329 across the two gates is the cost of this dependency, and when the stamp stops being trustworthy the difference does not close, it widens. The last lines say which part of the measurement is independent of the run. In the second draw the shares move: the user from 0.2701 to 0.2323, the retrieved document from 0.5417 to 0.5484, the tool output from 0.5484 to 0.5652. None of the values is fixed. What is fixed is the **ordering**: the user is lowest and the tool output highest in both runs. A source-based priority can safely be built on the ordering; a threshold built on a particular share moves out of place in the second run. ## Summary - In the unfiltered corpus 93 of 240 records are harmful, a share of 0.3875; lost clean records in the baseline are 0. - The harmful share per source is 0.2701 for the user, 0.5417 for the retrieved document, 0.5484 for the tool output; but volume reverses the order, and the lowest-share channel adds 37 harmful records. - The source gate lowers the share to 0.2701 and loses 47 clean records; the rule gate lowers it to 0.1329 and loses 23 clean records, so it is superior on both axes. The source is a weak proxy for the class. - Placing the source gate on top of the rule gate gives a share of 0.1383; the difference of 0.0054 is below the 0.0106 step of the 94-record corpus and is unmeasurable. Lost clean records rise from 23 to 66. - Rule filter recall is 0.6486 on the user channel and 0.9487 on the retrieved document; the weakness is a property of the rule list, not of the channel. - The shares move across two runs; the ordering of the channels does not. ## Next Step Everything counted in these two lessons was **on the input side**: the flag the record carried, the channel it came from, the gate sending it to a human or not taking it into the corpus. Both the rule filter and the source gate decided before anything reached the model. But the information a system leaks does not have to pass through the input. A model can put a field in its response without ever being asked, can reveal that a subscriber is in the set through an expression that carries no field directly about that subscriber, and none of this produces a flag on the input side. The next lesson moves the measure to the **output side**: how many responses carry a field of a secret field class, how many the output filter removes, and what is left after the filter.