Training Data Poisoning

What the previous lesson counted belonged to the request: a record arrived, carried the side-effect tool name flag, and the approval gate routed it to a human or did not. The decision rested on a single record, and once processed the record left no trace in the system. A system fed from its own traffic does not work this way. Answered support requests, retrieved documents, and tool outputs are collected in a corpus, that corpus becomes the input of the next adaptation, and from that point the harmful attribute profile stops being an event and becomes a feature.

This is exactly what training data poisoning is, and this lesson treats it not as a procedure but as a class and a count. The question is: what share of the records entering the corpus carries a harmful attribute profile, how does that share change by source class, and what does keeping a source out of the corpus gain and lose? No poisoning procedure, sample generation, or label corruption method is written.

The distinction decides the unit of measurement. When a defense at run time decides wrongly, a single record is affected and the decision can be reversed; a record that enters the corpus by mistake spreads to every response fed by that corpus and is not repaired by undoing one record. So on the corpus side the measured unit is not the record but the share.

• AS49. The record set, the source classes, and the gates are fictional; there is no real corpus or training run. Seed 20260218.
• AS50. An attack is not a text but an attribute profile. The text of the records is entirely harmless support requests and has no relation to the class; the class lives only in the attribute profile.
• AS51. Poisoning here is the harmful share of the corpus. What a record does to the model after entering the corpus is neither measured nor claimed in this lesson; the only thing measured is the composition of the corpus.
• AS52. The baseline is the unfiltered corpus: no gate, all 240 records are taken, stopped harmful is 0, lost clean is 0.
• AS53. The source stamp is read from the record itself and assumed correct. Spoofing the stamp is a separate class (source_spoofing) and this lesson counts that class’s stamp as correct too; so the count for filtering by source here is the most optimistic count.
• AS54. The source gate is a corpus gate, not a block at run time. Its cost is not blocked traffic but the number of records not taken into the corpus.
• AS55. Resolution: the step of the share is 0.0042 at 240 records, 0.0070 if the remaining corpus is 143 records, and 0.0106 at 94 records. The step per source is 0.0073 for the user, 0.0139 for the retrieved document, and 0.0323 for the tool output.
• AS56. Data quality, data leakage, and the training data preparation procedure were measured in earlier curricula and are not repeated; this lesson adds only the share per source.

Poisoning Is a Share, Not a Procedure

The first block builds the set, the classes, and the rule filter. The source of a record can be user, retrieved_document, or tool_output, and every class has a different source distribution: the indirect injection class comes mostly from retrieved documents, the corrupted input class mostly from the user. This distribution is fictional, and the poisoning count comes from it.

# FICTIONAL. Records, source classes and gates are defined inside the lesson; there is no
# real corpus or training run. No attack TEXT is written: it is a FEATURE PROFILE.
SEED, M32 = 20260218, 0xFFFFFFFF


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def pick(u, w):
    t = 0.0
    for i, x in enumerate(w):
        t += x
        if u < t:
            return i
    return len(w) - 1


CLASSES = ["clean", "indirect_instruction", "hidden_field_request", "privilege_escalation",
           "source_spoofing", "corrupted_input"]
SOURCES = ["user", "retrieved_document", "tool_output"]
# Features are structural; none of them is a working payload.
FEATURES = ["instruction_mode", "role_change_request", "hidden_field_name", "side_effect_tool_name",
            "source_stamp_mismatch", "unusual_stamp_distribution", "excessive_length"]
PROFILE = {
    "clean":                [0.06, 0.02, 0.05, 0.03, 0.02, 0.04, 0.05],
    "indirect_instruction": [0.88, 0.42, 0.10, 0.30, 0.35, 0.08, 0.12],
    "hidden_field_request": [0.35, 0.15, 0.86, 0.12, 0.10, 0.06, 0.08],
    "privilege_escalation": [0.30, 0.25, 0.12, 0.84, 0.14, 0.05, 0.07],
    "source_spoofing":      [0.22, 0.55, 0.08, 0.10, 0.82, 0.07, 0.06],
    "corrupted_input":      [0.05, 0.03, 0.04, 0.03, 0.06, 0.80, 0.62],
}
SOURCE_SHARE = {
    "clean":                [0.72, 0.20, 0.08],
    "indirect_instruction": [0.05, 0.70, 0.25],
    "hidden_field_request": [0.55, 0.30, 0.15],
    "privilege_escalation": [0.40, 0.35, 0.25],
    "source_spoofing":      [0.10, 0.60, 0.30],
    "corrupted_input":      [0.65, 0.20, 0.15],
}
# Entirely harmless support texts; they have NO relation to the class.
TEXT = ["bu ayki fatura tutari gecen aya gore yuksek geldi",
        "sayac endeksi yanlis girilmis olabilir",
        "mahallede kesinti ne zaman bitecek",
        "tarife basamagimi ogrenmek istiyorum",
        "ariza kaydim icin ekip ne zaman gelir",
        "adres degisikligi nasil yapilir"]


def build_set(seed=SEED):
    ks = []
    for i in range(240):
        r = rng(seed + 6151 * i)
        c = CLASSES[pick(r(), [0.62, 0.10, 0.07, 0.07, 0.07, 0.07])]
        source = SOURCES[pick(r(), SOURCE_SHARE[c])]
        feat = {o: (r() < p) for o, p in zip(FEATURES, PROFILE[c])}
        ks.append({"no": i + 1, "class": c, "source": source, "features": feat,
                   "text": TEXT[int(r() * len(TEXT))],
                   "length": 40 + int(r() * 260)})
    return ks


RECORDS = build_set()


def harmful(k):
    return k["class"] != "clean"


def rule_filter(k, rules=("instruction_mode", "hidden_field_name", "side_effect_tool_name",
                          "source_stamp_mismatch")):
    return any(k["features"][o] for o in rules)


def measure(records, decision):
    """What was stopped and what was stopped by mistake are written TOGETHER."""
    tp = fp = tn = fn = 0
    for k in records:
        flagged, actual = decision(k), harmful(k)
        if flagged and actual:
            tp += 1
        elif flagged and not actual:
            fp += 1
        elif not flagged and actual:
            fn += 1
        else:
            tn += 1
    precision = tp / (tp + fp) if tp + fp else 0.0
    recall = tp / (tp + fn) if tp + fn else 0.0
    return {"tp": tp, "fp": fp, "fn": fn, "tn": tn,
            "precision": round(precision, 4), "recall": round(recall, 4),
            "false_block_rate": round(fp / (fp + tn), 4) if fp + tn else 0.0}

Counting per Source

The second block first puts the harmful share per source, then four corpus gates side by side. A record caught by a gate is not blocked, only left out of the corpus; so the cost column is lost clean records.

def assemble(records, gate):
    """Corpus gate: a record caught by the gate is NOT taken into the corpus."""
    kept = [k for k in records if not gate(k)]
    dropped = [k for k in records if gate(k)]
    h = sum(1 for k in kept if harmful(k))
    return (len(kept), h, h / len(kept) if kept else 0.0,
            len(dropped), sum(1 for k in dropped if not harmful(k)))


def non_user(k):
    return k["source"] != "user"


print("source              records  harmful  harmful share")
for src in SOURCES:
    group = [k for k in RECORDS if k["source"] == src]
    h = sum(1 for k in group if harmful(k))
    print(f"{src:<19} {len(group):>7} {h:>8} {h / len(group):>14.4f}")
total = sum(1 for k in RECORDS if harmful(k))
print(f"{'whole set':<19} {len(RECORDS):>7} {total:>8} {total / len(RECORDS):>14.4f}")

print("\ncorpus gate       kept  harmful  harmful share  dropped  dropped clean")
for name, gate in (("no filter", lambda k: False), ("source gate", non_user),
                   ("rule gate", rule_filter),
                   ("source + rule", lambda k: non_user(k) or rule_filter(k))):
    n, h, share, nd, dc = assemble(RECORDS, gate)
    print(f"{name:<16} {n:>5} {h:>8} {share:>14.4f} {nd:>8} {dc:>14}")

print("\nrule filter per source")
for src in SOURCES:
    s = measure([k for k in RECORDS if k["source"] == src], rule_filter)
    print(f"  {src:<19} tp {s['tp']:>3}  fp {s['fp']:>3}"
          f"  precision {s['precision']:.4f}  recall {s['recall']:.4f}")

print("\nclasses of the 37 harmful records of the user channel")
uh = [k for k in RECORDS if k["source"] == "user" and harmful(k)]
print("  " + "  ".join(f"{c} {sum(1 for k in uh if k['class'] == c)}"
                       for c in CLASSES[1:] if any(k["class"] == c for k in uh)))

print("\nharmful share per source, two runs")
for t in (20260218, 20260219):
    ks = build_set(t)
    row = []
    for src in SOURCES:
        group = [k for k in ks if k["source"] == src]
        row.append(f"{src} {sum(1 for k in group if harmful(k)) / len(group):.4f}")
    print(f"  {t}: " + "  ".join(row))

source              records  harmful  harmful share
user                    137       37         0.2701
retrieved_document       72       39         0.5417
tool_output              31       17         0.5484
whole set               240       93         0.3875

corpus gate       kept  harmful  harmful share  dropped  dropped clean
no filter          240       93         0.3875        0              0
source gate        137       37         0.2701      103             47
rule gate          143       19         0.1329       97             23
source + rule       94       13         0.1383      146             66

rule filter per source
  user                tp  24  fp  19  precision 0.5581  recall 0.6486
  retrieved_document  tp  37  fp   4  precision 0.9024  recall 0.9487
  tool_output         tp  13  fp   0  precision 1.0000  recall 0.7647

classes of the 37 harmful records of the user channel
  indirect_instruction 2  hidden_field_request 13  privilege_escalation 3  source_spoofing 3  corrupted_input 16

harmful share per source, two runs
  20260218: user 0.2701  retrieved_document 0.5417  tool_output 0.5484
  20260219: user 0.2323  retrieved_document 0.5484  tool_output 0.5652

The Rate and the Volume of a Channel

The first table builds the case for filtering by source: the harmful share is 0.5417 for the retrieved document, 0.5484 for the tool output, and 0.2701 for the user. The two non-user channels carry a rate close to twice that of the user channel. The difference between the user and the retrieved document is 0.2716, more than nineteen times the 0.0139 step of the retrieved document channel; this is not a fluctuation but a measurement. Anyone reading up to here reaches a clear conclusion: the corpus should take only records coming from the user.

The second column of the same table breaks that conclusion. The user channel adds 37 harmful records, the retrieved document 39, and the tool output 17. The lowest-rate channel brings only two fewer harmful records than the retrieved document channel, because 137 of the records come from there. The rate is a property of a channel, the volume is its weight; when the two are not written separately, the sentence “what comes from the user is cleaner” is both true and misleading at once. Any decision that leaves the user channel open keeps about two fifths of the harmful records of the corpus in place.

The second table puts two gates side by side. The source gate lowers the corpus share from 0.3875 to 0.2701, drops 103 records, and 47 of them are clean. The rule gate lowers the same share to 0.1329, drops 97 records, and 23 of them are clean. The rule gate is superior on both axes: it leaves less harmful in the corpus and loses fewer clean records. The reason is structural. The source is a proxy for the class, and a weak one; the attribute profile stands much closer to the class itself. The farther a proxy criterion is from what it wants to measure, the more clean records it burns.

The difference between the third and the fourth row is the place in this lesson that needs the most careful reading. Placing the source gate on top of the rule gate moves the share from 0.1329 to 0.1383, that is, it does not improve it. But writing this difference as a finding would be wrong: in a 94-record corpus the step of the share is 0.0106 and the difference is 0.0054, below the step. The two shares are indistinguishable; the gain of the layer is unmeasurable. What can be measured is the cost: lost clean records rise from 23 to 66, forty-three more, and the corpus shrinks from 143 to 94. The second claim of the course appears here in its cleanest form — the added defense charged a measurable cost for an unmeasurable gain. Saying that a setup has two gates and counting two layers means, in this table, one layer and forty-three losses.

The third table shows why the rule filter behaves differently across channels. Recall is 0.9487 on the retrieved document, 0.7647 on the tool output, and only 0.6486 on the user. The filter is weakest in the channel from which most records come. The reason is written in the fourth table: 16 of the 37 harmful records of the user channel fall in the corrupted input class, and the two flags that class carries, unusual_stamp_distribution and excessive_length, are not in the rule list. The weakness is a property not of the channel but of the list, and it cannot be closed by closing a channel. The precision of 1.0000 in the same table also needs careful reading: there are only 14 clean records in the tool output channel, the step of the false block rate there is 0.0714, so that perfect number is a number read from fourteen records.

The same table also shows the weakest place of a source-based defense. The source spoofing class is, by its name, the class where the source stamp does not agree with the rest of the record; 3 of the 23 records of that class carry a user stamp and pass the source gate clean. Because this lesson counts the stamp as correct, the 0.2701 of the source gate is its most optimistic number. In a setup where the stamp is decided by the party sending the record, the gate reads its decision from a field written by the very party it wants to filter. The rule filter carries no such dependency: the flags it looks at come from the structure of the record, not from its stamp. Part of the difference between 0.2701 and 0.1329 across the two gates is the cost of this dependency, and when the stamp stops being trustworthy the difference does not close, it widens.

The last lines say which part of the measurement is independent of the run. In the second draw the shares move: the user from 0.2701 to 0.2323, the retrieved document from 0.5417 to 0.5484, the tool output from 0.5484 to 0.5652. None of the values is fixed. What is fixed is the ordering: the user is lowest and the tool output highest in both runs. A source-based priority can safely be built on the ordering; a threshold built on a particular share moves out of place in the second run.

Summary

• In the unfiltered corpus 93 of 240 records are harmful, a share of 0.3875; lost clean records in the baseline are 0.
• The harmful share per source is 0.2701 for the user, 0.5417 for the retrieved document, 0.5484 for the tool output; but volume reverses the order, and the lowest-share channel adds 37 harmful records.
• The source gate lowers the share to 0.2701 and loses 47 clean records; the rule gate lowers it to 0.1329 and loses 23 clean records, so it is superior on both axes. The source is a weak proxy for the class.
• Placing the source gate on top of the rule gate gives a share of 0.1383; the difference of 0.0054 is below the 0.0106 step of the 94-record corpus and is unmeasurable. Lost clean records rise from 23 to 66.
• Rule filter recall is 0.6486 on the user channel and 0.9487 on the retrieved document; the weakness is a property of the rule list, not of the channel.
• The shares move across two runs; the ordering of the channels does not.

Next Step

Everything counted in these two lessons was on the input side: the flag the record carried, the channel it came from, the gate sending it to a human or not taking it into the corpus. Both the rule filter and the source gate decided before anything reached the model. But the information a system leaks does not have to pass through the input. A model can put a field in its response without ever being asked, can reveal that a subscriber is in the set through an expression that carries no field directly about that subscriber, and none of this produces a flag on the input side. The next lesson moves the measure to the output side: how many responses carry a field of a secret field class, how many the output filter removes, and what is left after the filter.