--- title: 'Untrusted Content Marking' source: 'https://academia.sh/en/courses/ai-safety-and-evaluation/flagging-untrusted-content' course: 'AI Safety and Evaluation' language: en updated: '2026-10-21T05:02:28+00:00' license: 'CC BY-SA 4.0' objectives: - 'Define the distinction between data and instruction through the source stamp and show why marking is not a block.' - 'Measure together the share of harmful records marking neutralizes, the harmful that remains and the zero false alarm.' - 'Determine which attack classes marking does nothing against by reading its distribution per class.' - 'Compare how a layer that does not block lets the filter threshold be tightened, using the covered share and the false block rate.' --- # Untrusted Content Marking Counting a defense that blocks no record: text coming from a source other than the user is marked as data and is not counted as an instruction even if it carries the instruction mode. On the undefended baseline the harmful stopped is 0 and the false alarm is 0; marking neutralizes 42 of the 93 harmful records without blocking any record, 51 remain, and the false alarm is zero. Neutralization gathers in two classes: 0.9167 of indirect instruction, 0.8696 of source spoofing; 0.0000 in the hidden field request, privilege escalation and corrupted input classes. The real value of marking is that it lets the filter be tightened: the rule filter alone covers 0.7957 and blocks 23 clean records, marking with the 0.50 threshold of the model filter covers 0.7634 and blocks only 8 clean records. The cost is not a false alarm: 47 of the 103 marked records are clean and only 1 of them carries the instruction mode, so the path lost is a single record and exactly one step of the resolution. The previous lesson measured the permission gate and counted that the four narrowing steps blocked 49 clean records in total. Every defense in that lesson worked in the same way: it **stopped** a record. The filter, content moderation, masking and the permission gate each protected by not letting something through, and the cost of all four came from the same place — the clean record blocked by mistake. Yet stopping is not the only way to neutralize a harmful record. The harm of text that comes from a retrieved document often comes not from its content but from that content **being read as an instruction**. If the text is counted as data, the same record becomes harmless without being blocked at all. This lesson measures that defense and asks a single question: how many of the 93 harmful records does a layer that stops no record cover, and what does it pay in return. - **DF49.** The code is a **simulator**; no real model endpoint or document store is called. The record set, the classes and the features are **fictional**, the seed is **20260218**. - **DF50. An attack is not a text but a feature profile.** The six sentences the records carry are ordinary support requests and have **no relation to the class**; the class lives only in the structural flags. There is no working payload, template or instruction string in the code. - **DF51. The baseline is the undefended setup:** the harmful stopped is **0**, the false alarm is **0**. - **DF52. The rule of the course:** a defense that does not write the share it stopped and the share it stopped by mistake **together** counts as unmeasured. - **DF53. Marking is not a block.** Text coming from a source other than the user is stamped as **data**; even if it carries the instruction mode, it is not counted as an instruction. No record is stopped, no record is deleted, no field is removed. - **DF54. The neutralized record** is a record that is in a class whose harm rests on being read as an instruction and that comes from a source other than the user. Both conditions are required together. - **DF55. The covered harmful** is the **union** of what marking neutralizes and what the filter blocks; the same record is not counted twice. - **DF56. The cost of marking is not a false alarm**, because the blocked records are zero. The cost is the **instruction path** that a clean record counted as data loses, and it is counted separately. - **DF57. Resolution:** the step of recall is $1/93 = 0.0108$, the step of the false block rate is $1/147 = 0.0068$. There is no draw in the decision of marking and the measure is **independent of the run**. ## Shared Definition The record set, the classes, the defense layers and the measure are the same throughout the course. The set holds 240 fictional support records; the source field says where a record comes from, and marking looks only at this field. ```python SEED, M32 = 20260218, 0xFFFFFFFF def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def trace_id(m): h = 2166136261 for c in str(m): h = ((h ^ ord(c)) * 16777619) & M32 return h # ---- FICTIONAL records: no attack TEXT is written, an attack is a FEATURE PROFILE CLASSES = ["clean", "indirect_instruction", "hidden_field_request", "privilege_escalation", "source_spoofing", "corrupted_input"] SOURCES = ["user", "retrieved_document", "tool_output"] # Features are structural; none of them is a working payload. FEATURES = ["instruction_mode", "role_change_request", "hidden_field_name", "side_effect_tool_name", "source_stamp_mismatch", "unusual_stamp_distribution", "excessive_length"] PROFILE = { "clean": [0.06, 0.02, 0.05, 0.03, 0.02, 0.04, 0.05], "indirect_instruction": [0.88, 0.42, 0.10, 0.30, 0.35, 0.08, 0.12], "hidden_field_request": [0.35, 0.15, 0.86, 0.12, 0.10, 0.06, 0.08], "privilege_escalation": [0.30, 0.25, 0.12, 0.84, 0.14, 0.05, 0.07], "source_spoofing": [0.22, 0.55, 0.08, 0.10, 0.82, 0.07, 0.06], "corrupted_input": [0.05, 0.03, 0.04, 0.03, 0.06, 0.80, 0.62], } SOURCE_SHARE = { "clean": [0.72, 0.20, 0.08], "indirect_instruction": [0.05, 0.70, 0.25], "hidden_field_request": [0.55, 0.30, 0.15], "privilege_escalation": [0.40, 0.35, 0.25], "source_spoofing": [0.10, 0.60, 0.30], "corrupted_input": [0.65, 0.20, 0.15], } # Fictional, entirely harmless support texts. They have NO relation to the class; # the class information lives only in the feature profile. TEXT = ["bu ayki fatura tutari gecen aya gore yuksek geldi", "sayac endeksi yanlis girilmis olabilir", "mahallede kesinti ne zaman bitecek", "tarife basamagimi ogrenmek istiyorum", "ariza kaydim icin ekip ne zaman gelir", "adres degisikligi nasil yapilir"] RECORDS = [] for i in range(240): r = rng(SEED + 6151 * i) c = CLASSES[pick(r(), [0.62, 0.10, 0.07, 0.07, 0.07, 0.07])] source = SOURCES[pick(r(), SOURCE_SHARE[c])] feat = {o: (r() < p) for o, p in zip(FEATURES, PROFILE[c])} RECORDS.append({"no": i + 1, "class": c, "source": source, "features": feat, "text": TEXT[int(r() * len(TEXT))], "length": 40 + int(r() * 260)}) def harmful(k): return k["class"] != "clean" # ---- DEFENSE 1: rule filter (structural, explainable) def rule_filter(k, rules=("instruction_mode", "hidden_field_name", "side_effect_tool_name", "source_stamp_mismatch")): return any(k["features"][o] for o in rules) # ---- DEFENSE 2: model-based filter (score + threshold) WEIGHT = {"instruction_mode": 0.42, "role_change_request": 0.26, "hidden_field_name": 0.44, "side_effect_tool_name": 0.40, "source_stamp_mismatch": 0.38, "unusual_stamp_distribution": 0.30, "excessive_length": 0.12} SOURCE_BONUS = {"user": 0.00, "retrieved_document": 0.18, "tool_output": 0.12} def score(k): p = sum(WEIGHT[o] for o in FEATURES if k["features"][o]) + SOURCE_BONUS[k["source"]] r = rng(SEED ^ trace_id(k["no"])) return min(max(p + (r() - 0.5) * 0.20, 0.0), 1.6) def model_filter(k, threshold=0.40): return score(k) >= threshold # ---- DEFENSE 3: untrusted content marking (not counted as an instruction) def mark(k): """Text that comes from a source other than the user is marked as DATA; even if it carries the instruction mode, it is not counted as an instruction.""" return {"as_data": k["source"] != "user"} def marking_effect(records): """Marking closes only the 'read as an instruction' path.""" neutralized, remaining = 0, 0 for k in records: if not harmful(k): continue instruction_based = k["class"] in ("indirect_instruction", "source_spoofing") if instruction_based and mark(k)["as_data"]: neutralized += 1 else: remaining += 1 return neutralized, remaining # ---- MEASURE: what was blocked and what was blocked by mistake are written together def measure(records, decision): tp = fp = tn = fn = 0 for k in records: flagged, actual = decision(k), harmful(k) if flagged and actual: tp += 1 elif flagged and not actual: fp += 1 elif not flagged and actual: fn += 1 else: tn += 1 precision = tp / (tp + fp) if tp + fp else 0.0 recall = tp / (tp + fn) if tp + fn else 0.0 return {"tp": tp, "fp": fp, "fn": fn, "tn": tn, "precision": round(precision, 4), "recall": round(recall, 4), "false_block_rate": round(fp / (fp + tn), 4) if fp + tn else 0.0} print("records:", len(RECORDS), "- harmful:", sum(harmful(k) for k in RECORDS)) print("source:", {s: sum(k["source"] == s for k in RECORDS) for s in SOURCES}) ``` ``` records: 240 - harmful: 93 source: {'user': 137, 'retrieved_document': 72, 'tool_output': 31} ``` ## The Data-Instruction Distinction Marking breaks a single assumption: that all the text entering the model is read with the same weight. The user's own request is an instruction, because the user is the will the agent serves. Text that comes from a retrieved document or a tool output is **data** — it is read, quoted, summarized, but it does not change behavior. The `mark` function puts this stamp by looking only at the source field and stops no record. Whom the distinction affects depends on the class. If the harm of a record rests on **being read as an instruction**, the stamp removes that harm. If the harm rests on something else, the stamp does nothing: asking for a hidden field name is a request even when it is read as data. ```python def neutralized(k): """A record whose harm rests on BEING READ AS AN INSTRUCTION and that comes from a source other than the user stays neutral without anything being blocked: the text is counted as data, not as an instruction.""" return (harmful(k) and k["class"] in ("indirect_instruction", "source_spoofing") and mark(k)["as_data"]) neutralized_count, remaining = marking_effect(RECORDS) marked = [k for k in RECORDS if mark(k)["as_data"]] clean_marked = [k for k in marked if not harmful(k)] print("baseline : harmful stopped 0 , false alarm 0") print("marking : neutralized", neutralized_count, ", remaining harmful", remaining, ", covered share", round(neutralized_count / 93, 4)) print("marked records", len(marked), "- clean among them", len(clean_marked), "- carrying instruction mode", sum(k["features"]["instruction_mode"] for k in clean_marked)) print("blocked records 0 - false alarm 0") print("\nper-class share neutralized by marking") for s in CLASSES[1:]: g = [k for k in RECORDS if k["class"] == s] e = sum(neutralized(k) for k in g) print(f" {s:<20} {e:>2}/{len(g):<3} {e / len(g):.4f}") def setup(decision=None, marking=True): """Covered harmful = neutralized by marking OR blocked by the filter.""" s = measure(RECORDS, decision) if decision else {"tp": 0, "fp": 0, "false_block_rate": 0.0} covered = sum(1 for k in RECORDS if harmful(k) and ((marking and neutralized(k)) or (decision is not None and decision(k)))) return covered, s["tp"] + s["fp"], s["fp"], s["false_block_rate"] print("\nsetup covered share blocked false alarm false-block") for name, decision, mk in [("baseline", None, False), ("marking only", None, True), ("rule filter only", rule_filter, False), ("marking + rule", rule_filter, True), ("marking + model 0.50", lambda k: model_filter(k, 0.50), True), ("marking + model 0.60", lambda k: model_filter(k, 0.60), True), ("marking + model 0.80", lambda k: model_filter(k, 0.80), True)]: covered, blocked, fa, fbr = setup(decision, mk) print(f"{name:<24} {covered:>7} {covered / 93:>7.4f} {blocked:>11} {fa:>13} {fbr:>11.4f}") ``` ``` baseline : harmful stopped 0 , false alarm 0 marking : neutralized 42 , remaining harmful 51 , covered share 0.4516 marked records 103 - clean among them 47 - carrying instruction mode 1 blocked records 0 - false alarm 0 per-class share neutralized by marking indirect_instruction 22/24 0.9167 hidden_field_request 0/17 0.0000 privilege_escalation 0/10 0.0000 source_spoofing 20/23 0.8696 corrupted_input 0/19 0.0000 setup covered share blocked false alarm false-block baseline 0 0.0000 0 0 0.0000 marking only 42 0.4516 0 0 0.0000 rule filter only 74 0.7957 97 23 0.1565 marking + rule 77 0.8280 97 23 0.1565 marking + model 0.50 71 0.7634 74 8 0.0544 marking + model 0.60 67 0.7204 64 5 0.0340 marking + model 0.80 60 0.6452 47 1 0.0068 ``` ## The Number of a Defense That Does Not Block The second row is the most striking row of this course. Marking neutralizes **42** of the 93 harmful records — the covered share is **0.4516** — and it does so **without blocking any record**. The blocked records column is 0, the false alarm column is 0, the false block rate is 0.0000. The rule of the course asks for the two numbers of a defense together; the second number of this defense is exactly zero, and this is not a rounding but the count itself. For comparison the third row has to be read. The rule filter covers more — **0.7957** — but it blocks 97 records for this, and 23 of those blocks are on clean records. The two defenses are not of the same kind: one buys coverage with blocking, the other buys it free. The main finding is in the last three rows. Marking **allows the threshold of the filter behind it to be tightened**. Because most of the instruction-based harmful is already neutralized, the filter does not need to catch them; it can focus only on the remaining 51 records. Marking together with the 0.50 threshold of the model filter covers **0.7634** and blocks only **8** clean records. The difference from the 0.7957 of the rule filter alone is three records, that is $3 \times 0.0108 = 0.0323$; in return the false block rate falls from 0.1565 to 0.0544, that is, fifteen fewer clean records are blocked, and this is $15 \times 0.0068 = 0.1021$. A coverage difference of three records is given in exchange for a blocking difference of fifteen records. In the tightest row, at the 0.80 threshold, the blocked clean records fall to **one** and the coverage still stays at 0.6452. The fourth row shows one more thing: putting marking on top of the rule filter raises the coverage from 0.7957 to 0.8280 and **does not change the false block rate at all**. In the previous lesson, putting the permission gate on top of the filter raised recall by 0.0430 while it raised the false block rate by 0.0952. The difference comes from the kind of layer: stacking a layer that does not block is free. ## The Wall of Marking The class distribution says plainly where the defense ends. Neutralization happens in only two classes: **0.9167** of the `indirect_instruction` class and **0.8696** of the `source_spoofing` class. In the other three classes the number is **0.0000**. This is not a shortcoming but the consequence of the definition: marking closes only the "read as an instruction" path, and a hidden field request, privilege escalation or corrupted input does not pass through that path. Of the 51 remaining records, 46 come from these three classes. The remaining **5** records are in the first two classes but their source is **the user** — 2 of the indirect instruction, 3 of the source spoofing. Marking cannot touch them by its structure, because counting the user's own text as data leaves the agent useless. In the same set there are 37 harmful records from the user; this is the area this layer cannot cover, and it is the limit that holds the coverage at 0.4516. The cost column being zero does not mean the defense costs nothing. **47** of the 103 marked records are clean, and these records too are counted as data. Among them there is **one** record that carries the instruction mode: the only clean record that comes from a retrieved document and really asks for an operation. Marking closes its path as well. The capability lost is $1/147 = 0.0068$, that is, a whole step of the resolution of the set — the smallest measurable cost. Not writing this cost would show the defense as free; writing it is the only way to show how small it is. ## Summary - Untrusted content marking stamps text coming from a source other than the user as **data**; even if it carries the instruction mode it is not counted as an instruction, and no record is stopped. - Marking neutralizes 42 of the 93 harmful records (covered share 0.4516), 51 harmful remain, and the blocked records and the false alarm are **zero**. - Neutralization gathers in two classes: `indirect_instruction` 0.9167, `source_spoofing` 0.8696; in the other three classes it is 0.0000, because their harm does not rest on being read as an instruction. - Marking allows the threshold of the filter to be tightened: marking with the 0.50 threshold of the model filter covers 0.7634 and blocks 8 clean records; the rule filter alone covers 0.7957 and blocks 23 clean records. - Putting marking on top of the rule filter raises the coverage to 0.8280 and does not change the false block rate at all; adding a layer that does not block is free. - The cost is not zero, it is the smallest: 47 of the 103 marked records are clean and the 1 of them that carries the instruction mode loses its path, that is 0.0068 — a whole step of the resolution. ## Next Step Marking enlarged the coverage for free, but its wall was clear: it touches none of the remaining 51 records, and the most expensive class stands among them. Records that carry a privilege escalation or a hidden field request lead to operations that cannot be taken back once they pass; a deleted record, a written amount or a piece of information gone outside cannot be corrected afterwards. For such an action the certainty of an automatic decision is not enough. The next lesson opens the gate to a human and counts two things: how many of the records sent to the human are really harmful, and how many harmful records the gate **fails to catch** as the number of questions asked grows.