Untrusted Content Marking

The previous lesson measured the permission gate and counted that the four narrowing steps blocked 49 clean records in total. Every defense in that lesson worked in the same way: it stopped a record. The filter, content moderation, masking and the permission gate each protected by not letting something through, and the cost of all four came from the same place — the clean record blocked by mistake.

Yet stopping is not the only way to neutralize a harmful record. The harm of text that comes from a retrieved document often comes not from its content but from that content being read as an instruction. If the text is counted as data, the same record becomes harmless without being blocked at all. This lesson measures that defense and asks a single question: how many of the 93 harmful records does a layer that stops no record cover, and what does it pay in return.

• DF49. The code is a simulator; no real model endpoint or document store is called. The record set, the classes and the features are fictional, the seed is 20260218.
• DF50. An attack is not a text but a feature profile. The six sentences the records carry are ordinary support requests and have no relation to the class; the class lives only in the structural flags. There is no working payload, template or instruction string in the code.
• DF51. The baseline is the undefended setup: the harmful stopped is 0, the false alarm is 0.
• DF52. The rule of the course: a defense that does not write the share it stopped and the share it stopped by mistake together counts as unmeasured.
• DF53. Marking is not a block. Text coming from a source other than the user is stamped as data; even if it carries the instruction mode, it is not counted as an instruction. No record is stopped, no record is deleted, no field is removed.
• DF54. The neutralized record is a record that is in a class whose harm rests on being read as an instruction and that comes from a source other than the user. Both conditions are required together.
• DF55. The covered harmful is the union of what marking neutralizes and what the filter blocks; the same record is not counted twice.
• DF56. The cost of marking is not a false alarm, because the blocked records are zero. The cost is the instruction path that a clean record counted as data loses, and it is counted separately.
• DF57. Resolution: the step of recall is $1/93 = 0.0108$, the step of the false block rate is $1/147 = 0.0068$. There is no draw in the decision of marking and the measure is independent of the run.

Shared Definition

The record set, the classes, the defense layers and the measure are the same throughout the course. The set holds 240 fictional support records; the source field says where a record comes from, and marking looks only at this field.

SEED, M32 = 20260218, 0xFFFFFFFF


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def pick(u, w):
    t = 0.0
    for i, x in enumerate(w):
        t += x
        if u < t:
            return i
    return len(w) - 1


def trace_id(m):
    h = 2166136261
    for c in str(m):
        h = ((h ^ ord(c)) * 16777619) & M32
    return h


# ---- FICTIONAL records: no attack TEXT is written, an attack is a FEATURE PROFILE
CLASSES = ["clean", "indirect_instruction", "hidden_field_request", "privilege_escalation",
           "source_spoofing", "corrupted_input"]
SOURCES = ["user", "retrieved_document", "tool_output"]
# Features are structural; none of them is a working payload.
FEATURES = ["instruction_mode", "role_change_request", "hidden_field_name", "side_effect_tool_name",
            "source_stamp_mismatch", "unusual_stamp_distribution", "excessive_length"]
PROFILE = {
    "clean":                [0.06, 0.02, 0.05, 0.03, 0.02, 0.04, 0.05],
    "indirect_instruction": [0.88, 0.42, 0.10, 0.30, 0.35, 0.08, 0.12],
    "hidden_field_request": [0.35, 0.15, 0.86, 0.12, 0.10, 0.06, 0.08],
    "privilege_escalation": [0.30, 0.25, 0.12, 0.84, 0.14, 0.05, 0.07],
    "source_spoofing":      [0.22, 0.55, 0.08, 0.10, 0.82, 0.07, 0.06],
    "corrupted_input":      [0.05, 0.03, 0.04, 0.03, 0.06, 0.80, 0.62],
}
SOURCE_SHARE = {
    "clean":                [0.72, 0.20, 0.08],
    "indirect_instruction": [0.05, 0.70, 0.25],
    "hidden_field_request": [0.55, 0.30, 0.15],
    "privilege_escalation": [0.40, 0.35, 0.25],
    "source_spoofing":      [0.10, 0.60, 0.30],
    "corrupted_input":      [0.65, 0.20, 0.15],
}
# Fictional, entirely harmless support texts. They have NO relation to the class;
# the class information lives only in the feature profile.
TEXT = ["bu ayki fatura tutari gecen aya gore yuksek geldi",
        "sayac endeksi yanlis girilmis olabilir",
        "mahallede kesinti ne zaman bitecek",
        "tarife basamagimi ogrenmek istiyorum",
        "ariza kaydim icin ekip ne zaman gelir",
        "adres degisikligi nasil yapilir"]

RECORDS = []
for i in range(240):
    r = rng(SEED + 6151 * i)
    c = CLASSES[pick(r(), [0.62, 0.10, 0.07, 0.07, 0.07, 0.07])]
    source = SOURCES[pick(r(), SOURCE_SHARE[c])]
    feat = {o: (r() < p) for o, p in zip(FEATURES, PROFILE[c])}
    RECORDS.append({"no": i + 1, "class": c, "source": source, "features": feat,
                    "text": TEXT[int(r() * len(TEXT))],
                    "length": 40 + int(r() * 260)})


def harmful(k):
    return k["class"] != "clean"


# ---- DEFENSE 1: rule filter (structural, explainable)
def rule_filter(k, rules=("instruction_mode", "hidden_field_name", "side_effect_tool_name",
                          "source_stamp_mismatch")):
    return any(k["features"][o] for o in rules)


# ---- DEFENSE 2: model-based filter (score + threshold)
WEIGHT = {"instruction_mode": 0.42, "role_change_request": 0.26, "hidden_field_name": 0.44,
          "side_effect_tool_name": 0.40, "source_stamp_mismatch": 0.38,
          "unusual_stamp_distribution": 0.30, "excessive_length": 0.12}
SOURCE_BONUS = {"user": 0.00, "retrieved_document": 0.18, "tool_output": 0.12}


def score(k):
    p = sum(WEIGHT[o] for o in FEATURES if k["features"][o]) + SOURCE_BONUS[k["source"]]
    r = rng(SEED ^ trace_id(k["no"]))
    return min(max(p + (r() - 0.5) * 0.20, 0.0), 1.6)


def model_filter(k, threshold=0.40):
    return score(k) >= threshold


# ---- DEFENSE 3: untrusted content marking (not counted as an instruction)
def mark(k):
    """Text that comes from a source other than the user is marked as DATA;
    even if it carries the instruction mode, it is not counted as an instruction."""
    return {"as_data": k["source"] != "user"}


def marking_effect(records):
    """Marking closes only the 'read as an instruction' path."""
    neutralized, remaining = 0, 0
    for k in records:
        if not harmful(k):
            continue
        instruction_based = k["class"] in ("indirect_instruction", "source_spoofing")
        if instruction_based and mark(k)["as_data"]:
            neutralized += 1
        else:
            remaining += 1
    return neutralized, remaining


# ---- MEASURE: what was blocked and what was blocked by mistake are written together
def measure(records, decision):
    tp = fp = tn = fn = 0
    for k in records:
        flagged, actual = decision(k), harmful(k)
        if flagged and actual:
            tp += 1
        elif flagged and not actual:
            fp += 1
        elif not flagged and actual:
            fn += 1
        else:
            tn += 1
    precision = tp / (tp + fp) if tp + fp else 0.0
    recall = tp / (tp + fn) if tp + fn else 0.0
    return {"tp": tp, "fp": fp, "fn": fn, "tn": tn,
            "precision": round(precision, 4), "recall": round(recall, 4),
            "false_block_rate": round(fp / (fp + tn), 4) if fp + tn else 0.0}


print("records:", len(RECORDS), "- harmful:", sum(harmful(k) for k in RECORDS))
print("source:", {s: sum(k["source"] == s for k in RECORDS) for s in SOURCES})

records: 240 - harmful: 93
source: {'user': 137, 'retrieved_document': 72, 'tool_output': 31}

The Data-Instruction Distinction

Marking breaks a single assumption: that all the text entering the model is read with the same weight. The user’s own request is an instruction, because the user is the will the agent serves. Text that comes from a retrieved document or a tool output is data — it is read, quoted, summarized, but it does not change behavior. The mark function puts this stamp by looking only at the source field and stops no record.

Whom the distinction affects depends on the class. If the harm of a record rests on being read as an instruction, the stamp removes that harm. If the harm rests on something else, the stamp does nothing: asking for a hidden field name is a request even when it is read as data.

def neutralized(k):
    """A record whose harm rests on BEING READ AS AN INSTRUCTION and that comes from a source
    other than the user stays neutral without anything being blocked: the text is counted
    as data, not as an instruction."""
    return (harmful(k) and k["class"] in ("indirect_instruction", "source_spoofing")
            and mark(k)["as_data"])


neutralized_count, remaining = marking_effect(RECORDS)
marked = [k for k in RECORDS if mark(k)["as_data"]]
clean_marked = [k for k in marked if not harmful(k)]
print("baseline    : harmful stopped 0 , false alarm 0")
print("marking     : neutralized", neutralized_count, ", remaining harmful", remaining,
      ", covered share", round(neutralized_count / 93, 4))
print("marked records", len(marked), "- clean among them", len(clean_marked),
      "- carrying instruction mode", sum(k["features"]["instruction_mode"] for k in clean_marked))
print("blocked records 0 - false alarm 0")

print("\nper-class share neutralized by marking")
for s in CLASSES[1:]:
    g = [k for k in RECORDS if k["class"] == s]
    e = sum(neutralized(k) for k in g)
    print(f"  {s:<20} {e:>2}/{len(g):<3} {e / len(g):.4f}")


def setup(decision=None, marking=True):
    """Covered harmful = neutralized by marking OR blocked by the filter."""
    s = measure(RECORDS, decision) if decision else {"tp": 0, "fp": 0, "false_block_rate": 0.0}
    covered = sum(1 for k in RECORDS if harmful(k)
                  and ((marking and neutralized(k)) or (decision is not None and decision(k))))
    return covered, s["tp"] + s["fp"], s["fp"], s["false_block_rate"]


print("\nsetup                    covered   share     blocked   false alarm false-block")
for name, decision, mk in [("baseline", None, False),
                           ("marking only", None, True),
                           ("rule filter only", rule_filter, False),
                           ("marking + rule", rule_filter, True),
                           ("marking + model 0.50", lambda k: model_filter(k, 0.50), True),
                           ("marking + model 0.60", lambda k: model_filter(k, 0.60), True),
                           ("marking + model 0.80", lambda k: model_filter(k, 0.80), True)]:
    covered, blocked, fa, fbr = setup(decision, mk)
    print(f"{name:<24} {covered:>7} {covered / 93:>7.4f} {blocked:>11} {fa:>13} {fbr:>11.4f}")

baseline    : harmful stopped 0 , false alarm 0
marking     : neutralized 42 , remaining harmful 51 , covered share 0.4516
marked records 103 - clean among them 47 - carrying instruction mode 1
blocked records 0 - false alarm 0

per-class share neutralized by marking
  indirect_instruction 22/24  0.9167
  hidden_field_request  0/17  0.0000
  privilege_escalation  0/10  0.0000
  source_spoofing      20/23  0.8696
  corrupted_input       0/19  0.0000

setup                    covered   share     blocked   false alarm false-block
baseline                       0  0.0000           0             0      0.0000
marking only                  42  0.4516           0             0      0.0000
rule filter only              74  0.7957          97            23      0.1565
marking + rule                77  0.8280          97            23      0.1565
marking + model 0.50          71  0.7634          74             8      0.0544
marking + model 0.60          67  0.7204          64             5      0.0340
marking + model 0.80          60  0.6452          47             1      0.0068

The Number of a Defense That Does Not Block

The second row is the most striking row of this course. Marking neutralizes 42 of the 93 harmful records — the covered share is 0.4516 — and it does so without blocking any record. The blocked records column is 0, the false alarm column is 0, the false block rate is 0.0000. The rule of the course asks for the two numbers of a defense together; the second number of this defense is exactly zero, and this is not a rounding but the count itself.

For comparison the third row has to be read. The rule filter covers more — 0.7957 — but it blocks 97 records for this, and 23 of those blocks are on clean records. The two defenses are not of the same kind: one buys coverage with blocking, the other buys it free.

The main finding is in the last three rows. Marking allows the threshold of the filter behind it to be tightened. Because most of the instruction-based harmful is already neutralized, the filter does not need to catch them; it can focus only on the remaining 51 records. Marking together with the 0.50 threshold of the model filter covers 0.7634 and blocks only 8 clean records. The difference from the 0.7957 of the rule filter alone is three records, that is $3 \times 0.0108 = 0.0323$; in return the false block rate falls from 0.1565 to 0.0544, that is, fifteen fewer clean records are blocked, and this is $15 \times 0.0068 = 0.1021$. A coverage difference of three records is given in exchange for a blocking difference of fifteen records. In the tightest row, at the 0.80 threshold, the blocked clean records fall to one and the coverage still stays at 0.6452.

The fourth row shows one more thing: putting marking on top of the rule filter raises the coverage from 0.7957 to 0.8280 and does not change the false block rate at all. In the previous lesson, putting the permission gate on top of the filter raised recall by 0.0430 while it raised the false block rate by 0.0952. The difference comes from the kind of layer: stacking a layer that does not block is free.

The Wall of Marking

The class distribution says plainly where the defense ends. Neutralization happens in only two classes: 0.9167 of the indirect_instruction class and 0.8696 of the source_spoofing class. In the other three classes the number is 0.0000. This is not a shortcoming but the consequence of the definition: marking closes only the “read as an instruction” path, and a hidden field request, privilege escalation or corrupted input does not pass through that path.

Of the 51 remaining records, 46 come from these three classes. The remaining 5 records are in the first two classes but their source is the user — 2 of the indirect instruction, 3 of the source spoofing. Marking cannot touch them by its structure, because counting the user’s own text as data leaves the agent useless. In the same set there are 37 harmful records from the user; this is the area this layer cannot cover, and it is the limit that holds the coverage at 0.4516.

The cost column being zero does not mean the defense costs nothing. 47 of the 103 marked records are clean, and these records too are counted as data. Among them there is one record that carries the instruction mode: the only clean record that comes from a retrieved document and really asks for an operation. Marking closes its path as well. The capability lost is $1/147 = 0.0068$, that is, a whole step of the resolution of the set — the smallest measurable cost. Not writing this cost would show the defense as free; writing it is the only way to show how small it is.

Summary

• Untrusted content marking stamps text coming from a source other than the user as data; even if it carries the instruction mode it is not counted as an instruction, and no record is stopped.
• Marking neutralizes 42 of the 93 harmful records (covered share 0.4516), 51 harmful remain, and the blocked records and the false alarm are zero.
• Neutralization gathers in two classes: indirect_instruction 0.9167, source_spoofing 0.8696; in the other three classes it is 0.0000, because their harm does not rest on being read as an instruction.
• Marking allows the threshold of the filter to be tightened: marking with the 0.50 threshold of the model filter covers 0.7634 and blocks 8 clean records; the rule filter alone covers 0.7957 and blocks 23 clean records.
• Putting marking on top of the rule filter raises the coverage to 0.8280 and does not change the false block rate at all; adding a layer that does not block is free.
• The cost is not zero, it is the smallest: 47 of the 103 marked records are clean and the 1 of them that carries the instruction mode loses its path, that is 0.0068 — a whole step of the resolution.

Next Step

Marking enlarged the coverage for free, but its wall was clear: it touches none of the remaining 51 records, and the most expensive class stands among them. Records that carry a privilege escalation or a hidden field request lead to operations that cannot be taken back once they pass; a deleted record, a written amount or a piece of information gone outside cannot be corrected afterwards. For such an action the certainty of an automatic decision is not enough. The next lesson opens the gate to a human and counts two things: how many of the records sent to the human are really harmful, and how many harmful records the gate fails to catch as the number of questions asked grows.