Adversarial Examples

Up to here five classes have been counted: the indirect injection coming from retrieved content, the hidden field request, privilege escalation, source spoofing, and the field leaking on the output side. All five silently assumed the same thing: the input says something. The instruction mode is the mood of a sentence, a hidden field name is a name, a source stamp is a claim. The defense was built on this assumption too — the rule filter reads structural markers tied to the meaning of the text.

An input can be a surface without saying anything. A deliberately corrupted input targets the way the defense reads: it departs from the order the reading side is used to, and this departure has nothing to do with meaning. This lesson adds that class by name and asks a single question: how much of it does the same defense catch.

• AS81. The record set, the classes, the attributes, and the defense are fictional; no real model or filter is called. Seed 20260218, second run 20260219.
• AS82. An attack is not a text but an attribute profile. The text of the records is entirely harmless support requests and has no relation to the class; the class lives only in the attribute profile.
• AS83. No corruption procedure is written. The corrupted input class is represented by two flags: unusual_stamp_distribution and excessive_length. A flag is a marker, not content.
• AS84. All of the code that runs is on the defense side: classification, filter, count, and measure.
• AS85. The adversarial example itself was treated in the Deep Learning course; the architecture and the robustness procedure are not repeated. The only thing measured here is the share the defense catches in this class.
• AS86. The undefended baseline is the same in every lesson: no filter, blocked harmful is 0, false alarm is 0.
• AS87. The resolution of the set is the lower limit of the measurement band: the step of recall is 1/93 = 0.0108, the step of the false block rate is 1/147 = 0.0068. Within a class the step is larger and is written separately.

The Meaningful Text Assumption

The first block builds the common definition of the course. The record set is 240 fictional support requests; every record has a class, a source, and seven structural attributes. None of these attributes is a working payload — each is a flag, and the text a record carries is one of six harmless support sentences, independent of its class. The profile of the corrupted input class is structurally separate from the others: its instruction_mode share is 0.05 and its hidden_field_name share is 0.04 — that is, almost none of the markers the rule filter reads is present. In contrast, its unusual_stamp_distribution share is 0.80 and its excessive_length share is 0.62.

SEED, M32 = 20260218, 0xFFFFFFFF


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def pick(u, w):
    t = 0.0
    for i, x in enumerate(w):
        t += x
        if u < t:
            return i
    return len(w) - 1


def trace_id(m):
    h = 2166136261
    for c in str(m):
        h = ((h ^ ord(c)) * 16777619) & M32
    return h


# ---- FICTIONAL records: no attack TEXT is written, an attack is a FEATURE PROFILE
CLASSES = ["clean", "indirect_instruction", "hidden_field_request", "privilege_escalation",
           "source_spoofing", "corrupted_input"]
SOURCES = ["user", "retrieved_document", "tool_output"]
# Features are structural; none of them is a working payload.
FEATURES = ["instruction_mode", "role_change_request", "hidden_field_name", "side_effect_tool_name",
            "source_stamp_mismatch", "unusual_stamp_distribution", "excessive_length"]
# Per-class feature probability (fictional). Clean records also carry low shares:
# in the real world, well-intentioned text can carry these features too.
PROFILE = {
    "clean":                [0.06, 0.02, 0.05, 0.03, 0.02, 0.04, 0.05],
    "indirect_instruction": [0.88, 0.42, 0.10, 0.30, 0.35, 0.08, 0.12],
    "hidden_field_request": [0.35, 0.15, 0.86, 0.12, 0.10, 0.06, 0.08],
    "privilege_escalation": [0.30, 0.25, 0.12, 0.84, 0.14, 0.05, 0.07],
    "source_spoofing":      [0.22, 0.55, 0.08, 0.10, 0.82, 0.07, 0.06],
    "corrupted_input":      [0.05, 0.03, 0.04, 0.03, 0.06, 0.80, 0.62],
}
SOURCE_SHARE = {
    "clean":                [0.72, 0.20, 0.08],
    "indirect_instruction": [0.05, 0.70, 0.25],
    "hidden_field_request": [0.55, 0.30, 0.15],
    "privilege_escalation": [0.40, 0.35, 0.25],
    "source_spoofing":      [0.10, 0.60, 0.30],
    "corrupted_input":      [0.65, 0.20, 0.15],
}
# Fictional, entirely harmless support texts. They have NO relation to the class;
# the class information lives only in the feature profile.
TEXT = ["bu ayki fatura tutari gecen aya gore yuksek geldi",
        "sayac endeksi yanlis girilmis olabilir",
        "mahallede kesinti ne zaman bitecek",
        "tarife basamagimi ogrenmek istiyorum",
        "ariza kaydim icin ekip ne zaman gelir",
        "adres degisikligi nasil yapilir"]

RECORDS = []
for i in range(240):
    r = rng(SEED + 6151 * i)
    c = CLASSES[pick(r(), [0.62, 0.10, 0.07, 0.07, 0.07, 0.07])]
    source = SOURCES[pick(r(), SOURCE_SHARE[c])]
    feat = {o: (r() < p) for o, p in zip(FEATURES, PROFILE[c])}
    RECORDS.append({"no": i + 1, "class": c, "source": source, "features": feat,
                    "text": TEXT[int(r() * len(TEXT))],
                    "length": 40 + int(r() * 260)})


def harmful(k):
    return k["class"] != "clean"


# ---- DEFENSE 1: rule filter (structural, explainable)
def rule_filter(k, rules=("instruction_mode", "hidden_field_name", "side_effect_tool_name",
                          "source_stamp_mismatch")):
    return any(k["features"][o] for o in rules)


# ---- DEFENSE 2: model-based filter (score + threshold)
WEIGHT = {"instruction_mode": 0.42, "role_change_request": 0.26, "hidden_field_name": 0.44,
          "side_effect_tool_name": 0.40, "source_stamp_mismatch": 0.38,
          "unusual_stamp_distribution": 0.30, "excessive_length": 0.12}
SOURCE_BONUS = {"user": 0.00, "retrieved_document": 0.18, "tool_output": 0.12}


def score(k):
    p = sum(WEIGHT[o] for o in FEATURES if k["features"][o]) + SOURCE_BONUS[k["source"]]
    r = rng(SEED ^ trace_id(k["no"]))
    return min(max(p + (r() - 0.5) * 0.20, 0.0), 1.6)


def model_filter(k, threshold=0.40):
    return score(k) >= threshold


# ---- MEASURE: what was blocked and what was blocked by mistake are written together
def measure(records, decision):
    tp = fp = tn = fn = 0
    for k in records:
        flagged, actual = decision(k), harmful(k)
        if flagged and actual:
            tp += 1
        elif flagged and not actual:
            fp += 1
        elif not flagged and actual:
            fn += 1
        else:
            tn += 1
    precision = tp / (tp + fp) if tp + fp else 0.0
    recall = tp / (tp + fn) if tp + fn else 0.0
    return {"tp": tp, "fp": fp, "fn": fn, "tn": tn,
            "precision": round(precision, 4), "recall": round(recall, 4),
            "false_block_rate": round(fp / (fp + tn), 4) if fp + tn else 0.0}

Recall per Class

At the set level the recall of the rule filter is 0.7957, and this single number is an average. Under the average stand five separate classes; the second block breaks the same filter down class by class and writes next to each row the class’s own resolution.

HARMFUL_CLASSES = CLASSES[1:]
print("class                records     caught      recall  resolution")
for c in HARMFUL_CLASSES:
    group = [k for k in RECORDS if k["class"] == c]
    caught = sum(1 for k in group if rule_filter(k))
    print(f"{c:<20} {len(group):>7} {caught:>10} {caught / len(group):>11.4f} {1 / len(group):>11.4f}")

carriers = [k for k in RECORDS if k["features"]["unusual_stamp_distribution"]]
print("\nrecords carrying unusual_stamp_distribution:", len(carriers),
      " of them harmful:", sum(1 for k in carriers if harmful(k)),
      " of them corrupted_input:", sum(1 for k in carriers if k["class"] == "corrupted_input"))
print("records the rule filter caught in the corrupted_input class:")
for k in RECORDS:
    if k["class"] == "corrupted_input" and rule_filter(k):
        print("  record", k["no"], "-", ", ".join(o for o in FEATURES if k["features"][o]))

class                records     caught      recall  resolution
indirect_instruction      24         24      1.0000      0.0417
hidden_field_request      17         16      0.9412      0.0588
privilege_escalation      10          9      0.9000      0.1000
source_spoofing           23         20      0.8696      0.0435
corrupted_input           19          5      0.2632      0.0526

records carrying unusual_stamp_distribution: 29  of them harmful: 21  of them corrupted_input: 15
records the rule filter caught in the corrupted_input class:
  record 27 - hidden_field_name, unusual_stamp_distribution, excessive_length
  record 100 - instruction_mode, unusual_stamp_distribution, excessive_length
  record 212 - source_stamp_mismatch, unusual_stamp_distribution, excessive_length
  record 227 - side_effect_tool_name, unusual_stamp_distribution, excessive_length
  record 236 - side_effect_tool_name

Four classes stand in a narrow band: 1.0000, 0.9412, 0.9000, and 0.8696. The fifth is 0.2632. The distance is more than six times the coarsest class step and cannot be explained by the resolution of any class — the coarsest step is the 0.1000 of the privilege escalation class, and the observed difference is 0.6064.

What the last list says is heavier than the ratio. Four of the five records the rule filter catches in this class are caught because they carry hidden_field_name, instruction_mode, source_stamp_mismatch, and side_effect_tool_name; these are markers of other classes and happen to be present in these records. The fifth record carries neither: it has no unusual token distribution and no excessive length. So in this class the rule filter catches nothing specific to the class; the whole of 0.2632 comes from side information. A recall measured in a class does not mean that the defense sees that class.

The count line confirms this: there are 29 records carrying unusual_stamp_distribution in the set, 21 are harmful, 15 are corrupted input. The flag sits in the set and is not read.

Adding the Flag to the Rule

Since the flag is already in the record, the fix is seemingly a single line: it is added to the rule list. The third block does this and, by the rule of the course, prints two numbers together. The same block runs the model-based filter in two runs; in the second run the only thing that changes is the seed.

WIDE = ("instruction_mode", "hidden_field_name", "side_effect_tool_name",
        "source_stamp_mismatch", "unusual_stamp_distribution")
SEED2 = 20260219


def rule_wide(k):
    return rule_filter(k, WIDE)


def model_second(k, threshold=0.40):
    """The same model filter, second run: only the seed changes."""
    p = sum(WEIGHT[o] for o in FEATURES if k["features"][o]) + SOURCE_BONUS[k["source"]]
    r = rng(SEED2 ^ trace_id(k["no"]))
    return min(max(p + (r() - 0.5) * 0.20, 0.0), 1.6) >= threshold


SETUPS = (("undefended baseline", lambda k: False), ("rule", rule_filter),
          ("rule + flag", rule_wide), ("model threshold 0.40", model_filter),
          ("model second run", model_second))
print("setup                   blocked   false alarm escaped precision      recall false block rate")
for name, decision in SETUPS:
    m = measure(RECORDS, decision)
    print(f"{name:<21} {m['tp']:>10} {m['fp']:>13} {m['fn']:>7} {m['precision']:>9.4f}"
          f" {m['recall']:>11.4f} {m['false_block_rate']:>17.4f}")

group = [k for k in RECORDS if k["class"] == "corrupted_input"]
print("\nrecall in the corrupted_input class (resolution 1/19 = %.4f)" % (1 / len(group)))
for name, decision in SETUPS[1:]:
    caught = sum(1 for k in group if decision(k))
    print(f"  {name:<21} {caught:>2}/{len(group)} = {caught / len(group):.4f}")
scores = sorted(score(k) for k in group)
print("score band: %.4f - %.4f  between 0.35 and 0.50: %d records"
      % (scores[0], scores[-1], sum(1 for x in scores if 0.35 <= x < 0.50)))

setup                   blocked   false alarm escaped precision      recall false block rate
undefended baseline            0             0      93    0.0000      0.0000            0.0000
rule                          74            23      19    0.7629      0.7957            0.1565
rule + flag                   85            30       8    0.7391      0.9140            0.2041
model threshold 0.40          77            20      16    0.7938      0.8280            0.1361
model second run              77            19      16    0.8021      0.8280            0.1293

recall in the corrupted_input class (resolution 1/19 = 0.0526)
  rule                   5/19 = 0.2632
  rule + flag           16/19 = 0.8421
  model threshold 0.40   9/19 = 0.4737
  model second run       8/19 = 0.4211
score band: 0.0425 - 1.0132  between 0.35 and 0.50: 9 records

Adding the flag raises class recall from 0.2632 to 0.8421: caught records go from 5 to 16. At the set level recall rises from 0.7957 to 0.9140 and escaped harmful falls from 19 to 8. Not a single row of the other four classes changes — the whole gain lies in one class, and this is the proof that the flag really sees that class.

The cost stands in the same table. False alarm rises from 23 to 30, precision falls from 0.7629 to 0.7391, and the false block rate moves from 0.1565 to 0.2041. Seven extra clean records are blocked, because this flag is present in the profile of the clean class with a share of 0.04: a well-intentioned support request can also carry an unusual token distribution. Eleven harmful records were gained and seven clean ones lost. A defense that does not write these two numbers together counts as unmeasured; the sentence “recall 0.9140” alone hides what the setup cost. Whether the trade is worth it is not a number the set can give: whether eleven harmful records are worth seven clean ones depends on whom the blocked record belongs to and what happens once it is blocked. The set says only the ratio, not the decision.

The number of the model-based filter in the same class says something else. 0.4737 is twice the rule filter and half of the widened rule. The reason is in the score band: the weights of the two flags of this class are 0.30 and 0.12, their total is 0.42, and the threshold is 0.40. Nine records stand between 0.35 and 0.50, within one step of the threshold. When the whole weight of a class falls slightly above the threshold, the filter does not separate that class, it flips a coin.

Two Runs and the Resolution of the Class

In the second run only the seed changes, and set-level recall stays the same: 0.8280 in both runs. False alarm falls from 20 to 19, so precision rises from 0.7938 to 0.8021. This difference is the size of one record, and since the step of the false block rate is 0.0068 it stands barely above the measurable limit; yet it is not an improvement of the defense, it is the noise of the same filter.

Within the class the situation is sharper. In the corrupted input class the model-based filter gives 0.4737 in the first run and 0.4211 in the second. The difference is exactly one record and the resolution of the class is 1/19 = 0.0526. So the observed difference is equal to the smallest step the set can measure in this class, and it cannot be counted as a finding. The most honest sentence that can be said about the number of the same defense in this class is this: it is somewhere between 0.42 and 0.47, and the set cannot separate it more finely. The rule filter and the widened rule never meet this problem — both are noiseless, they give the same number from run to run, and this is also what makes them run-independent measures.

Summary

• Corrupted input is a class that forms a surface even without carrying meaning; in the lesson, the unusual_stamp_distribution and excessive_length flags are used, not a working corruption procedure.
• The recall of the rule filter is between 1.0000 and 0.8696 in four classes and drops to 0.2632 in the corrupted input class; four of the five records it catches there are caught because they carry the marker of another class, and one carries no corruption flag.
• Adding the flag to the rule raises class recall to 0.8421 and set recall to 0.9140; the same step moves false alarm from 23 to 30, the false block rate from 0.1565 to 0.2041, and drops precision from 0.7629 to 0.7391.
• The model-based filter stays at 0.4737 in this class, because the total weight of the class, 0.42, stands slightly above the threshold of 0.40 and nine records are squeezed between 0.35 and 0.50.
• The within-class difference between two runs is 0.4737 and 0.4211; since the class resolution is 0.0526, this difference is not a finding. The two rule-based setups are noiseless and give a run-independent measure.

Next Step

This lesson added a class by name and showed that the defense is almost blind in it. What closed the blind spot was not a discovery: the flag was already in the record, it was not in the rule list. A disturbing question follows from this. This class was counted only because someone thought of looking. How many classes were not counted because nobody looked, and what is the defense’s share in them? The next lesson treats this question not as a procedure but as a count: how many classes a trial campaign covers, what share passes the defense in each class, and with what number an uncovered class appears in the report.