Prompt Injection

The eleven lessons of the previous course measured whose hands the tool is in: the definition moved to the other side, the cost of the envelope was counted, and a definition reported by an untrusted server was checked item by item. All of that was outside the interface — the shell of the message was counted, not the text inside it. For a language model, however, data and instruction have no structural boundary: both enter the same context window in the same form. A user’s sentence, the body of a retrieved document, and a value returned by a tool are of the same kind in the model’s eyes. Prompt injection names this lack of distinction: the ability of text entering a model to make it do more than the authority of its path allows.

This lesson does not describe the surface; it counts it. In one record set, how many harmful records are blocked with no defense, how many with a rule-based filter, and how many clean records does that filter wrongly block? The rule of the course holds from the first line: a defense that does not write the share it blocks together with the share it blocks by mistake counts as unmeasured.

• AS1. The record set, classes, attributes, and defense layers are fictional; no real model endpoint or incident log is used. Seed 20260218.
• AS2. This is not an attack lesson. All code that runs is on the defense side: classification, filter, count, and measure. No working payload, template, instruction sequence, or bypass routine is written.
• AS3. The rule is structural: an attack is not a text but an attribute profile. The record text is entirely harmless support requests with no relation to the class.
• AS4. Class names are labels, not procedure recipes. When a new class is needed, a flag is added to the attribute list, not a text.
• AS5. Input validation was measured in the Secure Coding course and threat modeling in the Secure Development Lifecycle course; their procedures are not repeated. Only the surface specific to the language model is added.
• AS6. precision and recall are used with the classification meaning they have in the Machine Learning and Data Science curriculum. A false alarm is a clean record counted as harmful; a false negative is a harmful record that escapes.
• AS7. The rule-based filter has no sampling and gives the same number on the same set in every run; the second run regenerates the set with a second seed.
• AS8. The smallest step of recall is 1/93 and the step of the false block rate is 1/147; no difference smaller than these steps can be measured.

An Attack Is a Feature Profile, Not a Text

The usual way to teach an attack surface is to show an example. This course does not, and not as a matter of style but as a structural constraint: the record set has no field called attack text. What a record carries is seven flags — instruction mode, role change request, hidden field name, side-effect tool name, source stamp mismatch, unusual token distribution, excessive length. Each is a true-or-false value about the structure of a record; none runs.

The record also has a text field, and one of six fictional support requests sits there. Those six sentences are ordinary subscriber questions and have no relation to the class: which sentence falls on which record is decided by a separate draw. Because the text does not determine the class, the lesson never shows an attack text and the measure rests on the profile. In a real setup the filter reads the text; here what is measured is a profile that behaves like the output of that reading.

# FICTIONAL. The record set, classes, attributes and defense layers are defined inside
# the lesson; no real model endpoint is called. No attack TEXT is written.
SEED, M32 = 20260218, 0xFFFFFFFF


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def pick(u, w):
    t = 0.0
    for i, x in enumerate(w):
        t += x
        if u < t:
            return i
    return len(w) - 1


def trace_id(m):
    h = 2166136261
    for c in str(m):
        h = ((h ^ ord(c)) * 16777619) & M32
    return h


# ---- FICTIONAL records: no attack TEXT is written, an attack is a FEATURE PROFILE
CLASSES = ["clean", "indirect_instruction", "hidden_field_request", "privilege_escalation",
           "source_spoofing", "corrupted_input"]
SOURCES = ["user", "retrieved_document", "tool_output"]
# Features are structural; none of them is a working payload.
FEATURES = ["instruction_mode", "role_change_request", "hidden_field_name", "side_effect_tool_name",
            "source_stamp_mismatch", "unusual_stamp_distribution", "excessive_length"]
# Feature probability per class (fictional). Clean records carry small shares too:
# in the real world, well-meant text can carry these features as well.
PROFILE = {
    "clean":                [0.06, 0.02, 0.05, 0.03, 0.02, 0.04, 0.05],
    "indirect_instruction": [0.88, 0.42, 0.10, 0.30, 0.35, 0.08, 0.12],
    "hidden_field_request": [0.35, 0.15, 0.86, 0.12, 0.10, 0.06, 0.08],
    "privilege_escalation": [0.30, 0.25, 0.12, 0.84, 0.14, 0.05, 0.07],
    "source_spoofing":      [0.22, 0.55, 0.08, 0.10, 0.82, 0.07, 0.06],
    "corrupted_input":      [0.05, 0.03, 0.04, 0.03, 0.06, 0.80, 0.62],
}
SOURCE_SHARE = {
    "clean":                [0.72, 0.20, 0.08],
    "indirect_instruction": [0.05, 0.70, 0.25],
    "hidden_field_request": [0.55, 0.30, 0.15],
    "privilege_escalation": [0.40, 0.35, 0.25],
    "source_spoofing":      [0.10, 0.60, 0.30],
    "corrupted_input":      [0.65, 0.20, 0.15],
}
# Fictional, entirely harmless support texts. They have NO relation to the class;
# the class information lives only in the feature profile.
TEXT = ["bu ayki fatura tutari gecen aya gore yuksek geldi",
        "sayac endeksi yanlis girilmis olabilir",
        "mahallede kesinti ne zaman bitecek",
        "tarife basamagimi ogrenmek istiyorum",
        "ariza kaydim icin ekip ne zaman gelir",
        "adres degisikligi nasil yapilir"]

RECORDS = []
for i in range(240):
    r = rng(SEED + 6151 * i)
    c = CLASSES[pick(r(), [0.62, 0.10, 0.07, 0.07, 0.07, 0.07])]
    source = SOURCES[pick(r(), SOURCE_SHARE[c])]
    feat = {o: (r() < p) for o, p in zip(FEATURES, PROFILE[c])}
    RECORDS.append({"no": i + 1, "class": c, "source": source, "features": feat,
                    "text": TEXT[int(r() * len(TEXT))],
                    "length": 40 + int(r() * 260)})


def harmful(k):
    return k["class"] != "clean"


# ---- DEFENSE 1: rule filter (structural, explainable)
def rule_filter(k, rules=("instruction_mode", "hidden_field_name", "side_effect_tool_name",
                          "source_stamp_mismatch")):
    return any(k["features"][o] for o in rules)


# ---- DEFENSE 2: model-based filter (score + threshold)
WEIGHT = {"instruction_mode": 0.42, "role_change_request": 0.26, "hidden_field_name": 0.44,
          "side_effect_tool_name": 0.40, "source_stamp_mismatch": 0.38,
          "unusual_stamp_distribution": 0.30, "excessive_length": 0.12}
SOURCE_BONUS = {"user": 0.00, "retrieved_document": 0.18, "tool_output": 0.12}


def score(k):
    p = sum(WEIGHT[o] for o in FEATURES if k["features"][o]) + SOURCE_BONUS[k["source"]]
    r = rng(SEED ^ trace_id(k["no"]))
    return min(max(p + (r() - 0.5) * 0.20, 0.0), 1.6)


def model_filter(k, threshold=0.40):
    return score(k) >= threshold


# ---- MEASURE: what was blocked and what was blocked by mistake are written together
def measure(records, decision):
    tp = fp = tn = fn = 0
    for k in records:
        flagged, actual = decision(k), harmful(k)
        if flagged and actual:
            tp += 1
        elif flagged and not actual:
            fp += 1
        elif not flagged and actual:
            fn += 1
        else:
            tn += 1
    precision = tp / (tp + fp) if tp + fp else 0.0
    recall = tp / (tp + fn) if tp + fn else 0.0
    return {"tp": tp, "fp": fp, "fn": fn, "tn": tn,
            "precision": round(precision, 4), "recall": round(recall, 4),
            "false_block_rate": round(fp / (fp + tn), 4) if fp + tn else 0.0}

The Undefended Baseline and the Rule-Based Filter

The second block prints the composition of the set, then puts two setups side by side. The first is the undefended baseline: no filter, marking, or gate, so nothing is blocked. The second is a rule-based filter that looks at four flags.

from collections import Counter

print("records:", len(RECORDS), " harmful:", sum(1 for k in RECORDS if harmful(k)),
      " clean:", sum(1 for k in RECORDS if not harmful(k)))
print("class:", {c: sum(1 for k in RECORDS if k["class"] == c) for c in CLASSES})
print("source:", {x: sum(1 for k in RECORDS if k["source"] == x) for x in SOURCES})
print("number of classes in which each text appears:",
      [len({k["class"] for k in RECORDS if k["text"] == t}) for t in TEXT])


def undefended(k):
    """Baseline: no filter, marking or gate."""
    return False


print("\nsetup              blocked   false alarm  escaped  precision      recall  false block")
for name, decision in (("undefended", undefended), ("rule filter", rule_filter)):
    m = measure(RECORDS, decision)
    print(f"{name:<15} {m['tp']:>10} {m['fp']:>13} {m['fn']:>8}"
          f" {m['precision']:>10.4f} {m['recall']:>11.4f} {m['false_block_rate']:>12.4f}")

direct = [k for k in RECORDS if harmful(k) and k["source"] == "user"]
indirect = [k for k in RECORDS if harmful(k) and k["source"] != "user"]
print("\nform       harmful     rule caught      recall")
for name, g in (("direct", direct), ("indirect", indirect)):
    caught = sum(1 for k in g if rule_filter(k))
    print(f"{name:<10} {len(g):>7} {caught:>15} {caught / len(g):>11.4f}")

print("\nsource distribution per class")
for c in CLASSES:
    cnt = Counter(k["source"] for k in RECORDS if k["class"] == c)
    print(f"  {c:<20} " + "  ".join(f"{x}={cnt.get(x, 0):>3}" for x in SOURCES))

records: 240  harmful: 93  clean: 147
class: {'clean': 147, 'indirect_instruction': 24, 'hidden_field_request': 17, 'privilege_escalation': 10, 'source_spoofing': 23, 'corrupted_input': 19}
source: {'user': 137, 'retrieved_document': 72, 'tool_output': 31}
number of classes in which each text appears: [6, 5, 6, 6, 5, 6]

setup              blocked   false alarm  escaped  precision      recall  false block
undefended               0             0       93     0.0000      0.0000       0.0000
rule filter             74            23       19     0.7629      0.7957       0.1565

form       harmful     rule caught      recall
direct          37              24      0.6486
indirect        56              50      0.8929

source distribution per class
  clean                user=100  retrieved_document= 33  tool_output= 14
  indirect_instruction user=  2  retrieved_document= 15  tool_output=  7
  hidden_field_request user= 13  retrieved_document=  4  tool_output=  0
  privilege_escalation user=  3  retrieved_document=  2  tool_output=  5
  source_spoofing      user=  3  retrieved_document= 16  tool_output=  4
  corrupted_input      user= 16  retrieved_document=  2  tool_output=  1

The fourth line confirms the design: four of the six texts appear in all six classes and two in five. The text does not carry the class.

The baseline row sets the measurement frame of the course. In the undefended setup, true positives are 0, false alarms are 0, and escaped harmful records are 93. Precision and recall are zero. The row must be there because the undefended setup has zero false alarms too, and if false alarms are counted alone, doing nothing looks perfect. The number of a defense gains meaning only when the two columns are read together.

The rule-based filter blocks 74 of the 93 harmful records: recall is 0.7957. Its cost is written in two columns: 23 clean records were wrongly blocked, 0.1565 of the 147 clean records. A precision of 0.7629 means about one in four flagged records is harmless. The 19 escaped harmful records also stay in the table; the measure of a defense is what it misses as much as what it catches.

Direct and Indirect Form

The two forms of prompt injection are separated not by text but by source. In the direct form the harmful content comes from a request the user wrote. The indirect injection comes from a place the user did not write: the body of a retrieved document or a value returned by a tool. The user is the victim here, not the source. The distinction matters for authority: in the indirect form, the path that brings the text into the system is one the user approved, and it generally runs with broader authority.

The source distribution turns this distinction into a number. 37 of the 93 harmful records come from the user and 56 from a retrieved document or a tool output — most of the surface is indirect. The class table divides this among the classes: of the 24 records of the indirect injection class only 2 come from the user, and of the 23 of the source spoofing class, 3. By contrast, the hidden field request class (13 of 17) and the corrupted input class (16 of 19) are predominantly direct.

The filter’s number differs between the forms: recall is 0.8929 on the indirect form and 0.6486 on the direct form. The difference is 0.2443, more than twenty times the resolution of the set, so it is measurable. The reason is in the rule list: one of the four rules is source_stamp_mismatch, a flag that is frequent in the indirect classes and rare in the direct ones. The filter is markedly weaker on one side of the surface than on the other, and a single average recall hides this: the overall rate swallows the difference between the subsets.

The Band of the Measure

The rule-based filter contains no sampling; on the same set it gives exactly the same five numbers in every run. So the second run changes not the filter but the set: the same definition is regenerated with a second seed.

def build_set(seed):
    """Same definition, another seed: the record set is regenerated."""
    ks = []
    for i in range(240):
        r = rng(seed + 6151 * i)
        c = CLASSES[pick(r(), [0.62, 0.10, 0.07, 0.07, 0.07, 0.07])]
        source = SOURCES[pick(r(), SOURCE_SHARE[c])]
        feat = {o: (r() < p) for o, p in zip(FEATURES, PROFILE[c])}
        ks.append({"no": i + 1, "class": c, "source": source, "features": feat,
                   "text": TEXT[int(r() * len(TEXT))], "length": 40 + int(r() * 260)})
    return ks


SECOND = build_set(20260219)
print("run      harmful  clean     blocked   false alarm      recall  false block")
for name, ks in (("first ", RECORDS), ("second", SECOND)):
    m = measure(ks, rule_filter)
    print(f"{name}  {sum(1 for k in ks if harmful(k)):>8} {sum(1 for k in ks if not harmful(k)):>6}"
          f" {m['tp']:>11} {m['fp']:>13} {m['recall']:>11.4f}"
          f" {m['false_block_rate']:>12.4f}")

print("\nform split in the second run")
for name, cond in (("direct  ", lambda k: k["source"] == "user"),
                   ("indirect", lambda k: k["source"] != "user")):
    g = [k for k in SECOND if harmful(k) and cond(k)]
    caught = sum(1 for k in g if rule_filter(k))
    print(f"  {name}  harmful {len(g):>3}  caught {caught:>3}  recall {caught / len(g):.4f}")
print("\nresolution: recall step", round(1 / 93, 4),
      " false block step", round(1 / 147, 4))

run      harmful  clean     blocked   false alarm      recall  false block
first         93    147          74            23      0.7957       0.1565
second        83    157          58            31      0.6988       0.1975

form split in the second run
  direct    harmful  36  caught  18  recall 0.5000
  indirect  harmful  47  caught  40  recall 0.8511

resolution: recall step 0.0108  false block step 0.0068

The second run shows why a single number is not enough. The same filter, the same rules, the same generator; only the draw of the set changed, and recall went from 0.7957 to 0.6988 and the false block rate from 0.1565 to 0.1975. The reason is not a worse filter but a changed composition: the second set has 83 harmful records instead of 93, and the corrupted input class, where the filter is weak, carries 25 records instead of 19. The number of a defense is a property of the set it was measured on. Every rate in this course is written next to its set.

Only the ordering between forms holds: in the second run too, the indirect form (0.8511) is markedly higher than the direct form (0.5000). When the set changed, the rates slid and the relation stayed. What is independent of the run is the position of the two subsets relative to each other. The last line writes the lower bound of the band: no smaller difference is a finding on this set.

Summary

• Prompt injection arises from data and instruction standing without distinction in the same context window; whatever the source, the text enters in the same form.
• An attack is not a text but an attribute profile: the record text is a harmless support request, and four of the six texts appear in all six classes.
• In the undefended baseline, true positives are 0, false alarms are 0, and 93 harmful records escape; reading false alarms alone makes doing nothing look perfect.
• The rule-based filter blocks 74 of the 93 harmful records (recall 0.7957), blocks 23 clean records (false block rate 0.1565), and has a precision of 0.7629.
• The direct form has 37 harmful records and the indirect form 56; recall is 0.8929 on the indirect and 0.6486 on the direct, and a single average hides this.
• The resolution of the set is 0.0108 on recall and 0.0068 on the false block rate; in the second run the rates slide and the ordering between forms holds.

Next Step

The harshest side of the rule-based filter is its sharpness: a flag is present or absent, with no intermediate value, and every structure missing from the list passes silently. The opposite approach produces a score and decides with a threshold, and that threshold is a setting. The next lesson sweeps this setting and treats guardrail bypass not as a procedure but as a threshold question: as the threshold rises, precision goes up and recall goes down, and it counts which class the defense collapses on first and at which threshold the point most open to bypass appears.