--- title: 'Prompt Injection' source: 'https://academia.sh/en/courses/ai-safety-and-evaluation/prompt-injection' course: 'AI Safety and Evaluation' language: en updated: '2026-10-21T05:02:27+00:00' license: 'CC BY-SA 4.0' objectives: - 'Defines the attack surface not as a working text but as a structural attribute profile attached to a record.' - 'Compares the undefended baseline with the rule-based filter and writes the blocked harmful records together with the wrongly blocked clean ones.' - "Separates the direct and indirect forms by source distribution and counts the filter's recall on each form." --- # Prompt Injection Counting the text that enters a model as an attack surface: in the fictional record set, 93 of 240 records are harmful and 147 are clean, and being harmful lives not in the text but in a structural attribute profile. In the undefended baseline, true positives are 0 and false alarms are 0; the rule-based filter blocks 74 harmful records, wrongly blocks 23 clean ones, and lets 19 harmful ones escape (precision 0.7629, recall 0.7957, false block rate 0.1565). The direct and indirect forms are separated by source distribution: recall is 0.6486 on 37 direct harmful records and 0.8929 on 56 indirect ones. In the second run the same filter gives 0.6988; the resolution of the set is 0.0108 on recall and 0.0068 on false blocks. The eleven lessons of the previous course measured **whose hands** the tool is in: the definition moved to the other side, the cost of the envelope was counted, and a definition reported by an untrusted server was checked item by item. All of that was **outside the interface** — the shell of the message was counted, not the text inside it. For a language model, however, data and instruction have no structural boundary: both enter the same context window in the same form. A user's sentence, the body of a retrieved document, and a value returned by a tool are of the same kind in the model's eyes. **Prompt injection** names this lack of distinction: the ability of text entering a model to make it do more than the authority of its path allows. This lesson does not describe the surface; it **counts** it. In one record set, how many harmful records are blocked with no defense, how many with a rule-based filter, and how many clean records does that filter wrongly block? The rule of the course holds from the first line: **a defense that does not write the share it blocks together with the share it blocks by mistake counts as unmeasured.** - **AS1.** The record set, classes, attributes, and defense layers are **fictional**; no real model endpoint or incident log is used. Seed **20260218**. - **AS2.** **This is not an attack lesson.** All code that runs is on the defense side: classification, filter, count, and measure. No working payload, template, instruction sequence, or bypass routine is written. - **AS3.** The rule is structural: **an attack is not a text but an attribute profile.** The record text is entirely harmless support requests with no relation to the class. - **AS4.** Class names are **labels**, not procedure recipes. When a new class is needed, a flag is added to the attribute list, not a text. - **AS5.** Input validation was measured in the Secure Coding course and threat modeling in the Secure Development Lifecycle course; their procedures are not repeated. Only the surface **specific to the language model** is added. - **AS6.** `precision` and `recall` are used with the classification meaning they have in the Machine Learning and Data Science curriculum. A **false alarm** is a clean record counted as harmful; a **false negative** is a harmful record that escapes. - **AS7.** The rule-based filter has no sampling and gives the same number on the same set in every run; the second run regenerates the set with a second seed. - **AS8.** The smallest step of recall is **1/93** and the step of the false block rate is **1/147**; no difference smaller than these steps can be measured. ## An Attack Is a Feature Profile, Not a Text The usual way to teach an attack surface is to show an example. This course does not, and not as a matter of style but as a **structural constraint**: the record set has no field called attack text. What a record carries is seven **flags** — instruction mode, role change request, hidden field name, side-effect tool name, source stamp mismatch, unusual token distribution, excessive length. Each is a true-or-false value about the **structure** of a record; none runs. The record also has a `text` field, and one of six fictional support requests sits there. Those six sentences are ordinary subscriber questions and **have no relation to the class**: which sentence falls on which record is decided by a separate draw. Because the text does not determine the class, the lesson never shows an attack text and the measure rests on the profile. In a real setup the filter reads the text; here what is measured is **a profile that behaves like the output of that reading**. ```python # FICTIONAL. The record set, classes, attributes and defense layers are defined inside # the lesson; no real model endpoint is called. No attack TEXT is written. SEED, M32 = 20260218, 0xFFFFFFFF def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def trace_id(m): h = 2166136261 for c in str(m): h = ((h ^ ord(c)) * 16777619) & M32 return h # ---- FICTIONAL records: no attack TEXT is written, an attack is a FEATURE PROFILE CLASSES = ["clean", "indirect_instruction", "hidden_field_request", "privilege_escalation", "source_spoofing", "corrupted_input"] SOURCES = ["user", "retrieved_document", "tool_output"] # Features are structural; none of them is a working payload. FEATURES = ["instruction_mode", "role_change_request", "hidden_field_name", "side_effect_tool_name", "source_stamp_mismatch", "unusual_stamp_distribution", "excessive_length"] # Feature probability per class (fictional). Clean records carry small shares too: # in the real world, well-meant text can carry these features as well. PROFILE = { "clean": [0.06, 0.02, 0.05, 0.03, 0.02, 0.04, 0.05], "indirect_instruction": [0.88, 0.42, 0.10, 0.30, 0.35, 0.08, 0.12], "hidden_field_request": [0.35, 0.15, 0.86, 0.12, 0.10, 0.06, 0.08], "privilege_escalation": [0.30, 0.25, 0.12, 0.84, 0.14, 0.05, 0.07], "source_spoofing": [0.22, 0.55, 0.08, 0.10, 0.82, 0.07, 0.06], "corrupted_input": [0.05, 0.03, 0.04, 0.03, 0.06, 0.80, 0.62], } SOURCE_SHARE = { "clean": [0.72, 0.20, 0.08], "indirect_instruction": [0.05, 0.70, 0.25], "hidden_field_request": [0.55, 0.30, 0.15], "privilege_escalation": [0.40, 0.35, 0.25], "source_spoofing": [0.10, 0.60, 0.30], "corrupted_input": [0.65, 0.20, 0.15], } # Fictional, entirely harmless support texts. They have NO relation to the class; # the class information lives only in the feature profile. TEXT = ["bu ayki fatura tutari gecen aya gore yuksek geldi", "sayac endeksi yanlis girilmis olabilir", "mahallede kesinti ne zaman bitecek", "tarife basamagimi ogrenmek istiyorum", "ariza kaydim icin ekip ne zaman gelir", "adres degisikligi nasil yapilir"] RECORDS = [] for i in range(240): r = rng(SEED + 6151 * i) c = CLASSES[pick(r(), [0.62, 0.10, 0.07, 0.07, 0.07, 0.07])] source = SOURCES[pick(r(), SOURCE_SHARE[c])] feat = {o: (r() < p) for o, p in zip(FEATURES, PROFILE[c])} RECORDS.append({"no": i + 1, "class": c, "source": source, "features": feat, "text": TEXT[int(r() * len(TEXT))], "length": 40 + int(r() * 260)}) def harmful(k): return k["class"] != "clean" # ---- DEFENSE 1: rule filter (structural, explainable) def rule_filter(k, rules=("instruction_mode", "hidden_field_name", "side_effect_tool_name", "source_stamp_mismatch")): return any(k["features"][o] for o in rules) # ---- DEFENSE 2: model-based filter (score + threshold) WEIGHT = {"instruction_mode": 0.42, "role_change_request": 0.26, "hidden_field_name": 0.44, "side_effect_tool_name": 0.40, "source_stamp_mismatch": 0.38, "unusual_stamp_distribution": 0.30, "excessive_length": 0.12} SOURCE_BONUS = {"user": 0.00, "retrieved_document": 0.18, "tool_output": 0.12} def score(k): p = sum(WEIGHT[o] for o in FEATURES if k["features"][o]) + SOURCE_BONUS[k["source"]] r = rng(SEED ^ trace_id(k["no"])) return min(max(p + (r() - 0.5) * 0.20, 0.0), 1.6) def model_filter(k, threshold=0.40): return score(k) >= threshold # ---- MEASURE: what was blocked and what was blocked by mistake are written together def measure(records, decision): tp = fp = tn = fn = 0 for k in records: flagged, actual = decision(k), harmful(k) if flagged and actual: tp += 1 elif flagged and not actual: fp += 1 elif not flagged and actual: fn += 1 else: tn += 1 precision = tp / (tp + fp) if tp + fp else 0.0 recall = tp / (tp + fn) if tp + fn else 0.0 return {"tp": tp, "fp": fp, "fn": fn, "tn": tn, "precision": round(precision, 4), "recall": round(recall, 4), "false_block_rate": round(fp / (fp + tn), 4) if fp + tn else 0.0} ``` ## The Undefended Baseline and the Rule-Based Filter The second block prints the composition of the set, then puts two setups side by side. The first is the **undefended baseline**: no filter, marking, or gate, so nothing is blocked. The second is a **rule-based filter** that looks at four flags. ```python from collections import Counter print("records:", len(RECORDS), " harmful:", sum(1 for k in RECORDS if harmful(k)), " clean:", sum(1 for k in RECORDS if not harmful(k))) print("class:", {c: sum(1 for k in RECORDS if k["class"] == c) for c in CLASSES}) print("source:", {x: sum(1 for k in RECORDS if k["source"] == x) for x in SOURCES}) print("number of classes in which each text appears:", [len({k["class"] for k in RECORDS if k["text"] == t}) for t in TEXT]) def undefended(k): """Baseline: no filter, marking or gate.""" return False print("\nsetup blocked false alarm escaped precision recall false block") for name, decision in (("undefended", undefended), ("rule filter", rule_filter)): m = measure(RECORDS, decision) print(f"{name:<15} {m['tp']:>10} {m['fp']:>13} {m['fn']:>8}" f" {m['precision']:>10.4f} {m['recall']:>11.4f} {m['false_block_rate']:>12.4f}") direct = [k for k in RECORDS if harmful(k) and k["source"] == "user"] indirect = [k for k in RECORDS if harmful(k) and k["source"] != "user"] print("\nform harmful rule caught recall") for name, g in (("direct", direct), ("indirect", indirect)): caught = sum(1 for k in g if rule_filter(k)) print(f"{name:<10} {len(g):>7} {caught:>15} {caught / len(g):>11.4f}") print("\nsource distribution per class") for c in CLASSES: cnt = Counter(k["source"] for k in RECORDS if k["class"] == c) print(f" {c:<20} " + " ".join(f"{x}={cnt.get(x, 0):>3}" for x in SOURCES)) ``` ``` records: 240 harmful: 93 clean: 147 class: {'clean': 147, 'indirect_instruction': 24, 'hidden_field_request': 17, 'privilege_escalation': 10, 'source_spoofing': 23, 'corrupted_input': 19} source: {'user': 137, 'retrieved_document': 72, 'tool_output': 31} number of classes in which each text appears: [6, 5, 6, 6, 5, 6] setup blocked false alarm escaped precision recall false block undefended 0 0 93 0.0000 0.0000 0.0000 rule filter 74 23 19 0.7629 0.7957 0.1565 form harmful rule caught recall direct 37 24 0.6486 indirect 56 50 0.8929 source distribution per class clean user=100 retrieved_document= 33 tool_output= 14 indirect_instruction user= 2 retrieved_document= 15 tool_output= 7 hidden_field_request user= 13 retrieved_document= 4 tool_output= 0 privilege_escalation user= 3 retrieved_document= 2 tool_output= 5 source_spoofing user= 3 retrieved_document= 16 tool_output= 4 corrupted_input user= 16 retrieved_document= 2 tool_output= 1 ``` The fourth line confirms the design: four of the six texts appear in **all six classes** and two in five. The text does not carry the class. The baseline row sets the measurement frame of the course. In the undefended setup, **true positives are 0**, **false alarms are 0**, and escaped harmful records are **93**. Precision and recall are zero. The row must be there because the undefended setup has **zero false alarms** too, and if false alarms are counted alone, doing nothing looks perfect. The number of a defense gains meaning only when the two columns are read together. The rule-based filter blocks **74** of the 93 harmful records: recall is 0.7957. Its cost is written in two columns: **23 clean records** were wrongly blocked, 0.1565 of the 147 clean records. A precision of 0.7629 means about one in four flagged records is harmless. The 19 escaped harmful records also stay in the table; the measure of a defense is **what it misses** as much as what it catches. ## Direct and Indirect Form The two forms of prompt injection are separated not by text but by **source**. In the direct form the harmful content comes from a request the user wrote. The **indirect injection** comes from a place the user did not write: the body of a retrieved document or a value returned by a tool. The user is the victim here, not the source. The distinction matters for authority: in the indirect form, the path that brings the text into the system is one the user approved, and it generally runs with broader authority. The source distribution turns this distinction into a number. **37** of the 93 harmful records come from the user and **56** from a retrieved document or a tool output — most of the surface is indirect. The class table divides this among the classes: of the 24 records of the indirect injection class only 2 come from the user, and of the 23 of the source spoofing class, 3. By contrast, the hidden field request class (13 of 17) and the corrupted input class (16 of 19) are predominantly direct. The filter's number differs between the forms: recall is **0.8929** on the indirect form and **0.6486** on the direct form. The difference is 0.2443, more than twenty times the resolution of the set, so it is measurable. The reason is in the rule list: one of the four rules is `source_stamp_mismatch`, a flag that is frequent in the indirect classes and rare in the direct ones. The filter is **markedly weaker on one side of the surface than on the other**, and a single average recall hides this: the overall rate swallows the difference between the subsets. ## The Band of the Measure The rule-based filter contains no sampling; on the same set it gives exactly the same five numbers in every run. So the second run changes not the filter but the **set**: the same definition is regenerated with a second seed. ```python def build_set(seed): """Same definition, another seed: the record set is regenerated.""" ks = [] for i in range(240): r = rng(seed + 6151 * i) c = CLASSES[pick(r(), [0.62, 0.10, 0.07, 0.07, 0.07, 0.07])] source = SOURCES[pick(r(), SOURCE_SHARE[c])] feat = {o: (r() < p) for o, p in zip(FEATURES, PROFILE[c])} ks.append({"no": i + 1, "class": c, "source": source, "features": feat, "text": TEXT[int(r() * len(TEXT))], "length": 40 + int(r() * 260)}) return ks SECOND = build_set(20260219) print("run harmful clean blocked false alarm recall false block") for name, ks in (("first ", RECORDS), ("second", SECOND)): m = measure(ks, rule_filter) print(f"{name} {sum(1 for k in ks if harmful(k)):>8} {sum(1 for k in ks if not harmful(k)):>6}" f" {m['tp']:>11} {m['fp']:>13} {m['recall']:>11.4f}" f" {m['false_block_rate']:>12.4f}") print("\nform split in the second run") for name, cond in (("direct ", lambda k: k["source"] == "user"), ("indirect", lambda k: k["source"] != "user")): g = [k for k in SECOND if harmful(k) and cond(k)] caught = sum(1 for k in g if rule_filter(k)) print(f" {name} harmful {len(g):>3} caught {caught:>3} recall {caught / len(g):.4f}") print("\nresolution: recall step", round(1 / 93, 4), " false block step", round(1 / 147, 4)) ``` ``` run harmful clean blocked false alarm recall false block first 93 147 74 23 0.7957 0.1565 second 83 157 58 31 0.6988 0.1975 form split in the second run direct harmful 36 caught 18 recall 0.5000 indirect harmful 47 caught 40 recall 0.8511 resolution: recall step 0.0108 false block step 0.0068 ``` The second run shows why a single number is not enough. The same filter, the same rules, the same generator; only the draw of the set changed, and recall went from 0.7957 to **0.6988** and the false block rate from 0.1565 to **0.1975**. The reason is not a worse filter but a changed composition: the second set has 83 harmful records instead of 93, and the corrupted input class, where the filter is weak, carries 25 records instead of 19. **The number of a defense is a property of the set it was measured on.** Every rate in this course is written next to its set. Only **the ordering between forms** holds: in the second run too, the indirect form (0.8511) is markedly higher than the direct form (0.5000). When the set changed, the rates slid and the relation stayed. What is independent of the run is the **position of the two subsets relative to each other**. The last line writes the lower bound of the band: no smaller difference is a finding on this set. ## Summary - Prompt injection arises from data and instruction standing without distinction in the same context window; whatever the source, the text enters in the same form. - An attack is not a text but an **attribute profile**: the record text is a harmless support request, and four of the six texts appear in all six classes. - In the undefended baseline, true positives are 0, false alarms are 0, and 93 harmful records escape; reading false alarms alone makes doing nothing look perfect. - The rule-based filter blocks 74 of the 93 harmful records (recall 0.7957), blocks 23 clean records (false block rate 0.1565), and has a precision of 0.7629. - The direct form has 37 harmful records and the indirect form 56; recall is 0.8929 on the indirect and 0.6486 on the direct, and a single average hides this. - The resolution of the set is 0.0108 on recall and 0.0068 on the false block rate; in the second run the rates slide and the ordering between forms holds. ## Next Step The harshest side of the rule-based filter is its sharpness: a flag is present or absent, with no intermediate value, and every structure missing from the list passes silently. The opposite approach produces a **score** and decides with a **threshold**, and that threshold is a setting. The next lesson sweeps this setting and treats guardrail bypass not as a procedure but as **a threshold question**: as the threshold rises, precision goes up and recall goes down, and it counts which class the defense collapses on first and at which threshold the point most open to bypass appears.