--- title: 'AI Red Teaming' source: 'https://academia.sh/en/courses/ai-safety-and-evaluation/ai-red-teaming' course: 'AI Safety and Evaluation' language: en updated: '2026-10-21T05:02:27+00:00' license: 'CC BY-SA 4.0' objectives: - 'Defines a red team campaign as the number of covered classes, the number of trials, and the share that passes the defense.' - 'Compares the different numbers that a narrow and a wide campaign give for the same defense.' - "Computes the campaign's own resolution and the band that comes from trial selection." - 'Justifies why the count of a class with no row in the report is unknown, not zero.' --- # AI Red Teaming Red teaming treated not as an attack procedure but as scope and counting: a narrow campaign sweeping four classes with ten trials each reports 4 passing records in 40 trials and a passing share of 0.1000; when a fifth class is added to the same campaign the count becomes 12 passing in 50 trials and 0.2400, and the added class alone supplies 8 of them. Changing the trial selection gives 0.1800 at the same size, all harmful records give 0.2043, and the widened rule brings it down to 0.0860. A class with no row in the report has zero trials and its passing share is unmeasured; the scope of a campaign is a statement about the plan itself, not about the world. The lesson closes the topic and writes that the defense of eight lessons was a single filter. The previous lesson added a class by name and showed that the defense was almost blind in it. That class could have stayed uncounted for years because nobody looked for it; the only reason it turned into a number was that someone thought of looking. This leaves an uncomfortable gap on the defense side: saying that a system has been tested does not say what was not tested. Red teaming is the systematic trial that tries to close this gap. In this lesson it is **not treated as a procedure**. The procedure of penetration testing was measured in the Security Tooling and Penetration Testing course and is not repeated; here red teaming is reduced to a single thing: **scope and counting.** How many classes were tried, how many trials were made in each, what the share passing the defense is, and with what number the untested appears in the report. - **AS88.** The record set, the classes, the attributes, and the defense are **fictional**; no real model or filter is called. Seed **20260218**. - **AS89.** **An attack is not a text but an attribute profile**; the text of the records is entirely harmless support requests and has no relation to the class. - **AS90.** **There is no step called producing a trial.** A campaign is selecting, class by class, the records already standing in the set. No working attack, payload, or evasion method is written. - **AS91.** The only number a campaign measures is **the passing share**; this is the complement of recall in that class. - **AS92.** A campaign **does not measure false alarm**. It looks only at harmful records, so it produces only one of the course's two numbers; the second is read from the whole set. - **AS93.** The undefended baseline is the same: blocked harmful **0**, false alarm **0**, passing share **1.0000**. - **AS94.** Resolution has two layers: the step of the set is **1/93 = 0.0108**, the step of the campaign is **1/trials**, the step of a class is **1/trials in that class**. Every row writes its own step. ## A Campaign Is a Scope, Not a Procedure Three things determine a campaign: which classes were covered, how many trials were made in each class, and against which defense it was made. All three are plan decisions, and all three move the number in the report directly. The first block builds the common definition: 240 fictional support requests, six classes, seven structural attributes, and the rule filter. The attributes are flags, not content; the text of a record is independent of its class. ```python SEED, M32 = 20260218, 0xFFFFFFFF def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 # ---- FICTIONAL records: no attack TEXT is written, an attack is a FEATURE PROFILE CLASSES = ["clean", "indirect_instruction", "hidden_field_request", "privilege_escalation", "source_spoofing", "corrupted_input"] SOURCES = ["user", "retrieved_document", "tool_output"] # Features are structural; none of them is a working payload. FEATURES = ["instruction_mode", "role_change_request", "hidden_field_name", "side_effect_tool_name", "source_stamp_mismatch", "unusual_stamp_distribution", "excessive_length"] # Per-class feature probability (fictional). Clean records also carry low shares: # in the real world, well-intentioned text can carry these features too. PROFILE = { "clean": [0.06, 0.02, 0.05, 0.03, 0.02, 0.04, 0.05], "indirect_instruction": [0.88, 0.42, 0.10, 0.30, 0.35, 0.08, 0.12], "hidden_field_request": [0.35, 0.15, 0.86, 0.12, 0.10, 0.06, 0.08], "privilege_escalation": [0.30, 0.25, 0.12, 0.84, 0.14, 0.05, 0.07], "source_spoofing": [0.22, 0.55, 0.08, 0.10, 0.82, 0.07, 0.06], "corrupted_input": [0.05, 0.03, 0.04, 0.03, 0.06, 0.80, 0.62], } SOURCE_SHARE = { "clean": [0.72, 0.20, 0.08], "indirect_instruction": [0.05, 0.70, 0.25], "hidden_field_request": [0.55, 0.30, 0.15], "privilege_escalation": [0.40, 0.35, 0.25], "source_spoofing": [0.10, 0.60, 0.30], "corrupted_input": [0.65, 0.20, 0.15], } # Fictional, entirely harmless support texts. They have NO relation to the class; # the class information lives only in the feature profile. TEXT = ["bu ayki fatura tutari gecen aya gore yuksek geldi", "sayac endeksi yanlis girilmis olabilir", "mahallede kesinti ne zaman bitecek", "tarife basamagimi ogrenmek istiyorum", "ariza kaydim icin ekip ne zaman gelir", "adres degisikligi nasil yapilir"] RECORDS = [] for i in range(240): r = rng(SEED + 6151 * i) c = CLASSES[pick(r(), [0.62, 0.10, 0.07, 0.07, 0.07, 0.07])] source = SOURCES[pick(r(), SOURCE_SHARE[c])] feat = {o: (r() < p) for o, p in zip(FEATURES, PROFILE[c])} RECORDS.append({"no": i + 1, "class": c, "source": source, "features": feat, "text": TEXT[int(r() * len(TEXT))], "length": 40 + int(r() * 260)}) def harmful(k): return k["class"] != "clean" # ---- DEFENSE 1: rule filter (structural, explainable) def rule_filter(k, rules=("instruction_mode", "hidden_field_name", "side_effect_tool_name", "source_stamp_mismatch")): return any(k["features"][o] for o in rules) # ---- MEASURE: what was blocked and what was blocked by mistake are written together def measure(records, decision): tp = fp = fn = tn = 0 for k in records: flagged, actual = decision(k), harmful(k) if flagged and actual: tp += 1 elif flagged and not actual: fp += 1 elif not flagged and actual: fn += 1 else: tn += 1 precision = tp / (tp + fp) if tp + fp else 0.0 recall = tp / (tp + fn) if tp + fn else 0.0 return {"tp": tp, "fp": fp, "fn": fn, "tn": tn, "precision": round(precision, 4), "recall": round(recall, 4), "false_block_rate": round(fp / (fp + tn), 4) if fp + tn else 0.0} ``` ## Covered Classes and the Passing Share The second block reduces a campaign to a function and runs two plans side by side. The narrow plan is the one written when the corrupted input class was not yet known: four classes, ten trials per class. The wide plan adds the fifth class to the same campaign and changes nothing else. ```python HARMFUL_CLASSES = CLASSES[1:] WIDE = ("instruction_mode", "hidden_field_name", "side_effect_tool_name", "source_stamp_mismatch", "unusual_stamp_distribution") def rule_wide(k): return rule_filter(k, WIDE) def campaign(classes, per_class, decision, from_start=True): """Red team SCOPE count. Not producing trials but selecting records from the set; no attack procedure is run. The only thing measured is the passing share.""" rows, total_n, total_p = [], 0, 0 for c in classes: group = [k for k in RECORDS if k["class"] == c] sub = group[:per_class] if from_start else group[-per_class:] passed = sum(1 for k in sub if not decision(k)) rows.append((c, len(sub), passed, passed / len(sub), 1 / len(sub))) total_n, total_p = total_n + len(sub), total_p + passed return rows, total_n, total_p def report(name, classes, per_class, decision, from_start=True): rows, n, p = campaign(classes, per_class, decision, from_start) print(f"{name}: covered classes {len(rows)}/{len(HARMFUL_CLASSES)} trials {n} passed {p}" f" passing share {p / n:.4f} campaign step {1 / n:.4f}") for c, m, ps, share, step in rows: print(f" {c:<20} trials {m:>2} passed {ps:>2} share {share:.4f} step {step:.4f}") report("narrow campaign ", HARMFUL_CLASSES[:4], 10, rule_filter) report("wide campaign ", HARMFUL_CLASSES, 10, rule_filter) ``` ``` narrow campaign : covered classes 4/5 trials 40 passed 4 passing share 0.1000 campaign step 0.0250 indirect_instruction trials 10 passed 0 share 0.0000 step 0.1000 hidden_field_request trials 10 passed 1 share 0.1000 step 0.1000 privilege_escalation trials 10 passed 1 share 0.1000 step 0.1000 source_spoofing trials 10 passed 2 share 0.2000 step 0.1000 wide campaign : covered classes 5/5 trials 50 passed 12 passing share 0.2400 campaign step 0.0200 indirect_instruction trials 10 passed 0 share 0.0000 step 0.1000 hidden_field_request trials 10 passed 1 share 0.1000 step 0.1000 privilege_escalation trials 10 passed 1 share 0.1000 step 0.1000 source_spoofing trials 10 passed 2 share 0.2000 step 0.1000 corrupted_input trials 10 passed 8 share 0.8000 step 0.1000 ``` The report of the narrow campaign is a good report. Forty trials were made, four passed, the passing share is **0.1000**. None of the four rows raises an alarm: the worst class is 0.2000, the best is 0.0000. What this table tells its reader is plain — the defense stopped nine of every ten trials. The wide campaign carries the same defense, the same trial density, and the same four rows. The only difference is one more row. The passing share goes up to **0.2400**, more than double, and **eight** of the twelve passing records come from the new row. The four rows stayed identical letter for letter; the only thing that changed is what was looked at. The number of a campaign does not measure the strength of the defense, it measures **the scope of the plan**. ## The Count of the Untested Class The third block moves the scope in three more directions: another trial selection of the same size, all harmful records, and the widened rule. At the end it prints the two numbers of the set and the class that has no row in the report of the narrow campaign. ```python report("same campaign, picked from the end", HARMFUL_CLASSES, 10, rule_filter, from_start=False) report("all harmful records ", HARMFUL_CLASSES, 999, rule_filter) report("all records, wide rule ", HARMFUL_CLASSES, 999, rule_wide) print("\nthe two numbers of the defense (whole set):") for name, decision in (("undefended baseline", lambda k: False), ("rule", rule_filter), ("rule + flag", rule_wide)): m = measure(RECORDS, decision) print(f" {name:<19} blocked {m['tp']:>2} false alarm {m['fp']:>2}" f" recall {m['recall']:.4f} false block rate {m['false_block_rate']:.4f}") print("\nharmful classes defined in the set:", len(HARMFUL_CLASSES), " with no row in the narrow campaign's report:", HARMFUL_CLASSES[4:], " trials of that row:", 0) ``` ``` same campaign, picked from the end: covered classes 5/5 trials 50 passed 9 passing share 0.1800 campaign step 0.0200 indirect_instruction trials 10 passed 0 share 0.0000 step 0.1000 hidden_field_request trials 10 passed 0 share 0.0000 step 0.1000 privilege_escalation trials 10 passed 1 share 0.1000 step 0.1000 source_spoofing trials 10 passed 1 share 0.1000 step 0.1000 corrupted_input trials 10 passed 7 share 0.7000 step 0.1000 all harmful records : covered classes 5/5 trials 93 passed 19 passing share 0.2043 campaign step 0.0108 indirect_instruction trials 24 passed 0 share 0.0000 step 0.0417 hidden_field_request trials 17 passed 1 share 0.0588 step 0.0588 privilege_escalation trials 10 passed 1 share 0.1000 step 0.1000 source_spoofing trials 23 passed 3 share 0.1304 step 0.0435 corrupted_input trials 19 passed 14 share 0.7368 step 0.0526 all records, wide rule : covered classes 5/5 trials 93 passed 8 passing share 0.0860 campaign step 0.0108 indirect_instruction trials 24 passed 0 share 0.0000 step 0.0417 hidden_field_request trials 17 passed 1 share 0.0588 step 0.0588 privilege_escalation trials 10 passed 1 share 0.1000 step 0.1000 source_spoofing trials 23 passed 3 share 0.1304 step 0.0435 corrupted_input trials 19 passed 3 share 0.1579 step 0.0526 the two numbers of the defense (whole set): undefended baseline blocked 0 false alarm 0 recall 0.0000 false block rate 0.0000 rule blocked 74 false alarm 23 recall 0.7957 false block rate 0.1565 rule + flag blocked 85 false alarm 30 recall 0.9140 false block rate 0.2041 harmful classes defined in the set: 5 with no row in the narrow campaign's report: ['corrupted_input'] trials of that row: 0 ``` The first row gives the campaign's own band. The same defense, the same five classes, the same fifty trials; the only difference is which records were selected. The passing share comes out **0.1800** instead of 0.2400. This difference of three records has nothing to do with the defense, and since the total step of the campaign is 0.0200 it cannot be dissolved into noise either — even with the scope fixed, **the selection alone moves 0.0600.** The number of a campaign is not repeatable; a repeated campaign gives another number. The second row concentrates the scope: all records instead of ten per class, 93 trials. The passing share is 0.2043. The corrupted input row goes down from 0.8000 to **0.7368**, a difference of 0.0632; since the step of the ten-trial campaign is 0.1000, that campaign **could not have told these two numbers apart**. Writing a number with decimals does not make it precise. The third row runs the campaign against a change in the defense. The widened rule brings the passing share down from 0.2043 to **0.0860**; the whole gain again lies in one row, from 0.7368 to 0.1579. The campaign reports this as a success and **reports it incompletely**. The false alarm of the same defense rose from 23 to 30 and this number is in no row of the campaign, because the campaign looks only at harmful records. A red team report, by its structure, produces **only one** of the course's two numbers; the second must be read separately from the whole set. The last line carries the most important sentence of the lesson. The report of the narrow campaign has no row for `corrupted_input`, and the trial count of that row is zero. Someone who sums the table sees 40 trials, 4 passing, and 0.1000; the lost class contributes zero to the totals. But that zero **is not a measured zero** — it is the number of not having looked. As soon as the class entered the scope, it alone supplied eight of the twelve passing records. **The count of the untested class is not zero, it is unknown**, and a report has to be written in a way that tells the two apart: unless covered and uncovered classes are listed separately, a gap in scope looks like a success figure. There is a harsher version of this. The number of harmful classes defined in this set is five; if a sixth surface exists, it appears in no row of **any** campaign run on this set. Scope is not a number but a claim, and it says something not about the world but about **the plan itself**. ## Summary - Red teaming here is not a procedure but a count: covered classes, trials per class, and the share passing the defense. - The narrow campaign reports a passing share of 0.1000 in 40 trials over four classes; when the fifth class is added, the same defense gives 0.2400 in 50 trials, and eight of the twelve passing records come from the new row. - The number of a campaign carries a band from its own selection: another selection of the same size gives 0.1800, the difference is 0.0600 and has nothing to do with the defense. - As the scope concentrates, resolution grows: the passing share is 0.2043 in 93 trials and the corrupted input row is 0.7368; the ten-trial campaign could not have told it apart from 0.8000. - Since the campaign looks only at harmful records, it produces one of the course's two numbers; the widened rule brings the passing share down to 0.0860 while raising false alarm from 23 to 30, and this number is not in the report. - The trial count of a class with no row in the report is zero and its passing share is unknown; unless covered and uncovered are written separately, a gap looks like a success. ## Next Step In eight lessons the surface was counted. The text entering the model was divided into classes, distributed by its source, swept with a threshold, records carrying hidden field names were counted, the side-effect tool request was separated, the output side was measured, corruption that carries no meaning was added as a separate class, and in the end the scope itself turned into a number. In all of this the defense was **always the same thing**: a single filter. Sometimes a list of four rules, sometimes a list of five rules widened with a flag, sometimes a score with a threshold. The work it did never changed — it either passed the record or blocked it, and it always paid the cost in the same currency: false alarm. Yet blocking is not the only move of a defense. Deleting a field, not counting a text as an instruction, narrowing the authority of a tool, and asking a human before an action are defenses too, and none of them stops the record. The next topic builds these layers one by one and measures each with its own cost: what it catches, what it blocks by mistake, and which capability it takes away.