AI Red Teaming

The previous lesson added a class by name and showed that the defense was almost blind in it. That class could have stayed uncounted for years because nobody looked for it; the only reason it turned into a number was that someone thought of looking. This leaves an uncomfortable gap on the defense side: saying that a system has been tested does not say what was not tested.

Red teaming is the systematic trial that tries to close this gap. In this lesson it is not treated as a procedure. The procedure of penetration testing was measured in the Security Tooling and Penetration Testing course and is not repeated; here red teaming is reduced to a single thing: scope and counting. How many classes were tried, how many trials were made in each, what the share passing the defense is, and with what number the untested appears in the report.

• AS88. The record set, the classes, the attributes, and the defense are fictional; no real model or filter is called. Seed 20260218.
• AS89. An attack is not a text but an attribute profile; the text of the records is entirely harmless support requests and has no relation to the class.
• AS90. There is no step called producing a trial. A campaign is selecting, class by class, the records already standing in the set. No working attack, payload, or evasion method is written.
• AS91. The only number a campaign measures is the passing share; this is the complement of recall in that class.
• AS92. A campaign does not measure false alarm. It looks only at harmful records, so it produces only one of the course’s two numbers; the second is read from the whole set.
• AS93. The undefended baseline is the same: blocked harmful 0, false alarm 0, passing share 1.0000.
• AS94. Resolution has two layers: the step of the set is 1/93 = 0.0108, the step of the campaign is 1/trials, the step of a class is 1/trials in that class. Every row writes its own step.

A Campaign Is a Scope, Not a Procedure

Three things determine a campaign: which classes were covered, how many trials were made in each class, and against which defense it was made. All three are plan decisions, and all three move the number in the report directly. The first block builds the common definition: 240 fictional support requests, six classes, seven structural attributes, and the rule filter. The attributes are flags, not content; the text of a record is independent of its class.

SEED, M32 = 20260218, 0xFFFFFFFF


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def pick(u, w):
    t = 0.0
    for i, x in enumerate(w):
        t += x
        if u < t:
            return i
    return len(w) - 1


# ---- FICTIONAL records: no attack TEXT is written, an attack is a FEATURE PROFILE
CLASSES = ["clean", "indirect_instruction", "hidden_field_request", "privilege_escalation",
           "source_spoofing", "corrupted_input"]
SOURCES = ["user", "retrieved_document", "tool_output"]
# Features are structural; none of them is a working payload.
FEATURES = ["instruction_mode", "role_change_request", "hidden_field_name", "side_effect_tool_name",
            "source_stamp_mismatch", "unusual_stamp_distribution", "excessive_length"]
# Per-class feature probability (fictional). Clean records also carry low shares:
# in the real world, well-intentioned text can carry these features too.
PROFILE = {
    "clean":                [0.06, 0.02, 0.05, 0.03, 0.02, 0.04, 0.05],
    "indirect_instruction": [0.88, 0.42, 0.10, 0.30, 0.35, 0.08, 0.12],
    "hidden_field_request": [0.35, 0.15, 0.86, 0.12, 0.10, 0.06, 0.08],
    "privilege_escalation": [0.30, 0.25, 0.12, 0.84, 0.14, 0.05, 0.07],
    "source_spoofing":      [0.22, 0.55, 0.08, 0.10, 0.82, 0.07, 0.06],
    "corrupted_input":      [0.05, 0.03, 0.04, 0.03, 0.06, 0.80, 0.62],
}
SOURCE_SHARE = {
    "clean":                [0.72, 0.20, 0.08],
    "indirect_instruction": [0.05, 0.70, 0.25],
    "hidden_field_request": [0.55, 0.30, 0.15],
    "privilege_escalation": [0.40, 0.35, 0.25],
    "source_spoofing":      [0.10, 0.60, 0.30],
    "corrupted_input":      [0.65, 0.20, 0.15],
}
# Fictional, entirely harmless support texts. They have NO relation to the class;
# the class information lives only in the feature profile.
TEXT = ["bu ayki fatura tutari gecen aya gore yuksek geldi",
        "sayac endeksi yanlis girilmis olabilir",
        "mahallede kesinti ne zaman bitecek",
        "tarife basamagimi ogrenmek istiyorum",
        "ariza kaydim icin ekip ne zaman gelir",
        "adres degisikligi nasil yapilir"]

RECORDS = []
for i in range(240):
    r = rng(SEED + 6151 * i)
    c = CLASSES[pick(r(), [0.62, 0.10, 0.07, 0.07, 0.07, 0.07])]
    source = SOURCES[pick(r(), SOURCE_SHARE[c])]
    feat = {o: (r() < p) for o, p in zip(FEATURES, PROFILE[c])}
    RECORDS.append({"no": i + 1, "class": c, "source": source, "features": feat,
                    "text": TEXT[int(r() * len(TEXT))],
                    "length": 40 + int(r() * 260)})


def harmful(k):
    return k["class"] != "clean"


# ---- DEFENSE 1: rule filter (structural, explainable)
def rule_filter(k, rules=("instruction_mode", "hidden_field_name", "side_effect_tool_name",
                          "source_stamp_mismatch")):
    return any(k["features"][o] for o in rules)


# ---- MEASURE: what was blocked and what was blocked by mistake are written together
def measure(records, decision):
    tp = fp = fn = tn = 0
    for k in records:
        flagged, actual = decision(k), harmful(k)
        if flagged and actual:
            tp += 1
        elif flagged and not actual:
            fp += 1
        elif not flagged and actual:
            fn += 1
        else:
            tn += 1
    precision = tp / (tp + fp) if tp + fp else 0.0
    recall = tp / (tp + fn) if tp + fn else 0.0
    return {"tp": tp, "fp": fp, "fn": fn, "tn": tn,
            "precision": round(precision, 4), "recall": round(recall, 4),
            "false_block_rate": round(fp / (fp + tn), 4) if fp + tn else 0.0}

Covered Classes and the Passing Share

The second block reduces a campaign to a function and runs two plans side by side. The narrow plan is the one written when the corrupted input class was not yet known: four classes, ten trials per class. The wide plan adds the fifth class to the same campaign and changes nothing else.

HARMFUL_CLASSES = CLASSES[1:]
WIDE = ("instruction_mode", "hidden_field_name", "side_effect_tool_name",
        "source_stamp_mismatch", "unusual_stamp_distribution")


def rule_wide(k):
    return rule_filter(k, WIDE)


def campaign(classes, per_class, decision, from_start=True):
    """Red team SCOPE count. Not producing trials but selecting records from the set;
    no attack procedure is run. The only thing measured is the passing share."""
    rows, total_n, total_p = [], 0, 0
    for c in classes:
        group = [k for k in RECORDS if k["class"] == c]
        sub = group[:per_class] if from_start else group[-per_class:]
        passed = sum(1 for k in sub if not decision(k))
        rows.append((c, len(sub), passed, passed / len(sub), 1 / len(sub)))
        total_n, total_p = total_n + len(sub), total_p + passed
    return rows, total_n, total_p


def report(name, classes, per_class, decision, from_start=True):
    rows, n, p = campaign(classes, per_class, decision, from_start)
    print(f"{name}: covered classes {len(rows)}/{len(HARMFUL_CLASSES)}  trials {n}  passed {p}"
          f"  passing share {p / n:.4f}  campaign step {1 / n:.4f}")
    for c, m, ps, share, step in rows:
        print(f"   {c:<20} trials {m:>2}  passed {ps:>2}  share {share:.4f}  step {step:.4f}")


report("narrow campaign  ", HARMFUL_CLASSES[:4], 10, rule_filter)
report("wide campaign    ", HARMFUL_CLASSES, 10, rule_filter)

narrow campaign  : covered classes 4/5  trials 40  passed 4  passing share 0.1000  campaign step 0.0250
   indirect_instruction trials 10  passed  0  share 0.0000  step 0.1000
   hidden_field_request trials 10  passed  1  share 0.1000  step 0.1000
   privilege_escalation trials 10  passed  1  share 0.1000  step 0.1000
   source_spoofing      trials 10  passed  2  share 0.2000  step 0.1000
wide campaign    : covered classes 5/5  trials 50  passed 12  passing share 0.2400  campaign step 0.0200
   indirect_instruction trials 10  passed  0  share 0.0000  step 0.1000
   hidden_field_request trials 10  passed  1  share 0.1000  step 0.1000
   privilege_escalation trials 10  passed  1  share 0.1000  step 0.1000
   source_spoofing      trials 10  passed  2  share 0.2000  step 0.1000
   corrupted_input      trials 10  passed  8  share 0.8000  step 0.1000

The report of the narrow campaign is a good report. Forty trials were made, four passed, the passing share is 0.1000. None of the four rows raises an alarm: the worst class is 0.2000, the best is 0.0000. What this table tells its reader is plain — the defense stopped nine of every ten trials.

The wide campaign carries the same defense, the same trial density, and the same four rows. The only difference is one more row. The passing share goes up to 0.2400, more than double, and eight of the twelve passing records come from the new row. The four rows stayed identical letter for letter; the only thing that changed is what was looked at. The number of a campaign does not measure the strength of the defense, it measures the scope of the plan.

The Count of the Untested Class

The third block moves the scope in three more directions: another trial selection of the same size, all harmful records, and the widened rule. At the end it prints the two numbers of the set and the class that has no row in the report of the narrow campaign.

report("same campaign, picked from the end", HARMFUL_CLASSES, 10, rule_filter, from_start=False)
report("all harmful records              ", HARMFUL_CLASSES, 999, rule_filter)
report("all records, wide rule           ", HARMFUL_CLASSES, 999, rule_wide)
print("\nthe two numbers of the defense (whole set):")
for name, decision in (("undefended baseline", lambda k: False), ("rule", rule_filter),
                       ("rule + flag", rule_wide)):
    m = measure(RECORDS, decision)
    print(f"  {name:<19} blocked {m['tp']:>2}  false alarm {m['fp']:>2}"
          f"  recall {m['recall']:.4f}  false block rate {m['false_block_rate']:.4f}")
print("\nharmful classes defined in the set:", len(HARMFUL_CLASSES),
      " with no row in the narrow campaign's report:", HARMFUL_CLASSES[4:],
      " trials of that row:", 0)

same campaign, picked from the end: covered classes 5/5  trials 50  passed 9  passing share 0.1800  campaign step 0.0200
   indirect_instruction trials 10  passed  0  share 0.0000  step 0.1000
   hidden_field_request trials 10  passed  0  share 0.0000  step 0.1000
   privilege_escalation trials 10  passed  1  share 0.1000  step 0.1000
   source_spoofing      trials 10  passed  1  share 0.1000  step 0.1000
   corrupted_input      trials 10  passed  7  share 0.7000  step 0.1000
all harmful records              : covered classes 5/5  trials 93  passed 19  passing share 0.2043  campaign step 0.0108
   indirect_instruction trials 24  passed  0  share 0.0000  step 0.0417
   hidden_field_request trials 17  passed  1  share 0.0588  step 0.0588
   privilege_escalation trials 10  passed  1  share 0.1000  step 0.1000
   source_spoofing      trials 23  passed  3  share 0.1304  step 0.0435
   corrupted_input      trials 19  passed 14  share 0.7368  step 0.0526
all records, wide rule           : covered classes 5/5  trials 93  passed 8  passing share 0.0860  campaign step 0.0108
   indirect_instruction trials 24  passed  0  share 0.0000  step 0.0417
   hidden_field_request trials 17  passed  1  share 0.0588  step 0.0588
   privilege_escalation trials 10  passed  1  share 0.1000  step 0.1000
   source_spoofing      trials 23  passed  3  share 0.1304  step 0.0435
   corrupted_input      trials 19  passed  3  share 0.1579  step 0.0526

the two numbers of the defense (whole set):
  undefended baseline blocked  0  false alarm  0  recall 0.0000  false block rate 0.0000
  rule                blocked 74  false alarm 23  recall 0.7957  false block rate 0.1565
  rule + flag         blocked 85  false alarm 30  recall 0.9140  false block rate 0.2041

harmful classes defined in the set: 5  with no row in the narrow campaign's report: ['corrupted_input']  trials of that row: 0

The first row gives the campaign’s own band. The same defense, the same five classes, the same fifty trials; the only difference is which records were selected. The passing share comes out 0.1800 instead of 0.2400. This difference of three records has nothing to do with the defense, and since the total step of the campaign is 0.0200 it cannot be dissolved into noise either — even with the scope fixed, the selection alone moves 0.0600. The number of a campaign is not repeatable; a repeated campaign gives another number.

The second row concentrates the scope: all records instead of ten per class, 93 trials. The passing share is 0.2043. The corrupted input row goes down from 0.8000 to 0.7368, a difference of 0.0632; since the step of the ten-trial campaign is 0.1000, that campaign could not have told these two numbers apart. Writing a number with decimals does not make it precise.

The third row runs the campaign against a change in the defense. The widened rule brings the passing share down from 0.2043 to 0.0860; the whole gain again lies in one row, from 0.7368 to 0.1579. The campaign reports this as a success and reports it incompletely. The false alarm of the same defense rose from 23 to 30 and this number is in no row of the campaign, because the campaign looks only at harmful records. A red team report, by its structure, produces only one of the course’s two numbers; the second must be read separately from the whole set.

The last line carries the most important sentence of the lesson. The report of the narrow campaign has no row for corrupted_input, and the trial count of that row is zero. Someone who sums the table sees 40 trials, 4 passing, and 0.1000; the lost class contributes zero to the totals. But that zero is not a measured zero — it is the number of not having looked. As soon as the class entered the scope, it alone supplied eight of the twelve passing records. The count of the untested class is not zero, it is unknown, and a report has to be written in a way that tells the two apart: unless covered and uncovered classes are listed separately, a gap in scope looks like a success figure.

There is a harsher version of this. The number of harmful classes defined in this set is five; if a sixth surface exists, it appears in no row of any campaign run on this set. Scope is not a number but a claim, and it says something not about the world but about the plan itself.

Summary

• Red teaming here is not a procedure but a count: covered classes, trials per class, and the share passing the defense.
• The narrow campaign reports a passing share of 0.1000 in 40 trials over four classes; when the fifth class is added, the same defense gives 0.2400 in 50 trials, and eight of the twelve passing records come from the new row.
• The number of a campaign carries a band from its own selection: another selection of the same size gives 0.1800, the difference is 0.0600 and has nothing to do with the defense.
• As the scope concentrates, resolution grows: the passing share is 0.2043 in 93 trials and the corrupted input row is 0.7368; the ten-trial campaign could not have told it apart from 0.8000.
• Since the campaign looks only at harmful records, it produces one of the course’s two numbers; the widened rule brings the passing share down to 0.0860 while raising false alarm from 23 to 30, and this number is not in the report.
• The trial count of a class with no row in the report is zero and its passing share is unknown; unless covered and uncovered are written separately, a gap looks like a success.

Next Step

In eight lessons the surface was counted. The text entering the model was divided into classes, distributed by its source, swept with a threshold, records carrying hidden field names were counted, the side-effect tool request was separated, the output side was measured, corruption that carries no meaning was added as a separate class, and in the end the scope itself turned into a number. In all of this the defense was always the same thing: a single filter. Sometimes a list of four rules, sometimes a list of five rules widened with a flag, sometimes a score with a threshold. The work it did never changed — it either passed the record or blocked it, and it always paid the cost in the same currency: false alarm. Yet blocking is not the only move of a defense. Deleting a field, not counting a text as an instruction, narrowing the authority of a tool, and asking a human before an action are defenses too, and none of them stops the record. The next topic builds these layers one by one and measures each with its own cost: what it catches, what it blocks by mistake, and which capability it takes away.