--- title: 'Evaluation Set Design' source: 'https://academia.sh/en/courses/ai-safety-and-evaluation/evaluation-set-design' course: 'AI Safety and Evaluation' language: en updated: '2026-10-21T05:02:28+00:00' license: 'CC BY-SA 4.0' objectives: - 'Runs the same defense on evaluation sets of three different compositions and obtains three different numbers.' - 'Distinguishes with numbers that precision depends on the harmful share of the set while recall is independent of it.' - 'Computes the resolution of a set from the number of harmful and clean records and does not count differences below that step as findings.' --- # Evaluation Set Design Reading the number of a defense together with the composition of the set: the same rule-based filter, unchanged, gives three separate numbers on three sets. Precision comes out at 0.7629 on the representative set, 0.9136 on the harm-weighted set and 0.4444 on the borderline example set; recall stays exactly the same on the first two sets because the harmful subset did not change at all. On the borderline example set recall falls from 0.7957 to 0.6400 and the false block rate rises to 0.7143. The resolution of the set is written down too: the step of recall is 0.0108 on the representative set and 0.0400 on the borderline example set, and no difference smaller than that step can be measured. The fourteen lessons of this course built the defense. Input and output filters were written, content moderation was measured class by class, fields were masked in the data going to the model, tool permissions were narrowed, retrieved content was marked as data rather than instruction, and a human approval gate was placed in front of irreversible actions. For each layer, the share it stopped and the share it stopped by mistake were written side by side. One question was never asked: **where** do these numbers come from. Every ratio was computed on a set of records, and how that set was collected was never discussed. This lesson treats that set as a design object and makes a single claim: **the number of a defense is the number of the set as much as it is the number of the defense.** The defense will not change at all; only the composition of the set will, and three different ratios will be read from the same filter. - **EM1.** The record set, the classes and the defenses are **fictional** and are generated inside the lesson. There is no real institution, product or system. - **EM2.** An attack is **not a text but a feature profile**. The text of the records is entirely harmless support requests and has **no relation** to the class; the class lives only in the structural flags. For this reason no working attack procedure exists in the lesson. - **EM3.** The measure is not a single ratio: true positives (blocked), false positives, false negatives, precision, recall and the **false block rate** are written together. - **EM4.** The baseline on every set is the **undefended baseline**: no filter, true positives (blocked) 0 and false positives 0. - **EM5.** The three sets are derived **from the same pool** and the defense never changes. The **representative set** is the whole pool; the **harm-weighted set** is all the harmful records plus every third clean record; the **borderline example set** is the records that carry exactly one flag. - **EM6.** The borderline example criterion is **structural** and looks at the decision of no filter; taking the selection criterion from the output of a defense would mean measuring that defense on the set it chose itself. - **EM7.** The **resolution** of a set is the smallest difference a single record can move. The step of recall is the inverse of the number of harmful records, and the step of the false block rate is the inverse of the number of clean records. - **EM8.** The rule-based filter reads only flags and is **independent of the run**; the score of the model-based filter depends on the seed, and the second run is taken with seed 20260219. ## The Set Itself Is a Design Decision The pool is two hundred and forty records. Each record has a class, a source and a profile made of seven structural flags. The defense reads these flags, not the text. The first block builds the pool, separates the three sets and runs the same rule-based filter on all three. ```python # FICTIONAL. Records, classes and defenses are generated inside the lesson; no real # institution, product or system exists. An attack is not a TEXT, it is a FEATURE PROFILE. SEED, M32 = 20260218, 0xFFFFFFFF def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def trace_id(m): h = 2166136261 for c in str(m): h = ((h ^ ord(c)) * 16777619) & M32 return h CLASSES = ["clean", "indirect_instruction", "hidden_field_request", "privilege_escalation", "source_spoofing", "corrupted_input"] SOURCES = ["user", "retrieved_document", "tool_output"] FEATURES = ["instruction_mode", "role_change_request", "hidden_field_name", "side_effect_tool_name", "source_stamp_mismatch", "unusual_stamp_distribution", "excessive_length"] PROFILE = { "clean": [0.06, 0.02, 0.05, 0.03, 0.02, 0.04, 0.05], "indirect_instruction": [0.88, 0.42, 0.10, 0.30, 0.35, 0.08, 0.12], "hidden_field_request": [0.35, 0.15, 0.86, 0.12, 0.10, 0.06, 0.08], "privilege_escalation": [0.30, 0.25, 0.12, 0.84, 0.14, 0.05, 0.07], "source_spoofing": [0.22, 0.55, 0.08, 0.10, 0.82, 0.07, 0.06], "corrupted_input": [0.05, 0.03, 0.04, 0.03, 0.06, 0.80, 0.62], } SOURCE_SHARE = { "clean": [0.72, 0.20, 0.08], "indirect_instruction": [0.05, 0.70, 0.25], "hidden_field_request": [0.55, 0.30, 0.15], "privilege_escalation": [0.40, 0.35, 0.25], "source_spoofing": [0.10, 0.60, 0.30], "corrupted_input": [0.65, 0.20, 0.15], } # Fictional, entirely harmless support texts. They have NO relation to the class. TEXT = ["bu ayki fatura tutari gecen aya gore yuksek geldi", "sayac endeksi yanlis girilmis olabilir", "mahallede kesinti ne zaman bitecek", "tarife basamagimi ogrenmek istiyorum", "ariza kaydim icin ekip ne zaman gelir", "adres degisikligi nasil yapilir"] RECORDS = [] for i in range(240): r = rng(SEED + 6151 * i) c = CLASSES[pick(r(), [0.62, 0.10, 0.07, 0.07, 0.07, 0.07])] source = SOURCES[pick(r(), SOURCE_SHARE[c])] feat = {o: (r() < p) for o, p in zip(FEATURES, PROFILE[c])} RECORDS.append({"no": i + 1, "class": c, "source": source, "features": feat, "text": TEXT[int(r() * len(TEXT))], "length": 40 + int(r() * 260)}) def harmful(k): return k["class"] != "clean" def rule_filter(k, rules=("instruction_mode", "hidden_field_name", "side_effect_tool_name", "source_stamp_mismatch")): return any(k["features"][o] for o in rules) def measure(records, decision): tp = fp = tn = fn = 0 for k in records: flagged, actual = decision(k), harmful(k) if flagged and actual: tp += 1 elif flagged and not actual: fp += 1 elif not flagged and actual: fn += 1 else: tn += 1 precision = tp / (tp + fp) if tp + fp else 0.0 recall = tp / (tp + fn) if tp + fn else 0.0 return {"tp": tp, "fp": fp, "fn": fn, "tn": tn, "precision": round(precision, 4), "recall": round(recall, 4), "false_block_rate": round(fp / (fp + tn), 4) if fp + tn else 0.0} def flag_count(k): return sum(1 for o in FEATURES if k["features"][o]) CLEAN = [k for k in RECORDS if not harmful(k)] HARMFUL = [k for k in RECORDS if harmful(k)] SETS = {"representative": RECORDS, # the whole pool "harm-weighted": HARMFUL + CLEAN[::3], # every harmful, every third clean "borderline example": [k for k in RECORDS if flag_count(k) == 1]} print(f"{'set':<18} {'records':>8} {'harmful':>8} {'clean':>6} {'harmful share':>13}") for name, K in SETS.items(): h = sum(1 for k in K if harmful(k)) print(f"{name:<18} {len(K):>8} {h:>8} {len(K) - h:>6} {h / len(K):>13.4f}") print("\nundefended baseline (no filter), on all three sets") for name, K in SETS.items(): t = measure(K, lambda k: False) print(f"{name:<18} true positives (blocked) {t['tp']} , false positives {t['fp']} , " f"false negatives {t['fn']}") print("\nrule filter , same defense on three sets") print(f"{'set':<18} {'tp':>4} {'fp':>4} {'fn':>4} {'precision':>10} {'recall':>11}" f" {'false block rate':>17}") for name, K in SETS.items(): o = measure(K, rule_filter) print(f"{name:<18} {o['tp']:>4} {o['fp']:>4} {o['fn']:>4} {o['precision']:>10.4f}" f" {o['recall']:>11.4f} {o['false_block_rate']:>17.4f}") ``` ``` set records harmful clean harmful share representative 240 93 147 0.3875 harm-weighted 142 93 49 0.6549 borderline example 53 25 28 0.4717 undefended baseline (no filter), on all three sets representative true positives (blocked) 0 , false positives 0 , false negatives 93 harm-weighted true positives (blocked) 0 , false positives 0 , false negatives 93 borderline example true positives (blocked) 0 , false positives 0 , false negatives 25 rule filter , same defense on three sets set tp fp fn precision recall false block rate representative 74 23 19 0.7629 0.7957 0.1565 harm-weighted 74 7 19 0.9136 0.7957 0.1429 borderline example 16 20 9 0.4444 0.6400 0.7143 ``` ## Unchanging Recall, Moving Precision The second table gives the baseline and it is the same on all three sets: when no filter is built, true positives (blocked) are 0 and false positives are 0. That the baseline is this plain is not a convenience but a constraint; every record the defense gains is counted from here, and every clean record the defense blocks is also written against it. The number of false negatives changes from set to set because the sets have different numbers of harmful records: ninety-three in the representative and the harm-weighted set, twenty-five in the borderline example set. In the three rows of the third table the filter is the same: the same four rules, the same flags, not a single line of code changed. Even so, precision reads **0.7629**, **0.9136** and **0.4444**. The largest gap between them is **0.4692**, which means that when the "precision" of a defense is stated alone, almost nothing has been said. The first two rows make the source of this gap clear. On the harm-weighted set recall is **0.7957**, **exactly the same** as on the representative set. This is not surprising: the harmful subset of the two sets is the same ninety-three records, and all that was thrown away is clean records. Recall looks only at the harmful records, so diluting the clean side cannot move it. Precision, on the other hand, is the ratio `tp / (tp + fp)`, and its denominator shrinks together with the number of clean records: when ninety-eight clean records were thrown away, false positives dropped from twenty-three to seven and precision rose by **0.1507**. The filter did not catch one more record; only the place where it could be wrong narrowed. This is the source of the most common mistake made with evaluation sets. Collecting a harm-weighted set is a natural choice, because enough examples are needed from the rare classes. But the precision measured on that set is **not the precision in production**, and the gap between the two comes not from the quality of the defense but from the harmful share of the set. The rule is this: **precision cannot be reported without writing the harmful share of the set.** The false block rate falls from 0.1565 to 0.1429 between the two sets. The size of this difference is **0.0136**, and as will be seen shortly, it is below the resolution of the harm-weighted set; it is not a finding, it is a residue of the dilution. ## Borderline Examples and the Resolution of the Set The third set asks a different question: what does the defense do on **hard records**. Records carrying exactly one flag can be both clean and harmful; the evidence is thin. On this set the recall of the rule-based filter is **0.6400**, **0.1557** lower than on the representative set, and its precision is 0.4444, which means more than half of the records it flags are clean. The false block rate is **0.7143**: three quarters of the clean records in this set are blocked. What the defense breaks shows itself here. The second block runs the same three sets with the model-based filter, then writes the resolution of each set and the difference between two runs. ```python WEIGHT = {"instruction_mode": 0.42, "role_change_request": 0.26, "hidden_field_name": 0.44, "side_effect_tool_name": 0.40, "source_stamp_mismatch": 0.38, "unusual_stamp_distribution": 0.30, "excessive_length": 0.12} SOURCE_BONUS = {"user": 0.00, "retrieved_document": 0.18, "tool_output": 0.12} def score(k, seed=SEED): p = sum(WEIGHT[o] for o in FEATURES if k["features"][o]) + SOURCE_BONUS[k["source"]] r = rng(seed ^ trace_id(k["no"])) return min(max(p + (r() - 0.5) * 0.20, 0.0), 1.6) def model_filter(k, threshold=0.40, seed=SEED): return score(k, seed) >= threshold print("model filter (threshold 0.50) , same defense on three sets") print(f"{'set':<18} {'tp':>4} {'fp':>4} {'fn':>4} {'precision':>10} {'recall':>11}" f" {'false block rate':>17}") for name, K in SETS.items(): o = measure(K, lambda k: model_filter(k, 0.50)) print(f"{name:<18} {o['tp']:>4} {o['fp']:>4} {o['fn']:>4} {o['precision']:>10.4f}" f" {o['recall']:>11.4f} {o['false_block_rate']:>17.4f}") print("\nresolution of the set: the smallest difference one record moves") print(f"{'set':<18} {'harmful':>8} {'clean':>6} {'recall step':>17}" f" {'false block rate step':>23}") for name, K in SETS.items(): h = sum(1 for k in K if harmful(k)) print(f"{name:<18} {h:>8} {len(K) - h:>6} {1 / h:>17.4f} {1 / (len(K) - h):>23.4f}") print("\ntwo runs , representative set") print(f"{'defense':<30} {'run 1':>8} {'run 2':>9} {'diff':>8}") a = measure(RECORDS, rule_filter)["recall"] print(f"{'rule filter':<30} {a:>8.4f} {a:>9.4f} {0.0:>+8.4f}") b1 = measure(RECORDS, lambda k: model_filter(k, 0.50, SEED))["recall"] b2 = measure(RECORDS, lambda k: model_filter(k, 0.50, SEED + 1))["recall"] print(f"{'model filter (threshold 0.50)':<30} {b1:>8.4f} {b2:>9.4f} {b2 - b1:>+8.4f}") print(f"measured: recall . step 1/93 = {1 / 93:.4f} , diff {abs(b2 - b1) / (1 / 93):.0f} steps") ``` ``` model filter (threshold 0.50) , same defense on three sets set tp fp fn precision recall false block rate representative 66 8 27 0.8919 0.7097 0.0544 harm-weighted 66 2 27 0.9706 0.7097 0.0408 borderline example 10 5 15 0.6667 0.4000 0.1786 resolution of the set: the smallest difference one record moves set harmful clean recall step false block rate step representative 93 147 0.0108 0.0068 harm-weighted 93 49 0.0108 0.0204 borderline example 25 28 0.0400 0.0357 two runs , representative set defense run 1 run 2 diff rule filter 0.7957 0.7957 +0.0000 model filter (threshold 0.50) 0.7097 0.7419 +0.0322 measured: recall . step 1/93 = 0.0108 , diff 3 steps ``` The model-based filter repeats the same pattern and breaks down harder in one place. On the representative set its precision is **0.8919**, clearly higher than the **0.7629** of the rule-based filter; someone choosing would take the model-based filter. On the borderline example set its recall is **0.4000**, below the **0.6400** of the rule-based filter. In other words, the ranking of the two defenses **changes places depending on which set you look at**. This lesson does not choose a defense; it shows that the choice is given together with the set. The resolution table writes the lower bound of the measurement band. On the representative set recall can move only in steps of **0.0108**, because there are ninety-three harmful records and when the decision on one record changes, the ratio moves by exactly this much. On the borderline example set the same step is **0.0400**: a recall measured with twenty-five harmful records cannot show any improvement smaller than four records. The hard set is also the **coarsest** set, and this is not a contradiction but a direct consequence of the sample size. The two-run table gives the second source of the band: because the rule-based filter reads only flags, it gives **exactly the same** number in both runs, while the recall of the model-based filter moves from 0.7097 to 0.7419, that is **three steps**. In a defense where a model is involved, no difference smaller than three steps can be attributed to a change. From here the rule of the course is read once more, and this lesson adds a clause to it: **a defense that does not write together the share it stops and the share it stops by mistake is counted as unmeasured**; even if both are written, the number cannot be read unless **the set on which it was measured and the step of that set** are written. ## Summary - With the defense unchanged and only the composition of the set changed, three separate precisions were read from the rule-based filter: 0.7629 on the representative set, 0.9136 on the harm-weighted set, 0.4444 on the borderline example set. - Recall stayed **exactly the same** on the first two sets (0.7957), because the harmful subset did not change; the only thing that moved was the number of clean records. Precision cannot be reported without writing the harmful share of the set. - On the borderline example set the cost of the defense becomes visible: recall falls to 0.6400 and the false block rate rises to **0.7143**, that is, three quarters of the clean records in that set are blocked. - The ranking of the two defenses changes places depending on the set: the model-based filter has the higher precision on the representative set (0.8919) and the lower recall on the borderline example set (0.4000). - The resolution of the set is the lower bound of the measurement band: the step of recall is 0.0108 on the representative set and 0.0400 on the borderline example set. While the rule-based filter gives the same number in two runs, the recall of the model-based filter moves three steps. ## Next Step In this lesson the correctness of every decision was determined by looking at the true class of the record, and that class was taken as given. In production there is no such column: someone has to **say** whether a newly arrived record is harmful. There are two ways to do this; one is to write structural criteria, the other is to ask a language model. The next lesson puts these two ways side by side and tests both with the same two questions: how well the label they give agrees with the label a human gives, and how many decisions change when the same record is asked **twice**.