Unit 01 of 03
Attack Surface
Separating the attack surface into classes that carry a structural attribute profile rather than a working payload, and counting the share the defense catches in each class: zero blocked harmful and zero false alarm in the undefended baseline, writing together the harmful records the rule filter blocks and the clean records it blocks by mistake, separating the direct and indirect forms by source distribution, recall falling as precision rises in a threshold sweep and the threshold at which the easiest point of evasion appears, in records carrying hidden field names what masking removes versus what the filter catches versus the residue nothing catches, among records carrying side-effect tool names how many of those sent to the approval gate are truly harmful, the harmful share per source class and what filtering by source adds, the number of responses carrying a hidden field on the output side, the point in the adversarial class where recall falls relative to the other classes, and in red teaming the number of covered classes versus the passing share and the count of an untested class being unknown rather than zero.
Published Updated
Lessons
- 01 Prompt Injection Counting the text that enters a model as an attack surface: in the fictional record set, 93 of 240 records are harmful and 147 are clean, and being harmful lives not in the text but in a structural attribute profile. In the undefended baseline, true positives are 0 and false alarms are 0; the rule-based filter blocks 74 harmful records, wrongly blocks 23 clean ones, and lets 19 harmful ones escape (precision 0.7629, recall 0.7957, false block rate 0.1565). The direct and indirect forms are separated by source distribution: recall is 0.6486 on 37 direct harmful records and 0.8929 on 56 indirect ones. In the second run the same filter gives 0.6988; the resolution of the set is 0.0108 on recall and 0.0068 on false blocks. 14 min read
- 02 Jailbreak Techniques Measuring guardrail bypass as a threshold question, not a procedure: on the same 240-record fictional set, as the threshold rises from 0.30 to 0.80, precision climbs from 0.7568 to 0.9787, recall falls from 0.9032 to 0.4946, and the false block rate drops from 0.1837 to 0.0068. Raising the threshold is a one-to-one trade up to 0.50 (7 for 7, 11 for 12) and 2.33 and 3.25 above it; this is the region most open to bypass. Per-class recall is read separately, and the first class to collapse is corrupted input (0.6842 to 0.2105). In the second run, recall is unchanged at thresholds 0.40 and 0.60 while the decision of 13 and 9 records changes. 13 min read
- 03 Data Exfiltration Measuring system instruction disclosure and sensitive data disclosure by counting alone: in the 240-record fictional set, 32 records carry a hidden field name (23 harmful, 9 clean), split across field classes as access string 8, contact 10, subscriber identity 9, address 5. The exfiltration surface is 35 records, 26 harmful. The full rule filter blocks 25 and masking adds zero; with the hidden field rule removed, the narrow filter blocks 16 and masking removes the field from 9. In both setups the residue that none catches is 1 record: it carries no flag, scores 0.0088, and is not flagged even at the lowest threshold. Measured on its own as a decision, masking gives precision 0.7188, recall 0.2473, and a false block rate of 0.0612; in the second run the residue drops to 0. 14 min read
- 04 Tool Misuse Counting agent authority on the defense side: of 240 records, 27 carry the side-effect tool name flag; the no-gate baseline sends 0 requests to a human, the gate sends these 27 and 24 turn out to be harmful, so the gate has precision 0.8889 and recall 0.2581. The gate sees 8 of the 10 privilege escalation records and 1 of the 17 hidden field request records. A gate widened by role mode raises precision to 0.9138 and recall to 0.5699; a gate widened by source drops precision to 0.5586 and sends 49 clean records to a human. A gate placed on top of the rule filter moves none of the four numbers: the rule already flagged all 27 records going to the gate. Across two runs the gate's precision comes out 0.8889 and 0.6471. 13 min read
- 05 Training Data Poisoning Poisoning counted as the harmful share per source class, not as a procedure: 93 of 240 records in the unfiltered corpus are harmful, a share of 0.3875. The share is 0.2701 for the user, 0.5417 for the retrieved document, 0.5484 for the tool output, yet volume reverses the order: the lowest-share channel adds 37 harmful records and the highest adds 17. The source gate lowers the corpus share to 0.2701 and loses 47 clean records; the rule gate lowers it to 0.1329 and loses 23. Using both gives 0.1383, a difference below the 0.0106 step of a 94-record corpus, so unmeasurable, while lost clean records rise from 23 to 66. Rule filter recall is 0.6486 on the user channel and 0.9487 on retrieved documents. Shares move across two runs; the ordering does not. 13 min read
- 06 Model Inversion and Inference Attacks Leakage counted on the output side, not the input side: 71 of 240 responses carry a field of a secret field class, 106 fields in total. 31 of them come from the 32 records that asked for a field, and 40 come from the 208 records that never asked, so the model inversion class is larger than what was requested. The four-name output filter removes all 106 fields and the remaining secret field count is 0; yet 47 responses still carry an indirect identifier outside the secret field class, 57 responses still carry a membership signal that has no field name, and responses carrying a trace fall only from 137 to 94. A five-name list also removes the indirect identifier but cannot remove the membership signal. 58 of the 106 removed fields come from clean records, and 41 clean responses lose a field. Across two runs the field count is 106 and 104, and responses carrying a trace after the filter are 94 in both. 13 min read
- 07 Adversarial Examples Deliberate corruption of the input counted as a separate surface: the same rule filter gives recall between 1.0000 and 0.8696 in four classes but drops to 0.2632 in the corrupted input class, and four of the five records it catches there are caught because they carry a feature of another class. Adding the unusual token distribution flag to the rule raises class recall to 0.8421 and set recall from 0.7957 to 0.9140; the same step lowers precision from 0.7629 to 0.7391 and moves the false block rate from 0.1565 to 0.2041. The model-based filter stays at 0.4737 in the same class and moves between 0.4737 and 0.4211 across two runs; since this equals the class resolution of 0.0526, it is not counted as a finding. 9 min read
- 08 AI Red Teaming Red teaming treated not as an attack procedure but as scope and counting: a narrow campaign sweeping four classes with ten trials each reports 4 passing records in 40 trials and a passing share of 0.1000; when a fifth class is added to the same campaign the count becomes 12 passing in 50 trials and 0.2400, and the added class alone supplies 8 of them. Changing the trial selection gives 0.1800 at the same size, all harmful records give 0.2043, and the widened rule brings it down to 0.0860. A class with no row in the report has zero trials and its passing share is unmeasured; the scope of a campaign is a statement about the plan itself, not about the world. The lesson closes the topic and writes that the defense of eight lessons was a single filter. 13 min read