Skip to content
academia.sh
tr
Log in

Course Advanced

AI Safety and Evaluation

Published Updated

By the end of this course

Start course

01

Attack Surface

Separating the attack surface into classes that carry a structural attribute profile rather than a working payload, and counting the share the defense catches in each class: zero blocked harmful and zero false alarm in the undefended baseline, writing together the harmful records the rule filter blocks and the clean records it blocks by mistake, separating the direct and indirect forms by source distribution, recall falling as precision rises in a threshold sweep and the threshold at which the easiest point of evasion appears, in records carrying hidden field names what masking removes versus what the filter catches versus the residue nothing catches, among records carrying side-effect tool names how many of those sent to the approval gate are truly harmful, the harmful share per source class and what filtering by source adds, the number of responses carrying a hidden field on the output side, the point in the adversarial class where recall falls relative to the other classes, and in red teaming the number of covered classes versus the passing share and the count of an untested class being unknown rather than zero.

  1. 01 Prompt Injection Counting the text that enters a model as an attack surface: in the fictional record set, 93 of 240 records are harmful and 147 are clean, and being harmful lives not in the text but in a structural attribute profile. In the undefended baseline, true positives are 0 and false alarms are 0; the rule-based filter blocks 74 harmful records, wrongly blocks 23 clean ones, and lets 19 harmful ones escape (precision 0.7629, recall 0.7957, false block rate 0.1565). The direct and indirect forms are separated by source distribution: recall is 0.6486 on 37 direct harmful records and 0.8929 on 56 indirect ones. In the second run the same filter gives 0.6988; the resolution of the set is 0.0108 on recall and 0.0068 on false blocks.
  2. 02 Jailbreak Techniques Measuring guardrail bypass as a threshold question, not a procedure: on the same 240-record fictional set, as the threshold rises from 0.30 to 0.80, precision climbs from 0.7568 to 0.9787, recall falls from 0.9032 to 0.4946, and the false block rate drops from 0.1837 to 0.0068. Raising the threshold is a one-to-one trade up to 0.50 (7 for 7, 11 for 12) and 2.33 and 3.25 above it; this is the region most open to bypass. Per-class recall is read separately, and the first class to collapse is corrupted input (0.6842 to 0.2105). In the second run, recall is unchanged at thresholds 0.40 and 0.60 while the decision of 13 and 9 records changes.
  3. 03 Data Exfiltration Measuring system instruction disclosure and sensitive data disclosure by counting alone: in the 240-record fictional set, 32 records carry a hidden field name (23 harmful, 9 clean), split across field classes as access string 8, contact 10, subscriber identity 9, address 5. The exfiltration surface is 35 records, 26 harmful. The full rule filter blocks 25 and masking adds zero; with the hidden field rule removed, the narrow filter blocks 16 and masking removes the field from 9. In both setups the residue that none catches is 1 record: it carries no flag, scores 0.0088, and is not flagged even at the lowest threshold. Measured on its own as a decision, masking gives precision 0.7188, recall 0.2473, and a false block rate of 0.0612; in the second run the residue drops to 0.
  4. 04 Tool Misuse Counting agent authority on the defense side: of 240 records, 27 carry the side-effect tool name flag; the no-gate baseline sends 0 requests to a human, the gate sends these 27 and 24 turn out to be harmful, so the gate has precision 0.8889 and recall 0.2581. The gate sees 8 of the 10 privilege escalation records and 1 of the 17 hidden field request records. A gate widened by role mode raises precision to 0.9138 and recall to 0.5699; a gate widened by source drops precision to 0.5586 and sends 49 clean records to a human. A gate placed on top of the rule filter moves none of the four numbers: the rule already flagged all 27 records going to the gate. Across two runs the gate's precision comes out 0.8889 and 0.6471.
  5. 05 Training Data Poisoning Poisoning counted as the harmful share per source class, not as a procedure: 93 of 240 records in the unfiltered corpus are harmful, a share of 0.3875. The share is 0.2701 for the user, 0.5417 for the retrieved document, 0.5484 for the tool output, yet volume reverses the order: the lowest-share channel adds 37 harmful records and the highest adds 17. The source gate lowers the corpus share to 0.2701 and loses 47 clean records; the rule gate lowers it to 0.1329 and loses 23. Using both gives 0.1383, a difference below the 0.0106 step of a 94-record corpus, so unmeasurable, while lost clean records rise from 23 to 66. Rule filter recall is 0.6486 on the user channel and 0.9487 on retrieved documents. Shares move across two runs; the ordering does not.
  6. 06 Model Inversion and Inference Attacks Leakage counted on the output side, not the input side: 71 of 240 responses carry a field of a secret field class, 106 fields in total. 31 of them come from the 32 records that asked for a field, and 40 come from the 208 records that never asked, so the model inversion class is larger than what was requested. The four-name output filter removes all 106 fields and the remaining secret field count is 0; yet 47 responses still carry an indirect identifier outside the secret field class, 57 responses still carry a membership signal that has no field name, and responses carrying a trace fall only from 137 to 94. A five-name list also removes the indirect identifier but cannot remove the membership signal. 58 of the 106 removed fields come from clean records, and 41 clean responses lose a field. Across two runs the field count is 106 and 104, and responses carrying a trace after the filter are 94 in both.
  7. 07 Adversarial Examples Deliberate corruption of the input counted as a separate surface: the same rule filter gives recall between 1.0000 and 0.8696 in four classes but drops to 0.2632 in the corrupted input class, and four of the five records it catches there are caught because they carry a feature of another class. Adding the unusual token distribution flag to the rule raises class recall to 0.8421 and set recall from 0.7957 to 0.9140; the same step lowers precision from 0.7629 to 0.7391 and moves the false block rate from 0.1565 to 0.2041. The model-based filter stays at 0.4737 in the same class and moves between 0.4737 and 0.4211 across two runs; since this equals the class resolution of 0.0526, it is not counted as a finding.
  8. 08 AI Red Teaming Red teaming treated not as an attack procedure but as scope and counting: a narrow campaign sweeping four classes with ten trials each reports 4 passing records in 40 trials and a passing share of 0.1000; when a fifth class is added to the same campaign the count becomes 12 passing in 50 trials and 0.2400, and the added class alone supplies 8 of them. Changing the trial selection gives 0.1800 at the same size, all harmful records give 0.2043, and the widened rule brings it down to 0.0860. A class with no row in the report has zero trials and its passing share is unmeasured; the scope of a campaign is a statement about the plan itself, not about the world. The lesson closes the topic and writes that the defense of eight lessons was a single filter.

02

Defense Layers

Counting the cost of every defense layer together with the harmful it stops: stacking the rule-based filter and the model-based filter catches no new harmful record while it lowers precision and raises the false block rate; per-class precision and recall expose what a single average hides, and in at least one class the defense falls below the baseline; the records masking cleans against the tasks lost because it deletes a field the task needs; the harmful stopped against the tasks left incomplete as the permission set narrows; the number of records untrusted content marking neutralizes without blocking anything or producing a false alarm, and the harmful that remains; how many of the records sent to the human approval gate are really harmful, and how the harmful caught shrinks as the approval rate falls.

  1. 01 Input and Output Filters Counting the harmful records that rule-based and model-based checks block on the same record set together with the clean records they block by mistake: when the rule-based filter is split into an input side and an output side, one blocks 55 harmful records and the other 44, and their union rises to 74; the model-based filter at a 0.50 threshold blocks fewer harmful records but blocks far fewer clean ones; stacking the two is worse than the rule-based filter alone — recall stays at 0.7957, precision drops from 0.7629 to 0.7551, the false block rate rises from 0.1565 to 0.1633, and the layer blocks one extra clean record without catching a single new harmful one.
  2. 02 Content Moderation Counting that blocking harmful content and giving it a class name are two separate jobs: a moderation step that produces a class name gives 0.9355 recall and 0.7311 precision in the binary reading, but per class the precision spreads between 0.2917 and 0.6364 and the recall of the class label between 0.3333 and 0.8235; in the indirect instruction class the moderation step puts the label on 21 records and knows 8 of them correctly, and it marks 9 entirely clean records with the same label, so when the class label is tied to an action the action fires wrongly far more often than rightly and the defense falls below the baseline in that class; the recall of the corrupted input class rises from 0.2632 to 0.8421 while the false block rate rises from 0.1565 to 0.2177.
  3. 03 Personal Data Masking Reading the two numbers of reduction in the data that goes to the model together: flag-based masking removes a field from 32 of 240 records, and 14 of them belong to the hidden field request class and 9 are entirely clean records; the same policy makes 23 tasks that require the field unsolvable, and 7 of those 23 tasks belong to clean subscribers; 3 records of the hidden field request class do not carry the flag and so escape masking, the narrow policy raises the escaped count to 14 while lowering the broken tasks to 8, and the policy that masks every record misses no request but breaks 169 tasks, 104 of which belong to clean subscribers. Because masking blocks no record it has no false alarm, and reading it like a filter is misleading.
  4. 04 Permissions and Sandboxing Counting tool permission as a defense layer: the same fictional set of 240 records runs under five permission sets, and the harmful records stopped, the false alarms and the completed tasks are printed side by side. On the undefended baseline the harmful stopped is 0 and the false alarm is 0; closing the execution capability stops 12 harmful records, blocks no clean record and gives a precision of 1.0000; closing the side-effect capability gives 24 harmful against 3 false alarms; the allowlisted set stops 34 harmful and lowers the completed tasks to 0.8844, read-only stops 62 harmful and lowers them to 0.6667. The marginal precision of narrowing falls from 1.0000 to 0.4167: the last two steps block more than one clean record for every harmful record they stop. The class distribution shows the blind spot of the gate: 1.0000 of the privilege escalation class is stopped, only 0.0588 of the hidden field request class.
  5. 05 Untrusted Content Marking Counting a defense that blocks no record: text coming from a source other than the user is marked as data and is not counted as an instruction even if it carries the instruction mode. On the undefended baseline the harmful stopped is 0 and the false alarm is 0; marking neutralizes 42 of the 93 harmful records without blocking any record, 51 remain, and the false alarm is zero. Neutralization gathers in two classes: 0.9167 of indirect instruction, 0.8696 of source spoofing; 0.0000 in the hidden field request, privilege escalation and corrupted input classes. The real value of marking is that it lets the filter be tightened: the rule filter alone covers 0.7957 and blocks 23 clean records, marking with the 0.50 threshold of the model filter covers 0.7634 and blocks only 8 clean records. The cost is not a false alarm: 47 of the 103 marked records are clean and only 1 of them carries the instruction mode, so the path lost is a single record and exactly one step of the resolution.
  6. 06 Actions Requiring Human Approval Counting the precision and cost of a human approval gate on irreversible actions: the narrow gate sends 27 records to a human, 24 of them really harmful, precision 0.8889, recall 0.2581, 3 unnecessary interruptions. Widening raises recall to 0.7957 under unlimited attention, but under a fixed attention budget of 27 prompts the harmful caught falls to 23 and 17 in two runs and never passes the 24 of the narrow gate. As the review share goes from 1.00 to 0.20 the harmful caught falls from 24 to 4 and recall from 0.2581 to 0.0430. The cost is not a blocked task but an interruption: blocking the same 27 records would lose three clean tasks; asking loses none.

03

Evaluation and Monitoring

Measuring how we know the defense we built works and how it is monitored in production: the same defense giving three separate numbers on a representative, a harm-weighted and a borderline-example set, with the resolution of the set written down; the agreement of a model-as-judge with the human label read side by side with the judge's own difference between two runs under a criteria-based approach; inter-rater agreement changing with the scale and the guideline, and the class in which the disagreement concentrates; the number moved by prompt and model changes compared with the run band, and which change stays unmeasurable; how many records later each of three production indicators shows a degradation; the user feedback loop making the false alarm look smaller than it is because a blocked user cannot give a signal; and the measures a system card carries and does not carry, with every measure it does not carry remaining a claim.

  1. 01 Evaluation Set Design Reading the number of a defense together with the composition of the set: the same rule-based filter, unchanged, gives three separate numbers on three sets. Precision comes out at 0.7629 on the representative set, 0.9136 on the harm-weighted set and 0.4444 on the borderline example set; recall stays exactly the same on the first two sets because the harmful subset did not change at all. On the borderline example set recall falls from 0.7957 to 0.6400 and the false block rate rises to 0.7143. The resolution of the set is written down too: the step of recall is 0.0108 on the representative set and 0.0400 on the borderline example set, and no difference smaller than that step can be measured.
  2. 02 Automated Evaluation Measuring two ways of saying that a record is harmful: writing structural criteria and asking a language model. The criteria-based approach gives 0.7629 precision, 0.7957 recall and 0.8250 agreement with the human label, makes no calls and prints exactly the same number in two runs. The model-as-judge gives 0.7876 precision, 0.9570 recall and 0.8833 agreement; the entire gain comes from the corrupted input class, which the criteria never look at (1.0000 against 0.2632). The cost is 246 calls, 10779 input tokens and the judge's own uncertainty: between two runs the decision changes on 16 records, recall moves three steps, and the difference in false block rate between the two approaches stays far below this band.
  3. 03 Human Evaluation Measuring the human label the previous lesson rested on: three fictional raters looking at the same flag list give 0.8417 agreement on a binary scale, 0.7583 on a three-valued scale and 0.9306 with an instruction with examples. The instruction raises agreement by twenty-one steps, but the overlap of the majority label with the actual class stays at 0.8417 and the accuracy of unanimity falls from 0.9126 to 0.8744. Disagreement gathers in the harmful classes and in records with exactly one flag: 0.7368 in corrupted input, 0.0000 without flags. Measured against the majority label, the same rule-based filter reaches 1.0000 recall, because the label column shares its blind spot.
  4. 04 Regression Testing Measuring four changes against a fixed evaluation set and comparing every difference with the run band. Rewriting the instruction with the same meaning stays below the band; removing the format constraint drops recall to 0.7742 but does not move the false block rate; losing one trace record shows only in the per-class breakdown, where corrupted input recall falls from 1.0000 to 0.7895.
  5. 05 Production Monitoring Measuring how many records later three indicators show that a deployed defense has degraded: over a fictional stream of six hundred records the undefended baseline gives 0 blocks, 0 false alarms and 120 escaped harmful records, while the threshold 0.50 setup blocks 95 records, 11 of them false alarms, and lets 36 harmful ones through. After record three hundred and one two separate degradations begin: a traffic shift raises escaped harm from 36 to 84, a silent threshold change raises false alarms from 11 to 27. The escaped-harm proxy reports the wave after 20 records but never reports the threshold shift, the block rate sees the threshold after 29 records, and the label-dependent false alarm share gives 45 and 220 records over two runs. No indicator catches both degradations.
  6. 06 The Feedback Loop Measuring why the user signal comes back biased when it returns to evaluation: the threshold 0.50 setup blocks 74 records, 8 of them false alarms, and 27 harmful ones escape; 0.6296 of the escaped harm produces a complaint, but the covered share of the false alarms is 0.0000 with the appeal channel closed and 0.3750 with a narrow one, and only three of the six appeals are real false alarms. A threshold tuned for five rounds by the two numbers read from the signal drops from 0.50 to 0.30: false alarms rise from 8 to 27, the false block rate from 0.0544 to 0.1837, recall from 0.7097 to 0.9032, and the loop sees only the second. Total errors go from 35 to 36: the loop does not reduce errors, it changes who pays for them.
  7. 07 Responsible Release Measuring what the published number covers: the threshold 0.50 setup publishes a single recall number (0.7097), but per class the same defense spreads between 0.9583 and 0.2105, and the published number is 0.4992 above the weakest. A narrow system card ties 2 of nine commitments to a number, a wide card 6; the other 3 stay claims. This lesson closes this course and the AI Engineering curriculum, gathering twenty-one lessons in one table.

Final exam

AI Safety and Evaluation — Final Exam

In preparation

This course's question bank is not complete yet. The exam opens here automatically once enough questions are added.

Start typing to search.

↑↓ ↵ Esc navigate · open · close