Actions Requiring Human Approval

The previous lesson counted that a defense that does not block neutralizes 42 of the 93 harmful records for free, and showed its wall: it does not touch the remaining 51. Among them are the records that carry a privilege escalation and a hidden field request, and the target of these two classes is most often the same place — an operation that cannot be taken back once it happens.

An irreversible action is an action whose result cannot be corrected afterwards: a record written, a notification sent, information that has gone outside. For such an action a precision of 0.8889 is not enough, because the cost of a wrong decision is asymmetric. This lesson opens the gate to a human and counts three things: how many of the records sent to a human are really harmful, what happens as the gate widens, and what is left of the gate as the human’s attention decreases.

• DF58. The code is a simulator; no real tool or approval interface is called. The record set, the classes and the features are fictional, the seed is 20260218, the second run 20260219.
• DF59. An attack is not a text but a feature profile. The six sentences the records carry are ordinary support requests and have no relation to the class. There is no working payload, template or instruction string in the code.
• DF60. The baseline is the undefended setup: records sent to a human 0, harmful caught 0, unnecessary interruption 0.
• DF61. The rule of the course: a defense that does not write the share it stopped and the share it stopped by mistake together counts as unmeasured.
• DF62. The irreversible action request is represented in this set by the side_effect_tool_name flag; the approval_gate function of the shared definition looks at exactly this flag.
• DF63. The gate does not block, it asks. If approval is given the action proceeds, if not it does not. The cost of the gate is therefore not a blocked task but an interruption: every record sent to a human creates a wait, and some are unnecessary.
• DF64. The review order of the human is not related to harm. The order is fictional and differs between the two runs; this is the only source of the run band of the gate.
• DF65. Attention is a budget. The human can really review at most 27 prompts; prompts outside the budget are approved unseen. The budget is fictional, set equal to the prompts of the narrow gate.
• DF66. Resolution: the step of recall is $1/93 = 0.0108$, the step of the false block rate is $1/147 = 0.0068$; no difference smaller than these steps counts as a finding.

Shared Definition

The record set, the classes, the defense layers and the measure are the same throughout the course. In this lesson two defenses of the shared definition are used: the rule filter and the human approval gate.

SEED, M32 = 20260218, 0xFFFFFFFF


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def pick(u, w):
    t = 0.0
    for i, x in enumerate(w):
        t += x
        if u < t:
            return i
    return len(w) - 1


def trace_id(m):
    h = 2166136261
    for c in str(m):
        h = ((h ^ ord(c)) * 16777619) & M32
    return h


# ---- FICTIONAL records: no attack TEXT is written, an attack is a FEATURE PROFILE
CLASSES = ["clean", "indirect_instruction", "hidden_field_request", "privilege_escalation",
           "source_spoofing", "corrupted_input"]
SOURCES = ["user", "retrieved_document", "tool_output"]
# Features are structural; none of them is a working payload.
FEATURES = ["instruction_mode", "role_change_request", "hidden_field_name", "side_effect_tool_name",
            "source_stamp_mismatch", "unusual_stamp_distribution", "excessive_length"]
PROFILE = {
    "clean":                [0.06, 0.02, 0.05, 0.03, 0.02, 0.04, 0.05],
    "indirect_instruction": [0.88, 0.42, 0.10, 0.30, 0.35, 0.08, 0.12],
    "hidden_field_request": [0.35, 0.15, 0.86, 0.12, 0.10, 0.06, 0.08],
    "privilege_escalation": [0.30, 0.25, 0.12, 0.84, 0.14, 0.05, 0.07],
    "source_spoofing":      [0.22, 0.55, 0.08, 0.10, 0.82, 0.07, 0.06],
    "corrupted_input":      [0.05, 0.03, 0.04, 0.03, 0.06, 0.80, 0.62],
}
SOURCE_SHARE = {
    "clean":                [0.72, 0.20, 0.08],
    "indirect_instruction": [0.05, 0.70, 0.25],
    "hidden_field_request": [0.55, 0.30, 0.15],
    "privilege_escalation": [0.40, 0.35, 0.25],
    "source_spoofing":      [0.10, 0.60, 0.30],
    "corrupted_input":      [0.65, 0.20, 0.15],
}
# Fictional, entirely harmless support texts. They have NO relation to the class;
# the class information lives only in the feature profile.
TEXT = ["bu ayki fatura tutari gecen aya gore yuksek geldi",
        "sayac endeksi yanlis girilmis olabilir",
        "mahallede kesinti ne zaman bitecek",
        "tarife basamagimi ogrenmek istiyorum",
        "ariza kaydim icin ekip ne zaman gelir",
        "adres degisikligi nasil yapilir"]

RECORDS = []
for i in range(240):
    r = rng(SEED + 6151 * i)
    c = CLASSES[pick(r(), [0.62, 0.10, 0.07, 0.07, 0.07, 0.07])]
    source = SOURCES[pick(r(), SOURCE_SHARE[c])]
    feat = {o: (r() < p) for o, p in zip(FEATURES, PROFILE[c])}
    RECORDS.append({"no": i + 1, "class": c, "source": source, "features": feat,
                    "text": TEXT[int(r() * len(TEXT))],
                    "length": 40 + int(r() * 260)})


def harmful(k):
    return k["class"] != "clean"


# ---- DEFENSE 1: rule filter (structural, explainable)
def rule_filter(k, rules=("instruction_mode", "hidden_field_name", "side_effect_tool_name",
                          "source_stamp_mismatch")):
    return any(k["features"][o] for o in rules)


# ---- DEFENSE 5: human approval gate (irreversible actions)
def approval_gate(k, required=True):
    """A record that carries a side-effecting tool request is sent to a human."""
    return required and k["features"]["side_effect_tool_name"]


# ---- MEASURE: what was blocked and what was blocked by mistake are written together
def measure(records, decision):
    tp = fp = tn = fn = 0
    for k in records:
        flagged, actual = decision(k), harmful(k)
        if flagged and actual:
            tp += 1
        elif flagged and not actual:
            fp += 1
        elif not flagged and actual:
            fn += 1
        else:
            tn += 1
    precision = tp / (tp + fp) if tp + fp else 0.0
    recall = tp / (tp + fn) if tp + fn else 0.0
    return {"tp": tp, "fp": fp, "fn": fn, "tn": tn,
            "precision": round(precision, 4), "recall": round(recall, 4),
            "false_block_rate": round(fp / (fp + tn), 4) if fp + tn else 0.0}


print("records:", len(RECORDS), "- harmful:", sum(harmful(k) for k in RECORDS))
print("sent to approval gate:", sum(approval_gate(k) for k in RECORDS))

records: 240 - harmful: 93
sent to approval gate: 27

The Precision and the Width of the Gate

The AI Agents course measured the approval point in the agent loop: with approval off, 3 wrong records are written, with approval on, 1; the loop had 7 approval points. That procedure is not repeated; its numbers are taken as input. The question here is different — what the gate gives when it is measured like a classifier.

The width of the gate is a design decision: which flags a record must carry to be asked of a human. Four widths are tried; the narrowest is the approval_gate function of the shared definition, the widest is the rule set of the rule filter itself. The difference lies in the kind of defense: the filter blocks, the gate asks.

RULE_SET = ("instruction_mode", "hidden_field_name", "side_effect_tool_name",
            "source_stamp_mismatch")
GATE = {"narrow":  ("side_effect_tool_name",),
        "medium":  ("side_effect_tool_name", "hidden_field_name"),
        "wide":    ("side_effect_tool_name", "hidden_field_name", "source_stamp_mismatch"),
        "widest":  RULE_SET}
BUDGET = 27


def asks(k, fields):
    """The gate does not block; it sends the record to a human and waits for the answer."""
    return any(k["features"][o] for o in fields)


def review_order(records, seed):
    """The order in which the human looks is not related to harm; it is a fictional order."""
    return sorted(records, key=lambda k: rng(seed ^ trace_id(k["no"]))())


print("narrow gate is the approval gate of the shared definition:",
      all(asks(k, GATE["narrow"]) == approval_gate(k) for k in RECORDS))
print("widest gate is the rule set of the rule filter:",
      all(asks(k, GATE["widest"]) == rule_filter(k) for k in RECORDS))
print("baseline: sent to a human 0 , harmful caught 0 , unnecessary interruption 0\n")
print("gate      prompts  harmful precision      recall  needless reviewed   A shareA   B shareB")
for name, fields in GATE.items():
    s = measure(RECORDS, lambda k, fields=fields: asks(k, fields))
    q = [k for k in RECORDS if asks(k, fields)]
    n = min(BUDGET, len(q))
    a = sum(harmful(k) for k in review_order(q, SEED)[:n])
    b = sum(harmful(k) for k in review_order(q, 20260219)[:n])
    print(f"{name:<9} {s['tp'] + s['fp']:>7} {s['tp']:>8} {s['precision']:>9.4f}"
          f" {s['recall']:>11.4f} {s['fp']:>9} {n:>8} {a:>3} {a / 93:.4f} {b:>3} {b / 93:.4f}")

q = [k for k in RECORDS if approval_gate(k)]
print("\nnarrow gate as the review share falls (27 prompts)")
print(" review share reviewed     caught A shareA     caught B shareB passed harmful")
for p in (1.00, 0.80, 0.60, 0.40, 0.20, 0.00):
    n = int(p * len(q))
    a = sum(harmful(k) for k in review_order(q, SEED)[:n])
    b = sum(harmful(k) for k in review_order(q, 20260219)[:n])
    print(f"{p:>13.2f} {n:>8} {a:>12} {a / 93:>6.4f} {b:>12} {b / 93:>6.4f} {93 - a:>14}")

narrow gate is the approval gate of the shared definition: True
widest gate is the rule set of the rule filter: True
baseline: sent to a human 0 , harmful caught 0 , unnecessary interruption 0

gate      prompts  harmful precision      recall  needless reviewed   A shareA   B shareB
narrow         27       24    0.8889      0.2581         3       27  24 0.2581  24 0.2581
medium         56       44    0.7857      0.4731        12       27  22 0.2366  18 0.1935
wide           78       62    0.7949      0.6667        16       27  22 0.2366  19 0.2043
widest         97       74    0.7629      0.7957        23       27  23 0.2473  17 0.1828

narrow gate as the review share falls (27 prompts)
 review share reviewed     caught A shareA     caught B shareB passed harmful
         1.00       27           24 0.2581           24 0.2581             69
         0.80       21           18 0.1935           18 0.1935             75
         0.60       16           13 0.1398           13 0.1398             80
         0.40       10            8 0.0860            8 0.0860             85
         0.20        5            4 0.0430            4 0.0430             89
         0.00        0            0 0.0000            0 0.0000             93

The Cost of the Gate Is Interruption

The first row is the precision of the gate: 24 of the 27 records sent to a human are really harmful, precision 0.8889. This is among the highest precisions measured in this course, for a narrow reason — the gate looks at a single flag that clean traffic almost never visits. Its recall is only 0.2581; the gate is not a filter and does not try to be one.

The cost column is where the gate parts from the filter. The gate blocks no record, it asks; if approval is given the action proceeds. The cost is therefore not a blocked task but 27 interruptions, and 3 of them are unnecessary. The previous lesson gives the comparison: had the permission gate blocked the same 27 records, the completed task would have fallen from 1.0000 to 0.9796, that is, three clean tasks would have been lost. When asked, none is lost. The gate turns one false block into one unnecessary interruption; whether this is a good exchange depends on the waiting being cheaper than the lost task, and this is not a number but a decision.

A consequence is that the gate adds nothing in coverage. Its flag is one of the four rules of the rule filter, so the 27 records sent to a human are a subset of the 97 records the filter flags. If the filter is in place, the gate catches no harmful record for the first time; it applies another operation to those 27 records. Where the classifier’s precision is stuck at 0.8889, handing the decision to a human may be the only remedy for an irreversible action — but this is not a gain in recall and reporting it as one would be wrong.

The Backfire of Widening

The next three rows widen the gate. The recall column brings good news: 0.2581 rises to 0.4731, 0.6667 and 0.7957. But this column assumes unlimited attention — that the human really looks at all 97 prompts. The reviewed column removes that assumption: the budget is 27 prompts, and in a widened gate more than half of the prompts are approved without being reviewed at all.

The columns of the harmful really caught reverse the result. The narrow gate catches 24 in both runs; the medium gate 22 and 18, the wide gate 22 and 19, the widest gate 23 and 17. Under a fixed attention budget no widening passes the 24 of the narrow gate, in either run. In the same step the unnecessary interruptions rise from 3 to 23. This is the evidence this lesson contributes to the second claim of the course: widening the gate, with attention fixed, lowers what is caught and multiplies the interruptions.

The run band limits the measure here. The narrow gate gives the same number in both runs, its band is zero. In the widened gates the difference between the two runs is 4, 3 and 6 records, that is, 0.0430, 0.0323 and 0.0645. The differences among 22, 22, 23 in the first run are below this band and do not count as a finding: which widening is better has not been measured. The only thing measured is that all of them stay below the narrow gate.

As the Approval Rate Falls

The last table sweeps the quietest failure of the gate. An approval gate works on the day it is set up; then the prompts pile up, most turn out harmless, and in time they are passed by looking away. This is not a defect but a setting, and it can be swept: the review share is how much of the incoming prompts is really read.

As the share goes from 1.00 to 0.20, the harmful caught falls from 24 to 4, the recall from 0.2581 to 0.0430; the harmful passed rises from 69 to 89. In the row where the share is zero the gate is still in place, still produces 27 prompts, and the harmful it catches is 0 — the same number as the undefended baseline, with 27 interruptions added on top. The fall is nearly linear, because the review order is not related to harm: a gate that looks away misses the harmful at random. The two runs give the same number in every row, because only 3 of the 27 prompts are unnecessary and even when the order changes the harmful share of the reviewed slice stays the same.

A single rule follows: the recall of an approval gate is the joint number of the gate and the attention given to it. The number 0.2581 is true only when the review share is 1.00. If the number of prompts and the observed review share are not written together, the reported recall is not measured but assumed.

Summary

• The gate for irreversible actions does not block, it asks; its cost is not a blocked task but an interruption.
• The narrow gate sends 27 records to a human and 24 are really harmful: precision 0.8889, recall 0.2581, unnecessary interruption 3; on the undefended baseline both are 0. These 27 records are a subset of the filter’s 97, so the gate adds nothing in coverage; it changes the operation.
• Had the same 27 records been blocked, the completed task would have fallen to 0.9796; when asked, no task is lost. The gate turns one false block into one unnecessary interruption.
• Under a fixed attention budget of 27 prompts, widening lowers what is caught: the narrow gate 24 in both runs, the widest 23 and 17; unnecessary interruptions rise from 3 to 23.
• The differences 22, 22, 23 among the widened gates are below the run band of 4 to 6 records and do not count as a finding.
• As the review share goes from 1.00 to 0.20, the harmful caught falls from 24 to 4 and the recall from 0.2581 to 0.0430; the recall of a gate is the joint number of the gate and the attention.

Next Step

This lesson closes the defense layers. In fourteen lessons the attack surface was first counted by its classes, then six defense layers were built, each with its cost written beside the share it stopped: for the filter the false alarm, for content moderation the hidden class, for masking the lost field, for the permission gate the dropped task, for marking the single lost path, for the approval gate the interruption. In every row two numbers stood side by side.

But in none of the fourteen lessons was this asked: how do we know these numbers are right. All were measured on a single fictional set, with a single class distribution and a single source of labels. How much would the number move if the composition changed; who decides whether an output is good; do two evaluators give one record the same grade; is yesterday’s number still valid after a prompt or model change. The next topic leaves the defense and takes up the measurement itself, and starts from the beginning: what an evaluation set is built on, and how much the form it is built in moves the measure.