Tool Misuse

In three lessons the attack surface was read through the text entering the model: whether a record carrying the instruction mode came directly or from an indirect source, how far the threshold held in the guardrail bypass class, and the residue left after masking in records carrying a hidden field name. In all three the model was a reader: it took the text, produced a response, and the job ended there.

A model that can call tools is not only a reader. If it can write a record, close a request, or assign a field crew, what the model holds is not text but authority, and spending that authority in the wrong place is a surface independent of how the text is read. This lesson counts that surface: how many records carry a side-effect tool name, how many of them the human approval gate sends to a human, how many of those sent are actually harmful, and what the precision of the gate is.

In the Model Context Protocol and Tool Ecosystem course, three definitions reported by an untrusted server were audited and gave six findings; the same audit also produced two findings on three trusted servers. That count was on the declaration side: it audited the party that says what a tool is. Here the declaration is treated as closed, and what is counted is the request itself. The definition audit is not repeated; its numbers are inputs.

• AS41. The record set, the classes, and the defenses are fictional; no real system, server, or tool is called. Seed 20260218.
• AS42. An attack is not a text but an attribute profile. The text of the records is entirely harmless support requests and has no relation to the class; the class lives only in the attribute profile. No working exploit, payload, or privilege escalation procedure is written.
• AS43. side_effect_tool_name is a flag, not a tool name. Which tool carries which side effect is not counted here; what is counted is whether a record carrying the flag goes to the gate.
• AS44. The baseline is the no-gate setup: no request goes to a human, blocked harmful is 0, false alarm is 0.
• AS45. The gate is not a block but a routing; its cost is not blocked traffic but the number of records going to a human.
• AS46. The gate’s decision does not rest on scoring, it looks at a single flag; so on the same set the measure is independent of the run. The second run redraws the set with seed 20260219.
• AS47. Resolution: with 93 harmful and 147 clean records, the step of recall is 0.0108 and the step of the false block rate is 0.0068. The gate’s precision is read from a base of 27 records, with a step of 0.0370.
• AS48. The sandbox, the principle of least privilege, and the allowlist procedure were measured in earlier curricula and are not repeated; this lesson adds only the count of the request.

Authority Is a Flag, Not a Text

The first block builds the set and the gate. Nothing in the record runs: the side_effect_tool_name field is True or False, and this flag says that the record carries a wish to turn into a side-effect tool call. Which tool the flag corresponds to is not written, because what is measured is not the tool but the gate. The text field is an ordinary support request and has no relation to the class.

# FICTIONAL. Records, classes and defenses are defined inside the lesson; no real
# system is called. No attack TEXT is written: an attack is a FEATURE PROFILE.
SEED, M32 = 20260218, 0xFFFFFFFF


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def pick(u, w):
    t = 0.0
    for i, x in enumerate(w):
        t += x
        if u < t:
            return i
    return len(w) - 1


CLASSES = ["clean", "indirect_instruction", "hidden_field_request", "privilege_escalation",
           "source_spoofing", "corrupted_input"]
SOURCES = ["user", "retrieved_document", "tool_output"]
# Features are structural; none of them is a working payload.
FEATURES = ["instruction_mode", "role_change_request", "hidden_field_name", "side_effect_tool_name",
            "source_stamp_mismatch", "unusual_stamp_distribution", "excessive_length"]
PROFILE = {
    "clean":                [0.06, 0.02, 0.05, 0.03, 0.02, 0.04, 0.05],
    "indirect_instruction": [0.88, 0.42, 0.10, 0.30, 0.35, 0.08, 0.12],
    "hidden_field_request": [0.35, 0.15, 0.86, 0.12, 0.10, 0.06, 0.08],
    "privilege_escalation": [0.30, 0.25, 0.12, 0.84, 0.14, 0.05, 0.07],
    "source_spoofing":      [0.22, 0.55, 0.08, 0.10, 0.82, 0.07, 0.06],
    "corrupted_input":      [0.05, 0.03, 0.04, 0.03, 0.06, 0.80, 0.62],
}
SOURCE_SHARE = {
    "clean":                [0.72, 0.20, 0.08],
    "indirect_instruction": [0.05, 0.70, 0.25],
    "hidden_field_request": [0.55, 0.30, 0.15],
    "privilege_escalation": [0.40, 0.35, 0.25],
    "source_spoofing":      [0.10, 0.60, 0.30],
    "corrupted_input":      [0.65, 0.20, 0.15],
}
# Entirely harmless support texts; they have NO relation to the class.
TEXT = ["bu ayki fatura tutari gecen aya gore yuksek geldi",
        "sayac endeksi yanlis girilmis olabilir",
        "mahallede kesinti ne zaman bitecek",
        "tarife basamagimi ogrenmek istiyorum",
        "ariza kaydim icin ekip ne zaman gelir",
        "adres degisikligi nasil yapilir"]


def build_set(seed=SEED):
    ks = []
    for i in range(240):
        r = rng(seed + 6151 * i)
        c = CLASSES[pick(r(), [0.62, 0.10, 0.07, 0.07, 0.07, 0.07])]
        source = SOURCES[pick(r(), SOURCE_SHARE[c])]
        feat = {o: (r() < p) for o, p in zip(FEATURES, PROFILE[c])}
        ks.append({"no": i + 1, "class": c, "source": source, "features": feat,
                   "text": TEXT[int(r() * len(TEXT))],
                   "length": 40 + int(r() * 260)})
    return ks


RECORDS = build_set()


def harmful(k):
    return k["class"] != "clean"


def rule_filter(k, rules=("instruction_mode", "hidden_field_name", "side_effect_tool_name",
                          "source_stamp_mismatch")):
    return any(k["features"][o] for o in rules)


def approval_gate(k, required=True):
    """A record that carries a side-effecting tool request is sent to a human."""
    return required and k["features"]["side_effect_tool_name"]


def measure(records, decision):
    """What was blocked and what was blocked by mistake are written TOGETHER."""
    tp = fp = tn = fn = 0
    for k in records:
        flagged, actual = decision(k), harmful(k)
        if flagged and actual:
            tp += 1
        elif flagged and not actual:
            fp += 1
        elif not flagged and actual:
            fn += 1
        else:
            tn += 1
    precision = tp / (tp + fp) if tp + fp else 0.0
    recall = tp / (tp + fn) if tp + fn else 0.0
    return {"tp": tp, "fp": fp, "fn": fn, "tn": tn,
            "precision": round(precision, 4), "recall": round(recall, 4),
            "false_block_rate": round(fp / (fp + tn), 4) if fp + tn else 0.0}

What the Gate Counts

The second block runs four setups side by side: the no-gate baseline, the gate that looks at a single flag, and two widenings. Then it prints the share the gate sees per class, what it adds when placed on top of the rule filter, and the band of the two runs.

def gate_role(k):
    """The gate widens: a role change request is added to the side-effect tool name."""
    return k["features"]["side_effect_tool_name"] or k["features"]["role_change_request"]


def gate_source(k):
    """The gate widens further: every source other than the user goes to a human."""
    return k["features"]["side_effect_tool_name"] or k["source"] != "user"


print("records", len(RECORDS), " harmful", sum(1 for k in RECORDS if harmful(k)),
      " carrying a side-effect tool name", sum(1 for k in RECORDS if approval_gate(k)))
print("\ngate              to human  harmful  clean  precision   recall")
for name, decision in (("no-gate baseline", lambda k: False), ("side-effect name", approval_gate),
                       ("+ role mode", gate_role), ("+ source", gate_source)):
    m = measure(RECORDS, decision)
    print(f"{name:<17} {m['tp'] + m['fp']:>8} {m['tp']:>8} {m['fp']:>6}"
          f" {m['precision']:>10.4f} {m['recall']:>8.4f}")

print("\nshare of each class going to the gate")
for c in CLASSES:
    t = [k for k in RECORDS if k["class"] == c]
    sent = [k for k in t if approval_gate(k)]
    print(f"  {c:<20} {len(t):>3} {len(sent):>4}   {len(sent) / len(t):.4f}")

rule = measure(RECORDS, rule_filter)
stacked = measure(RECORDS, lambda k: rule_filter(k) or approval_gate(k))
print("\nrule filter       tp", rule["tp"], " fp", rule["fp"],
      " precision", rule["precision"], " recall", rule["recall"])
print("rule + gate       tp", stacked["tp"], " fp", stacked["fp"],
      " precision", stacked["precision"], " recall", stacked["recall"])
print("records sent to the gate that the rule did not see:",
      sum(1 for k in RECORDS if approval_gate(k) and not rule_filter(k)))

for t in (20260218, 20260219):
    ks = build_set(t)
    m = measure(ks, approval_gate)
    print(f"run {t}: harmful {sum(1 for k in ks if harmful(k))}"
          f"  to human {m['tp'] + m['fp']}  harmful sent {m['tp']}"
          f"  precision {m['precision']:.4f}")

records 240  harmful 93  carrying a side-effect tool name 27

gate              to human  harmful  clean  precision   recall
no-gate baseline         0        0      0     0.0000   0.0000
side-effect name        27       24      3     0.8889   0.2581
+ role mode             58       53      5     0.9138   0.5699
+ source               111       62     49     0.5586   0.6667

share of each class going to the gate
  clean                147    3   0.0204
  indirect_instruction  24   10   0.4167
  hidden_field_request  17    1   0.0588
  privilege_escalation  10    8   0.8000
  source_spoofing       23    3   0.1304
  corrupted_input       19    2   0.1053

rule filter       tp 74  fp 23  precision 0.7629  recall 0.7957
rule + gate       tp 74  fp 23  precision 0.7629  recall 0.7957
records sent to the gate that the rule did not see: 0
run 20260218: harmful 93  to human 27  harmful sent 24  precision 0.8889
run 20260219: harmful 83  to human 17  harmful sent 11  precision 0.6471

The Precision of the Gate

The baseline row is empty, and being empty is a finding: in the no-gate setup, the 27 requests that carry a side-effect tool name run without ever being asked, blocked harmful is 0, false alarm is 0. Everything the gate adds is written on top of this zero.

The gate sends these 27 records to a human and 24 of them are actually harmful: precision 0.8889. More than three of every four records going to the gate deserve review, which is a high rate for a human queue. The recall of the same row is 0.2581: the gate sees a quarter of the 93 harmful records. The gate cannot be discussed without writing these two numbers side by side: someone who looks at precision alone takes the gate for a good defense, someone who looks at recall alone takes it for useless. Together they say what the gate is: a narrow and accurate routing, not a broad filter.

The class table shows where this narrowness looks. 8 of the 10 records of the privilege escalation class go to the gate (0.8000); 1 of the 17 records of the hidden field request class goes (0.0588). The gate does not see a request for a field, and it is right not to: asking for a hidden field is not a side-effect action, and the defense of that class is another layer. 10 of the 24 records of the indirect instruction class go to the gate (0.4167): an instruction from a retrieved document most often needs a tool to run to be effective, so these two classes overlap structurally. The share in clean records is 0.0204, that is, 3 of the 147 clean records go to a human.

The widening rows tell two separate stories. Adding the role mode to the gate does the opposite of what is expected: precision rises from 0.8889 to 0.9138 and recall from 0.2581 to 0.5699, twenty-nine steps of 0.0108. The human queue grows from 27 to 58, but 29 of the 31 added records are actually harmful. This is a rare row in the course: the widened defense improved both numbers at once.

Adding the source to the gate does not do the same. The records going to a human are 111, precision is 0.5586, and 49 of the clean records fall into the human queue. Compared with the gate with the role mode, caught harmful rises from 53 to 62, nine steps; clean records going to a human rise from 5 to 49, forty-four steps. Forty-four reviews were bought for nine harmful records. At this point the gate stops being a gate and becomes a queue, and the human looking at the queue is also a defense layer: since all 111 records will not be looked at with the same attention, the real recall of the gate is below the 0.6667 in the table.

The count in the last two rows is the quietest finding of this lesson. When the gate is placed on top of the rule filter, none of the four numbers moves: tp 74, fp 23, precision 0.7629, recall 0.7957 — the setup with the gate and the one without it are exactly the same. The reason is in the last line: none of the 27 records going to the gate escapes the rule filter, because side_effect_tool_name is already one of the rules. Saying that a setup has both a rule filter and an approval gate and counting two layers is counting an addition of zero on the detection side as two. The value of the gate is elsewhere and does not show in these four numbers: it turns a blocked record into a reviewed one. The measure of that value belongs to the defense topic and is not in this table.

The count on the declaration side and the count on the request side do not measure the same thing, and substituting one for the other is the easiest mistake on this topic. What the previous course audited was three definitions, and it gave six findings; findings came to two per definition, and the same audit produced two findings on trusted servers as well. That number is independent of traffic: it does not change unless the set of definitions changes, it does not depend on how many requests arrive, and it is read once. The number here depends entirely on traffic: 27 of 240 records carry the flag, about one in every nine, and this share changes when the composition of the set changes. A setup that gives zero findings on the definition side says nothing about the request side, because a tool with a clean definition can also be called with a wrong request. The reverse is also true: the gate’s precision of 0.8889 does not close the question of who wrote the tool’s definition. The two numbers belong to two separate surfaces, and the security of a setup can be read only when both are written together.

The two runs show how much the gate’s precision moves. With the same gate and the same rule, the second draw sends 17 records to a human and 11 of them are harmful: precision 0.6471. The difference from the first run’s 0.8889 is 0.2418, more than six times the 0.0370 step of the base of 27 records. The gate’s precision is not a property of the gate but a property of how the draw distributed the side-effect flag across classes. A 0.8889 read from one run is not a measurement but an observation.

Summary

• In the no-gate baseline, 27 requests that carry a side-effect tool name run without being asked; blocked harmful is 0, false alarm is 0.
• The gate sends 27 records to a human and 24 are actually harmful: precision 0.8889, recall 0.2581. The gate is a narrow and accurate routing, not a broad filter.
• The gate sees 8 of the 10 privilege escalation records and 1 of the 17 hidden field request records; 10 of the 24 indirect instruction records go to the gate.
• A gate widened by role mode improves both numbers at once; a gate widened by source buys forty-four clean reviews for nine harmful records and drops precision to 0.5586.
• A gate placed on top of the rule filter moves none of the four numbers; the rule already flagged all 27 records going to the gate.
• The gate’s precision comes out 0.8889 and 0.6471 in two runs; the difference is far above the resolution of the 27-record base.

Next Step

What was counted in this lesson belonged to the request: a record arrived, carried a flag, and the gate routed it to a human or did not. The gate’s decision and the rule filter’s decision both looked at the record itself. But the record itself is not the only thing that determines which class it falls into: what the model counts as a legitimate request, how much weight it gives to text from which source, and which pattern it finds familiar come from the data it was fed. The sources are not all the same, and no lesson so far has asked this: how different from each other is the harmful share carried by records from the user, from a retrieved document, and from a tool output, and what does closing one source gain and lose? The next lesson counts this.