--- title: 'Permissions and Sandboxing' source: 'https://academia.sh/en/courses/ai-safety-and-evaluation/permissions-and-sandboxing' course: 'AI Safety and Evaluation' language: en updated: '2026-10-21T05:02:28+00:00' license: 'CC BY-SA 4.0' objectives: - 'Define the permission surface of a tool as the pair of capability and scope, and draw the boundary of the sandbox with that pair.' - 'Compare the five permission sets on the axes of harmful stopped, false alarm and completed task.' - 'Compute the marginal precision of a narrowing step and show where narrowing permissions stops paying.' - 'Determine which attack class the gate is blind to by reading its recall per class.' --- # Permissions and Sandboxing Counting tool permission as a defense layer: the same fictional set of 240 records runs under five permission sets, and the harmful records stopped, the false alarms and the completed tasks are printed side by side. On the undefended baseline the harmful stopped is 0 and the false alarm is 0; closing the execution capability stops 12 harmful records, blocks no clean record and gives a precision of 1.0000; closing the side-effect capability gives 24 harmful against 3 false alarms; the allowlisted set stops 34 harmful and lowers the completed tasks to 0.8844, read-only stops 62 harmful and lowers them to 0.6667. The marginal precision of narrowing falls from 1.0000 to 0.4167: the last two steps block more than one clean record for every harmful record they stop. The class distribution shows the blind spot of the gate: 1.0000 of the privilege escalation class is stopped, only 0.0588 of the hidden field request class. In three lessons the defense worked **on the text**. The input and output filters read a record and flagged it, content moderation separated it per class, masking deleted the field going to the model. What all three touched was the same: the **text** entering or leaving the model. What the agent **does** after reading that text was never touched. This lesson moves the defense from the text to the action. Even if a record passes the filter, it is harmless if the capability it wants to touch was never granted. The only thing measured is this: as permissions narrow, how many harmful records stop, how many clean records stop by mistake, and how many tasks can still be completed. - **DF41.** The code is a **simulator**; no real tool, file system or runtime is called. The record set, the classes and the features are **fictional**, the seed is **20260218**. - **DF42. An attack is not a text but a feature profile.** The six sentences the records carry are ordinary support requests and have **no relation to the class**; the class lives only in the structural flags. Nowhere in the code is there a working payload, template or instruction string. - **DF43. The baseline is the undefended setup:** no filter, marking or gate; the harmful stopped is **0**, the false alarm is **0**. - **DF44. The rule of the course:** a defense that does not write the share it stopped and the share it stopped by mistake **together** counts as unmeasured. - **DF45. The permission surface in this lesson is a pair:** the **capability** the tool declares (read, external access, write, execution) and the **scope** it can touch (the source class). The capability a record asks for is derived only from the flags that exist; no new text is added to the set. - **DF46. The sandbox is the boundary this pair draws**, and the gate stands in front of the executor: a request that does not pass never reaches the tool. The gate **only stops**; there is no path that leaves the boundary. - **DF47. The completed task** is the share of clean records that are not stopped: how many of the 147 clean records can finish their job. - **DF48. Resolution:** the step of recall is $1/93 = 0.0108$, the step of the false block rate is $1/147 = 0.0068$. There is no draw in the decision of the gate, so the measure is **independent of the run**; the same set gives the same number in every run. ## Shared Definition The record set, the classes, the defense layers and the measure are the same throughout the course. The set holds 240 fictional support records; each record has a class, a source and seven structural features. The measure function `measure` returns every decision with five numbers and never separates what is stopped from what is stopped by mistake. ```python SEED, M32 = 20260218, 0xFFFFFFFF def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def trace_id(m): h = 2166136261 for c in str(m): h = ((h ^ ord(c)) * 16777619) & M32 return h # ---- FICTIONAL records: no attack TEXT is written, an attack is a FEATURE PROFILE CLASSES = ["clean", "indirect_instruction", "hidden_field_request", "privilege_escalation", "source_spoofing", "corrupted_input"] SOURCES = ["user", "retrieved_document", "tool_output"] # Features are structural; none of them is a working payload. FEATURES = ["instruction_mode", "role_change_request", "hidden_field_name", "side_effect_tool_name", "source_stamp_mismatch", "unusual_stamp_distribution", "excessive_length"] # Feature probability per class (fictional). Clean records carry small shares too: # well-meant text can carry these features as well. PROFILE = { "clean": [0.06, 0.02, 0.05, 0.03, 0.02, 0.04, 0.05], "indirect_instruction": [0.88, 0.42, 0.10, 0.30, 0.35, 0.08, 0.12], "hidden_field_request": [0.35, 0.15, 0.86, 0.12, 0.10, 0.06, 0.08], "privilege_escalation": [0.30, 0.25, 0.12, 0.84, 0.14, 0.05, 0.07], "source_spoofing": [0.22, 0.55, 0.08, 0.10, 0.82, 0.07, 0.06], "corrupted_input": [0.05, 0.03, 0.04, 0.03, 0.06, 0.80, 0.62], } SOURCE_SHARE = { "clean": [0.72, 0.20, 0.08], "indirect_instruction": [0.05, 0.70, 0.25], "hidden_field_request": [0.55, 0.30, 0.15], "privilege_escalation": [0.40, 0.35, 0.25], "source_spoofing": [0.10, 0.60, 0.30], "corrupted_input": [0.65, 0.20, 0.15], } # Fictional, entirely harmless support texts. They have NO relation to the class; # the class information lives only in the feature profile. TEXT = ["bu ayki fatura tutari gecen aya gore yuksek geldi", "sayac endeksi yanlis girilmis olabilir", "mahallede kesinti ne zaman bitecek", "tarife basamagimi ogrenmek istiyorum", "ariza kaydim icin ekip ne zaman gelir", "adres degisikligi nasil yapilir"] RECORDS = [] for i in range(240): r = rng(SEED + 6151 * i) c = CLASSES[pick(r(), [0.62, 0.10, 0.07, 0.07, 0.07, 0.07])] source = SOURCES[pick(r(), SOURCE_SHARE[c])] feat = {o: (r() < p) for o, p in zip(FEATURES, PROFILE[c])} RECORDS.append({"no": i + 1, "class": c, "source": source, "features": feat, "text": TEXT[int(r() * len(TEXT))], "length": 40 + int(r() * 260)}) def harmful(k): return k["class"] != "clean" # ---- DEFENSE 1: rule filter (structural, explainable) def rule_filter(k, rules=("instruction_mode", "hidden_field_name", "side_effect_tool_name", "source_stamp_mismatch")): return any(k["features"][o] for o in rules) # ---- MEASURE: what was blocked and what was blocked by mistake are written together def measure(records, decision): tp = fp = tn = fn = 0 for k in records: flagged, actual = decision(k), harmful(k) if flagged and actual: tp += 1 elif flagged and not actual: fp += 1 elif not flagged and actual: fn += 1 else: tn += 1 precision = tp / (tp + fp) if tp + fp else 0.0 recall = tp / (tp + fn) if tp + fn else 0.0 return {"tp": tp, "fp": fp, "fn": fn, "tn": tn, "precision": round(precision, 4), "recall": round(recall, 4), "false_block_rate": round(fp / (fp + tn), 4) if fp + tn else 0.0} print("records:", len(RECORDS), "- harmful:", sum(harmful(k) for k in RECORDS), "- clean:", sum(not harmful(k) for k in RECORDS)) print("source:", {s: sum(k["source"] == s for k in RECORDS) for s in SOURCES}) ``` ``` records: 240 - harmful: 93 - clean: 147 source: {'user': 137, 'retrieved_document': 72, 'tool_output': 31} ``` ## The Permission Surface and the Boundary of the Sandbox The principle of least privilege was established in the Introduction to Cybersecurity course, sandboxes and allowlists were measured in the Endpoint and System Hardening course, and file permissions in the Introduction to Linux course. The AI Agents course measured the same gate in the agent loop: at full permission 0.9583 of the tasks were completed, in the allowlisted set 0.8333, and the gate stopped 20 calls. None of these procedures is repeated; the numbers are input. The question here is different: how many harmful records does the same gate stop as a **defense**, and what does it cost. Note that the gate only stops. No path that crosses the boundary is defined in the code; the `sandbox` function decides whether a request can pass, and a request that does not pass never reaches the tool. The five permission sets narrow step by step: each row is a subset of the one before. ```python # ---- Specific to this lesson: the CAPABILITY a record wants to touch is read from existing flags def requested_capability(k): """No new text or payload is added; the capability is derived only from structural flags.""" if k["features"]["side_effect_tool_name"]: return "execution" if k["features"]["instruction_mode"] else "write" if k["source"] != "user": return "external_access" return "read" ALL_CAPABILITIES = ("read", "external_access", "write", "execution") ALL_SCOPES = ("user", "retrieved_document", "tool_output") PERMISSIONS = { "full access": (ALL_CAPABILITIES, ALL_SCOPES), "no execution": (("read", "external_access", "write"), ALL_SCOPES), "no side effects": (("read", "external_access"), ALL_SCOPES), "allowlisted": (("read", "external_access"), ("user", "retrieved_document")), "read-only": (("read",), ("user",)), } def sandbox(k, p): """The gate stands in front of the executor and ONLY stops; there is no path that leaves the boundary. A request that does not pass never reaches the tool.""" capability, scope = p return requested_capability(k) not in capability or k["source"] not in scope print("requested capability:", {g: sum(requested_capability(k) == g for k in RECORDS) for g in ALL_CAPABILITIES}) print("\npermission set block false precision recall false-block task marginal") prev_tp = prev_fp = 0 for name, p in PERMISSIONS.items(): s = measure(RECORDS, lambda k, p=p: sandbox(k, p)) d_tp, d_fp = s["tp"] - prev_tp, s["fp"] - prev_fp marginal = f"{d_tp / (d_tp + d_fp):.4f}" if d_tp + d_fp else "-" print(f"{name:<15} {s['tp']:>6} {s['fp']:>7} {s['precision']:>10.4f} {s['recall']:>11.4f}" f" {s['false_block_rate']:>12.4f} {s['tn'] / 147:>6.4f} {marginal:>10}") prev_tp, prev_fp = s["tp"], s["fp"] print("\nshare stopped per class (allowlisted set)") for s in CLASSES[1:]: g = [k for k in RECORDS if k["class"] == s] d = sum(sandbox(k, PERMISSIONS["allowlisted"]) for k in g) print(f" {s:<20} {d:>2}/{len(g):<3} {d / len(g):.4f}") def allowlisted(k): return sandbox(k, PERMISSIONS["allowlisted"]) print("\nlayer on layer block false precision recall false-block") for name, decision in (("rule filter", rule_filter), ("permission gate", allowlisted), ("rule OR permission", lambda k: rule_filter(k) or allowlisted(k))): s = measure(RECORDS, decision) print(f"{name:<24} {s['tp']:>6} {s['fp']:>7} {s['precision']:>10.4f}" f" {s['recall']:>11.4f} {s['false_block_rate']:>12.4f}") added = [k for k in RECORDS if allowlisted(k) and not rule_filter(k)] print("added by permission only:", len(added), "records,", sum(harmful(k) for k in added), "harmful") ``` ``` requested capability: {'read': 129, 'external_access': 84, 'write': 15, 'execution': 12} permission set block false precision recall false-block task marginal full access 0 0 0.0000 0.0000 0.0000 1.0000 - no execution 12 0 1.0000 0.1290 0.0000 1.0000 1.0000 no side effects 24 3 0.8889 0.2581 0.0204 0.9796 0.8000 allowlisted 34 17 0.6667 0.3656 0.1156 0.8844 0.4167 read-only 62 49 0.5586 0.6667 0.3333 0.6667 0.4667 share stopped per class (allowlisted set) indirect_instruction 14/24 0.5833 hidden_field_request 1/17 0.0588 privilege_escalation 10/10 1.0000 source_spoofing 6/23 0.2609 corrupted_input 3/19 0.1579 layer on layer block false precision recall false-block rule filter 74 23 0.7629 0.7957 0.1565 permission gate 34 17 0.6667 0.3656 0.1156 rule OR permission 78 37 0.6783 0.8387 0.2517 added by permission only: 18 records, 4 harmful ``` ## The Marginal Value of Narrowing The first row is the baseline: the gate stops nothing, all 93 harmful records pass and all 147 clean records finish their job. Every row after that stops something and pays a price. The second row is the cheapest gain of this lesson. Closing the execution capability stops **12** harmful records, blocks **no** clean record, leaves the precision at **1.0000** and the completed task at **1.0000**. The reason is in the structure: the execution capability requires two flags to be present together, and that is not where clean traffic goes. This is the measurable core of the principle of least privilege — **closing a capability that clean traffic does not visit is a free defense.** The last column says where that ends. **Marginal precision** is the ratio of the extra harmful records a narrowing step stops to the total extra records stopped in that step. The sequence runs 1.0000, 0.8000, 0.4167, 0.4667. At the third step the ratio falls **below 0.5000**: moving to the allowlist stops 10 more harmful records while blocking 14 more clean records. The fourth step is heavier still — 32 clean records for 28 harmful. This is the evidence this lesson contributes to the second claim of the course: **past a certain point, continuing to narrow stops more tasks than harmful records.** The number is still growing, because recall rises from 0.3656 to 0.6667; but in the same step the completed task drops from 0.8844 to 0.6667. Reading recall alone would count this step as an improvement. Resolution tells whether the differences are measured. The last jump of 0.3011 in recall is twenty-eight times the step of 0.0108; the drop of 0.2177 in the completed task is thirty-two times the step of 0.0068. Both are far above the band. On the other hand, the difference in false blocks between `no execution` and `full access` is **exactly zero**, and that is not rounding but the count itself. ## Where the Gate Is Blind The last block opens what an average hides. The allowlisted set gives a recall of 0.3656 in general, but the distribution per class is not flat at all: **1.0000** of the `privilege_escalation` class is stopped, 0.5833 of the `indirect_instruction` class, and only **0.0588** of the `hidden_field_request` class. The reason is what the gate measures. The permission gate looks at **what a record wants to do**, not **what it says**. A record carrying a hidden field request mostly asks only for the read capability; read stays open in every set, because when it is closed no task remains. The permission gate is blind to that class by its structure, and this blindness is not closed by narrowing. Two conclusions follow. First, the permission layer does **not take the place** of a filter: the two layers look at different classes, and the recall of one does not fill the blind spot of the other. Second, reporting a defense with a single average rate is misleading — the number 0.3656 contains both 1.0000 and 0.0588 and does not say which attack is stopped by which layer. ## Layer on Layer The first answer that comes to mind for the blind spot is to stack two layers: let the rule filter look at the text and the permission gate at the action, and a record that is caught by either is enough. The last table gives the number, and the result is the opposite of what was expected. The rule filter alone stops 74 harmful records and blocks 23 clean records. Putting the permission gate on top raises recall from 0.7957 to **0.8387**, that is, four more harmful records. In the same step the false block rate rises from 0.1565 to **0.2517** and the precision falls from 0.7629 to 0.6783. The last line says why: of the 18 records that the permission gate **adds** to the filter, only **4** are harmful and 14 are clean. The marginal precision of the layer is $4/18 = 0.2222$. Four harmful records are four times the step of 0.0108 and are a measured gain; fourteen clean records are fourteen times the step of 0.0068 and are a measured loss. Both are real; which one weighs more is read not from the number but from how expensive a blocked task is. The one thing the number says for certain is this: **stacking two defenses does not enlarge recall for free.** Reporting this table with the recall column alone would have made the gain of the added layer look larger than it is. ## Summary - The permission surface is the pair of the **capability** a tool declares and the **scope** it can touch; the sandbox is the boundary this pair draws, and the gate stands in front of the executor and only stops. - On the undefended baseline the harmful stopped is 0 and the false alarm is 0; closing the execution capability stops 12 harmful records and blocks no clean record, with precision 1.0000 and completed task 1.0000. The allowlisted set stops 34 harmful and lowers the task to 0.8844, read-only stops 62 and lowers it to 0.6667: while recall rises to 0.6667, a third of the tasks disappear. - Marginal precision falls from 1.0000 to 0.4167: the last two narrowing steps block more than one clean record for every harmful record they stop. - The permission gate is class-selective: it stops 1.0000 of the `privilege_escalation` class and only 0.0588 of the `hidden_field_request` class; a single average rate hides this distinction. - Putting the permission gate on top of the rule filter raises recall from 0.7957 to 0.8387 while it moves the false block rate from 0.1565 to 0.2517; of the 18 added records only 4 are harmful. - There is no draw in the decision of the gate, the measure is independent of the run; the recall step is 0.0108, the false block step is 0.0068. ## Next Step Every defense in this lesson worked in the same way: it **stopped** a record. The cost of stopping is also written in the table — the four narrowing steps blocked 49 clean records in total. Yet stopping is not the only way to neutralize a harmful record. The harm of a record often comes not from its content but from that content **being read as an instruction**; if the text that comes from a retrieved document is counted as data, it becomes harmless without being blocked at all. The next lesson measures this defense that does not block and counts two things: how many of the 93 harmful records marking neutralizes without stopping any record, which classes remain, and what the false alarm counterpart of this is.