--- title: 'Sandbox Analysis' source: 'https://academia.sh/en/courses/penetration-testing/sandbox-analysis' course: 'Security Tooling and Penetration Testing' language: en updated: '2026-09-26T03:02:55+00:00' license: 'CC BY-SA 4.0' objectives: - "Measures an item's hiding by the number of environment hints it checks against observed behavior count" - 'Separately counts items that turn into evidence, come out clean, and stay undetermined across three analysis environments' - 'Computes prep, observation, and manual-review time together with the trace left on the other side' --- # Sandbox Analysis Measuring sandbox analysis in terms of suppressed behavior: 5 of 14 items noticing the environment on a stock image and not showing their decisive behavior, the number of resolved items rising from 7 to 12 while total time rises from 288 minutes to 965 minutes, one environment hint being impossible to hide in any environment, and analysis leaving zero records on the target while dropping a trace on the other side. The previous three tools worked over the network. A portion of the items that need examining during an assessment, though, come from a file: an unknown executable found on an endpoint, an archive extracted from an attachment, a helper tool the field team obtained from outside. There is no way to learn what these do by sending requests; the only way to learn is to **run** the item in an isolated place and observe its behavior. The sandbox's isolation strength was measured in the Endpoint and System Hardening course and is taken as input here; what is measured in this lesson is not isolation but **analysis's yield**. The only question is: how many of an item's behaviors can be observed, how many of them notice that the environment is an analysis environment and hide, and how many items end up unable to be decided. ## Environment Hints and Hiding An item can look at the environment it is running in while it runs. What it looks at are the traces that separate analysis environments from real endpoints: the machine having been up for a very short time, the low number of devices, an empty user history, traces left by analysis tools, an accelerated clock, uniform network responses. If even one of these traces is still standing, the item recognizes the environment and **never shows its decisive behavior** — it runs, does a few harmless things, and stops. Every trace can be hidden, and each one carries a **prep cost**: refurbishing the image, building a fake user history, cleaning up tool traces. One trace falls outside this. The only way to fix uniform network responses is to let the item reach a real target; in an authorized assessment this step is out of scope, and connecting to another party's infrastructure without written authorization is both illegal and unprofessional. This trace is a boundary that prep cannot close. - **AT22.** The hint catalog, environments, item list, and behaviors are a **model**; no real item was run. Items are referred to only by the hints they check and the class of behavior observed. - **AT23.** Isolation strength is not measured here; it is taken as input. - **AT24.** An item that has **at least one** of the hints it checks not hidden notices the environment and does not show its decisive behaviors. It keeps showing its harmless behaviors. - **AT25.** For a behavior to be visible, the observation window has to be larger than its delay. Delays and windows were written by hand; there is no randomness. - **AT26.** The decision is three-valued. **Evidence**: a decisive behavior was observed. **Clean**: the item was not hidden, all its behaviors fit in the window, and none was decisive. **Undetermined**: every remaining case. - **AT27.** The decisive marker belongs to the behavior, not the item; a legitimate item can also show a behavior counted as decisive. - **AT28.** Analysis does not run in the target environment; it does not produce records on target systems. ```js // alan/observe.mjs — this is a model of SANDBOX ANALYSIS. No real item is run; // items are referred to only by the environment hints they check and the class of behavior // observed. Isolation strength is not measured here; it is taken as input from the Endpoint // and System Hardening course. // hint -> the prep cost of hiding that hint (min) export const HINT = { 'short-uptime': 8, 'low-device-count': 12, 'known-tool-traces': 15, 'empty-user-history': 20, 'accelerated-clock': 10, 'uniform-network-response': 0, }; // Analysis environment: which hints it hides, how many minutes the observation window is. // 'uniform-network-response' is not hidden in any environment: hiding it requires the item // to reach a real target, and that is out of scope in an authorized assessment. export const ENVIRONMENT = { 'stock-image': { hides: ['short-uptime'], window: 5 }, 'refurbished': { hides: ['short-uptime', 'low-device-count', 'known-tool-traces'], window: 20 }, 'near-real': { hides: ['short-uptime', 'low-device-count', 'known-tool-traces', 'empty-user-history', 'accelerated-clock'], window: 60 }, }; export const MANUAL_MIN = 30; // manual review per undetermined item // row: item | actual | hints it checks | behaviors (name:delay, * = decisive) export const ITEM = ` k01 bad short-uptime file-write:0,*persistence:2 k02 bad low-device-count,known-tool-traces process-creation:0,*credential-read:5 k03 bad empty-user-history file-write:0,*external-connection:10 k04 bad accelerated-clock file-write:0,*timed-trigger:45 k05 bad uniform-network-response process-creation:0,*external-connection:3 k06 bad - file-write:0,*persistence:1,external-connection:4 k07 bad short-uptime,uniform-network-response file-write:0,*credential-read:8 k08 clean - file-write:0,process-creation:1 k09 clean - file-write:0,external-connection:2 k10 clean - file-write:0,process-creation:0,external-connection:1 k11 clean - file-write:0,*credential-read:3 k12 clean - file-write:0,timed-trigger:40 k13 clean - process-creation:0,external-connection:35 k14 clean - file-write:0 `.trim().split('\n').map((r) => { const [name, actual, hint, dav] = r.trim().split(/\s+/); return { name, bad: actual === 'bad', controls: hint === '-' ? [] : hint.split(','), behavior: dav.split(',').map((d) => { const [n, g] = d.split(':'); return { name: n.replace('*', ''), decisive: n.startsWith('*'), delay: +g }; }) }; }); // An item that has one of the hints it checks not hidden notices the environment and HIDES: // it never shows its decisive behaviors. export const hides = (k, o) => k.controls.some((i) => !ENVIRONMENT[o].hides.includes(i)); export function observe(k, o) { const p = ENVIRONMENT[o].window; const g = hides(k, o); const observed = k.behavior.filter((d) => d.delay <= p && !(g && d.decisive)); const suppressed = k.behavior.length - observed.length; let decision = 'undetermined'; if (observed.some((d) => d.decisive)) decision = 'evidence'; else if (!g && suppressed === 0) decision = 'clean'; return { hidden: g, observed: observed.length, suppressed, decision, externalContact: observed.some((d) => d.name === 'external-connection') }; } export function measureEnvironment(o) { const s = { prep: 0, hidden: 0, observed: 0, suppressed: 0, evidence: 0, clean: 0, undetermined: 0, falseEvidence: 0, escaped: 0, externalTrace: 0 }; s.prep = ENVIRONMENT[o].hides.reduce((t, i) => t + HINT[i], 0); for (const k of ITEM) { const r = observe(k, o); if (r.hidden) s.hidden += 1; s.observed += r.observed; s.suppressed += r.suppressed; s[r.decision] += 1; if (r.decision === 'evidence' && !k.bad) s.falseEvidence += 1; if (r.decision !== 'evidence' && k.bad) s.escaped += 1; if (r.externalContact) s.externalTrace += 1; } s.observeMin = ITEM.length * ENVIRONMENT[o].window; s.manualMin = s.undetermined * MANUAL_MIN; s.totalMin = s.prep + s.observeMin + s.manualMin; return s; } ``` ## Fourteen Items, Three Environments ```js // alan/measure.mjs — fourteen items are observed in three analysis environments. import { HINT, ENVIRONMENT, ITEM, MANUAL_MIN, hides, observe, measureEnvironment } from './observe.mjs'; const y = (g, ...s) => console.log(s.map((v, i) => (g[i] < 0 ? String(v).padEnd(-g[i]) : String(v).padStart(g[i]))).join('').trimEnd()); console.log(`${ITEM.length} items (${ITEM.filter((k) => k.bad).length} bad), ` + `${Object.keys(HINT).length} environment hints, ${Object.keys(ENVIRONMENT).length} analysis environments`); console.log(`hint never hidden in any environment: ` + Object.keys(HINT).filter((i) => !Object.values(ENVIRONMENT).some((o) => o.hides.includes(i))).join(', ')); const A = [-14, 9, 10, 10, 12, 10, 8, 14, 16]; console.log('\n1. result per environment'); y(A, 'environment', 'window', 'hidden', 'observed', 'suppressed', 'evidence', 'clean', 'undetermined', 'false evidence'); for (const o of Object.keys(ENVIRONMENT)) { const s = measureEnvironment(o); y(A, o, ENVIRONMENT[o].window, s.hidden, s.observed, s.suppressed, s.evidence, s.clean, s.undetermined, s.falseEvidence); } const B = [-7, 8, 23, 23, 23]; console.log('\n2. decision per item'); y(B, 'item', 'actual', ...Object.keys(ENVIRONMENT)); for (const k of ITEM) y(B, k.name, k.bad ? 'bad' : 'clean', ...Object.keys(ENVIRONMENT).map((o) => { const r = observe(k, o); return r.hidden ? `${r.decision} (hidden)` : r.decision; })); const C = [-14, 8, 12, 11, 11, 13, 16]; console.log('\n3. cost and trace'); y(C, 'environment', 'prep', 'observe min', 'manual min', 'total min', 'escaped bad', 'external trace'); for (const o of Object.keys(ENVIRONMENT)) { const s = measureEnvironment(o); y(C, o, s.prep, s.observeMin, s.manualMin, s.totalMin, s.escaped, s.externalTrace); } const h = measureEnvironment('stock-image'), g = measureEnvironment('near-real'); console.log(`\nitems resolved ${h.evidence + h.clean} -> ${g.evidence + g.clean} / ${ITEM.length}`); console.log(`items detecting env ${h.hidden} -> ${g.hidden} / ${ITEM.length}`); console.log(`records produced on target 0 (analysis does not run in the target environment)`); console.log(`manual review time ${MANUAL_MIN} min/item`); ``` ``` 14 items (7 bad), 6 environment hints, 3 analysis environments hint never hidden in any environment: uniform-network-response 1. result per environment environment window hidden observed suppressed evidence clean undetermined false evidence stock-image 5 5 22 7 3 4 7 1 refurbished 20 4 23 6 4 4 6 1 near-real 60 2 27 2 6 6 2 1 2. decision per item item actual stock-image refurbished near-real k01 bad evidence evidence evidence k02 bad undetermined (hidden) evidence evidence k03 bad undetermined (hidden) undetermined (hidden) evidence k04 bad undetermined (hidden) undetermined (hidden) evidence k05 bad undetermined (hidden) undetermined (hidden) undetermined (hidden) k06 bad evidence evidence evidence k07 bad undetermined (hidden) undetermined (hidden) undetermined (hidden) k08 clean clean clean clean k09 clean clean clean clean k10 clean clean clean clean k11 clean evidence evidence evidence k12 clean undetermined undetermined clean k13 clean undetermined undetermined clean k14 clean clean clean clean 3. cost and trace environment prep observe min manual min total min escaped bad external trace stock-image 8 70 210 288 5 3 refurbished 35 280 180 495 4 3 near-real 65 840 60 965 2 5 items resolved 7 -> 12 / 14 items detecting env 5 -> 2 / 14 records produced on target 0 (analysis does not run in the target environment) manual review time 30 min/item ``` ## The Suppressed Behavior The `hidden` column of the first table is this lesson's measure. On a stock image, **five** of fourteen items notice the environment and never show their decisive behavior. As the environment is refurbished this number drops to 4, then to 2. While observed behavior rises from 22 to 27, suppressed behavior falls from 7 to 2. Hiding is a different thing from a short observation window, and the two need to be told apart. If the window is short, the behavior has not been seen **yet**; lengthening it fixes that. If the item is hiding, the behavior will **never** be seen; lengthening the window changes nothing, the hint has to be hidden. The second table reads this distinction item by item: `k04` has a behavior with a forty-five-minute delay and checks the clock hint; seeing it requires both the window to reach sixty minutes and the accelerated clock to be hidden — one without the other is not enough. By contrast, `k12` and `k13` check nothing at all; for them, lengthening the window alone is enough, and both come out clean in the widest environment. Which prep step unlocks which item is also read from this table, and the gain is not distributed evenly. Moving from the stock image to the refurbished environment asks for 27 extra minutes of prep and turns a single item — `k02`, which checks both device count and tool traces — into evidence. Moving from refurbished to near-real asks for 30 extra minutes of prep and a window three times as long; in return, `k03` and `k04` turn into evidence, and `k12` and `k13` turn clean. The second step does four times the work for a prep cost only slightly above the first, but it does so by pushing the window from 280 minutes to 840 minutes. Prep is cheap, the window is expensive, and they are two separate line items. Two items cannot be resolved in any environment: `k05` and `k07`. Both check the uniformity of network responses, and that hint is not hidden in any environment. Two of the seven bad items stay **undetermined** regardless of the analysis environment's quality. These two items get handed off to manual review and appear in the report with the line "analysis could not resolve this." ## Evidence, Clean, and False Evidence The candidate-evidence gap is read here as three values. Fourteen items arrive at analysis as candidates. On the stock image, 3 turn into evidence, 4 come out clean, 7 stay undetermined. In the best environment there are 6 evidence, 6 clean, and 2 undetermined. The number of resolved items goes from **7 to 12**. Items that come out clean are worth as much as evidence. If all of an item's behaviors were seen and none was decisive, that item drops out of the queue — this is the only mechanism that shortens the candidate list. But the condition for a "clean" decision is strict: the item must not be hidden **and** none of its behaviors may have been suppressed. If even a single behavior was suppressed, the decision goes to undetermined, not clean. This strictness is deliberate, because relaxing it would let `k05` and `k07` quietly count as clean. An error in the opposite direction stands in every environment: the **false evidence** column is 1 in all three. A legitimate item shows a behavior counted as decisive, and analysis flags it as evidence. Raising the prep, lengthening the window, and hiding hints do not change this line, because what is missing is not observation, it is **context**. The sandbox sees the behavior; it does not see whether that behavior belongs there, what job the item was set up to do. That is why analysis output alone is not a decision — it is input to one. ## Cost and the Trace Left on the Other Side The third table gives the cost, and it rises in one direction only. Prep goes from 8 minutes to 65; observation goes from 70 to 840. Manual review goes the other way — from 210 minutes to 60 — but the drop does not offset the rise: the total goes from 288 minutes to 965 minutes, more than tripling. In return, the number of resolved items rises by five, and the number of escaped bad items falls from 5 to 2. The cost paid per item is roughly 135 minutes. The last column is a kind of trace seen for the first time in this course. Records produced on target systems are **zero**; analysis does not run in the target environment, none of its requests reach the target. But the number of items that attempt an external connection during observation rises from 3 to 5, and those attempts go somewhere. Even if the controlled network does not let them reach a real target, the attempt itself and its timing can leak outward in a way that tells the other side something. The closer the analysis environment gets to reality, the more behavior an item shows; and every behavior it shows is also **a sign that it is being examined**. This splits the course's trace column in two: the trace the defense sees, and the trace the other side can see. The two point in opposite directions. The trace the defense sees is a sign that the assessment is running transparently inside the organization, and its excess only creates noise. The trace left on the other side is irreversible: learning that an item is being examined is enough for whoever is using it to change their behavior, and it corrupts every measurement after that. The next lesson measures the second one far more sharply. ## Summary - Of fourteen items, five notice the environment on a stock image and never show their decisive behavior; as the environment is refurbished this number falls to 2, and observed behavior rises from 22 to 27. - Hiding and a short window are separate problems: the first is solved by hiding a hint, the second by lengthening the window, and some items need both at once. - One environment hint cannot be hidden in any environment, because hiding it requires the item to reach a real target, which is out of scope; two of the seven bad items stay undetermined in every environment for this reason. - The number of resolved items goes from 7 to 12; items that come out clean are the only mechanism that shortens the candidate list, and its condition is strict. - False evidence is 1 in all three environments; what is missing is not observation, it is context, and prep does not close that gap. - Total time goes from 288 minutes to 965 minutes; records produced on target are zero, but the number of items attempting an external connection rises from 3 to 5, leaving a trace on the other side. ## Next Step This lesson measured a single analysis environment's decision, and two items stayed unresolved. There is a second way to resolve an item: ask the same item to **a large number of independent engines** and combine the results that come back. This route brings analysis time down from minutes to seconds, and in return raises a new question — how to combine results that contradict each other. The next lesson takes up multi-engine reputation services on this axis. The same set of items is run through a large number of engines, the false-positive and false-negative rate of the **majority decision** is measured threshold by threshold, and the cost of trusting a single engine is counted. Alongside this, a privacy cost is computed: sending a file to an outside service is publishing it, and a portion of the items in the fictional measurement network carry subscriber data.