Sandbox Analysis

The previous three tools worked over the network. A portion of the items that need examining during an assessment, though, come from a file: an unknown executable found on an endpoint, an archive extracted from an attachment, a helper tool the field team obtained from outside. There is no way to learn what these do by sending requests; the only way to learn is to run the item in an isolated place and observe its behavior.

The sandbox’s isolation strength was measured in the Endpoint and System Hardening course and is taken as input here; what is measured in this lesson is not isolation but analysis’s yield. The only question is: how many of an item’s behaviors can be observed, how many of them notice that the environment is an analysis environment and hide, and how many items end up unable to be decided.

Environment Hints and Hiding

An item can look at the environment it is running in while it runs. What it looks at are the traces that separate analysis environments from real endpoints: the machine having been up for a very short time, the low number of devices, an empty user history, traces left by analysis tools, an accelerated clock, uniform network responses. If even one of these traces is still standing, the item recognizes the environment and never shows its decisive behavior — it runs, does a few harmless things, and stops.

Every trace can be hidden, and each one carries a prep cost: refurbishing the image, building a fake user history, cleaning up tool traces. One trace falls outside this. The only way to fix uniform network responses is to let the item reach a real target; in an authorized assessment this step is out of scope, and connecting to another party’s infrastructure without written authorization is both illegal and unprofessional. This trace is a boundary that prep cannot close.

• AT22. The hint catalog, environments, item list, and behaviors are a model; no real item was run. Items are referred to only by the hints they check and the class of behavior observed.
• AT23. Isolation strength is not measured here; it is taken as input.
• AT24. An item that has at least one of the hints it checks not hidden notices the environment and does not show its decisive behaviors. It keeps showing its harmless behaviors.
• AT25. For a behavior to be visible, the observation window has to be larger than its delay. Delays and windows were written by hand; there is no randomness.
• AT26. The decision is three-valued. Evidence: a decisive behavior was observed. Clean: the item was not hidden, all its behaviors fit in the window, and none was decisive. Undetermined: every remaining case.
• AT27. The decisive marker belongs to the behavior, not the item; a legitimate item can also show a behavior counted as decisive.
• AT28. Analysis does not run in the target environment; it does not produce records on target systems.

// alan/observe.mjs — this is a model of SANDBOX ANALYSIS. No real item is run;
// items are referred to only by the environment hints they check and the class of behavior
// observed. Isolation strength is not measured here; it is taken as input from the Endpoint
// and System Hardening course.
// hint -> the prep cost of hiding that hint (min)
export const HINT = {
  'short-uptime': 8, 'low-device-count': 12, 'known-tool-traces': 15,
  'empty-user-history': 20, 'accelerated-clock': 10, 'uniform-network-response': 0,
};

// Analysis environment: which hints it hides, how many minutes the observation window is.
// 'uniform-network-response' is not hidden in any environment: hiding it requires the item
// to reach a real target, and that is out of scope in an authorized assessment.
export const ENVIRONMENT = {
  'stock-image':   { hides: ['short-uptime'], window: 5 },
  'refurbished':   { hides: ['short-uptime', 'low-device-count', 'known-tool-traces'], window: 20 },
  'near-real':     { hides: ['short-uptime', 'low-device-count', 'known-tool-traces',
    'empty-user-history', 'accelerated-clock'], window: 60 },
};
export const MANUAL_MIN = 30;   // manual review per undetermined item

// row: item | actual | hints it checks | behaviors (name:delay, * = decisive)
export const ITEM = `
  k01 bad   short-uptime                                    file-write:0,*persistence:2
  k02 bad   low-device-count,known-tool-traces               process-creation:0,*credential-read:5
  k03 bad   empty-user-history                               file-write:0,*external-connection:10
  k04 bad   accelerated-clock                                file-write:0,*timed-trigger:45
  k05 bad   uniform-network-response                         process-creation:0,*external-connection:3
  k06 bad   -                                                 file-write:0,*persistence:1,external-connection:4
  k07 bad   short-uptime,uniform-network-response            file-write:0,*credential-read:8
  k08 clean -                                                 file-write:0,process-creation:1
  k09 clean -                                                 file-write:0,external-connection:2
  k10 clean -                                                 file-write:0,process-creation:0,external-connection:1
  k11 clean -                                                 file-write:0,*credential-read:3
  k12 clean -                                                 file-write:0,timed-trigger:40
  k13 clean -                                                 process-creation:0,external-connection:35
  k14 clean -                                                 file-write:0
`.trim().split('\n').map((r) => {
  const [name, actual, hint, dav] = r.trim().split(/\s+/);
  return { name, bad: actual === 'bad',
    controls: hint === '-' ? [] : hint.split(','),
    behavior: dav.split(',').map((d) => {
      const [n, g] = d.split(':');
      return { name: n.replace('*', ''), decisive: n.startsWith('*'), delay: +g };
    }) };
});

// An item that has one of the hints it checks not hidden notices the environment and HIDES:
// it never shows its decisive behaviors.
export const hides = (k, o) => k.controls.some((i) => !ENVIRONMENT[o].hides.includes(i));

export function observe(k, o) {
  const p = ENVIRONMENT[o].window;
  const g = hides(k, o);
  const observed = k.behavior.filter((d) => d.delay <= p && !(g && d.decisive));
  const suppressed = k.behavior.length - observed.length;
  let decision = 'undetermined';
  if (observed.some((d) => d.decisive)) decision = 'evidence';
  else if (!g && suppressed === 0) decision = 'clean';
  return { hidden: g, observed: observed.length, suppressed, decision,
    externalContact: observed.some((d) => d.name === 'external-connection') };
}

export function measureEnvironment(o) {
  const s = { prep: 0, hidden: 0, observed: 0, suppressed: 0, evidence: 0, clean: 0,
    undetermined: 0, falseEvidence: 0, escaped: 0, externalTrace: 0 };
  s.prep = ENVIRONMENT[o].hides.reduce((t, i) => t + HINT[i], 0);
  for (const k of ITEM) {
    const r = observe(k, o);
    if (r.hidden) s.hidden += 1;
    s.observed += r.observed; s.suppressed += r.suppressed; s[r.decision] += 1;
    if (r.decision === 'evidence' && !k.bad) s.falseEvidence += 1;
    if (r.decision !== 'evidence' && k.bad) s.escaped += 1;
    if (r.externalContact) s.externalTrace += 1;
  }
  s.observeMin = ITEM.length * ENVIRONMENT[o].window;
  s.manualMin = s.undetermined * MANUAL_MIN;
  s.totalMin = s.prep + s.observeMin + s.manualMin;
  return s;
}

Fourteen Items, Three Environments

// alan/measure.mjs — fourteen items are observed in three analysis environments.
import { HINT, ENVIRONMENT, ITEM, MANUAL_MIN, hides, observe, measureEnvironment } from './observe.mjs';

const y = (g, ...s) => console.log(s.map((v, i) =>
  (g[i] < 0 ? String(v).padEnd(-g[i]) : String(v).padStart(g[i]))).join('').trimEnd());

console.log(`${ITEM.length} items (${ITEM.filter((k) => k.bad).length} bad), ` +
  `${Object.keys(HINT).length} environment hints, ${Object.keys(ENVIRONMENT).length} analysis environments`);
console.log(`hint never hidden in any environment: ` +
  Object.keys(HINT).filter((i) => !Object.values(ENVIRONMENT).some((o) => o.hides.includes(i))).join(', '));

const A = [-14, 9, 10, 10, 12, 10, 8, 14, 16];
console.log('\n1. result per environment');
y(A, 'environment', 'window', 'hidden', 'observed', 'suppressed', 'evidence', 'clean', 'undetermined', 'false evidence');
for (const o of Object.keys(ENVIRONMENT)) {
  const s = measureEnvironment(o);
  y(A, o, ENVIRONMENT[o].window, s.hidden, s.observed, s.suppressed, s.evidence, s.clean, s.undetermined,
    s.falseEvidence);
}

const B = [-7, 8, 23, 23, 23];
console.log('\n2. decision per item');
y(B, 'item', 'actual', ...Object.keys(ENVIRONMENT));
for (const k of ITEM)
  y(B, k.name, k.bad ? 'bad' : 'clean', ...Object.keys(ENVIRONMENT).map((o) => {
    const r = observe(k, o);
    return r.hidden ? `${r.decision} (hidden)` : r.decision;
  }));

const C = [-14, 8, 12, 11, 11, 13, 16];
console.log('\n3. cost and trace');
y(C, 'environment', 'prep', 'observe min', 'manual min', 'total min', 'escaped bad', 'external trace');
for (const o of Object.keys(ENVIRONMENT)) {
  const s = measureEnvironment(o);
  y(C, o, s.prep, s.observeMin, s.manualMin, s.totalMin, s.escaped, s.externalTrace);
}

const h = measureEnvironment('stock-image'), g = measureEnvironment('near-real');
console.log(`\nitems resolved         ${h.evidence + h.clean} -> ${g.evidence + g.clean} / ${ITEM.length}`);
console.log(`items detecting env     ${h.hidden} -> ${g.hidden} / ${ITEM.length}`);
console.log(`records produced on target 0 (analysis does not run in the target environment)`);
console.log(`manual review time      ${MANUAL_MIN} min/item`);

14 items (7 bad), 6 environment hints, 3 analysis environments
hint never hidden in any environment: uniform-network-response

1. result per environment
environment      window    hidden  observed  suppressed  evidence   clean  undetermined  false evidence
stock-image           5         5        22           7         3       4             7               1
refurbished          20         4        23           6         4       4             6               1
near-real            60         2        27           2         6       6             2               1

2. decision per item
item     actual            stock-image            refurbished              near-real
k01         bad               evidence               evidence               evidence
k02         bad  undetermined (hidden)               evidence               evidence
k03         bad  undetermined (hidden)  undetermined (hidden)               evidence
k04         bad  undetermined (hidden)  undetermined (hidden)               evidence
k05         bad  undetermined (hidden)  undetermined (hidden)  undetermined (hidden)
k06         bad               evidence               evidence               evidence
k07         bad  undetermined (hidden)  undetermined (hidden)  undetermined (hidden)
k08       clean                  clean                  clean                  clean
k09       clean                  clean                  clean                  clean
k10       clean                  clean                  clean                  clean
k11       clean               evidence               evidence               evidence
k12       clean           undetermined           undetermined                  clean
k13       clean           undetermined           undetermined                  clean
k14       clean                  clean                  clean                  clean

3. cost and trace
environment       prep observe min manual min  total min  escaped bad  external trace
stock-image          8          70        210        288            5               3
refurbished         35         280        180        495            4               3
near-real           65         840         60        965            2               5

items resolved         7 -> 12 / 14
items detecting env     5 -> 2 / 14
records produced on target 0 (analysis does not run in the target environment)
manual review time      30 min/item

The Suppressed Behavior

The hidden column of the first table is this lesson’s measure. On a stock image, five of fourteen items notice the environment and never show their decisive behavior. As the environment is refurbished this number drops to 4, then to 2. While observed behavior rises from 22 to 27, suppressed behavior falls from 7 to 2.

Hiding is a different thing from a short observation window, and the two need to be told apart. If the window is short, the behavior has not been seen yet; lengthening it fixes that. If the item is hiding, the behavior will never be seen; lengthening the window changes nothing, the hint has to be hidden. The second table reads this distinction item by item: k04 has a behavior with a forty-five-minute delay and checks the clock hint; seeing it requires both the window to reach sixty minutes and the accelerated clock to be hidden — one without the other is not enough. By contrast, k12 and k13 check nothing at all; for them, lengthening the window alone is enough, and both come out clean in the widest environment.

Which prep step unlocks which item is also read from this table, and the gain is not distributed evenly. Moving from the stock image to the refurbished environment asks for 27 extra minutes of prep and turns a single item — k02, which checks both device count and tool traces — into evidence. Moving from refurbished to near-real asks for 30 extra minutes of prep and a window three times as long; in return, k03 and k04 turn into evidence, and k12 and k13 turn clean. The second step does four times the work for a prep cost only slightly above the first, but it does so by pushing the window from 280 minutes to 840 minutes. Prep is cheap, the window is expensive, and they are two separate line items.

Two items cannot be resolved in any environment: k05 and k07. Both check the uniformity of network responses, and that hint is not hidden in any environment. Two of the seven bad items stay undetermined regardless of the analysis environment’s quality. These two items get handed off to manual review and appear in the report with the line “analysis could not resolve this.”

Evidence, Clean, and False Evidence

The candidate-evidence gap is read here as three values. Fourteen items arrive at analysis as candidates. On the stock image, 3 turn into evidence, 4 come out clean, 7 stay undetermined. In the best environment there are 6 evidence, 6 clean, and 2 undetermined. The number of resolved items goes from 7 to 12.

Items that come out clean are worth as much as evidence. If all of an item’s behaviors were seen and none was decisive, that item drops out of the queue — this is the only mechanism that shortens the candidate list. But the condition for a “clean” decision is strict: the item must not be hidden and none of its behaviors may have been suppressed. If even a single behavior was suppressed, the decision goes to undetermined, not clean. This strictness is deliberate, because relaxing it would let k05 and k07 quietly count as clean.

An error in the opposite direction stands in every environment: the false evidence column is 1 in all three. A legitimate item shows a behavior counted as decisive, and analysis flags it as evidence. Raising the prep, lengthening the window, and hiding hints do not change this line, because what is missing is not observation, it is context. The sandbox sees the behavior; it does not see whether that behavior belongs there, what job the item was set up to do. That is why analysis output alone is not a decision — it is input to one.

Cost and the Trace Left on the Other Side

The third table gives the cost, and it rises in one direction only. Prep goes from 8 minutes to 65; observation goes from 70 to 840. Manual review goes the other way — from 210 minutes to 60 — but the drop does not offset the rise: the total goes from 288 minutes to 965 minutes, more than tripling. In return, the number of resolved items rises by five, and the number of escaped bad items falls from 5 to 2. The cost paid per item is roughly 135 minutes.

The last column is a kind of trace seen for the first time in this course. Records produced on target systems are zero; analysis does not run in the target environment, none of its requests reach the target. But the number of items that attempt an external connection during observation rises from 3 to 5, and those attempts go somewhere. Even if the controlled network does not let them reach a real target, the attempt itself and its timing can leak outward in a way that tells the other side something. The closer the analysis environment gets to reality, the more behavior an item shows; and every behavior it shows is also a sign that it is being examined.

This splits the course’s trace column in two: the trace the defense sees, and the trace the other side can see. The two point in opposite directions. The trace the defense sees is a sign that the assessment is running transparently inside the organization, and its excess only creates noise. The trace left on the other side is irreversible: learning that an item is being examined is enough for whoever is using it to change their behavior, and it corrupts every measurement after that. The next lesson measures the second one far more sharply.

Summary

• Of fourteen items, five notice the environment on a stock image and never show their decisive behavior; as the environment is refurbished this number falls to 2, and observed behavior rises from 22 to 27.
• Hiding and a short window are separate problems: the first is solved by hiding a hint, the second by lengthening the window, and some items need both at once.
• One environment hint cannot be hidden in any environment, because hiding it requires the item to reach a real target, which is out of scope; two of the seven bad items stay undetermined in every environment for this reason.
• The number of resolved items goes from 7 to 12; items that come out clean are the only mechanism that shortens the candidate list, and its condition is strict.
• False evidence is 1 in all three environments; what is missing is not observation, it is context, and prep does not close that gap.
• Total time goes from 288 minutes to 965 minutes; records produced on target are zero, but the number of items attempting an external connection rises from 3 to 5, leaving a trace on the other side.

Next Step

This lesson measured a single analysis environment’s decision, and two items stayed unresolved. There is a second way to resolve an item: ask the same item to a large number of independent engines and combine the results that come back. This route brings analysis time down from minutes to seconds, and in return raises a new question — how to combine results that contradict each other.

The next lesson takes up multi-engine reputation services on this axis. The same set of items is run through a large number of engines, the false-positive and false-negative rate of the majority decision is measured threshold by threshold, and the cost of trusting a single engine is counted. Alongside this, a privacy cost is computed: sending a file to an outside service is publishing it, and a portion of the items in the fictional measurement network carry subscriber data.