Test Types

The previous lesson turned scope into numbers: 20 items in scope, 11 items on the border, a net window of 70 person-hours. That calculation wrote down what could be done, not how it would be done. The same 20 items can be assessed with no prior information at all, with credentials supplied, or with configuration and source access opened up. The three paths do not carry the same name and do not produce the same result.

This lesson compares the three paths on a single target. Black-box assessment starts with no internal information at all, gray box receives credentials and limited documentation, white box sees the configuration and the source. The measure of comparison is not the number of findings; the number of findings varies from type to type in a misleading way. The measure is the gap between the candidates produced and the findings that turn into evidence.

• TP7. The assessment runs on the 20 in-scope items from the previous lesson. Both the item set and the flaw set are fictional.
• TP8. Each flaw carries two separate access levels: the level required to see it and the level required to prove it. The levels are ordered as external, credentialed and internal.
• TP9. Some flaws have a proof level of impact: their evidence can only be obtained through an action that would affect the service. Because the rules of engagement forbid that action, no type can reach this level.
• TP10. Each item carries a noise base: the number of signals counted as candidates before validation. The less internal information a type sees, the higher its noise multiplier.
• TP11. Time is made up of two components: setup per item and validation per candidate. All durations are fictional and given in person-hours.
• TP12. Trace is the number of records a step produces on the defense side. The trace multiplier per candidate depends on the type; a type that proceeds by reading documentation produces fewer records.

Three Types on the Same Target

// types.mjs — candidate production of black, gray and white box assessment on the same target.
// THIS IS A MODEL; both the item set and the flaw set are fictional.
export const SEED = 20260811, ITEM = 20;
const makeRng = (t) => { let s = t >>> 0;
  return () => (s = (s * 1664525 + 1013904223) >>> 0) / 4294967296; };
const r = makeRng(SEED);

// Each flaw carries two separate access levels: the one required to SEE it and the one
// required to PROVE it. The proof level can exceed the visibility level; this is where
// the candidate-versus-evidence gap comes from.
const LEVEL = ["external", "credentialed", "internal"];
export const FLAW = [];
for (let k = 0; k < ITEM; k++) {
  const n = 1 + Math.floor(r() * 3);                // 1-3 real flaws per item
  for (let j = 0; j < n; j++) {
    const g = Math.floor(r() * 3);
    // "impact": a flaw whose evidence can only be obtained through an action that would
    // affect the service. Because the rules of engagement forbid this, no type can reach it.
    const proof = r() < 0.15 ? "impact" : LEVEL[Math.min(2, g + (r() < 0.42 ? 1 : 0))];
    FLAW.push({ item: k, visible: LEVEL[g], proof });
  }
}
// Noise base per item: the number of signals counted as a candidate before validation.
export const NOISE = Array.from({ length: ITEM }, () => 1 + Math.floor(r() * 5));

// Type definition: the levels it accesses, the noise multiplier, setup time per item and
// validation time per candidate (hours), trace left on the defense side per candidate.
export const TYPE = [
  { name: "black box", access: ["external"], noise: 1.0, setup: 0.5, validation: 0.55, trace: 6.0 },
  { name: "gray box", access: ["external", "credentialed"], noise: 0.6, setup: 1.1,
    validation: 0.40, trace: 3.5 },
  { name: "white box", access: ["external", "credentialed", "internal"], noise: 0.3, setup: 2.0,
    validation: 0.30, trace: 1.5 },
];

// A type is measured over the given item set; if no list is given, the full set is used.
export function measure(t, items = [...Array(ITEM).keys()]) {
  const set = new Set(items);
  const flaws = FLAW.filter((a) => set.has(a.item));
  const visible = flaws.filter((a) => t.access.includes(a.visible));
  const evidenced = visible.filter((a) => t.access.includes(a.proof));
  const falsePositive = items.reduce((s, k) => s + Math.round(NOISE[k] * t.noise), 0);
  const candidates = visible.length + falsePositive;
  const time = items.length * t.setup + candidates * t.validation;
  return { candidates, evidenced: evidenced.length, falsePositive,
    pending: visible.length - evidenced.length, missed: flaws.length - visible.length, time,
    trace: Math.round(candidates * t.trace) };
}

if (import.meta.url === `file://${process.argv[1]}`) {
  const gen = (B) => (h) => h.map((v, j) => (j ? String(v).padStart(B[j])
    : String(v).padEnd(B[0]))).join("");
  const yz = gen([13, 7, 8, 9, 9, 9, 9, 11]);
  console.log(`model (seed ${SEED}): ${ITEM} in-scope items, ${FLAW.length} real flaws`);
  const d = (k) => FLAW.filter((a) => a.visible === k).length;
  console.log(`by visibility level: external ${d("external")}, credentialed ${d("credentialed")}, ` +
    `internal ${d("internal")}; in ${FLAW.filter((a) => a.visible !== a.proof).length} flaws the ` +
    `proof level exceeds the visibility level`);
  console.log("\n" + yz(["type", "cand.", "evid.", "false+", "pending", "missed", "hours",
    "evid./hour"]));
  for (const t of TYPE) {
    const o = measure(t);
    console.log(yz([t.name, o.candidates, o.evidenced, o.falsePositive, o.pending, o.missed,
      o.time.toFixed(1), (o.evidenced / o.time).toFixed(2)]));
  }
  const [a, b, c] = TYPE.map((t) => measure(t));
  console.log(`\nevidence rate: black ${(100 * a.evidenced / a.candidates).toFixed(1)}%, gray ` +
    `${(100 * b.evidenced / b.candidates).toFixed(1)}%, white ${(100 * c.evidenced / c.candidates).toFixed(1)}%`);
}

model (seed 20260811): 20 in-scope items, 41 real flaws
by visibility level: external 16, credentialed 11, internal 14; in 19 flaws the proof level exceeds the visibility level

type           cand.   evid.   false+  pending   missed    hours evid./hour
black box         78       6       62       10       25     52.9       0.11
gray box          63      20       36        7       14     47.2       0.42
white box         60      31       19       10        0     58.0       0.53

evidence rate: black 7.7%, gray 31.7%, white 51.7%

The first column of the table is this course’s most misleading number. The type that produces the most candidates yields the fewest evidenced findings. Black box leaves 78 candidates: 62 of them are false positives, 10 are items that are seen but cannot be proven, and what remains is 6 evidenced findings — 7.7 percent. On the same target, white box turns 60 candidates into 31 evidenced findings. The gap between the “78 findings” written on a report’s cover and “60 findings” runs in the opposite direction from what the organization actually receives.

The source of this gap is in the second line: in 19 of the 41 flaws, the proof level is higher than the visibility level. Seeing a flaw and proving it require separate access, and every signal the reconnaissance stage produces stands on the wrong side of this distinction. Black box can see all 16 flaws visible from outside, but it can only reach the evidence for 6 of them from outside; the remaining 10 stay pending. A pending candidate that goes into the report is an unvalidated claim; if it is left out, the organization never hears about it at all. Neither option is correct on its own, and what this lesson measures is exactly the size of this third column.

Gray box’s place is also legible in this table. With 63 candidates, 20 evidenced findings and 14 missed flaws, it sits between the two extremes, but its real contribution is not in the numbers: gray box answers the question of how far an account with compromised credentials can reach. Black box cannot ask this question at all, because it has no access to the credentialed level; white box can ask it, but it reads the answer from the configuration, whereas the question belongs to runtime. Gray box’s 6 internal evidenced findings fall exactly into this class and mark the boundary of a credential-compromise scenario.

White box’s 10 pending items are a separate class. White box sees every level, so the 10 candidates it leaves pending are flaws whose evidence can only be obtained through an action that would affect the service. The rules of engagement forbid that action, and the moment the prohibition is lifted, what gets produced is not an assessment but an outage. These 10 items cannot be evidenced by any type; in the report they have to stand not as an “evidenced finding” but as a “candidate with its conditions written down.” The missed column shows the opposite: black box never sees 25 real flaws at all, gray box misses 14 of them, white box misses none.

The Cost of Access and the Meaning of Evidence

// compare.mjs — the trace types leave and the external reachability of evidence.
import { TYPE, ITEM, FLAW, measure } from "./types.mjs";

const gen = (B) => (h) => h.map((v, j) => (j ? String(v).padStart(B[j])
  : String(v).padEnd(B[0]))).join("");
const yz = gen([13, 8, 10, 13, 10, 10, 13]);
const full = [...Array(ITEM).keys()];

// How many of the evidenced flaws are visible at an externally-reachable level: evidence an
// attacker could reach without prior access is separated from evidence meaningful only from inside.
const spread = (t) => {
  const evidenced = FLAW.filter((a) => t.access.includes(a.visible) && t.access.includes(a.proof));
  return { external: evidenced.filter((a) => a.visible === "external").length,
    internal: evidenced.filter((a) => a.visible !== "external").length };
};
console.log(`${ITEM} items, ${FLAW.length} real flaws; trace is the record count on the defense side`);
console.log(yz(["type", "evid.", "trace", "trace/evid.", "external", "internal", "hours/evid."]));
for (const t of TYPE) {
  const o = measure(t), d = spread(t);
  console.log(yz([t.name, o.evidenced, o.trace, (o.trace / o.evidenced).toFixed(1), d.external, d.internal,
    (o.time / o.evidenced).toFixed(2)]));
}

// Mix: white box on the first m items, gray box on the remaining items.
const [, gray, white] = TYPE;
const mix = (m) => {
  const a = measure(white, full.slice(0, m)), b = measure(gray, full.slice(m));
  return { candidates: a.candidates + b.candidates, evidenced: a.evidenced + b.evidenced,
    pending: a.pending + b.pending, missed: a.missed + b.missed, time: a.time + b.time,
    trace: a.trace + b.trace };
};
const y2 = gen([17, 8, 8, 8, 8, 9, 10]);
console.log("\n" + y2(["mix white / gray", "cand.", "evid.", "pending", "missed", "hours",
  "trace"]));
for (let m = 0; m <= ITEM; m += 4) {
  const k = mix(m);
  console.log(y2([`${m} / ${ITEM - m}`, k.candidates, k.evidenced, k.pending, k.missed, k.time.toFixed(1),
    k.trace]));
}
const s = mix(0), b20 = mix(ITEM);
console.log(`\nmoving from gray box to white box adds ${(b20.time - s.time).toFixed(1)} hours, ` +
  `raises evidence from ${s.evidenced} to ${b20.evidenced}, and drops missed flaws from ${s.missed} to 0`);
console.log(`the same move brings the trace down from ${s.trace} to ${b20.trace}; pending candidates ` +
  `go from ${s.pending} to ${b20.pending}, because white box sees more flaws`);

20 items, 41 real flaws; trace is the record count on the defense side
type            evid.     trace  trace/evid.  external  internal  hours/evid.
black box           6       468         78.0         6         0         8.82
gray box           20       221         11.1        14         6         2.36
white box          31        90          2.9        14        17         1.87

mix white / gray    cand.   evid. pending  missed    hours     trace
0 / 20                 63      20       7      14     47.2       221
4 / 16                 64      24       7      10     49.9       199
8 / 12                 62      26       7       8     51.6       169
12 / 8                 63      30       7       4     54.2       145
16 / 4                 61      31       8       2     55.8       114
20 / 0                 60      31      10       0     58.0        90

moving from gray box to white box adds 10.8 hours, raises evidence from 20 to 31, and drops missed flaws from 14 to 0
the same move brings the trace down from 221 to 90; pending candidates go from 7 to 10, because white box sees more flaws

The time column corrects the previous lesson’s rough plan. There, a single 76-hour plan covered 20 items with no distinction by type; once the type is distinguished, the same 20 items range between 47.2 and 58.0 hours, and all three fit inside the 70-hour net window. So the choice is not a time constraint. The gap opens up when time is measured per evidenced finding: one evidenced finding takes 8.82 hours in black box and 1.87 hours in white box. Black box’s time does not go toward validation — it goes toward eliminating 62 false positives.

The trace column writes down the blue-team side. Black box leaves 468 records on the defense side, white box leaves 90. The ratio between them is 5.2, and the reason is not only the candidate count: black box has to test every candidate over the network, while white box closes off part of its candidates through documentation. Trace per evidenced finding drops from 78 to 2.9. This is the correct unit for measuring how much an assessment occupies the defense: not the total record count, but records per evidenced finding.

The last two columns show white box’s limit. 17 of the 31 evidenced findings are meaningful only from the inside: it has been shown that they exist in the configuration, but not that an external request can reach them. All 6 of black box’s 6 evidenced findings, by contrast, fall in the externally-reachable class. The two types answer different questions — one asks “does this flaw exist,” the other asks “can this flaw be reached from outside” — and in a remediation queue, the two cannot be given the same priority.

This distinction turns the choice of type from a preference into a question-matching exercise. If the question asked is “what does someone looking from outside see, and where can they reach,” black box is the right tool, and its 6 evidenced findings answer that question directly; the 468 records are separately valuable as a measure of whether the defense noticed this kind of attempt at all. If the question asked is “how many flaws does the inventory have, and which ones need to go into the remediation queue,” white box is the right tool, because it is the only type that zeroes out the missed column. Evaluating both under the same contract with the same metric amounts to writing the answers to two different questions into the same table.

The mix table shows the transition along a single axis. Every four items shifted to white box reduces the missed flaws and lowers the trace; in exchange, the time increases. The pending candidates do not move from seven across the first three transitions, then climb to eight and ten in the last two, because as more flaws become visible, more of the items whose evidence is forbidden also become visible. White box’s real cost does not show up in the table: the organization has to open its configuration and source to the test team, and that is not a time line item but an organizational decision. This is also why the table’s intermediate rows are useful — opening four or eight items instead of the whole inventory brings the missed flaws down from 14 to 10 and to 8, while keeping the amount of documentation opened at roughly a fifth of the total. Which items get opened, though, is a decision made on the business side, not the technical one.

Summary

• On the same 20 items, black box produces 78 candidates and leaves 6 evidenced findings, while white box turns 60 candidates into 31; the candidate count and the evidenced-finding count move in opposite directions.
• The source of the gap is that in 19 of the 41 flaws, the proof level is higher than the visibility level: seeing a flaw and proving it require separate access.
• 10 flaws cannot be evidenced by any type, because their evidence can only be obtained through an action that would affect the service and the rules of engagement forbid it; these items stand in the report not as evidenced findings but as candidates with their conditions written down.
• All three types fit inside the 70-hour window; the distinction is time per evidenced finding — 8.82 hours versus 1.87 hours.
• Black box leaves 468 records on the defense side, white box leaves 90; trace per evidenced finding drops from 78 to 2.9.
• 17 of white box’s 31 evidenced findings are meaningful only from the inside; all 6 of black box’s evidenced findings fall in the externally-reachable class, and the two types answer different questions.

Next Step

All three types share one common residue: false positives. In black box, 62 candidates do not correspond to a real flaw; in gray box, 36; in white box, 19 — and in this lesson these numbers were produced by a single multiplier. In reality, which candidate gets eliminated is determined not by a multiplier but by what evidence the candidate rests on: a candidate produced by reading version information and a candidate produced by observing behavior do not carry the same weight. The next lesson runs the candidate list through validation rules and counts the eliminated false positives one by one: how many candidates rest only on version information, how many are confirmed by behavior, and how much risk the validation itself adds to the target.