Black, White and Grey Box

The previous lesson asked how far a test reaches. There is an independent second question: what did the person who wrote the test look at? The three tests there were also derived from the specification — the upper-limit rule was read and asked about. Had they been written by looking into the code, different questions would have come out, because the code can have paths that are never in the specification.

This lesson establishes that information-level distinction and shows, on the same function, that the two approaches find different kinds of defects.

The Information-Level Distinction

Black-box testing uses only what is visible from the outside: the specification, the interface, input and output. How the code is written is unknown. This is why it is also called specification-based testing.

White-box testing is written by looking at the source code: which conditions exist, which branches exist, which loop bounds exist? The name structure-based testing describes this view better — its criterion is the structure of the program.

Grey-box testing lies between the two. The source code is not read, but part of the internal structure is known: the data schema, the record format, log entries, error codes. This partial knowledge guides input selection.

The distinction is independent of level. A unit test can be written as black-box or white-box; the same holds for a system test.

The Traced Fee Computation

For the comparison to be measurable, the fee computation is written in a form that records which branches run. The TRACE set holds the numbers of the branches that ran.

// fee-traced.mjs — the fee computation that records which branches ran
export const BRANCHES = ['D1', 'D2', 'D3', 'D4'];
export const TRACE = new Set();

export function lateFee(lateDays, memberNo, offBranch = false) {
  if (memberNo === 0) { TRACE.add('D1'); return 0; }
  TRACE.add('D2');
  let fee = Math.max(lateDays - 3, 0) * 2;
  if (fee > 20) { TRACE.add('D3'); fee = 20; } else { TRACE.add('D4'); }
  return fee;
}

The specification has four items: the first three days are free, every following day costs 2 units, the fee does not exceed 20 units, and an off-branch return incurs a 5-unit surcharge. The last item is the library’s new decision and carries the number K12.

What Black-Box Finds

Black-box checks are derived only from these four items. Because the memberNo condition inside the code is unknown, no check targets it.

// black-box.mjs — checks derived only from the specification
import assert from 'node:assert/strict';
import { lateFee, BRANCHES, TRACE } from './fee-traced.mjs';

const CHECKS = {
  'K7a first 3 days are free': () => assert.equal(lateFee(3, 41), 0),
  'K7b 2 units per day': () => assert.equal(lateFee(6, 41), 6),
  'K7c fee does not exceed 20 units': () => assert.equal(lateFee(100, 41), 20),
  'K12 off-branch return incurs a 5-unit surcharge': () => assert.equal(lateFee(4, 41, true), 7),
};

let passed = 0;
const failed = [];
for (const [name, check] of Object.entries(CHECKS)) {
  try { check(); passed += 1; } catch { failed.push(name); }
}

console.log(`black-box      : ${passed}/${Object.keys(CHECKS).length} checks passed`);
for (const name of failed) console.log(`  failed: ${name}`);
console.log(`branch coverage: ${TRACE.size}/${BRANCHES.length} branches ran`);
console.log(`  unreached: ${BRANCHES.filter((d) => !TRACE.has(d)).join(', ')}`);

black-box      : 3/4 checks passed
  failed: K12 off-branch return incurs a 5-unit surcharge
branch coverage: 3/4 branches ran
  unreached: D1

Black-box found a function that does not exist at all in the code: the off-branch return rule exists in the specification but not in the implementation. White-box cannot find this defect by its very structure — a test cannot be written for a branch that does not exist. A missing function is visible only by comparing against the specification.

The last line of the output, however, shows black-box’s blind spot: one of the four branches never ran. A branch with no counterpart in the specification is not the target of any check derived from the specification. The role of the branch coverage measurement here is not to find a defect but to name the region nobody is looking at.

What White-Box Finds

Someone looking at the source code sees the D1 branch and writes a check that targets it directly.

// white-box.mjs — a check targeting the unreached branch seen from the source code
import assert from 'node:assert/strict';
import { lateFee, BRANCHES, TRACE } from './fee-traced.mjs';

const fee = lateFee(100, 0);
try {
  assert.ok(fee === 20, `fee for member number 0 is ${fee}, expected 20`);
  console.log('D1 check: passed');
} catch (error) {
  console.log(`D1 check: failed — ${error.message}`);
}
console.log(`branch coverage: ${TRACE.size}/${BRANCHES.length} branches ran`);

D1 check: failed — fee for member number 0 is 0, expected 20
branch coverage: 1/4 branches ran

What it finds is a processing shortcut: member number zero is never charged a fee at all. A branch like this is usually code that was added during some experiment and never removed; because it has no counterpart in the specification, no black-box check ever asks about it, and it settles into the product unnoticed.

The second line of the same output shows white-box’s own blind spot. A set that targets a single branch runs only that branch; tests written by looking at structure cover the code’s existing paths, not its missing ones. Work that takes structure as its criterion never looks for a rule that was never written.

Grey-Box Uses the Knowledge In Between

In the grey-box approach the source code is not read, but the schema of member records is known: the member number is an integer starting from zero. This knowledge is enough to guide input selection.

// grey-box.mjs — probes derived from knowledge of the data schema
import { lateFee, BRANCHES, TRACE } from './fee-traced.mjs';

const MEMBER_NO_RANGE = { min: 0, max: 9999 };
const DELAY_SAMPLES = [0, 4, 100];

const deviations = [];
for (const memberNo of [MEMBER_NO_RANGE.min, MEMBER_NO_RANGE.max]) {
  for (const days of DELAY_SAMPLES) {
    const expected = Math.min(Math.max(days - 3, 0) * 2, 20);
    const observed = lateFee(days, memberNo);
    if (observed !== expected) deviations.push(`memberNo=${memberNo} days=${days}: ${observed} (expected ${expected})`);
  }
}

console.log(`grey-box       : ${deviations.length} deviations found`);
for (const s of deviations) console.log(`  ${s}`);
console.log(`branch coverage: ${TRACE.size}/${BRANCHES.length} branches ran`);

grey-box       : 2 deviations found
  memberNo=0 days=4: 0 (expected 2)
  memberNo=0 days=100: 0 (expected 20)
branch coverage: 4/4 branches ran

Schema knowledge gives no access to the source code; it only says which inputs are boundaries. Knowing that zero is the low end of a member-number range is enough to try that value, and the undocumented branch surfaces without ever looking at the source code. All four of the four branches ran.

Which Approach, When

The three approaches do not replace one another; they close different gaps.

Black-box finds the gap between the specification and the product, and it is unaffected by changes to the code; these tests survive when the code is refactored. It is the only valid approach at the acceptance level and at external interfaces, because the criterion is what the user sees.

White-box finds paths with no counterpart in the specification and names coverage gaps. Its cost is being tied to the structure of the code: when the structure changes, the test itself changes.

Grey-box works from the external interface while using internal knowledge to select input. This is the most common case in practice; the person writing the test usually knows the schema, the error codes, and the log entries.

The practical criterion is this: the body of the checks is written from the specification, and the coverage measurement is read from the structure. When a coverage gap is seen, the question to ask should not be “should I write a test here” but “does this branch have a counterpart in the specification.” If it does not, what should be written is not a test but the removal of that branch.

Summary

• Black-box relies on the specification, white-box on the structure of the source code, grey-box on partial internal knowledge such as schema and record format; the distinction is independent of test level.
• Black-box finds a function that does not exist in the code at all; white-box cannot see this defect because it cannot write a test for a branch that does not exist.
• White-box finds branches with no counterpart in the specification; in the example, the undocumented shortcut that leaves member number zero charged no fee at all surfaced this way.
• The branch coverage measurement does not find defects; it names the region nobody is looking at.
• The right question for a coverage gap is not “should I write a test” but “does this branch have a counterpart in the specification.”

Next Step

The level and the information level determine where a test sits, not its purpose. The same black-box unit test can be run to see whether the release is standing, or to check whether a fixed defect has come back. The test is the same in both cases; what changes is why it is run and what is done with its result. The next lesson names tests by this purpose: smoke, sanity, regression, and acceptance.