Functional Test Types

The level and the information level determine where a test sits, not its purpose. The same black-box unit test can be run to see whether the release is standing, or to check whether a fixed defect has come back. The test is the same in both cases; what changes is why it is run and what is done with its result.

This lesson names tests by their purpose. The function of the names is to build a shared understanding of when a set is run and what happens when it fails.

Purpose Names the Test

Smoke test is wide and shallow: it tries every main path once, testing none of them deeply. Its question is one sentence — is this release worth working on? The decision when it fails is also single: the release is rejected, and no detailed testing follows.

Sanity test is narrow and deep: it probes the region a specific change touches, in a few minutes. Its question is — does this fix look reasonable? It is a pre-filter done before running the full set.

Regression test checks whether behavior that used to work has broken. Its set accumulates over time: every fixed defect leaves behind a sentinel that will warn if it comes back.

Confirmation test checks that a single defect reported as fixed is actually fixed. It is a step-by-step repeat of the defect report, and its coverage is exactly that report.

Acceptance test is the condition of delivery, and its criterion comes from the party who will use the product; this is the place defined for it in the previous two lessons.

Smoke and Regression in the Same Release

Three versions of the library are set up: a sound version, a version with a core operation broken, and a version where a previously fixed defect has come back.

// version.mjs — the library's sound version and two broken versions
export function baseLibrary() {
  return {
    findMember: (no) => (no >= 0 ? { no, type: no % 3 === 0 ? 'student' : 'member' } : null),
    lateFee: (days) => Math.min(Math.max(days - 3, 0) * 2, 20),
    lend: (member, book) => ({ memberNo: member.no, book, status: 'lent' }),
  };
}

export const VERSIONS = {
  'S0 sound': baseLibrary(),
  'S1 core broken': { ...baseLibrary(), lend: () => { throw new Error('record could not be opened'); } },
  'S2 old defect returned': { ...baseLibrary(), lateFee: (days) => Math.max(days - 3, 0) * 2 },
};

The smoke set tries the three main paths once each. The regression set is the sentinel of three defect reports; each check’s name carries the number of the report it comes from.

// types.mjs — comparing the smoke and regression sets across three versions
import assert from 'node:assert/strict';
import { VERSIONS } from './version.mjs';

const SMOKE = {
  'a member record opens': (k) => assert.ok(k.findMember(41)),
  'a loan is issued': (k) => assert.equal(k.lend(k.findMember(41), 'K-1').status, 'lent'),
  'a fee is computed': (k) => assert.equal(typeof k.lateFee(5), 'number'),
};

const REGRESSION = {
  'H-14 fee does not exceed the upper limit': (k) => assert.equal(k.lateFee(100), 20),
  'H-22 grace period is three days': (k) => assert.equal(k.lateFee(3), 0),
  'H-31 an invalid member number returns null': (k) => assert.equal(k.findMember(-1), null),
};

function run(set, library) {
  const failed = [];
  for (const [name, check] of Object.entries(set)) {
    try { check(library); } catch { failed.push(name); }
  }
  return failed;
}

for (const [versionName, library] of Object.entries(VERSIONS)) {
  const line = [];
  for (const [setName, set] of [['smoke', SMOKE], ['regression', REGRESSION]]) {
    const failed = run(set, library);
    line.push(`${setName} ${Object.keys(set).length - failed.length}/${Object.keys(set).length}${failed.length ? ` (${failed[0]})` : ''}`);
  }
  console.log(`${versionName.padEnd(24)}: ${line.join(' | ')}`);
}

S0 sound                : smoke 3/3 | regression 3/3
S1 core broken          : smoke 2/3 (a loan is issued) | regression 3/3
S2 old defect returned  : smoke 3/3 | regression 2/3 (H-14 fee does not exceed the upper limit)

Two defects were caught by two separate sets. When the core operation breaks, the regression set stays quiet, because its sentinels sit in the region of past defects; lending a book never produced a defect there, so there is no sentinel for it. When the old defect comes back, the smoke set stays quiet, because it only asked whether the fee computation returned a number.

Both sets are correct in purpose, and both are incomplete. This incompleteness is not a defect; it follows from the definition: the shallow set does not know depth, the narrow set does not know breadth.

Confirmation Testing Alone Is Not Enough

A fix is made for the fee defect. The person writing the fix restores the upper limit, but drops the grace period.

// sanity.mjs — the outcome of confirmation, sanity, and regression sets on the same fix
import assert from 'node:assert/strict';
import { baseLibrary } from './version.mjs';

const fixed = { ...baseLibrary(), lateFee: (days) => Math.min(days * 2, 20) };

const CONFIRMATION_TEST = {
  'H-14 the state in the report': () => assert.equal(fixed.lateFee(100), 20),
};

const SANITY = {
  'fee at 0 days is 0': () => assert.equal(fixed.lateFee(0), 0),
  'fee at 3 days is 0': () => assert.equal(fixed.lateFee(3), 0),
  'fee at 4 days is 2': () => assert.equal(fixed.lateFee(4), 2),
  'fee at 13 days is 20': () => assert.equal(fixed.lateFee(13), 20),
  'fee at 14 days is 20': () => assert.equal(fixed.lateFee(14), 20),
};

const REGRESSION = {
  'H-14 fee does not exceed the upper limit': () => assert.equal(fixed.lateFee(100), 20),
  'H-22 grace period is three days': () => assert.equal(fixed.lateFee(3), 0),
  'H-31 an invalid member number returns null': () => assert.equal(fixed.findMember(-1), null),
};

for (const [name, set] of [['confirmation', CONFIRMATION_TEST], ['sanity test ', SANITY], ['regression  ', REGRESSION]]) {
  const failed = Object.entries(set).filter(([, d]) => { try { d(); return false; } catch { return true; } });
  console.log(`${name}: ${Object.keys(set).length - failed.length}/${Object.keys(set).length} passed${failed.length ? ` — first failed: ${failed[0][0]}` : ''}`);
}

confirmation: 1/1 passed
sanity test : 3/5 passed — first failed: fee at 3 days is 0
regression  : 2/3 passed — first failed: H-22 grace period is three days

Confirmation testing passed: the state in the defect report now gives the correct result. The report could be closed. The sanity test, however, showed a second breakage in the same region within a few seconds — the fix damaged its own surroundings.

Two rules follow from this. When a defect is fixed, not only the state in the report but also the boundary values of that rule are tested. And confirmation testing passing is not enough to close the report; the region of the fixed defect is also probed.

The Maintenance Cost of the Sets

The regression set grows in one direction only: every new defect adds a sentinel, and none of them leaves on its own. Over time, the set becomes slow enough to push the feedback point back.

There are three levers. Regression test selection runs only the checks that touch the changed region; the full set is left for a point further back. Pruning removes the sentinel for a rule that no longer has a counterpart — no rule, no sentinel. Pruning flaky tests protects trust as well: a check that fails for unrelated reasons turns, when it fails, into noise nobody looks at, and it lowers the value of the whole set.

The maintenance of the smoke set runs the opposite way: it must stay small. As a smoke set grows, it stops being able to do its own job, because giving a fast answer is its only job.

Summary

• Test types are separated by purpose: smoke is wide and shallow, sanity is narrow and deep, regression is the sentinel of past defects, confirmation testing is the repeat of a single report.
• In the example, only the smoke set caught the core breakage, and only the regression set caught the returning defect; the shortfall of both sets follows from their definitions.
• Confirmation testing passing shows the defect was fixed, not that the fix left its surroundings undamaged; in the example, the sanity test surfaced the second breakage.
• The regression set grows in one direction; selective runs, pruning checks with no remaining counterpart, and removing flaky tests are all parts of its maintenance.
• The maintenance of the smoke set is staying small; once it grows, it loses its function of giving a fast answer.

Next Step

So far, the inputs to the checks were chosen as examples: 3 days, 4 days, 100 days. The reason for the choice was intuition. There are systematic ways to select a finite subset from an infinite input set, and these ways justify which inputs are tried. The next lesson establishes these techniques: grouping inputs that behave the same into classes, probing the edges of the classes, and counting condition combinations exhaustively.