Fast and Independent Tests

Every test in the previous lesson was correct on its own, and stays correct as long as it stays on its own. As tests multiply, this condition does not hold by itself. If two tests use the same object, the same module variable, or the same file, the state the first one leaves behind mixes into the second one’s input. The result is that run order turns into an input.

Order dependence is an insidious defect, because most of the time it is invisible: as long as the runner runs the tests in the same order every time, the suite stays green. The dependence surfaces when a new test is added to the file or when tests are distributed in parallel, and at that moment it is not clear whether what broke is the code or the test. This lesson’s first task is to turn that order into a variable and make the dependence visible.

Turning Order into a Variable

The way to search for order dependence is to change run order in a controlled way. This does not need randomness; a permutation derived from a fixed seed is enough, and it is more useful for being repeatable. The linear congruential generator introduced in the Test Stability lesson in the Frontend Quality course is a sufficient source for this job.

// runner.mjs — a small runner with a fixed-seed order shuffle
export function generator(seed) {
  let state = (seed * 2654435761) % 2147483647;
  return () => {
    state = (state * 48271) % 2147483647;
    return state / 2147483647;
  };
}

export function shuffle(array, seed) {
  const random = generator(seed);
  const copy = [...array];
  for (let i = copy.length - 1; i > 0; i -= 1) {
    const j = Math.floor(random() * (i + 1));
    [copy[i], copy[j]] = [copy[j], copy[i]];
  }
  return copy;
}

export function run(tests, seed) {
  const order = shuffle(Object.keys(tests), seed);
  let passed = 0;
  const lines = [];
  for (const name of order) {
    try {
      tests[name]();
      passed += 1;
      lines.push(`  ok      ${name}`);
    } catch (error) {
      lines.push(`  not ok  ${name}  (${error.message.split('\n')[0]})`);
    }
  }
  console.log(`seed ${seed}: ${passed} passed, ${order.length - passed} failed`);
  console.log(lines.join('\n'));
}

The same seed always gives the same order; when a failing run is reported, sharing the seed is enough to reproduce the problem.

The Dependence Produced by Shared State

The catalog module below keeps loan records in a module-level map. The module loads once, and there is a single map for the life of the process.

// catalog.mjs — module-level shared loan record
const records = new Map();

export function addLoan(memberId, bookId) {
  if (records.has(bookId)) throw new Error(`book is already checked out: ${bookId}`);
  records.set(bookId, memberId);
}

export function returnBook(bookId) {
  records.delete(bookId);
}

export function openLoanCount() {
  return records.size;
}

Three tests exercise three separate rules of this module. Each one is correct read on its own.

// shared.mjs — run order determined by seed, over shared state
import assert from 'node:assert/strict';
import { run } from './runner.mjs';
import { addLoan, returnBook, openLoanCount } from './catalog.mjs';

const TESTS = {
  'a new loan appears in the catalog': () => {
    addLoan('U-17', 'K-903');
    assert.equal(openLoanCount(), 1);
  },
  'a returned book drops from the catalog': () => {
    addLoan('U-17', 'K-101');
    returnBook('K-101');
    assert.equal(openLoanCount(), 0);
  },
  'the same book cannot be loaned out twice': () => {
    addLoan('U-42', 'K-903');
    assert.throws(() => addLoan('U-58', 'K-903'), /already checked out/);
  },
};

run(TESTS, Number(process.argv[2] ?? 3));

Running it in two separate processes with two different seeds is enough. A separate process is needed because the module-level map lives for the process’s lifetime.

node shared.mjs 3
node shared.mjs 4

seed 3: 1 passed, 2 failed
  ok      a new loan appears in the catalog
  not ok  a returned book drops from the catalog  (Expected values to be strictly equal:)
  not ok  the same book cannot be loaned out twice  (book is already checked out: K-903)
seed 4: 2 passed, 1 failed
  ok      a returned book drops from the catalog
  ok      a new loan appears in the catalog
  not ok  the same book cannot be loaned out twice  (book is already checked out: K-903)

Same code, same three tests, two different results. A test that passes at seed three fails at seed four. There is no answer, in this table, to the question of which result is “correct”, because the tests are measuring not a rule but what ran before them.

Two separate forms of contamination show up. The a returned book drops from the catalog test expects an absolute number: it assumes the catalog is empty. The the same book cannot be loaned out twice test, conversely, assumes the catalog does not already contain the book it is about to add. Both carry an unwritten assumption about the starting state, and that assumption is set by whichever tests ran before them.

A Fresh Fixture

The fix is not to make the tests insensitive to order, but to have every test set up its own starting state. For this, instead of a single shared map, a factory that produces a fresh catalog on every call is used.

// catalog.mjs — version 2: a fresh catalog is produced on every call
export function createCatalog() {
  const records = new Map();
  return {
    addLoan(memberId, bookId) {
      if (records.has(bookId)) throw new Error(`book is already checked out: ${bookId}`);
      records.set(bookId, memberId);
    },
    returnBook(bookId) {
      records.delete(bookId);
    },
    openLoanCount() {
      return records.size;
    },
  };
}

This starting state a test sets up before it begins running is called a fixture. The test runner provides hooks to set up the fixture and, if needed, tear it down; the hook that runs before every test corresponds to setup, and the one that runs after every test corresponds to teardown.

// independent.test.mjs — a fresh fixture per test, registration order comes from the seed
import { test, beforeEach } from 'node:test';
import assert from 'node:assert/strict';
import { shuffle } from './runner.mjs';
import { createCatalog } from './catalog.mjs';

let catalog;
beforeEach(() => {
  catalog = createCatalog();
});

const TESTS = {
  'a new loan appears in the catalog': () => {
    catalog.addLoan('U-17', 'K-903');
    assert.equal(catalog.openLoanCount(), 1);
  },
  'a returned book drops from the catalog': () => {
    catalog.addLoan('U-17', 'K-101');
    catalog.returnBook('K-101');
    assert.equal(catalog.openLoanCount(), 0);
  },
  'the same book cannot be loaned out twice': () => {
    catalog.addLoan('U-42', 'K-903');
    assert.throws(() => catalog.addLoan('U-58', 'K-903'), /already checked out/);
  },
};

for (const name of shuffle(Object.keys(TESTS), Number(process.env.SEED ?? 3))) {
  test(name, TESTS[name]);
}

The tests now run with the built-in runner; the registration order comes from the seed, and the fixture is rebuilt before every test.

SEED=3 node --test --test-reporter=tap independent.test.mjs | grep -E '^ *(ok|not ok|# (tests|pass|fail))'
SEED=4 node --test --test-reporter=tap independent.test.mjs | grep -E '^ *(ok|not ok|# (tests|pass|fail))'

ok 1 - a new loan appears in the catalog
ok 2 - a returned book drops from the catalog
ok 3 - the same book cannot be loaned out twice
# tests 3
# pass 3
# fail 0
ok 1 - a returned book drops from the catalog
ok 2 - a new loan appears in the catalog
ok 3 - the same book cannot be loaned out twice
# tests 3
# pass 3
# fail 0

The order changed, the result did not. This is the measure of independence: once run order stops being an input, what a test reports is only the rule it tests.

The same principle has two practical consequences. Because no writable state remains shared between tests, the tests can be run in parallel. A failing test also fails when run on its own; reproducing it does not require running the whole suite.

A Fixture’s Share of the Duration

Independence and speed look like separate virtues, but they draw on the same source. Every test setting up its own fixture requires the fixture to be cheap. Setting up a map in memory takes a few microseconds; setting up the same fixture on disk requires stepping outside the process.

// speed.mjs — comparing an in-memory fixture with a disk-backed fixture
import { mkdtempSync, writeFileSync, readFileSync, rmSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';

const TEST_COUNT = 200;

function inMemoryFixture() {
  let fileOps = 0;
  for (let i = 0; i < TEST_COUNT; i += 1) {
    const records = new Map([['K-903', 'U-17']]);
    if (records.size !== 1) throw new Error('fixture broken');
  }
  return fileOps;
}

function diskBackedFixture() {
  let fileOps = 0;
  const root = mkdtempSync(join(tmpdir(), 'catalog-'));
  for (let i = 0; i < TEST_COUNT; i += 1) {
    const path = join(root, `catalog-${i}.json`);
    writeFileSync(path, JSON.stringify({ 'K-903': 'U-17' }));
    fileOps += 1;
    const records = new Map(Object.entries(JSON.parse(readFileSync(path, 'utf8'))));
    fileOps += 1;
    if (records.size !== 1) throw new Error('fixture broken');
  }
  rmSync(root, { recursive: true, force: true });
  return fileOps;
}

function measure(fn) {
  const start = performance.now();
  const fileOps = fn();
  return { duration: performance.now() - start, fileOps };
}

const memory = measure(inMemoryFixture);
const disk = measure(diskBackedFixture);

console.log(`in-memory fixture   : ${TEST_COUNT} tests, ${memory.fileOps} file operations`);
console.log(`disk-backed fixture : ${TEST_COUNT} tests, ${disk.fileOps} file operations`);
console.log(`disk-backed version took longer       : ${disk.duration > memory.duration}`);
console.log(`duration ratio is at least twentyfold : ${disk.duration / memory.duration >= 20}`);

in-memory fixture   : 200 tests, 0 file operations
disk-backed fixture : 200 tests, 400 file operations
disk-backed version took longer       : true
duration ratio is at least twentyfold : true

Absolute durations depend on the machine and the filesystem; that is why the output reports a comparison result instead of raw milliseconds. What does not change is the direction of the relationship: a suite of two hundred tests, at two file operations per test, produces four hundred out-of-process calls, and each of these calls carries a fixed baseline cost.

The rule that follows from this is not a speed target but a boundary definition. The first lesson said that the “unit” boundary keeps out-of-process resources out; this measurement shows the cost of that constraint. It does not mean tests that work with disk, network, or a database will not be written — those tests are called integration tests, and they run in a separate layer, less often.

Summary

• Shared writable state turns run order into one of the test’s inputs.
• A permutation derived from a fixed seed surfaces order dependence in a repeatable way; the same three tests gave two different results at two seeds.
• Contamination works in both directions: one test assumed the catalog was empty, another assumed it was not already occupied; the prior tests determined both assumptions.
• Every test setting up its own fixture makes order independent of the result and makes the tests runnable in parallel.
• Independence requires a cheap fixture: in a suite of two hundred tests, an out-of-process fixture added four hundred file operations and noticeably lengthened the run.

Next Step

In this lesson the fixture was in the test’s hands: the catalog object was produced by a factory call. When the code under test creates its dependency internally, this option disappears — a clock, a source of randomness, or a store the test cannot change stops being one of the test’s inputs. The next lesson writes the same business rule in two forms, compares the version that creates its dependency internally with the version it is supplied from outside, and measures the fixture a test sets up by its line count.