Flaky Tests

In the previous lesson, tests were written without looking inside the code: input was given, the return value was checked. In the sixth lesson, by contrast, spies and mocks looked inside the code — they verified which call was made. Both approaches can verify the same behavior, but they do not behave the same way in the face of the same change.

A test being flaky means it fails even though the behavior it tests has not changed. Flakiness has two sources: the test binding to something non-deterministic, and the test binding to the implementation. The fourth and fifth lessons showed the first. This lesson measures the second with a refactor.

The Unit Under Test

The reservation queue holds members waiting for a book that is out on loan. Its contract has four clauses: a member is added to the queue and learns their waiting position, the book is assigned to the first member in the queue, the same member cannot join the queue twice, and a notification goes to the member on every state change. Every operation is also written to a log.

// reservation.mjs — version 1: array-based reservation queue
export function createReservationQueue({ notification, log }) {
  const queue = [];
  return {
    queue,

    add(memberId) {
      if (queue.includes(memberId)) throw new Error(`member already in queue: ${memberId}`);
      queue.push(memberId);
      log.write(`add ${memberId} position ${queue.length}`);
      notification.send(memberId, `queue: ${queue.length}`);
      return queue.length;
    },

    next() {
      const memberId = queue.shift();
      if (memberId === undefined) return undefined;
      log.write(`next ${memberId}`);
      notification.send(memberId, 'book assigned');
      return memberId;
    },

    length: () => queue.length,
  };
}

// spy.mjs — small call-recording test double
export function spy(...methods) {
  const calls = [];
  const obj = { calls };
  for (const name of methods) obj[name] = (...args) => calls.push([name, ...args]);
  return obj;
}

Two Test Sets

The first set tests only what can be observed from outside: return values, the thrown error, and the notification sent to the member. The notification sent to the member is a side effect and is verified with a spy — but because it is part of the contract, it counts as behavior.

// behavior.test.mjs — tests only the externally observed behavior
import { test } from 'node:test';
import assert from 'node:assert/strict';
import { createReservationQueue } from './reservation.mjs';
import { spy } from './spy.mjs';

const setup = () => {
  const notification = spy('send');
  return { queue: createReservationQueue({ notification, log: spy('write') }), notification };
};

test('the order of adding determines the waiting position', () => {
  const { queue } = setup();

  assert.equal(queue.add('U-17'), 1);
  assert.equal(queue.add('U-42'), 2);
});

test('the book is assigned to the first member in the queue', () => {
  const { queue } = setup();
  queue.add('U-17');
  queue.add('U-42');

  assert.equal(queue.next(), 'U-17');
  assert.equal(queue.next(), 'U-42');
  assert.equal(queue.next(), undefined);
});

test('the same member cannot join the queue twice', () => {
  const { queue } = setup();
  queue.add('U-17');

  assert.throws(() => queue.add('U-17'), /member already in queue: U-17/);
  assert.equal(queue.length(), 1);
});

test('a message telling the member their position is sent', () => {
  const { queue, notification } = setup();

  queue.add('U-17');
  queue.next();

  assert.deepEqual(notification.calls, [
    ['send', 'U-17', 'queue: 1'],
    ['send', 'U-17', 'book assigned'],
  ]);
});

The second set tests the same unit, but binds to the internal structure and the log text. Both are outside the contract: whether the queue is kept as an array or a map, and which sentence is written to the log, are of no concern to the member.

// implementation.test.mjs — tests bound to internal structure and log text
import { test } from 'node:test';
import assert from 'node:assert/strict';
import { createReservationQueue } from './reservation.mjs';
import { spy } from './spy.mjs';

const setup = () => {
  const log = spy('write');
  return { queue: createReservationQueue({ notification: spy('send'), log }), log };
};

test("the queue's internal array keeps the order of adding", () => {
  const { queue } = setup();

  queue.add('U-17');
  queue.add('U-42');

  assert.deepEqual(queue.queue, ['U-17', 'U-42']);
});

test('the lines written to the log have a specific format', () => {
  const { queue, log } = setup();

  queue.add('U-17');
  queue.next();

  assert.deepEqual(log.calls, [
    ['write', 'add U-17 position 1'],
    ['write', 'next U-17'],
  ]);
});

node --test --test-reporter=tap behavior.test.mjs implementation.test.mjs | grep -E '^ *(ok|not ok|# (tests|pass|fail))'

ok 1 - the order of adding determines the waiting position
ok 2 - the book is assigned to the first member in the queue
ok 3 - the same member cannot join the queue twice
ok 4 - a message telling the member their position is sent
ok 5 - the queue's internal array keeps the order of adding
ok 6 - the lines written to the log have a specific format
# tests 6
# pass 6
# fail 0

All six tests are green. At this point the difference between the two sets is invisible; both appear to verify the same code.

The Refactor

The queue is moved to a map, the internal array is no longer exposed, and the log lines are fit into a single pattern. All four clauses of the contract are preserved exactly: the return values, the thrown error, and the messages sent to the member do not change.

// reservation.mjs — version 2: queue moved to a map, log format collapsed to one pattern
export function createReservationQueue({ notification, log }) {
  const entries = new Map();
  let sequence = 0;
  const queued = () => [...entries.entries()]
    .sort((a, b) => a[1] - b[1])
    .map(([memberId]) => memberId);

  return {
    add(memberId) {
      if (entries.has(memberId)) throw new Error(`member already in queue: ${memberId}`);
      sequence += 1;
      entries.set(memberId, sequence);
      const position = entries.size;
      log.write(`reservation member=${memberId} status=added position=${position}`);
      notification.send(memberId, `queue: ${position}`);
      return position;
    },

    next() {
      const [first] = queued();
      if (first === undefined) return undefined;
      entries.delete(first);
      log.write(`reservation member=${first} status=assigned`);
      notification.send(first, 'book assigned');
      return first;
    },

    length: () => entries.size,
  };
}

Same two files, same command, a refactored library.

node --test --test-reporter=tap behavior.test.mjs implementation.test.mjs | grep -E '^ *(ok|not ok|# (tests|pass|fail))'

ok 1 - the order of adding determines the waiting position
ok 2 - the book is assigned to the first member in the queue
ok 3 - the same member cannot join the queue twice
ok 4 - a message telling the member their position is sent
not ok 5 - the queue's internal array keeps the order of adding
not ok 6 - the lines written to the log have a specific format
# tests 6
# pass 4
# fail 2

The measurement is clean: four behavior tests stayed green, two implementation tests failed. The library’s contract did not change — no member sees a different result. The two failing tests did not report a regression; they reported that their own assumptions had broken.

This is the cost of flakiness. Refactoring means improving the internal structure while preserving behavior; a test suite that fails every time this is done makes refactoring expensive. Once it is expensive enough, it stops happening, and the code becomes unimprovable because of the very tests meant to protect it.

The boundary question that follows is this: is an assertion bound to something that is part of the contract? The fourth behavior test also uses a spy, and it too verifies a side effect — but the notification sent to the member is in the contract. What draws the line is not the type of tool but whether the thing being verified is something someone else observes.

The Measure of Flakiness

The other source of flakiness is non-deterministic input, and there the measure is the flake rate: the percentage of drops across a large number of runs of the same test on the same code.

// flake-rate.mjs — a flaky test's flake rate measured with a fixed seed
function generator(seed) {
  let state = (seed * 2654435761) % 2147483647;
  return () => {
    state = (state * 48271) % 2147483647;
    return state / 2147483647;
  };
}

// A batch job's duration varies by machine and load: between forty and sixty-nine.
const durationFor = (random) => 40 + Math.floor(random() * 30);

function budgetBoundTest(random) {
  const duration = durationFor(random);
  if (duration > 68) throw new Error(`duration budget exceeded: ${duration}`);
}

function behaviorBoundTest(random) {
  const duration = durationFor(random);
  if (Number.isInteger(duration) === false || duration < 0) throw new Error(`invalid duration: ${duration}`);
}

const RUNS = 1000;
for (const [label, testFn] of [['duration-budget-bound', budgetBoundTest], ['behavior-bound', behaviorBoundTest]]) {
  const random = generator(20);
  let failed = 0;
  for (let i = 0; i < RUNS; i += 1) {
    try {
      testFn(random);
    } catch {
      failed += 1;
    }
  }
  const rate = ((failed / RUNS) * 100).toFixed(1);
  console.log(`${label.padEnd(21)} ${RUNS} runs, ${failed} drops, flake rate ${rate}%`);
}

duration-budget-bound 1000 runs, 27 drops, flake rate 2.7%
behavior-bound        1000 runs, 0 drops, flake rate 0.0%

A rate of two point seven percent is dangerous precisely because it is hard to notice. A test that is green in forty-nine runs out of fifty gets called “flaky now and then,” and the run where it fails gets re-run. At that point the meaning of the suite breaks down: a green run no longer says “the code is correct” — it starts saying “we got lucky this time.”

Once the flake rate is measured, the work to do is clear. The flaky test is put into quarantine — removed from the main run so the remaining tests’ signal stays clean — and its source is fixed. In the example above, the fix is to stop loading a duration budget onto a unit test; duration is a performance metric and is measured separately.

Summary

• A flaky test is a test that fails even though the behavior it tests has not changed; its two sources are non-deterministic input and binding to the implementation.
• When the queue array was moved to a map and the log format changed, four behavior tests stayed green and two implementation tests failed; the library’s contract had not changed at all.
• Implementation-bound tests make refactoring expensive; once expensive enough, the code becomes unimprovable because of the very tests meant to protect it.
• The question that draws the line is scope, not tool: is the thing being verified part of the contract, or an internal detail?
• The measure of non-deterministic flakiness is the flake rate; in the measurement, the test bound to the duration budget failed in twenty-seven of a thousand runs, and the test bound to behavior never failed.

Next Step

The Unit Testing Practice topic closes here. Up to this point, in every test the code came first: a bug report, a rule, or a refactor found an implementation already there, ready to be tested. The next topic reverses this order. When the test is written first, the test determines what code gets written; a cycle that starts with a red run ends, after reaching green, with a refactor. The next lesson runs these three steps one at a time.