---
title: 'Flaky Tests'
source: 'https://academia.sh/en/courses/unit-testing/flaky-tests'
course: 'Unit Testing and Test-Driven Development'
language: en
updated: '2026-08-23T14:25:21+00:00'
license: 'CC BY-SA 4.0'
---

# Flaky Tests

Implementation-bound tests breaking under a refactor with no behavior change, behavior-bound tests staying green under the same refactor, and flakiness measured with the flake rate.

In the previous lesson, tests were written without looking inside the code: input was
given, the return value was checked. In the sixth lesson, by contrast, spies and mocks
looked inside the code — they verified which call was made. Both approaches can verify the
same behavior, but they do not behave the same way in the face of the same change.

A test being **flaky** means it fails even though the behavior it tests has not changed.
Flakiness has two sources: the test binding to something non-deterministic, and the test
binding to the implementation. The fourth and fifth lessons showed the first. This lesson
measures the second with a refactor.

## The Unit Under Test

The reservation queue holds members waiting for a book that is out on loan. Its contract
has four clauses: a member is added to the queue and learns their waiting position, the
book is assigned to the first member in the queue, the same member cannot join the queue
twice, and a notification goes to the member on every state change. Every operation is
also written to a log.

```js
// reservation.mjs — version 1: array-based reservation queue
export function createReservationQueue({ notification, log }) {
  const queue = [];
  return {
    queue,

    add(memberId) {
      if (queue.includes(memberId)) throw new Error(`member already in queue: ${memberId}`);
      queue.push(memberId);
      log.write(`add ${memberId} position ${queue.length}`);
      notification.send(memberId, `queue: ${queue.length}`);
      return queue.length;
    },

    next() {
      const memberId = queue.shift();
      if (memberId === undefined) return undefined;
      log.write(`next ${memberId}`);
      notification.send(memberId, 'book assigned');
      return memberId;
    },

    length: () => queue.length,
  };
}
```

```js
// spy.mjs — small call-recording test double
export function spy(...methods) {
  const calls = [];
  const obj = { calls };
  for (const name of methods) obj[name] = (...args) => calls.push([name, ...args]);
  return obj;
}
```

## Two Test Sets

The first set tests only what can be observed from outside: return values, the thrown
error, and the notification sent to the member. The notification sent to the member is a
side effect and is verified with a spy — but because it is part of the contract, it counts
as behavior.

```js
// behavior.test.mjs — tests only the externally observed behavior
import { test } from 'node:test';
import assert from 'node:assert/strict';
import { createReservationQueue } from './reservation.mjs';
import { spy } from './spy.mjs';

const setup = () => {
  const notification = spy('send');
  return { queue: createReservationQueue({ notification, log: spy('write') }), notification };
};

test('the order of adding determines the waiting position', () => {
  const { queue } = setup();

  assert.equal(queue.add('U-17'), 1);
  assert.equal(queue.add('U-42'), 2);
});

test('the book is assigned to the first member in the queue', () => {
  const { queue } = setup();
  queue.add('U-17');
  queue.add('U-42');

  assert.equal(queue.next(), 'U-17');
  assert.equal(queue.next(), 'U-42');
  assert.equal(queue.next(), undefined);
});

test('the same member cannot join the queue twice', () => {
  const { queue } = setup();
  queue.add('U-17');

  assert.throws(() => queue.add('U-17'), /member already in queue: U-17/);
  assert.equal(queue.length(), 1);
});

test('a message telling the member their position is sent', () => {
  const { queue, notification } = setup();

  queue.add('U-17');
  queue.next();

  assert.deepEqual(notification.calls, [
    ['send', 'U-17', 'queue: 1'],
    ['send', 'U-17', 'book assigned'],
  ]);
});
```

The second set tests the same unit, but binds to the internal structure and the log text.
Both are outside the contract: whether the queue is kept as an array or a map, and which
sentence is written to the log, are of no concern to the member.

```js
// implementation.test.mjs — tests bound to internal structure and log text
import { test } from 'node:test';
import assert from 'node:assert/strict';
import { createReservationQueue } from './reservation.mjs';
import { spy } from './spy.mjs';

const setup = () => {
  const log = spy('write');
  return { queue: createReservationQueue({ notification: spy('send'), log }), log };
};

test("the queue's internal array keeps the order of adding", () => {
  const { queue } = setup();

  queue.add('U-17');
  queue.add('U-42');

  assert.deepEqual(queue.queue, ['U-17', 'U-42']);
});

test('the lines written to the log have a specific format', () => {
  const { queue, log } = setup();

  queue.add('U-17');
  queue.next();

  assert.deepEqual(log.calls, [
    ['write', 'add U-17 position 1'],
    ['write', 'next U-17'],
  ]);
});
```

```sh
node --test --test-reporter=tap behavior.test.mjs implementation.test.mjs | grep -E '^ *(ok|not ok|# (tests|pass|fail))'
```

```
ok 1 - the order of adding determines the waiting position
ok 2 - the book is assigned to the first member in the queue
ok 3 - the same member cannot join the queue twice
ok 4 - a message telling the member their position is sent
ok 5 - the queue's internal array keeps the order of adding
ok 6 - the lines written to the log have a specific format
# tests 6
# pass 6
# fail 0
```

All six tests are green. At this point the difference between the two sets is invisible;
both appear to verify the same code.

## The Refactor

The queue is moved to a map, the internal array is no longer exposed, and the log lines
are fit into a single pattern. All four clauses of the contract are preserved exactly: the
return values, the thrown error, and the messages sent to the member do not change.

```js
// reservation.mjs — version 2: queue moved to a map, log format collapsed to one pattern
export function createReservationQueue({ notification, log }) {
  const entries = new Map();
  let sequence = 0;
  const queued = () => [...entries.entries()]
    .sort((a, b) => a[1] - b[1])
    .map(([memberId]) => memberId);

  return {
    add(memberId) {
      if (entries.has(memberId)) throw new Error(`member already in queue: ${memberId}`);
      sequence += 1;
      entries.set(memberId, sequence);
      const position = entries.size;
      log.write(`reservation member=${memberId} status=added position=${position}`);
      notification.send(memberId, `queue: ${position}`);
      return position;
    },

    next() {
      const [first] = queued();
      if (first === undefined) return undefined;
      entries.delete(first);
      log.write(`reservation member=${first} status=assigned`);
      notification.send(first, 'book assigned');
      return first;
    },

    length: () => entries.size,
  };
}
```

Same two files, same command, a refactored library.

```sh
node --test --test-reporter=tap behavior.test.mjs implementation.test.mjs | grep -E '^ *(ok|not ok|# (tests|pass|fail))'
```

```
ok 1 - the order of adding determines the waiting position
ok 2 - the book is assigned to the first member in the queue
ok 3 - the same member cannot join the queue twice
ok 4 - a message telling the member their position is sent
not ok 5 - the queue's internal array keeps the order of adding
not ok 6 - the lines written to the log have a specific format
# tests 6
# pass 4
# fail 2
```

The measurement is clean: four behavior tests stayed green, two implementation tests
failed. The library's contract did not change — no member sees a different result. The two
failing tests did not report a regression; they reported that **their own assumptions**
had broken.

This is the cost of flakiness. Refactoring means improving the internal structure while
preserving behavior; a test suite that fails every time this is done makes refactoring
expensive. Once it is expensive enough, it stops happening, and the code becomes
unimprovable because of the very tests meant to protect it.

The boundary question that follows is this: is an assertion bound to something that is
part of the contract? The fourth behavior test also uses a spy, and it too verifies a side
effect — but the notification sent to the member is in the contract. What draws the line
is not the type of tool but whether the thing being verified is something someone else
observes.

## The Measure of Flakiness

The other source of flakiness is non-deterministic input, and there the measure is the
**flake rate**: the percentage of drops across a large number of runs of the same test on
the same code.

```js
// flake-rate.mjs — a flaky test's flake rate measured with a fixed seed
function generator(seed) {
  let state = (seed * 2654435761) % 2147483647;
  return () => {
    state = (state * 48271) % 2147483647;
    return state / 2147483647;
  };
}

// A batch job's duration varies by machine and load: between forty and sixty-nine.
const durationFor = (random) => 40 + Math.floor(random() * 30);

function budgetBoundTest(random) {
  const duration = durationFor(random);
  if (duration > 68) throw new Error(`duration budget exceeded: ${duration}`);
}

function behaviorBoundTest(random) {
  const duration = durationFor(random);
  if (Number.isInteger(duration) === false || duration < 0) throw new Error(`invalid duration: ${duration}`);
}

const RUNS = 1000;
for (const [label, testFn] of [['duration-budget-bound', budgetBoundTest], ['behavior-bound', behaviorBoundTest]]) {
  const random = generator(20);
  let failed = 0;
  for (let i = 0; i < RUNS; i += 1) {
    try {
      testFn(random);
    } catch {
      failed += 1;
    }
  }
  const rate = ((failed / RUNS) * 100).toFixed(1);
  console.log(`${label.padEnd(21)} ${RUNS} runs, ${failed} drops, flake rate ${rate}%`);
}
```

```
duration-budget-bound 1000 runs, 27 drops, flake rate 2.7%
behavior-bound        1000 runs, 0 drops, flake rate 0.0%
```

A rate of two point seven percent is dangerous precisely because it is hard to notice. A
test that is green in forty-nine runs out of fifty gets called "flaky now and then," and
the run where it fails gets re-run. At that point the meaning of the suite breaks down: a
green run no longer says "the code is correct" — it starts saying "we got lucky this
time."

Once the flake rate is measured, the work to do is clear. The flaky test is put into
**quarantine** — removed from the main run so the remaining tests' signal stays clean —
and its source is fixed. In the example above, the fix is to stop loading a duration
budget onto a unit test; duration is a performance metric and is measured separately.

## Summary

- A flaky test is a test that fails even though the behavior it tests has not changed; its
  two sources are non-deterministic input and binding to the implementation.
- When the queue array was moved to a map and the log format changed, four behavior tests
  stayed green and two implementation tests failed; the library's contract had not changed
  at all.
- Implementation-bound tests make refactoring expensive; once expensive enough, the code
  becomes unimprovable because of the very tests meant to protect it.
- The question that draws the line is scope, not tool: is the thing being verified part of
  the contract, or an internal detail?
- The measure of non-deterministic flakiness is the flake rate; in the measurement, the
  test bound to the duration budget failed in twenty-seven of a thousand runs, and the
  test bound to behavior never failed.

## Next Step

The Unit Testing Practice topic closes here. Up to this point, in every test the code came
first: a bug report, a rule, or a refactor found an implementation already there, ready to
be tested. The next topic reverses this order. When the test is written first, the test
determines what code gets written; a cycle that starts with a red run ends, after reaching
green, with a refactor. The next lesson runs these three steps one at a time.