Test Stability

The tests in the previous four lessons rested on one assumption: the same code gives the same result. This assumption breaks often. A test passes in nineteen of twenty runs and fails in one; no one changed anything. It turns green on the second run and the work continues.

This behavior looks harmless but erodes the signal the suite gives. Once the meaning of a red result becomes uncertain, the team first tries rerunning it, then reruns without even looking, and eventually mistakes a real defect for flakiness and lets it through. The entire value of a test suite lies in red meaning something.

Flaky Tests and Their Measure

A flaky test is a test that gives a different result from run to run even though the code it tests has not changed. The defect is in the test itself: the test’s result depends on something outside the behavior it tests.

This should be treated as a number, not an impression. Flake rate is how many times a test fails across a given number of runs. If a test fails at unknown intervals, running it many times under the same conditions and measuring the rate proves both that the problem exists and that a fix works.

Counting the Three Causes

The following simulation models three causes of flakiness separately and counts, with a pseudo-random but deterministic generator, how many times each fails across two hundred runs. The generator is a linear congruential method: the same seed always gives the same sequence, so the results are reproducible.

// flakiness.mjs — a deterministic simulation of three flakiness causes
export function generator(seed) {
  let s = seed >>> 0;
  return () => {
    s = (Math.imul(s, 1664525) + 1013904223) >>> 0;
    return s / 4294967296;
  };
}

function shuffle(array, random) {
  const copy = [...array];
  for (let i = copy.length - 1; i > 0; i -= 1) {
    const j = Math.floor(random() * (i + 1));
    [copy[i], copy[j]] = [copy[j], copy[i]];
  }
  return copy;
}

export function runSuite({ seed = 7, runs = 200, fixed = false, shuffling = true }) {
  const random = generator(seed);
  const names = ['tableLoads', 'filterNarrows', 'formSubmits'];
  const failed = { tableLoads: 0, filterNarrows: 0, formSubmits: 0 };
  let draft = null; // shared state living at module scope

  for (let n = 0; n < runs; n += 1) {
    const order = shuffling ? shuffle(names, random) : names;
    const setUp = new Set();
    if (fixed) draft = null; // each run starts with its own state

    for (const name of order) {
      let passed;
      if (name === 'tableLoads') {
        const responseTime = 40 + 80 * random();
        passed = fixed ? responseTime <= 500 : responseTime <= 100;
        setUp.add('table');
      } else if (name === 'filterNarrows') {
        if (fixed) setUp.add('table'); // the test sets up its own data
        passed = setUp.has('table');
      } else {
        passed = draft === null; // the form must open with an empty draft
        const submitSucceeded = random() < 0.5;
        draft = submitSucceeded ? null : { station: 'north-slope-01' };
      }
      if (passed === false) failed[name] += 1;
    }
  }
  return failed;
}

// report.mjs — how many times each test failed across 200 runs
import { runSuite } from './flakiness.mjs';

const scenarios = [
  ['fixed order, unfixed', { shuffling: false }],
  ['shuffled order, unfixed', { shuffling: true }],
  ['shuffled order, fixed', { shuffling: true, fixed: true }],
];

console.log('scenario                       tableLoads  filterNarrows  formSubmits');
for (const [label, options] of scenarios) {
  const result = runSuite({ seed: 7, runs: 200, ...options });
  console.log(
    label.padEnd(30) +
      String(result.tableLoads).padStart(9) +
      String(result.filterNarrows).padStart(16) +
      String(result.formSubmits).padStart(16),
  );
}

node report.mjs

scenario                       tableLoads  filterNarrows  formSubmits
fixed order, unfixed                 50               0              95
shuffled order, unfixed              48             102             104
shuffled order, fixed                 0               0               0

Three rows give three separate lessons.

Timing

The tableLoads test waits a fixed hundred milliseconds for the list to arrive. In the simulation, the response time varies between forty and a hundred twenty milliseconds; the test fails when the duration exceeds the wait. It failed in fifty and forty-eight of the two hundred runs — a rate close to one in four, independent of run order.

A fixed wait is wrong in two ways. It is not enough on a slow run; it is wasted on a fast one. Adding a hundred milliseconds for each of a hundred tests adds ten seconds to the suite’s duration and guarantees nothing.

The correct form is a conditional wait: checking at short intervals and waiting until an observable result exists, with a generous upper bound. In the fixed version, the upper bound is five hundred milliseconds, and it was not exceeded in any of the two hundred runs. A generous upper bound does not slow the test down — when the condition is met early, the wait ends early. Removing the upper bound entirely is not acceptable: in a genuinely frozen application, the test would wait forever.

The microtask queue from the Asynchronous JavaScript and the Runtime course applies here too. A promise settling and the screen updating may not happen in the same turn; what should be waited for is not the promise itself but the result the user will see.

Order Dependence

The filterNarrows test expects the table to have already loaded — a different test sets up the data. In the first row, this test never failed: in the fixed order, tableLoads always runs first. In the second row, the order was shuffled, and it failed in a hundred two of the two hundred runs.

The real finding here is this: a fixed order hides the defect. The test suite can stay green for months, then suddenly turn red once a file name changes or the runner parallelizes. Deliberately shuffling the order is a diagnostic tool; once the seed used for shuffling is written to the output, the failing order can be reproduced exactly.

The fix is not to fix the order but to remove the dependency. Every test sets up the data it needs itself. In the fixed version, filterNarrows adds its own measurement and becomes independent of the order.

Shared State

The formSubmits test expects the form to open with an empty draft. The draft lives at module scope and is cleared only when the submission succeeds; on a failed submission, it carries over to the next run. It failed in ninety-five and a hundred four of the two hundred runs.

This cause is different from the other two in one way: on its own, in a clean environment, the test always passes. For it to fail, some earlier run first has to have contaminated the environment. This is exactly the test that passes when run alone on a developer’s machine but fails inside the suite — and it is the behavior that makes diagnosis hardest, because there is no defect in the failing test itself.

The sources of shared state can be enumerated: values held at module scope, local storage and session storage, cookies, program-controlled cache, backend data, and the file system. The fix is a fixture that is set up before every test and torn down after. In the fixed version, the draft is reset at the start of every run and the failure rate drops to zero.

Parallel execution magnifies this problem. Two runners sharing the same backend record or the same port corrupt each other’s environment; isolation is achieved with a separate data set per run.

What to Do About Flakiness

Retrying is the first solution that comes to mind, and on its own it is wrong. Silently rerunning a failing test hides the flakiness; hidden flakiness grows over time and hides the real defects mixed in among it. The only acceptable form is retrying that is recorded, with the flake rate tracked as a measure.

If a test is too complex to fix, it is put into quarantine: it is separated from the suite, its result does not block a release, but it has an owner and a deadline on the list. Ownerless quarantine is the long way of silently deleting the test.

There are also general measures that increase determinism. The clock is fixed, the locale and time zone are given explicitly, randomness is seeded as in this lesson, and the seed is written to the output. The goal is the same: leaving a test’s result dependent on nothing outside the behavior it tests.

Summary

• A flaky test is a test that gives a different result from run to run even though the code it tests has not changed; its cost is that a red result loses its meaning.
• A fixed wait fails at the tail of the response-time distribution; a conditional wait with a generous upper bound zeroed out this cause.
• A fixed run order hides order dependence: the same test never failed in the fixed order but failed in a hundred two of two hundred runs in the shuffled order.
• The signature of shared state is a test that passes alone but fails inside the suite; every test setting up and tearing down its own fixture zeroed out this cause.
• Retrying hides flakiness; the acceptable form is retrying that is recorded and tracked as a flake rate, while quarantine must be owned and time-boxed.

Next Step

A stable test suite says that expected behavior is preserved in a controlled environment. What it does not say is how the application behaves in the hands of real users. The test machine is fast, its network is clean, its cache is in a known state; an observer in the field works over a weak connection, with an aging device, and a crowded screen. The next lesson places these two data sources — lab data from a controlled run and field data from real sessions — side by side and measures the difference between them with a percentile computation.