Flakiness Management

The last three lessons saw the same fact from separate angles: wait policy shifted the drop rate, device selection shifted defect coverage, threshold choice shifted the false alarm rate. All three are pieces of a single problem — an end-to-end test carries an instability that the lower-level tests do not carry.

The definition of a flaky test, the drop rate, and quarantine were established in the Unit Testing and Test-Driven Development course. Here the scale changes. There, a single test’s drop rate was measured; here, the run of a forty-test team is measured, and two new questions arise: what proportion of drops each cause accounts for, and how much quarantining flaky tests raises the rate of escaped defects.

Distribution at Team Scale

The model’s input is written out explicitly. KY1 — forty end-to-end tests. Twelve are exposed to a tight wait timeout (0.02 per run), six depend on state left behind by the previous test (0.01), eight share the same member and book record (0.015), five make a large number of network calls (a 0.005 per-call stall probability across six calls, that is, 0.0296), and nine are stable. A real regression occurs in one run in four and falls under one of forty protecting tests. The proportions across causes come from the mechanism: the network cause grows with the number of calls, the shared-data cause with the number of tests using the same record.

// team.mjs — two hundred runs of forty end-to-end tests and a quarantine scan
function createRng(seed) {
  let state = (seed * 2654435761) % 2147483647;
  return () => {
    state = (state * 48271) % 2147483647;
    return state / 2147483647;
  };
}

const tests = [];
const add = (count, cause, rate) => {
  for (let i = 0; i < count; i += 1) tests.push({ cause, rate, drops: 0 });
};
add(12, 'wait', 0.02);
add(6, 'ordering', 0.01);
add(8, 'shared data', 0.015);
add(5, 'network', 1 - (1 - 0.005) ** 6);
add(9, 'stable', 0);

const RUNS = 200;
const random = createRng(47);
for (let k = 0; k < RUNS; k += 1) {
  for (const t of tests) if (random() < t.rate) t.drops += 1;
}

const s = (n, g) => String(n).padStart(g);
console.log(`${'cause'.padEnd(17)}${s('test', 6)}${s('drops', 7)}${s('rate per test', 18)}`);
for (const cause of ['wait', 'ordering', 'shared data', 'network', 'stable']) {
  const group = tests.filter((t) => t.cause === cause);
  const drops = group.reduce((a, t) => a + t.drops, 0);
  const rate = ((drops / (group.length * RUNS)) * 100).toFixed(2);
  console.log(`${cause.padEnd(17)}${s(group.length, 6)}${s(drops, 7)}${s(`%${rate}`, 18)}`);
}

const REAL_DEFECT = 0.25;
const scenario = (threshold) => {
  const quarantined = tests.map((t) => (t.drops / RUNS) * 100 > threshold);
  const random2 = createRng(53);
  let red = 0;
  let escaped = 0;
  for (let k = 0; k < RUNS; k += 1) {
    let dropped = false;
    tests.forEach((t, i) => {
      if (quarantined[i] === false && random2() < t.rate) dropped = true;
    });
    if (dropped) red += 1;
    if (random2() < REAL_DEFECT) {
      const protector = Math.floor(random2() * tests.length);
      if (quarantined[protector]) escaped += 1;
    }
  }
  return { count: quarantined.filter(Boolean).length, red, escaped };
};

console.log(`${'quarantine threshold'.padEnd(23)}${s('quarantined', 13)}${s('red run', 15)}${s('escaped defect', 19)}`);
for (const threshold of [100, 2.5, 1.5, 0.5]) {
  const r = scenario(threshold);
  const name = threshold === 100 ? 'none' : `%${threshold.toFixed(1)}`;
  console.log(`${name.padEnd(23)}${s(r.count, 13)}${s(`%${((r.red / RUNS) * 100).toFixed(1)}`, 15)}`
    + `${s(`%${((r.escaped / RUNS) * 100).toFixed(1)}`, 19)}`);
}

cause              test  drops     rate per test
wait                 12     37             %1.54
ordering              6     10             %0.83
shared data           8     32             %2.00
network               5     28             %2.80
stable                9      0             %0.00
quarantine threshold     quarantined        red run     escaped defect
none                               0          %47.5               %0.0
%2.5                               8          %36.0               %4.5
%1.5                              13          %26.5               %9.5
%0.5                              25           %9.0              %16.5

The individual rows of the first table look harmless: even the most flaky group drops only three times in a hundred runs. The first row of the second table breaks that impression — with no quarantine, 47.5 percent of runs are red because of flakiness. That growth comes from multiplication: for a single run to pass clean, all forty of the forty tests have to pass, and multiplying the individual pass probabilities together yields a 43.67 percent expectation of red. The 47.5 percent measured across two hundred runs is that expectation’s sampling fluctuation.

This is where the end-to-end scale departs from the unit scale. A two-percent drop rate is a tolerable flaw in a single unit test; in a forty-test end-to-end team, it is a system property that turns roughly every second run red. And the causes of drops are not fixed: the largest shares in the measurement are network and shared data, and neither can be fixed inside a single test — one is tied to the call count, the other to the fixture being shared.

The Two Faces of Quarantine

The remaining rows of the second table scan the quarantine threshold. As the threshold drops, the number of quarantined tests rises, the red-run rate falls, and the escaped real defect rate rises. A 2.5 percent threshold removes eight tests from the main run: the red-run rate falls from 47.5 percent to 36.0 percent, while in exchange 4.5 percent of runs let a real regression through unseen. A half-percent threshold quarantines a quarter of the team; the signal is nearly clean — a 9.0 percent red-run rate — but 16.5 percent of runs carry a regression and still turn green.

Quarantine is therefore not a solution but a trade: it buys down noise at the cost of coverage. The measure of that decision is visible in the table too — whichever threshold is chosen, the escaped-defect rate approaches the share of tests taken into quarantine.

There is one more detail. The quarantine decision looks at the rate measured over two hundred runs, not the true drop rate. The measured values deviate from the model’s true values: the wait group’s true rate is two percent, the measured rate is 1.54 percent. That deviation grows as the sample shrinks, and stable tests start entering the quarantine list while flaky ones start staying out of it.

Green Run, Standing Defect

The concrete face of this trade-off shows up in a single run. The borrowing flow has four tests; the second checks that a warning is shown to the member when the loan limit is exceeded. The flow’s first version gives the correct status code for a rejected request, but does not carry the warning.

// flow.mjs — version 1: end-to-end visible result of a loan request
export const LIMIT = 5;

export function requestLoan(member) {
  if (member.openLoans >= LIMIT) return { status: 409, warning: null };
  return { status: 201, warning: null };
}

export const cardStatus = (result) => (result.status === 201 ? 'loaned' : 'shelved');

// quarantine.mjs — version 1: quarantined test names
export const quarantine = new Set();

// team.test.mjs — the end-to-end team's four tests, checking the quarantine list
import { test } from 'node:test';
import assert from 'node:assert/strict';
import { requestLoan, cardStatus } from './flow.mjs';
import { quarantine } from './quarantine.mjs';

const runTest = (name, body) => test(name, { skip: quarantine.has(name) }, body);

runTest('an eligible request is approved', () => {
  assert.equal(requestLoan({ openLoans: 1 }).status, 201);
});

runTest('a warning is shown when the loan limit is exceeded', () => {
  const result = requestLoan({ openLoans: 5 });

  assert.equal(result.status, 409);
  assert.equal(result.warning, 'loan-limit-exceeded');
});

runTest('the card shows loaned after approval', () => {
  assert.equal(cardStatus(requestLoan({ openLoans: 1 })), 'loaned');
});

runTest('the card stays shelved on a rejected request', () => {
  assert.equal(cardStatus(requestLoan({ openLoans: 5 })), 'shelved');
});

node --test --test-reporter=tap team.test.mjs | grep -E '^ *(ok|not ok|# (tests|pass|fail|skipped))'

ok 1 - an eligible request is approved
not ok 2 - a warning is shown when the loan limit is exceeded
ok 3 - the card shows loaned after approval
ok 4 - the card stays shelved on a rejected request
# tests 4
# pass 3
# fail 1
# skipped 0

The second test caught the defect. Now suppose this test makes a large number of network calls and comes out flaky in measurement; by the rule, it gets quarantined.

// quarantine.mjs — version 2: the test found to be flaky is quarantined
export const quarantine = new Set(['a warning is shown when the loan limit is exceeded']);

node --test --test-reporter=tap team.test.mjs | grep -E '^ *(ok|not ok|# (tests|pass|fail|skipped))'

ok 1 - an eligible request is approved
ok 2 - a warning is shown when the loan limit is exceeded # SKIP
ok 3 - the card shows loaned after approval
ok 4 - the card stays shelved on a rejected request
# tests 4
# pass 3
# fail 0
# skipped 1

The code has not changed, the defect stands, and the run is green. This is the class that escapes, and it is a single instance of the “escaped real defect” column measured in the table: every test taken into quarantine takes the defect class it was protecting along with it. That is why it matters for a skipped test to stay visible in the output — if how many tests a green run skipped cannot be read, the trade-off becomes invisible.

The discipline has two parts, and both are completed in this run: the source of flakiness is fixed (here, the number of network calls and the timeout), and then the test is taken out of quarantine. Below, both happen together: the flow is made to carry the warning, and the list is cleared.

// flow.mjs — version 2: rejected request now carries the warning too
export const LIMIT = 5;

export function requestLoan(member) {
  if (member.openLoans >= LIMIT) return { status: 409, warning: 'loan-limit-exceeded' };
  return { status: 201, warning: null };
}

export const cardStatus = (result) => (result.status === 201 ? 'loaned' : 'shelved');

// quarantine.mjs — version 3: source of flakiness fixed, list cleared
export const quarantine = new Set();

node --test --test-reporter=tap team.test.mjs | grep -E '^ *(ok|not ok|# (tests|pass|fail|skipped))'

ok 1 - an eligible request is approved
ok 2 - a warning is shown when the loan limit is exceeded
ok 3 - the card shows loaned after approval
ok 4 - the card stays shelved on a rejected request
# tests 4
# pass 4
# fail 0
# skipped 0

A quarantine entry becomes permanent when it has no owner, no justification, and no exit criterion; permanent quarantine is a slow drift toward the state in the tables’ last row — a clean signal and a 16.5 percent escaped-defect rate.

The run-dependent side of the cost is small here: the four tests took about 33 ms. The run-independent numbers are in the measurement — forty tests across two hundred runs produced eight thousand test executions, and every quarantine entry means one maintenance debt, one lost defect class.

Summary

• At end-to-end scale, flakiness is a property of the team, not of a single test: forty tests each dropping around two percent per run turned 47.5 percent of runs red.
• The largest shares of measured drops were the network (2.80%) and shared-data (2.00%) causes; neither can be fixed inside a single test.
• As the quarantine threshold falls, red runs fall and escaped real defects rise: 36.0% and 4.5% at the 2.5 percent threshold; 9.0% and 16.5% at the half-percent threshold.
• The quarantine decision rests on the measured rate, not the true rate; the two-hundred-run sample showed the wait group’s true two-percent rate as 1.54 percent.
• The single-run example made the trade-off concrete: when the quarantined test was skipped, the defect stayed in place while the run turned green.

Next Step

Every test here ran one at a time: forty tests in sequence, one finishing before the next started. The same number always stood behind the quarantine discussion — running time. A flaky test could have been rerun instead of quarantined, more cases could have been added to cut the escaped-defect rate; both draw on the same budget, elapsed time. That changes the question itself: is running tests in sequence a necessity, or a choice? The next lesson takes on that question through the run order itself, and measures what is paid, and what isolation requirement, in exchange for shortening the duration.