---
title: 'Flakiness Management'
source: 'https://academia.sh/en/courses/integration-testing/flakiness-management'
course: 'Integration, Contract and End-to-End Testing'
language: en
updated: '2026-08-23T14:25:14+00:00'
license: 'CC BY-SA 4.0'
---

# Flakiness Management

Measuring flakiness's distribution across causes at the scale of an end-to-end team; how individually small drop rates compound at team scale; and the trade-off the quarantine threshold sets between red runs and escaped real defects.

The last three lessons saw the same fact from separate angles: wait policy shifted the
drop rate, device selection shifted defect coverage, threshold choice shifted the false
alarm rate. All three are pieces of a single problem — an end-to-end test carries an
instability that the lower-level tests do not carry.

The definition of a flaky test, the drop rate, and quarantine were established in the
Unit Testing and Test-Driven Development course. Here the scale changes. There, a single
test's drop rate was measured; here, the run of a forty-test team is measured, and two
new questions arise: what proportion of drops each cause accounts for, and how much
quarantining flaky tests raises the rate of escaped defects.

## Distribution at Team Scale

The model's input is written out explicitly. **KY1 — forty end-to-end tests. Twelve are
exposed to a tight wait timeout (0.02 per run), six depend on state left behind by the
previous test (0.01), eight share the same member and book record (0.015), five make a
large number of network calls (a 0.005 per-call stall probability across six calls, that
is, 0.0296), and nine are stable. A real regression occurs in one run in four and falls
under one of forty protecting tests.** The proportions across causes come from the
mechanism: the network cause grows with the number of calls, the shared-data cause with
the number of tests using the same record.

```js
// team.mjs — two hundred runs of forty end-to-end tests and a quarantine scan
function createRng(seed) {
  let state = (seed * 2654435761) % 2147483647;
  return () => {
    state = (state * 48271) % 2147483647;
    return state / 2147483647;
  };
}

const tests = [];
const add = (count, cause, rate) => {
  for (let i = 0; i < count; i += 1) tests.push({ cause, rate, drops: 0 });
};
add(12, 'wait', 0.02);
add(6, 'ordering', 0.01);
add(8, 'shared data', 0.015);
add(5, 'network', 1 - (1 - 0.005) ** 6);
add(9, 'stable', 0);

const RUNS = 200;
const random = createRng(47);
for (let k = 0; k < RUNS; k += 1) {
  for (const t of tests) if (random() < t.rate) t.drops += 1;
}

const s = (n, g) => String(n).padStart(g);
console.log(`${'cause'.padEnd(17)}${s('test', 6)}${s('drops', 7)}${s('rate per test', 18)}`);
for (const cause of ['wait', 'ordering', 'shared data', 'network', 'stable']) {
  const group = tests.filter((t) => t.cause === cause);
  const drops = group.reduce((a, t) => a + t.drops, 0);
  const rate = ((drops / (group.length * RUNS)) * 100).toFixed(2);
  console.log(`${cause.padEnd(17)}${s(group.length, 6)}${s(drops, 7)}${s(`%${rate}`, 18)}`);
}

const REAL_DEFECT = 0.25;
const scenario = (threshold) => {
  const quarantined = tests.map((t) => (t.drops / RUNS) * 100 > threshold);
  const random2 = createRng(53);
  let red = 0;
  let escaped = 0;
  for (let k = 0; k < RUNS; k += 1) {
    let dropped = false;
    tests.forEach((t, i) => {
      if (quarantined[i] === false && random2() < t.rate) dropped = true;
    });
    if (dropped) red += 1;
    if (random2() < REAL_DEFECT) {
      const protector = Math.floor(random2() * tests.length);
      if (quarantined[protector]) escaped += 1;
    }
  }
  return { count: quarantined.filter(Boolean).length, red, escaped };
};

console.log(`${'quarantine threshold'.padEnd(23)}${s('quarantined', 13)}${s('red run', 15)}${s('escaped defect', 19)}`);
for (const threshold of [100, 2.5, 1.5, 0.5]) {
  const r = scenario(threshold);
  const name = threshold === 100 ? 'none' : `%${threshold.toFixed(1)}`;
  console.log(`${name.padEnd(23)}${s(r.count, 13)}${s(`%${((r.red / RUNS) * 100).toFixed(1)}`, 15)}`
    + `${s(`%${((r.escaped / RUNS) * 100).toFixed(1)}`, 19)}`);
}
```

```
cause              test  drops     rate per test
wait                 12     37             %1.54
ordering              6     10             %0.83
shared data           8     32             %2.00
network               5     28             %2.80
stable                9      0             %0.00
quarantine threshold     quarantined        red run     escaped defect
none                               0          %47.5               %0.0
%2.5                               8          %36.0               %4.5
%1.5                              13          %26.5               %9.5
%0.5                              25           %9.0              %16.5
```

The individual rows of the first table look harmless: even the most flaky group drops
only three times in a hundred runs. The first row of the second table breaks that
impression — with no quarantine, 47.5 percent of runs are red because of flakiness. That
growth comes from multiplication: for a single run to pass clean, all forty of the forty
tests have to pass, and multiplying the individual pass probabilities together yields a
43.67 percent expectation of red. The 47.5 percent measured across two hundred runs is
that expectation's sampling fluctuation.

This is where the end-to-end scale departs from the unit scale. A two-percent drop rate
is a tolerable flaw in a single unit test; in a forty-test end-to-end team, it is a
system property that turns roughly every second run red. And the causes of drops are not
fixed: the largest shares in the measurement are network and shared data, and neither
can be fixed inside a single test — one is tied to the call count, the other to the
fixture being shared.

## The Two Faces of Quarantine

The remaining rows of the second table scan the quarantine threshold. As the threshold
drops, the number of quarantined tests rises, the red-run rate falls, and the **escaped
real defect** rate rises. A 2.5 percent threshold removes eight tests from the main run:
the red-run rate falls from 47.5 percent to 36.0 percent, while in exchange 4.5 percent
of runs let a real regression through unseen. A half-percent threshold quarantines a
quarter of the team; the signal is nearly clean — a 9.0 percent red-run rate — but 16.5
percent of runs carry a regression and still turn green.

Quarantine is therefore not a solution but a trade: it buys down noise at the cost of
coverage. The measure of that decision is visible in the table too — whichever threshold
is chosen, the escaped-defect rate approaches the share of tests taken into quarantine.

There is one more detail. The quarantine decision looks at the rate **measured** over
two hundred runs, not the true drop rate. The measured values deviate from the model's
true values: the wait group's true rate is two percent, the measured rate is 1.54
percent. That deviation grows as the sample shrinks, and stable tests start entering the
quarantine list while flaky ones start staying out of it.

## Green Run, Standing Defect

The concrete face of this trade-off shows up in a single run. The borrowing flow has
four tests; the second checks that a warning is shown to the member when the loan limit
is exceeded. The flow's first version gives the correct status code for a rejected
request, but does not carry the warning.

```js
// flow.mjs — version 1: end-to-end visible result of a loan request
export const LIMIT = 5;

export function requestLoan(member) {
  if (member.openLoans >= LIMIT) return { status: 409, warning: null };
  return { status: 201, warning: null };
}

export const cardStatus = (result) => (result.status === 201 ? 'loaned' : 'shelved');
```

```js
// quarantine.mjs — version 1: quarantined test names
export const quarantine = new Set();
```

```js
// team.test.mjs — the end-to-end team's four tests, checking the quarantine list
import { test } from 'node:test';
import assert from 'node:assert/strict';
import { requestLoan, cardStatus } from './flow.mjs';
import { quarantine } from './quarantine.mjs';

const runTest = (name, body) => test(name, { skip: quarantine.has(name) }, body);

runTest('an eligible request is approved', () => {
  assert.equal(requestLoan({ openLoans: 1 }).status, 201);
});

runTest('a warning is shown when the loan limit is exceeded', () => {
  const result = requestLoan({ openLoans: 5 });

  assert.equal(result.status, 409);
  assert.equal(result.warning, 'loan-limit-exceeded');
});

runTest('the card shows loaned after approval', () => {
  assert.equal(cardStatus(requestLoan({ openLoans: 1 })), 'loaned');
});

runTest('the card stays shelved on a rejected request', () => {
  assert.equal(cardStatus(requestLoan({ openLoans: 5 })), 'shelved');
});
```

```bash
node --test --test-reporter=tap team.test.mjs | grep -E '^ *(ok|not ok|# (tests|pass|fail|skipped))'
```

```
ok 1 - an eligible request is approved
not ok 2 - a warning is shown when the loan limit is exceeded
ok 3 - the card shows loaned after approval
ok 4 - the card stays shelved on a rejected request
# tests 4
# pass 3
# fail 1
# skipped 0
```

The second test caught the defect. Now suppose this test makes a large number of
network calls and comes out flaky in measurement; by the rule, it gets quarantined.

```js
// quarantine.mjs — version 2: the test found to be flaky is quarantined
export const quarantine = new Set(['a warning is shown when the loan limit is exceeded']);
```

```bash
node --test --test-reporter=tap team.test.mjs | grep -E '^ *(ok|not ok|# (tests|pass|fail|skipped))'
```

```
ok 1 - an eligible request is approved
ok 2 - a warning is shown when the loan limit is exceeded # SKIP
ok 3 - the card shows loaned after approval
ok 4 - the card stays shelved on a rejected request
# tests 4
# pass 3
# fail 0
# skipped 1
```

The code has not changed, the defect stands, and the run is green. This is the class
that escapes, and it is a single instance of the "escaped real defect" column measured
in the table: every test taken into quarantine takes the defect class it was protecting
along with it. That is why it matters for a skipped test to stay visible in the output —
if how many tests a green run skipped cannot be read, the trade-off becomes invisible.

The discipline has two parts, and both are completed in this run: **the source of
flakiness is fixed** (here, the number of network calls and the timeout), and then
**the test is taken out of quarantine**. Below, both happen together: the flow is made
to carry the warning, and the list is cleared.

```js
// flow.mjs — version 2: rejected request now carries the warning too
export const LIMIT = 5;

export function requestLoan(member) {
  if (member.openLoans >= LIMIT) return { status: 409, warning: 'loan-limit-exceeded' };
  return { status: 201, warning: null };
}

export const cardStatus = (result) => (result.status === 201 ? 'loaned' : 'shelved');
```

```js
// quarantine.mjs — version 3: source of flakiness fixed, list cleared
export const quarantine = new Set();
```

```bash
node --test --test-reporter=tap team.test.mjs | grep -E '^ *(ok|not ok|# (tests|pass|fail|skipped))'
```

```
ok 1 - an eligible request is approved
ok 2 - a warning is shown when the loan limit is exceeded
ok 3 - the card shows loaned after approval
ok 4 - the card stays shelved on a rejected request
# tests 4
# pass 4
# fail 0
# skipped 0
```

A quarantine entry becomes permanent when it has no owner, no justification, and no
exit criterion; permanent quarantine is a slow drift toward the state in the tables'
last row — a clean signal and a 16.5 percent escaped-defect rate.

The run-dependent side of the cost is small here: the four tests took about 33 ms. The
run-independent numbers are in the measurement — forty tests across two hundred runs
produced eight thousand test executions, and every quarantine entry means one
maintenance debt, one lost defect class.

## Summary

- At end-to-end scale, flakiness is a property of the team, not of a single test: forty
  tests each dropping around two percent per run turned 47.5 percent of runs red.
- The largest shares of measured drops were the network (2.80%) and shared-data (2.00%)
  causes; neither can be fixed inside a single test.
- As the quarantine threshold falls, red runs fall and escaped real defects rise: 36.0%
  and 4.5% at the 2.5 percent threshold; 9.0% and 16.5% at the half-percent threshold.
- The quarantine decision rests on the measured rate, not the true rate; the
  two-hundred-run sample showed the wait group's true two-percent rate as 1.54 percent.
- The single-run example made the trade-off concrete: when the quarantined test was
  skipped, the defect stayed in place while the run turned green.

## Next Step

Every test here ran one at a time: forty tests in sequence, one finishing before the
next started. The same number always stood behind the quarantine discussion — running
time. A flaky test could have been rerun instead of quarantined, more cases could have
been added to cut the escaped-defect rate; both draw on the same budget, elapsed time.
That changes the question itself: is running tests in sequence a necessity, or a
choice? The next lesson takes on that question through the run order itself, and
measures what is paid, and what isolation requirement, in exchange for shortening the
duration.