Testing in Continuous Integration

The previous topic left a distribution behind: scope was written, a risk order was built, an exploratory session’s charter was set, acceptance criteria were tied to a definition of done. What all of these share is this — the distribution is on paper. A plan is only an intention until it is wired into a mechanism that runs on its own with every change; otherwise who runs which test, and when, is left to human memory.

This lesson builds that mechanism. The chain that takes a change and runs it through the tests in sequence is called a pipeline; each part of the pipeline is a stage, and the point where a stage’s outcome forces a decision is a gate. The pipe from the Shell Programming course connects one process’s output to another’s input; the pipeline here is the ordered set of tests a change must pass — it carries a decision, not data. The question can be written as a single number: for a given test running at a given stage, where does the trade-off between feedback time and escaped defect (the same concept as the previous courses’ escaped real defect) fall.

Test Inventory

Four courses wrote tests across thirty lessons, and each lesson’s wrap-up table left behind that test’s run-independent cost. Those numbers are collected here into nine test teams. Process count, network requests, data access, and manual review steps are read directly from the wrap-up tables: the integration team’s 816 accesses is the sum of the rows for The Scope of Integration Testing (4 queries), Test Environment Management (6 steps), Test Data Management (772 queries), Database Testing (21 queries, 13 migration steps), and Mocking External Services (0); the security team’s 20 manual steps is the sum of the manual-review counts from five security lessons.

TP6 — run cost is model minutes: standing up one process is 1 minute, a thousand network requests is 1 minute, a thousand data accesses is 1 minute, one manual review step is 15 minutes; the floor is 1 minute. A manual step weighs fifteen times the others, because manual work cannot be placed at a gate — it waits its turn.

TP7 — occurrences per defect class in a hundred changes. Classes the lower layers see are written more often, because every change touches code; capacity and recovery classes are rare. Two classes have an empty caught-by list: these come from the source lessons’ “missed” column and are defects no team can see.

// inventory.mjs -- the team list, run cost, and defect set of tests written across four courses
const roundUp = (n) => Math.ceil(n / 1000);

// TP6: process 1 min, thousand requests 1 min, thousand accesses 1 min, manual step 15 min; floor 1 min.
export const cost = (t) => Math.max(1,
  t.process + roundUp(t.request) + roundUp(t.access) + t.manual * 15);

// process / request / access / manual counts read from the M21/K03 and K04 wrap-up tables.
export const teams = [
  { name: 'unit', process: 0, request: 0, access: 9, manual: 0 },
  { name: 'contract', process: 6, request: 126, access: 0, manual: 0 },
  { name: 'integration', process: 1, request: 0, access: 816, manual: 0 },
  { name: 'end-to-end', process: 12, request: 24, access: 25511, manual: 0 },
  { name: 'perf-run', process: 44, request: 78939, access: 0, manual: 0 },
  { name: 'perf-sample', process: 0, request: 0, access: 536000, manual: 0 },
  { name: 'security-auto', process: 9, request: 8, access: 183, manual: 0 },
  { name: 'security-manual', process: 0, request: 0, access: 0, manual: 20 },
  { name: 'resilience', process: 0, request: 639, access: 20, manual: 0 },
];

// TP7: occurrences per defect class in a hundred changes.
export const defects = [
  { defectClass: 'boundary comparison', catches: ['unit'], count: 13 },
  { defectClass: 'schema mismatch', catches: ['integration'], count: 8 },
  { defectClass: 'card status not updating', catches: ['end-to-end'], count: 7 },
  { defectClass: 'status code mapping', catches: ['end-to-end'], count: 6 },
  { defectClass: 'breaking change', catches: ['unit', 'contract'], count: 6 },
  { defectClass: 'request shape drift', catches: ['contract'], count: 5 },
  { defectClass: 'accessibility criterion', catches: ['security-manual'], count: 5 },
  { defectClass: 'field lost from contract', catches: ['contract'], count: 4 },
  { defectClass: 'missing warning on rejected request', catches: ['end-to-end'], count: 4, quarantine: true },
  { defectClass: 'latency regression', catches: ['perf-run'], count: 4 },
  { defectClass: 'data loss in migration', catches: ['integration'], count: 3 },
  { defectClass: 'concatenated query', catches: ['security-auto'], count: 3 },
  { defectClass: 'semantic mismatch', catches: [], count: 3 },
  { defectClass: 'check-then-act race', catches: ['end-to-end'], count: 2 },
  { defectClass: 'distribution-dependent report defect', catches: ['integration'], count: 2 },
  { defectClass: 'process-lifetime state', catches: [], count: 2 },
  { defectClass: 'percentile read from few samples', catches: ['perf-sample'], count: 1 },
  { defectClass: 'recovery gap', catches: ['resilience'], count: 1 },
  { defectClass: 'capacity limit', catches: ['resilience'], count: 1 },
];

// cost.mjs -- run cost per team and the total of the defect set
import { teams, defects, cost } from './inventory.mjs';

const s = (x, n) => String(x).padStart(n);
console.log(`${'team'.padEnd(20)}${s('process', 9)}${s('request', 9)}${s('access', 8)}${s('manual', 8)}${s('minutes', 9)}`);
for (const t of teams) {
  console.log(`${t.name.padEnd(20)}${s(t.process, 9)}${s(t.request, 9)}${s(t.access, 8)}${s(t.manual, 8)}${s(cost(t), 9)}`);
}
const total = teams.reduce((a, t) => a + cost(t), 0);
const defectTotal = defects.reduce((a, k) => a + k.count, 0);
const unseen = defects.filter((k) => k.catches.length === 0);
console.log(`\nteam ${teams.length}, full team ${total} minutes`);
console.log(`defect class ${defects.length}, ${defectTotal} defects in a hundred changes, `
  + `${100 - defectTotal} clean changes`);
console.log(`classes no team sees ${unseen.length}, `
  + `${unseen.reduce((a, k) => a + k.count, 0)} defects`);

team                  process  request  access  manual  minutes
unit                        0        0       9       0        1
contract                    6      126       0       0        7
integration                 1        0     816       0        2
end-to-end                 12       24   25511       0       39
perf-run                   44    78939       0       0      123
perf-sample                 0        0  536000       0      536
security-auto               9        8     183       0       11
security-manual             0        0       0      20      300
resilience                  0      639      20       0        2

team 9, full team 1021 minutes
defect class 19, 80 defects in a hundred changes, 20 clean changes
classes no team sees 2, 5 defects

The full team is 1021 minutes, that is, seventeen hours. This number has already made the decision: running the full team on every change is not an option. The pyramid’s arithmetic was done in The Test Pyramid lesson; here the pyramid is converted into time — the unit team is 1 minute, the end-to-end team 39, the performance team’s sampling run alone is 536 minutes.

Gate Arrangements

An arrangement sets the order of stages and each stage’s frequency: on every change, nightly, or per release. Frequency produces a wait.

TP8 — wait times: 240 minutes for the nightly gate, 1200 minutes for the release gate. Changes spread evenly across the workday; the nightly run fires once at day’s end, so the average wait is half a workday. Release is once a week, so the average wait is two and a half workdays.

TP9 — a hundred changes spread across twenty workdays: five changes a day, four releases. This assumption sets the run count and directly affects the machine cost.

TP10 — a runner works 480 minutes a day. The required runner count follows from this and is the arrangement’s hardware cost.

Fail fast is when a stage turning red keeps the following stages from running at all. The measurement below toggles this as an option.

// gate.mjs -- comparing seven gate arrangements over the same defect set
import { teams, defects, cost } from './inventory.mjs';

const minute = Object.fromEntries(teams.map((t) => [t.name, cost(t)]));
const WAIT = { change: 0, nightly: 240, release: 1200 };   // TP8
const RUNS = { change: 100, nightly: 20, release: 4 };      // TP9

const A = ['unit', 'contract'];
const B = ['integration', 'end-to-end'];
const C = ['perf-run', 'security-auto', 'resilience'];
const D = ['perf-sample', 'security-manual'];

const arrangements = [
  { name: 'single gate', failFast: true, stages: [['all', 'change', [...A, ...B, ...C, ...D]]] },
  { name: 'three stages', failFast: true, stages: [['fast', 'change', A], ['mid', 'change', B], ['heavy', 'change', [...C, ...D]]] },
  { name: 'three stages, no cut', failFast: false, stages: [['fast', 'change', A], ['mid', 'change', B], ['heavy', 'change', [...C, ...D]]] },
  { name: 'heavy stage nightly', failFast: true, stages: [['fast', 'change', A], ['mid', 'change', B], ['heavy', 'nightly', [...C, ...D]]] },
  { name: 'three frequencies', failFast: true, stages: [['fast', 'change', A], ['mid', 'change', B], ['heavy', 'nightly', C], ['release', 'release', D]] },
  { name: 'manual off pipeline', failFast: true, stages: [['fast', 'change', A], ['mid', 'change', B], ['heavy', 'nightly', C], ['release', 'release', ['perf-sample']]] },
  { name: 'three frequencies, quarantine', failFast: true, quarantine: true, stages: [['fast', 'change', A], ['mid', 'change', B], ['heavy', 'nightly', C], ['release', 'release', D]] },
];

const measure = (d) => {
  const stage = d.stages.map(([name, frequency, tk]) => ({ name, frequency, tk, duration: tk.reduce((a, x) => a + minute[x], 0) }));
  const priorDuration = (i) => {
    const f = stage[i].frequency;
    const moreFrequent = stage.filter((x) => RUNS[x.frequency] > RUNS[f]).reduce((a, x) => a + x.duration, 0);
    const same = stage.slice(0, i + 1).filter((x) => x.frequency === f).reduce((a, x) => a + x.duration, 0);
    return moreFrequent + WAIT[f] + same;
  };
  let missed = 0, weight = 0, totalFeedback = 0, worst = 0, cancelled = 0;
  for (const k of defects) {
    const visible = (t) => k.catches.includes(t) && (d.quarantine !== true || k.quarantine !== true);
    const i = stage.findIndex((x) => x.tk.some(visible));
    if (i === -1) { missed += k.count; continue; }
    const feedback = priorDuration(i);
    weight += k.count;
    totalFeedback += feedback * k.count;
    worst = Math.max(worst, feedback);
    // fail fast: change stages after a red stage do not run
    if (d.failFast && stage[i].frequency === 'change') {
      cancelled += k.count * stage.slice(i + 1).filter((x) => x.frequency === 'change').reduce((a, x) => a + x.duration, 0);
    }
  }
  const machine = (stage.reduce((a, x) => a + x.duration * RUNS[x.frequency], 0) - cancelled) / 100;
  return { machine, runner: Math.ceil((machine * 5) / 480), feedback: totalFeedback / weight, worst, missed };
};

const s = (x, n) => String(x).padStart(n);
console.log(`${'arrangement'.padEnd(32)}${s('min/change', 12)}${s('runner', 8)}${s('avg feedback', 14)}${s('worst', 9)}${s('missed/100', 12)}`);
for (const d of arrangements) {
  const r = measure(d);
  console.log(`${d.name.padEnd(32)}${s(r.machine.toFixed(1), 12)}${s(r.runner, 8)}`
    + `${s(r.feedback.toFixed(1), 14)}${s(r.worst, 9)}${s(r.missed, 12)}`);
}
const duration = (g) => g.reduce((a, x) => a + minute[x], 0);
console.log(`\nstage durations: fast ${duration(A)}, mid ${duration(B)}, nightly ${duration(C)}, release ${duration(D)} minutes`);
console.log(`night window 480 minutes; heavy stage in full ${duration([...C, ...D])} minutes, `
  + `${duration([...C, ...D]) <= 480 ? 'fits' : 'does not fit'}`);

arrangement                       min/change  runner  avg feedback    worst  missed/100
single gate                           1021.0      11        1021.0     1021           5
three stages                           426.3       5         228.1     1021           5
three stages, no cut                  1021.0      11         228.1     1021           5
heavy stage nightly                    231.9       3         276.1     1261           5
three frequencies                       98.2       2         252.6     2221           5
manual off pipeline                     86.2       1         107.7     1921          10
three frequencies, quarantine           98.2       2         264.0     2221           9

stage durations: fast 8, mid 41, nightly 136, release 836 minutes
night window 480 minutes; heavy stage in full 972 minutes, does not fit

Reading the Table

The first two rows give what splitting into stages buys. In the single-gate arrangement, every defect surfaces after 1021 minutes, because the result only comes out once the whole team finishes. Once the same team is split into three stages, average feedback drops to 228.1 minutes — escaped defects do not change, the missed defect set is the same. Splitting alone is a free gain, with no loss.

The third row isolates fail fast’s share. The same arrangement asks for 1021 minutes and eleven runners per change with the cut off; 426.3 minutes and five runners with it on. Feedback time is identical in both rows: fail fast does not tell anyone anything sooner, it only cancels the 1013 minutes of work that would have run after a red change. The gain is fifty-eight percent and comes directly from defect frequency — in a defect-free repository, fail fast buys nothing.

The fourth and fifth rows show frequency’s cost. Moving the heavy stage to the nightly gate leaves 231.9 minutes per change, but the last row reports a problem: the heavy stage in full is 972 minutes and does not fit the 480-minute night window. A nightly gate has to finish before night ends. So the heavy stage splits in two — the performance run, the automated security scan, and the resilience tests go nightly (136 minutes, fits the window), performance sampling and the manual security review go to the release gate. The three-frequency arrangement drops to 98.2 minutes per change and two runners; in exchange, the worst feedback rises from 1021 to 2221 minutes.

The sixth row is this course’s rule in its plainest form. When the manual security review is dropped from the pipeline entirely, 86.2 minutes and a single runner per change suffice, and average feedback drops to 107.7 minutes — and escaped defects rise from five to ten. The average improving is not a gain, it is a measurement trap: the latest-caught class is no longer caught at all, so it has also dropped out of the average’s denominator. Read on its own, feedback time always makes deleting a test look good.

The last row ties quarantine to the gate decision. When the end-to-end test quarantined in the Flakiness Management lesson is dropped from the pipeline, run cost does not change (98.2 minutes is the same, because the team’s duration is set by other tests) but the defect class it protected escapes: from five to nine. Quarantine looks like a maintenance decision; in the table, it is a gate decision.

Across the first five rows, escaped defects stay fixed at five. These five defects belong to two classes that no arrangement catches — semantic mismatch and process-lifetime state, both from the source lessons’ “missed” column. No arrangement asks for zero escaped defects; even the most expensive one misses five, because no test in the inventory was written to see that class.

The Return on the Decision

A gate’s value is in what its red result does. The fast and mid stages block merging: if the result is red, the change does not enter the main branch, and its owner is still at that work. The nightly gate cannot block merging, because the change has already entered; a red result opens a record, and that record’s owner is the set of changes that went into the nightly run — no single one of them is identified, and that is the 240-minute wait’s second cost. The release gate stops the publish, and the decision’s owner is not one person but the team approving the release.

These three returns are the real criterion for stage placement: a test is placed at the earliest stage where the decision its result corresponds to can actually be made. The manual security review sitting at the release gate is not a shortcoming — it is manual work’s nature: it waits its turn and cannot be placed at an automated gate.

Summary

• Four courses’ thirty lessons were collected into nine test teams; the full team is 1021 model minutes, and every cost was read from the source lessons’ wrap-up tables.
• Splitting the team into three stages dropped average feedback from 1021 to 228.1 minutes and did not change escaped defects.
• Fail fast, without changing feedback at all, dropped the machine cost per change from 1021 to 426.3 minutes and the runner count from eleven to five.
• The heavy stage in full is 972 minutes and does not fit the 480-minute night window; split in two, the cost per change drops to 98.2 minutes and the worst feedback rises to 2221.
• Dropping the manual review from the pipeline lowered average feedback to 107.7 minutes and raised escaped defects from five to ten; dropping the quarantined test moved escaped defects to nine.
• No arrangement zeroed out escaped defects: two defect classes are not seen by any test in the inventory.

Next Step

Every row in this table rests on an assumption: a stage’s duration is just the tests’ own run cost. But every team besides the fast stage wants a test environment — the integration team a dependent service, the end-to-end team a live system, the performance team a production-like environment. These environments are not unlimited. When two changes arrive at once, one waits, and its wait is added to the stage’s duration; adding environments lowers the wait but the cost grows linearly. The next lesson counts how many environments to keep at which stage, the wait a shared environment produces, and a reserved environment’s cost.