Pipeline Stages

The previous lesson measured the manual approval gate at the end of the pipeline and referred to everything before it as a single box, under the name “automated gates.” Inside that box there are several gates, and each one has its own account. This lesson opens the box: an eight-gate pipeline is built, a known defect set is run through it, and which gate catches each defect is tracked.

The question asked is the same for every gate. How many defects does it hold? Of those, how many does it hold alone — that is, if the gate were removed, how many would escape to production? How many defects does it pass, and how many clean changes does it stop for nothing? And how many minutes does it add? The fourth question can make the third worthless: a gate that holds a lot adds nothing if a later gate holds all the same defects anyway. At the end of the lesson each of the eight gates is removed one at a time, and the gate whose removal is better is shown by the numbers.

The Pipeline and the Defect Set

The model is the pipeline of the regional measurement network: the meter reading collector, verifier, billing, and work-order services; the development, staging, and production environments. Both the network and the pipeline are fictional; the defect set below is a model. Durations are model minutes, independent of the machine the measurement runs on.

PD13 — the pipeline is eight gates, in a fixed order: format checking, build, unit tests, contract tests, integration tests, packaging, a smoke test in the staging environment, and a release-health window in production. Fail-fast is on: gates after a red gate do not run.

PD14 — the known defect set is fourteen classes. For every class, the appearance count per hundred changes, the model minutes it costs if it escapes to production, and the catch probability per gate are written down. A gate not listed for a class never sees it. A single change can carry more than one defect.

PD15 — a defect caught after a red gate gets fixed, and the change runs again from the start, up to six attempts. The fix itself does not give birth to a new defect.

PD16 — every gate has a false-hold probability; a falsely held change reruns and spends 35 model minutes of human time.

PD17 — the total cost is the sum of three items: runner minutes, false holds × 35, and the class cost of defects that escape to production. The gate decision is made on this total.

PD18 — 400 changes are run; every draw is generated in advance, so that when a gate is removed, the remaining gates see the same numbers and the comparison isolates the gate.

// pipeline.mjs — eight-gate pipeline model and a known defect set
const generator = (seed) => () => {
  seed = (seed * 1664525 + 1013904223) >>> 0;
  return seed / 4294967296;
};

// PD13: gate order, model minutes per gate, false-hold probability per gate.
export const gates = [
  { name: 'format', min: 1, falseRate: 0.005 },
  { name: 'build', min: 4, falseRate: 0.004 },
  { name: 'unit', min: 3, falseRate: 0.012 },
  { name: 'contract', min: 11, falseRate: 0.040 },
  { name: 'integration', min: 14, falseRate: 0.030 },
  { name: 'packaging', min: 6, falseRate: 0.008 },
  { name: 'smoke', min: 9, falseRate: 0.045 },
  { name: 'health', min: 20, falseRate: 0.035 },
].map((g, gi) => ({ ...g, gi }));   // gi: the gate's fixed position in the full pipeline

// PD14: known defect set. count = appearances per hundred changes, cost = model minutes if it
// escapes to production, seen = catch probability per gate (an unlisted gate never sees it).
export const classes = [
  { name: 'format violation', count: 14, cost: 40, seen: { format: 1.0 } },
  { name: 'type mismatch', count: 8, cost: 300, seen: { build: 0.95, unit: 0.40 } },
  { name: 'missing dependency', count: 5, cost: 500, seen: { build: 0.80, packaging: 0.90 } },
  { name: 'boundary condition', count: 7, cost: 900, seen: { unit: 0.85, integration: 0.25 } },
  { name: 'null handling', count: 6, cost: 800, seen: { unit: 0.60, integration: 0.35, smoke: 0.40 } },
  { name: 'field rename', count: 4, cost: 1600, seen: { contract: 0.90, integration: 0.85 } },
  { name: 'request format drift', count: 3, cost: 1400, seen: { contract: 0.85, integration: 0.75, smoke: 0.35 } },
  { name: 'schema mismatch', count: 4, cost: 1800, seen: { integration: 0.80, smoke: 0.50 } },
  { name: 'migration ordering', count: 2, cost: 2600, seen: { integration: 0.35, smoke: 0.55, health: 0.40 } },
  { name: 'missing configuration', count: 5, cost: 1200, seen: { packaging: 0.45, smoke: 0.80 } },
  { name: 'package integrity', count: 2, cost: 1000, seen: { packaging: 0.90 } },
  { name: 'latency regression', count: 3, cost: 1500, seen: { smoke: 0.15, health: 0.70 } },
  { name: 'memory leak', count: 2, cost: 2300, seen: { health: 0.55 } },
  { name: 'business rule error', count: 3, cost: 2000, seen: { unit: 0.20, integration: 0.15, smoke: 0.10 } },
];

export const CHANGES = 400, ATTEMPTS = 6;    // PD15: up to six attempts
export const FALSE_MIN = 35;                 // PD16: human cost of a falsely held change

// PD18: every draw is generated in advance; removing a gate reuses the same numbers.
export const generateFlow = (seed) => {
  const r = generator(seed);
  return Array.from({ length: CHANGES }, () => ({
    defects: classes.filter((s) => r() < s.count / 100)
      .map((s) => ({ s, roll: Array.from({ length: ATTEMPTS * gates.length }, () => r()) })),
    falseRoll: Array.from({ length: ATTEMPTS * gates.length }, () => r()),
  }));
};

Four Numbers per Gate

The measurement runs twice: first the full pipeline, then eight times with one gate missing each. The marginal column is read directly from the removal trial — how much the defects escaping to production increase when the gate is taken out.

// gate.mjs — which gate catches each defect, three numbers per gate, and the removal trial
import { gates, classes, generateFlow, CHANGES, ATTEMPTS, FALSE_MIN } from './pipeline.mjs';

const runOnce = (lineup, flow) => {
  const tally = Object.fromEntries(lineup.map((g) => [g.name,
    { held: 0, passed: 0, falseHeld: 0, min: 0 }]));
  let escaped = 0, escapedCost = 0, runs = 0;
  for (const c of flow) {
    let remaining = c.defects.slice();
    for (let attempt = 0; attempt < ATTEMPTS; attempt++) {
      runs++;
      let stopper = null;
      for (const g of lineup) {
        const i = attempt * gates.length + g.gi;   // the roll's index is tied to the gate's fixed position
        tally[g.name].min += g.min;
        if (c.falseRoll[i] < g.falseRate) {                    // PD16: false hold
          tally[g.name].falseHeld++; stopper = { g, real: false }; break;
        }
        const caught = remaining.filter((x) => x.roll[i] < (x.s.seen[g.name] ?? 0));
        tally[g.name].passed += remaining.length - caught.length;
        if (caught.length > 0) {
          tally[g.name].held += caught.length;
          remaining = remaining.filter((x) => !caught.includes(x));  // PD15: fixed, reruns
          stopper = { g, real: true }; break;
        }
      }
      if (stopper === null) break;                             // green: what remains ships to production
    }
    escaped += remaining.length;
    escapedCost += remaining.reduce((a, x) => a + x.s.cost, 0);
  }
  const min = lineup.reduce((a, g) => a + tally[g.name].min, 0);
  const falseHeld = lineup.reduce((a, g) => a + tally[g.name].falseHeld, 0);
  return { tally, escaped, escapedCost, runs, min, falseHeld,
    total: min + falseHeld * FALSE_MIN + escapedCost };
};

const flow = generateFlow(88123);
const full = runOnce(gates, flow);
const defectTotal = flow.reduce((a, c) => a + c.defects.length, 0);
const s = (x, w) => String(x).padStart(w);

console.log(`changes ${CHANGES}, defects ${defectTotal}, defect classes ${classes.length}, gates ${gates.length}`);
console.log(`full pipeline: runs ${full.runs}, runner ${full.min} min, false holds ${full.falseHeld}, `
  + `escaped to production ${full.escaped} defects / ${full.escapedCost} min, total ${full.total} min\n`);

console.log(`${'gate'.padEnd(13)}${s('min', 5)}${s('held', 7)}${s('marginal', 10)}${s('passed', 9)}${s('false', 8)}${s('added min', 13)}`);
const removed = {};
for (const g of gates) {
  const y = runOnce(gates.filter((x) => x.name !== g.name), flow);
  removed[g.name] = y;
  console.log(`${g.name.padEnd(13)}${s(g.min, 5)}${s(full.tally[g.name].held, 7)}${s(y.escaped - full.escaped, 10)}`
    + `${s(full.tally[g.name].passed, 9)}${s(full.tally[g.name].falseHeld, 8)}${s(full.tally[g.name].min, 13)}`);
}

console.log(`\n${'lineup'.padEnd(18)}${s('runner min', 15)}${s('false', 8)}${s('escaped', 9)}${s('escaped min', 13)}${s('total min', 11)}${s('diff', 8)}`);
console.log(`${'full pipeline'.padEnd(18)}${s(full.min, 15)}${s(full.falseHeld, 8)}${s(full.escaped, 9)}${s(full.escapedCost, 13)}${s(full.total, 11)}${s('-', 8)}`);
for (const g of gates) {
  const y = removed[g.name];
  console.log(`${('no ' + g.name).padEnd(18)}${s(y.min, 15)}${s(y.falseHeld, 8)}${s(y.escaped, 9)}${s(y.escapedCost, 13)}`
    + `${s(y.total, 11)}${s(y.total - full.total, 8)}`);
}
const removalHelps = gates.filter((g) => removed[g.name].total < full.total);
console.log(`\ngate whose removal lowers the total: ${removalHelps.length ? removalHelps.map((g) => g.name).join(', ') : 'none'}`);
console.log('seed scan (gate whose removal lowers the total):');
for (const seed of [88123, 4402, 71905, 3311]) {
  const a = generateFlow(seed), t0 = runOnce(gates, a);
  const d = gates.filter((g) => runOnce(gates.filter((x) => x.name !== g.name), a).total < t0.total);
  console.log(`  ${seed}: ${d.length ? d.map((g) => g.name).join(', ') : 'none'} (escaped ${t0.escaped})`);
}

changes 400, defects 256, defect classes 14, gates 8
full pipeline: runs 713, runner 34140 min, false holds 89, escaped to production 27 defects / 39600 min, total 76855 min

gate           min   held  marginal   passed   false    added min
format           1     51        51      284       7          713
build            4     46         8      213       3         2620
unit             3     42        27      150       7         1818
contract        11     18         1      119      17         6138
integration     14     25        14       85      15         7336
packaging        6     19        13       63       1         2910
smoke            9     17        15       40      28         4185
health          20     11        12       28      11         8420

lineup                 runner min   false  escaped  escaped min  total min    diff
full pipeline               34140      89       27        39600      76855       -
no format                   33688      87       78        40440      77173     318
no build                    32422      87       35        43700      79167    2312
no unit                     32350      76       54        67500     102510   25655
no contract                 28180      74       28        40800      71570   -5285
no integration              26571      75       41        62100      91296   14441
no packaging                30906      90       40        51100      85156    8301
no smoke                    28216      51       42        59800      89801   12946
no health                   24520      76       39        60400      87580   10725

gate whose removal lowers the total: contract
seed scan (gate whose removal lowers the total):
  88123: contract (escaped 27)
  4402: contract (escaped 16)
  71905: format, contract (escaped 27)
  3311: none (escaped 25)

Holding versus Holding Alone

The held and marginal columns do not track each other. The build gate holds 46 defects, but when it is removed, defects escaping to production increase by only 8: most of what it holds is also seen by the unit and packaging gates. The contract gate holds 18, and its marginal is 1. The format gate, by contrast, holds 51, and all 51 are marginal — no other gate in the pipeline sees format violations.

The ranking reverses with a third number: the marginal defect’s cost. The format violation’s class cost is 40 minutes, the memory leak’s is 2,300, the migration ordering’s is 2,600. When the format gate is removed, the escape cost rises by 840 minutes; when the health gate is removed, it rises by 20,800 minutes — 1,733 minutes per defect for twelve defects. This is why the gate that holds the most is not the most valuable gate: removing the format gate raises the total by 318 minutes, removing the health gate by 10,725. A gate’s value is not in the number of defects it holds, but in the cost of the defects only it holds.

The passed column is a natural consequence of order, and it is not, on its own, an accusation. The format gate passes 284 defects because it stands first and sees every defect in the pipeline. The health gate passes 28 because few defects even reach it. This column only carries meaning when it is read together with the classes the gate is expected to see.

The added min column depends on how many runs the gate’s duration is multiplied by, and fail-fast pulls that multiplier down. Dividing the column by the gate’s duration gives the run count per gate: format 713, build 655, unit 606, contract 558, integration 524, packaging 485, smoke 465, health 421. The run count drops moving down the pipeline, because a change that stops at an earlier gate never reaches the later ones. The health gate is the most expensive per run (20 minutes) but runs the least; still, a quarter of the pipeline’s total runner load belongs to it.

The false-hold column is an independent axis. The smoke test gate stops a clean change for nothing 28 times, the highest count in the pipeline; yet its marginal is 15 defects, and removing it raises the total by 12,946 minutes. A gate with high flakiness stays in place if its marginal is enough — the fix is not to remove it, but to lower its flakiness.

The Gate Whose Removal Is Better

The second table ranks the eight removal trials by total cost. For seven gates, the diff column is positive: removal comes at a cost. The unit gate is the most expensive loss, 25,655 minutes — because it is the gate that actually sees the boundary condition and null handling classes, and it costs 3 minutes. The best trade-off per gate sits at the start of the pipeline.

One gate stands apart. Removing the contract gate lowers the total by 5,285 minutes. The gate’s own account explains why: it holds 18 defects, but when it is removed the integration gate catches 16 of them and two escape to production; removal also shifts the flow of attempts, so another defect starts being caught at the smoke gate and the marginal stays at 1 defect. In exchange it adds 6,138 runner minutes and stops 17 clean changes for nothing. When it is removed, defects escaping to production rise from 27 to 28 — 5,285 minutes for one defect. This gate is not badly built; its place in the pipeline is wrong. If it sees a field rename with probability 0.90 and integration sees the same with 0.85, the gain from holding both is measurably small.

The seed scan shows how robust the result is: the contract gate comes up in three of four seeds, in one seed no gate comes up, and in one the format gate also enters the list. So the decision is not “the contract test is unnecessary”; for this defect set, this pipeline order, and these gate durations, the contract gate’s marginal is under its cost. If the defect set changes — for instance, if contracts between services break more often — the same calculation brings the gate back. What carries the decision is not the gate’s name, but the four numbers in the table.

Summary

• The eight-gate pipeline was run with 400 changes and 256 defects: 713 runs, 34,140 runner minutes, 89 false holds, 27 defects escaping to production, and a 39,600-minute escape cost.
• What a gate holds and what it alone holds are different: build holds 46, marginal 8; contract holds 18, marginal 1; format holds 51, and all 51 are marginal.
• The gate that holds the most is not the most valuable gate: removing the format gate raises the total by 318 minutes, removing the health gate by 10,725.
• A false hold alone does not disqualify a gate: the smoke test gate stops 28 clean changes for nothing, but stays in place because its marginal is 15 defects.
• Removing the contract gate lowered the total by 5,285 minutes; 6,138 runner minutes and 17 false holds were being paid for one defect.
• The result depends on the gate’s four numbers, not its name: three of four seeds pointed to the same gate, one pointed to none.

Next Step

Across its 713 runs the pipeline produced something, and this lesson never counted it: the build artifact the packaging gate outputs. At the end of every run that finishes green, there is a package, and that package is what goes to production. Two questions remain unanswered. What does this package’s identity depend on — which commit, which branch, which run number — and can it be traced back from that identity to the source that produced it? Second, when the same package causes a failure in production, rolling back to the previous version requires that version to still be there; the stricter the retention rule, the narrower the rollback window, and the looser it is, the bigger the repository grows. The next lesson measures these two numbers together.