Continuous Delivery and Continuous Deployment

The previous lesson measured the gate at entry to the main branch and answered a single question: should this change get in? It never touched the question of what happens after a green main branch. There are two answers, and the difference between them lives in a single gate.

Continuous delivery is every green run producing an output that is release-ready: the output is packaged, tested, and fit to put into production, but putting it there depends on a human decision. Continuous deployment is every green run being in production: every gate is automated, and there is no human between them. The only structural difference between the two is the manual approval gate at the end of the pipeline. This lesson measures that gate: how many real defects it holds, how many of what it holds the automated gates already saw, how many clean changes it stops for nothing, and how long it holds things up.

Flow and Defect Classes

The measurement runs on the pipeline of the regional measurement network: the meter reading collector, verifier, billing, and work-order services. Both the network and the pipeline are fictional; the flow below is a model. Durations are model minutes.

PD7 — 320 changes enter the main branch in twenty workdays, one every thirty minutes. This frequency comes from the four-hour interval that gave the previous lesson’s lowest total: eight developers, once every four hours each. Below are the appearance counts, out of a hundred changes, for seven defect classes, and four probabilities — the probability the automated gate stops it, the probability the nightly suite catches it in one run, the probability the approver sees it in a small batch, and the probability the automated gate produces a warning without stopping.

PD8 — the pipeline’s own duration is 26 model minutes per change.

PD9 — the approver fully sees six changes; if the batch is larger than that, attention dilutes at a ratio of 6/n. Review examines a batch, not individual changes one by one. Review time is 15 + 4n model minutes, and two people make the decision: the approver and the release owner.

PD10 — the probability a clean batch is falsely rejected is 0.08. In the mode that turns the warning into a hard gate, the probability a clean change produces a false warning is 0.06.

PD11 — diagnosing a defect that escapes to production takes 30 + 12×(batch−1) model minutes. As the batch grows, the number of suspect changes grows, and the search is for which one brought the defect.

PD12 — the nightly suite runs once at the end of every day. Every run that fits within the wait time makes an independent draw against the class’s nightly probability.

// flow.mjs — change flow and defect classes; the same draws are used in every mode
const generator = (seed) => () => {
  seed = (seed * 1664525 + 1013904223) >>> 0;
  return seed / 4294967296;
};

// PD7: appearances per hundred changes. gate = probability the automated gates running on
// every run turn red, nightly = probability the nightly suite catches it in one run,
// manual = probability the approver sees it in a small batch, warn = probability the
// automated gates produce a warning without stopping.
export const classes = [
  { name: 'condition outside unit coverage', count: 9, gate: 0.90, nightly: 0.50, manual: 0.05, warn: 0.30 },
  { name: 'schema mismatch', count: 6, gate: 0.75, nightly: 0.60, manual: 0.10, warn: 0.40 },
  { name: 'configuration drift', count: 5, gate: 0.15, nightly: 0.35, manual: 0.60, warn: 0.55 },
  { name: 'migration-release ordering', count: 4, gate: 0.10, nightly: 0.20, manual: 0.75, warn: 0.45 },
  { name: 'business rule error', count: 4, gate: 0.20, nightly: 0.15, manual: 0.30, warn: 0.10 },
  { name: 'threshold error', count: 3, gate: 0.10, nightly: 0.30, manual: 0.12, warn: 0.35 },
  { name: 'concurrency race', count: 3, gate: 0.05, nightly: 0.10, manual: 0.00, warn: 0.05 },
];

export const CHANGES = 320, INTERVAL = 30;    // PD7: twenty workdays, one merge every 30 minutes
export const DAY_MIN = 480, WEEK_MIN = 2400;

// Every change's full set of draws is generated in advance: modes are compared on the same numbers.
export const generateFlow = (seed) => {
  const r = generator(seed);
  return Array.from({ length: CHANGES }, (_, i) => {
    let x = r() * 100, cls = null;
    for (const s of classes) { x -= s.count; if (x <= 0) { cls = s; break; } }
    return { t: i * INTERVAL, cls, gateRoll: r(), manualRoll: r(), warnRoll: r(),
      falseRoll: r(), nightlyRoll: Array.from({ length: 8 }, () => r()) };
  });
};

Six Modes

The six modes being compared split two axes: the release window (waiting) and review (human). The third row is the key to this split — there is a daily window but no review, so the wait’s own gain is measured apart from the human’s.

// mode.mjs — the same pipeline in six modes: every green run in production (deployment) or production-ready (delivery)
import { generateFlow, CHANGES, DAY_MIN, WEEK_MIN } from './flow.mjs';

const PIPELINE_MIN = 26;                                  // PD8: the pipeline's own duration
const ATTENTION = 6;                                       // PD9: fully-seen batch size
const REVIEW_BASE = 15, REVIEW_UNIT = 4, REVIEW_PEOPLE = 2; // PD9: review time and deciding people
const FALSE_REJECT = 0.08, FALSE_WARN = 0.06;               // PD10: false-stop probabilities
const DIAGNOSIS_BASE = 30, DIAGNOSIS_UNIT = 12;             // PD11: diagnosis time for a production defect

const modes = [
  { name: 'deployment', window: 0, review: false },
  { name: 'delivery, every run', window: 0, review: true },
  { name: 'daily window, no review', window: DAY_MIN, review: false },
  { name: 'delivery, daily review', window: DAY_MIN, review: true },
  { name: 'delivery, weekly review', window: WEEK_MIN, review: true },
  { name: 'deployment, warning hard gate', window: 0, review: false, hard: true },
];

const measure = (mode, flow) => {
  const batch = new Map();
  let hardHeld = 0, falseRejects = 0;
  for (const c of flow) {
    if (c.cls && c.gateRoll < c.cls.gate) continue;          // automated gate turned red
    let extra = 0;
    if (mode.hard) {
      if (c.cls && c.warnRoll < c.cls.warn) { hardHeld++; continue; }
      if (!c.cls && c.falseRoll < FALSE_WARN) { falseRejects++; extra = PIPELINE_MIN; }
    }
    const ready = c.t + PIPELINE_MIN + extra;
    const release = mode.window === 0 ? ready : Math.ceil(ready / mode.window) * mode.window;
    (batch.get(release) ?? batch.set(release, []).get(release)).push({ c, ready });
  }
  let shipped = 0, decisions = 0, wait = 0, worst = 0, held = 0, alreadyWarned = 0;
  let reviewMin = 0, escaped = 0, diagnosisMin = 0, batchTotal = 0, releaseCount = 0;
  for (const [release, records] of [...batch.entries()].sort((a, b) => a[0] - b[0])) {
    const n = records.length;
    releaseCount++; batchTotal += n;
    const attention = Math.min(1, ATTENTION / n);             // PD9: dilutes as the batch grows
    let rejected = false;
    if (mode.review) { decisions += REVIEW_PEOPLE; reviewMin += (REVIEW_BASE + REVIEW_UNIT * n) * REVIEW_PEOPLE; }
    for (const { c, ready } of records) {
      // PD12: how many nightly runs fit within the wait
      const nightlyRuns = Math.floor(release / DAY_MIN) - Math.floor(ready / DAY_MIN);
      let stopped = false;
      if (c.cls) {
        for (let k = 0; k < Math.min(nightlyRuns, 8); k++) {
          if (c.nightlyRoll[k] < c.cls.nightly) { stopped = true; break; }
        }
        if (!stopped && mode.review && c.manualRoll < c.cls.manual * attention) {
          stopped = true; rejected = true; held++;
          if (c.warnRoll < c.cls.warn) alreadyWarned++;        // automated gate had already warned
        }
      }
      if (stopped) continue;
      if (c.cls) { escaped++; diagnosisMin += DIAGNOSIS_BASE + DIAGNOSIS_UNIT * (n - 1); }
      shipped++; wait += release - ready; worst = Math.max(worst, release - ready);
    }
    if (mode.review && !rejected && records[0].c.falseRoll < FALSE_REJECT) falseRejects += n;
  }
  return { releaseCount, batch: batchTotal / releaseCount, shipped, decisions, wait: wait / shipped,
    worst, held, alreadyWarned, falseRejects, reviewMin, escaped, diagnosisMin, hardHeld };
};

const flow = generateFlow(51301);
const results = modes.map((m) => ({ name: m.name, r: measure(m, flow) }));
const find = (name) => results.find((x) => x.name === name).r;
const s = (x, w) => String(x).padStart(w);
const f = (x, w) => String(x.toFixed(1)).padStart(w);

console.log(`changes ${CHANGES}, defect-bearing ${flow.filter((c) => c.cls).length}\n`);
console.log(`${'mode'.padEnd(32)}${s('release', 9)}${s('avg batch', 11)}${s('shipped', 15)}${s('human decisions', 17)}${s('avg wait', 13)}${s('worst', 9)}`);
for (const { name, r } of results) {
  console.log(`${name.padEnd(32)}${s(r.releaseCount, 9)}${f(r.batch, 11)}${s(r.shipped, 15)}`
    + `${s(r.decisions, 17)}${f(r.wait, 13)}${s(r.worst, 9)}`);
}
console.log(`\n${'mode'.padEnd(32)}${s('review held', 13)}${s('already warned', 16)}${s('escaped', 15)}${s('false rejected', 16)}${s('human min', 11)}${s('diag min', 11)}`);
for (const { name, r } of results) {
  console.log(`${name.padEnd(32)}${s(r.held, 13)}${s(r.alreadyWarned, 16)}${s(r.escaped, 15)}`
    + `${s(r.falseRejects, 16)}${s(r.reviewMin, 11)}${s(r.diagnosisMin, 11)}`);
}
const hard = find('deployment, warning hard gate');
const summary = (seed) => {
  const a = generateFlow(seed), h = (name) => measure(modes.find((m) => m.name === name), a);
  return { single: h('deployment').escaped - h('delivery, every run').escaped,
    day: h('daily window, no review').escaped - h('delivery, daily review').escaped,
    hard: h('deployment, warning hard gate').escaped, dep: h('deployment').escaped };
};
console.log(`\nreview's marginal: no window ${find('deployment').escaped - find('delivery, every run').escaped} defects, `
  + `daily window ${find('daily window, no review').escaped - find('delivery, daily review').escaped} defects`);
console.log(`daily window's no-review gain ${find('deployment').escaped - find('daily window, no review').escaped} defects, human minutes 0`);
console.log(`hard gate stopped ${hard.hardHeld} changes, ${hard.falseRejects} of them were for nothing; escaped to production ${hard.escaped}`);
console.log('seed scan (review marginal single/daily, hard gate escaped, no-review deployment escaped):');
for (const seed of [51301, 8802, 33107, 6449]) {
  const o = summary(seed);
  console.log(`  ${seed}: ${o.single}/${o.day}, ${o.hard}, ${o.dep}`);
}

changes 320, defect-bearing 116

mode                              release  avg batch        shipped  human decisions     avg wait    worst
deployment                            264        1.0            264                0          0.0        0
delivery, every run                   264        1.0            242              528          0.0        0
daily window, no review                20       13.2            247                0        224.8      454
delivery, daily review                 20       13.2            242               40        226.5      454
delivery, weekly review                 4       66.0            239                8       1152.0     2374
deployment, warning hard gate         236        1.0            236                0          0.0        0

mode                              review held  already warned        escaped  false rejected  human min   diag min
deployment                                  0               0             60               0          0       1800
delivery, every run                        22              14             38              16      10032       1140
daily window, no review                     0               0             43               0          0       7890
delivery, daily review                      5               3             38              26       2712       6972
delivery, weekly review                     2               2             35               0       2232      28662
deployment, warning hard gate               0               0             32               3          0        960

review's marginal: no window 22 defects, daily window 5 defects
daily window's no-review gain 17 defects, human minutes 0
hard gate stopped 28 changes, 3 of them were for nothing; escaped to production 32
seed scan (review marginal single/daily, hard gate escaped, no-review deployment escaped):
  51301: 22/5, 32, 60
  8802: 20/6, 45, 60
  33107: 14/6, 50, 64
  6449: 21/6, 48, 69

Who Gains From the Wait

The first table gives the flow difference between the two modes. In deployment mode, twenty workdays produce 264 releases, 264 changes reach production, and there is no human decision at all. In weekly-reviewed delivery, the release count is four, the batch averages 66 changes, human decisions number eight, and a change that is ready but not yet released waits an average of 1,152 model minutes, 2,374 in the worst case — roughly five workdays. Same pipeline, same output, same tests; the difference is only in the gate at the end.

At first glance, the second table’s escaped column justifies waiting: 60 in deployment, 35 in weekly. The critical row is the third one. The daily window, no review mode brings escaped defects down from 60 to 43, and it does this with zero human minutes. A human did not find these 17 defects; the nightly runs that fit inside the wait time did. Most of the improvement credited to manual approval’s account is actually the gain from the extra time granted to the machine — and that time can be granted without a human too.

The wait’s bill, meanwhile, is in the diag min column. In deployment, diagnosing 60 defects takes 1,800 minutes, 30 minutes per defect: a release has a single change, and the suspect is obvious. In weekly mode, diagnosing 35 defects takes 28,662 minutes, 819 minutes per defect — twenty-seven times as much. As the batch grows, every failure in production has to be picked out from among 66 candidates. Waiting shrinks the number of defects that escape, and grows the cost of the ones that do.

Manual Approval’s Marginal Value

The approval gate’s marginal value is the difference in escaped defects between the reviewed and unreviewed run on the same draws. When the batch is a single change, this difference is 22 defects — approval is a serious gate. In the daily window, the same difference drops to 5 defects. The gate itself has not changed; what changed is the batch it is examining. The approval gate generates its own wait, the wait grows the batch, and the batch dilutes attention: for a batch of 13.2 changes, attention is 6/13.2, that is, 0.45. By its own existence, the gate cuts its own effect by 77 percent. In the weekly window, at a batch of 66, the marginal is 2 defects.

The second number is even sharper. The already warned column counts, out of the defects approval holds, how many the automated gates had already produced a warning for: 14 of 22 when the batch is one, 3 of 5 in the daily window. For these defects, the human just read a line the machine had already written. Had the warning been turned into a hard gate, these would have been stopped without approval too. Approval’s contribution that belongs only to itself is 8 and 2 defects in the two cases.

The last row carries this thought to its end. In the deployment mode where warnings are turned into a hard gate, 28 changes are stopped, three of them for nothing; escaped defects drop to 32. This beats every mode that includes waiting or review: zero human minutes, zero wait, 30 minutes of diagnosis per defect. The seed scan shows the numbers move — the hard gate’s escaped count ranges from 32 to 50, unreviewed deployment’s from 60 to 69 — but the ranking is the same across all four seeds.

The manual approval gate’s three numbers can now be written. What it holds: 5 defects in the daily window, 2 of them held only by itself. What it misses and falsely holds: 38 defects go to production, 26 clean changes are stopped for nothing. The time it adds: 2,712 human minutes and 226.5 minutes of wait per change. Once these three lines are written side by side, the gate becomes indefensible; the sentence “let someone approve it” carries no number on its own.

Summary

• The only structural difference between continuous delivery and continuous deployment is the manual approval gate standing between a green run and production; the measurement was made on this gate.
• Manual approval’s marginal value depends on the batch: 22 defects when the batch is one change, 5 in the daily window, 2 in the weekly window. The gate generates its own wait and cuts its own effect by 77 percent.
• For most of the defects approval holds, the automated gates had already produced a warning (14 of 22, 3 of 5); approval’s unique contribution is 8 and 2 defects.
• The gain from waiting belongs to the machine, not the human: the unreviewed daily window brought escaped defects down from 60 to 43, at zero human minutes.
• Waiting grows the cost of what escapes: diagnosis per defect rose from 30 minutes to 819 minutes, because a batch of 66 has 66 suspects.
• Unreviewed deployment with the warning turned into a hard gate brought escaped defects down to 32; with zero human minutes and zero wait, it beat every reviewed mode.

Next Step

The gate this lesson measured sat at the end of the pipeline, and everything before it was referred to as a single box, under the name “automated gates.” Inside that box there are several gates: build, test, packaging, and release. Since the marginal value of the approval at the end could be calculated, the marginal value of every gate inside it can be calculated too — which defect only that gate holds, which one it misses, how many clean changes it stops for nothing, and even how many minutes it adds. The next lesson runs a known set of defects through a pipeline model, tracks which gate catches each defect, and tries removing one of the gates.