--- title: 'Continuous Integration and Delivery' source: 'https://academia.sh/en/courses/ci-cd' language: en updated: '2026-09-13T05:08:17+00:00' license: 'CC BY-SA 4.0' lessons: 21 outcomes: - 'Connects the build, test, and release steps into a single pipeline' - 'Chooses deployment strategies according to the risk level' - "Ties the pipeline's own security and output integrity to numbers" --- # Continuous Integration and Delivery Who changes the desired state, on what evidence, and through which check: frequent merging and fast feedback, the separation of delivery from deployment, the pipeline stages made of build, test, packaging, and release gates, build artifact versioning and retention, shortening pipeline time with caching and parallelism, runner selection, the declarative definition of the pipeline itself, triggers, branching strategy, environment promotion, progressive delivery, blue-green deployment, separating deployment from release with feature flags, the rollback plan, coordination with data migrations, repository-defined desired state, pull-based deployment, secret management, the dependency inventory, build artifact signing, and pipeline security. ## Pipeline Design Building the gates on a change's path to production: frequent merging's effect on conflict and feedback, the separation of continuous delivery from continuous deployment, the defects the build, test, packaging, and release gates hold and miss, versioning and retaining the build artifact, the limit on shortening pipeline time with caching and parallelism, the cost of managed and self-hosted runners, the pipeline's declarative definition, and event, schedule, and manual triggering. ### Continuous Integration Source: https://academia.sh/en/courses/ci-cd/continuous-integration Scanning the merge interval: how conflicting files, resolution rounds, and errors born in resolution grow as the interval grows, how many runs and how many runner minutes frequency's own cost takes, and what the merge gate alone catches and how many clean changes it stops for nothing. The previous course left a cluster behind: the desired state is written down somewhere, a reconciler keeps applying it, and the actual state is pulled back whenever it drifts. The question left standing at its close was — who changes that desired state, **on what evidence, and through which check**? This course builds that path. Every step that stands on the route from a developer's machine to production, and either lets a change through or does not, is a **gate**, and the gate is this course's unit of measure. The first gate on the path is the oldest one: a change entering the main branch. Continuous integration is every developer returning their work to the main branch at frequent intervals and running an automated check on every return. "Frequent" is not a number but an axis, and this lesson scans that axis. What gets scanned is the **merge interval**: how long a branch lives apart from the main branch. As the interval grows, two things grow together — the number of conflicting files and the errors born while resolving the conflict. As the interval shrinks, something else grows — the number of runs and runner time. This is where frequency stops being free. The **feedback delay** measured in the Process Models topic is not remeasured here; what this lesson measures is the conflict itself. #### The Repository Model The measurement runs on the repository of the regional measurement network: the service that collects water-meter readings, the service that verifies the readings, and the billing and work-order services. **This network is fictional; the repository below is a model too, and no real repository was scanned.** Durations are model minutes and are independent of the machine the measurement runs on. **PD1 — the repository is 120 files split across four services; eight developers work twenty workdays, and a day is 480 model minutes.** Each service's first three files are **interface files**: they carry another service's expectations. **PD2 — a developer touches 0.5 files per hour.** Total work is independent of the interval: every scan row has 640 file touches, and the only thing that changes is how many merges this work is split into. **PD3 — one fifth of the files are hot and are touched eight times as often; a developer prefers their own service 75 percent of the time.** The generator is self-written and the seed is visible. **PD4 — resolution per conflicting file is 20 model minutes.** The main branch keeps advancing while resolution runs; if a new merge touches the same files during that time, a new round opens, up to five rounds. **PD5 — the probability that resolving a conflicting file gives birth to an error is 0.09.** Finding and fixing, later on, either an error born in resolution or an interface mismatch the gate missed costs 180 model minutes. **PD6 — one pipeline run spends 14 model minutes of runner time; two percent of runs flake red** and stop a clean change for nothing, demanding one more run. ```js // repo.mjs — model of the fictional regional measurement network's repository: service, file, developer // Seeded generator: 32-bit linear congruential map; no external library is used. export const generator = (seed) => () => { seed = (seed * 1664525 + 1013904223) >>> 0; return seed / 4294967296; }; // PD1: four services, files per service; the first three files are interface files. export const services = [ { name: 'collector', files: 38 }, { name: 'verifier', files: 26 }, { name: 'billing', files: 31 }, { name: 'work-order', files: 25 }, ]; export const files = services.flatMap((s, si) => Array.from({ length: s.files }, (_, i) => ({ name: `${s.name}/${i}`, service: si, interface: i < 3, // PD3: one fifth of the files are hot and touched eight times as often. weight: i % 5 === 0 ? 8 : 1, }))); // Weighted file pick: a developer prefers their own service 75 percent of the time. export const pickFile = (r, developer) => { const own = r() < 0.75; const pool = own ? files.filter((f) => f.service === developer % 4) : files; const total = pool.reduce((a, f) => a + f.weight, 0); let x = r() * total; for (const f of pool) { x -= f.weight; if (x <= 0) return f; } return pool[pool.length - 1]; }; export const DEVELOPERS = 8; export const DAY = 20, DAY_MIN = 480; // PD1: twenty workdays, 480 model minutes per day export const RATE = 0.5; // PD2: 0.5 files touched per hour export const RESOLVE_MIN = 20; // PD4: resolution time per conflicting file export const ERROR_PROBABILITY = 0.09; // PD5: probability an error is born during resolution export const LATER_MIN = 180; // PD5: cost of a later-found error export const FLAKY = 0.02; // PD6: two percent of runs falsely hold export const RUN_MIN = 14; // PD6: runner time of one pipeline run ``` #### Scanning the Interval The scan runs the same work for six intervals: two hours, four hours, one day, two days, four days, and ten workdays. Every row starts with the same seed, so what is being compared is the interval itself. ```js // interval.mjs — conflict and gate cost when the merge interval is scanned import * as R from './repo.mjs'; const simulate = (hours, seed = 7391) => { const r = R.generator(seed); // every row starts with the same seed const T = hours * 60, horizon = R.DAY * R.DAY_MIN, n = R.RATE * hours; const trunk = [], events = []; // trunk: merged files {t, file, ...} for (let dev = 0; dev < R.DEVELOPERS; dev++) { for (let k = 0; k < horizon / T; k++) events.push({ dev, start: k * T + dev * (T / R.DEVELOPERS) }); } events.sort((a, b) => a.start - b.start); let merges = 0, conflicts = 0, rounds = 0, errors = 0, semantic = 0, falseHold = 0, runs = 0, resolveMin = 0; for (const e of events) { const touched = new Set(), ifaceServices = new Set(); while (touched.size < n) { // each branch touches n files const f = R.pickFile(r, e.dev); if (touched.has(f.name)) continue; touched.add(f.name); if (f.interface) ifaceServices.add(f.service); } merges++; let t = e.start + T, window = e.start, step = 0; while (true) { const rivals = trunk.filter((g) => g.t > window && g.t <= t && g.dev !== e.dev); const c = new Set(rivals.filter((g) => touched.has(g.file)).map((g) => g.file)).size; runs++; if (r() < R.FLAKY) { falseHold++; runs++; } // flaky red: stops a clean change for nothing if (step === 0) { // interface mismatch with no textual conflict semantic += new Set(rivals.filter((g) => g.interface && ifaceServices.has(g.service) && !touched.has(g.file)) .map((g) => g.file)).size; } if (c === 0 || step >= 5) break; conflicts += c; rounds++; for (let i = 0; i < c; i++) if (r() < R.ERROR_PROBABILITY) errors++; const duration = c * R.RESOLVE_MIN; resolveMin += duration; window = t; t += duration; step++; } for (const name of touched) { const f = R.files.find((x) => x.name === name); trunk.push({ t, file: name, dev: e.dev, service: f.service, interface: f.interface }); } } const laterCost = (errors + semantic) * R.LATER_MIN; return { hours, n, merges, laterCost, conflicts, rounds, errors, semantic, falseHold, runs, resolveMin, runMin: runs * R.RUN_MIN, touches: trunk.length }; }; const pad = (x, w) => String(x).padStart(w); const rows = [2, 4, 8, 16, 32, 80].map((h) => simulate(h)); const f2 = (x, w) => String(x.toFixed(2)).padStart(w); console.log(`${'interval'.padEnd(9)}${pad('branch files', 14)}${pad('merges', 12)}${pad('conflicting', 13)}${pad('conflict/mrg', 14)}${pad('round/mrg', 11)}${pad('resolve errs', 14)}${pad('iface missed', 16)}`); for (const x of rows) { console.log(`${(x.hours + ' hr').padEnd(9)}${pad(x.n, 14)}${pad(x.merges, 12)}${pad(x.conflicts, 13)}` + `${f2(x.conflicts / x.merges, 14)}${f2(x.rounds / x.merges, 11)}${pad(x.errors, 14)}${pad(x.semantic, 16)}`); } console.log(`\n${'interval'.padEnd(9)}${pad('runs', 7)}${pad('runner min', 15)}${pad('resolve min', 13)}${pad('late-found min', 16)}${pad('total min', 11)}${pad('false hold', 14)}`); for (const x of rows) { console.log(`${(x.hours + ' hr').padEnd(9)}${pad(x.runs, 7)}${pad(x.runMin, 15)}${pad(x.resolveMin, 13)}${pad(x.laterCost, 16)}` + `${pad(x.runMin + x.resolveMin + x.laterCost, 11)}${pad(x.falseHold, 14)}`); } const best = rows.reduce((a, x) => (x.runMin + x.resolveMin + x.laterCost < a.runMin + a.resolveMin + a.laterCost ? x : a)); console.log(`\ntouched files in every row ${rows.map((x) => x.touches).join('/')} (work is constant); ` + `for seed 7391 the lowest total is in the ${best.hours}-hour interval`); const at = (seed) => [2, 4, 8, 16, 32, 80].map((h) => simulate(h, seed)) .reduce((a, x) => (x.runMin + x.resolveMin + x.laterCost < a.runMin + a.resolveMin + a.laterCost ? x : a)); console.log('other seeds: ' + [20260801, 4517, 66049, 131].map((s) => `${s} -> ${at(s).hours} hr`).join(', ')); ``` ``` interval branch files merges conflicting conflict/mrg round/mrg resolve errs iface missed 2 hr 1 640 74 0.12 0.12 12 3 4 hr 2 320 90 0.28 0.27 4 10 8 hr 4 160 205 1.28 0.81 22 17 16 hr 8 80 320 4.00 0.97 31 26 32 hr 16 40 484 12.10 1.52 41 32 80 hr 40 16 654 40.88 1.63 55 43 interval runs runner min resolve min late-found min total min false hold 2 hr 731 10234 1480 2700 14414 17 4 hr 407 5698 1800 2520 10018 2 8 hr 297 4158 4100 7020 15278 8 16 hr 163 2282 6400 10260 18942 5 32 hr 103 1442 9680 13140 24262 2 80 hr 43 602 13080 17640 31322 1 touched files in every row 640/640/640/640/640/640 (work is constant); for seed 7391 the lowest total is in the 4-hour interval other seeds: 20260801 -> 4 hr, 4517 -> 8 hr, 66049 -> 4 hr, 131 -> 4 hr ``` #### Conflict Does Not Grow Linearly The first table's real column is `conflict/mrg`. When branch lifetime grows forty-fold from two hours, conflicting files per merge climb from 0.12 to 40.88 — a factor of 354. The growth is not proportional to files per branch, because conflict is the **intersection** of two sets: both the branch's own set and the total work landing on the main branch during that time grow. Two long-lived branches both raise the odds of touching the same hot files. The `round/mrg` column does not, at first glance, keep pace with this growth: it climbs from 0.12 to 1.63 and saturates there. The reason is inside the measurement — a resolution round opens when the main branch moves while resolution is running. When branches are long-lived, merges are sparse, so the main branch usually does not move during resolution and one round is enough. The cost of a long branch does not show up in the round count; it shows up in **round length**: resolution time per merge is 2.3 model minutes on the two-hour branch and 818 model minutes on the ten-workday branch. Errors born in resolution also grow with the interval: from 12 to 55. These are defects the merge itself **produces**; nobody wrote wrong code, they were born while two correct changes were reconciled by hand. The value of 4 in the four-hour row sits below its neighbors; because these numbers are small, that is seed noise, not a trend. #### Frequency's Own Cost The second table gives the reverse direction. In the two-hour interval the pipeline runs 731 times and spends 10,234 model minutes of runner time; in the ten-workday interval, 43 runs and 602 minutes. Frequent merging is seventeen times more expensive to the machine. The number that has to be read alongside that column is `false hold`: a flaky red is a fixed probability per run, so as runs grow, false stops grow with them — 17 against 1. On a team that merges frequently, the number of changes the gate unfairly stops is seventeen times that of a team that merges rarely, and the urge to skip the gate is born right there. The total column combines three items: runner time, resolution time, and the cost of late-found errors. The curve rises from both ends, and the lowest value sits at four hours, 10,018 minutes. Four of five seeds pick the same interval; one picks eight hours. **The result is not "the more often, the better"**: in the two-hour interval, runner time and false holds together inflate the total by more than 4,000 minutes; in the ten-workday interval, resolution and late finding push it up to 30,000 minutes. The optimum of the interval sits where the cost of a run intersects the cost of a conflict; as a run gets cheaper, the optimum shifts toward more frequent merging, and that is the real reason later lessons work to shorten pipeline time. #### The Merge Gate's Three Numbers This gate is the entry gate to the main branch, and it is written with three numbers. **What it holds:** in the four-hour interval, 90 conflicting files become visible at merge time. All of this belongs **only to this gate** — no later gate in the pipeline can see a textual conflict. The build gate can only report a badly resolved merge as "broken"; it cannot say which two changes collided. The conflict information exists at merge time, then it is gone. **What it misses:** interface mismatches. These are cases where two developers touch **different** files — no textual conflict — but one's expectation diverges from what the other delivers. 10 of these pass through the gate in the four-hour interval, 43 in the ten-workday interval. Errors born in resolution also pass through the gate, because the gate does not check the resolution itself. **What it falsely holds:** 2 clean changes in the four-hour interval, 17 in the two-hour interval. **The time it adds:** 14 minutes of runner time per run and 20 minutes of human time per conflicting file; 23 minutes per change in the four-hour interval. The gate cannot be defended unless these four numbers are read together. An evaluation that looks only at runner time finds frequent merging expensive and grows the interval; one that looks only at conflict count wants to shrink the interval to zero. Both are wrong, because the gate's value is the difference between what it holds and what it misses, and that difference depends on the interval. The same gate holds 74 conflicts and misses 3 mismatches in the two-hour interval, and holds 654 while missing 43 in the ten-workday interval: holding more does not mean holding better — most of what it holds, the interval itself produced. The two items it misses say this gate is not enough on its own, and that is the reason for the rest of the pipeline: the contract test is what will catch the interface mismatch, and the test gate is what will catch the error born in resolution. The order of the gates and what each one catches is the subject of the course's later lessons. #### Summary - When the merge interval grew from two hours to ten workdays, conflicting files per merge rose from 0.12 to 40.88, errors born in resolution from 12 to 55, and interface mismatches the gate missed from 3 to 43. - The resolution-round count saturated at 1.63; the cost of a long branch is not in the round count but in round length — resolution time per merge rose from 2.3 minutes to 818 model minutes. - Frequency's own cost was counted: the two-hour interval takes 731 runs and 10,234 runner minutes, the ten-workday interval takes 43 runs and 602 minutes. - False holds are proportional to run count: 17 against 1; the urge to skip the gate is born from this number. - When the three items are added together, the lowest value sits in the four-hour interval (10,018 minutes), and four of five seeds point to the same place; as a run gets cheaper, the optimum shifts toward merging more often. #### Next Step This lesson's gate sits at entry to the main branch, and its decision is the answer to a single question: should this change get in? The question of what happens after the main branch turns green was never touched. Is a green main branch something that can be put into production, or only something that is a **candidate** for production? The two answers build two different arrangements, and the difference between them collects into a single gate: the human approval standing between a green run and production. The next lesson runs the same pipeline in two modes and counts the real defects that approval catches, together with the time it holds up. ### Continuous Delivery and Continuous Deployment Source: https://academia.sh/en/courses/ci-cd/continuous-delivery-and-continuous-deployment Running the same pipeline in two modes: the difference between every green run being production-ready (delivery) and every green run being in production (deployment) is measured by the change reaching production, the number of people deciding, the wait time, and the real defects manual approval catches; the approval's marginal value is tied to batch size. The previous lesson measured the gate at entry to the main branch and answered a single question: should this change get in? It never touched the question of what happens after a green main branch. There are two answers, and the difference between them lives in a single gate. Continuous delivery is every green run producing an output that is **release-ready**: the output is packaged, tested, and fit to put into production, but putting it there depends on a human decision. Continuous deployment is every green run being **in production**: every gate is automated, and there is no human between them. The only structural difference between the two is the **manual approval gate** at the end of the pipeline. This lesson measures that gate: how many real defects it holds, how many of what it holds the automated gates already saw, how many clean changes it stops for nothing, and how long it holds things up. #### Flow and Defect Classes The measurement runs on the pipeline of the regional measurement network: the meter reading collector, verifier, billing, and work-order services. **Both the network and the pipeline are fictional; the flow below is a model.** Durations are model minutes. **PD7 — 320 changes enter the main branch in twenty workdays, one every thirty minutes.** This frequency comes from the four-hour interval that gave the previous lesson's lowest total: eight developers, once every four hours each. Below are the appearance counts, out of a hundred changes, for seven defect classes, and four probabilities — the probability the automated gate stops it, the probability the nightly suite catches it in one run, the probability the approver sees it in a small batch, and the probability the automated gate **produces a warning without stopping**. **PD8 — the pipeline's own duration is 26 model minutes per change.** **PD9 — the approver fully sees six changes; if the batch is larger than that, attention dilutes at a ratio of 6/n.** Review examines a batch, not individual changes one by one. Review time is 15 + 4n model minutes, and **two people make the decision**: the approver and the release owner. **PD10 — the probability a clean batch is falsely rejected is 0.08.** In the mode that turns the warning into a hard gate, the probability a clean change produces a false warning is 0.06. **PD11 — diagnosing a defect that escapes to production takes 30 + 12×(batch−1) model minutes.** As the batch grows, the number of suspect changes grows, and the search is for which one brought the defect. **PD12 — the nightly suite runs once at the end of every day.** Every run that fits within the wait time makes an independent draw against the class's nightly probability. ```js // flow.mjs — change flow and defect classes; the same draws are used in every mode const generator = (seed) => () => { seed = (seed * 1664525 + 1013904223) >>> 0; return seed / 4294967296; }; // PD7: appearances per hundred changes. gate = probability the automated gates running on // every run turn red, nightly = probability the nightly suite catches it in one run, // manual = probability the approver sees it in a small batch, warn = probability the // automated gates produce a warning without stopping. export const classes = [ { name: 'condition outside unit coverage', count: 9, gate: 0.90, nightly: 0.50, manual: 0.05, warn: 0.30 }, { name: 'schema mismatch', count: 6, gate: 0.75, nightly: 0.60, manual: 0.10, warn: 0.40 }, { name: 'configuration drift', count: 5, gate: 0.15, nightly: 0.35, manual: 0.60, warn: 0.55 }, { name: 'migration-release ordering', count: 4, gate: 0.10, nightly: 0.20, manual: 0.75, warn: 0.45 }, { name: 'business rule error', count: 4, gate: 0.20, nightly: 0.15, manual: 0.30, warn: 0.10 }, { name: 'threshold error', count: 3, gate: 0.10, nightly: 0.30, manual: 0.12, warn: 0.35 }, { name: 'concurrency race', count: 3, gate: 0.05, nightly: 0.10, manual: 0.00, warn: 0.05 }, ]; export const CHANGES = 320, INTERVAL = 30; // PD7: twenty workdays, one merge every 30 minutes export const DAY_MIN = 480, WEEK_MIN = 2400; // Every change's full set of draws is generated in advance: modes are compared on the same numbers. export const generateFlow = (seed) => { const r = generator(seed); return Array.from({ length: CHANGES }, (_, i) => { let x = r() * 100, cls = null; for (const s of classes) { x -= s.count; if (x <= 0) { cls = s; break; } } return { t: i * INTERVAL, cls, gateRoll: r(), manualRoll: r(), warnRoll: r(), falseRoll: r(), nightlyRoll: Array.from({ length: 8 }, () => r()) }; }); }; ``` #### Six Modes The six modes being compared split two axes: the release window (waiting) and review (human). The third row is the key to this split — there is a daily window but no review, so the wait's own gain is measured apart from the human's. ```js // mode.mjs — the same pipeline in six modes: every green run in production (deployment) or production-ready (delivery) import { generateFlow, CHANGES, DAY_MIN, WEEK_MIN } from './flow.mjs'; const PIPELINE_MIN = 26; // PD8: the pipeline's own duration const ATTENTION = 6; // PD9: fully-seen batch size const REVIEW_BASE = 15, REVIEW_UNIT = 4, REVIEW_PEOPLE = 2; // PD9: review time and deciding people const FALSE_REJECT = 0.08, FALSE_WARN = 0.06; // PD10: false-stop probabilities const DIAGNOSIS_BASE = 30, DIAGNOSIS_UNIT = 12; // PD11: diagnosis time for a production defect const modes = [ { name: 'deployment', window: 0, review: false }, { name: 'delivery, every run', window: 0, review: true }, { name: 'daily window, no review', window: DAY_MIN, review: false }, { name: 'delivery, daily review', window: DAY_MIN, review: true }, { name: 'delivery, weekly review', window: WEEK_MIN, review: true }, { name: 'deployment, warning hard gate', window: 0, review: false, hard: true }, ]; const measure = (mode, flow) => { const batch = new Map(); let hardHeld = 0, falseRejects = 0; for (const c of flow) { if (c.cls && c.gateRoll < c.cls.gate) continue; // automated gate turned red let extra = 0; if (mode.hard) { if (c.cls && c.warnRoll < c.cls.warn) { hardHeld++; continue; } if (!c.cls && c.falseRoll < FALSE_WARN) { falseRejects++; extra = PIPELINE_MIN; } } const ready = c.t + PIPELINE_MIN + extra; const release = mode.window === 0 ? ready : Math.ceil(ready / mode.window) * mode.window; (batch.get(release) ?? batch.set(release, []).get(release)).push({ c, ready }); } let shipped = 0, decisions = 0, wait = 0, worst = 0, held = 0, alreadyWarned = 0; let reviewMin = 0, escaped = 0, diagnosisMin = 0, batchTotal = 0, releaseCount = 0; for (const [release, records] of [...batch.entries()].sort((a, b) => a[0] - b[0])) { const n = records.length; releaseCount++; batchTotal += n; const attention = Math.min(1, ATTENTION / n); // PD9: dilutes as the batch grows let rejected = false; if (mode.review) { decisions += REVIEW_PEOPLE; reviewMin += (REVIEW_BASE + REVIEW_UNIT * n) * REVIEW_PEOPLE; } for (const { c, ready } of records) { // PD12: how many nightly runs fit within the wait const nightlyRuns = Math.floor(release / DAY_MIN) - Math.floor(ready / DAY_MIN); let stopped = false; if (c.cls) { for (let k = 0; k < Math.min(nightlyRuns, 8); k++) { if (c.nightlyRoll[k] < c.cls.nightly) { stopped = true; break; } } if (!stopped && mode.review && c.manualRoll < c.cls.manual * attention) { stopped = true; rejected = true; held++; if (c.warnRoll < c.cls.warn) alreadyWarned++; // automated gate had already warned } } if (stopped) continue; if (c.cls) { escaped++; diagnosisMin += DIAGNOSIS_BASE + DIAGNOSIS_UNIT * (n - 1); } shipped++; wait += release - ready; worst = Math.max(worst, release - ready); } if (mode.review && !rejected && records[0].c.falseRoll < FALSE_REJECT) falseRejects += n; } return { releaseCount, batch: batchTotal / releaseCount, shipped, decisions, wait: wait / shipped, worst, held, alreadyWarned, falseRejects, reviewMin, escaped, diagnosisMin, hardHeld }; }; const flow = generateFlow(51301); const results = modes.map((m) => ({ name: m.name, r: measure(m, flow) })); const find = (name) => results.find((x) => x.name === name).r; const s = (x, w) => String(x).padStart(w); const f = (x, w) => String(x.toFixed(1)).padStart(w); console.log(`changes ${CHANGES}, defect-bearing ${flow.filter((c) => c.cls).length}\n`); console.log(`${'mode'.padEnd(32)}${s('release', 9)}${s('avg batch', 11)}${s('shipped', 15)}${s('human decisions', 17)}${s('avg wait', 13)}${s('worst', 9)}`); for (const { name, r } of results) { console.log(`${name.padEnd(32)}${s(r.releaseCount, 9)}${f(r.batch, 11)}${s(r.shipped, 15)}` + `${s(r.decisions, 17)}${f(r.wait, 13)}${s(r.worst, 9)}`); } console.log(`\n${'mode'.padEnd(32)}${s('review held', 13)}${s('already warned', 16)}${s('escaped', 15)}${s('false rejected', 16)}${s('human min', 11)}${s('diag min', 11)}`); for (const { name, r } of results) { console.log(`${name.padEnd(32)}${s(r.held, 13)}${s(r.alreadyWarned, 16)}${s(r.escaped, 15)}` + `${s(r.falseRejects, 16)}${s(r.reviewMin, 11)}${s(r.diagnosisMin, 11)}`); } const hard = find('deployment, warning hard gate'); const summary = (seed) => { const a = generateFlow(seed), h = (name) => measure(modes.find((m) => m.name === name), a); return { single: h('deployment').escaped - h('delivery, every run').escaped, day: h('daily window, no review').escaped - h('delivery, daily review').escaped, hard: h('deployment, warning hard gate').escaped, dep: h('deployment').escaped }; }; console.log(`\nreview's marginal: no window ${find('deployment').escaped - find('delivery, every run').escaped} defects, ` + `daily window ${find('daily window, no review').escaped - find('delivery, daily review').escaped} defects`); console.log(`daily window's no-review gain ${find('deployment').escaped - find('daily window, no review').escaped} defects, human minutes 0`); console.log(`hard gate stopped ${hard.hardHeld} changes, ${hard.falseRejects} of them were for nothing; escaped to production ${hard.escaped}`); console.log('seed scan (review marginal single/daily, hard gate escaped, no-review deployment escaped):'); for (const seed of [51301, 8802, 33107, 6449]) { const o = summary(seed); console.log(` ${seed}: ${o.single}/${o.day}, ${o.hard}, ${o.dep}`); } ``` ``` changes 320, defect-bearing 116 mode release avg batch shipped human decisions avg wait worst deployment 264 1.0 264 0 0.0 0 delivery, every run 264 1.0 242 528 0.0 0 daily window, no review 20 13.2 247 0 224.8 454 delivery, daily review 20 13.2 242 40 226.5 454 delivery, weekly review 4 66.0 239 8 1152.0 2374 deployment, warning hard gate 236 1.0 236 0 0.0 0 mode review held already warned escaped false rejected human min diag min deployment 0 0 60 0 0 1800 delivery, every run 22 14 38 16 10032 1140 daily window, no review 0 0 43 0 0 7890 delivery, daily review 5 3 38 26 2712 6972 delivery, weekly review 2 2 35 0 2232 28662 deployment, warning hard gate 0 0 32 3 0 960 review's marginal: no window 22 defects, daily window 5 defects daily window's no-review gain 17 defects, human minutes 0 hard gate stopped 28 changes, 3 of them were for nothing; escaped to production 32 seed scan (review marginal single/daily, hard gate escaped, no-review deployment escaped): 51301: 22/5, 32, 60 8802: 20/6, 45, 60 33107: 14/6, 50, 64 6449: 21/6, 48, 69 ``` #### Who Gains From the Wait The first table gives the flow difference between the two modes. In deployment mode, twenty workdays produce 264 releases, 264 changes reach production, and there is no human decision at all. In weekly-reviewed delivery, the release count is four, the batch averages 66 changes, human decisions number eight, and a change that is ready but not yet released waits an average of 1,152 model minutes, 2,374 in the worst case — roughly five workdays. Same pipeline, same output, same tests; the difference is only in the gate at the end. At first glance, the second table's `escaped` column justifies waiting: 60 in deployment, 35 in weekly. The critical row is the third one. The **daily window, no review** mode brings escaped defects down from 60 to 43, and it does this with zero human minutes. A human did not find these 17 defects; the nightly runs that fit inside the wait time did. Most of the improvement credited to manual approval's account is actually the gain from **the extra time granted to the machine** — and that time can be granted without a human too. The wait's bill, meanwhile, is in the `diag min` column. In deployment, diagnosing 60 defects takes 1,800 minutes, 30 minutes per defect: a release has a single change, and the suspect is obvious. In weekly mode, diagnosing 35 defects takes 28,662 minutes, 819 minutes per defect — twenty-seven times as much. As the batch grows, every failure in production has to be picked out from among 66 candidates. Waiting shrinks the number of defects that escape, and grows the cost of the ones that do. #### Manual Approval's Marginal Value The approval gate's marginal value is the difference in escaped defects between the reviewed and unreviewed run on the same draws. When the batch is a single change, this difference is 22 defects — approval is a serious gate. In the daily window, the same difference drops to 5 defects. The gate itself has not changed; what changed is the batch it is examining. The approval gate **generates its own wait**, the wait grows the batch, and the batch dilutes attention: for a batch of 13.2 changes, attention is 6/13.2, that is, 0.45. By its own existence, the gate cuts its own effect by 77 percent. In the weekly window, at a batch of 66, the marginal is 2 defects. The second number is even sharper. The `already warned` column counts, out of the defects approval holds, how many the automated gates had already produced a warning for: 14 of 22 when the batch is one, 3 of 5 in the daily window. For these defects, the human just read a line the machine had already written. Had the warning been turned into a hard gate, these would have been stopped without approval too. Approval's **contribution that belongs only to itself** is 8 and 2 defects in the two cases. The last row carries this thought to its end. In the deployment mode where warnings are turned into a hard gate, 28 changes are stopped, three of them for nothing; escaped defects drop to 32. This beats every mode that includes waiting or review: zero human minutes, zero wait, 30 minutes of diagnosis per defect. The seed scan shows the numbers move — the hard gate's escaped count ranges from 32 to 50, unreviewed deployment's from 60 to 69 — but the ranking is the same across all four seeds. The manual approval gate's three numbers can now be written. **What it holds:** 5 defects in the daily window, 2 of them held only by itself. **What it misses and falsely holds:** 38 defects go to production, 26 clean changes are stopped for nothing. **The time it adds:** 2,712 human minutes and 226.5 minutes of wait per change. Once these three lines are written side by side, the gate becomes indefensible; the sentence "let someone approve it" carries no number on its own. #### Summary - The only structural difference between continuous delivery and continuous deployment is the manual approval gate standing between a green run and production; the measurement was made on this gate. - Manual approval's marginal value depends on the batch: 22 defects when the batch is one change, 5 in the daily window, 2 in the weekly window. The gate generates its own wait and cuts its own effect by 77 percent. - For most of the defects approval holds, the automated gates had already produced a warning (14 of 22, 3 of 5); approval's unique contribution is 8 and 2 defects. - The gain from waiting belongs to the machine, not the human: the unreviewed daily window brought escaped defects down from 60 to 43, at zero human minutes. - Waiting grows the cost of what escapes: diagnosis per defect rose from 30 minutes to 819 minutes, because a batch of 66 has 66 suspects. - Unreviewed deployment with the warning turned into a hard gate brought escaped defects down to 32; with zero human minutes and zero wait, it beat every reviewed mode. #### Next Step The gate this lesson measured sat at the end of the pipeline, and everything before it was referred to as a single box, under the name "automated gates." Inside that box there are several gates: build, test, packaging, and release. Since the marginal value of the approval at the end could be calculated, the marginal value of every gate inside it can be calculated too — which defect **only** that gate holds, which one it misses, how many clean changes it stops for nothing, and even how many minutes it adds. The next lesson runs a known set of defects through a pipeline model, tracks which gate catches each defect, and tries removing one of the gates. ### Pipeline Stages Source: https://academia.sh/en/courses/ci-cd/pipeline-stages Measuring the build, test, packaging, and release gates with a known defect set: what each gate holds, what it alone holds, what it passes, and what it stops for nothing are counted, the time it adds is written in runner minutes, and each of the eight gates is removed one at a time to find the gate whose removal is better. The previous lesson measured the manual approval gate at the end of the pipeline and referred to everything before it as a single box, under the name "automated gates." Inside that box there are several gates, and each one has its own account. This lesson opens the box: an eight-gate pipeline is built, a known defect set is run through it, and which gate catches each defect is tracked. The question asked is the same for every gate. How many defects does it hold? Of those, how many does it hold **alone** — that is, if the gate were removed, how many would escape to production? How many defects does it pass, and how many clean changes does it stop for nothing? And how many minutes does it add? The fourth question can make the third worthless: a gate that holds a lot adds nothing if a later gate holds all the same defects anyway. At the end of the lesson each of the eight gates is removed one at a time, and the **gate whose removal is better** is shown by the numbers. #### The Pipeline and the Defect Set The model is the pipeline of the regional measurement network: the meter reading collector, verifier, billing, and work-order services; the development, staging, and production environments. **Both the network and the pipeline are fictional; the defect set below is a model.** Durations are model minutes, independent of the machine the measurement runs on. **PD13 — the pipeline is eight gates, in a fixed order:** format checking, build, unit tests, contract tests, integration tests, packaging, a smoke test in the staging environment, and a release-health window in production. Fail-fast is on: gates after a red gate do not run. **PD14 — the known defect set is fourteen classes.** For every class, the appearance count per hundred changes, the model minutes it costs if it escapes to production, and the **catch probability per gate** are written down. A gate not listed for a class never sees it. A single change can carry more than one defect. **PD15 — a defect caught after a red gate gets fixed, and the change runs again from the start**, up to six attempts. The fix itself does not give birth to a new defect. **PD16 — every gate has a false-hold probability**; a falsely held change reruns and spends 35 model minutes of human time. **PD17 — the total cost is the sum of three items:** runner minutes, false holds × 35, and the class cost of defects that escape to production. The gate decision is made on this total. **PD18 — 400 changes are run; every draw is generated in advance**, so that when a gate is removed, the remaining gates see the same numbers and the comparison isolates the gate. ```js // pipeline.mjs — eight-gate pipeline model and a known defect set const generator = (seed) => () => { seed = (seed * 1664525 + 1013904223) >>> 0; return seed / 4294967296; }; // PD13: gate order, model minutes per gate, false-hold probability per gate. export const gates = [ { name: 'format', min: 1, falseRate: 0.005 }, { name: 'build', min: 4, falseRate: 0.004 }, { name: 'unit', min: 3, falseRate: 0.012 }, { name: 'contract', min: 11, falseRate: 0.040 }, { name: 'integration', min: 14, falseRate: 0.030 }, { name: 'packaging', min: 6, falseRate: 0.008 }, { name: 'smoke', min: 9, falseRate: 0.045 }, { name: 'health', min: 20, falseRate: 0.035 }, ].map((g, gi) => ({ ...g, gi })); // gi: the gate's fixed position in the full pipeline // PD14: known defect set. count = appearances per hundred changes, cost = model minutes if it // escapes to production, seen = catch probability per gate (an unlisted gate never sees it). export const classes = [ { name: 'format violation', count: 14, cost: 40, seen: { format: 1.0 } }, { name: 'type mismatch', count: 8, cost: 300, seen: { build: 0.95, unit: 0.40 } }, { name: 'missing dependency', count: 5, cost: 500, seen: { build: 0.80, packaging: 0.90 } }, { name: 'boundary condition', count: 7, cost: 900, seen: { unit: 0.85, integration: 0.25 } }, { name: 'null handling', count: 6, cost: 800, seen: { unit: 0.60, integration: 0.35, smoke: 0.40 } }, { name: 'field rename', count: 4, cost: 1600, seen: { contract: 0.90, integration: 0.85 } }, { name: 'request format drift', count: 3, cost: 1400, seen: { contract: 0.85, integration: 0.75, smoke: 0.35 } }, { name: 'schema mismatch', count: 4, cost: 1800, seen: { integration: 0.80, smoke: 0.50 } }, { name: 'migration ordering', count: 2, cost: 2600, seen: { integration: 0.35, smoke: 0.55, health: 0.40 } }, { name: 'missing configuration', count: 5, cost: 1200, seen: { packaging: 0.45, smoke: 0.80 } }, { name: 'package integrity', count: 2, cost: 1000, seen: { packaging: 0.90 } }, { name: 'latency regression', count: 3, cost: 1500, seen: { smoke: 0.15, health: 0.70 } }, { name: 'memory leak', count: 2, cost: 2300, seen: { health: 0.55 } }, { name: 'business rule error', count: 3, cost: 2000, seen: { unit: 0.20, integration: 0.15, smoke: 0.10 } }, ]; export const CHANGES = 400, ATTEMPTS = 6; // PD15: up to six attempts export const FALSE_MIN = 35; // PD16: human cost of a falsely held change // PD18: every draw is generated in advance; removing a gate reuses the same numbers. export const generateFlow = (seed) => { const r = generator(seed); return Array.from({ length: CHANGES }, () => ({ defects: classes.filter((s) => r() < s.count / 100) .map((s) => ({ s, roll: Array.from({ length: ATTEMPTS * gates.length }, () => r()) })), falseRoll: Array.from({ length: ATTEMPTS * gates.length }, () => r()), })); }; ``` #### Four Numbers per Gate The measurement runs twice: first the full pipeline, then eight times with one gate missing each. The `marginal` column is read directly from the removal trial — how much the defects escaping to production increase when the gate is taken out. ```js // gate.mjs — which gate catches each defect, three numbers per gate, and the removal trial import { gates, classes, generateFlow, CHANGES, ATTEMPTS, FALSE_MIN } from './pipeline.mjs'; const runOnce = (lineup, flow) => { const tally = Object.fromEntries(lineup.map((g) => [g.name, { held: 0, passed: 0, falseHeld: 0, min: 0 }])); let escaped = 0, escapedCost = 0, runs = 0; for (const c of flow) { let remaining = c.defects.slice(); for (let attempt = 0; attempt < ATTEMPTS; attempt++) { runs++; let stopper = null; for (const g of lineup) { const i = attempt * gates.length + g.gi; // the roll's index is tied to the gate's fixed position tally[g.name].min += g.min; if (c.falseRoll[i] < g.falseRate) { // PD16: false hold tally[g.name].falseHeld++; stopper = { g, real: false }; break; } const caught = remaining.filter((x) => x.roll[i] < (x.s.seen[g.name] ?? 0)); tally[g.name].passed += remaining.length - caught.length; if (caught.length > 0) { tally[g.name].held += caught.length; remaining = remaining.filter((x) => !caught.includes(x)); // PD15: fixed, reruns stopper = { g, real: true }; break; } } if (stopper === null) break; // green: what remains ships to production } escaped += remaining.length; escapedCost += remaining.reduce((a, x) => a + x.s.cost, 0); } const min = lineup.reduce((a, g) => a + tally[g.name].min, 0); const falseHeld = lineup.reduce((a, g) => a + tally[g.name].falseHeld, 0); return { tally, escaped, escapedCost, runs, min, falseHeld, total: min + falseHeld * FALSE_MIN + escapedCost }; }; const flow = generateFlow(88123); const full = runOnce(gates, flow); const defectTotal = flow.reduce((a, c) => a + c.defects.length, 0); const s = (x, w) => String(x).padStart(w); console.log(`changes ${CHANGES}, defects ${defectTotal}, defect classes ${classes.length}, gates ${gates.length}`); console.log(`full pipeline: runs ${full.runs}, runner ${full.min} min, false holds ${full.falseHeld}, ` + `escaped to production ${full.escaped} defects / ${full.escapedCost} min, total ${full.total} min\n`); console.log(`${'gate'.padEnd(13)}${s('min', 5)}${s('held', 7)}${s('marginal', 10)}${s('passed', 9)}${s('false', 8)}${s('added min', 13)}`); const removed = {}; for (const g of gates) { const y = runOnce(gates.filter((x) => x.name !== g.name), flow); removed[g.name] = y; console.log(`${g.name.padEnd(13)}${s(g.min, 5)}${s(full.tally[g.name].held, 7)}${s(y.escaped - full.escaped, 10)}` + `${s(full.tally[g.name].passed, 9)}${s(full.tally[g.name].falseHeld, 8)}${s(full.tally[g.name].min, 13)}`); } console.log(`\n${'lineup'.padEnd(18)}${s('runner min', 15)}${s('false', 8)}${s('escaped', 9)}${s('escaped min', 13)}${s('total min', 11)}${s('diff', 8)}`); console.log(`${'full pipeline'.padEnd(18)}${s(full.min, 15)}${s(full.falseHeld, 8)}${s(full.escaped, 9)}${s(full.escapedCost, 13)}${s(full.total, 11)}${s('-', 8)}`); for (const g of gates) { const y = removed[g.name]; console.log(`${('no ' + g.name).padEnd(18)}${s(y.min, 15)}${s(y.falseHeld, 8)}${s(y.escaped, 9)}${s(y.escapedCost, 13)}` + `${s(y.total, 11)}${s(y.total - full.total, 8)}`); } const removalHelps = gates.filter((g) => removed[g.name].total < full.total); console.log(`\ngate whose removal lowers the total: ${removalHelps.length ? removalHelps.map((g) => g.name).join(', ') : 'none'}`); console.log('seed scan (gate whose removal lowers the total):'); for (const seed of [88123, 4402, 71905, 3311]) { const a = generateFlow(seed), t0 = runOnce(gates, a); const d = gates.filter((g) => runOnce(gates.filter((x) => x.name !== g.name), a).total < t0.total); console.log(` ${seed}: ${d.length ? d.map((g) => g.name).join(', ') : 'none'} (escaped ${t0.escaped})`); } ``` ``` changes 400, defects 256, defect classes 14, gates 8 full pipeline: runs 713, runner 34140 min, false holds 89, escaped to production 27 defects / 39600 min, total 76855 min gate min held marginal passed false added min format 1 51 51 284 7 713 build 4 46 8 213 3 2620 unit 3 42 27 150 7 1818 contract 11 18 1 119 17 6138 integration 14 25 14 85 15 7336 packaging 6 19 13 63 1 2910 smoke 9 17 15 40 28 4185 health 20 11 12 28 11 8420 lineup runner min false escaped escaped min total min diff full pipeline 34140 89 27 39600 76855 - no format 33688 87 78 40440 77173 318 no build 32422 87 35 43700 79167 2312 no unit 32350 76 54 67500 102510 25655 no contract 28180 74 28 40800 71570 -5285 no integration 26571 75 41 62100 91296 14441 no packaging 30906 90 40 51100 85156 8301 no smoke 28216 51 42 59800 89801 12946 no health 24520 76 39 60400 87580 10725 gate whose removal lowers the total: contract seed scan (gate whose removal lowers the total): 88123: contract (escaped 27) 4402: contract (escaped 16) 71905: format, contract (escaped 27) 3311: none (escaped 25) ``` #### Holding versus Holding Alone The `held` and `marginal` columns do not track each other. The build gate holds 46 defects, but when it is removed, defects escaping to production increase by only 8: most of what it holds is also seen by the unit and packaging gates. The contract gate holds 18, and its marginal is 1. The format gate, by contrast, holds 51, and all 51 are marginal — no other gate in the pipeline sees format violations. The ranking reverses with a third number: the marginal defect's **cost**. The format violation's class cost is 40 minutes, the memory leak's is 2,300, the migration ordering's is 2,600. When the format gate is removed, the escape cost rises by 840 minutes; when the health gate is removed, it rises by 20,800 minutes — 1,733 minutes per defect for twelve defects. This is why **the gate that holds the most is not the most valuable gate**: removing the format gate raises the total by 318 minutes, removing the health gate by 10,725. A gate's value is not in the number of defects it holds, but in the cost of the defects only it holds. The `passed` column is a natural consequence of order, and it is not, on its own, an accusation. The format gate passes 284 defects because it stands first and sees every defect in the pipeline. The health gate passes 28 because few defects even reach it. This column only carries meaning when it is read together with the classes the gate is **expected to see**. The `added min` column depends on how many runs the gate's duration is multiplied by, and fail-fast pulls that multiplier down. Dividing the column by the gate's duration gives the run count per gate: format 713, build 655, unit 606, contract 558, integration 524, packaging 485, smoke 465, health 421. The run count drops moving down the pipeline, because a change that stops at an earlier gate never reaches the later ones. The health gate is the most expensive per run (20 minutes) but runs the least; still, a quarter of the pipeline's total runner load belongs to it. The false-hold column is an independent axis. The smoke test gate stops a clean change for nothing 28 times, the highest count in the pipeline; yet its marginal is 15 defects, and removing it raises the total by 12,946 minutes. A gate with high flakiness stays in place if its marginal is enough — the fix is not to remove it, but to lower its flakiness. #### The Gate Whose Removal Is Better The second table ranks the eight removal trials by total cost. For seven gates, the `diff` column is positive: removal comes at a cost. The unit gate is the most expensive loss, 25,655 minutes — because it is the gate that actually sees the boundary condition and null handling classes, and it costs 3 minutes. The best trade-off per gate sits at the start of the pipeline. One gate stands apart. **Removing the contract gate lowers the total by 5,285 minutes.** The gate's own account explains why: it holds 18 defects, but when it is removed the integration gate catches 16 of them and two escape to production; removal also shifts the flow of attempts, so another defect starts being caught at the smoke gate and the marginal stays at 1 defect. In exchange it adds 6,138 runner minutes and stops 17 clean changes for nothing. When it is removed, defects escaping to production rise from 27 to 28 — **5,285 minutes for one defect**. This gate is not badly built; its place in the pipeline is wrong. If it sees a field rename with probability 0.90 and integration sees the same with 0.85, the gain from holding both is measurably small. The seed scan shows how robust the result is: the contract gate comes up in three of four seeds, in one seed no gate comes up, and in one the format gate also enters the list. So the decision is not "the contract test is unnecessary"; **for this defect set, this pipeline order, and these gate durations**, the contract gate's marginal is under its cost. If the defect set changes — for instance, if contracts between services break more often — the same calculation brings the gate back. What carries the decision is not the gate's name, but the four numbers in the table. #### Summary - The eight-gate pipeline was run with 400 changes and 256 defects: 713 runs, 34,140 runner minutes, 89 false holds, 27 defects escaping to production, and a 39,600-minute escape cost. - What a gate holds and what it alone holds are different: build holds 46, marginal 8; contract holds 18, marginal 1; format holds 51, and all 51 are marginal. - The gate that holds the most is not the most valuable gate: removing the format gate raises the total by 318 minutes, removing the health gate by 10,725. - A false hold alone does not disqualify a gate: the smoke test gate stops 28 clean changes for nothing, but stays in place because its marginal is 15 defects. - Removing the contract gate lowered the total by 5,285 minutes; 6,138 runner minutes and 17 false holds were being paid for one defect. - The result depends on the gate's four numbers, not its name: three of four seeds pointed to the same gate, one pointed to none. #### Next Step Across its 713 runs the pipeline produced something, and this lesson never counted it: the **build artifact** the packaging gate outputs. At the end of every run that finishes green, there is a package, and that package is what goes to production. Two questions remain unanswered. What does this package's identity depend on — which commit, which branch, which run number — and can it be traced back from that identity to the source that produced it? Second, when the same package causes a failure in production, rolling back to the previous version requires that version to still be there; the stricter the retention rule, the narrower the rollback window, and the looser it is, the bigger the repository grows. The next lesson measures these two numbers together. ### Build Artifact Management Source: https://academia.sh/en/courses/ci-cd/build-artifact-management The version identity and retention rule of the pipeline's output: the uniqueness of six identity schemes and their rate of resolving back to source are measured, and seven retention rules are scanned to count the rollback window, unbroken depth, and wastefully retained output together. The previous lesson measured eight gates and never counted what the 713 runs produced: the **build artifact** the packaging gate outputs. At the end of every run that finishes green, there is a package, and that package is what goes to production. This package has two questions, and both of them are gate questions. The first is **identity**: what is the package named after, and can that name be traced back to the source that produced the package? The second is **retention**: how long does the package stay, and when a failure occurs in production, is the version to roll back to still there? The two are linked: a deleted output can only be rebuilt if its identity resolves back to the source. The build artifact repository's growth in bytes was measured in the Fundamental Concepts topic, and the resolution of the movable tag in the Images topic; those measurements are not repeated here, and repository size is counted as the **number of retained outputs**. #### Output History The model is the history of outputs the regional measurement network's pipeline produces over twenty workdays. **Both the network and the history are fictional.** Durations are model minutes. **PD19 — 400 build artifacts are produced in twenty workdays, one every 24 minutes.** 70 percent of runs are from the main branch, the rest from feature branches; 85 percent of feature branches are deleted once the work is done. 6 percent of runs are a rerun of the same commit. Every fifteenth run carries a version tag. **PD20 — six identity schemes are compared.** What information a scheme carries is written with a single field: does the identity text contain the commit hash. **PD21 — resolving from an identity back to the source requires three conditions together:** the identity must be unique, it must carry the commit or a pipeline log must still exist (0.92 probability), and the commit must be accessible in the repository. A deleted branch's commit is not accessible. **PD22 — 40 rollback events occur in twenty workdays.** How many production outputs back the target is drawn with these weights: 1 step back 50, 2 steps 18, 3 steps 10, 5 steps 7, 10 steps 6, 25 steps 5, 60 steps 4. **PD23 — rolling back from a retained output takes 6 model minutes**; if the output has been deleted, rebuilding from source takes 55 minutes and is possible only if the identity resolves to the source. If it does not resolve, the request cannot be met. **PD24 — repository size is measured by the number of retained outputs**, not in bytes. ```js // history.mjs — fictional history of build artifacts the pipeline produces over twenty workdays const generator = (seed) => () => { seed = (seed * 1664525 + 1013904223) >>> 0; return seed / 4294967296; }; export const OUTPUTS = 400, INTERVAL = 24; // PD19: one output every 24 minutes, a 9600-minute window export const LOG = 0.92; // PD21: probability the pipeline log still exists const hex = (r, n) => Array.from({ length: n }, () => '0123456789abcdef'[Math.floor(r() * 16)]).join(''); export const generateHistory = (seed) => { const r = generator(seed); const branches = new Map(); const list = []; let commit = hex(r, 12), version = [1, 4, 0]; for (let i = 0; i < OUTPUTS; i++) { const rerun = r() < 0.06; // PD19: a rerun of the same commit if (!rerun) commit = hex(r, 12); const mainBranch = r() < 0.70; const branch = mainBranch ? 'main' : `feature-${Math.floor(r() * 40)}`; if (!branches.has(branch)) branches.set(branch, { deleted: mainBranch ? false : r() < 0.85 }); if (i % 15 === 0 && i > 0) version = [version[0], version[1], version[2] + 1]; list.push({ number: i + 1, commit, branch, mainBranch, t: i * INTERVAL, version: version.join('.'), tagged: i % 15 === 0, accessible: !branches.get(branch).deleted, // a deleted branch's commit does not stay in the repository logged: r() < LOG }); } return list; }; // PD20: identity schemes. commit = does the identity carry the commit, make = the identity text. export const schemes = [ { name: 'movable tag', commit: false, make: () => 'latest' }, { name: 'run number', commit: false, make: (c) => `${c.number}` }, { name: 'semantic version', commit: false, make: (c) => c.version }, { name: 'branch + run number', commit: false, make: (c) => `${c.branch}-${c.number}` }, { name: 'semantic version + commit', commit: true, make: (c) => `${c.version}+${c.commit.slice(0, 7)}` }, { name: 'commit + run number', commit: true, make: (c) => `${c.commit.slice(0, 7)}-${c.number}` }, ]; // PD21: resolving back to the source requires the identity to be unique, to carry the commit or // have a pipeline log, and for the commit to be accessible in the repository. export const resolve = (scheme, history) => { const counter = new Map(); for (const c of history) counter.set(scheme.make(c), (counter.get(scheme.make(c)) ?? 0) + 1); let unique = 0, source = 0, sourceMain = 0, main = 0; const resolved = new Set(); for (const c of history) { const single = counter.get(scheme.make(c)) === 1; if (single) unique++; const resolves = single && (scheme.commit || c.logged) && c.accessible; if (resolves) { source++; resolved.add(c.number); } if (c.mainBranch) { main++; if (resolves) sourceMain++; } } return { unique, source, sourceMain, main, resolved, colliding: history.length - unique, length: scheme.make(history[history.length - 1]).length }; }; ``` ```js // identity.mjs — uniqueness per scheme and the rate at which identity resolves back to source import { generateHistory, schemes, resolve, OUTPUTS } from './history.mjs'; const history = generateHistory(30719); const pad = (x, w) => String(x).padStart(w); const pct = (a, b) => `${(100 * a / b).toFixed(1)}`; const stranded = history.filter((c) => !c.accessible).length; console.log(`outputs ${OUTPUTS}, main branch ${history.filter((c) => c.mainBranch).length}, ` + `stranded in a deleted branch ${stranded}, reruns ${OUTPUTS - new Set(history.map((c) => c.commit)).size}`); console.log(`\n${'scheme'.padEnd(28)}${pad('length', 9)}${pad('colliding', 11)}${pad('resolves to source', 20)}${pad('on main', 11)}`); for (const scheme of schemes) { const c = resolve(scheme, history); console.log(`${scheme.name.padEnd(28)}${pad(c.length, 9)}${pad(c.colliding, 11)}` + `${pad(`${c.source} (${pct(c.source, OUTPUTS)}%)`, 20)}${pad(`${pct(c.sourceMain, c.main)}%`, 11)}`); } ``` ``` outputs 400, main branch 274, stranded in a deleted branch 103, reruns 28 scheme length colliding resolves to source on main movable tag 6 400 0 (0.0%) 0.0% run number 3 0 269 (67.3%) 90.9% semantic version 6 400 0 (0.0%) 0.0% branch + run number 8 0 269 (67.3%) 90.9% semantic version + commit 14 50 261 (65.3%) 87.6% commit + run number 11 0 297 (74.3%) 100.0% ``` #### What Identity Carries The first two rows separate out the **uniqueness** condition of identity. The movable tag is rewritten on every run: 400 of the 400 outputs share the same identity, and none of them can resolve back to source. Semantic version is in the same position: a number that increments every fifteenth run cannot tell the fourteen outputs in between apart. The run number is unique, and resolution back to source rises to 67.3 percent. The remaining gap has two sources: since the identity does not carry the commit, the pipeline log it relies on stands only with 0.92 probability, and the commit of the 103 outputs in deleted branches is not accessible in the repository. The second constraint is eliminated in the main-branch row — for the run number, resolution on the main branch is 90.9 percent, meaning the remaining 9.1 points to reach 100 percent come entirely from a lost pipeline log. The last scheme, which carries the commit hash, closes this gap: 100 percent on the main branch. When the identity carries the source within itself, no outside record is needed. The overall rate still sitting at 74.3 percent is a matter of deleted feature branches, and that is not an identity problem. The fifth scheme is a middle ground: when semantic version is combined with the commit hash, 50 outputs collide, because rerunning the same commit produces the same identity. The identity shows the source, but it does not show **which run** produced it; adding the run number solves exactly that. #### The Retention Rule Is a Gate The retention rule decides which output stays available to roll back to, which makes it a gate, and it is measured with the same three numbers. The scan below runs seven rules under two identity schemes. ```js // retention.mjs — rollback window and repository size when the retention rule is scanned import { generateHistory, schemes, resolve, OUTPUTS, INTERVAL } from './history.mjs'; const DAY_MIN = 480, DIRECT_MIN = 6, REBUILD_MIN = 55; // PD23: rollback and rebuild const generator = (seed) => () => { seed = (seed * 1664525 + 1013904223) >>> 0; return seed / 4294967296; }; // PD22: 40 rollback events in twenty workdays; how many production outputs back the target is. const DEPTH = [[1, 50], [2, 18], [3, 10], [5, 7], [10, 6], [25, 5], [60, 4]]; const generateEvents = (seed, releases) => { const r = generator(seed), total = DEPTH.reduce((a, d) => a + d[1], 0); return Array.from({ length: 40 }, () => { const T = 1200 + r() * (OUTPUTS * INTERVAL - 1200); let x = r() * total, depth = 1; for (const [d, w] of DEPTH) { x -= w; if (x <= 0) { depth = d; break; } } const priorReleases = releases.filter((c) => c.t < T); return { T, depth, target: priorReleases[priorReleases.length - depth] ?? null }; }); }; const rules = [ { name: 'last 10 runs', keep: (c, n) => c.number > n - 10 }, { name: 'last 25 runs', keep: (c, n) => c.number > n - 25 }, { name: 'last 50 runs', keep: (c, n) => c.number > n - 50 }, { name: 'last 100 runs', keep: (c, n) => c.number > n - 100 }, { name: 'last 7 days', keep: (c, n, T) => T - c.t <= 7 * DAY_MIN }, { name: 'tagged + last 25', keep: (c, n) => c.tagged || c.number > n - 25 }, { name: 'all', keep: () => true }, ]; const history = generateHistory(30719); const releases = history.filter((c) => c.mainBranch); const events = generateEvents(60417, releases); const now = OUTPUTS * INTERVAL; const measure = (rule, resolved) => { let direct = 0, rebuilt = 0, unmet = 0, min = 0; for (const e of events) { if (e.target === null) { unmet++; continue; } const n = history.filter((c) => c.t < e.T).length; if (rule.keep(e.target, n, e.T)) { direct++; min += DIRECT_MIN; } else if (resolved.has(e.target.number)) { rebuilt++; min += REBUILD_MIN; } else unmet++; } const retained = history.filter((c) => rule.keep(c, OUTPUTS, now)).length; const remainingReleases = releases.filter((c) => rule.keep(c, OUTPUTS, now)); // unbroken depth: how many are retained without a gap, counting back from the latest release let unbroken = 0; for (let i = releases.length - 1; i >= 0 && rule.keep(releases[i], OUTPUTS, now); i--) unbroken++; // wasted: retained output that is not the target of any rollback event const targets = new Set(events.filter((e) => e.target).map((e) => e.target.number)); const wasted = history.filter((c) => rule.keep(c, OUTPUTS, now) && !targets.has(c.number)).length; return { direct, rebuilt, unmet, min, retained, unbroken, wasted, window: remainingReleases.length ? (now - remainingReleases[0].t) / DAY_MIN : 0 }; }; const pad = (x, w) => String(x).padStart(w); console.log(`releases ${releases.length}, rollback events ${events.length}, ` + `deepest target ${Math.max(...events.map((e) => e.depth))} versions back`); console.log(`\n${'retention rule'.padEnd(20)}${pad('retained', 10)}${pad('window days', 13)}${pad('unbroken depth', 21)}${pad('wasted retained', 17)}`); for (const r of rules) { const m = measure(r, new Set()); console.log(`${r.name.padEnd(20)}${pad(m.retained, 10)}${pad(m.window.toFixed(2), 13)}${pad(m.unbroken, 21)}${pad(m.wasted, 17)}`); } for (const name of ['movable tag', 'commit + run number']) { const scheme = schemes.find((x) => x.name === name); const { resolved } = resolve(scheme, history); console.log(`\nidentity scheme: ${name} (resolves to source for ${resolved.size} outputs)`); console.log(`${'retention rule'.padEnd(20)}${pad('direct', 10)}${pad('rebuild', 16)}${pad('unmet', 14)}${pad('recovery min', 14)}`); for (const r of rules) { const m = measure(r, resolved); console.log(`${r.name.padEnd(20)}${pad(m.direct, 10)}${pad(m.rebuilt, 16)}${pad(m.unmet, 14)}${pad(m.min, 14)}`); } } ``` ``` releases 274, rollback events 40, deepest target 60 versions back retention rule retained window days unbroken depth wasted retained last 10 runs 10 0.50 8 10 last 25 runs 25 1.25 17 23 last 50 runs 50 2.50 31 47 last 100 runs 100 5.00 67 89 last 7 days 140 7.00 98 125 tagged + last 25 50 20.00 17 44 all 400 20.00 274 365 identity scheme: movable tag (resolves to source for 0 outputs) retention rule direct rebuild unmet recovery min last 10 runs 35 0 5 210 last 25 runs 37 0 3 222 last 50 runs 38 0 2 228 last 100 runs 40 0 0 240 last 7 days 40 0 0 240 tagged + last 25 37 0 3 222 all 40 0 0 240 identity scheme: commit + run number (resolves to source for 297 outputs) retention rule direct rebuild unmet recovery min last 10 runs 35 5 0 485 last 25 runs 37 3 0 387 last 50 runs 38 2 0 338 last 100 runs 40 0 0 240 last 7 days 40 0 0 240 tagged + last 25 37 3 0 387 all 40 0 0 240 ``` #### The Advertised Window versus the Unbroken Depth The most instructive row of the second table is `tagged + last 25`. This rule retains 50 outputs and advertises a **20-day** window: the oldest retained production output sits at the start of the window. The `last 50 runs` rule, which retains the same 50 outputs, has a window of only 2.5 days. The eight-times-wider window looks better. Rollback events say the opposite: the tagged rule meets 37 requests directly, while `last 50 runs` meets 38. The `unbroken depth` column explains the difference. The tagged rule's window has holes in it; only 17 production outputs are retained without a gap counting backward, against 31 for the `last 50 runs` rule. A rollback request does not say "go back to twenty days ago," it says "go back three versions"; if the version in between has been deleted, the window's length is useless. **A retention rule is defended by unbroken depth, not by window length.** The `wasted retained` column is the gate's false hold. The `all` rule retains 400 outputs, 365 of which are never the target of a rollback event, and it does not meet a single request more than the `last 100 runs` rule: four times the repository for zero marginal gain. The difference between `last 100 runs` and `last 50 runs`, meanwhile, is 2 requests. #### Where Identity and Retention Intersect The last two tables combine the two gates. Under the movable-tag scheme, the `last 10 runs` rule **cannot meet** five requests: the output has been deleted, and since the identity cannot resolve to the source, it cannot be rebuilt either. When the same retention rule runs with an identity that carries the commit hash, unmet requests drop to zero; the five requests are met with a 55-minute rebuild each, and recovery time rises from 210 minutes to 485 minutes. The trade-off is clear. A strict retention rule brings the repository down from 400 to 10 and costs 245 extra minutes of recovery time — **but only if the identity scheme allows it.** If the identity does not resolve, the same strictness loses five requests entirely. The identity scheme sets the upper bound on how strict the retention rule can be. The retention gate's four numbers can now be written. For the `last 50 runs` rule: **what it holds** is 38 rollback requests, **what it alone holds** — when chosen over `last 25` — is 1 request, **what it misses** is 2 requests, **what it falsely holds** is 47 outputs, and **what it adds** is a 50-output repository and 338 minutes of recovery time. #### Summary - Under the movable-tag and semantic-version schemes, all 400 outputs collide and the rate of resolving back to source is zero; the run number is unique and gives 67.3 percent, and the scheme carrying the commit hash gives 74.3 percent. - The main-branch row isolates the constraint: the run number stays at 90.9 percent (a lost pipeline log), while the identity carrying the commit rises to 100 percent. The gap in the overall rate belongs to deleted feature branches, and no identity scheme closes it. - The advertised window is misleading: the `tagged + last 25` rule shows a 20-day window, but its unbroken depth is 17, and with the same repository size it meets one fewer request than the rule that gives a 2.5-day window. - Quadrupling the repository can have zero marginal value: the `all` rule retains 400 outputs, 365 of which are never used, and it meets the same 40 requests as `last 100 runs`. - The identity scheme sets retention's upper bound: the `last 10 runs` rule produces zero unmet requests and 485 minutes with a resolvable identity, and five unmet requests with one that does not resolve. #### Next Step The four lessons so far counted what the pipeline **holds** and always wrote duration as a cost item: the minutes a gate adds, the minutes waiting costs, the minutes a rebuild takes. Duration itself was never the target. Yet the first lesson's table said something — as a run gets cheaper, the best merge interval shifts toward more frequent merging, meaning shortening pipeline time does not just make the wait easier, it also changes the gates' economics. The next lesson looks directly at this duration: what work can be reused between stages, where the limit of running in parallel sits, and past what point adding resources stops shortening the pipeline. ### Caching and Parallelism Source: https://academia.sh/en/courses/ci-cd/caching-and-parallelism The pipeline's own duration is measured: stage caching derives a key from the content hash of real files, a twelve-run sequence counts cached and uncached duration, hit rate, and the hit rate's distribution by change type, and then parallelism is applied to show the limit the critical path sets. The previous lesson versioned and retained the output the pipeline produces: every run's output sits under its own identity, and which change gave birth to it can be read. One thing was never counted — how many minutes that output took to produce. The pipeline runs every stage from scratch on every run, yet most stages' inputs never change between two runs. This lesson measures the pipeline's own duration. Two levers are available: not rerunning a stage that does the same work on the same input — **stage caching** — and running stages that do not depend on each other at the same time — **parallelism**. Both have a limit, and the limits do not sit in the same place. The gate measure is two-edged here: when a stage comes from the cache, that gate **does not run at all**, and what the gate misses is born exactly there. **PD25.** The pipeline is a model set up with `node`. Stage durations are the nominal values the model declares, not measured from the machine the run happens on; the environment-independent quantities are the number of stages that ran, the number of hits, and the cache entries. **PD26.** The stage key is computed from real files: files are written to disk, read, and hashed. **PD27.** The change sequence stands in for the fictional regional measurement network's two-week patch flow; there is no randomness, the sequence is hand-written. **PD28.** The defect set is eight defects, and which gates can catch each one is declared; a gate that runs catches its own defect, a gate that does not run does not. **PD29.** Cache entries are never deleted, and no retention rule is applied. **PD30.** Runners are identical and queue wait is zero — that item is measured in the next lesson. The caching of image layers was measured outside this course and is not repeated here; the unit measured here is the **pipeline stage** — a unit of work with its own input, its own duration, and its own output. #### The Stage Key The cache has a single rule: **a stage's key must cover everything that could change that stage's result.** When an input that is not covered changes, the key stays the same, the stage does not run, and the pipeline mistakes the old result for correct. In the model, every stage **declares** its own inputs. ```js // measurement-network/pipeline.mjs — pipeline stages and stage caching (model) import { mkdirSync, writeFileSync, readFileSync, readdirSync, statSync, appendFileSync, rmSync } from "node:fs"; import { createHash } from "node:crypto"; const ROOT = "project"; const hash = (s) => createHash("sha256").update(s).digest("hex").slice(0, 12); const text = (name, n) => Array.from({ length: n }, (_, i) => `// ${name} ${i}\n`).join(""); const TREE = { "dependency-list": "measurement-client 3.1\nqueue-connector 1.9\ncsv-parser 2.4\n", "src/collector.js": text("collector", 260), "src/verifier.js": text("verifier", 180), "src/billing.js": text("billing", 240), "test/unit/collector.js": text("unit", 300), "test/integration/flow.js": text("integration", 220), "test/contract/api.js": text("contract", 90), "config/env.json": '{"timeout":30}\n', "rules/static.json": '{"rule-count":48}\n', "contracts/api.json": '{"field-count":17}\n', }; // duration: declared nominal stage duration (seconds). base: output written to the cache (bytes). // inputs: what the stage DECLARES as its inputs — rules/ and contracts/ are declared by no stage. export const STAGES = [ { name: "dependency-resolution", inputs: ["dependency-list"], deps: [], duration: 42, base: 41e6 }, { name: "build", inputs: ["src"], deps: ["dependency-resolution"], duration: 65, base: 8.6e6 }, { name: "static-analysis", inputs: ["src"], deps: ["dependency-resolution"], duration: 27, base: 12e4 }, { name: "unit-test", inputs: ["test/unit"], deps: ["build"], duration: 38, base: 9e4 }, { name: "integration-test", inputs: ["test/integration", "config"], deps: ["build"], duration: 96, base: 26e4 }, { name: "packaging", inputs: ["config"], deps: ["build"], duration: 18, base: 9.1e6 }, { name: "contract-test", inputs: ["test/contract"], deps: ["packaging"], duration: 31, base: 6e4 }, { name: "image-build", inputs: [], deps: ["packaging"], duration: 24, base: 12.3e6 }, ]; export const RESTORE = 2; // seconds to restore an output from the cache on every hit export const SERIAL = STAGES.reduce((t, s) => t + s.duration, 0); export const buildTree = () => { rmSync(ROOT, { recursive: true, force: true }); for (const [p, content] of Object.entries(TREE)) { mkdirSync(`${ROOT}/${p}`.split("/").slice(0, -1).join("/"), { recursive: true }); writeFileSync(`${ROOT}/${p}`, content); } }; export const modify = (path, no) => appendFileSync(`${ROOT}/${path}`, `// fix ${no}\n`); const walk = (sub) => readdirSync(`${ROOT}/${sub}`, { withFileTypes: true }) .flatMap((g) => (g.isDirectory() ? walk(`${sub}/${g.name}`) : [`${sub}/${g.name}`])).sort(); const paths = (p) => (statSync(`${ROOT}/${p}`).isDirectory() ? walk(p) : [p]); // Stage key: stage name + the real content of the declared inputs + the previous stages' keys. export const runPipeline = (cache) => { const key = {}, result = { hits: [], ran: [], duration: 0 }; for (const s of STAGES) { const files = s.inputs.flatMap(paths); key[s.name] = hash(s.name + files.map((p) => p + readFileSync(`${ROOT}/${p}`)).join() + s.deps.map((dep) => key[dep]).join()); if (cache.has(key[s.name])) { result.hits.push(s.name); result.duration += RESTORE; continue; } cache.set(key[s.name], s.base + files.reduce((t, p) => t + statSync(`${ROOT}/${p}`).size, 0)); result.ran.push(s.name); result.duration += s.duration; } return result; }; ``` Feeding the prior stages' keys into a stage's own key builds a chain: when a stage misses, **every** stage depending on it misses too, even if its own input has not changed. This is the cache's correct side; its wrong side is in the declaration list — `rules/` and `contracts/` are real inputs to the pipeline, but no stage declares them. #### Twelve Runs The sequence runs the fictional network's patch flow across twelve runs: source and test changes, a dependency list, a config, a run where nothing changes, and two changes touching undeclared directories. Eight defects are scattered through the sequence. ```js // measurement-network/run.mjs — a twelve-run change sequence, cached and uncached import { buildTree, modify, runPipeline, STAGES, SERIAL } from "./pipeline.mjs"; // Fictional two-week sequence: file touched, change category, defect carried, gates that // could catch the defect. No randomness; the sequence is hand-written. const SEQUENCE = [ ["", "first run"], ["src/collector.js", "src", "D1", ["unit-test", "integration-test"]], ["src/verifier.js", "src", "D2", ["static-analysis"]], ["test/unit/collector.js", "test"], ["", "no change"], ["dependency-list", "dependency", "D3", ["integration-test"]], ["rules/static.json", "rule", "D4", ["static-analysis"]], ["src/billing.js", "src", "D5", ["unit-test", "static-analysis"]], ["config/env.json", "config"], ["contracts/api.json", "contract", "D6", ["contract-test"]], ["test/integration/flow.js", "test", "D7", ["integration-test"]], ["src/collector.js", "src", "D8", ["unit-test"]], ]; buildTree(); const cache = new Map(), category = {}, missed = []; const gates = Object.fromEntries(STAGES.map((s) => [s.name, { runs: 0, duration: 0, held: 0, alone: 0 }])); let totalHits = 0, totalDuration = 0; console.log(" no change".padEnd(33) + "category".padEnd(15) + "hits ran dur(s)"); SEQUENCE.forEach(([file, cat, defect, catchableBy], i) => { if (file) modify(file, i); const r = runPipeline(cache); totalHits += r.hits.length; totalDuration += r.duration; category[cat] = category[cat] || { attempts: 0, hits: 0 }; category[cat].attempts += STAGES.length; category[cat].hits += r.hits.length; for (const name of r.ran) { gates[name].runs++; gates[name].duration += STAGES.find((s) => s.name === name).duration; } if (defect) { const caught = catchableBy.filter((name) => r.ran.includes(name)); if (!caught.length) missed.push(defect); else for (const name of caught) { gates[name].held++; if (caught.length === 1) gates[name].alone++; } } console.log(` ${String(i + 1).padStart(2)} ${file || "(" + cat + ")"}`.padEnd(33) + cat.padEnd(15) + String(r.hits.length).padStart(4) + String(r.ran.length).padStart(6) + String(r.duration).padStart(7)); }); const attempts = SEQUENCE.length * STAGES.length, uncached = SEQUENCE.length * SERIAL; console.log(`\nuncached ${uncached} s, cached ${totalDuration} s (${(100 - (totalDuration * 100) / uncached).toFixed(1)}% shorter); ` + `hits ${totalHits}/${attempts} (${((totalHits * 100) / attempts).toFixed(1)}%); ` + `cache ${cache.size} entries ${([...cache.values()].reduce((a, b) => a + b, 0) / 1e6).toFixed(1)} MB`); console.log("\nhits / change category" + " ".repeat(10) + "gate runs dur held alone"); const rows = Object.entries(category).map(([cat, d]) => ` ${cat.padEnd(15)}${d.hits}/${d.attempts} ${((d.hits * 100) / d.attempts).toFixed(0)}%`); STAGES.forEach((s, i) => { const g = gates[s.name]; console.log((rows[i] || "").padEnd(33) + s.name.padEnd(24) + String(g.runs).padStart(3) + String(g.duration + "s").padStart(7) + String(g.held).padStart(7) + String(g.alone).padStart(8)); }); console.log(`\ndefect set 8, missed ${missed.length}: ${missed.join(", ")} — undeclared input, stale hit`); ``` ``` no change category hits ran dur(s) 1 (first run) first run 0 8 341 2 src/collector.js src 1 7 301 3 src/verifier.js src 1 7 301 4 test/unit/collector.js test 7 1 52 5 (no change) no change 8 0 16 6 dependency-list dependency 0 8 341 7 rules/static.json rule 8 0 16 8 src/billing.js src 1 7 301 9 config/env.json config 4 4 177 10 contracts/api.json contract 8 0 16 11 test/integration/flow.js test 7 1 110 12 src/collector.js src 1 7 301 uncached 4092 s, cached 2273 s (44.5% shorter); hits 46/96 (47.9%); cache 50 entries 287.4 MB hits / change category gate runs dur held alone first run 0/8 0% dependency-resolution 2 84s 0 0 src 4/32 13% build 6 390s 0 0 test 14/16 88% static-analysis 6 162s 2 1 no change 8/8 100% unit-test 7 266s 3 1 dependency 0/8 0% integration-test 8 768s 3 2 rule 8/8 100% packaging 7 126s 0 0 config 4/8 50% contract-test 7 217s 0 0 contract 8/8 100% image-build 7 168s 0 0 defect set 8, missed 2: D4, D6 — undeclared input, stale hit ``` #### Reading the Hit Rate Total duration drops **from 4,092 seconds to 2,273 seconds**: a 44.5 percent reduction, 46 hits in 96 stage attempts. This single number carries no decision on its own; the decision comes from the distribution by category. For test changes, the hit rate is **88 percent** — touching one test file only runs that one test stage. For source changes, the hit rate drops to **13 percent**: source sits at the bottom of the chain, and touching it reruns seven stages. For a dependency-list change, the hit rate is **zero**; that file is the input of the very bottom stage and invalidates the entire pipeline. The gain depends on which layer of the pipeline is touched, and how often. Only one of the three rows showing 100 percent is correct. In the run where nothing changes, eight of eight stages hit and the pipeline finishes in 16 seconds; that is exactly what the cache is supposed to do. Rule and contract changes also show eight hits — but something has changed there. `rules/static.json` is the rule set for static analysis, `contracts/api.json` is the contract test's criterion; neither is an input any stage declares. The key does not change, the gate does not run, the pipeline finishes green. **The two change types with the highest hit rate are the ones the cache cannot see.** The gate account writes down the cost of this. Six of eight defects are caught, **two escape**: D4 is a defect that becomes visible when the rule set is tightened, D6 is born from a change in the contract field; both pass through because the gate that could catch them came from the cache. The marginal-value column separates the gates: the integration test spends **768 seconds** across eight runs, holds three defects, and **catches two of them alone** — it is the most expensive and most necessary gate in the pipeline. The unit test holds three defects in 266 seconds, but only one of them alone; of the other two, one would also have been caught by the integration test and the other by static analysis. Four stages — dependency resolution, build, packaging, image build — never hold a defect; they are production steps, not gates, and they do not enter this accounting. If the declaration is fixed, the cost can be read off the table: on the seventh run, static analysis takes 27 seconds instead of a 2-second hit; on the tenth run, the contract test takes 31 instead of 2. The total is **54 seconds**, 2.4 percent of the cached duration — in exchange, two defects do not escape. The cache's own cost has three items. Storage: across twelve runs, **50 entries and 287.4 MB** accumulate, none of them ever deleted. Restore: each of the 46 hits costs 2 seconds, 92 seconds in total. The third item does not reduce to a number, and it is the most expensive — the two defects that escaped above are the bill for the invalidation error. #### The Critical Path Caching removed the repeated work; the remaining work still runs in a single line. The stages form a dependency graph, the ones that do not depend on each other can run at the same time, and the graph's longest chain sets the limit. ```js // measurement-network/parallel.mjs — stage graph, runner count, and the critical path import { STAGES, RESTORE } from "./pipeline.mjs"; const find = (name) => STAGES.find((s) => s.name === name); const successors = (name) => STAGES.filter((b) => b.deps.includes(name)); // Longest remaining duration from a stage to the pipeline's end; the critical path is the chain whose head is largest. const remaining = (s) => s.duration + Math.max(0, ...successors(s.name).map(remaining)); const chain = (s) => (successors(s.name).length ? `${s.name}(${s.duration}) -> ` + chain(successors(s.name).reduce((x, y) => (remaining(x) >= remaining(y) ? x : y))) : `${s.name}(${s.duration})`); // List scheduling: among FINISHED stages, the one whose remaining path is longest gets its // dependents assigned to a free runner first. Runner count is n. const schedule = (duration, n) => { const done = new Set(), running = [], pending = STAGES.map((s) => s.name); let clock = 0; while (done.size < STAGES.length) { const ready = pending.filter((name) => find(name).deps.every((d) => done.has(d))) .sort((x, y) => remaining(find(y)) - remaining(find(x))); while (running.length < n && ready.length) { const name = ready.shift(); pending.splice(pending.indexOf(name), 1); running.push({ name, ends: clock + duration(name) }); } clock = Math.min(...running.map((r) => r.ends)); for (let i = running.length - 1; i >= 0; i--) if (running[i].ends <= clock) { done.add(running[i].name); running.splice(i, 1); } } return clock; }; const start = STAGES.filter((s) => !s.deps.length).reduce((x, y) => (remaining(x) >= remaining(y) ? x : y)); const limit = remaining(start), serial = STAGES.reduce((t, s) => t + s.duration, 0); const cold = (name) => find(name).duration; // On a source change, the cache only hits at dependency-resolution (table above, run 2). const cached = (name) => (name === "dependency-resolution" ? RESTORE : find(name).duration); console.log(`serial ${serial} s, critical path ${limit} s: ${chain(start)}`); console.log("\nrunners cold distance from critical path gain from one more runner cached (src change)"); let prev = null; for (let n = 1; n <= 6; n++) { const s = schedule(cold, n); console.log(String(n).padStart(9) + String(s + " s").padStart(9) + String(s - limit + " s").padStart(18) + (prev === null ? "-" : prev - s + " s").padStart(24) + String(schedule(cached, n) + " s").padStart(28)); prev = s; } ``` ``` serial 341 s, critical path 203 s: dependency-resolution(42) -> build(65) -> integration-test(96) runners cold distance from critical path gain from one more runner cached (src change) 1 341 s 138 s - 301 s 2 218 s 15 s 123 s 178 s 3 203 s 0 s 15 s 163 s 4 203 s 0 s 0 s 163 s 5 203 s 0 s 0 s 163 s 6 203 s 0 s 0 s 163 s ``` The second runner gains **123 seconds**, the third **15 seconds**, the fourth **zero**. Duration flattens at three runners and stops at 203 seconds; this number comes from the graph, not from the runner count. The chain that sets the limit is dependency resolution, build, and the integration test: 42 + 65 + 96. All three must wait on each other, so no runner count can bring the pipeline below 203 seconds. The highest speedup is **1.68 times**, and everything past the fourth runner is paid-for idle capacity. The only way past the limit is to shorten the chain: if the integration test is split into two independent parts, the critical path drops to 42 + 65 + 48. This is a pipeline design decision, not a decision about buying more runners. The two levers multiply each other, but they do not grip the same place. The last column shows the source-change run: the cache brings the 42 seconds at the head of the chain down to 2, and with three runners duration settles at **163 seconds**. Caching shortens the chain's **head**; parallelism removes the work **outside** the chain; neither can touch the 65 and 96 seconds in the middle. That is the pipeline's real floor. #### Summary - The stage key derives from the content of a stage's **declared** inputs and the keys of the prior stages. When an undeclared input changes, the key does not change, the gate does not run, and the pipeline finishes green. - Across twelve runs, caching brought duration down from 4,092 seconds to 2,273 seconds (44.5 percent), and hits reached 46 in 96 attempts. The distribution carries the decision: 88 percent for tests, 13 percent for source, zero for the dependency list. - The 100 percent hit rate shown for rule and contract changes is the cache's blindness. Two defects (D4, D6) escaped because the gate that could catch them came from the cache; fixing the declaration adds 54 seconds to twelve runs, that is, 2.4 percent. - Gates' marginal value is not equal: the integration test held three defects in 768 seconds, two of them alone; the unit test held three defects in 266 seconds but only one of them alone. The cache's own cost is 50 entries, 287.4 MB of storage, and 92 seconds of restoring. - Parallelism cannot bring the pipeline below 203 seconds. The limit is set by the dependency resolution → build → integration test chain; past the third runner, the gain from one more runner is zero. #### Next Step This measurement counted two things as free. Every one of the forty-six hits assumes a machine that can still find the previous run's output; PD30 also took queue wait as zero and assumed three runners were always ready. Both assumptions come down to the same decision: where the pipeline runs. If the pipeline runs on a machine set up clean on every run, the cache is empty every time and the 1,819-second gain measured here evaporates; if it runs on self-hosted machines, the cache stays, but the machines wait, age, and demand maintenance. The next lesson runs the same pipeline on two runner types and counts four items — queue wait, cost per run, the cache's persistence, and maintenance load — then calculates the threshold: past how many runs a day, which type gets cheaper. ### Pipeline Runners Source: https://academia.sh/en/courses/ci-cd/pipeline-runners The same pipeline runs on two runner types: queue wait, the cost line item per run, whether the cache persists between runs, and the maintenance line item are counted, the number of runs a day after which each type gets cheaper is calculated, and the items the measure leaves out are named. The previous lesson brought pipeline duration down from 4,092 seconds to 2,273 seconds and showed the limit stopping at 203 seconds. Two silent assumptions sat under that measurement: the cache stayed in place between runs, and runners were always ready. Neither is free, and both come down to the same decision — **where** the pipeline runs. There are two types to choose from. A **managed runner** is a machine set aside per run and torn down when the work is done; every run starts on a clean disk. A **self-hosted runner** is a machine that stays up, one you own; its disk persists between runs. The difference between them is not a preference, it is four separate columns: queue wait, the item paid per run, the cache's persistence, and maintenance load. **PD31.** Both types are a model set up with `node`; they are referred to as types, and no product or vendor name appears. **PD32.** Run durations are taken from the previous lesson: the managed type is 203 seconds because it starts cold on every run, the self-hosted type is 163 seconds because it keeps its cache. **PD33.** Arrival times are produced by a self-written generator, seed **4831**; the fictional workday forms three peaks, because the fictional measurement network's team sends its commits in the morning, in the afternoon, and at day's close. **PD34.** Costs are in **units**; no currency is named. **PD35.** The measurement covers a single day; machine failure, spare capacity, and rerunning a run that finished failed do not enter the model. **PD36.** The gate account carries forward the previous lesson's eight-defect set and twelve-run sequence. #### Two Types, the Same Pipeline In the model, a type is defined by four numbers: how many concurrent slots it has, how many seconds a run takes, how the run is billed, and how many maintenance items it carries. ```js // runner.mjs — two runner types and a day's queue (model) export const SEED = 4831; export const generator = (t) => () => ((t = (t * 48271) % 2147483647) / 2147483647); // duration: three-slot run duration measured in 05 (seconds). managed starts cold every run // (203 s), self-hosted keeps its cache and runs warm (163 s). hitRate: the rate measured in 05. export const TYPE = { managed: { slots: 8, duration: 203, hitRate: 0, perMinuteCost: 0.42, fixedDaily: 0, lineItems: 1, maintenance: [] }, selfHosted: { slots: 2, duration: 163, hitRate: 0.479, perMinuteCost: 0, fixedDaily: 2 * 11 + (6 * 28) / 30, lineItems: 2, maintenance: ["OS patching", "disk filling up", "runner version", "network and certificates", "capacity/hardware"], }, }; // Fictional workday: runs land in three peaks (10:00, 14:00, 17:00) under a triangular distribution. export const arrivals = (n, rnd) => Array.from({ length: n }, () => { const m = [10, 14, 17][Math.floor(rnd() * 3)] * 3600; return Math.round(m + (rnd() + rnd() - 1) * 1200); // ±20 minutes }).sort((a, b) => a - b); // Slot schedule: every arrival goes to the earliest-free slot; wait = free moment - arrival moment. export const day = (n, type, rnd) => { const free = Array(type.slots).fill(0), wait = []; for (const a of arrivals(n, rnd)) { const i = free.indexOf(Math.min(...free)); const w = Math.max(0, free[i] - a); wait.push(w); free[i] = a + w + type.duration; } return { avgWait: wait.reduce((a, b) => a + b, 0) / n, longest: Math.max(...wait), waiting: wait.filter((w) => w > 0).length, cost: type.fixedDaily + (n * type.duration * type.perMinuteCost) / 60, }; }; ``` The self-hosted type's fixed daily cost sums two items: the daily cost of two machines and the day's share of six hours of monthly maintenance labor. In the managed type, the fixed cost is zero and the only item is run duration. This is the real difference between the two types — one charges per **run**, the other per **time**. Slot count sits on the same axis. In the self-hosted type, a slot is a purchased machine, and the count is capped at two; in the managed type, eight slots is a ceiling the model declares, not a machine. This ceiling itself is counted as a line item — eight concurrent runs are not free, where that number comes from is declared in the model, and it is a matter of negotiation in a real setup. #### One Day's Queue Same pipeline, same arrival times, two types. It runs at six separate intensities: from 10 runs a day to 80. ```js // threshold.mjs — the same pipeline on two runner types: wait, cost, threshold import { TYPE, SEED, generator, day } from "./runner.mjs"; const min = (s) => (s / 60).toFixed(1); console.log(`seed ${SEED}; slots: managed ${TYPE.managed.slots}, self-hosted ${TYPE.selfHosted.slots}`); console.log("\n managed (cold, 203 s) self-hosted (warm, 163 s)"); console.log("runs/day wait longest waiting cost wait longest waiting cost"); for (const n of [10, 20, 30, 40, 60, 80]) { const y = day(n, TYPE.managed, generator(SEED)), b = day(n, TYPE.selfHosted, generator(SEED)); console.log(String(n).padStart(7) + [min(y.avgWait) + " m", min(y.longest) + " m", `${y.waiting}/${n}`, y.cost.toFixed(1), min(b.avgWait) + " m", min(b.longest) + " m", `${b.waiting}/${n}`, b.cost.toFixed(1)] .map((h, i) => h.padStart(i === 4 ? 12 : 9)).join("")); } let threshold = 0; while (day(++threshold, TYPE.managed, generator(SEED)).cost <= day(threshold, TYPE.selfHosted, generator(SEED)).cost); const y = day(threshold, TYPE.managed, generator(SEED)), b = day(threshold, TYPE.selfHosted, generator(SEED)); console.log(`\nthreshold: at ${threshold} runs/day managed rises to ${y.cost.toFixed(1)} units, ` + `self-hosted stays flat at ${b.cost.toFixed(1)} units.`); console.log(`cost line items per run: managed ${TYPE.managed.lineItems} (runner-minutes), ` + `self-hosted ${TYPE.selfHosted.lineItems} (machine + maintenance labor)`); console.log(`maintenance line items: managed ${TYPE.managed.maintenance.length}, self-hosted ` + `${TYPE.selfHosted.maintenance.length} — ${TYPE.selfHosted.maintenance.join(", ")}`); // Gate account: in 05 the cache had missed two defects (D4, D6). A runner that starts clean // runs every gate on every run, so those two defects do not escape. const seq = 12, diff = TYPE.managed.duration - TYPE.selfHosted.duration; console.log(`\ncache hit rate: managed ${TYPE.managed.hitRate * 100}%, self-hosted ${(TYPE.selfHosted.hitRate * 100).toFixed(1)}%`); console.log(`gate account (${seq}-run sequence): managed 8/8 defects, self-hosted 6/8 (D4, D6 stale hit)`); console.log(`cost of the two defects: ${diff} s per run, ${seq * diff} s over the sequence ` + `(${min(seq * diff)} m); at 30 runs/day ${min(30 * diff)} m and ` + `${(day(30, TYPE.managed, generator(SEED)).cost - day(30, TYPE.selfHosted, generator(SEED)).cost).toFixed(1)} unit difference`); // Managed type + remote cache store: the hit rate comes back, transfer time and storage cost are added. const TRANSFER = 24, STORAGE = 3; const remote = { ...TYPE.managed, duration: TYPE.selfHosted.duration + TRANSFER, fixedDaily: STORAGE }; console.log(`\nmanaged type + remote cache store: run ${remote.duration} s, at 30 runs/day ` + `${day(30, remote, generator(SEED)).cost.toFixed(1)} units — but the stale hit comes back, so defects 6/8`); ``` ``` seed 4831; slots: managed 8, self-hosted 2 managed (cold, 203 s) self-hosted (warm, 163 s) runs/day wait longest waiting cost wait longest waiting cost 10 0.0 m 0.0 m 0/10 14.2 0.1 m 0.6 m 1/10 27.6 20 0.0 m 0.0 m 0/20 28.4 0.1 m 1.1 m 2/20 27.6 30 0.0 m 0.0 m 0/30 42.6 1.0 m 5.3 m 12/30 27.6 40 0.0 m 0.0 m 0/40 56.8 1.5 m 8.0 m 21/40 27.6 60 0.0 m 1.0 m 2/60 85.3 4.3 m 15.1 m 44/60 27.6 80 0.1 m 1.0 m 8/80 113.7 6.6 m 20.9 m 63/80 27.6 threshold: at 20 runs/day managed rises to 28.4 units, self-hosted stays flat at 27.6 units. cost line items per run: managed 1 (runner-minutes), self-hosted 2 (machine + maintenance labor) maintenance line items: managed 0, self-hosted 5 — OS patching, disk filling up, runner version, network and certificates, capacity/hardware cache hit rate: managed 0%, self-hosted 47.9% gate account (12-run sequence): managed 8/8 defects, self-hosted 6/8 (D4, D6 stale hit) cost of the two defects: 40 s per run, 480 s over the sequence (8.0 m); at 30 runs/day 20.0 m and 15.0 unit difference managed type + remote cache store: run 187 s, at 30 runs/day 42.3 units — but the stale hit comes back, so defects 6/8 ``` #### Threshold and Queue If the cost column is read alone, the decision looks clear: the threshold is **20 runs a day**. Below it, the managed type is cheaper — 14.2 units against 27.6 units at 10 runs a day. Above it, the gap widens fast: 113.7 units against 27.6 at 80 runs a day, **4.1 times**. The side paying a fixed cost gains as run count grows, because what it pays for is not the run, but time. When the same numbers are read per run, the curve becomes visible. In the managed type, cost per run is **1.42 units** at every intensity and never changes — 14.2/10 and 113.7/80 give the same number. In the self-hosted type, cost per run drops from 2.76 at 10 runs, to 1.38 at 20, to **0.35 units** at 80. This also separates the behavior the two types produce on a team: under a fixed cost, running one more run is close to free, and nobody counts runs; under per-run billing, every extra run has a price, and unnecessary runs show up in the account. This same measure will feed into trigger design in a later lesson. The queue column reverses the decision. The two-slot self-hosted type starts clogging at 30 runs a day: **12 of 30 runs wait**, the average wait is 1 minute, the longest wait is 5.3 minutes. At 80 runs a day, **63 runs wait**, averaging 6.6 minutes with the longest wait at 20.9 minutes. The eight-slot managed type, on the same arrivals, makes only 8 of 80 runs wait, and the longest wait is 1 minute. The total wait is striking: at 80 runs, 6.6 × 80 = **528 minutes**, more than a full workday of waiting, against a total of 5 minutes for the managed type. It is also worth counting that the queue does not come from the daily average. At 30 runs a day, total work is 30 × 163 = 4,890 seconds; two slots over a nine-hour day offer 64,800 seconds of capacity, that is, 8 percent utilization. Yet 12 runs wait. The reason is the shape of the arrivals: runs land in three peaks, each within a twenty-minute window, and at the center of a peak both slots are full at once. If the capacity decision is made against the **daily total**, the queue never shows up; what the decision needs to look at is the busiest twenty minutes. This wait feeds directly into the gate account. The "time a gate adds," which the previous lesson measured, was only pipeline duration; the queue is added in front of it. At 80 runs a day on the self-hosted type, a change's feedback averages 163 + 396 seconds — **two and a half times the run duration in waiting**. The effort spent shortening the pipeline by 40 seconds becomes invisible next to what is lost in the queue. The queue can be closed with money, and the arithmetic is simple. If four machines are kept instead of two, the fixed cost rises to 4 × 11 + 5.6 = **49.6 units**, still less than half of the 113.7 units at 80 runs a day. The threshold thus stops being a single number: the decision sits at the intersection of run count and an acceptable wait time. #### The Gate Value of a Clean Start The cache column is where the two types are really split apart. The self-hosted runner keeps the **47.9 percent** hit rate the previous lesson measured; the managed runner starts clean on every run, and its hit rate is **zero**. The 40-second-per-run difference comes from here. The gate account shows the other side of this difference. In the previous lesson, two defects (D4, D6) escaped, because when an undeclared input changed, the gate in question came from the cache and never ran at all. On a runner with no cache, that cannot happen: every gate runs on every run, and in the twelve-run sequence, **eight of eight defects** are caught. The two defects cost 40 seconds per run, 480 seconds over the sequence, 20 minutes and 15 units at 30 runs a day. A middle path can be tried: the managed type can run by restoring its cache from a remote store. The measurement counts this option too — the run drops to 187 seconds, costing 42.3 units at 30 runs a day. The gain is not as large as advertised: the drop from 203 to 187 seconds is paid for with transfer time and storage cost, so the daily cost falls only from 42.6 to 42.3. More importantly, **the stale hit comes back** — the defect count reverts from 8/8 to 6/8. The managed type's two-defect edge was not born from its speed at all; it was born exactly from its **lack of a cache**. When these three columns come together, the decision rule does not collapse into one sentence, but it does fall into an order. If run count is below the threshold, the managed type is both cheaper and ahead in the gate account; the choice is not up for debate. If it is clearly above the threshold, the money points to the self-hosted type, but two items still have to be paid back: the extra slots needed to close the queue, and the defect the stale hit misses. The second cannot be closed with money; the way to close it is where the previous lesson pointed — declaring gates' inputs completely. In the twelve-run sequence, that cost 54 seconds; catching the same two defects by changing runner type instead costs 480 seconds. **The same defect, a nine-times more expensive fix.** What the measure leaves out is named. On the managed side: moments when the concurrency ceiling is exceeded, data transfer fees, the pipeline stopping entirely during a provider outage, the pipeline's secret values being resolved outside it, and the pipeline definition being adapted when the runner version changes. On the self-hosted side: the initial setup effort, spare machines, days lost to failure, delayed security patches, the electricity and floor-space cost of idle capacity, and capacity-planning error. Both sides share two uncounted items: the context loss of the person waiting in the queue, and rerunning a run that finished failed. Five maintenance items are listed, but they are priced only for six hours of labor a month; a month where maintenance is skipped never shows up in this column. #### Summary - The threshold is **20 runs a day**: below it the managed type is cheaper (14.2 against 27.6 units at 10 runs), above it the self-hosted type is cheaper (27.6 against 113.7 units at 80 runs, 4.1 times). - The threshold does not decide alone. At 30 runs a day, the two-slot self-hosted type makes 12 runs wait; at 80 runs, 63 runs wait, averaging 6.6 minutes with a longest of 20.9. On the same arrivals, the eight-slot managed type makes only 8 runs wait, up to 1 minute. - The queue is written in front of the time a gate adds: at 80 runs a day, the wait is two and a half times the 163-second run. Moving to four machines brings the fixed cost to 49.6 units and closes the queue with money. - The cost items sit on different axes: one item for the managed type (runner-minutes, per run), two items for the self-hosted type (machine and maintenance labor, per time), plus five maintenance items. - A clean-starting runner zeroes the cache hit rate (47.9 percent against 0) and catches 8/8 defects in the twelve-run sequence; the self-hosted type stays at 6/8. The two defects cost 40 seconds per run, 20 minutes and 15 units at 30 runs a day. - The remote cache store middle path brings the run down to 187 seconds but brings the stale hit back, and the defect count reverts to 6/8; the managed type's edge comes not from its speed but from its lack of a cache. #### Next Step In this measurement the pipeline always ran the same pipeline — eight stages, the same dependencies, the same gates. One question went unasked: where does this definition live, and who changes it? Raising the slot count from two to four, removing a gate, or splitting the integration test all touch the pipeline's **definition**. If the definition lives as something clicked through in an interface, who made the change, when, and why is written down nowhere; and when the pipeline breaks, there is no prior state to go back to. If the definition lives as a file in the repository, all of that exists — but a file per service for five services means the same lines repeated five times. The next lesson defines the schema itself and counts two forms: the traceability of a change, the fraction of pipeline changes that go through without review, the rollback step when the pipeline itself breaks, and the line cost of duplicating the definition. ### Pipeline as Code Source: https://academia.sh/en/courses/ci-cd/pipeline-as-code The same pipeline is defined in two forms — by hand through an interface and as a file in the repository: a schema is defined within the lesson and six faulty changes are run through a real validator, and traceability, the fraction of pipeline changes that pass without review, the rollback step, and the line cost of duplicating the definition are counted. The previous two lessons shortened pipeline duration and decided where it runs. Both touched the same object, but the object itself was never questioned: eight stages, the dependencies between them, which stage is a gate, and how many slots are used are **written down** somewhere. Raising the slot count from two to four, and removing a gate, both mean touching that written thing. Where it is written down is a choice between two options. The pipeline can be configured **by hand through an interface**: fields are filled in, saved, and take effect immediately. Or it can sit **as a file in the repository**: the change arrives as a commit, is reviewed, and is versioned. This lesson compares the two forms on four numbers — traceability, the fraction of changes that pass without review, the rollback step, and the duplication of the definition across five services. **PD37.** The schema, the validator, and the file form are defined in this lesson; the field names `pipeline`, `slots`, `stage`, `command`, `deps`, `gate` are not the fields of a real tool. **PD38.** Twenty pipeline changes stand in for a quarter of the fictional measurement network; six are faulty. **PD39.** The interface form's picker structurally blocks three kinds of error — an unknown field cannot be written, a nonexistent stage cannot be linked to, a number field cannot take text; against that, the interface has no review step. **PD40.** In the repository form, changes 3, 11, and 17 passed without review on grounds of urgency. **PD41.** Review wait is 42 minutes, schema validation is 4 seconds; the rollback steps are counted, their minutes are fictional. **PD42.** Line counts are counted from the actually produced text. #### The Schema Itself The first thing gained when a pipeline definition sits as a file is this: the definition is now **data**, and data can be read. The schema declares fields and their types; the validator reads the schema and checks the definition. ```js // schema.mjs — pipeline definition schema, validator, and writer (field names defined in this lesson) export const SCHEMA = { pipeline: "text", slots: "number", stages: "array" }; const STAGE = { name: "text", command: "text", deps: "array", gate: "yn" }; const types = { text: (d) => typeof d === "string", number: (d) => Number.isInteger(d), array: (d) => Array.isArray(d), yn: (d) => d === "yes" || d === "no" }; const checkFields = (obj, schema, prefix, errors) => { for (const key of Object.keys(obj)) if (!schema[key]) errors.push(`unknown field: ${prefix}${key}`); for (const [key, type] of Object.entries(schema)) if (!(key in obj)) errors.push(`missing field: ${prefix}${key}`); else if (!types[type](obj[key])) errors.push(`type error: ${prefix}${key}, expected ${type}`); }; export const validate = (def) => { const errors = []; checkFields(def, SCHEMA, "", errors); if (!Array.isArray(def.stages)) return errors; const names = def.stages.map((s) => s.name); for (const s of def.stages) { checkFields(s, STAGE, `${s.name}.`, errors); for (const dep of s.deps || []) if (!names.includes(dep)) errors.push(`unknown dep: ${s.name} -> ${dep}`); } const inCycle = (name, seen) => (seen.includes(name) ? true : (def.stages.find((s) => s.name === name)?.deps || []).some((dep) => inCycle(dep, [...seen, name]))); for (const s of def.stages) if (inCycle(s.name, [])) { errors.push(`cycle: ${s.name}`); break; } return errors; }; export const BASE = { pipeline: "reading-collector", slots: 3, stages: [ ["dependency-resolution", "resolve", [], "no"], ["build", "build", ["dependency-resolution"], "no"], ["static-analysis", "analyze-static", ["dependency-resolution"], "yes"], ["unit-test", "test-unit", ["build"], "yes"], ["integration-test", "test-integration", ["build"], "yes"], ["packaging", "package", ["build"], "no"], ["contract-test", "test-contract", ["packaging"], "yes"], ["image-build", "build-image", ["packaging"], "no"], ].map(([name, command, deps, gate]) => ({ name, command, deps, gate })) }; // The definition's file form: every stage is three or four lines. export const write = (def) => [`pipeline ${def.pipeline}`, `slots ${def.slots}`, ...def.stages.flatMap((s) => [`stage ${s.name}`, ` command ${s.command}`, ...(s.deps.length ? [` deps ${s.deps.join(" ")}`] : []), ` gate ${s.gate}`])]; // Five services of the fictional measurement network: base pipeline + service-specific diffs. export const SERVICES = [ { name: "reading-collector", diff: [] }, { name: "verifier", diff: [["slots", 2]] }, { name: "billing", diff: [["remove", "contract-test"]] }, { name: "work-order", diff: [["add", { name: "latency-test", command: "test-latency", deps: ["packaging"], gate: "yes" }]] }, { name: "nightly-batch", diff: [["slots", 4], ["remove", "static-analysis"]] }, ]; export const apply = (s) => { const def = { ...BASE, pipeline: s.name, stages: BASE.stages.map((a) => ({ ...a })) }; for (const [kind, data] of s.diff) { if (kind === "slots") def.slots = data; if (kind === "remove") def.stages = def.stages.filter((a) => a.name !== data); if (kind === "add") def.stages.push(data); } return def; }; ``` The validator checks four things: unknown and missing fields, type mismatches, a `deps` reference to a stage that does not exist, and a cycle in the dependencies. All four are decided from the definition's **text**; there is no need to run the pipeline. #### Two Forms Six faulty changes are actually constructed, actually run through the validator, and for each one, what the two forms hold is written down separately. ```js // forms.mjs — the same pipeline in two forms: traceability, review, gate, rollback import { validate, BASE } from "./schema.mjs"; const clone = () => ({ ...BASE, stages: BASE.stages.map((a) => ({ ...a })) }); const N = 20, PASSED = [3, 11, 17]; // three pipeline changes that pass without review const TRACE = ["who", "when", "why", "what changed"]; // Six faulty pipeline changes. no: which change number it arrives on. iface: an error the interface // form cannot write by construction (a picker prevents it). human: an error only a human can see. const ERRORS = [ ["E1 unknown field", 2, 1, 0, (def) => { def.stage = def.stages; delete def.stages; }], ["E2 cycle", 5, 0, 1, (def) => { def.stages[1].deps = ["packaging"]; }], ["E3 unknown dep", 8, 1, 0, (def) => { def.stages[3].deps = ["build-old"]; }], ["E4 type error (slots)", 11, 1, 0, (def) => { def.slots = "three"; }], ["E5 gate removed", 14, 0, 1, (def) => { def.stages.splice(2, 1); }], ["E6 gate disabled", 17, 0, 1, (def) => { def.stages[4].gate = "no"; }], ]; const tally = { iface: 0, repo: 0, schema: 0, alone: 0 }; console.log("error no what the schema validator says iface repo"); for (const [name, no, iface, human, corrupt] of ERRORS) { const def = clone(); corrupt(def); const findings = validate(def), schemaCaught = findings.length > 0; const repoCaught = schemaCaught || (human && !PASSED.includes(no)); // a human does not look at a passed change tally.schema += schemaCaught; tally.iface += iface; tally.repo += repoCaught; tally.alone += repoCaught && !schemaCaught; console.log(name.padEnd(24) + String(no).padStart(2) + " " + (findings[0] || "— (schema passed)").slice(0, 42).padEnd(44) + (iface ? "holds" : "escapes").padEnd(9) + (repoCaught ? "holds" : "escapes")); } const FORMS = { interface: { trace: TRACE.slice(0, 2), unreviewed: N, min: 35, rollback: ["find the change", "recall the old value", "write it back by hand", "save", "trigger manually", "verify"] }, repo: { trace: TRACE, unreviewed: PASSED.length, min: 4, rollback: ["revert to the previous version", "let the pipeline run itself"] }, }; console.log(`\ntraceability (${N} changes x ${TRACE.length} fields):`); for (const [name, f] of Object.entries(FORMS)) console.log(` ${name.padEnd(11)}${f.trace.length * N}/${TRACE.length * N} fields — ${f.trace.join(", ")}`); console.log(`passes without review: interface ${N}/${N} (100%), repo ${PASSED.length}/${N} (${(PASSED.length * 100) / N}%)`); console.log(`gate account (6 errors): interface ${tally.iface}/6, repo ${tally.repo}/6 (schema ${tally.schema}, review alone ${tally.alone})`); console.log("\nrollback when the pipeline breaks:"); for (const [name, f] of Object.entries(FORMS)) console.log(` ${name.padEnd(11)}${f.rollback.length} steps, ${f.min} min — ${f.rollback.join(" / ")}`); console.log(`\nthe time the gate adds: schema 4 s x ${N} = ${4 * N} s; review 42 min x ${N - PASSED.length} = ${42 * (N - PASSED.length)} min`); ``` ``` error no what the schema validator says iface repo E1 unknown field 2 unknown field: stage holds holds E2 cycle 5 cycle: build escapes holds E3 unknown dep 8 unknown dep: unit-test -> build-old holds holds E4 type error (slots) 11 type error: slots, expected number holds holds E5 gate removed 14 — (schema passed) escapes holds E6 gate disabled 17 — (schema passed) escapes escapes traceability (20 changes x 4 fields): interface 40/80 fields — who, when repo 80/80 fields — who, when, why, what changed passes without review: interface 20/20 (100%), repo 3/20 (15%) gate account (6 errors): interface 3/6, repo 5/6 (schema 4, review alone 1) rollback when the pipeline breaks: interface 6 steps, 35 min — find the change / recall the old value / write it back by hand / save / trigger manually / verify repo 2 steps, 4 min — revert to the previous version / let the pipeline run itself the time the gate adds: schema 4 s x 20 = 80 s; review 42 min x 17 = 714 min ``` #### The Gate's Account The table starts from an unexpected place: the interface form holds three errors **by construction**. An unknown field cannot be written because the field list is fixed; a nonexistent stage cannot be linked to because the choice comes from a picker; a number field cannot take text. In the repository form, these three errors **can be written**, and are caught only if the schema validator runs. Without a validator, the repository form is **worse** than the interface on these three errors. Once the validator runs, the table turns. The schema holds four errors; the fourth is the **cycle**, an error the interface cannot see: when the `build` stage is linked to `packaging`, the graph closes on itself, but every field is individually valid. An interface checks fields; it does not check the **graph**. Two errors remain, and both touch the gate itself: E5 deletes the static analysis stage, E6 turns the integration test's `gate` field to `no`. Both are flawless definitions according to the schema. Only a human can ask "why does this change remove a gate?" The repository form holds E5; E6 escapes, because change 17 is one of the three that pass without review. The total is **interface 3/6, repository 5/6**. The gate's marginal value is read separately: the schema holds four errors and adds a total of 80 seconds over 20 changes. Review holds **only one error** alone, and holds up 17 changes for **714 minutes**. Twelve hours for one defect — by this course's measure, this is the hardest gate to defend. Its defense sits in one place: the error it holds is a change that **removes a gate** — exactly the kind of thing that silently erases what every measurement in this course rests on. Against that, a much cheaper way to hold the same error is visible — adding a "gate count cannot decrease" rule to the schema. This lesson does not measure that rule; but if it did, review's marginal value would drop to zero, and the 714-minute gate could be removed. Traceability splits on two fields. The interface records **40 of 80 fields** — who and when. The two missing are "why" and "what changed," exactly the two questions to ask when the pipeline breaks at night. The rollback table is the second bill for the same gap: in the interface, rollback's first step is "find the change," and its answer is written down nowhere — six steps, 35 minutes. In the repository, reverting to the previous version is two steps and 4 minutes, because "what changed" is already on record. #### Five Services The fictional measurement network's five services use almost the same pipeline. The definition can be copied to every service, or the shared part can be separated out and only the diffs declared. The two forms' lines are actually produced and counted. ```js // duplication.mjs — five services: full copy vs. shared-part form import { SERVICES, apply, write, BASE } from "./schema.mjs"; const full = SERVICES.map((s) => ({ name: s.name, lines: write(apply(s)) })); const totalFull = full.reduce((t, s) => t + s.lines.length, 0); // Shared part: everything except the pipeline name. Service file: inherit + name + diffs only. const shared = write(BASE).filter((l) => !l.startsWith("pipeline ")); const split = SERVICES.map((s) => ({ name: s.name, lines: ["inherit shared", `pipeline ${s.name}`, ...s.diff.flatMap(([kind, d]) => kind === "slots" ? [`slots ${d}`] : kind === "remove" ? [`stage-remove ${d}`] : [`stage ${d.name}`, ` command ${d.command}`, ` deps ${d.deps.join(" ")}`, ` gate ${d.gate}`])], })); const totalSplit = shared.length + split.reduce((t, s) => t + s.lines.length, 0); // Lines identical across all five files (counting repeats). const count = (a) => a.reduce((m, l) => m.set(l, (m.get(l) || 0) + 1), new Map()); const counts = full.map((s) => count(s.lines)); let inAll = 0; for (const [l, c] of counts[0]) inAll += Math.min(c, ...counts.slice(1).map((m) => m.get(l) || 0)); console.log("service full copy with shared part"); full.forEach((s, i) => console.log(s.name.padEnd(20) + String(s.lines.length).padStart(6) + String(split[i].lines.length).padStart(12))); console.log("shared file".padEnd(20) + " —" + String(shared.length).padStart(12)); console.log("total".padEnd(20) + String(totalFull).padStart(6) + String(totalSplit).padStart(12) + ` (${(((totalFull - totalSplit) * 100) / totalFull).toFixed(1)}% fewer lines)`); console.log(`\nlines identical across all five files: ${inAll} — in the full copy, ` + `${inAll * 4} of the ${inAll * 5} lines are repeats`); console.log("\nreadability cost"); console.log(` reading one service's pipeline: full copy 1 file, ${full[1].lines.length} lines; ` + `shared part 2 files, ${shared.length + split[1].lines.length} lines`); console.log(` services affected when one line in the shared file changes: ${SERVICES.length}/${SERVICES.length}`); console.log(` adding a new gate to five services: full copy ${5 * 4} lines / 5 separate changes; ` + `shared part 4 lines / 1 change`); ``` ``` service full copy with shared part reading-collector 33 2 verifier 33 3 billing 29 3 work-order 37 6 nightly-batch 29 4 shared file — 32 total 161 50 (68.9% fewer lines) lines identical across all five files: 24 — in the full copy, 96 of the 120 lines are repeats readability cost reading one service's pipeline: full copy 1 file, 33 lines; shared part 2 files, 35 lines services affected when one line in the shared file changes: 5/5 adding a new gate to five services: full copy 20 lines / 5 separate changes; shared part 4 lines / 1 change ``` The gain is large: **161 lines drop to 50**, a 68.9 percent reduction. Its source is plain — **24 lines** are identical across all five files, and in the full copy, these lines are written 120 times, meaning **96 lines are repeats**. The cost is not visible in the line count. Reading one service's pipeline means a **single** 33-line file in the full copy, and 35 lines across **two** files once the shared part is separated. The difference is two lines, but the line `slots 2` in that service's file says nothing on its own; what it overrides is only visible in the other file, and the effective definition sits nowhere in one whole piece. The real cost is blast radius. A single line changed in the shared file affects **all five of the five services** at once — which is also the gain: adding a new gate to five services is 20 lines and five separate changes in the full copy, 4 lines and a single change with the shared part. Duplication is scattered, but it lets one service be changed on its own; sharing makes every change a joint decision of all five services. If the five pipelines are meant to genuinely stay identical, sharing is correct; if they are expected to diverge, the 96 lines of repetition are a paid price for independence. #### Summary - Once the definition is a file, it becomes **data**: the schema declares fields and types, and the validator catches unknown fields, type errors, a reference to a nonexistent stage, and a dependency cycle without running the pipeline. - The interface form holds three errors **by construction** (the field list is fixed, the choice comes from a picker), but it does not check the graph and misses the cycle. Total: interface 3/6, repository 5/6. - The gate's marginal value: the schema holds four errors and adds 80 seconds over 20 changes; review holds only one error alone and holds up 17 changes for 714 minutes. The error it holds is a change that removes a gate; if a "gate count cannot decrease" rule is added to the schema, this gate becomes indefensible. - Traceability is 40/80 fields against 80/80. The two fields missing in the interface are "why" and "what changed"; rollback's first step is exactly that too — 6 steps and 35 minutes against 2 steps and 4 minutes. - For five services, the full copy is 161 lines, the shared-part form is 50 (68.9 percent fewer); the 96 repeats of the 24 lines identical across all five files are removed. The cost is readability and blast radius: reading one service grows to 2 files and 35 lines, and a single line in the shared file affects all five services at once. #### Next Step The pipeline is now defined, validated, versioned, and can be rolled back. One question still remains: **when** does this pipeline run? The previous lesson gave a run count between 10 and 80 a day and calculated the threshold from that number, but where the number came from was never said. What starts a run is a trigger, and a trigger is a gate too: it decides whether the run happens at all. There are three types — bound to an event, bound to a schedule, and pulled by hand — and the three produce very different numbers on the same calendar: how many runs, how many of them with nothing changed, how much delay between a change and its run, how many runs overlap each other. The next lesson runs all three on the same two-week fictional calendar, then adds a path-based filter and counts, together, the drop in unnecessary runs and the real change the filter misses. ### Triggers Source: https://academia.sh/en/courses/ci-cd/triggers Event, schedule, and manual triggering are all run on the same two-week fictional calendar: run count, wasted runs made when nothing has changed or when a change does not affect the pipeline's output, the delay between a change and its run, and overlapping runs are counted, and then a path-based filter is added and the drop in wasted runs is measured together with the real change it misses. The pipeline is defined, validated, and versioned. The previous two lessons used run counts between 10 and 80 a day and calculated thresholds from those numbers, but never said where the number came from. What starts a run is a **trigger**, and a trigger is a gate too — what it holds is the run itself. A pipeline that never runs catches no defect; one that runs needlessly only spends money and queue. There are three types: bound to an **event** (when a change lands in the repository), bound to a **schedule** (at set intervals), and pulled **manually** (when someone presses a button). The three produce very different numbers on the same calendar. This lesson counts four of them: run count, wasted runs, delayed feedback, and overlapping runs — then adds a path-based filter and reads the same numbers again. **PD43.** The calendar is two weeks of the fictional measurement network, produced with `node`; the generator is self-written, seed **9173**. There are ten workdays and no weekend changes. **PD44.** Path groups and their proportions are declared in the model; `docs/` does not affect the pipeline's output, the others do. **PD45.** Run duration is 203 seconds — the critical path measured in earlier lessons; an overlapping run starts before the previous one ends. **PD46.** Delay is the time between a change and the **start** of the first run covering it; a run's own duration is the same for every trigger, so it stays out of the delay. **PD47.** Manual triggering is modeled at 17:30 on every workday and skipped with a probability of one in four. **PD48.** A filtered-out run does not advance; uncovered changes carry over to the next run. #### Three Triggers, the Same Calendar ```js // calendar.mjs — two-week fictional change calendar and triggers (model) export const SEED = 9173; export const generator = (t) => () => ((t = (t * 48271) % 2147483647) / 2147483647); export const DAY = 86400, RUN = 203; // critical path duration measured in 05 (seconds) // Path groups: an effective one changes the pipeline's output, an ineffective one does not. export const PATHS = [["src/", 45, 1], ["test/", 20, 1], ["docs/", 15, 0], ["config/", 10, 1], ["shared-lib/", 10, 1]]; export const FILTER = ["src/", "test/", "config/"]; // shared-lib/ was not added to the list export const calendar = () => { const rnd = generator(SEED), changes = []; for (let day = 0; day < 14; day++) { if (day % 7 >= 5) continue; // weekend for (let i = 3 + Math.floor(rnd() * 6); i > 0; i--) { const p = rnd() * 100; let t = 0, path = PATHS[PATHS.length - 1]; for (const entry of PATHS) { t += entry[1]; if (p < t) { path = entry; break; } } changes.push({ at: day * DAY + Math.round((9 + rnd() * 9) * 3600), path: path[0], effective: path[2] }); } } return changes.sort((a, b) => a.at - b.at); }; export const TRIGGER = { event: (changes) => changes.map((x) => x.at), "time-4h": () => Array.from({ length: 84 }, (_, i) => Math.floor(i / 6) * DAY + (i % 6) * 4 * 3600), "time-nightly": () => Array.from({ length: 14 }, (_, day) => day * DAY + 2 * 3600), manual: () => { const rnd = generator(SEED + 1); return [...Array(14).keys()].filter((day) => day % 7 < 5 && rnd() > 0.25).map((day) => day * DAY + 17.5 * 3600); }, }; // Empty run: nothing has changed since the previous run. Irrelevant run: none of the changes it // covers affect the pipeline's output. Overlapping: a run that starts before the previous one ends. export const measure = (runs, changes) => { let empty = 0, irrelevant = 0, overlapping = 0, prev = -Infinity; runs.forEach((run, i) => { const covered = changes.filter((c) => c.at > prev && c.at <= run); if (!covered.length) empty++; else if (!covered.some((c) => c.effective)) irrelevant++; if (i > 0 && run - runs[i - 1] < RUN) overlapping++; prev = run; }); const delay = []; let uncovered = 0; for (const c of changes.filter((x) => x.effective)) { const run = runs.find((t) => t >= c.at); if (run === undefined) uncovered++; else delay.push(run - c.at); } return { runs: runs.length, empty, irrelevant, overlapping, uncovered, avg: delay.reduce((a, b) => a + b, 0) / (delay.length || 1), longest: Math.max(0, ...delay) }; }; // Path-based filter: a run happens only if one of the changes it covers matches the list. // A filtered-out run does not advance "prev"; uncovered changes carry over to the next run. export const filter = (runs, changes, list = FILTER) => { const kept = []; let prev = -Infinity; for (const run of runs) if (changes.some((c) => c.at > prev && c.at <= run && list.includes(c.path))) { kept.push(run); prev = run; } return kept; }; ``` ```js // trigger.mjs — three trigger types on the same two-week calendar import { calendar, TRIGGER, measure, PATHS, SEED } from "./calendar.mjs"; const changes = calendar(); const hr = (s) => (s / 3600).toFixed(1); const distribution = PATHS.map(([prefix]) => `${prefix}${changes.filter((x) => x.path === prefix).length}`).join(" "); console.log(`seed ${SEED}; ${changes.length} changes in two weeks, ${changes.filter((x) => x.effective).length} effective`); console.log(`path distribution: ${distribution}`); console.log("\ntrigger runs empty irrelevant wasted overlap avg delay longest uncovered"); const results = {}; for (const [name, make] of Object.entries(TRIGGER)) { const r = (results[name] = measure(make(changes), changes)); console.log(name.padEnd(16) + String(r.runs).padStart(6) + String(r.empty).padStart(7) + String(r.irrelevant).padStart(12) + String(r.empty + r.irrelevant).padStart(10) + String(r.overlapping).padStart(9) + (hr(r.avg) + " hr").padStart(13) + (hr(r.longest) + " hr").padStart(9) + String(r.uncovered).padStart(13)); } // The time trigger's added contribution when the event trigger is already there. const eventRuns = TRIGGER.event(changes); for (const name of ["time-4h", "time-nightly"]) { const combined = [...eventRuns, ...TRIGGER[name]()].sort((a, b) => a - b); const r = measure(combined, changes); console.log(`\nevent + ${name}: ${r.runs} runs (event alone ${results.event.runs}), ` + `empty ${r.empty}, avg delay ${hr(r.avg)} hr — the event trigger's own delay ${hr(results.event.avg)} hr`); console.log(` of the ${r.runs - results.event.runs} runs added, ${r.empty} are empty; ` + `contribution to delay ${(results.event.avg - r.avg).toFixed(1)} seconds`); } ``` ``` seed 9173; 53 changes in two weeks, 46 effective path distribution: src/21 test/12 docs/7 config/3 shared-lib/10 trigger runs empty irrelevant wasted overlap avg delay longest uncovered event 53 0 7 7 2 0.0 hr 0.0 hr 0 time-4h 84 58 1 59 0 2.0 hr 3.9 hr 0 time-nightly 14 4 0 4 0 12.4 hr 16.8 hr 0 manual 6 0 0 0 0 22.2 hr 103.1 hr 0 event + time-4h: 137 runs (event alone 53), empty 84, avg delay 0.0 hr — the event trigger's own delay 0.0 hr of the 84 runs added, 84 are empty; contribution to delay 0.0 seconds event + time-nightly: 67 runs (event alone 53), empty 14, avg delay 0.0 hr — the event trigger's own delay 0.0 hr of the 14 runs added, 14 are empty; contribution to delay 0.0 seconds ``` #### Reading the Four Numbers The event trigger makes 53 runs and its delay is **zero**: every change gives birth to its own run. Its wasted runs are 7, all of one kind — changes that only touch `docs/` cost a full run without changing the pipeline's output. Its overlapping runs are 2: two changes arrived less than 203 seconds apart and the second run started before the first finished. Overlap does not only spend a slot; when the first run finishes, its result already belongs to a stale tree. The four-hour schedule trigger is bad **on both axes at once**. It makes 84 runs — 58 percent more than the event trigger — and **58 of them are completely empty**, meaning no change has come in since the previous run. Its wasted-run rate is 59/84, that is, 70 percent. Against that, its average delay is not zero but **2 hours**, with the longest wait at 3.9 hours. It runs more, and it reports later. Manual triggering is the most deceptive row in the table. Its wasted runs are **zero** and its run count is only 6; on the cost column it wins by a wide margin. The delay column makes the call: an average of 22.2 hours, a longest of **103.1 hours** — more than four days. Zero wasted runs is not a virtue either: runs are so sparse that each one covers dozens of changes at once, so none of them go to waste. That same sparseness leaves it unclear **which change broke it** when a run turns red. #### The Gate That Can Be Removed Triggers can also be used together, and in practice the most common combination is event plus schedule. The measurement asks what this combination adds. When the four-hour schedule trigger is added alongside the event trigger, run count rises from 53 to **137**; **all 84 of the 84 added runs are empty**, because every change has already triggered its own run. Its contribution to delay is **0.0 seconds**. By this course's measure, the decision is clear: a gate is worth only what it alone catches, and this gate has nothing that belongs only to itself. The four-hour schedule trigger, with the event trigger already present, is **the gate that should be removed** — 84 empty runs, zero gain. The nightly run adds 14 runs by the same measure and also shows zero contribution on this calendar. Whether it should be removed cannot be measured here, and that is written down: this calendar only models changes that **arrive in the repository**. The nightly run's defense is a change outside it — a dependency's content growing without its list file changing. Earlier lessons measured that the cache key misses exactly this kind of change. The nightly run is the antidote to that blindness, but it is invisible here; deciding it needs a different calendar. #### The Path Filter The event trigger's only flaw was wasted runs: 7 runs were spent on `docs/` changes. The known remedy is a **path filter**, which ties a run to the change of only certain paths. The filter list carries three paths; the shared library directory, added to the repository later, was not written into the list. ```js // filter.mjs — path-based filter: the drop in wasted runs and the real change it misses import { calendar, TRIGGER, measure, filter, FILTER } from "./calendar.mjs"; const changes = calendar(), hr = (s) => (s / 3600).toFixed(1); const arrow = (a, b) => `${a} -> ${b}`.padStart(12); console.log(`filter list: ${FILTER.join(" ")} (shared-lib/ is not on the list)`); console.log("\ntrigger runs wasted uncovered avg delay"); for (const [name, make] of Object.entries(TRIGGER)) { const runs = make(changes), before = measure(runs, changes), after = measure(filter(runs, changes), changes); console.log(name.padEnd(16) + arrow(before.runs, after.runs) + arrow(before.empty + before.irrelevant, after.empty + after.irrelevant) + arrow(before.uncovered, after.uncovered) + ` ${hr(before.avg)} -> ${hr(after.avg)} hr`); } // Changes on a path not in the filter list: cannot trigger their own run, ride along on another one. const filtered = filter(TRIGGER.event(changes), changes); const outside = changes.filter((x) => x.path === "shared-lib/"); const wait = outside.map((x) => { const run = filtered.find((t) => t >= x.at); return run === undefined ? null : run - x.at; }); const riding = wait.filter((w) => w !== null); console.log(`\nshared-lib/ changes: ${outside.length}, none of them triggered its own run`); console.log(` ${riding.length} of them rode along on another change's run, averaging ` + `${hr(riding.reduce((a, b) => a + b, 0) / riding.length)} hr of delay, the longest being ${hr(Math.max(...riding))} hr`); console.log(` ${wait.length - riding.length} of them never ran at all across the two weeks`); // The gate's account: the wasted runs it holds, the real changes it falsely holds, the delay it adds. const eventRaw = measure(TRIGGER.event(changes), changes), eventFiltered = measure(filtered, changes); const corrected = measure(filter(TRIGGER.event(changes), changes, [...FILTER, "shared-lib/"]), changes); console.log(`\ngate account (event trigger + path filter)`); console.log(` holds : ${eventRaw.empty + eventRaw.irrelevant} wasted runs (docs/ changes)`); console.log(` falsely holds: ${outside.length} real changes, ${eventFiltered.uncovered} of them never ran at all`); console.log(` adds : average delay ${hr(eventRaw.avg)} -> ${hr(eventFiltered.avg)} hr`); console.log(` once the list is fixed: ${corrected.runs} runs, ${corrected.empty + corrected.irrelevant} wasted, ` + `${corrected.uncovered} uncovered, delay ${hr(corrected.avg)} hr`); ``` ``` filter list: src/ test/ config/ (shared-lib/ is not on the list) trigger runs wasted uncovered avg delay event 53 -> 36 7 -> 0 0 -> 1 0.0 -> 0.6 hr time-4h 84 -> 23 59 -> 0 0 -> 0 2.0 -> 2.6 hr time-nightly 14 -> 10 4 -> 0 0 -> 0 12.4 -> 12.4 hr manual 6 -> 6 0 -> 0 0 -> 0 22.2 -> 22.2 hr shared-lib/ changes: 10, none of them triggered its own run 9 of them rode along on another change's run, averaging 3.1 hr of delay, the longest being 19.9 hr 1 of them never ran at all across the two weeks gate account (event trigger + path filter) holds : 7 wasted runs (docs/ changes) falsely holds: 10 real changes, 1 of them never ran at all adds : average delay 0.0 -> 0.6 hr once the list is fixed: 46 runs, 0 wasted, 0 uncovered, delay 0.0 hr ``` #### The Filter's Two Faces If the first row is read alone, the filter is a success: on the event trigger, runs drop from 53 to 36, and wasted runs from 7 to zero. Once the second number is looked at, the arithmetic falls apart. The runs that dropped number **17**, yet only **7** were wasted. The 10 runs in between were runs of changes that really did affect the pipeline's output. All ten changes are in the shared library directory and **none triggered its own run**. Nine entered the pipeline by riding along on a later change's run, with an average delay of 3.1 hours and a longest of 19.9 hours. One was **never run at all** across the two weeks. Average delay rose from 0.0 to 0.6 hours — small-looking, because 36 of the 46 effective changes were never delayed; the average hides what happened to the ten. The gate account is complete in three lines: the filter **holds 7 wasted runs**, **falsely holds 10 real changes**, and one of them never runs at all. What it falsely holds outnumbers what it correctly holds. As it stands, the gate is harmful, and removing it would be better. Removing it is still not the right fix: the problem is not in the gate but in its list. Once the shared library path is added, the measurement gives the last row: **46 runs, 0 wasted, 0 uncovered, 0.0 hours of delay** — seven fewer runs than the raw event trigger, with no loss. That is the filter's value set up correctly; its cost set up wrong is the three lines above. The filter's effect on the schedule trigger is even larger: 84 runs drop to 23, 59 wasted runs drop to zero, and no uncovered change is born, because the changes a filtered-out run would have covered carry over to the next one. On manual triggering, nothing changes at all: all six of the six runs already cover a change that matches the filter. This measurement's real finding is hiding in the list. The pipeline's inputs are declared in **two separate places** across this course: in the cache key, and in the trigger filter. Both are hand-written, both go stale as the repository grows, and both produce the same result when they are incomplete — a change exists, but the gate does not run. In the earlier measurement, this let two defects escape; here it delayed ten changes and never ran one at all. #### Summary - On the same two-week calendar: the event trigger gave 53 runs and zero delay, the four-hour schedule trigger 84 runs and 2 hours, the nightly run 14 runs and 12.4 hours, manual triggering 6 runs and 22.2 hours (longest 103.1). - The four-hour schedule trigger is bad on both axes: it runs 58 percent more than the event trigger (59/84 wasted) and reports later. Manual triggering's zero wasted runs come from sparseness, not virtue, and that sparseness hides which change broke it. - With the event trigger present, the four-hour schedule trigger is **the gate that should be removed**: all 84 runs it adds are empty and its contribution to delay is 0.0 seconds. The nightly run's defense is a change outside the repository, which this calendar does not measure. - The path filter brought the event trigger's runs down from 53 to 36, but of the 17 dropped runs only 7 were wasted. Nine of the 10 changes in the unlisted directory rode along on another run with an average delay of 3.1 hours; one never ran across the two weeks. - Gate account: 7 wasted runs held, 10 real changes falsely held. With the list fixed the numbers reach 46 runs, 0 wasted, 0 uncovered, 0.0 hours of delay — the defect is not in the gate but in the list. The pipeline's inputs are declared by hand in two places (the cache key and the trigger filter), and an incomplete list gives the same result in both: the change exists, the gate does not run. #### Next Step The pipeline is complete at the end of this topic: its gates were counted, what each gate alone catches was measured, its duration was brought down to the 203-second critical path, and where and when it runs was decided. All of it rested on one silent assumption: that a change coming out of a green run **had reached production**. But a green run only says the change is ready in the pipeline, not that it is in production. How it enters production is a separate decision — all at once, to a slice first, by switching between two environments, or from behind a flag in the code? Each brings its own rollback path and observation criterion. The pipeline's last gate stands there, and what it holds, what it misses, and how long it holds are measured with the same three numbers. The next topic takes up that decision. ## Release Strategies How the same artifact enters production: branching strategy's relationship to integration frequency, the same artifact being promoted across environments, the balance that staged rollout strikes between escaped defects and affected requests, two-environment deployment's instant-rollback capability and its cost, feature flags separating deployment from release and the flag debt that accumulates, how many steps a rollback plan runs, and coordinating the ordering of database migrations. ### Branching Strategy and Pipeline Relationship Source: https://academia.sh/en/courses/ci-cd/branching-strategy-and-pipeline-relationship Long-lived branching and trunk-based development run on the same ninety-change stream: branch lifetime, conflicting files, gate runs per integration, and when the integration gate runs under each scheme are counted. Trunk-based development's own cost — the concealment record — is measured separately. The pipeline is built: its stages are ordered, each stage's gate is defined, and what each gate catches, misses, and holds up is counted. This does not say **how** a change will enter production. The pipeline's existence is not a flow scheme; the flow scheme is set by when a change joins the shared trunk and by what steps it advances from there to production. This topic measures those decisions, starting with the first one. The first decision is this: when does a developer's written change enter the shared trunk. There are two extreme schemes. In long-lived branching, work accumulates on its own branch and merges all at once, weeks later. In trunk-based development, every change joins the trunk on the day it is written. The two schemes carry the same work, with the same people, through the same defect set — but the **integration gate** over them runs at different moments, on pieces of different size. The gate here is an acceptance decision: it either lets the change into the shared trunk or it does not. It is the first promotion decision in the chain. - **RS1.** The fictional regional measurement network's verification service is kept in a single repository. Six developers, sixty workdays, ninety changes, forty source files. Both the network and the figures are fictional. - **RS2.** A feature belongs to a single author and is written in one to five pieces over consecutive days. Handed-off work, jointly written features, and pair programming are outside the model. - **RS3.** The defect set has three classes: an **isolated defect** (also visible on the branch), an **interaction defect** (visible only once both changes have entered the trunk), and a **production-data-dependent defect** (invisible at every integration gate). - **RS4.** The local gate running on the branch catches every isolated defect. The integration gate's marginal contribution is therefore the interaction defects. - **RS5.** One gate run takes fourteen minutes; a red batch is fixed and runs once more. The human time spent fixing it and queue waiting are outside the model. - **RS6.** Conflicts are counted at the file level. Line-level overlap, renaming, and file moves are not modeled. The probability of a flaky red is six percent, and it is only distinguishable when no real defect is present. #### Measurement Setup What is measured is not a real repository but a model set up with `node`. The model produces a change stream, consumes that stream under two different merge schemes, and counts when and on how many changes the gate runs in each scheme. The random generator is written by hand, and its seed is visible. ```js // branch.mjs — model of two branching schemes on the same change stream; not a real repository, a model export const SEED = 20260901; export const DAYS = 60, CHANGES = 90, DEVELOPERS = 6, FILES = 40; export const GATE_MIN = 14, LOCAL = 0.10, PRODUCTION = 0.08, PAIR = 0.06, FLAKY = 0.06; export function generator(seed) { // linear congruential generator, seed visible let x = seed >>> 0; return () => { x = (Math.imul(x, 1664525) + 1013904223) >>> 0; return x / 4294967296; }; } // The stream is produced once: both schemes consume the same changes and the same defect set. // A feature belongs to a single author and is written in a few pieces over consecutive days. export function stream() { const rnd = generator(SEED), l = []; while (l.length < CHANGES) { const author = Math.floor(rnd() * DEVELOPERS), start = 1 + Math.floor(rnd() * (DAYS - 5)); const pieces = 1 + Math.floor(rnd() * 5), feat = l.length && l[l.length - 1].feature + 1 || 0; for (let j = 0; j < pieces && l.length < CHANGES; j += 1) { const d = new Set(); // hot-file bias: squaring skews toward low indices for (let k = 0, n = 1 + Math.floor(rnd() * 3); k < n; k += 1) d.add(Math.floor(rnd() ** 2 * FILES)); l.push({ no: l.length, author, day: start + j, file: [...d], feature: feat, local: rnd() < LOCAL, production: rnd() < PRODUCTION }); } } return l.sort((a, b) => a.day - b.day || a.no - b.no); } // An interaction defect belongs to a pair: two changes from different authors touching the same file. // Invisible on an isolated branch; it only surfaces once both have entered the trunk. export function pairs(l) { const c = []; for (let i = 0; i < l.length; i += 1) for (let j = i + 1; j < l.length; j += 1) { if (l[i].author === l[j].author || !l[i].file.some((f) => l[j].file.includes(f))) continue; let h = Math.imul(l[i].no + 1, 2654435761) ^ Math.imul(l[j].no + 1, 40503); if ((Math.imul(h ^ (h >>> 13), 1274126177) >>> 0) / 4294967296 < PAIR) c.push([l[i].no, l[j].no]); } return c; } // interval 0: trunk-based, every change merges on its own. interval k: an author's branch merges every k days. export function run(l, pair, interval) { const rnd = generator(SEED + 7), merge = new Map(), batch = new Map(); for (const c of l) { const day = interval === 0 ? c.day : Math.ceil(c.day / interval) * interval; const key = interval === 0 ? `t${c.no}` : `${c.author}:${day}`; merge.set(c.no, day); if (!batch.has(key)) batch.set(key, { day, author: c.author, member: [] }); batch.get(key).member.push(c); } const s = { batch: batch.size, runs: 0, red: 0, cleanBlocked: 0, flakyBlocked: 0, conflicts: 0, local: 0, interaction: 0, escaped: 0, age: 0 }; const trunk = new Map(); for (const p of [...batch.values()].sort((a, b) => a.day - b.day)) { const base = p.day - (interval || 1), guilty = new Set(); for (const f of new Set(p.member.flatMap((c) => c.file))) { const seen = trunk.get(f); if (seen && seen.author !== p.author && seen.day > base) s.conflicts += 1; trunk.set(f, { author: p.author, day: p.day }); } for (const c of p.member) { s.age += p.day - c.day; if (c.production) s.escaped += 1; if (c.local) { guilty.add(c.no); s.local += 1; } } for (const [a, b] of pair) { // shows up in the batch of whichever member merges later if (Math.max(merge.get(a), merge.get(b)) !== p.day) continue; const later = p.member.find((c) => c.no === b) || p.member.find((c) => c.no === a); if (!later || merge.get(a) === merge.get(b) && !p.member.some((c) => c.no === a)) continue; guilty.add(later.no); s.interaction += 1; } s.runs += 1; const flaky = guilty.size === 0 && rnd() < FLAKY; if (guilty.size || flaky) { // a red batch is fixed and reruns s.red += 1; s.runs += 1; s.cleanBlocked += p.member.length - guilty.size; if (flaky) s.flakyBlocked += p.member.length; } } return { ...s, age: s.age / l.length, size: l.length / batch.size, min: s.runs * GATE_MIN }; } ``` The model has a single knob: `interval`. At zero, every change integrates on its own; this is trunk-based development. At ten, everything an author writes in ten days merges into a single batch; this is long-lived branching. The defect set, on the other hand, is the same no matter what `interval` is — an interaction defect belongs to a pair, and both members of that pair enter the trunk sooner or later. #### Two Schemes, One Stream ```js // run1.mjs — the same stream is run through two branching schemes import { stream, pairs, run, SEED, CHANGES, DEVELOPERS, GATE_MIN } from './branch.mjs'; const l = stream(), c = pairs(l); console.log(`seed ${SEED} | ${CHANGES} changes, ${DEVELOPERS} developers, 60 workdays`); console.log(`defect set: ${l.filter((x) => x.local).length} isolated defects, ${c.length} interaction defects, ` + `${l.filter((x) => x.production).length} production-dependent defects`); console.log(`integration gate run ${GATE_MIN} min\n`); console.log('scheme integrations batch branch life conflicting gate gate-only wasted gate'); console.log(' size (days) files runs catches blocked minutes'); for (const [ad, interval] of [['trunk-based', 0], ['long-lived branch', 10]]) { const s = run(l, c, interval); console.log(`${ad.padEnd(22)} ${String(s.batch).padEnd(13)} ${s.size.toFixed(2).padEnd(7)} ` + `${s.age.toFixed(2).padEnd(12)} ${String(s.conflicts).padEnd(12)} ${String(s.runs).padEnd(9)} ` + `${String(s.interaction).padEnd(10)} ${String(s.cleanBlocked).padEnd(9)} ${s.min}`); } ``` ``` seed 20260901 | 90 changes, 6 developers, 60 workdays defect set: 10 isolated defects, 24 interaction defects, 8 production-dependent defects integration gate run 14 min scheme integrations batch branch life conflicting gate gate-only wasted gate size (days) files runs catches blocked minutes trunk-based 90 1.00 0.00 8 117 24 2 1638 long-lived branch 20 4.50 4.47 44 35 24 54 490 ``` Both schemes caught the same twenty-four interaction defects. What the gate **catches** does not change; what changes is when it catches it and how much clean work it stops to do so. The branch-life column gives the timing. Under trunk-based development, a change enters the gate on the day it is written; the average age is zero. Under branching, it waits four and a half days on average, because the gate runs only at merge time. The local gate on the branch runs during that whole wait, but it can see nothing beyond the ten isolated defects: an interaction defect, by definition, waits for two changes to meet. So under the branching scheme, **half the gates run early, the discriminating one runs late**. The cost shows up in the wasted-blocked column. Under trunk-based development a batch is a single change; when the gate is red, the only thing that stops is the guilty change, and only two changes are wastefully blocked — both from a flaky red. Under branching a batch is four and a half changes on average; when the gate is red, the whole batch stops, and fifty-four clean changes wait for something that is not their own fault. Catching the same twenty-four defects costs two unnecessary stops under one scheme and fifty-four under the other. The time column reverses direction. Trunk-based development spent 1,638 gate-minutes, branching spent 490. Machine time per gate run is the same; the difference is in the run count. Branching pays about twenty-four and a half minutes of gate time per integration, trunk-based pays eighteen — but trunk-based pays it ninety times over. Running the gate rarely cuts machine time to a third, and also delays the moment it discovers the defect it is there to find. #### Merge Interval Scan To see what lies between the two extremes, the interval is scanned. The same scan also produces trunk-based development's own bill: unfinished work sits in the trunk and needs a separate scheme so it does not leak into production. ```js // run2.mjs — the merge interval is scanned; the cost of trunk-based — half-finished work — is counted separately import { stream, pairs, run } from './branch.mjs'; // A feature left half-done in the trunk: its first piece has entered, its last piece has not. function unfinished(l, interval) { const dayOf = (c) => (interval === 0 ? c.day : Math.ceil(c.day / interval) * interval); const feat = new Map(); for (const c of l) { if (!feat.has(c.feature)) feat.set(c.feature, []); feat.get(c.feature).push({ d: dayOf(c), author: c.author }); } let records = 0, featCount = 0; for (const p of feat.values()) { const last = Math.max(...p.map((x) => x.d)); // every piece entering before the last piece must be concealed const early = p.filter((x) => x.d < last).length; records += early; featCount += early > 0 ? 1 : 0; } return { records, featCount, features: feat.size }; } const l = stream(), c = pairs(l); console.log(`${l.length} changes belong to ${unfinished(l, 0).features} features; piece count 1-5\n`); console.log('interval batch branch life conflicting gate red wasted flaky gate hidden features'); console.log('(days) size (days) files runs batches blocked blocked minutes records hidden'); for (const a of [0, 1, 2, 3, 5, 10, 20]) { const s = run(l, c, a), y = unfinished(l, a); console.log(`${String(a || 'trunk').padEnd(9)} ${s.size.toFixed(2).padEnd(6)} ${s.age.toFixed(2).padEnd(12)} ` + `${String(s.conflicts).padEnd(12)} ${String(s.runs).padEnd(7)} ${String(s.red).padEnd(8)} ` + `${String(s.cleanBlocked).padEnd(8)} ${String(s.flakyBlocked).padEnd(9)} ${String(s.min).padEnd(10)} ` + `${String(y.records).padEnd(8)} ${y.featCount}`); } ``` ``` 90 changes belong to 30 features; piece count 1-5 interval batch branch life conflicting gate red wasted flaky gate hidden features (days) size (days) files runs batches blocked blocked minutes records hidden trunk 1.00 0.00 8 117 27 2 2 1638 60 23 1 1.30 0.00 8 95 26 9 2 1330 60 23 2 2.09 0.47 18 67 24 31 2 938 46 20 3 2.73 0.98 24 55 22 43 1 770 38 16 5 3.33 2.08 37 46 19 48 0 644 25 12 10 4.50 4.47 44 35 15 54 0 490 13 6 20 6.43 9.80 59 25 11 56 0 350 1 1 ``` The scan sets two curves against each other. As the interval grows, gate minutes fall from 1,638 to 350, conflicting files rise from eight to fifty-nine, and wastefully blocked changes rise from two to fifty-six. A three-day interval is an interesting middle point: gate minutes are roughly halved, but wastefully blocked changes have already gone from two to forty-three, more than twentyfold. The trade-off is not linear — every minute saved from machine time turns, past a certain point, into wait time for work that carries no fault of its own. The flaky column carries a quiet finding. Once the interval reaches five days, the changes a flaky run wastefully blocks drop to zero. Flakiness has not gone down; close to three quarters of the batches already carry a real defect, so a flaky run can no longer be told apart from a real one. Counting what a gate wrongly holds requires the gate to be green most of the time. In a scheme where red is the normal state, wrongly holding a change becomes unmeasurable — and unmeasurable wrong-holding turns into a team habit of skipping the gate. The last two columns are trunk-based development's own bill. Ninety changes belong to thirty features; in the trunk-based scheme, sixty of them enter the trunk before the feature they belong to is complete. Twenty-three features sit half-done in the trunk. For these changes not to leak into production, every one of them needs to carry a concealment record: the code will be in the trunk, the behavior will stay off. At a ten-day interval this number drops to thirteen, at a twenty-day interval to one — because unfinished work stays hidden on the branch instead. Trunk-based development's cost is exactly here, and it is as concrete as gate minutes: sixty concealment records, a switch to open and close for each of twenty-three features, and twenty-three separate cleanup jobs to remove those switches once each feature is done. This is not a side effect of trunk-based development; it is an entry requirement. A team that moves to trunk-based development without first setting up a concealment scheme has built a scheme that leaks unfinished work into production. #### The Gate's Three Numbers The integration gate's account closes with this measurement. **What it catches:** across the ninety-change stream, ten isolated and twenty-four interaction defects, thirty-four in total. **What only it catches:** twenty-four — the local gate on the branch would also catch the isolated defects, but only the integration gate can see an interaction defect. The gate's marginal value is this twenty-four, and it is the same under both schemes. **What it misses:** eight production-data-dependent defects, under both schemes. This number does not change with branching strategy; merge frequency does not make visible what the gate cannot see in the first place. **What it wrongly holds:** two clean changes under trunk-based development, fifty-four under the ten-day branching scheme. **The time it adds:** between eighteen and twenty-four and a half minutes per integration; between 490 and 1,638 minutes in total. Looked at through these numbers, the decision is not "which scheme is better." The gate's value is fixed; what gets chosen is which currency that fixed value gets paid in — machine minutes, or the wait time and conflict resolution of clean work. #### Summary - The same stream was run through two schemes: the integration gate caught the same twenty-four interaction defects under both, and missed the same eight production-dependent defects under both. - The split is in when the gate runs: at zero days old under trunk-based development, at an average of four and a half days under the ten-day branching scheme. The local gate on the branch runs early but cannot see interaction defects. - As batches grow, the cost of wrongly holding a change grows with them: wastefully blocked clean changes go from two under trunk-based development to fifty-four at a ten-day interval. Conflicting files go from eight to forty-four. - Gate minutes run in the opposite direction: 1,638 for trunk-based development, 490 for the ten-day interval. A gate that runs rarely is cheap, and it speaks late. - Past a five-day interval, a flaky run becomes indistinguishable, because most batches are already red; unmeasurable wrong-holding turns into a habit of skipping the gate. - Trunk-based development's cost was counted: sixty concealment records, twenty-three features, and twenty-three cleanup jobs afterward. Moving to trunk-based development without this scheme in place opens production to unfinished work. #### Next Step This measurement ends the moment a change enters the trunk. What enters the trunk is not yet in production: at least one build artifact and a few environments stand between. The next question is how that artifact moves between environments. Does the same artifact get promoted, or does every environment compile its own build? The next lesson runs both schemes on the same candidate sequence: the number of environment-specific defects, what the promotion gate catches, whether the artifact's identity is preserved across environments, and how long an artifact waits at the promotion gate. ### Environment Promotion Source: https://academia.sh/en/courses/ci-cd/environment-promotion The scheme where the same build artifact promotes between environments is run against the scheme where every environment rebuilds, on the same forty-candidate sequence: environment-specific defects, what the promotion gate catches, whether the artifact's identity is preserved across environments, and how long an artifact waits at the promotion gate are counted. The change entered the shared trunk and passed the integration gate. This does not mean the change is in production. At least one build artifact and a few environments stand between: development, staging, production. This lesson's question is **how** that artifact advances between environments. There are two schemes. In the first, the artifact is produced once and the same object is promoted to every environment. In the second, every environment does its own build; the source is the same, the artifact does not have to be. Whether environments are equivalent to one another is a separate measure's subject and is not repeated here. This lesson measures the difference that remains even when equivalence is achieved: **the difference in the running object itself**. In this lesson the promotion gate is an approval decision — it either lets an artifact move to the next environment or it does not — and the gate's value depends on whether the object it approves is the same object it tested. - **RS7.** Forty release candidates are generated for the fictional regional measurement network's verification service; there are three environments, and the candidates spread across a forty-day horizon. Both the network and the figures are fictional. - **RS8.** Environments are taken as equivalent in configuration and data. Environment-equivalence measurement is not repeated; the only source of difference here is the artifact itself. - **RS9.** Dependency resolution picks the newest version at the moment of resolution. The version schedule for the eight dependencies is fictional; lockfiles, mirroring, and manual version pinning are outside the model. - **RS10.** The defect classes are three: a **source defect** (present in every artifact), a **dependency-version defect** (present only in an artifact that includes that version), and a **production-scale-dependent defect** (invisible at every promotion gate). - **RS11.** The promotion gate runs in a window that opens every four hours; production promotion also waits zero to eight hours for approval. A gate run takes eighteen minutes, a build twenty-two. - **RS12.** Artifact identity is the first twelve hexadecimal digits of the digest of the input record. Digest collision is outside the model. #### Measurement Setup What is measured is not a real pipeline but a model set up with `node`. The model produces a dependency version schedule, resolves the candidates against that schedule, and actually computes artifact identity with `node:crypto`. The random generator is written by hand, and its seed is visible. ```js // promotion.mjs — model of the scheme where the same artifact promotes vs. every environment rebuilding import { createHash } from 'node:crypto'; export const SEED = 20260902; export const CANDIDATES = 40, DEPENDENCIES = 8, HORIZON_HOURS = 960, INTERVAL_HOURS = 24; export const BUILD_MIN = 22, GATE_MIN = 18, WINDOW_HOURS = 4; export const SOURCE = 0.25, SCALE = 0.10, VERSION_DEFECT = 0.12, FLAKY = 0.05; export const DEV_CATCH = 0.45, STAGING_CATCH = 0.75; export function generator(seed) { // linear congruential generator, seed visible let x = seed >>> 0; return () => { x = (Math.imul(x, 1664525) + 1013904223) >>> 0; return x / 4294967296; }; } export const digest = (candidate, resolved) => // the build artifact's identity: a digest of its inputs createHash('sha256').update(JSON.stringify({ candidate, resolved: resolved.map((v) => v.name) })).digest('hex').slice(0, 12); // Every dependency has a version schedule; resolution picks the newest version at that moment. export function schedule(rnd) { return Array.from({ length: DEPENDENCIES }, (_, i) => { const versions = [{ hour: 0, name: `d${i}v1`, defect: false }]; for (let hour = 0; ;) { hour += 40 + Math.floor(rnd() * 200); if (hour >= HORIZON_HOURS) break; versions.push({ hour, name: `d${i}v${versions.length + 1}`, defect: rnd() < VERSION_DEFECT }); } return versions; }); } export const resolve = (sched, hour) => sched.map((v) => v.filter((x) => x.hour <= hour).pop()); export function candidates(rnd) { return Array.from({ length: CANDIDATES }, (_, i) => ({ no: i + 1, hour: Math.floor((i + 0.5) * (HORIZON_HOURS / CANDIDATES) + rnd() * 8), source: rnd() < SOURCE, scale: rnd() < SCALE, devRoll: rnd(), stagingRoll: rnd(), depRoll: rnd(), approval: Math.floor(rnd() * 9), })); } export const wait = (hour) => (WINDOW_HOURS - (hour % WINDOW_HOURS)) % WINDOW_HOURS; // rebuild=false: one build, the same artifact promotes. rebuild=true: every environment builds its own. export function run(rebuild, interval = INTERVAL_HOURS) { const rnd = generator(SEED), sched = schedule(rnd), cand = candidates(rnd), flakyRnd = generator(SEED + 3); const s = { digestSame: 0, sourceEscaped: 0, scaleEscaped: 0, devMarginal: 0, stagingMarginal: 0, uniqueEscaped: 0, testedEscaped: 0, caught: 0, wronglyHeld: 0, versionDrift: 0, min: 0, wait: 0, wait1: 0, peak: 0 }; for (const c of cand) { const staging = rebuild ? resolve(sched, c.hour + 6) : resolve(sched, c.hour); const prod = rebuild ? resolve(sched, c.hour + 6 + interval) : resolve(sched, c.hour); if (digest(c.no, staging) === digest(c.no, prod)) s.digestSame += 1; if (c.source) { // a source defect is present in every build const g = c.devRoll < DEV_CATCH, h = c.stagingRoll < STAGING_CATCH; if (g || h) s.caught += 1; else s.sourceEscaped += 1; if (g && !h) s.devMarginal += 1; // caught only at the dev gate if (h && !g) s.stagingMarginal += 1; // caught only at the staging gate } if (c.scale) s.scaleEscaped += 1; // a production-scale defect: invisible at every gate const tested = new Set(staging.map((v) => v.name)); s.versionDrift += prod.filter((v) => !tested.has(v.name)).length; for (const v of prod.filter((x) => x.defect)) { if (!tested.has(v.name)) s.uniqueEscaped += 1; // present in production, absent from the tested artifact else if (c.depRoll < STAGING_CATCH) s.caught += 1; else s.testedEscaped += 1; } for (let k = 0; k < 2; k += 1) if (flakyRnd() < FLAKY) s.wronglyHeld += 1; s.min += (rebuild ? 3 : 1) * BUILD_MIN + 2 * GATE_MIN; const ready = c.hour + (rebuild ? BUILD_MIN / 60 : 0), w1 = wait(ready); const w2 = wait(ready + w1 + GATE_MIN / 60 + c.approval + (rebuild ? BUILD_MIN / 60 : 0)); s.wait += w1 + w2; s.wait1 += w1; s.peak = Math.max(s.peak, w1 + w2); } return { ...s, avgWait: s.wait / CANDIDATES }; } ``` The model has a single switch: `rebuild`. When it is off, the artifact resolves once, the moment the candidate is ready, and that resolution holds for all three environments. When it is on, staging does its own build at hour six, production twenty-four hours later. The source is the same across all three builds; the only thing that changes is **when** it gets resolved. #### Two Schemes, One Candidate Sequence ```js // run1.mjs — the same candidate sequence is run through two promotion schemes import { run, digest, schedule, candidates, generator, resolve, SEED, CANDIDATES, INTERVAL_HOURS } from './promotion.mjs'; console.log(`seed ${SEED} | ${CANDIDATES} candidates, 8 dependencies, three environments, production build runs ${INTERVAL_HOURS} h after staging`); console.log('scheme digest version env-uniq tested source scale wrongly pipeline avg peak'); console.log(' match drift escape escape escape escape held min wait/h (h)'); for (const [label, rebuild] of [['same artifact', false], ['rebuild each env', true]]) { const s = run(rebuild); console.log(`${label.padEnd(16)} ${`${s.digestSame}/${CANDIDATES}`.padEnd(7)} ${String(s.versionDrift).padEnd(8)} ` + `${String(s.uniqueEscaped).padEnd(9)} ${String(s.testedEscaped).padEnd(8)} ${String(s.sourceEscaped).padEnd(7)} ` + `${String(s.scaleEscaped).padEnd(7)} ${String(s.wronglyHeld).padEnd(8)} ${String(s.min).padEnd(9)} ` + `${s.avgWait.toFixed(2).padEnd(8)} ${s.peak.toFixed(2)}`); } const t = run(false); console.log(`\nmarginal value of the promotion gates (the same source-defect set under both schemes):`); console.log(` caught only at the dev gate: ${t.devMarginal}`); console.log(` caught only at the staging gate: ${t.stagingMarginal}`); console.log(` the dev gate's bill: ${CANDIDATES * 18} min of gate time + ${t.wait1.toFixed(1)} h of window waiting`); const rnd = generator(SEED), sched = schedule(rnd), cand = candidates(rnd); const c = cand.find((x) => digest(x.no, resolve(sched, x.hour + 6)) !== digest(x.no, resolve(sched, x.hour + 6 + INTERVAL_HOURS))); const staging = resolve(sched, c.hour + 6), prod = resolve(sched, c.hour + 6 + INTERVAL_HOURS); console.log(`\ncandidate ${c.no} (rebuild scheme):`); console.log(` artifact tested in staging: ${digest(c.no, staging)}`); console.log(` artifact that reached production: ${digest(c.no, prod)}`); console.log(` drifted dependency: ${staging.map((v, i) => v.name === prod[i].name ? null : `${v.name} -> ${prod[i].name}`).filter(Boolean).join(', ')}`); ``` ``` seed 20260902 | 40 candidates, 8 dependencies, three environments, production build runs 24 h after staging scheme digest version env-uniq tested source scale wrongly pipeline avg peak match drift escape escape escape escape held min wait/h (h) same artifact 40/40 0 0 5 3 4 5 2320 3.93 6.70 rebuild each env 9/40 56 2 4 3 4 5 4080 4.19 6.97 marginal value of the promotion gates (the same source-defect set under both schemes): caught only at the dev gate: 1 caught only at the staging gate: 7 the dev gate's bill: 720 min of gate time + 59.0 h of window waiting candidate 2 (rebuild scheme): artifact tested in staging: 7f8036842703 artifact that reached production: 8c914f3974ca drifted dependency: d6v1 -> d6v2 ``` The first column is this lesson's core. In the same-artifact scheme, in all forty of the forty candidates, the object in production is the object tested in staging. In the rebuild scheme this number drops to nine: **in thirty-one candidates, production ran an artifact no gate had ever seen**. Drifted dependency versions number fifty-six — across the forty candidates, fifty-six times a dependency in the artifact that reached production differed from the one in the artifact tested in staging. The damage from this drift shows up in the third column: two environment-unique defects. The number looks small, but its nature is different. The four tested-and-escaped defects are defects the gate **looked at and could not see**; they fall as the gate's catch rate rises. The two environment-unique defects, by contrast, are defects the gate **never looked at** — that version never ran in any environment. Raising the gate's catch rate to one hundred percent does not zero out either of these two. The source-defect and scale-defect columns are the same under both schemes: three and four. This is the expected result, and it is the measurement's consistency check. The promotion scheme neither reduces nor multiplies the source's own defects; it only forges or breaks the link between the tested object and the running object. The block below shows a single candidate. Candidate two's staging-tested artifact digest is `7f8036842703`, the one that reached production is `8c914f3974ca`. The source is the same, the candidate is the same; the only difference is that the sixth dependency resolved to `d6v2` instead of `d6v1`. No one made a decision — twenty-four hours passed, nothing more. #### What Drift Is a Function Of ```js // run2.mjs — the interval between the staging build and the production build is scanned import { run, CANDIDATES } from './promotion.mjs'; console.log('interval digest version env-uniq tested pipeline avg'); console.log('(h) match drift escape escape min wait/h'); for (const a of [2, 6, 12, 24, 48, 96]) { const s = run(true, a); console.log(`${String(a).padEnd(9)} ${`${s.digestSame}/${CANDIDATES}`.padEnd(7)} ${String(s.versionDrift).padEnd(8)} ` + `${String(s.uniqueEscaped).padEnd(9)} ${String(s.testedEscaped).padEnd(8)} ${String(s.min).padEnd(9)} ${s.avgWait.toFixed(2)}`); } console.log('\nin the same-artifact scheme, every one of these rows reads: 40/40 digest match, 0 version drift, 0 env-unique escape'); ``` ``` interval digest version env-uniq tested pipeline avg (h) match drift escape escape min wait/h 2 36/40 4 0 5 4080 4.19 6 24/40 18 0 4 4080 4.19 12 17/40 31 1 4 4080 4.19 24 9/40 56 2 4 4080 4.19 48 0/40 104 4 4 4080 4.19 96 0/40 192 8 1 4080 4.19 in the same-artifact scheme, every one of these rows reads: 40/40 digest match, 0 version drift, 0 env-unique escape ``` The scan says what drift is a function of: **elapsed time**. Even at a two-hour interval, four of the forty candidates lose their identity. At forty-eight hours, no candidate keeps its identity. No one changes a version, no one touches the configuration; resolution happens twice, and the world moves on in between. The last row shows a quiet shift. At the ninety-six-hour interval, tested-and-escaped defects drop from five to one — not because the gate is working better, but because those defects have moved into the **untested** column instead. A drop in what a gate misses is not, by itself, good news; the set the gate is looking at may have shrunk instead. The pipeline-minutes and wait columns stay constant across the scan: 4,080 minutes, and four hours eleven minutes per candidate. Shortening the interval buys neither machine time nor wait time. In the rebuild scheme, the only way to reduce drift is to bring the two builds closer together, and once that closing reaches zero, what is left is a single build. So the scan shows there is no middle ground between the two schemes: however small the interval gets, the probability of drift never reaches zero — the only point where it does is the point where the second build never happens at all. #### The Gate's Three Numbers, and a Removable Gate **What it catches:** across the forty-candidate sequence, the two promotion gates together caught ten of thirteen source defects and most of the dependency defects in the tested artifact. **What only it catches:** the staging gate seven, the dev gate one. **What it misses:** three source defects, four scale defects, plus two environment-unique defects in the rebuild scheme. **What it wrongly holds:** five clean candidates. **The time it adds:** an average of three hours fifty-six minutes of wait per candidate, a peak of six hours forty-two minutes; pipeline time of 2,320 minutes, 4,080 in the rebuild scheme. The dev gate's arithmetic writes itself: across forty candidates, it caught only one defect on its own, and in exchange asked for 720 minutes of gate time and fifty-nine hours of window waiting. The staging gate delivers seven defects for the same span of time. With these numbers, removing the dev gate is a defensible decision: in exchange for one defect being caught later, across forty candidates, every candidate advances one window earlier. What decides this is not an opinion about the gate, but its marginal value. The same arithmetic gives a different result for the rebuild scheme. There, the staging gate's seven-defect marginal value holds **for the object it tested**; in thirty-one candidates, the object that reaches production is a different one. The gate works, and its numbers look good — but its evidence does not travel with the object. What needs removing is not the gate, but the second build that invalidates the gate's evidence. #### Summary - In the same-artifact scheme, in all forty of the forty candidates production ran the object tested in staging; in the rebuild scheme this number was nine, and drifted dependency versions numbered fifty-six. - Drift produced two environment-unique defects: a defect class the gate never looks at, and one that will not shrink just because the catch rate rises. - Source defects three, scale defects four: the same under both schemes. The promotion scheme does not change the source's own defects; it only forges or breaks the link between the tested object and the running object. - Drift is a function of time: at a two-hour interval, four of the forty candidates lost their identity; at forty-eight hours, all of them did. At ninety-six hours, the drop in escaped defects is not the gate improving — it is the set it looks at shrinking. - The dev promotion gate caught only one defect on its own across forty candidates, and asked for 720 minutes plus fifty-nine hours of window waiting; with these numbers, removing it is defensible. - The promotion gate's waiting bill runs an average of three hours fifty-six minutes per candidate, with a peak of six hours forty-two minutes; the rebuild scheme adds two extra builds per candidate on top of that. #### Next Step In this measurement, promotion to production was a single decision: the artifact either passed or it did not, and when it passed, it opened to everyone at once. But promotion does not have to happen in a single step. A new version can first open to a small percentage of requests, be watched for a while, and then have that percentage raised. This turns the gate from a single decision into a **sequence of decisions**, and it opens a new question: what metric drives the decision to raise the percentage, over what window is that metric collected, and what is its threshold. The next lesson scans these three settings, and also counts how often a clean version gets stopped while the metric is noisy. ### Staged Rollout Source: https://academia.sh/en/courses/ci-cd/staged-rollout The decision side of percentage rollout is measured: the metric that raises the percentage, the metric window, and the decision threshold are scanned. Across three ladders, the stop rate for pronounced and faint defects, touched requests, the wrongful rollback of a clean version, and total rollout duration are counted. The promotion gate was a single decision: the artifact either passed or it did not, and when it passed, it opened to everyone at once. Promotion does not have to happen in a single step. A new version can first open to a small percentage of requests, be watched for a while, and then have its share raised. This turns the gate from a single decision into a **sequence of decisions**. The rollout mechanics themselves — how the share shifts, how many replicas sit on which version, how many requests an escaped defect touches — are an already-measured subject and are not repeated here; they are this lesson's input. This lesson's question sits one step further back: **who raises the share, and on what evidence.** At the end of every step, a metric is read, compared against a threshold, and one of two decisions is made: raise the percentage, or roll back. That is the gate. What gets measured is this gate's three numbers — which defect it catches, which clean version it wrongfully stops, and how many minutes it adds to the rollout. - **RS13.** The fictional regional measurement network's billing service receives five hundred requests per minute. Time-of-day variation, regional difference, and request mix are not modeled. Both the network and the figures are fictional. - **RS14.** The broken version has passed the promotion gates; its defect is visible only in production traffic. Two severities are measured: a **pronounced** defect errors on 1.6% of requests, a **faint** defect on 0.8%. The baseline error rate is 0.4%. - **RS15.** The metric is a single item: the new version's error rate. Latency, saturation, and business metrics, and a multi-metric decision, are outside the model. - **RS16.** The window is read once, at the end of the step. A sliding window, a consecutive-violation counter, and metric-aggregation lag are not modeled. - **RS17.** Rollback is instant and free. The round cost of rollback is not this lesson's subject. - **RS18.** Every setting is run two hundred times; the rates are the share of these two hundred runs. The seed is visible. #### Measurement Setup What is measured is not a real rollout but a model set up with `node`. The model climbs a ladder step by step, generates the requests falling in the window one by one at each step, and makes the decision at the end of the step. The random generator is written by hand, and its seed is visible. ```js // rollout.mjs — model of the decision side of a staged rollout; not real traffic, a model export const SEED = 20260903; export const REQ_MIN = 500, BASE = 0.004, PRONOUNCED = 0.016, FAINT = 0.008; export const WINDOW = 10, MULTIPLIER = 2.5, REPEAT = 200; export function generator(seed) { // linear congruential generator, seed visible let x = seed >>> 0; return () => { x = (Math.imul(x, 1664525) + 1013904223) >>> 0; return x / 4294967296; }; } export const ladders = { // percentage steps fast: [5, 50, 100], balanced: [1, 5, 25, 50, 100], careful: [1, 2, 5, 10, 25, 50, 100], }; // One rollout: the ladder climbs step by step, the metric window is read at the end of each step. // The decision gate is here: if the error rate seen in the window exceeds the threshold, roll back; if not, raise the percentage. export function attempt(rnd, ladder, window, threshold, rate) { let touched = 0; for (let i = 0; i < ladder.length; i += 1) { const n = Math.round(REQ_MIN * window * ladder[i] / 100); let h = 0; for (let r = 0; r < n; r += 1) if (rnd() < rate) h += 1; touched += n; if (n > 0 && h / n > threshold) return { decision: 'roll back', step: i + 1, touched }; } return { decision: 'ok', step: ladder.length, touched }; } export function run(ladder, rate, window = WINDOW, multiplier = MULTIPLIER) { const threshold = BASE * multiplier, rnd = generator(SEED); let stopped = 0, step = 0, touched = 0; const distribution = ladder.map(() => 0); for (let k = 0; k < REPEAT; k += 1) { const s = attempt(rnd, ladder, window, threshold, rate); if (s.decision === 'roll back') { stopped += 1; step += s.step; distribution[s.step - 1] += 1; } touched += s.touched; } return { distribution, rate: stopped / REPEAT, avgStep: stopped ? step / stopped : 0, avgTouched: Math.round(touched / REPEAT), duration: ladder.length * window, threshold }; } ``` The same `run` is called for three separate versions. For the pronounced- and faint-defect versions, the resulting `rate` is what the gate **catches**; called with the baseline rate, that same number is what the gate **wrongly holds** — because that version has no defect. #### Three Ladders ```js // run1.mjs — three ladders, three versions: pronounced-defect, faint-defect, clean import { run, ladders, SEED, REQ_MIN, BASE, PRONOUNCED, FAINT, WINDOW, MULTIPLIER, REPEAT } from './rollout.mjs'; console.log(`seed ${SEED} | ${REQ_MIN} req/min, window ${WINDOW} min, ${REPEAT} repeats`); console.log(`baseline error rate ${BASE * 100}% | pronounced defect ${PRONOUNCED * 100}% | faint defect ${FAINT * 100}%`); console.log(`decision threshold: ${MULTIPLIER} times baseline, i.e. ${BASE * MULTIPLIER * 100}%\n`); console.log('ladder percentage steps pronounced defect faint defect clean'); console.log(' stopped step touched stopped touched wrongly stopped'); for (const [label, m] of Object.entries(ladders)) { const b = run(m, PRONOUNCED), s = run(m, FAINT), t = run(m, BASE); console.log(`${label.padEnd(10)} ${m.join('-').padEnd(23)} ${`${(b.rate * 100).toFixed(0)}%`.padEnd(8)} ` + `${b.avgStep.toFixed(2).padEnd(7)} ${String(b.avgTouched).padEnd(10)} ${`${(s.rate * 100).toFixed(0)}%`.padEnd(8)} ` + `${String(s.avgTouched).padEnd(11)} ${`${(t.rate * 100).toFixed(0)}%`.padEnd(9)} ${b.duration} min`); } console.log('\nrequests falling in the metric window per step, and the smallest observable non-zero rate:'); for (const p of [1, 2, 5, 10, 25, 50, 100]) { const n = REQ_MIN * WINDOW * p / 100; console.log(` ${String(p).padStart(3)}% share: ${String(n).padStart(5)} req/window, one failure = ${(100 / n).toFixed(2)}%` + ` -> threshold ${(BASE * MULTIPLIER * 100).toFixed(1)}% ${100 / n > BASE * MULTIPLIER * 100 ? 'indistinguishable' : 'distinguishable'}`); } ``` ``` seed 20260903 | 500 req/min, window 10 min, 200 repeats baseline error rate 0.4% | pronounced defect 1.6% | faint defect 0.8% decision threshold: 2.5 times baseline, i.e. 1% ladder percentage steps pronounced defect faint defect clean stopped step touched stopped touched wrongly stopped fast 5-50-100 100% 1.23 825 40% 5375 9% 30 min balanced 1-5-25-50-100 100% 1.60 306 70% 3423 27% 50 min careful 1-2-5-10-25-50-100 100% 1.72 188 82% 2403 32% 70 min requests falling in the metric window per step, and the smallest observable non-zero rate: 1% share: 50 req/window, one failure = 2.00% -> threshold 1.0% indistinguishable 2% share: 100 req/window, one failure = 1.00% -> threshold 1.0% distinguishable 5% share: 250 req/window, one failure = 0.40% -> threshold 1.0% distinguishable 10% share: 500 req/window, one failure = 0.20% -> threshold 1.0% distinguishable 25% share: 1250 req/window, one failure = 0.08% -> threshold 1.0% distinguishable 50% share: 2500 req/window, one failure = 0.04% -> threshold 1.0% distinguishable 100% share: 5000 req/window, one failure = 0.02% -> threshold 1.0% distinguishable ``` On the pronounced defect, all three ladders scored perfectly: the version was stopped in all two hundred of the two hundred runs. Where they diverge is touched-request count. The fast ladder opened its first step at 5%, so it touched 825 requests; the careful ladder started at 1% and stayed at 188. The cost is duration: thirty minutes against seventy. The faint-defect column turns the table around. A version with a 0.8% error rate sits below the threshold; the gate catches it only when noise happens to push it over. The fast ladder stopped this version in four of ten runs, the careful ladder in eight. But when it fails to stop it, the touched-request count is 5,375 for the fast ladder and 2,403 for the careful one. On a defect the gate misses, a fine-grained ladder still pays off — because even at the moment of missing it, the share stays small. The last column writes the bill. The careful ladder stopped a clean version in three of ten runs. Thirty-two percent. Why is answered by the table below: a 1% share, in a ten-minute window, gets fifty requests. The smallest observable non-zero error rate at fifty requests is 2% — twice the decision threshold. At that step, a single baseline failure alone triggers the gate. There, the gate is not measuring the version; it is measuring noise. #### Window and Threshold ```js // run2.mjs — on the balanced ladder, the metric window and the decision threshold are scanned separately import { run, ladders, BASE, PRONOUNCED, FAINT, WINDOW, MULTIPLIER } from './rollout.mjs'; const m = ladders.balanced; console.log(`ladder ${m.join('-')} | threshold multiplier ${MULTIPLIER} fixed`); console.log('window pronounced stopped faint stopped clean wrongly stopped touched rollout duration'); for (const p of [5, 10, 20, 40, 80]) { const b = run(m, PRONOUNCED, p), s = run(m, FAINT, p), t = run(m, BASE, p); console.log(`${`${p} min`.padEnd(9)} ${`${(b.rate * 100).toFixed(0)}%`.padEnd(21)} ${`${(s.rate * 100).toFixed(0)}%`.padEnd(15)} ` + `${`${(t.rate * 100).toFixed(0)}%`.padEnd(21)} ${String(b.avgTouched).padEnd(9)} ${b.duration} min`); } console.log(`\nwindow ${WINDOW} min fixed | threshold = baseline x multiplier`); console.log('multiplier threshold pronounced stopped faint stopped clean wrongly stopped touched'); for (const c of [1.5, 2, 2.5, 3, 4, 6]) { const b = run(m, PRONOUNCED, WINDOW, c), s = run(m, FAINT, WINDOW, c), t = run(m, BASE, WINDOW, c); console.log(`${String(c).padEnd(11)} ${`${(BASE * c * 100).toFixed(1)}%`.padEnd(10)} ${`${(b.rate * 100).toFixed(0)}%`.padEnd(21)} ` + `${`${(s.rate * 100).toFixed(0)}%`.padEnd(15)} ${`${(t.rate * 100).toFixed(0)}%`.padEnd(21)} ${b.avgTouched}`); } const t2 = run(m, BASE), b2 = run(m, PRONOUNCED); console.log(`\nwindow ${WINDOW} min, threshold multiplier ${MULTIPLIER}: which step the stop happened at`); console.log(` percentage step: ${m.map((x) => `${x}%`.padStart(6)).join('')}`); console.log(` clean version stopped: ${t2.distribution.map((x) => String(x).padStart(6)).join('')}`); console.log(` pronounced defect stopped:${b2.distribution.map((x) => String(x).padStart(6)).join('')}`); ``` ``` ladder 1-5-25-50-100 | threshold multiplier 2.5 fixed window pronounced stopped faint stopped clean wrongly stopped touched rollout duration 5 min 100% 64% 20% 341 25 min 10 min 100% 70% 27% 306 50 min 20 min 100% 48% 9% 563 100 min 40 min 100% 40% 5% 770 200 min 80 min 100% 34% 2% 890 400 min window 10 min fixed | threshold = baseline x multiplier multiplier threshold pronounced stopped faint stopped clean wrongly stopped touched 1.5 0.6% 100% 100% 53% 236 2 0.8% 100% 91% 26% 306 2.5 1.0% 100% 70% 27% 306 3 1.2% 100% 45% 23% 565 4 1.6% 94% 37% 19% 1373 6 2.4% 31% 7% 2% 6333 window 10 min, threshold multiplier 2.5: which step the stop happened at percentage step: 1% 5% 25% 50% 100% clean version stopped: 37 16 0 0 0 pronounced defect stopped: 100 79 21 0 0 ``` The window scan does not give a straight curve. Going from a five-minute to a ten-minute window, wrongful stopping **rises** from 20% to 27%, then falls. The reason is in the decision rule: at a 1% share, a smaller window brings fewer requests, but a single failure still crosses the threshold. Growing the window first raises the odds of "seeing one failure"; but once the window grows large enough to require two failures, the curve turns. At twenty minutes, wrongful stopping drops to 9%, at eighty minutes to 2% — in exchange, rollout duration climbs from one hundred minutes to four hundred, and touched requests rise from 563 to 890. The price of waiting is the requests touched while waiting. The threshold scan shows a clean trade-off. Dropping the multiplier to 1.5 catches all of the faint defect, but stops more than half of clean versions. Raising it to 6 drops wrongful stopping to 2%; but this time even the pronounced defect is caught only a third of the time, and touched requests jump to 6,333. The rows between 2 and 2.5 stand out: wrongful stopping stays almost the same while faint-defect catch rate rises from 70% to 91%. Pulling the threshold from 2.5 down to 2 is a gain with no offsetting cost. The reason is in the last block. Of the clean version's fifty-three wrongful stops, thirty-seven happened at the first step, sixteen at the second; past the third step, not one. **Small-share steps**, not the threshold, produce the wrongful stops. Moving the threshold does not fix those steps, because the number measured there already jumps around independently of the threshold. The pronounced-defect row concentrates in the same place: a hundred runs stopped at the first step, seventy-nine at the second. So the first two steps produce nearly all of both the decisions and the mistakes; the remaining three steps only add duration. #### The Gate's Three Numbers **What it catches:** the pronounced-defect version was stopped in all two hundred of the two hundred runs. **What only it catches:** every one of these defects — by definition, the broken version has already passed the promotion gates, and its defect is visible only in production traffic. The percentage-raising gate is the first gate to turn real traffic into a metric; its marginal value is therefore one hundred percent. **What it misses:** 18% to 60% of the faint defect depending on the ladder; even 69% of the pronounced defect once the threshold is pulled to 6. **What it wrongly holds:** 27% of clean versions on the balanced ladder, 32% on the careful ladder; that drops to 2% at an eighty-minute window. **The time it adds:** thirty, fifty, and seventy minutes per ladder; up to four hundred minutes once the window is widened. None of these three numbers can be read off the pattern's name. The sentence "we did a canary" is empty until it says which metric the gate watches, over what window, at what threshold — and the worst of these three settings is trying to make the decision at a share too small for the decision to be made at all. #### Summary - Three ladders were run on the same defect severities: all three stopped the pronounced defect across two hundred runs, but touched requests dropped from 825 to 188, and duration rose from thirty minutes to seventy. - For the faint defect, the stop rate ranged from 40% to 82%; even when the gate missed it, the fine-grained ladder cut touched requests from 5,375 to 2,403. - Wrongful rollback was measured: 27% on the balanced ladder, 32% on the careful one. All fifty-three wrongful stops happened in the first two steps; past the third, not one. - The window scan is not one-directional: going from five to ten minutes raised wrongful stopping, past twenty it fell; at eighty minutes it reached 2%, at the cost of a four-hundred-minute rollout. - In the threshold scan, a multiplier of 2 instead of 2.5 raised faint-defect catch rate from 70% to 91% without raising wrongful stopping; raising the threshold further grew missed defects fast. - A 1% share, in a ten-minute window, gets fifty requests; a single failure reads as 2%, the threshold is 1%. At that step the gate measures noise, not the version — a small share limits blast radius, it does not deliver a decision. #### Next Step This entire measurement leaned on one assumption: rollback is instant and free. The moment the gate said "roll back," nothing happened in the model. In reality, rollback has a round, a cost, and a limit. If the old version is still standing, rollback is a matter of flipping a single switch; if it is not, rollback means a fresh release. The next lesson measures the scheme that keeps two full environments standing side by side: rollback's round cost, the resource bill of keeping two environments up continuously, the state-carrying requests cut off at the moment of the switch, and — when the data layer is shared — what rollback actually fails to undo. ### Two-Environment Deployment Source: https://academia.sh/en/courses/ci-cd/two-environment-deployment Rollback's round cost and the standing bill of keeping two full environments up are counted together: the rollback window is scanned, sessions and long-running jobs cut off at the moment of cutover are measured, and the rows that rollback fails to undo in the shared data layer are shown. The previous measurement leaned on one assumption: rollback is instant and free. The moment the gate said "roll back," nothing happened in the model. This lesson lifts that assumption. Rollback has a round, a price, and a limit — and most of that price gets paid even when rollback **never turns out to be needed**. Two-environment deployment's promise fits one sentence: the new version stands up beside the old one, traffic switches over with a single flip, and if something goes wrong, the switch flips back. What that promise costs is that two full environments doing the same work stand together for a while. In this lesson the gate is the **rollback decision**: how many rounds the decision takes, under what condition it drops to a single round, and what that single round fails to cover are measured. - **RS19.** The fictional regional measurement network's work-order service runs with sixteen copies; sixty releases are measured, and one deployment round is three minutes. Both the network and the figures are fictional. - **RS20.** Thirty percent of releases are defective. The defect's time-to-appear is exponentially distributed, with a mean of twenty-five minutes. Which metric surfaces the defect is a separate measure's subject; only the moment of appearance is modeled here. - **RS21.** The second environment stands up twelve minutes before the switch flips, and stays up for the rollback window. Once the window closes, rollback means a fresh release: five rounds. - **RS22.** The staged scheme climbs five steps in fifty minutes with four extra copies; rollback takes as many rounds as the steps climbed by then. - **RS23.** At the moment of cutover there are three classes of work unit, and their durations are exponentially distributed. Session affinity carrying over to the new environment is not modeled; what gets measured is how many are cut off when it does not carry over. - **RS24.** The data layer is shared between the two environments. The write rate is 140 rows per minute; 35% of rows are updated after rollback. Migration ordering and backward-compatible writes are not this lesson's subject. #### Measurement Setup What is measured is not a real cluster but a model set up with `node`. The model carries three separate things: a release sequence's resource cost and rollback rounds, the work units in flight at the moment the switch flips, and the rows written to the shared data layer. The random generator is written by hand, and its seed is visible. ```js // twoenv.mjs — cost and cutover model of two-environment deployment; not a real cluster, a model export const SEED = 20260904; export const COPIES = 16, RELEASES = 60, ROUND_MIN = 3, PREP_MIN = 12, REDEPLOY_ROUNDS = 5; export const STAGED_EXTRA = 4, STAGED_DURATION = 50, STAGED_STEPS = 5; export const DEFECT_MEAN = 25, WINDOW_MIN = 30; // the defect's time-to-appear is exponentially distributed export function generator(seed) { // linear congruential generator, seed visible let x = seed >>> 0; return () => { x = (Math.imul(x, 1664525) + 1013904223) >>> 0; return x / 4294967296; }; } export const exponential = (rnd, mean) => -mean * Math.log(1 - rnd()); // Release sequence: whether each release is defective, and in which minute the defect appears. export function releases(rnd, brokenRate = 0.30) { return Array.from({ length: RELEASES }, (_, i) => ({ no: i + 1, broken: rnd() < brokenRate, appear: exponential(rnd, DEFECT_MEAN), })); } // Two-environment scheme: the old environment is kept up for the window after the switch is flipped. export function twoEnv(list, window) { let replicaMin = 0, round = 0, inWindow = 0, outOfWindow = 0; for (const y of list) { replicaMin += COPIES * (PREP_MIN + window); // how long the second environment stays up if (!y.broken) continue; if (y.appear <= window) { round += 1; inWindow += 1; } else { round += REDEPLOY_ROUNDS; outOfWindow += 1; } } const broken = list.filter((y) => y.broken).length; return { name: `two-environment (window ${window} min)`, replicaMin, inWindow, outOfWindow, avgRound: broken ? round / broken : 0, rollbackMin: broken ? round * ROUND_MIN / broken : 0 }; } // Staged scheme: the extra copies stay up for the rollout's duration; rollback takes as many rounds as were climbed by then. export function staged(list) { let replicaMin = 0, round = 0; for (const y of list) { replicaMin += STAGED_EXTRA * STAGED_DURATION; if (!y.broken) continue; const step = Math.min(STAGED_STEPS, 1 + Math.floor(y.appear / (STAGED_DURATION / STAGED_STEPS))); round += step; } const broken = list.filter((y) => y.broken).length; return { name: 'staged (single environment)', replicaMin, inWindow: 0, outOfWindow: broken, avgRound: broken ? round / broken : 0, rollbackMin: broken ? round * ROUND_MIN / broken : 0 }; } // The moment of cutover: units in flight when the switch is flipped. Three classes, exponential duration. export const CLASSES = [ { name: 'short request', rate: 300, mean: 0.04 }, // rate: units starting per minute { name: 'session', rate: 8, mean: 12 }, { name: 'long-running job', rate: 0.5, mean: 25 }, ]; export function inFlight(rnd, cls, T = 240) { // T minutes of warm-up, switch flips at T const open = []; for (let min = 0; min < T; min += 1) { for (let k = 0; k < cls.rate; k += 1) { const start = min + rnd(), duration = exponential(rnd, cls.mean); if (start + duration > T) open.push({ start, end: start + duration }); } } return open; } // Shared data layer: the new version writes in the new format, the rollback code reverts, it does not revert rows. export function dataLayer(rnd, openMin, writeRate = 140, updateRate = 0.35) { let newFormat = 0, laterUpdated = 0; for (let min = 0; min < openMin; min += 1) { for (let k = 0; k < writeRate; k += 1) { newFormat += 1; if (rnd() < updateRate) laterUpdated += 1; // updated by the old version after rollback } } return { newFormat, lost: laterUpdated, silent: newFormat - laterUpdated }; } ``` The model's setting is `window`: how many minutes the old environment stays up after the switch flips. This single number sets both rollback's round cost and the scheme's standing bill — and it sets them in opposite directions. #### Resource Cost and Rollback Round ```js // run1.mjs — the resource cost and rollback round cost of two schemes; the rollback window is scanned import { generator, releases, twoEnv, staged, SEED, COPIES, RELEASES, ROUND_MIN, WINDOW_MIN, DEFECT_MEAN } from './twoenv.mjs'; const l = releases(generator(SEED)); const broken = l.filter((y) => y.broken).length; console.log(`seed ${SEED} | ${RELEASES} releases, ${COPIES} copies, ${broken} releases defective`); console.log(`the defect's time-to-appear is exponential, mean ${DEFECT_MEAN} min | one round ${ROUND_MIN} min\n`); console.log('scheme replica-min in window out of window avg round avg rollback'); for (const s of [twoEnv(l, WINDOW_MIN), staged(l)]) { console.log(`${s.name.padEnd(34)} ${String(s.replicaMin).padEnd(12)} ${String(s.inWindow).padEnd(10)} ` + `${String(s.outOfWindow).padEnd(14)} ${s.avgRound.toFixed(2).padEnd(10)} ${s.rollbackMin.toFixed(1)} min`); } console.log('\nthe rollback window is scanned (two-environment scheme):'); console.log('window replica-min single switch redeploy avg round avg rollback extra replica-min'); for (const p of [0, 5, 15, 30, 60, 120, 240]) { const s = twoEnv(l, p), a = staged(l); console.log(`${`${p} min`.padEnd(8)} ${String(s.replicaMin).padEnd(12)} ${String(s.inWindow).padEnd(14)} ` + `${String(s.outOfWindow).padEnd(9)} ${s.avgRound.toFixed(2).padEnd(10)} ${`${s.rollbackMin.toFixed(1)} min`.padEnd(14)} ${s.replicaMin - a.replicaMin}`); } ``` ``` seed 20260904 | 60 releases, 16 copies, 18 releases defective the defect's time-to-appear is exponential, mean 25 min | one round 3 min scheme replica-min in window out of window avg round avg rollback two-environment (window 30 min) 40320 11 7 2.56 7.7 min staged (single environment) 12000 0 18 2.78 8.3 min the rollback window is scanned (two-environment scheme): window replica-min single switch redeploy avg round avg rollback extra replica-min 0 min 11520 0 18 5.00 15.0 min -480 5 min 16320 3 15 4.33 13.0 min 4320 15 min 25920 8 10 3.22 9.7 min 13920 30 min 40320 11 7 2.56 7.7 min 28320 60 min 69120 17 1 1.22 3.7 min 57120 120 min 126720 18 0 1.00 3.0 min 114720 240 min 241920 18 0 1.00 3.0 min 229920 ``` The top table puts a limit on the promise. At a thirty-minute window, only eleven of the eighteen defective releases were rolled back with a single switch; seven were noticed after the window closed and needed a five-round redeploy. The average rollback round is 2.56, and 2.78 in the staged scheme. The difference between the two schemes at this setting is eight percent. In exchange, the two-environment scheme burned three times the resources: 40,320 replica-minutes against 12,000. The phrase "instant rollback" becomes a number once the window is kept longer than the defect's time-to-appear. The scan shows the price. At a sixty-minute window, seventeen of the eighteen defects were rolled back in a single round, and the average round dropped to 1.22 — an extra 57,120 replica-minutes. At one hundred twenty minutes, every one of them dropped to a single round, at an extra cost of 114,720 replica-minutes. Every round gained carries a price: at the thirty-minute window, forty-four rounds were shaved off, and that cost 655 replica-minutes per round. The first row says the same thing from the other direction. At a zero-minute window, the two-environment scheme is **cheaper** than the staged scheme (11,520 against 12,000), because the second environment stands up only for the twelve-minute prep. But in that scheme, not one release could be rolled back with a single switch; the average round climbed to five. Shutting down the old environment shuts down two-environment deployment's one reason to exist. #### Work Cut Off at the Moment of Cutover ```js // run2.mjs — state-carrying requests cut off at the moment of cutover, and the shared data layer import { generator, inFlight, dataLayer, CLASSES, SEED, COPIES } from './twoenv.mjs'; const open = CLASSES.map((s) => ({ s, list: inFlight(generator(SEED + s.rate), s) })); console.log('units in flight at the moment the switch is flipped:'); for (const { s, list } of open) console.log(` ${s.name.padEnd(18)} ${String(list.length).padStart(5)} units (average duration ${s.mean} min)`); console.log('\nthe drain duration is scanned: a unit still running at the end of the duration gets cut'); console.log('drain short request session long-running job total cut two environments together'); for (const d of [0, 0.5, 1, 5, 15, 30]) { const k = open.map(({ list }) => list.filter((u) => u.end > 240 + d).length); console.log(`${`${d} min`.padEnd(9)} ${String(k[0]).padEnd(15)} ${String(k[1]).padEnd(8)} ${String(k[2]).padEnd(17)} ` + `${String(k[0] + k[1] + k[2]).padEnd(10)} ${2 * COPIES * d} replica-min`); } const v = dataLayer(generator(SEED + 11), 11); console.log('\nshared data layer: the switch was flipped, rollback happened at minute 11'); console.log(` rows written in the new format: ${v.newFormat}`); console.log(` rollback round: 1 (code) | rows reverted for data: 0`); console.log(` rows updated by the old version after rollback, dropping the new field: ${v.lost}`); console.log(` rows still carrying the new field, invisible to the old version: ${v.silent}`); ``` ``` units in flight at the moment the switch is flipped: short request 10 units (average duration 0.04 min) session 89 units (average duration 12 min) long-running job 24 units (average duration 25 min) the drain duration is scanned: a unit still running at the end of the duration gets cut drain short request session long-running job total cut two environments together 0 min 10 89 24 123 0 replica-min 0.5 min 0 88 24 112 16 replica-min 1 min 0 86 24 110 32 replica-min 5 min 0 68 20 88 160 replica-min 15 min 0 31 14 45 480 replica-min 30 min 0 7 5 12 960 replica-min shared data layer: the switch was flipped, rollback happened at minute 11 rows written in the new format: 1540 rollback round: 1 (code) | rows reverted for data: 0 rows updated by the old version after rollback, dropping the new field: 491 rows still carrying the new field, invisible to the old version: 1049 ``` At the moment the switch flipped, 123 work units were in flight. Ten of them were short requests, eighty-nine sessions, twenty-four long-running jobs. A half-minute drain saved every short request — and only one session. A one-minute drain brought the eighty-nine down to eighty-six, and touched none of the long-running jobs. The reason is a difference in scale. The drain duration is tuned to the short request's lifetime; a session lives twelve minutes, a long-running job twenty-five — two to three orders of magnitude longer. Bringing 123 units down to twelve needs a thirty-minute drain, and for those thirty minutes, two versions write to the same data layer together. The cost of a seamless cutover is not just 960 replica-minutes; it is a half hour during which two versions write at once. One more calculation is needed. If every cutover cuts off eighty-nine sessions, then **wrongfully rolling back a clean version** means two cutovers: 178 sessions and forty-eight long-running jobs. The cost of a wrongful rollback can outweigh the cost of the defect itself — and how often a wrongful rollback happens depends not on the rollback decision, but on the setting of the gate that came before it. #### What Rollback Does Not Undo The last block shows the single switch's limit. The switch flipped, the defect was seen at minute eleven, rollback completed in one round. The code returned to the old version. In those eleven minutes, 1,540 rows had been written to the shared data layer in the new format; rollback undid **zero** of them. There are two separate kinds of damage. 491 rows were updated by the old version after rollback; because the old version does not recognize the new field, that field dropped during the write and the information was lost for good. The remaining 1,049 rows keep carrying their new field; the old version does not see them, so they stay silent — but once the new version comes back, there will be a format split between these two sets. Two-environment deployment reduces rolling back **the code** to a single round. When the data layer is shared, rollback's scope is limited to code, and that limit never shows up in the round cost: a one-round rollback can leave behind a 1,540-row residue. #### The Gate's Three Numbers **What it catches:** all eighteen defective releases were eventually rolled back. **What only it catches:** eleven releases rolled back in a single round at the thirty-minute window — possible only while the old environment is still standing — for a total of forty-four rounds saved, or 132 minutes. The marginal value's price is 655 replica-minutes per round. **What it misses:** the seven releases noticed after the window closed, and the rows written to the data layer during every rollback — 1,540 rows in the example release, 491 of them a permanent loss. **What it wrongly holds:** rolling back a clean version means two cutovers, 178 sessions, and forty-eight long-running jobs. **The time it adds:** twelve minutes of prep per release plus the window duration; up to thirty minutes of drain for a seamless cutover; 40,320 replica-minutes in total. #### Summary - At a thirty-minute window, eleven of the eighteen defective releases were rolled back with a single switch, seven with a five-round redeploy; the average round is 2.56, and 2.78 in the staged scheme. - The resource cost is threefold: 40,320 replica-minutes against 12,000. Raising the window to one hundred twenty minutes drops every release to a single round, at an extra cost of 114,720 replica-minutes. - At a zero-minute window, the two-environment scheme comes out cheaper than the staged one, but no release remains rollback-able with a single switch: shutting down the old environment shuts down the pattern's reason to exist. - At the moment of cutover, 123 work units were in flight. A half-minute drain saved every short request, and only one of eighty-nine sessions; getting down to twelve means a thirty-minute drain and two versions writing together for half an hour. - In the shared data layer, a one-round rollback undid zero of 1,540 rows; 491 of them were updated by the old version and lost for good. - Wrongfully rolling back a clean version means two cutovers, 178 sessions, and forty-eight interrupted long-running jobs; this number depends not on the rollback gate, but on the setting of the gate before it. #### Next Step Every measurement in this lesson rested on a single necessity: turning on new behavior requires putting the new version into traffic, and turning it off requires pulling it out. As long as releasing and enabling behavior are the same operation, rollback is always a deployment operation — it has a round, a cutover moment, and a data residue. These two can be separated. If behavior is carried in the code and kept off with a record, turning it on and off is no longer a deployment but a setting change; rollback's round cost approaches zero. The next lesson measures this separation: the rounds it saves, the enabling decision's own gate, and the debt every record left off accumulates over time. ### Feature Flags Source: https://academia.sh/en/courses/ci-cd/feature-flags The same change sequence is released under two schemes, with and without flags; whether the rollback unit is code or behavior is measured in rounds and affected invoices. The eight code paths opened by three flags are counted, the tested-path share is derived, and a period's flag debt is written down together with its dead code paths. In the previous lesson's two-environment scheme, rollback dropped to a single switch: traffic flips to the old environment, and the previous behavior returns within minutes. What that switch rolled back was **the entire code**. A release carries a version; every change inside that version ships together and comes back together. But what usually needs rolling back is a single behavior. A feature flag separates this unit: the code deploys to production, the behavior turns on with a separate switch. This lesson's measure is not whether a flag exists, but **which number this separation changes**. The same change sequence is run under both schemes, the rollback unit and round are counted, the code paths a flag opens are counted, and at the end of a period, the flag debt left behind is written down. - **RS25.** The flag value lives outside the pipeline, in a config record the running process reads; changing it does not require a new release. Propagating the setting to every node takes one round. - **RS26.** Flags are independent of one another: any combination is a valid code path. - **RS27.** Drift in production is seen from the verification service's mismatch report; the report is produced once every two deployment rounds. Detection time is the same under both schemes. - **RS28.** In the without-flags scheme, the rollback unit is a version: every change inside that version goes back together. - **RS29.** A flag's read point is where the condition is written in the code; a flag can be read at more than one point, and cleanup cost is proportional to that number. - **RS30.** The fiction is the regional measurement network's billing service: an invoice amount is produced from a meter reading. The service, the ledger, and the figures below are all fictional. #### The Rollback Unit A period of six changes gets released. The third — the late fee — is defective, and the defect is seen only after the following changes have also reached production. In the without-flags scheme, the changes split into three releases of two. In the with-flags scheme, every change releases separately, and the defective one ships dark behind a flag before it is turned on. ```js // scheme.mjs — the same change sequence is released under two schemes; the release scheme is a MODEL const CHANGES = ['rate table', 'tiered rate', 'late fee', 'invoice summary field', 'work order link', 'batch job report']; const DEFECTIVE = 2; // late fee drifts once the tier is on const INVOICE_ROUND = 40; // invoices computed in one deployment round const CONFIG_ROUND = 4; // invoices computed in one config-propagation round const NOTICE = 2; // the mismatch report surfacing the defect: two deployment rounds function withoutFlags() { const release = [[0, 1], [2, 3], [4, 5]]; // three releases, each release a version let version = release.length, round = 0; while (release.slice(0, version).flat().includes(DEFECTIVE)) { version -= 1; round += 1; } const remaining = release.slice(0, version).flat(); const pulled = release.flat().filter((i) => !remaining.includes(i)); return { name: 'without flags', unit: 'code', round, pulled, affected: (NOTICE + round) * INVOICE_ROUND }; } function withFlags() { // every change released separately; the defective one shipped dark behind a flag, then turned on const flag = { 'late fee': 'on' }; let round = 0; flag['late fee'] = 'off'; round += 1; const pulled = Object.entries(flag).filter(([, d]) => d === 'off') .map(([name]) => CHANGES.indexOf(name)); return { name: 'with flags', unit: 'behavior', round, pulled, affected: NOTICE * INVOICE_ROUND + round * CONFIG_ROUND }; } console.log('scheme rollback unit round pulled of these sound affected invoices'); for (const d of [withoutFlags(), withFlags()]) { const sound = d.pulled.filter((i) => i !== DEFECTIVE); console.log(`${d.name.padEnd(15)} ${d.unit.padEnd(14)} ${String(d.round).padEnd(6)} ` + `${String(d.pulled.length).padEnd(7)} ${String(sound.length).padEnd(17)} ${d.affected}`); console.log(` pulled from production: ${d.pulled.map((i) => CHANGES[i]).join(', ')}`); } ``` ``` scheme rollback unit round pulled of these sound affected invoices without flags code 2 4 3 160 pulled from production: late fee, invoice summary field, work order link, batch job report with flags behavior 1 1 0 84 pulled from production: late fee ``` In the without-flags scheme, two rounds of rewinding were needed, because the defective change sat in the middle version and one more version had shipped after it. Four changes were pulled from production; three of them were sound. The owners of those three changes had nothing to do with the defect, and their work got pulled from production for a reason that was not their fault. In the with-flags scheme, the only thing rolled back is a behavior: not one sound change left production. The difference in affected-invoice count comes from how long the rollback round takes. Detection takes the same time under both schemes — eighty invoices. The split happens after that: rewinding a version takes two deployment rounds, turning off a flag takes one config-propagation round. 160 against 84. All of this difference comes from the rollback path getting shorter, not from the defect being seen any earlier. #### The Code Paths a Flag Opens A flag is a condition, and every condition splits the code in two. Three flags open eight code paths. The computation below actually runs: the invoice amount is computed separately for every combination of the three flags, and compared against a separately written expected behavior for that combination. ```js // flag.mjs — the code paths opened by three flags are counted; the billing computation is a MODEL // Regional measurement network fiction: an invoice amount is produced from a meter reading. const BASE = 12.4; const TIER = [[10, 12.4], [25, 18.6], [Infinity, 27.9]]; const READING = [ { no: 'S-0431', m3: 8, delayMo: 0 }, { no: 'S-0432', m3: 8, delayMo: 2 }, { no: 'S-0433', m3: 17, delayMo: 0 }, { no: 'S-0434', m3: 17, delayMo: 1 }, { no: 'S-0435', m3: 34, delayMo: 3 }, { no: 'S-0436', m3: 52, delayMo: 0 }, ]; const FLAG = ['tier', 'delay', 'summary']; const cents = (x) => Math.round(x * 100) / 100; function tiered(m3) { let remaining = m3, floor = 0, amount = 0; for (const [ceiling, unit] of TIER) { const slice = Math.min(remaining, ceiling - floor); amount += slice * unit; remaining -= slice; floor = ceiling; if (remaining <= 0) break; } return amount; } // Code in production: every flag is a branch point. function production(o, b) { let amount = b.tier ? tiered(o.m3) : o.m3 * BASE; const base = o.m3 * BASE; if (b.delay && o.delayMo > 0) amount += base * 0.015 * o.delayMo; const f = { amount: cents(amount) }; if (b.summary) f.summary = { m3: o.m3, delayMo: o.delayMo }; return f; } // Every combination's expected behavior is written separately: the fee runs on its own amount. function expected(o, b) { let amount = b.tier ? tiered(o.m3) : o.m3 * BASE; if (b.delay && o.delayMo > 0) amount += amount * 0.015 * o.delayMo; return cents(amount); } const paths = []; for (let m = 0; m < 2 ** FLAG.length; m += 1) { const b = Object.fromEntries(FLAG.map((name, i) => [name, Boolean(m & (1 << i))])); const diverging = READING.filter((o) => production(o, b).amount !== expected(o, b)); paths.push({ mask: FLAG.map((name) => (b[name] ? '1' : '0')).join(''), b, diverging }); } // Tested set: production's current state, and every flag on its own. const TESTED = new Set(['000', '100', '010', '001']); console.log('path tier delay summary diverging tested'); for (const y of paths) { console.log(`${y.mask} ${FLAG.map((a) => (y.b[a] ? 'on ' : 'off ')).join(' ')}` + ` ${String(y.diverging.length).padEnd(11)} ${TESTED.has(y.mask) ? 'yes' : 'no'}`); } const defective = paths.filter((y) => y.diverging.length > 0); console.log(''); console.log(`total code paths : ${paths.length}`); console.log(`tested paths : ${TESTED.size} (${(TESTED.size / paths.length * 100).toFixed(0)}% share)`); console.log(`paths producing diverging behavior: ${defective.length} — ${defective.map((y) => y.mask).join(', ')}`); console.log(`of these, tested : ${defective.filter((y) => TESTED.has(y.mask)).length}`); console.log(''); console.log('per-invoice difference on the first path where the defect appears'); for (const o of defective[0].diverging) { const y = defective[0]; console.log(` ${o.no} ${String(o.m3).padStart(3)} m3 ${o.delayMo} mo delay` + ` production=${production(o, y.b).amount} expected=${expected(o, y.b)}`); } ``` ``` path tier delay summary diverging tested 000 off off off 0 yes 100 on off off 0 yes 010 off on off 0 yes 110 on on off 2 no 001 off off on 0 yes 101 on off on 0 no 011 off on on 0 no 111 on on on 2 no total code paths : 8 tested paths : 4 (50% share) paths producing diverging behavior: 2 — 110, 111 of these, tested : 0 per-invoice difference on the first path where the defect appears S-0434 17 m3 1 mo delay production=257.36 expected=258.01 S-0435 34 m3 3 mo delay production=673.07 expected=683.53 ``` The defect does not sit in a single flag; it sits **at the intersection of two flags**: the late fee runs on the base amount instead of the tiered amount. While the tier is off, the two values are equal, so the computation comes out right; once the tier turns on, it drifts. Because the test set tries each flag on its own, it never sees four of the eight paths, and both of the two defective paths sit inside that four. The size of the drift is also worth noting: between 0.65 and 10.46 per invoice. This is a drift that produces no error message, runs from a quarter of a percent to one and a half percent as a share, and shows up only for subscribers who happen to have a delay. That is why the question of how many of the eight paths get tested matters: adding a flag does not add to testing obligation, it multiplies it. #### The Gate's Three Numbers The flag condition is a gate, and like the other gates it is evaluated on three numbers. First, what it catches. The number of defects the flag condition catches is **zero**: the condition lets both branches through, and tests nothing. The only thing it catches is exposure — 84 invoices instead of 160, a difference of 76 invoices. All of that difference comes from the rollback path, and it is what only this gate delivers: none of the pipeline's other gates can change the rollback unit. Second, what it misses and what it wrongly holds. The flag gate produces the defect it misses by itself: the four untested paths are paths that did not exist before the three flags were added. Wrongly holding is measured by sound behavior sitting off — in the ledger below, at the end of the twelve-week period, there is a change with nothing to do with the defect, complete and deployed, that is still sitting off. Third, the time it adds. This gate adds no minutes to lead time: the code still ships in a single release, the pipeline's stages do not change. What it adds is a **deferred round** — writing the read points, and later deleting them. Because deferral looks free, it goes unpaid and accumulates as debt. In this lesson, the gate's third number is not minutes; it is the count of uncleared read points. #### Flag Debt A flag's cost is not paid the moment it is added; it is paid for as long as it is not removed. A twelve-week period's ledger is counted: which flag was added when, whether it was cleared, and at how many points it gets read. ```js // debt.mjs — one period's flag ledger is counted; the ledger is a MODEL // [name, week added, week cleared (null: still standing), read points, today's value] const PERIOD = 12; const LEDGER = [ ['tiered-rate', 1, 6, 3, 'on'], ['late-fee', 2, null, 2, 'off'], ['invoice-summary', 2, 7, 1, 'on'], ['work-order-link', 3, null, 4, 'on'], ['batch-job-report', 4, null, 2, 'on'], ['verifier-second-pass', 5, 9, 2, 'on'], ['reading-retry', 7, null, 5, 'on'], ['contract-warning', 8, null, 3, 'off'], ['summary-new-field', 10, null, 2, 'on'], ]; const cleared = LEDGER.filter(([, , t]) => t !== null); const remaining = LEDGER.filter(([, , t]) => t === null); const points = remaining.reduce((t, [, , , n]) => t + n, 0); const oldest = remaining.reduce((e, k) => (k[1] < e[1] ? k : e)); console.log(`period : ${PERIOD} weeks`); console.log(`added / cleared / remaining : ${LEDGER.length} / ${cleared.length} / ${remaining.length}`); console.log(`average lifetime of cleared flags: ` + `${(cleared.reduce((t, [, e, k]) => t + (k - e), 0) / cleared.length).toFixed(1)} weeks`); console.log(`paths opened by remaining flags : ${2 ** remaining.length}`); console.log(`path running today : 1 (remaining flags' values are fixed)`); console.log(`dead code paths : ${2 ** remaining.length - 1}`); console.log(`if every flag is tested alone : ${remaining.length + 1} paths` + ` (${((remaining.length + 1) / 2 ** remaining.length * 100).toFixed(0)}%)`); console.log(`read points left to clean : ${points}`); console.log(`off flags waiting to be turned on: ` + remaining.filter(([, , , , d]) => d === 'off').map(([a, e]) => `${a} (${PERIOD - e} weeks)`).join(', ')); console.log(`oldest uncleared flag : ${oldest[0]} — ${PERIOD - oldest[1]} weeks`); ``` ``` period : 12 weeks added / cleared / remaining : 9 / 3 / 6 average lifetime of cleared flags: 4.7 weeks paths opened by remaining flags : 64 path running today : 1 (remaining flags' values are fixed) dead code paths : 63 if every flag is tested alone : 7 paths (11%) read points left to clean : 18 off flags waiting to be turned on: late-fee (10 weeks), contract-warning (4 weeks) oldest uncleared flag : late-fee — 10 weeks ``` Six flags remained standing and opened sixty-four code paths. Only one of these paths runs today, because all six flags' values are fixed. The remaining sixty-three are dead code paths: none of them execute, all of them sit in the code, and each is assumed to be a behavior that can be tested. If the test set widens with the same pattern — every flag on its own — it covers seven of the sixty-four paths; the share drops from 50% to 11%. As the flag count grows linearly, coverage thins out exponentially. Eighteen read points is the number of conditions that need deleting to clear these flags. The longer this work is deferred, the less readable the code becomes — but more importantly, which branch is live stops being readable from the code itself. The three cleared flags' average lifetime was 4.7 weeks; one of the remaining ones has been sitting for ten weeks. Two flags are waiting, off: one is defective, one is complete but never turned on. The second is the change this gate wrongly holds — a behavior that has sat in production for four weeks and reached not one subscriber. The gate that needs removing becomes visible right here. `tiered-rate` was cleared in week six: the flag was turned on, the behavior became permanent, and the condition was deleted. Removing a gate belongs on the plan exactly as much as building one does; when it does not, the ledger's every number grows. #### Summary - The flag reduced the rollback unit from the entire code to a single behavior: in the without-flags scheme, four changes were pulled from production and three were sound; in the with-flags scheme, the only thing pulled was the defective behavior. - The rollback round dropped from two to one, affected invoices from 160 to 84; all of that difference came from the rollback path getting shorter, not from the defect being seen any earlier. - Three flags opened eight code paths; a set testing each flag on its own covered four of them, and both of the two paths producing diverging behavior sat outside that coverage. - A flag condition catches no defects — what it catches is exposure; the paths it misses, it produces by itself. - In twelve weeks, nine flags were added, three were cleared; the remaining six flags opened sixty-four paths, sixty-three of which sat as dead code, and eighteen read points waited to be cleared. #### Next Step Turning off a flag works when what needs rolling back is a behavior. If the defect does not sit behind a flag — configuration changed, data was transformed, or the change shipped without a flag at all — the way back is not a single switch, it is an ordered list of steps. How many steps that list has, how many rounds each step takes, and after which step the way back closes are each measured separately. The next lesson treats rollback not as a button but as a **plan**, and counts how many steps a plan's not having been rehearsed costs on its first run. ### Rollback Plan Source: https://academia.sh/en/courses/ci-cd/rollback-plan Rollback is modeled not as a button but as a plan: three scenarios (code only, code and config, code and data) are run step by step, every step's round is counted, and which step is the point of no return is found. How many steps an unrehearsed plan gets stuck on during its first run, and the precondition check's value as a gate, are measured. Turning off a flag takes one round when what needs rolling back is a behavior. If the defect sits in a change that was never flagged, if the configuration also changed, or if data was transformed along with the release, the way back is not a single switch. There are ordered steps, the steps have preconditions, and after some of them the way back closes. What this lesson measures is not release mechanics but **the plan itself**. Zero-downtime release schemes and migration-versus-release ordering were measured in **M16/K08 Server Security and Going to Production**; the question asked here is how many steps and rounds it takes to come back from those schemes. Three scenarios are run, every step's round is counted, which step is the point of no return is found, and how many rounds an unrehearsed plan costs on its first run is measured. - **RS31.** Rollback is a plan: ordered steps, each step with one round and zero or more preconditions. Steps cannot be skipped. - **RS32.** A round is the unit of time paid at the moment of the incident; in the model, every round is the same length. A round's minute equivalent varies by environment; the ratio does not. - **RS33.** The config record does not keep versions: an overwritten value cannot be recovered. - **RS34.** A rehearsal is running the plan start to finish in an environment outside the incident. It finds permanent gaps; it does not find ones tied to the moment of the incident — on-call, authority, that night's team. - **RS35.** A rollback left halfway leaves a mixed state: some nodes are on the old version, some on the new. Getting out of this state adds a fixed round. - **RS36.** The fiction is the regional measurement network's billing service; recovery is underway from a broken release. The service, and the plan and attempt ledgers below, are all fictional. #### The Plan's Three Scenarios The same incident produces three separate rollback plans. If only the code changed, the plan is short. If a configuration value also changed along with the release, two steps get added. If the release carried a data migration, three steps get added. The executor below runs the plan step by step, gets stuck on a step whose precondition is not met, and adds the repair round. ```js // plan.mjs — rollback is modeled as a plan and run across three scenarios; the plan is a MODEL // Regional measurement network fiction: recovering from the billing service's broken release. const STATE = { // the real state at the moment of the incident 'previous-artifact-in-storage': true, // the build artifact inside the retention policy still stands 'health-endpoint-compatible': true, // the old version's health endpoint matches the new check 'pre-migration-backup': true, // a snapshot backup was taken before the migration 'config-previous-version': false, // the config record does not keep versions 'oncall-authority': false, // rollback authority sits only with the day team 'reverse-migration-script': false, // the migration's reverse step was never written }; const REPAIR = { 'config-previous-version': 3, 'oncall-authority': 2, 'reverse-migration-script': 6 }; // [name, round, preconditions, is the way back open after this step] const CODE = [ ['defect confirmed', 1, ['health-endpoint-compatible'], true], ['rollback decision and authority', 1, ['oncall-authority'], true], ['previous build artifact selected', 1, ['previous-artifact-in-storage'], true], ['nodes reverted to the old artifact', 2, [], true], ['health and consistency verified', 1, ['health-endpoint-compatible'], true], ]; const CONFIG = [ ['config written back to previous value', 2, ['config-previous-version'], false], ['config propagated to nodes', 1, [], true], ]; const DATA = [ ['writes stopped', 1, [], true], ['reverse migration runs', 3, ['reverse-migration-script'], false], ['writes reopened, counts compared', 2, ['pre-migration-backup'], true], ]; const SCENARIO = { 'code only': CODE, 'code + config': [...CODE.slice(0, 4), ...CONFIG, CODE[4]], 'code + data': [...CODE.slice(0, 4), ...DATA, CODE[4]], }; function run(plan, rehearsed) { const state = { ...STATE }; if (rehearsed) for (const k of Object.keys(REPAIR)) state[k] = true; // rehearsal found it first, fixed it first let round = 0, extra = 0; const stuck = []; plan.forEach(([name, t, cond], i) => { for (const k of cond) { if (!state[k]) { stuck.push([i + 1, name, k, REPAIR[k]]); extra += REPAIR[k]; state[k] = true; } } round += t; }); const d = plan.findIndex(([, , , g]) => !g); return { steps: plan.length, round, extra, stuck, pointOfNoReturn: d < 0 ? '—' : String(d + 1) }; } console.log('scenario steps unrehearsed stuck extra round unrehearsed rehearsed point of no return'); for (const [name, plan] of Object.entries(SCENARIO)) { const y = run(plan, false), p = run(plan, true); console.log(`${name.padEnd(19)} ${String(y.steps).padEnd(5)} ${String(y.stuck.length).padEnd(19)} ` + `${String(y.extra).padEnd(13)} ${String(y.round + y.extra).padEnd(13)} ${String(p.round + p.extra).padEnd(10)} ${y.pointOfNoReturn}`); } console.log(''); console.log('code + data, first unrehearsed run — step by step'); const kv = run(SCENARIO['code + data'], false); for (const [no, name, cond, repairRound] of kv.stuck) { console.log(` step ${no} stuck: ${name.padEnd(38)} missing precondition: ${cond} (+${repairRound} round)`); } console.log(` total ${kv.steps} steps, ${kv.round} plan rounds, ${kv.extra} repair rounds`); console.log(` no way back after step ${kv.pointOfNoReturn}: the reverse migration overwrites the data`); ``` ``` scenario steps unrehearsed stuck extra round unrehearsed rehearsed point of no return code only 5 1 2 8 6 — code + config 7 2 5 14 9 5 code + data 8 2 8 20 12 6 code + data, first unrehearsed run — step by step step 2 stuck: rollback decision and authority missing precondition: oncall-authority (+2 round) step 6 stuck: reverse migration runs missing precondition: reverse-migration-script (+6 round) total 8 steps, 12 plan rounds, 8 repair rounds no way back after step 6: the reverse migration overwrites the data ``` Three scenarios come from the same incident and take five, seven, eight steps. The rise in step count looks small; the rise in round count does not. In the unrehearsed run, an eight-round plan climbs to twenty rounds — two and a half times. All of that difference comes not from the plan's steps, but from the steps' preconditions. The real number here is that rollback **does not carry the same weight as release**. A release goes through the pipeline: its stages are ordered, its preconditions are supplied by the pipeline, and every step has already run hundreds of times before. Rollback, by contrast, usually runs for the first time. This asymmetry between the two directions of the same system shows up in the three scenarios' unrehearsed column. Three of the model's six preconditions are not met, and all three are unmet for the same reason: none of them is required in the forward direction. The config record keeping history is not required for release — the latest value is enough going forward. The on-call engineer's rollback authority is not required for release — the pipeline does the releasing. The migration's reverse script is not required for release — the migration runs forward. All three are needed only on the way back, and something needed only on the way back is something no one notices until the way back is taken. The moment a missing precondition is discovered is, most of the time, the moment it is needed the most. #### The Point of No Return Not every step in a rollback plan is reversible. In the code-only scenario there is no point of no return: every step moves to the next state without destroying the previous one; someone who wants to change their mind at round eight can release forward to the new version again. In the config scenario, the point of no return is step five. Because the config record does not keep versions, writing over the previous value means losing the current one. The value lost is the new version's value, and once rollback fails and someone wants to go back to the new version, that value is no longer there. The step itself takes one second; its irreversibility comes not from its duration, but from the fact that where it writes keeps no history. In the data scenario, the point of no return is step six. The reverse migration converts the records accumulated in the new schema back to the old format and overwrites them. After this step, the rest of the plan becomes mandatory: if steps seven and eight do not run, the system is in neither the old state nor the new one. That is why the eight-step plan's last three steps count as a single unit. Where the point of no return sits inside the plan is direct information for the decision. A rollback that has reached step five or six is no longer "abandonable"; the decision that has to be made before that point is whether rollback is truly necessary. Knowing up to which step a plan is free also says when the decision has to be made. #### The Cost of an Unrehearsed Plan The unrehearsed and rehearsed columns run the same plans, with the same step count, in the same order. The only difference is whether the preconditions are found before the incident or during it. In the code-only scenario the difference is two rounds, in the data scenario eight. The unrehearsed run's step-by-step breakdown shows where it gets stuck. At step two, the on-call engineer lacks rollback authority: two rounds pass while someone else is reached and the authority is opened up. At step six, the migration's reverse script has not been written: six rounds pass while the script gets written during the incident. Where the second stuck point falls is worth noting — right on top of the point of no return. The script gets written at the exact step where the way back closes, and it runs having never been tested. Rehearsal is not free. The plan's twelve rounds get paid during rehearsal too, plus the repair of the gaps. The difference is **when** the payment happens: rehearsal rounds pass outside the incident, at a chosen time, with no one kept waiting. The unrehearsed plan's eight repair rounds, by contrast, pass inside the incident window, while the defect is still running in production. The same work, at a different price. Rehearsal's limit is also part of the measurement. The script not being written gets found by rehearsal, because it is a permanent gap. The on-call engineer's authority is not: if the rehearsal runs during the day, it passes with the day team's authority, and the gap that opens up on the night shift stays invisible. #### The Gate: Pre-Step Precondition Check Checking every plan step's precondition before the step starts is a gate: a step that does not pass does not start. This gate, too, is evaluated on three numbers. A ledger of twelve rollback attempts is run in two modes — checked and unchecked. ```js // check.mjs — the precondition check is a gate; twelve rollback attempts run in two modes // [attempt, scenario, actually missing preconditions, what the check wrongly calls "missing"] MODEL const ATTEMPT = [ ['O-01', 'code only', [], []], ['O-02', 'code only', ['oncall-authority'], []], ['O-03', 'code + config', ['config-previous-version'], []], ['O-04', 'code only', [], ['previous-artifact-in-storage']], ['O-05', 'code + data', ['reverse-migration-script'], []], ['O-06', 'code only', [], []], ['O-07', 'code + config', ['oncall-authority', 'config-previous-version'], []], ['O-08', 'code only', [], []], ['O-09', 'code + data', ['pre-migration-backup'], []], ['O-10', 'code only', ['oncall-authority'], []], ['O-11', 'code + config', [], ['config-previous-version']], ['O-12', 'code + data', [], []], ]; const REPAIR = { 'config-previous-version': 3, 'oncall-authority': 2, 'reverse-migration-script': 6, 'pre-migration-backup': 4, 'previous-artifact-in-storage': 2 }; const CHECKED = new Set(['previous-artifact-in-storage', 'health-endpoint-compatible', 'config-previous-version', 'oncall-authority', 'reverse-migration-script']); const REHEARSAL_CATCHES = new Set(['config-previous-version', 'reverse-migration-script', 'previous-artifact-in-storage']); const HALFWAY = 5; // the round cost of getting out of a half-finished rollback's mixed state const CHECK = 1; // the round the precondition check adds per plan const count = { caught: 0, onlyThis: 0, missed: 0, wrong: 0, wrongRounds: 0, uncheckedExtra: 0, checkedExtra: 0 }; for (const [, , missing, wrongMissing] of ATTEMPT) { const repair = missing.reduce((t, k) => t + REPAIR[k], 0); count.uncheckedExtra += repair + (missing.length ? HALFWAY : 0); const seen = missing.filter((k) => CHECKED.has(k)); const missed = missing.filter((k) => !CHECKED.has(k)); const wasted = wrongMissing.reduce((t, k) => t + REPAIR[k], 0); count.wrongRounds += wasted; count.checkedExtra += CHECK + repair + (missed.length ? HALFWAY : 0) + wasted; if (seen.length) { count.caught += 1; if (seen.every((k) => !REHEARSAL_CATCHES.has(k))) count.onlyThis += 1; } if (missed.length) count.missed += 1; if (wrongMissing.length) count.wrong += 1; } console.log(`${'attempts'.padEnd(46)}: ${ATTEMPT.length}`); console.log(`${'what the gate catches (half-finished avoided)'.padEnd(46)}: ${count.caught}`); console.log(`${'of these, only this gate catches'.padEnd(46)}: ${count.onlyThis} (what rehearsal cannot catch)`); console.log(`${'what the gate misses (unchecked precondition)'.padEnd(46)}: ${count.missed}`); console.log(`${'what the gate wrongly holds'.padEnd(46)}: ${count.wrong}`); console.log(''); console.log(`${'rounds added to the incident, unchecked mode'.padEnd(46)}: ${count.uncheckedExtra}`); console.log(`${'rounds added to the incident, checked mode'.padEnd(46)}: ${count.checkedExtra}`); console.log(`${' from the check itself'.padEnd(46)}: ${ATTEMPT.length * CHECK}`); console.log(`${' from wrongly holding'.padEnd(46)}: ${count.wrongRounds}`); console.log(`${"the gate's net gain".padEnd(46)}: ${count.uncheckedExtra - count.checkedExtra} rounds`); console.log(`${'net gain per attempt'.padEnd(46)}: ${((count.uncheckedExtra - count.checkedExtra) / ATTEMPT.length).toFixed(2)} rounds`); ``` ``` attempts : 12 what the gate catches (half-finished avoided) : 5 of these, only this gate catches : 2 (what rehearsal cannot catch) what the gate misses (unchecked precondition) : 1 what the gate wrongly holds : 2 rounds added to the incident, unchecked mode : 52 rounds added to the incident, checked mode : 44 from the check itself : 12 from wrongly holding : 5 the gate's net gain : 8 rounds net gain per attempt : 0.67 rounds ``` The gate prevented a halfway rollback in five of twelve attempts. But rehearsal would have caught three of these five too: they were permanent gaps, and running the plan start to finish once would have been enough. The two attempts **only this gate** catches are the ones where on-call authority happened to be missing that night — a gap tied to the moment of the incident, invisible to a rehearsal that ran during the day. The marginal value is not five, it is two. The gate missed one attempt: the pre-migration backup not having been taken is not on the check list. The check list is itself a piece of code, and what it does not check has to be counted as much as what it does. In two attempts, the gate wrongly held — it took a precondition that had actually been met for missing, and made the attempt pay a five-round repair for nothing. The time the gate adds is a separate line in this ledger: one round per attempt, twelve rounds across twelve attempts. This time is paid in a window where the defect keeps running in production, and in seven of the attempts it passes without finding a single gap. The five rounds of wrongly holding are paid in the same window. The gate's gross gain is twenty-five rounds; the audit's own cost is twelve rounds and wrongful holding's is five — seventeen together. The net gain is forty-four rounds against fifty-two, per-attempt two-thirds of a round. This is a thin margin, and the gate's own cost — twelve rounds — already exceeds that gain. What keeps the gate justified is not the average but the distribution: in the attempts it catches, the cost is a five-round mixed state, and that cost is paid inside the incident window, while the defect is running. The gate's only defensible form is splitting the check list with rehearsal: permanent gaps go to rehearsal, ones tied to the moment of the incident go to the gate. #### Summary - The same incident produced three plans: code only five steps, code with config seven steps, code with data eight steps; their unrehearsed total durations were eight, fourteen, and twenty rounds. - There is no point of no return in the code-only scenario; it is step five in the config scenario and step six in the data scenario, after which the rest of the plan becomes mandatory. - The unrehearsed run got stuck at two steps and added eight repair rounds; the second stuck point sat right above the point of no return, at the step where the reverse migration script got written during the incident. - Rehearsal makes you pay for the same work, at a different time: twelve plan rounds pass outside the incident, while eight repair rounds pass while the defect is running in production. - The precondition-check gate caught five of twelve attempts, caught two of those by itself alone, missed one, wrongly held two; its net gain was eight rounds, its own cost twelve. #### Next Step Two of the three scenarios had a point of no return, and both were tied to data. Code is reversible; configuration is reversible if it is versioned; for a schema change, it is not that simple. Some migration types are fully reversible, some look like they work but bring the data back incomplete, and some have no way back at all. The next lesson ranks migration types on an axis of reversibility, runs the expand–migrate–contract scheme step by step, and at every step counts whether the old and new versions can run at the same time. ### Coordination with Database Migrations Source: https://academia.sh/en/courses/ci-cd/coordination-with-database-migrations Six migration types are actually applied and reversed, and reversibility is measured not by whether the command runs without error but by whether the table's content comes out identical. The expand–migrate–contract scheme runs step by step; at every step, whether the old and new versions can run together is counted, and the incompatible step's effect on the release window and the rollback plan is written down. Two of the previous lesson's three scenarios had a point of no return, and in the data scenario, that point was the reverse migration step. A rollback plan is only as reversible as every step inside it, and a schema change is the most fragile of those steps. Code comes back by selecting an artifact, configuration by writing a value. For schema, this is not always true. This lesson's axis is **reversibility**. Migration-versus-code-release ordering, and how many requests that ordering errors on, were measured in **M16/K08 Server Security and Going to Production**; the question asked here is not ordering but class: which migration types are fully reversible, which look like they work but bring the data back incomplete, and which have no way back at all. The second measure is whether the steps **carry two versions together** — this is where the rollback plan intersects with the release window. - **RS37.** The schema lives in a single database, and every version reads and writes the same table. - **RS38.** Migration steps run separately from the application release; the two are not part of the same operation. - **RS39.** Reversibility is measured by the table's content being **identical** once the reverse operation has run. The command running without error is not enough on its own. - **RS40.** Two versions are compatible when each can correctly read the record the other one wrote. The query not raising an error is not enough; the value it reads back has to be correct too. - **RS41.** The transitional versions' reads fall back: if the new column is empty, they look at the old one. - **RS42.** The fiction is the regional measurement network's meter reading table; the reading value is a single column in cubic meters, being split into a first/last index pair in liters. The figures are fictional. #### The Reversibility of Migration Types Six migration types are applied to a real table, their reverse runs immediately after, and the table's content is compared against what it started as. The measure is not whether the reverse command runs, but whether the rows come out identical. ```js // migration-types.mjs — six migration types are actually applied, reversed, and the content is compared // Regional measurement network fiction: a meter reading table. Measure: identity after the reverse operation. import { DatabaseSync } from 'node:sqlite'; import { createHash } from 'node:crypto'; const READING = [ ['R-101', 'S-0431', 8.432, 'visual'], ['R-102', 'S-0432', 17.06, 'visual'], ['R-103', 'S-0433', 34.915, 'remote'], ['R-104', 'S-0434', 52.5, 'remote'], ['R-105', 'S-0435', 6.004, 'visual'], ['R-106', 'S-0436', 21.37, 'estimated'], ]; function setUp() { const db = new DatabaseSync(':memory:'); db.exec('CREATE TABLE reading (id TEXT PRIMARY KEY, meter TEXT, m3 REAL, method TEXT)'); const insert = db.prepare('INSERT INTO reading VALUES (?, ?, ?, ?)'); for (const o of READING) insert.run(...o); return db; } const rows = (db) => db.prepare('SELECT * FROM reading ORDER BY id').all() .map((s) => Object.keys(s).sort().map((k) => `${k}=${s[k]}`).join('|')); const digest = (db) => createHash('sha256').update(rows(db).join('\n')).digest('hex').slice(0, 10); const TYPE = [ ['add column', (db) => db.exec('ALTER TABLE reading ADD COLUMN liters INTEGER'), (db) => db.exec('ALTER TABLE reading DROP COLUMN liters')], ['add index', (db) => db.exec('CREATE INDEX ix_meter ON reading(meter)'), (db) => db.exec('DROP INDEX ix_meter')], ['rename column', (db) => db.exec('ALTER TABLE reading RENAME COLUMN m3 TO consumption'), (db) => db.exec('ALTER TABLE reading RENAME COLUMN consumption TO m3')], ['backfill', (db) => { db.exec('ALTER TABLE reading ADD COLUMN liters INTEGER'); db.exec('UPDATE reading SET liters = CAST(m3 * 1000 AS INTEGER)'); }, (db) => db.exec('ALTER TABLE reading DROP COLUMN liters')], ['in-place precision narrowing', (db) => db.exec('UPDATE reading SET m3 = ROUND(m3, 1)'), (db) => db.exec('UPDATE reading SET m3 = m3')], ['drop column', (db) => db.exec('ALTER TABLE reading DROP COLUMN method'), (db) => db.exec('ALTER TABLE reading ADD COLUMN method TEXT')], ]; console.log('migration type before after after reverse identical different rows'); for (const [name, forward, reverse] of TYPE) { const db = setUp(); const o1 = rows(db), h1 = digest(db); forward(db); const h2 = digest(db); reverse(db); const o3 = rows(db), h3 = digest(db); const diff = o1.filter((s, i) => s !== o3[i]).length; console.log(`${name.padEnd(31)} ${h1} ${h2} ${h3} ` + `${(h1 === h3 ? 'yes' : 'no').padEnd(6)} ${diff}`); db.close(); } ``` ``` migration type before after after reverse identical different rows add column 7ea0a3f6ba bd4e22eb22 7ea0a3f6ba yes 0 add index 7ea0a3f6ba 7ea0a3f6ba 7ea0a3f6ba yes 0 rename column 7ea0a3f6ba 94357d3ccb 7ea0a3f6ba yes 0 backfill 7ea0a3f6ba 55cf38998f 7ea0a3f6ba yes 0 in-place precision narrowing 7ea0a3f6ba 2ddb95ed6d 2ddb95ed6d no 5 drop column 7ea0a3f6ba 6f470ab65d 75633d1eb4 no 6 ``` Four of the six types are fully reversible: the digest returns to its previous value, and not one row differs. What these four have in common is that they never lose data. Adding a column or an index does not touch existing rows. Renaming only changes a name. Backfilling writes to a new column while the old one stays in place. The last two types are not reversible, and each fails for a different reason. Precision narrowing overwrote the value **in place**: once the decimals are truncated, there is no reverse command that can bring them back. An identity operation stands in for the reverse, and five rows' values never returned to their previous state — the second and third digests came out the same. Dropping a column is the example of a type where the command runs without error but still does not restore anything: the column was re-added, and all six rows came back empty. The distinction sits right here: a rollback command existing is not the same as being reversible. In both failing rows, the reverse command ran and returned exit code zero. Building the measure on identity turns the phrase "a rollback script was written" into something that can actually be tested. The table's third row also carries a warning. Renaming a column is fully reversible — zero different rows — but that does not mean the step is safe. A version using the old name cannot find the column the moment the rename goes through. Reversibility looks at what comes **after** a step; compatibility looks at **during**. A step can be reversible and still fail to carry two versions together, or the reverse. That is why the second measure runs separately. #### Expand–Migrate–Contract, Step by Step A single reading column in cubic meters gets split into a first/last index pair in liters. The scheme has six steps: the schema expands, the data migrates, the code advances through four versions, the schema contracts. At every step, the version currently running and the one before it are tested together — one writes, the other reads, then the reverse. ```js // expand-migrate-contract.mjs — expand-migrate-contract runs step by step; two versions are tested together at each step // Fiction: a meter reading's single m3 column is split into a first/last index pair, in liters. import { DatabaseSync } from 'node:sqlite'; const M3 = 12.345, FIRST = 100000, LAST = 112345; const READ_FALLBACK = 'SELECT COALESCE((last_index - first_index) / 1000.0, m3) AS value' + ' FROM reading WHERE id = ?'; const VERSION = { v1: { field: 'm3', read: 'SELECT m3 AS value FROM reading WHERE id = ?', write: 'INSERT INTO reading (id, meter, m3) VALUES (?, ?, ?)' }, v2: { field: 'm3+index', read: READ_FALLBACK, write: 'INSERT INTO reading (id, meter, m3, first_index, last_index) VALUES (?, ?, ?, ?, ?)' }, v3: { field: 'index', read: READ_FALLBACK, write: 'INSERT INTO reading (id, meter, first_index, last_index) VALUES (?, ?, ?, ?)' }, v4: { field: 'index', read: 'SELECT (last_index - first_index) / 1000.0 AS value FROM reading WHERE id = ?', write: 'INSERT INTO reading (id, meter, first_index, last_index) VALUES (?, ?, ?, ?)' }, }; const VALUE = { m3: ['S-0440', M3], 'm3+index': ['S-0440', M3, FIRST, LAST], index: ['S-0440', FIRST, LAST] }; const SCHEMA = { expand: [(db) => db.exec('ALTER TABLE reading ADD COLUMN first_index INTEGER;' + ' ALTER TABLE reading ADD COLUMN last_index INTEGER'), true], migrate: [(db) => db.exec('UPDATE reading SET first_index = 0,' + ' last_index = CAST(m3 * 1000 AS INTEGER) WHERE last_index IS NULL'), true], contract: [(db) => db.exec('ALTER TABLE reading DROP COLUMN m3'), false], }; // [name, schema step, running version, one-version-back target, two-version-back target] const STEP = [ ['expand: index columns added', 'expand', 'v1', 'v1'], ['dual-writing version released', null, 'v2', 'v1'], ['migrate: old rows backfilled', 'migrate', 'v2', 'v1'], ['version that stops writing m3 released', null, 'v3', 'v2', 'v1'], ['version that removes the fallback released', null, 'v4', 'v3'], ['contract: m3 column dropped', 'contract', 'v4', 'v3'], ]; const db = new DatabaseSync(':memory:'); db.exec('CREATE TABLE reading (id TEXT PRIMARY KEY, meter TEXT, m3 REAL)'); db.prepare('INSERT INTO reading (id, meter, m3) VALUES (?, ?, ?)').run('R-101', 'S-0431', 8.432); function test(A, B, suffix) { const finding = []; for (const [j, [writer, reader]] of [[A, B], [B, A]].entries()) { const id = `T-${suffix}-${j}`; try { db.prepare(VERSION[writer].write).run(id, ...VALUE[VERSION[writer].field]); } catch (e) { finding.push(`${writer} failed to write: ${e.message}`); continue; } try { const d = db.prepare(VERSION[reader].read).get(id)?.value; if (Math.abs((d ?? -1) - M3) > 1e-9) finding.push(`${reader} read ${writer}'s record as ${d}`); } catch (e) { finding.push(`${reader} failed to read: ${e.message}`); } } return finding; } console.log('step schema running target compat schema reversible'); let compat = 0, reversible = 0, schemaSteps = 0; STEP.forEach(([name, schema, A, B, C], i) => { if (schema) { SCHEMA[schema][0](db); schemaSteps += 1; if (SCHEMA[schema][1]) reversible += 1; } const finding = test(A, B, i + 1); if (finding.length === 0) compat += 1; console.log(`${(i + 1)} ${name.padEnd(38)} ${(schema ?? '—').padEnd(9)} ${A} ${B} ` + `${(finding.length ? 'no' : 'yes').padEnd(7)} ${schema ? (SCHEMA[schema][1] ? 'yes' : 'no') : '—'}`); for (const b of finding) console.log(` ${b}`); if (C) { const twoBack = test(A, C, `${i + 1}x`); console.log(` two versions back (${A} with ${C}): ${twoBack.length ? 'incompatible' : 'compatible'}`); for (const b of twoBack) console.log(` ${b}`); } }); console.log(''); console.log(`steps where two versions ran together : ${compat} / ${STEP.length}`); console.log(`step where they could not : ${STEP.length - compat}`); console.log(`schema steps : ${schemaSteps}, of which reversible ${reversible}`); ``` ``` step schema running target compat schema reversible 1 expand: index columns added expand v1 v1 yes yes 2 dual-writing version released — v2 v1 yes — 3 migrate: old rows backfilled migrate v2 v1 yes yes 4 version that stops writing m3 released — v3 v2 yes — two versions back (v3 with v1): incompatible v1 read v3's record as null 5 version that removes the fallback released — v4 v3 yes — 6 contract: m3 column dropped contract v4 v3 no no v3 failed to read: no such column: m3 steps where two versions ran together : 5 / 6 step where they could not : 1 schema steps : 3, of which reversible 2 ``` In five of the six steps, two versions ran together. The one incompatible step is the last one, and it is also the one schema step that cannot be reversed. Two properties meet at the same step: dropping the old column both leaves the previous code version unable to run and cannot itself be undone. The extra row at step four shows that compatibility depends on **how many versions back** you ask. One version back — the running version and the one before it — is compatible. Two versions back is not: the version that stops writing to the m3 column has its record read as empty by the version two generations earlier. No error, the record exists, no value. That is why the rollback window is a single version; a plan that wants to go back two versions has to climb down the code steps one at a time. Two of the three schema steps are reversible. Expansion is reversed by dropping the columns, migration by clearing the new columns — in both, no loss occurs because the values keep standing in the old column. Contraction is the catalog's sixth row itself. The code ladder having four rungs also falls out of this measurement. Because the dual-writing version keeps filling the old column, the previous version can keep reading. The version that stops writing m3 is the next rung, the version that removes the fallback the one after that. Without the fourth rung, the contract step cannot run: the fallback read refers to a column that no longer exists, and whatever version is running in production at that moment breaks. If one of the rungs is skipped, the incompatible-step count goes from one to two. #### The Cost of the Incompatible Step The incompatible step charges twice. First, the release window. A step where two versions cannot run together cannot roll out in stages: every moment where some nodes are on the old version and some on the new is a moment of failure. Each of the five compatible steps can advance node by node; the sixth step has to go through in one shot, on every node at the same time. The gradual observation window staged rollout provides does not exist at this step. Second, the rollback plan. For the first five steps, rollback is the previous lesson's "code only" scenario: five steps, eight rounds, no point of no return. Past the sixth step, the plan shifts to the "code and data" scenario: eight steps, twenty unrehearsed rounds, and a way back that closes at step six. A single migration step stretches the rollback plan two and a half times and puts an irreversible point inside it. The gate here is the **compatibility check** that runs the old version's queries against the new schema before the migration step ships to production. What it catches is one step: the sixth of six. Only this gate catches this step — the migration step's own tests run against the new schema and never see the old query at all. What it misses is the catalog's fifth row: precision narrowing is compatible with both versions, the gate lets it through, but it is not reversible. Compatibility and reversibility are separate axes, and this gate looks at only one of them. The gate wrongly holding comes from its choice of target. If the check compares the running version against the oldest deployable version instead of the one before it, step four also gets stopped — as the output showed, that comparison comes out incompatible. One of the six steps ends up waiting for nothing. The time it adds is three rounds for three schema steps; what gets paid when it misses is a twenty-round rollback plan. #### Summary - Four of six migration types left the table identical after the reverse operation; precision narrowing failed to restore five rows, dropping a column failed to restore six — in both, the reverse command ran without error. - Reversibility is measured by content identity, not by a command's existence; with this measure, the phrase "a rollback script was written" becomes testable. - Expand–migrate–contract carried two versions together in five of six steps; the one incompatible step was the same step as the one irreversible schema step. - The rollback window is a single version: step four, compatible one version back, made the old version read an empty value once you went back two. - The incompatible step dropped the release window from staged to one-shot, and stretched the rollback plan from five steps and eight rounds to eight steps and twenty. #### Next Step Across this topic, where the release decision comes from was measured: what metric is watched to promote, what the rollback unit is, how many steps a plan has, whether a migration step carries two versions. What comes out of all these decisions is a desired state — this version, this flag value, this schema. But that desired state itself is not written down anywhere. What carries it to production is still a button clicked in an interface, or a command in someone's hands; the only way to learn what is actually running in production is to look at production. The next topic opens with this question: where does the desired state live, who changes it, and how is it measured whether the real state has drifted from it. ## Repository-Based Deployment and the Supply Chain Keeping the desired state in version control and verifying the source of the output: the repository-defined desired state principle, pull-based deployment with an in-cluster reconciler agent, approaches to storing sealed secrets in the repository, generating the dependency inventory, signing and verifying the build artifact, least-privileged identities, and the pipeline's own security with an isolated build environment. ### Repository-Defined Desired State Source: https://academia.sh/en/courses/ci-cd/repository-based-desired-state The same change flow runs in two arrangements — manual application and committing to the repository — and across 180 changes the two arrangements' traceability, unrecorded-change rate, end-of-period drift, and steps per change are counted. The review gate's marginal value — the defect it alone catches, the clean change it wrongly holds, and the 796 hours it adds — is measured separately. The previous topics established how a release is decided: which metric promotion looks at, at what threshold a percentage rollout stops, how many steps a rollback plan has. All of these decisions shared one thing — in the end, someone pressed a button in an interface or typed a shell command. The decision itself was on record; its **application** was not. This lesson measures that gap. The question is: does the desired state the cluster is running exist **in writing** somewhere, or only in the running system's memory? The **repository-defined desired state** principle is a single sentence — the definition of what runs in an environment lives in a repository, and it changes through no path outside that repository. The principle itself is short; its bill is long. This lesson draws up the bill. The measurement is over the **regional measurement network** and is **fictional**: the meter-reading collector, verifier, billing, and work-order services, across three environments. What runs is a **model** built with `node`, not a real system. **RD1.** The desired state is 60 items: four services × three environments × five fields (replica count, threshold, version, network rule, schedule). **RD2.** 180 changes are generated over ninety days; the generator is self-written, seed 20260801 is visible. **RD3.** A portion of the changes are **urgent** (off-shift intervention); 20% carry a known defect: format, threshold, environment, runtime. **RD4.** In the manual arrangement, recording is voluntary — the person applying it writes it down somewhere if they choose to. In the repository arrangement, the commit record mandatorily carries who, when, and rationale fields. **RD5.** In the repository arrangement too, a portion of the urgent changes bypass the repository; the principle is not applied 100%. **RD6.** There are two gates and both are only on the repository path: the schema validator (automated) and review (human). #### The Same Change, Two Paths ```js // flow.mjs — a 90-day change flow modeled over the (fictional) regional measurement // network. The same flow runs in two arrangements: manual application and committing // to the repository. What runs is a model, not a real system; seed is visible. export const generator = (t) => { let s = t >>> 0; return () => (s = (s * 1664525 + 1013904223) >>> 0) / 4294967296; }; const SERVICE = ["collector", "verifier", "billing", "workorder"]; const ENV = ["development", "staging", "production"]; const FIELD = ["replica", "threshold", "version", "network-rule", "schedule"]; export const ITEM = SERVICE.flatMap((s) => ENV.flatMap((e) => FIELD.map((f) => `${s}/${e}/${f}`))); export function flow(seed = 20260801, count = 180) { // 90-day change flow const r = generator(seed), list = []; for (let i = 0; i < count; i++) { const item = ITEM[Math.floor(r() * ITEM.length)]; const urgent = r() < 0.22; // off-shift intervention const p = r(); // known defect set: 20% const defect = p < 0.06 ? "format" : p < 0.12 ? "threshold" : p < 0.16 ? "environment" : p < 0.20 ? "runtime" : null; list.push({ i, day: Math.floor(r() * 90), item, urgent, defect }); } return list; } // repo=false: every change is applied manually, recording is voluntary. // repo=true : the change is committed to the repository; some urgent ones still skip it. export function arrange(list, { repo, seed = 7717 }) { const r = generator(seed); const actual = new Map(), ledger = new Map(); // item -> version number let who = 0, rationale = 0, unrecorded = 0, skipped = 0, step = 0, reviewed = 0; for (const d of list) { const repoPath = repo && !(d.urgent && r() < 0.35); const s = (actual.get(d.item) ?? 0) + 1; actual.set(d.item, s); if (repoPath) { // branch, change, commit, ledger.set(d.item, s); // proposal, review, who++; rationale++; reviewed++; step += 7; // merge, apply } else { skipped++; step += 2; // write command, apply const records = r() < (d.urgent ? 0.25 : 0.7); if (records) { who++; if (r() < 0.45) rationale++; } else unrecorded++; if (r() < (d.urgent ? 0.1 : 0.3)) ledger.set(d.item, s); // logged later into the ledger } } let drifted = 0; for (const [k, s] of actual) if (ledger.get(k) !== s) drifted++; return { n: list.length, who, rationale, unrecorded, skipped, reviewed, drifted, touched: actual.size, step: step / list.length }; } if (import.meta.url === `file://${process.argv[1]}`) { const list = flow(); const urgent = list.filter((d) => d.urgent).length; const defective = list.filter((d) => d.defect).length; console.log(`seed 20260801 | ${list.length} changes / 90 days | ${ITEM.length} items`); console.log(`urgent: ${urgent} | carrying a known defect: ${defective}`); const HEAD = ["arrangement", "who", "rationale", "unrecorded", "skipped", "reviewed", "drifted item", "step"]; const W = [21, 5, 11, 12, 9, 10, 14, 6]; const row = (h) => h.map((v, j) => j ? String(v).padStart(W[j]) : String(v).padEnd(W[0])).join(""); console.log("\n" + row(HEAD)); for (const repo of [false, true]) { const x = arrange(list, { repo }); console.log(row([repo ? "repository commit" : "manual application", x.who, x.rationale, x.unrecorded, x.skipped, x.reviewed, `${x.drifted}/${x.touched}`, x.step.toFixed(2)])); } } ``` ``` seed 20260801 | 180 changes / 90 days | 60 items urgent: 35 | carrying a known defect: 36 arrangement who rationale unrecorded skipped reviewed drifted item step manual application 117 54 63 180 0 45/56 2.00 repository commit 175 172 5 9 171 1/56 6.75 ``` The traceability columns diverge in the expected direction. In the manual arrangement, 117 of the 180 changes have who-and-when information landing somewhere, but only 54 carry a rationale; 63 changes — 35% of the flow — were applied leaving no trace at all. In the repository arrangement, rationale climbs to 172, because a commit record cannot be left empty: the who, when, and why fields are the change's own carrier, not a separately filled-out form. The real number sits three columns from the end. Over ninety days, 56 of the 60 items were touched; in the manual arrangement, at period's end, **45** of those 56 items differ from the repository's value. In other words, the document that describes the system is wrong for four-fifths of it. In the repository arrangement, the drifted-item count is 1, and that one item is left over from the nine urgent changes that bypassed the repository. The operational weight of this gap is heavier than a stale document. In the manual arrangement, the only place that knows an item's correct value is the running system itself; when an environment is rebuilt, or an item is reset, there is no source to fall back to. If the billing service's production threshold is changed by hand one night and no one writes it down, that threshold is no longer anyone's decision — it is an accident. The first payoff of repository-defined desired state is not auditing, it is **reversibility**: for 55 of the 56 items, the repository's value is the system's last known correct state. #### What the Review Gate Holds Only By Itself ```js // gate.mjs — ledger of the repository-path review gate: what it holds, what it holds // ONLY BY ITSELF, what it misses, wrongly holds, and the time it adds. Same flow, new seed. import { flow, generator } from "./flow.mjs"; const SCHEMA = { format: 1, threshold: 0, environment: 0, runtime: 0 }; // schema validator const REVIEW = { format: 0.85, threshold: 0.55, environment: 0.75, runtime: 0.05 }; function run(list, { escape, seed = 4231 }) { const r = generator(seed), actual = new Map(), ledger = new Map(); const byEnv = {}, byDefect = {}; let skipped = 0, gated = 0, schemaOnly = 0, marginal = 0, both = 0, passed = 0, wrong = 0, minutes = 0, triggered = false; const apply = (d, withRepo) => { const s = (actual.get(d.item) ?? 0) + 1; actual.set(d.item, s); if (withRepo) ledger.set(d.item, s); return s; }; for (const d of list) { const e = (byEnv[d.item.split("/")[1]] ??= { gated: 0, marginal: 0, wrong: 0, min: 0 }); const k = d.defect ? (byDefect[d.defect] ??= { n: 0, review: 0, schema: 0, passed: 0 }) : null; if (k) k.n++; const skip = r() < (d.urgent ? 0.35 : (escape && triggered ? 0.5 : 0)); triggered = false; if (skip) { // ungated path skipped++; const s = apply(d, false); if (k) { passed++; k.passed++; } if (r() < 0.2) ledger.set(d.item, s); continue; } gated++; e.gated++; const wait = d.urgent ? 5 + r() * 55 : 25 + r() * 575; // review wait time minutes += wait; e.min += wait; const schema = d.defect ? r() < SCHEMA[d.defect] : false; const review = d.defect ? r() < REVIEW[d.defect] : false; if (k && (schema || review)) { // held, not applied if (schema) k.schema++; if (review) k.review++; if (schema && review) both++; else if (schema) schemaOnly++; else { marginal++; e.marginal++; } continue; } if (!k && r() < 0.06) { // clean but held wrong++; e.wrong++; triggered = true; minutes += wait * 0.6; e.min += wait * 0.6; // second round } apply(d, true); if (k) { passed++; k.passed++; } } let drifted = 0; for (const [a, s] of actual) if (ledger.get(a) !== s) drifted++; return { skipped, gated, schemaOnly, marginal, both, passed, wrong, drifted, min: minutes / gated, hours: minutes / 60, byEnv, byDefect }; } const row = (w, cells) => cells.map((v, j) => j ? String(v).padStart(w[j]) : String(v).padEnd(w[0])).join(""); if (import.meta.url === `file://${process.argv[1]}`) { const list = flow(), defective = list.filter((d) => d.defect).length; const W_MAIN = [8, 9, 7, 13, 13, 6, 8, 7, 9, 12]; console.log(`${list.length} changes, ${defective} known defect; two gates: schema + review`); console.log("\n" + row(W_MAIN, ["escape", "skipped", "gated", "schema only", "review only", "both", "passed", "wrong", "drifted", "min/review"])); for (const escape of [false, true]) { const x = run(list, { escape }); console.log(row(W_MAIN, [escape ? "yes" : "no", x.skipped, x.gated, x.schemaOnly, x.marginal, x.both, x.passed, x.wrong, x.drifted, x.min.toFixed(0)])); } const x = run(list, { escape: false }); const W_DEFECT = [13, 7, 8, 8, 8]; console.log("\n" + row(W_DEFECT, ["defect", "count", "schema", "review", "passed"])); for (const [ad, k] of Object.entries(x.byDefect)) console.log(row(W_DEFECT, [ad, k.n, k.schema, k.review, k.passed])); const W_ENV = [13, 7, 13, 14, 13, 16]; console.log("\n" + row(W_ENV, ["environment", "gated", "review only", "wrongly held", "total hours", "hours/marginal"])); for (const [ad, e] of Object.entries(x.byEnv)) console.log(row(W_ENV, [ad, e.gated, e.marginal, e.wrong, (e.min / 60).toFixed(1), e.marginal ? (e.min / 60 / e.marginal).toFixed(1) : "-"])); console.log(`\ntotal review load: ${x.hours.toFixed(0)} hours / 90 days`); } ``` ``` 180 changes, 36 known defect; two gates: schema + review escape skipped gated schema only review only both passed wrong drifted min/review no 13 167 1 13 11 11 7 4 286 yes 20 160 1 15 9 11 9 7 270 defect count schema review passed threshold 8 0 3 5 format 12 12 11 0 environment 11 0 9 2 runtime 5 0 1 4 environment gated review only wrongly held total hours hours/marginal staging 67 6 2 312.5 52.1 development 44 1 2 193.7 193.7 production 56 6 3 289.9 48.3 total review load: 796 hours / 90 days ``` Both gates do work, but their ledgers do not resemble each other. The schema validator held 12 format defects; review would have caught 11 of those too, so the number the **schema alone** holds is 1. Review held 24 defects and caught 13 of them by itself — these are the defects that would pass whether or not the other gate existed. The marginal-value ranking is clear: schema gate 1, review 13. The defect-class table shows why. All 12 format defects were held, none got through; both gates see this class, and the schema gate does it in seconds. Threshold and environment defects are invisible to the schema: the value is valid, the field is correct, only the number or the target is wrong. Only a human reading catches these — 3 of 8 threshold defects, 9 of 11 environment defects, were held. The 11 defects the gate misses also read by class: 5 threshold, 2 environment, 4 runtime. The runtime class let four of its five defects through, because this class cannot be seen by reading the written text — the value only turns out wrong once the service meets real data. This is a boundary standing in front of a static gate, and extending the review's duration does not push that boundary back. The wrongly-held column is the gate's other face: 7 clean changes were stopped, and each one produced a second round. The average wait per review is 286 minutes; over ninety days that is **796 hours** total, roughly a nine-hour wait load per day. This time is not a labor cost — it is lead time itself: after a change is written, it goes nowhere for an average of four hours and forty-six minutes. #### The Measured Trace of Escaping the Gate The table's second row models the behavior wrongful holding produces in the team: after a change is stopped for no reason, the change that follows it becomes more likely to go through the ungated path, even if it is not urgent. The change skipping the gate rises from 13 to 20, and the drifted item at period's end rises from 4 to 7. The number of defects that reach production is 11 in both runs; at the scale of 180 changes, this difference sits under the generator's noise and should be presented that way. The measurable effect is not in the defect count — it is in the **drift the gate produces itself**: when trust in a gate falls, the loss shows up first in traceability. The environment table, in turn, points directly at the gate that should be removed. 44 changes headed to the development environment were reviewed, those reviews totaled 193.7 hours, and **only 1** defect was caught marginally — 193.7 hours per defect. The same measure is 48.3 hours in production, 52.1 in staging. What is more, the 2 wrongful holds in the development environment are twice the marginal catch: here the gate stops more for nothing than it holds. The decision that follows does not touch the principle, it touches the gate: development-environment changes keep being committed to the repository, the review gate comes off. This distinction is the most practical conclusion this lesson carries — **committing to the repository and review are not the same thing.** The traceability, rationale, and reversibility payoff comes from the commit record; the only thing the gate contributes is the defect it holds, and that is priced very differently by environment. #### The Principle's Price: Extra Steps Per Change The first table's last column is the principle's direct bill. Manual application is 2 steps per change: a command is written, applied. The repository path is 7 steps — a branch is opened, the file is changed, committed, a proposal is opened, review is waited on, merged, applied. The mixed average is 6.75; the difference is 4.75 steps per change, 855 extra steps over ninety days. Not all of these steps carry equal weight. Six of them are the writer's own minutes; only one — the review wait — depends on someone else's calendar, and the entire measured 286 minutes comes from there. This is why the urgent path continues to exist: 13 of the 35 urgent changes skipped the gate, because a four-hour wait at midnight extends the outage. Applying the principle 100% brings the measured unrecorded-change rate down from 5 to 0, but adds four hours to the urgent intervention's duration; whichever side is chosen cannot be defended without measuring it. #### Summary - In the manual arrangement, 63 of the 180 changes (35%) were applied leaving no trace at all, and rationale existed for only 54 changes; in the repository arrangement, unrecorded changes fell to 5 and unjustified changes to 8. - Of the 56 items touched at period's end, 45 had drifted from the repository's value in the manual arrangement, versus 1 in the repository arrangement; the principle's first payoff is not auditing, it is reversibility. - The schema gate held 12 defects but caught only 1 of them by itself; review held 24, catching 13 by itself — the marginal-value ranking is 1 against 13. - The gate missed 11 defects (5 threshold, 2 environment, 4 runtime) and wrongly held 7 clean changes for nothing; it added 286 minutes per review, 796 hours over ninety days. - In the development environment, the gate caught 1 marginal defect in 193.7 hours and stopped 2 clean changes; the same measure is 48.3 hours in production. This gate is removed in the development environment; the commit record stays. - The principle's step cost rises from 2 to 6.75 per change; only one of the extra steps depends on someone else's calendar, and the entire measured wait comes from there. #### Next Step This lesson measured **where the desired state is written** and showed the value of the repository's writing through the drifted-item count. The question left open is the next lesson's subject: who carries that writing in the repository over to the cluster? In the arrangement so far, the carrier is the pipeline — the pipeline connects to the cluster and writes. There is an alternative: an agent running inside the cluster reads the repository and closes the gap itself. The items to measure are set: the pipeline's authority surface over the cluster, the detection delay of a manual change, the direction of the network connection, and what happens when the pipeline goes down. ### Pull-Based Deployment Source: https://academia.sh/en/courses/ci-cd/pull-based-deployment Two arrangements that carry the desired state from the repository to the cluster — the pipeline writing to the cluster and an in-cluster agent reading the repository — are counted on the same items: authority surface, connection direction, the detection delay of a manual drift, and what happens when the pipeline goes down. The reconciliation round's marginal value comes out to 17 drifts, and the genuine intervention it wrongly reverts to 9. The previous lesson counted what keeping the desired state in the repository buys: unrecorded changes fell from 63 to 5, drifted items at period's end from 45 to 1. One item stayed silent in that measurement — who carries the repository's writing over to the cluster. In every arrangement so far, the carrier has been the pipeline: the pipeline connects to the cluster's control plane, identifies itself with credentials, and writes objects. This is called **push**. There is a second way: a **reconciler agent** running inside the cluster reads the repository, compares it against its own environment's actual state, and closes the gap itself. This is called **pull**. The mechanics of the reconciliation loop were measured in the cluster architecture topic — the period, the lagging snapshot, the oscillation — and are not repeated here. The only thing this lesson measures is what **the source being a repository** changes across these two directions of transport. The measurement is again over the **regional measurement network** and is **fictional**; what runs is a **model** built with `node`. **RD7.** The model carries 4 services × 3 environments = 12 namespaces, 6 object kinds, 72 object instances, and 4 operations. **RD8.** In push, a single subject — the pipeline runner — is authorized across all 12 namespaces. In pull, each environment has one agent, authorized only in its own environment's 4 namespaces. **RD9.** 60 manual changes occur over thirty days; the generator is self-written, seed 31337. Three classes: the field the deployment writes on every run, the field the deployment never writes, and the object the agent does not own. **RD10.** A portion of the manual changes are **genuine** interventions that close a real outage; a portion are short-lived — the operator reverts them themselves within minutes. **RD11.** The deployment schedule is fixed in writing: the four services deploy every 720, 480, 2880, and 4320 minutes respectively. **RD12.** On the twelfth day there is a 24-hour pipeline outage; the deployment does not run during that window, the agent does. #### The Authority Surface and the Direction of the Connection ```js // authority.mjs — counts the authority surface in push and pull arrangements. No // randomness; the count runs over a model's object tree, not a real cluster. const SERVICE = ["collector", "verifier", "billing", "workorder"]; const ENV = ["development", "staging", "production"]; const OBJECT = ["deployment", "service", "config", "secret-claim", "scheduled-job", "network-rule"]; const WRITE = ["write", "delete", "patch"], OP = ["read", ...WRITE]; const namespace = ENV.flatMap((e) => SERVICE.map((s) => `${e}/${s}`)); const instance = namespace.flatMap((n) => OBJECT.map((o) => `${n}/${o}`)); // object instances const arrangement = { push: { // pipeline writes to the cluster subject: "pipeline runner", scope: namespace, exposed: ENV.length, // each environment's control plane is externally reachable pipelineHeld: ENV.length, // cluster identities the pipeline holds direction: "pipeline -> cluster (outside-in)", }, pull: { // agent reads the repository subject: "one agent per environment", scope: namespace.filter((n) => n.startsWith("production/")), exposed: 0, pipelineHeld: 0, direction: "cluster -> repository (inside-out)", }, }; const count = (k) => ({ authEntries: k.length * OBJECT.length * OP.length, writeEntries: k.length * OBJECT.length * WRITE.length, writable: instance.filter((o) => k.some((n) => o.startsWith(n + "/"))).length }); console.log(`model: ${SERVICE.length} services x ${ENV.length} environments = ${namespace.length} namespaces, ` + `${OBJECT.length} object kinds, ${instance.length} object instances, ${OP.length} operations`); console.log("\n" + "arrangement".padEnd(13) + "subject".padEnd(28) + "namespace".padStart(11) + "auth".padStart(6) + "write".padStart(7) + "writable if compromised".padStart(26)); for (const [ad, d] of Object.entries(arrangement)) { const s = count(d.scope); console.log(ad.padEnd(13) + d.subject.padEnd(28) + String(d.scope.length).padStart(11) + String(s.authEntries).padStart(6) + String(s.writeEntries).padStart(7) + String(s.writable).padStart(26)); } console.log("\n" + "arrangement".padEnd(13) + "exposed control planes".padStart(25) + "pipeline-held cluster identities".padStart(35) + " connection direction"); for (const [ad, d] of Object.entries(arrangement)) console.log(ad.padEnd(13) + String(d.exposed).padStart(25) + String(d.pipelineHeld).padStart(35) + " " + d.direction); ``` ``` model: 4 services x 3 environments = 12 namespaces, 6 object kinds, 72 object instances, 4 operations arrangement subject namespace auth write writable if compromised push pipeline runner 12 288 216 72 pull one agent per environment 4 96 72 24 arrangement exposed control planes pipeline-held cluster identities connection direction push 3 3 pipeline -> cluster (outside-in) pull 0 0 cluster -> repository (inside-out) ``` The table's first reading is misleading. In the pull arrangement, 96 authority entries appear per agent, versus 288 in push; but since there is one agent per environment, the total authority entries in pull are also 3 × 96 = **288**. The authority surface does not shrink, it **splits**. What changes is the last column: a compromised pipeline runner can write to all 72 object instances, a compromised agent only to its own environment's 24. The production environment, among the three, sits behind a separate subject, and a problem in the development environment cannot reach it. The second table shows a sharper difference. In push, three control planes must be externally reachable and the pipeline holds three cluster identities; in pull, both are zero, because the cluster establishes the connection. The operational counterpart of this is concrete: in push, wherever the pipeline runs has to be treated as if it were inside the security boundary — the pipeline runner is a subject that can write to three production environments. In pull, the pipeline's only authority is to write to the repository; the clusters do not even recognize it. This is not a gate count, but it determines the ground the gates stand on: in push, the repository-defined desired state principle is a **contract** — if it wants to, the pipeline can bypass the gates and write directly. In pull, the principle is a **constraint**: there is no authority to write from outside the cluster. #### Who Closes the Drift, and When ```js // drift.mjs — a 30-day manual-change flow is counted in two arrangements: push (pipeline // writes to cluster) and pull (agent reads repository). Model runs; generator self-written, seed visible. const generator = (t) => { let s = t >>> 0; // same linear congruential generator return () => (s = (s * 1664525 + 1013904223) >>> 0) / 4294967296; }; const DURATION = 30 * 1440, OUTAGE = [17820, 19260]; // day 12, a 24-hour pipeline outage const SCHEDULE = { collector: 720, verifier: 480, billing: 2880, workorder: 4320 }; const SERVICE = Object.keys(SCHEDULE); function events(seed = 31337, count = 60) { const r = generator(seed), l = []; for (let i = 0; i < count; i++) { const p = r(); l.push({ t: Math.floor(r() * DURATION), service: SERVICE[Math.floor(r() * 4)], class: p < 0.45 ? "written" : p < 0.85 ? "not-written" : "out-of-scope", genuine: r() < 0.2, // a fix that closes a real outage lifespan: r() < 0.25 ? 5 + Math.floor(r() * 25) : Infinity });// reverted by the operator themselves } return l; } const nextDeploy = (service, t) => { // the outage window is skipped const p = SCHEDULE[service]; for (let k = Math.floor(t / p) + 1; k * p <= DURATION; k++) if (!(k * p >= OUTAGE[0] && k * p < OUTAGE[1])) return k * p; return null; }; function run(l, U) { // U=null means push arrangement const s = { closed: 0, marginal: 0, delay: 0, unseen: 0, missed: 0, wrong: 0, remaining: 0, windowOpen: 0, windowClosed: 0 }; for (const o of l) { const closesAt = U === null ? (o.class === "written" ? nextDeploy(o.service, o.t) : null) : (o.class === "out-of-scope" ? null : (Math.floor(o.t / U) + 1) * U); if (o.lifespan !== Infinity && (closesAt === null || o.t + o.lifespan < closesAt)) { s.unseen++; continue; } const end = (closesAt === null || closesAt > DURATION) ? DURATION : closesAt; if (o.t < OUTAGE[1] && end > OUTAGE[0]) s.windowOpen++; // open drift throughout the outage if (closesAt !== null && closesAt >= OUTAGE[0] && closesAt < OUTAGE[1]) s.windowClosed++; if (closesAt === null || closesAt > DURATION) { s.remaining++; if (o.class === "out-of-scope") s.missed++; continue; } s.closed++; s.delay += closesAt - o.t; if (o.class === "not-written") s.marginal++; // the class push can never close if (o.genuine) s.wrong++; // a genuine fix reverted/overwritten } s.avgDelay = s.closed ? s.delay / s.closed : 0; s.propagation = U === null ? 3 : U / 2 + 1; // commit -> apply lag return s; } const W = [13, 8, 10, 11, 8, 8, 7, 11, 13]; const row = (h) => h.map((v, j) => j ? String(v).padStart(W[j]) : String(v).padEnd(W[0])).join(""); const l = events(); console.log(`seed 31337 | ${l.length} manual changes / 30 days | ` + `${l.filter((o) => o.genuine).length} are genuine fixes closing a real outage`); console.log(`class: written ${l.filter((o) => o.class === "written").length}, ` + `not-written ${l.filter((o) => o.class === "not-written").length}, ` + `out of scope ${l.filter((o) => o.class === "out-of-scope").length}`); console.log("\n" + row(["arrangement", "closed", "marginal", "avg delay", "unseen", "missed", "wrong", "remaining", "propagation"])); console.log(row(["push", ...(({ closed, marginal, avgDelay, unseen, missed, wrong, remaining, propagation }) => [closed, marginal, avgDelay.toFixed(0), unseen, missed, wrong, remaining, propagation])(run(l, null))])); for (const U of [1, 5, 15, 60, 240]) { const x = run(l, U); console.log(row([`pull U=${U}`, x.closed, x.marginal, x.avgDelay.toFixed(0), x.unseen, x.missed, x.wrong, x.remaining, x.propagation.toFixed(1)])); } const missedDeploy = SERVICE.reduce((a, sv) => a + Object.keys([...Array(DURATION / SCHEDULE[sv] | 0)]) .filter((k) => (+k + 1) * SCHEDULE[sv] >= OUTAGE[0] && (+k + 1) * SCHEDULE[sv] < OUTAGE[1]).length, 0); const i = run(l, null), c = run(l, 15); console.log(`\n24-hour pipeline outage: ${missedDeploy} scheduled deploys missed, ` + `agent ran ${(OUTAGE[1] - OUTAGE[0]) / 15} rounds`); console.log(`open drift throughout the window — push ${i.windowOpen}, pull(U=15) ${c.windowOpen}; ` + `closed within the window — push ${i.windowClosed}, pull ${c.windowClosed}`); ``` ``` seed 31337 | 60 manual changes / 30 days | 10 are genuine fixes closing a real outage class: written 32, not-written 17, out of scope 11 arrangement closed marginal avg delay unseen missed wrong remaining propagation push 26 0 1206 16 6 4 18 3 pull U=1 49 17 1 5 6 9 6 1.5 pull U=5 49 17 3 5 6 9 6 3.5 pull U=15 48 17 8 6 6 9 6 8.5 pull U=60 42 14 30 12 6 8 6 31.0 pull U=240 39 12 135 15 6 7 6 121.0 24-hour pipeline outage: 5 scheduled deploys missed, agent ran 96 rounds open drift throughout the window — push 11, pull(U=15) 7; closed within the window — push 0, pull 3 ``` The push row does not look like a gate, because it is not one. 26 of the sixty manual changes closed, but closing is not a matter of noticing — it is an **overwrite**: the drift is erased because the next deployment rewrites that field. The average duration is 1206 minutes — twenty hours — and the entire duration depends on the deployment schedule. No one noticed the drift, no one kept a record; 18 items remained drifted at period's end. The reconciliation round's gate ledger stands against this. In a fifteen-minute period, 48 drifts closed, with an average delay of 8 minutes. The **marginal** column gives the work this gate does only by itself: 17 drifts in fields the deployment never writes at all. These are items that, under the push arrangement, would never close over thirty days — a manually added network rule, a manually enlarged replica count, a manually disabled scheduled job. The gate's value is not in the 48, it is in the **17**. What it misses is just as clear: 6 drifts in object kinds the agent does not own were missed at every period, and shrinking the period did not change this. Scope is not a setting, it is a definition; the agent cannot see what it does not manage. The second class of misses is the `unseen` column: changes made between two rounds and reverted by the operator themselves. 5 of them were never seen at U=1, 15 at U=240. An agent with a long period is not merely slow; it **keeps no record at all** of short-lived interventions. #### What the Reconciliation Round Wrongly Holds The `wrong` column is this gate's price. Over thirty days, 10 manual changes are genuine interventions closing a real outage. In the pull arrangement, 9 of these were reverted within an average of 9 minutes — the agent looks at the repository's writing, does not know why the intervention was made, and brings the outage back. In push, the same number is 4, and the reversion happens silently, thirty-four hours later on average. Both are wrongful holding, but their forms differ, and this is where the confusion starts. In pull, the wrongful hold is **fast and visible**: the agent leaves a record, the person who made the intervention sees it reverted within nine minutes and learns the correct behavior — commit the fix to the repository. In push, the wrongful hold is **slow and silent**: the intervention runs for thirty-four hours, then a deployment overwrites it, and no one connects the two. The gate's count is twice as high in pull; the harm the gate does is greater in push, because it does not teach. The correct tuning for this gate is not lengthening the period. In the U=240 row, wrongful reversion falls from 9 to 7, but marginal value also drops from 17 to 12 and unseen drift rises to 15. The correct tuning for the gate is **scope**: if the fields where urgent intervention happens are moved into the set the agent does not own, both wrongful reversion and learning disappear. The measurement gives this trade-off's price — the drifts in those fields are now added to the 6 items no one closes at all. #### When the Pipeline Goes Down, and the Price of Pull The 24-hour pipeline outage separates the two arrangements' resilience. 5 scheduled deploys were missed in that window; in the push arrangement, **none** of the 11 drifts left open throughout the window closed, because the closing mechanism was the deployment itself. The agent ran 96 rounds in the same window and closed 3 of the 7 open drifts. The distinction is this: in push, the pipeline both carries the new change and closes drift, so the pipeline's outage stops both at once. In pull, the pipeline only carries the new change; **preserving the last known state is independent of the pipeline.** New releases stop in both arrangements — this is not an advantage, it is a separation of concerns. Pull's bill is in the last column. In push, a commit turning into an application takes 3 minutes; in pull it is half the period plus application, i.e. 8.5 minutes at U=15 and 121 minutes at U=240. Keeping the reconciliation round's marginal value high requires shortening the period, but a short period improves propagation while placing a constant read load on the repository and the control plane. The second item is the agent itself: one agent per environment, three separate upgrade windows, three separate sets of authority entries, and a separate watch to notice when the agent itself goes down. The repository-defined desired state principle does not come bundled with the agent; **the agent is only one of the ways to implement the principle**, and it demands its own upkeep. #### Summary - Total authority entries are 288 in both arrangements; in pull, authority splits across three subjects, and the object instances a compromised subject can write to fall from 72 to 24. Externally exposed control planes drop from 3 to 0. - In push, drift goes unnoticed, it gets overwritten: 26 of 60 manual changes were erased after an average of 1206 minutes, and 18 were still standing at period's end. - The reconciliation round's marginal value is 17 drifts — fields the deployment never writes. The gate's value is not in the 48 it closes, it is in the 17 push could never close. - The gate misses two classes: 6 objects the agent does not own (independent of period), and changes made and reverted between two rounds (5 at U=1, 15 at U=240). - Pull wrongly reverted a genuine emergency intervention 9 times in an average of 9 minutes; push silently overwrote it 4 times after thirty-four hours; pull's wrongful hold is more frequent, push's is more harmful because it does not teach. - In the 24-hour pipeline outage, push closed 0 of 11 open drifts, the agent closed 3 of 7; the price is propagation delay rising from 3 minutes to 8.5 minutes, and the upkeep of one agent per environment. #### Next Step Two lessons measured the desired state staying in the repository and being carried from there to the cluster. One item remains, and its entry into the repository is problematic from the start: the secrets services need. The database connection, the credentials going out to the measurement network's remote endpoints, and the signing keys cannot sit in the repository as plain text — but if they do not sit in the repository, the desired state stays incomplete and the agent cannot apply it. The next lesson measures the sealed secret: where the key sits, how many subjects can decrypt it, how many records get re-encrypted and how many files get touched when the key is rotated, and how long a stale encrypted value remains readable. ### Secret Management Source: https://academia.sh/en/courses/ci-cd/secret-management When secrets are kept sealed in the repository, where the key sits, how many subjects can decrypt how many records, and how many records get re-encrypted and how many files get touched when the key is rotated, are counted with real encryption. A sealed record's staying in the past is measured: if the old key leaks, 24 versions in history decrypt and 11 are today's value. The previous two lessons measured the desired state staying in the repository and being carried from there to the cluster. Both silently left one item out: the **secrets** the services need. The credentials the collector carries to connect to the measurement network's endpoints, billing's database password, the work-order service's archive key — these are part of the desired state, but they cannot be put into the repository as plain text. If they are not put into the repository, the desired state stays incomplete and the reconciler agent cannot apply it. The shape of the solution is set: the secret sits **sealed** in the repository, and decrypt authority is managed on a separate axis. What this lesson measures is that axis — where the key sits, how many subjects can decrypt how many records, what happens when the key is rotated, and what the **history** of a sealed record staying in the repository means. Which paths a secret leak can take through was scanned earlier — log lines, environment variables, build-output layers — and that scan is not repeated here. The measurement is over the **regional measurement network** and is **fictional**. Encryption runs for real with `node:crypto`; **all secret values are fabricated** and written with a `fixture-` prefix. **RD13.** The inventory is 33 secret records: four services, three environments, two to three secret types per service. **RD14.** Encryption is AES-256-GCM, the key is derived with `scrypt`. The initialization vector is derived from the record ID so the output is reproducible; in a real deployment it is generated randomly. **RD15.** Three key layouts are compared: single key, per-environment key, per-env-service key. **RD16.** There are 29 subjects that can read the repository; 14 of them carry no key at all. Each subject's needed (environment, service) pairs are fixed in writing. **RD17.** Over a 180-day history, each record changed one to four times; each change sits in a separate commit. **RD18.** Production on-call needs two services' secrets; there are 24 night events over 180 days, and every event left without a key adds 35 minutes of escalation. #### Where the Key Sits, Who Can Decrypt ```js // secret.mjs — sealed secret record in the repository: inventory, key layout, decrypt // authority. SECRET VALUES ARE FICTIONAL, no real secret is written. Encryption runs // for real with node:crypto. The IV is derived from the record ID and key version so the output is reproducible. import { scryptSync, createCipheriv, createDecipheriv, createHash } from "node:crypto"; export const ENV = ["development", "staging", "production"]; export const SERVICE = { collector: ["database", "measurement-network-token", "cache"], verifier: ["database", "signing"], billing: ["database", "payment-token", "signing"], workorder: ["database", "notification-token", "archive"] }; export const RECORD = ENV.flatMap((e) => Object.entries(SERVICE).flatMap(([s, tl]) => tl.map((t) => ({ id: `${e}/${s}/${t}`, env: e, service: s, value: `fixture-${t}-${s}-${e}-0001` })))); // fictional value, not a real secret export const counter = { derive: 0, encrypt: 0, decrypt: 0 }; const keyStore = new Map(); export const key = (name, version = 1) => { // password is fictional too const k = `${name}#${version}`; if (!keyStore.has(k)) { counter.derive++; keyStore.set(k, scryptSync(`fixture-password-${k}`, `salt-${name}`, 32, { N: 16384 })); } return keyStore.get(k); }; export const LAYOUT = { single: () => "all", env: (k) => k.env, "env-service": (k) => `${k.env}/${k.service}` }; export const encrypt = (record, name, version, value = record.value) => { const iv = createHash("sha256").update(`${record.id}#${name}#${version}#${value}`).digest().subarray(0, 12); const c = createCipheriv("aes-256-gcm", key(name, version), iv); const g = Buffer.concat([c.update(value, "utf8"), c.final()]); counter.encrypt++; return { iv, body: g, tag: c.getAuthTag(), name, version }; }; export const decrypt = (k) => { const d = createDecipheriv("aes-256-gcm", key(k.name, k.version), k.iv); d.setAuthTag(k.tag); counter.decrypt++; return Buffer.concat([d.update(k.body), d.final()]).toString("utf8"); }; // subjects: each needs specific (env, service) pairs. the repo reader has no keys. export const SUBJECT = [ { name: "agent/development", n: 1, need: [["development", "*"]] }, { name: "agent/staging", n: 1, need: [["staging", "*"]] }, { name: "agent/production", n: 1, need: [["production", "*"]] }, { name: "measurement team", n: 6, need: [["development", "collector"], ["development", "verifier"], ["staging", "collector"]] }, { name: "billing team", n: 4, need: [["development", "billing"], ["staging", "billing"]] }, { name: "production on-call", n: 2, need: [["production", "billing"], ["production", "collector"]] }, { name: "repo reader", n: 14, need: [] }, ]; const matches = (k, [e, s]) => k.env === e && (s === "*" || k.service === s); export const needed = (z) => RECORD.filter((k) => z.need.some((c) => matches(k, c))); export function authorize(layout) { // records decryptable under a layout const f = LAYOUT[layout]; return SUBJECT.map((z) => { const carries = new Set(needed(z).map(f)); // key names granted to the subject return { z, decryptable: RECORD.filter((k) => carries.has(f(k))), need: needed(z).length }; }); } if (import.meta.url === `file://${process.argv[1]}`) { const subjectCount = SUBJECT.reduce((a, z) => a + z.n, 0); console.log(`inventory: ${RECORD.length} secret records (fictional values), ${subjectCount} repo-reader subjects`); const sample = encrypt(RECORD[0], "development", 1); console.log(`sample record ${RECORD[0].id} -> ${sample.body.toString("base64")} ` + `| decrypted: ${decrypt(sample)}`); const B = [13, 6, 15, 12, 9, 9, 16]; const yz = (h) => h.map((v, j) => j ? String(v).padStart(B[j]) : String(v).padEnd(B[0])).join(""); console.log("\n" + yz(["layout", "keys", "decrypt pairs", "need pairs", "surplus", "blocked", "widest subject"])); for (const layout of Object.keys(LAYOUT)) { const y = authorize(layout); const keys = new Set(RECORD.map(LAYOUT[layout])).size; const pair = y.reduce((a, x) => a + x.z.n * x.decryptable.length, 0); const need = y.reduce((a, x) => a + x.z.n * x.need, 0); for (const x of y) if (x.decryptable.length) decrypt(encrypt(x.decryptable[0], LAYOUT[layout](x.decryptable[0]), 1)); console.log(yz([layout, keys, pair, need, pair - need, subjectCount * RECORD.length - pair, Math.max(...y.map((x) => x.decryptable.length))])); } console.log(`\noperation counter: ${counter.derive} key derivations, ` + `${counter.encrypt} encryptions, ${counter.decrypt} decryptions`); } ``` ``` inventory: 33 secret records (fictional values), 29 repo-reader subjects sample record development/collector/database -> 2w41e6WkaaQMnq/d3iUgX3C7n5OAdHW0TUpYFVHQUBKjJ0HWGhgaAridVQ== | decrypted: fixture-database-collector-development-0001 layout keys decrypt pairs need pairs surplus blocked widest subject single 1 495 117 378 462 33 env 3 275 117 158 682 22 env-service 12 117 117 0 840 11 operation counter: 8 key derivations, 19 encryptions, 19 decryptions ``` Encryption genuinely ran: the sample record was encrypted, decrypted back, and the fixture value returned unchanged. The real table is below, and it answers a single question three times — **how many subjects can decrypt how many records.** The unit of measure is the (subject, record) pair; with 29 subjects and 33 records, the largest possible value is 957. The key does not sit inside the repository — if it did, encryption would lose its meaning. The key sits in the subjects' hands, and the table counts that distribution's result. In the single-key layout, the 15 subjects carrying the key can decrypt all 33 records: 495 pairs. The number of pairs these subjects actually need is **117**, and this number is the same across all three layouts, because need is independent of the key layout. The 378 pairs in between are pure surplus — authority no one asked for, granted only because the key is coarse. When switching to a per-environment key, surplus falls from 378 to 158; in the per-env-service key, it is **zeroed out**, and a subject that can decrypt, decrypts exactly what it needs. The widest subject's authority also drops from 33 to 11: the number of records a single person's key opens shrinks threefold. The price of this is the key count — from 1 to 12 — and that number is the subject of the next section. #### The Cost of Key Rotation and the Record Left in the Past ```js // rotate.mjs — the cost of key rotation and how a sealed record stays in the past. // Historical encrypted values are genuinely decrypted; the values are fictional. import { createHash } from "node:crypto"; import { RECORD, LAYOUT, SUBJECT, authorize, encrypt, decrypt, counter } from "./secret.mjs"; const FILE = (k) => `${k.env}/${k.service}/secrets`; // records are spread across files like this const times = (k) => 1 + (createHash("sha256").update(k.id).digest()[0] % 4); const value = (k, i) => `fixture-${k.id.split("/")[2]}-${k.service}-${k.env}-` + String(i).padStart(4, "0"); // 180-day history: each record changed times(k) times; each change sits in a separate commit. const history = RECORD.flatMap((k) => Array.from({ length: times(k) }, (_, i) => ({ record: k, plain: value(k, i + 1) }))); const current = new Map(RECORD.map((k) => [k.id, value(k, times(k))])); const W = [13, 14, 17, 15, 15]; const row = (h) => h.map((v, j) => j ? String(v).padStart(W[j]) : String(v).padEnd(W[0])).join(""); console.log(`history: ${history.length} encrypted record versions / ${RECORD.length} records, ` + `${new Set(RECORD.map(FILE)).size} files`); console.log("\n" + row(["layout", "keys", "single rotation", "files", "rotate all"])); for (const [layout, f] of Object.entries(LAYOUT)) { const target = f(RECORD.find((k) => k.env === "production" && k.service === "billing")); const affected = RECORD.filter((k) => f(k) === target); console.log(row([layout, new Set(RECORD.map(f)).size, affected.length, new Set(affected.map(FILE)).size, `${RECORD.length} / ${new Set(RECORD.map(FILE)).size} files`])); } // in the env layout, the production key is rotated 1 -> 2. Current records are re-encrypted. const f = LAYOUT.env; const production = RECORD.filter((k) => k.env === "production"); const historyEncrypted = history.map((g) => ({ ...g, box: encrypt(g.record, f(g.record), 1, g.plain) })); const rotated = production.map((k) => encrypt(k, f(k), 2, current.get(k.id))); console.log(`\nproduction key 1 -> 2: ${rotated.length} records re-encrypted, ` + `${new Set(production.map(FILE)).size} files touched, ${new Set(production.map(FILE)).size} new commits`); // what the history yields if the old key leaks const old = historyEncrypted.filter((g) => g.box.name === "production" && g.box.version === 1); const opened = old.map((g) => decrypt(g.box)); const correct = opened.every((a, i) => a === old[i].plain); const stillCurrent = old.filter((g, i) => opened[i] === current.get(g.record.id)).length; const idx = old.findIndex((g, i) => opened[i] === current.get(g.record.id)); console.log(`if the old key (version 1) leaks: ${old.length} versions in history decrypt ` + `(all correct: ${correct}); ${stillCurrent} of these are the TODAY'S value of ${production.length} records`); console.log(`example: ${old[idx].record.id} from an old commit -> ${opened[idx]}`); // what the decrypt-authority gate wrongly holds: events left unkeyed during the night shift const generator = (t) => { let s = t >>> 0; return () => (s = (s * 1664525 + 1013904223) >>> 0) / 4294967296; }; const r = generator(90210), SRV = ["collector", "verifier", "billing", "workorder"]; const event = Array.from({ length: 24 }, () => SRV[Math.floor(r() * 4)]); const ONCALL = ["billing", "collector"]; // what production on-call needs console.log("\n" + row(["layout", "night events", "unkeyed", "extra minutes", "blocked pairs"])); for (const layout of Object.keys(LAYOUT)) { const carries = (s) => layout === "env-service" ? ONCALL.includes(s) : true; const missing = event.filter((s) => !carries(s)).length; const pair = authorize(layout).reduce((a, x) => a + x.z.n * x.decryptable.length, 0); console.log(row([layout, event.length, missing, missing * 35, SUBJECT.reduce((a, z) => a + z.n, 0) * RECORD.length - pair])); } console.log(`\noperation counter: ${counter.derive} key derivations, ${counter.encrypt} encryptions, ` + `${counter.decrypt} decryptions`); ``` ``` history: 76 encrypted record versions / 33 records, 12 files layout keys single rotation files rotate all single 1 33 12 33 / 12 files env 3 11 4 33 / 12 files env-service 12 3 1 33 / 12 files production key 1 -> 2: 11 records re-encrypted, 4 files touched, 4 new commits if the old key (version 1) leaks: 24 versions in history decrypt (all correct: true); 11 of these are the TODAY'S value of 11 records example: production/collector/database from an old commit -> fixture-database-collector-production-0002 layout night events unkeyed extra minutes blocked pairs single 24 0 0 462 env 24 0 0 682 env-service 24 11 385 840 operation counter: 4 key derivations, 87 encryptions, 24 decryptions ``` The rotation table gives the second face of key count. In the single-key layout, rotating one key re-encrypts all 33 records and touches 12 files; in the env-service layout, the same operation is 3 records and 1 file. But the last column honestly shows the trade-off: rotating **all** keys is 33 records and 12 files in all three layouts. Fine-grained key partitioning does not reduce total work, it narrows the echo of a single leak. When the production key was rotated from 1 to 2, 11 records were re-encrypted, 4 files were touched, and 4 new commits were produced. Everything up to here is expected. This lesson's core is in the next line, and it is not a guess, it is a **decrypt that was actually run**: if the old key leaks, all of the 24 encrypted versions in history decrypt — the `all correct: true` line shows this was tested — and **11 of these 24 versions are the today's value of the 11 production records.** The result is: rotating the key does not close off the past. The new commit holds the new encrypted value, but the old commit still holds the old encrypted value as it was, and when that value is opened with the old key, it yields **a secret that is still valid today.** The example line makes this concrete: the value decrypted from the collector's production database record's old commit is the same value the service running today uses. The only operation that actually closes off the past is changing the secret's **value** at its source — rotating the password in the database, revoking the token on the measurement network's side. Key rotation touches 4 files; value rotation touches every consumer, and the 4 files on the repository side are the cheapest part of that work. #### The Decrypt-Authority Gate's Ledger The gate in this topic is not a review or a reconciliation round, it is a **decrypt authority.** Its ledger reads in three items. **What it holds, and holds only by itself.** In the per-environment key layout, the blocked (subject, record) pairs number 682. The entirety of this number is marginal, because all 29 subjects can already read the repository — the repository-access gate stops none of these pairs. If encryption were removed, 682 reads would become free; even under a coarse layout, the gate does work entirely on its own. In the env-service layout, the same number rises to 840. **What it misses.** There are two classes. The first is the past: 24 encrypted versions can be read through a path the gate never enters at all, because there is no authority check in place when the old key leaks. The second is surplus: in the env layout, 158 pairs open more than the legitimately key-carrying subjects need. The gate does not hold these, it lets them through openly — the measurement team's 6 members can also decrypt the development environment's billing secrets. **What it wrongly holds, and the time it adds.** The last table closes the trade-off. The env-service layout zeroes out the surplus, but in 11 of the 24 night events the production on-call did not carry the key it needed, and each event added a 35-minute escalation: 385 minutes. In the env layout this number is zero, in exchange for 158 open surplus pairs. The decision is made between these two numbers, and the sentence "a finer-grained key is more secure" does not carry the decision by itself. The gate's operating load is also counted: the 180-day history of 33 records costs 87 encryption operations, its verification costs 24 decrypt operations; every secret change asks for one encryption, every reconciliation round asks the agent for one decryption. #### Summary - Of the 957 possible (subject, record) pairs from 29 subjects and 33 records, the single-key layout opens 495, the per-environment key 275, the per-env-service key 117; need is 117 in all three layouts. - Surplus authority falls from 378 to 158, then to 0; the records a compromised widest subject can decrypt fall from 33 to 11. - Rotating one key is 33 records and 12 files in the single-key layout, 3 records and 1 file in the env-service layout; but rotating all keys is 33 records and 12 files in all three layouts. - After the production key is rotated, if the old key leaks, 24 encrypted versions in history decrypt, and 11 of them are the today's value of 11 production records — key rotation does not close off the past, value rotation does. - The decrypt-authority gate's marginal value is 682 blocked pairs in the env layout, and the entirety of it is marginal, because all 29 subjects can already read the repository. - The gate's price was measured: in the env-service layout, 11 of 24 night events were left without a key and added 385 minutes of escalation; in the env layout, delay is 0, open surplus is 158 pairs. #### Next Step Secrets can now stay sealed in the repository, and who can decrypt what is countable. This was a question about the **content** of the repository's desired state. The next lesson turns to another face of content: what components a release actually consists of. An application is not made up of its own code alone — direct and transitive dependencies, runtime libraries, base-layer content. What gets measured is this inventory's coverage: how many dependencies show up, how many are transitive, how long producing the inventory takes, and how many components go entirely uncounted in a pipeline that does not produce an inventory. ### Software Bill of Materials Source: https://academia.sh/en/courses/ci-cd/software-bill-of-materials The components inside a build artifact are produced as an inventory and measured with three numbers: the direct, transitive, and build-time components the inventory sees, the three classes it never sees at all, and how many queries the response takes when a defect advisory arrives. The previous lesson measured that a secret can stay sealed in the repository: the key sits in one place, the subjects that can decrypt it are counted, rotation rewrites a fixed number of records. The configuration entering the repository is now declared — but what is declared is only the configuration. **What is inside** the output the pipeline produces is still written down nowhere. This lesson produces that declaration. The measure is three numbers: how many components the inventory sees, how many it can never see at all, and how many queries the response takes when a defect advisory arrives. M22/K02 counted the defects scanning caught and missed; that count is not repeated. The object here is the inventory itself — as **an output the pipeline produces**. **RD19.** The fictional regional measurement network is four services and shares a pool of 64 components; the tree is written to disk with `node` and is a model. **RD20.** The inventory generator reads only the manifest files: manifest, resolved, and build. **RD21.** There are three undeclared paths — vendored into source, embedded in the binary, fetched at runtime; the generator's seed `20260214` is visible. #### Producing the Inventory ```js // measurement-network/inventory.mjs — a fictional dependency tree is written to disk, inventory is produced import { mkdirSync, rmSync, writeFileSync, readFileSync, statSync } from "node:fs"; import { join, dirname } from "node:path"; const ROOT = join(process.env.TMPDIR ?? "/tmp", "measurement-network-inventory"); rmSync(ROOT, { recursive: true, force: true }); const generator = (seed) => () => (seed = (seed * 1103515245 + 12345) % 2147483648) / 2147483648; const r = generator(20260214); // seed visible const pick = (d) => d[Math.floor(r() * d.length)]; // Fictional regional measurement network: four services. The pool is shared, the same component appears in multiple services. const POOL = Array.from({ length: 64 }, (_, i) => `component-${String(i + 1).padStart(2, "0")}`); const VERSION = ["1.4", "2.0", "2.3", "2.6", "3.1", "3.4"]; const SERVICE = [ // vendored/embedded/runtime: undeclared components { name: "collector", direct: 9, tool: 5, vendored: 2, embedded: 1, runtime: 1, ownCopy: 1 }, { name: "verifier", direct: 7, tool: 4, vendored: 1, embedded: 1, runtime: 0, ownCopy: 1 }, { name: "billing", direct: 12, tool: 6, vendored: 2, embedded: 2, runtime: 2, ownCopy: 1 }, { name: "workorder", direct: 6, tool: 4, vendored: 1, embedded: 0, runtime: 1, ownCopy: 0 }, ]; const write = (path, content) => { mkdirSync(dirname(path), { recursive: true }); writeFileSync(path, content); }; for (const s of SERVICE) { const taken = new Set(); const fresh = () => { let a; do { a = pick(POOL); } while (taken.has(a)); taken.add(a); return a; }; const direct = {}, resolved = {}, tools = {}; for (let i = 0; i < s.direct; i++) { const name = fresh(); resolved[name] = { version: (direct[name] = pick(VERSION)), requiredBy: "" }; for (let k = 0, n = Math.floor(r() * 4); k < n; k++) // transitive children resolved[fresh()] = { version: pick(VERSION), requiredBy: name }; } for (let i = 0; i < s.tool; i++) tools[fresh()] = pick(VERSION); write(join(ROOT, s.name, "manifest.json"), JSON.stringify({ service: s.name, direct }, null, 1)); write(join(ROOT, s.name, "resolved.json"), JSON.stringify(resolved, null, 1)); write(join(ROOT, s.name, "build.json"), JSON.stringify({ tools }, null, 1)); for (let i = 0; i < s.vendored; i++) // third party vendored into source write(join(ROOT, s.name, "source", `k${i}.js`), `// vendored: ${fresh()}@${pick(VERSION)}\n`); for (let i = 0; i < s.ownCopy; i++) // own code, not third party write(join(ROOT, s.name, "source", `i${i}.js`), `// vendored: internal-library-measurement@1.0\n`); for (let i = 0; i < s.embedded; i++) // embedded inside the binary write(join(ROOT, s.name, "output", "embedded", `g${i}.bin`), Buffer.concat( [Buffer.from(`EMBEDDED:${fresh()}@${pick(VERSION)}\n`), Buffer.alloc(2048, "x")])); const fetched = Array.from({ length: s.runtime }, () => `${fresh()}@${pick(VERSION)}`); write(join(ROOT, s.name, "output", "downloads.json"), JSON.stringify({ fetched }, null, 1)); write(join(ROOT, s.name, "output", "body.js"), `// ${s.name} build output\n`); } // The inventory generator reads ONLY the declared manifests; files and bytes read are counted. let file = 0, byte = 0; const read = (y) => { file++; byte += statSync(y).size; return JSON.parse(readFileSync(y)); }; const inventory = []; for (const s of SERVICE) { const manifest = read(join(ROOT, s.name, "manifest.json")).direct; const resolved = read(join(ROOT, s.name, "resolved.json")); for (const [name, k] of Object.entries(resolved)) inventory.push({ service: s.name, name, version: k.version, kind: name in manifest ? "direct" : "transitive", requiredBy: k.requiredBy }); for (const [name, ver] of Object.entries(read(join(ROOT, s.name, "build.json")).tools)) inventory.push({ service: s.name, name, version: ver, kind: "build", requiredBy: "(build)" }); } const inventoryPath = join(ROOT, "inventory.json"); write(inventoryPath, JSON.stringify(inventory, null, 1)); write(join(ROOT, "cost.json"), JSON.stringify({ file, byte })); const count = (t) => inventory.filter((k) => k.kind === t).length; console.log(`seed 20260214; services ${SERVICE.length}, pool ${POOL.length} components`); console.log(`manifest files read to produce the inventory ${file}, ${byte} bytes`); console.log(`inventory records ${inventory.length} -> direct ${count("direct")}, ` + `transitive ${count("transitive")}, build time ${count("build")}`); console.log(`inventory file ${statSync(inventoryPath).size} bytes, ` + `unique components ${new Set(inventory.map((k) => k.name)).size}`); ``` ``` seed 20260214; services 4, pool 64 components manifest files read to produce the inventory 12, 7674 bytes inventory records 107 -> direct 34, transitive 54, build time 19 inventory file 13511 bytes, unique components 56 ``` The four services declare **34 direct dependencies** in total. The inventory produces 107 records: alongside that 34 come **54 transitive** components and **19 build-time tools**. The ratio between what is declared and what is carried is right here — the team writes 34 names, 88 components enter the output. A gate that looks only at direct dependencies sees **34** of the declared 88 runtime components (39%). The cost of production is written in quantities independent of the environment: **12 manifest files, 7,674 bytes read, 107 records, a 13,511-byte inventory.** Duration is not printed: duration depends on the machine, not on the number of files read. Behind the 107 records stand **56 unique components**: the same component appears across multiple services. The ability to answer "which services are affected" from a single file rests on this. #### What the Inventory Does Not See The generator reads the manifest files. A manifest describes **intent**: what was meant to be pulled in. The output tree, in turn, carries the **result**. The two do not have to be the same. **RD22.** Build-time tools do not run in production; they do not enter the runtime component set. **RD23.** The auditor walks the tree and counts every file carrying a `vendored:` or `EMBEDDED:` header as undeclared; a copy of its own code carries the same header too. ```js // measurement-network/coverage.mjs — what the inventory does not see: the output tree is walked, compared against the manifest import { readdirSync, readFileSync, writeFileSync, statSync } from "node:fs"; import { join } from "node:path"; const ROOT = join(process.env.TMPDIR ?? "/tmp", "measurement-network-inventory"); const inventory = JSON.parse(readFileSync(join(ROOT, "inventory.json"))); const cost = JSON.parse(readFileSync(join(ROOT, "cost.json"))); // what the generator read const SERVICE = ["collector", "verifier", "billing", "workorder"]; const walk = (root, sub = "") => readdirSync(join(root, sub), { withFileTypes: true }) .flatMap((g) => g.isDirectory() ? walk(root, join(sub, g.name)) : [join(sub, g.name)]); // The auditor reads not the manifest files, but the tree itself. let walkedFile = 0, walkedByte = 0; const finding = []; for (const s of SERVICE) { for (const path of walk(join(ROOT, s))) { if (path.endsWith(".json")) continue; // manifest files are not the auditor's input const full = join(ROOT, s, path); walkedFile++; walkedByte += statSync(full).size; const head = readFileSync(full).subarray(0, 120).toString("latin1"); const k = head.match(/vendored: (\S+)@(\S+)/); if (k) finding.push({ service: s, name: k[1], version: k[2], how: "vendored into source" }); const g = head.match(/EMBEDDED:(\S+)@(\S+)/); if (g) finding.push({ service: s, name: g[1], version: g[2], how: "embedded in binary" }); } const { fetched } = JSON.parse(readFileSync(join(ROOT, s, "output", "downloads.json"))); for (const record of fetched) { const [name, version] = record.split("@"); finding.push({ service: s, name, version, how: "fetched at runtime" }); } } // A declared runtime component: build tools do not run in production, they are not in the runtime set. const declared = new Set(inventory.filter((k) => k.kind !== "build").map((k) => `${k.service}|${k.name}`)); const thirdParty = (name) => name.startsWith("component-"); const undeclared = finding.filter((b) => thirdParty(b.name) && !declared.has(`${b.service}|${b.name}`)); const wronglyHeld = finding.filter((b) => !thirdParty(b.name)); const count = (how) => undeclared.filter((b) => b.how === how).length; const declaredRuntime = inventory.filter((k) => k.kind !== "build").length; const runtimeTotal = declaredRuntime + undeclared.length; console.log(`runtime components the inventory sees ${declaredRuntime}, ` + `build time ${inventory.length - declaredRuntime}`); console.log(`the inventory does not see ${undeclared.length} -> ` + `vendored into source ${count("vendored into source")}, ` + `embedded in binary ${count("embedded in binary")}, fetched at runtime ${count("fetched at runtime")}`); console.log(`coverage ${(100 * declaredRuntime / runtimeTotal).toFixed(0)}% ` + `(${declaredRuntime}/${runtimeTotal} runtime components)`); const COLUMN = [23, 26, 7, 7, 0]; const row = (s) => console.log(s.map((v, i) => String(v).padEnd(COLUMN[i])).join("")); console.log(""); row(["gate", "reads", "holds", "alone", "wrongly holds"]); row(["reads manifest files", `${cost.file} files / ${cost.byte} bytes`, 0, 0, 0]); row(["walks the output tree", `${walkedFile} files / ${walkedByte} bytes`, undeclared.length, undeclared.length, wronglyHeld.length]); console.log(`wrongly held ${wronglyHeld.length} records: ` + `${wronglyHeld.map((b) => b.service).join(", ")} -> ${wronglyHeld[0].name}`); // The actual runtime set is carried into the next block: declared + undeclared. writeFileSync(join(ROOT, "actual.json"), JSON.stringify([ ...inventory.filter((k) => k.kind !== "build") .map(({ service, name, version }) => ({ service, name, version, visible: true })), ...undeclared.map(({ service, name, version }) => ({ service, name, version, visible: false })), ])); ``` ``` runtime components the inventory sees 88, build time 19 the inventory does not see 14 -> vendored into source 6, embedded in binary 4, fetched at runtime 4 coverage 86% (88/102 runtime components) gate reads holds alone wrongly holds reads manifest files 12 files / 7674 bytes 0 0 0 walks the output tree 17 files / 8715 bytes 14 14 3 wrongly held 3 records: collector, verifier, billing -> internal-library-measurement ``` Of the 102 running components, 88 are visible in the inventory; **14 are invisible** — 6 vendored into source, 4 embedded in the binary, 4 fetched at runtime. **Coverage is 86%.** The missing 14% is the measurement's most expensive part, because the inventory reports these components not as "absent," but by never showing them at all. The table sets the two gates side by side. The gate that reads manifest files holds **zero** undeclared components; it cannot, because where it looks is the manifest itself. **Its marginal value as a gate is zero,** and it has no need to stand as a gate: producing the inventory is enough, the authority to permit passage earns it nothing. **This is the gate that is better off removed.** The gate that walks the output tree holds 14 of the 14, and **holds all of them only by itself** — its marginal value is the entirety of what it holds. The reading cost is small: 12 files and 7,674 bytes against 17 files and 8,715 bytes — a different read, not a bigger one. **The real cost is wrongful holding:** 3 of the 17 findings are copies of its own code. One in every six findings is a wasted review, and the behavior this ratio produces is a familiar one — the list gets marked "known" wholesale once, and the next real finding stays under the same mark. #### When an Advisory Arrives The inventory's value is measured not at the moment it is produced, but at the moment it is asked. The question always takes the same form: versions below such-and-such a component's such-and-such release are exposed — do we have it, where? **RD24.** A defect advisory is a component name and a threshold version; versions below the threshold count as exposed. The advisory draw's seed `884301` is visible. ```js // measurement-network/advisory.mjs — the response when a defect advisory arrives: the inventoried path and the uninventoried path import { readFileSync } from "node:fs"; import { join } from "node:path"; const ROOT = join(process.env.TMPDIR ?? "/tmp", "measurement-network-inventory"); const SERVICE = ["collector", "verifier", "billing", "workorder"]; const VERSION = ["1.4", "2.0", "2.3", "2.6", "3.1", "3.4"]; const read = (name) => JSON.parse(readFileSync(join(ROOT, name))); const inventory = read("inventory.json"), cost = read("cost.json"); const actual = read("actual.json"); // previous block's audit: declared + undeclared const generator = (seed) => () => (seed = (seed * 1103515245 + 12345) % 2147483648) / 2147483648; const r = generator(884301); // the advisory draw's seed is visible const NAMES = [...new Set(actual.map((k) => k.name))].sort(); const ADVISORY = Array.from({ length: 9 }, () => ({ name: NAMES[Math.floor(r() * NAMES.length)], threshold: VERSION[1 + Math.floor(r() * (VERSION.length - 1))], // versions below this are exposed })); const affected = (set, d) => new Set(set .filter((k) => k.name === d.name && VERSION.indexOf(k.version) < VERSION.indexOf(d.threshold)) .map((k) => k.service)); console.log(`actual runtime records ${actual.length} ` + `(invisible ${actual.filter((k) => !k.visible).length}), advisories ${ADVISORY.length}; seed 884301`); console.log(""); console.log("advisory".padEnd(20) + "inventory answer".padEnd(18) + "actual".padEnd(14) + "status"); let correct = 0, incomplete = 0, impactful = 0; for (const d of ADVISORY) { const answer = affected(actual.filter((k) => k.visible), d), real = affected(actual, d); if (real.size) impactful++; answer.size === real.size ? correct++ : incomplete++; console.log(`${d.name} < ${d.threshold}`.padEnd(20) + `${answer.size} service(s)`.padEnd(18) + `${real.size} service(s)`.padEnd(14) + (answer.size === real.size ? "complete" : "incomplete")); } const DECLARE_FILES = SERVICE.length * 3; // manifest, resolved, build per repo console.log(""); console.log("path".padEnd(28) + "per advisory".padEnd(28) + `for ${ADVISORY.length} advisories`); console.log("query the inventory file".padEnd(28) + `1 query / ${inventory.length} records`.padEnd(28) + `1 build (${cost.file} files) + ${ADVISORY.length} queries`); console.log("open repos one by one".padEnd(28) + `${SERVICE.length} repos / ${DECLARE_FILES} files`.padEnd(28) + `${ADVISORY.length * SERVICE.length} repo openings, ${ADVISORY.length * DECLARE_FILES} files`); console.log(""); console.log(`advisories with an affected service ${impactful}/${ADVISORY.length}; ` + `inventory answered fully ${correct}, incompletely ${incomplete}`); ``` ``` actual runtime records 102 (invisible 14), advisories 9; seed 884301 advisory inventory answer actual status component-47 < 2.6 0 service(s) 1 service(s) incomplete component-49 < 3.4 0 service(s) 1 service(s) incomplete component-59 < 2.3 0 service(s) 0 service(s) complete component-41 < 2.0 0 service(s) 0 service(s) complete component-12 < 2.0 0 service(s) 0 service(s) complete component-12 < 2.3 0 service(s) 0 service(s) complete component-05 < 3.1 1 service(s) 1 service(s) complete component-58 < 2.3 0 service(s) 0 service(s) complete component-15 < 2.6 1 service(s) 2 service(s) incomplete path per advisory for 9 advisories query the inventory file 1 query / 107 records 1 build (12 files) + 9 queries open repos one by one 4 repos / 12 files 36 repo openings, 108 files advisories with an affected service 4/9; inventory answered fully 6, incompletely 3 ``` **Five of the nine advisories affect no service at all.** The most frequent answer is "we do not have it," and the inventory's value lies primarily here: that answer arrives in a single query. To reach the same answer, the inventory-less path **opens four repositories, reads twelve manifest files** for every advisory — 36 repository openings and 108 files for nine advisories. This sets the direction of the time the gate adds. Producing the inventory adds a stage to the pipeline, and that stage reads 12 files on **every change**; against that, the answer drops from four repository openings **per advisory** to a single query. **The gate's cost is per change, its payoff is per advisory**; the ratio is set by advisory frequency, not by the gate itself. The last line is the price of coverage. The inventory answers **6 of 9 advisories fully, 3 incompletely.** In two, the answer is "0 services," the actual is "1 service"; the advisory gets closed as irrelevant. The third is worse: for `component-15`, the inventory says "1 service," the actual is "2 services." A zero answer sometimes gets questioned, **a number does not** — the work looks done. This is the price of the 14 invisible components, and it comes back the moment the gate that walks the output tree is removed. #### Summary - The four services declare 34 direct dependencies; the inventory produces 107 records — 34 direct, 54 transitive, 19 build-time tools. A gate that looks only at direct dependencies sees 34 of the declared 88 runtime components. - Production cost is written independent of environment: 12 manifest files, 7,674 bytes read, 107 records, a 13,511-byte inventory, 56 unique components. - Of the 102 running components, 88 are visible in the inventory, 14 are invisible: 6 vendored into source, 4 embedded in the binary, 4 fetched at runtime. Coverage is 86%. - The gate that reads manifest files holds zero undeclared components and can be removed as a gate; the gate that walks the output tree holds 14, holding all of them only by itself, at the cost of wrongly holding 3 of its 17 findings. - The answer drops from 4 repository openings per advisory to 1 query: 36 repository openings against a single build plus 9 queries for 9 advisories. The inventory answers 6 advisories fully, 3 incompletely; one of the incomplete ones says "1" instead of "0." #### Next Step The inventory now exists, its coverage is counted and queryable. But there is **no link at all** between this 13,511-byte file and the output it describes. The inventory is a separate file: another inventory could be placed next to it, a record inside it could be changed, an entirely different build artifact could be paired with the same inventory. None of the 107 records says that the bytes it describes are actually those bytes. The same gap exists at the chain's other links too: between the source commit and the build artifact, between the build artifact and what gets deployed. The next lesson binds these links with a real key pair and turns two questions into numbers: at which point in the chain verification happens determines how many forged outputs get through, and how many subjects hold that key? ### Build Artifact Signing and Verification Source: https://academia.sh/en/courses/ci-cd/build-artifact-signing-and-verification The chain running from the source commit to the deployed output is built with a real key pair, and two questions are turned into numbers: however many forged outputs get through depends on the point in the chain where verification happens, and how many subjects with access to that key can sign. The previous lesson produced the inventory and ended with a gap: there was no link at all between the 107-record file and the output it describes. The same gap exists at every link in the chain. Between the source commit the pipeline reads and the build output it produces, between that output and the bytes written to the repository, between the bytes in the repository and what gets deployed — at every handoff the object changes hands, and **there is no proof that it changed hands.** This lesson builds that link and turns two questions into numbers: however many forged outputs get through depends on the point in the chain where verification happens, and how many subjects can produce the signature. M22/K02 built the signing chain inside the repository and counted the path an unverified link opens up; that setup is not repeated here. The axis here is **where verification happens** and **who holds the key**. **RD25.** Signing is done with a real key pair, but the key is regenerated on every run; what gets printed is the verification result, not the signature itself. No real key or secret is written. **RD26.** The chain is four links: source commit, build output, inventory, and statement; the statement binds three digests, the signature binds the statement. #### The Chain's Four Links ```js // chain.mjs — source commit, build output, inventory, statement: the chain is built and tested import { generateKeyPairSync, sign, verify, createHash } from "node:crypto"; const digest = (b) => createHash("sha256").update(b).digest(); const short = (b) => digest(b).toString("hex").slice(0, 12); // Fictional key pair regenerated on every run; no real key or secret is written. const { publicKey: pub, privateKey: priv } = generateKeyPairSync("ed25519"); const { privateKey: otherPriv } = generateKeyPairSync("ed25519"); // Link 1: source commit — the tree contents are hashed. Link 2: build output. Link 3: inventory. const source = Buffer.from(["app.js", "config.json", "measure.js"] .map((y) => `${y}\n// fictional regional measurement network\n`).join("")); const output = Buffer.concat([Buffer.from("body\n"), Buffer.alloc(24576, "u")]); const inventory = Buffer.from( // previous lesson's inventory, 107 records Array.from({ length: 107 }, (_, i) => `component-${i}@2.3`).join("\n")); // Link 4: the statement binds three digests; the signature binds the statement. const writeStatement = (version, src, out, inv) => Buffer.from([`service=collector`, `version=${version}`, `source=${short(src)}`, `output=${short(out)}`, `inventory=${short(inv)}`].join("\n")); const STATEMENT = writeStatement(41, source, output, inventory); const SIGNATURE = sign(null, digest(STATEMENT), priv); console.log(`statement ${STATEMENT.length} bytes, signature ${SIGNATURE.length} bytes; ` + `output ${output.length} bytes, inventory ${inventory.length} bytes`); // The verifier runs three separate checks: signature, output digest, inventory digest. const check = (b, i, pk, out, inv) => ({ signature: verify(null, digest(b), pk, i), output: b.includes(`output=${short(out)}`), inventory: b.includes(`inventory=${short(inv)}`), }); const corruptedOutput = Buffer.from(output); corruptedOutput[9000] = 0x41; // one byte const corruptedInventory = Buffer.from(inventory.toString().replace("component-3@2.3", "component-3@1.4")); const SCENARIO = [ ["unmodified", STATEMENT, SIGNATURE, pub, output, inventory], ["version changed in statement", writeStatement(39, source, output, inventory), SIGNATURE, pub, output, inventory], ["output's single byte changed", STATEMENT, SIGNATURE, pub, corruptedOutput, inventory], ["inventory changed", STATEMENT, SIGNATURE, pub, output, corruptedInventory], ["signed with another key", STATEMENT, sign(null, digest(STATEMENT), otherPriv), pub, output, inventory], ]; const COLUMN = [31, 11, 15, 19, 0]; const row = (s) => console.log(s.map((v, i) => String(v).padEnd(COLUMN[i])).join("")); console.log(""); row(["scenario", "signature", "output digest", "inventory digest", "accepted"]); for (const [name, b, i, pk, out, inv] of SCENARIO) { const s = check(b, i, pk, out, inv); row([name, s.signature, s.output, s.inventory, s.signature && s.output && s.inventory]); } ``` ``` statement 91 bytes, signature 64 bytes; output 24581 bytes, inventory 1815 bytes scenario signature output digest inventory digest accepted unmodified true true true true version changed in statement false true true false output's single byte changed true false true false inventory changed true true false false signed with another key false true true false ``` The link's cost is two items: **a 91-byte statement and a 64-byte signature.** What they bind is a 24,581-byte output and a 1,815-byte inventory; 155 bytes nails down 26 kilobytes. The table shows exactly where the chain comes apart. **Each of the three checks catches exactly one scenario, and the others miss that scenario.** The signature check fails when the version in the statement is changed and when the signature comes from another key — it catches two of the four forgeries. But when a single byte of the output changes, the signature is still valid: the statement is not corrupted, what is corrupted is what the statement **describes**. The same holds when a single version number in the inventory is changed — the signature still verifies. This is the gate's marginal value applied inside the chain. When the signature check is used alone, two of the four forgeries get through. The output-digest check catches only its own scenario, the inventory-digest check only its own. All three together hold all four, and the cost of that is three digest computations. One detail determines all three checks: **two of them require the object itself.** The signature can be tested with the statement alone, but the output digest requires re-reading the output's bytes, and the inventory digest requires re-reading the inventory's bytes. A verifier that does not have the object in hand cannot perform those two checks — and the question of where verification happens comes exactly from this. #### Where Verification Happens **RD27.** The deployment attempts are fictional; every attempt is genuinely signed, forgery is applied **after signing**, and seed `31071` is visible. **RD28.** After the thirtieth attempt the key is rotated and the service's authorized subject changes. ```js // deployment.mjs — however many forged outputs get through depends on where verification happens import { generateKeyPairSync, sign, verify, createHash } from "node:crypto"; const digest = (b) => createHash("sha256").update(b).digest(); const short = (b) => digest(b).toString("hex").slice(0, 12); const A = generateKeyPairSync("ed25519"); // the service's first authorized subject const B = generateKeyPairSync("ed25519"); // trusted, but unauthorized for this service const C = generateKeyPairSync("ed25519"); // the authorized subject after rotation const der = (k) => k.export({ type: "spki", format: "der" }).toString("base64"); const TRUSTED = [A, B, C].map((k) => k.publicKey); // keys the verifier recognizes const generator = (seed) => () => (seed = (seed * 1103515245 + 12345) % 2147483648) / 2147483648; const r = generator(31071); // seed visible const FORGERY = ["output tampered in the repository", "inventory tampered", "another subject's signature", "redeployment of an old version"]; // Fictional deployment attempts: each is genuinely signed, forgery is applied AFTER signing. // Key rotated after attempt 30: the authorized subject is C, not A. const ATTEMPT = []; for (let i = 0, version = 40; i < 60; i++) { const kind = r() < 0.6 ? "valid" : FORGERY[Math.floor(r() * FORGERY.length)]; const next = kind === "redeployment of an old version" ? version - 3 : ++version; let output = Buffer.alloc(4096, String.fromCharCode(97 + (i % 26))); let inv = Buffer.from(`inventory-${i}\n` + "record\n".repeat(107)); const signer = kind === "another subject's signature" ? B : (i < 30 ? A : C); const statement = Buffer.from(`service=collector\nversion=${next}\n` + `output=${short(output)}\ninventory=${short(inv)}`); const signature = sign(null, digest(statement), signer.privateKey); if (kind === "output tampered in the repository") output = Buffer.alloc(4096, "z"); if (kind === "inventory tampered") inv = Buffer.concat([inv, Buffer.from("extra\n")]); ATTEMPT.push({ kind, version: next, output, inv, statement, signature, pub: signer.publicKey }); } // Five disciplines: the chain is the same, what changes is WHERE verification happens and the subject list. const STALE = [A].map((k) => der(k.publicKey)), CURRENT = [A, C].map((k) => der(k.publicKey)); const DISCIPLINE = [ ["no verification", {}], ["only at pipeline end", { end: 1 }], ["at deploy time: signature and digests", { signature: 1, digest: 1 }], ["+ subject list (stale)", { signature: 1, digest: 1, subject: STALE, fresh: 1 }], ["+ subject list (current)", { signature: 1, digest: 1, subject: CURRENT, fresh: 1 }], ]; const FORGED_COUNT = ATTEMPT.filter((d) => d.kind !== "valid").length; const VALID_COUNT = ATTEMPT.length - FORGED_COUNT; const COLUMN = [40, 18, 18, 0]; const row = (s) => console.log(s.map((v, i) => String(v).padEnd(COLUMN[i])).join("")); row(["discipline", "forged accepted", "valid stopped", "forgery types that passed"]); for (const [name, k] of DISCIPLINE) { let lastVersion = 40, validPassed = 0; const passed = []; for (const d of ATTEMPT) { // Pipeline-end verification: the output was correct the moment it was produced; forgery applied later. if (k.signature && !TRUSTED.some((a) => verify(null, digest(d.statement), a, d.signature))) continue; if (k.digest && !(d.statement.includes(`output=${short(d.output)}`) && d.statement.includes(`inventory=${short(d.inv)}`))) continue; if (k.subject && !k.subject.includes(der(d.pub))) continue; if (k.fresh && d.version <= lastVersion) continue; lastVersion = Math.max(lastVersion, d.version); d.kind === "valid" ? validPassed++ : passed.push(d.kind); } row([name, `${passed.length}/${FORGED_COUNT}`, `${VALID_COUNT - validPassed}/${VALID_COUNT}`, [...new Set(passed)].map((t) => t.split(" ")[0]).join(", ") || "-"]); } console.log(""); console.log(`valid ${VALID_COUNT}; ` + FORGERY.map((t) => `${t.split(" ")[0]} ${ATTEMPT.filter((d) => d.kind === t).length}`).join(", ")); ``` ``` discipline forged accepted valid stopped forgery types that passed no verification 18/18 0/42 inventory, redeployment, another, output only at pipeline end 18/18 0/42 inventory, redeployment, another, output at deploy time: signature and digests 9/18 0/42 redeployment, another + subject list (stale) 0/18 18/42 - + subject list (current) 0/18 0/42 - valid 42; output 4, inventory 5, another 6, redeployment 3 ``` The first two rows are this lesson's harshest number: **whether there is no verification at all, or verification happens only at the end of the pipeline, eighteen of the eighteen forgeries get through.** The pipeline verifies the output it itself produced and finds it correct — because at that moment the output genuinely is correct. Forgery is applied after the signing. **A gate that stands before the event it means to catch has a marginal value of zero,** and that comes not from the gate being weak, but from it standing in the wrong place. The third row moves verification to the moment of deployment, and the number is cut in half: 9 of 18 forgeries get through. The nine held are the output tampered in the repository (4) and the inventory tampered (5) — both are caught because the object's bytes are re-read. The nine that get through are the ones **whose signature is valid.** In six of them the signature is from another subject's valid key; the verifier says "a key I recognize signed this" and lets it through. In three, everything is genuine, only stale: a past version's statement and signature still verify today. The fourth row adds two rules — the signer must be the service's authorized subject, the version must be greater than the last deployed one — and the forgery count falls to **zero.** But a second number appears in the same row: **18 of 42 valid deployments get stopped.** The key was rotated after the thirtieth attempt, and the verifier's subject list stayed on the old key. The gate holds all of the forgeries and also holds 43% of the valid deployments. This writes the gate's third number. Verification itself is a digest computation and a list lookup; the time it adds is negligible. What causes the wait is not verification, it is **the freshness of the list**: after the key rotates, every valid deployment stops until the list is updated. The behavior a gate that stops eighteen consecutive valid deployments produces in a team is a familiar one — the gate gets bypassed once, and the bypass becomes permanent. The last row runs the same chain with the current list: 0 forgeries, 0 wrongful stops. #### In How Many Subjects Does the Key Sit The fourth row leaves a question open: is "authorized subject" a singular thing? It is anyone who can produce the signature — anyone with access to the private key. **RD29.** The storage models are fictional; the number of subjects with access to the key is the model's input. ```js // key.mjs — the key itself is a gate: how many subjects can sign import { generateKeyPairSync, sign, verify, createHash } from "node:crypto"; const digest = (b) => createHash("sha256").update(b).digest(); const der = (k) => k.export({ type: "spki", format: "der" }).toString("base64"); const statement = (i) => Buffer.from(`service=collector\nversion=${100 + i}`); // Fictional storage models: whoever can reach where the key sits, that subject can sign. const STORAGE = [ ["private key in the repository", 23], ["environment variable on the pipeline runner", 11], ["signing service, three subjects with call authority", 3], ["in hardware, a single automated subject", 1], ]; const COLUMN = [55, 10, 8, 13, 0]; const row = (s) => console.log(s.map((v, i) => String(v).padEnd(COLUMN[i])).join("")); row(["storage", "subjects", "shared", "per-subject", "distinguishable"]); for (const [name, n] of STORAGE) { // (a) Shared single key: n subjects sign with the same private key. const singleKey = generateKeyPairSync("ed25519"); const sharedCount = Array.from({ length: n }, (_, i) => sign(null, digest(statement(i)), singleKey.privateKey)) .filter((im, i) => verify(null, digest(statement(i)), singleKey.publicKey, im)).length; // (b) Per-subject key: the authorized list is a single subject, the rest sign with their own keys. const keys = Array.from({ length: n }, () => generateKeyPairSync("ed25519")); const LIST = [der(keys[0].publicKey)]; const perSubjectCount = keys .map((a, i) => [a, sign(null, digest(statement(i)), a.privateKey), i]) .filter(([a, im, i]) => LIST.includes(der(a.publicKey)) && verify(null, digest(statement(i)), a.publicKey, im)).length; // Distinguishable signer: the number of distinct public keys the verifier sees across n signatures. const distinctSeen = (d) => new Set(d).size; row([name, n, `${sharedCount}/${n}`, `${perSubjectCount}/${n}`, `${distinctSeen(Array(n).fill(der(singleKey.publicKey)))} -> ${distinctSeen(keys.map((a) => der(a.publicKey)))}`]); } ``` ``` storage subjects shared per-subject distinguishable private key in the repository 23 23/23 1/23 1 -> 23 environment variable on the pipeline runner 11 11/11 1/11 1 -> 11 signing service, three subjects with call authority 3 3/3 1/3 1 -> 3 in hardware, a single automated subject 1 1/1 1/1 1 -> 1 ``` With a shared single key, **every subject** with access to the key produces a signature that verifies: 23 of 23, 11 of 11. The verifier can reject none of them, because they are all the same key. The last column turns this into a quantity — across 23 signatures, the verifier sees **a single public key.** There is no path from a signature to a signer; the sentence "the authorized subject signed this" is an unmeasurable sentence. A per-subject key and a one-name authorized list make the accepted count **independent of the number of subjects**: acceptance is 1 whether there are 23, 11, or 3. The number of distinguishable signers, on the other hand, equals the number of subjects. This difference is achieved with the same algorithm, the same signature length; the only thing that changes is how many subjects the key sits in. M22/K02 showed that signing does not eliminate trust, it carries it over to the key. The table here gives the **magnitude** of that carry-over. While the key sits in the repository, the signature says "one of the 23 subjects who can read the repository signed this"; while it sits in hardware, in a single automated subject, it says "that subject signed this." The difference between the two sentences is a factor of 23, and, as with the previous lesson's inventory coverage, the upper bound on the gate's strength is set not by the gate itself, but by a number outside it. #### Summary - The chain is four links, and the cost of the bond is a 91-byte statement and a 64-byte signature; what they bind is a 24,581-byte output and a 1,815-byte inventory. - Each of the three checks catches exactly one scenario: the signature check holds two of four forgeries, the output digest and the inventory digest each hold one of the remaining two. Two of them require re-reading the object's bytes. - With no verification, 18 of 18 forgeries get through; with verification only at pipeline end, 18 also get through, because forgery is applied after signing. A gate standing in the wrong place has a marginal value of zero. - When signature and digests are checked at deploy time, 9 forgeries get through: 6 are another subject's valid signature, 3 are the redeployment of a past version. With a subject list and a freshness rule, the forgery count falls to 0. - A stale subject list holds all of the forgeries but also stops 18 of 42 valid deployments; what causes the wait is not verification, it is the freshness of the list. - With a shared key, 23 of 23 subjects' signatures are accepted and the verifier sees a single public key; with a per-subject key, acceptance falls to 1 and the number of distinguishable signers rises to 23. #### Next Step The chain is now built, tested at deployment time, and the key sits in a single subject. The measured numbers hold up to a point: 0 forgeries, 0 wrongful stops, a 155-byte bond. But that single subject is not a person, it is **the pipeline itself** — and every number in this section stands on the assumption that the pipeline runs honestly. That assumption has never been counted so far. The workflow definition the pipeline runs comes from the repository, and how many subjects can change the repository? The pipeline runner accesses the signing key, the build-output repository, and the deployment target all at once; can a dependency pulled in by one stage use all of those accesses? The next lesson turns this course's gates onto **the machine that places the gates** and asks the same three numbers for the pipeline itself: what does it hold, what does it miss, how long does it hold things up? ### Pipeline Security Source: https://academia.sh/en/courses/ci-cd/pipeline-security The pipeline itself is measured: how many secrets and targets nine job steps reach, the access that drops and the cost paid when moving to least privilege, the state carried between jobs run back to back on the same runner, and, across a known set of misuse attempts, what each constraint closes, misses, and how much legitimate work it stops. The previous lesson signed the output and verified it at deployment time: all eighteen of the eighteen forgeries were stopped, the key stayed in a single subject. That subject was not a person, it was **the pipeline itself.** Every gate measured in this course rested on the assumption that the machine running the gate behaves honestly, and that assumption has never been counted. This lesson counts it: how many secrets the pipeline reaches, what gets carried between two jobs, how many misuse attempts get closed. The measure is still the gate — what stands in front of it is no longer a change, it is the pipeline's own step. **RD31.** The fictional regional measurement network's pipeline is nine steps; 12 secrets and 6 targets are modeled with `node`, and it is written that it is a model — **secret values are not written, only their names appear.** **RD32.** A step's *actual need* is what it genuinely uses, its *declared* set is what is written into the pipeline definition; the two diverge at two steps. #### The Pipeline's Authority Surface ```js // measurement-network/authority.mjs — the pipeline's authority surface: which step reaches which secret and target (model) const SECRET = ["source-repo-read", "package-repo-read", "package-repo-write", "object-store-write", "signing-key", "deploy-token-development", "deploy-token-staging", "deploy-token-production", "database-staging", "database-production", "notification-token", "collector-key"]; const TARGET = ["development", "staging", "production", "build-output-repo", "object-store", "notification-channel"]; // Fictional regional measurement network pipeline: nine steps. `need` is what the step actually // uses, `declared` is the set written into the pipeline definition. The two diverge at two steps. const STEP = [ ["format-parsing", [], []], ["build", ["source-repo-read", "package-repo-read"], []], ["unit-test", [], []], ["contract-test", [], []], ["integration-test", ["database-staging"], ["staging"]], ["packaging", ["package-repo-write"], ["build-output-repo"]], ["inventory-generation", ["package-repo-read"], ["build-output-repo"]], ["signing", ["signing-key"], ["build-output-repo"]], ["deployment", ["deploy-token-production", "object-store-write", "notification-token"], ["production", "object-store", "notification-channel"]], ].map(([name, secrets, targets]) => ({ name, secrets, targets })); const MISSING = { "inventory-generation": ["package-repo-read"], deployment: ["notification-token"] }; const declared = (a) => a.secrets.filter((s) => !(MISSING[a.name] ?? []).includes(s)); const total = (f) => STEP.reduce((t, a) => t + f(a), 0); const singleSecret = STEP.length * SECRET.length, singleTarget = STEP.length * TARGET.length; const perStepSecret = total((a) => declared(a).length), perStepTarget = total((a) => a.targets.length); const widest = Math.max(...STEP.map((a) => declared(a).length + a.targets.length)); const COLUMN = [26, 12, 15, 15, 0]; const row = (s) => console.log(s.map((v, i) => String(v).padEnd(COLUMN[i])).join("")); row(["layout", "identities", "secret access", "target access", "reach if a step is compromised"]); row(["single pipeline identity", 1, `${singleSecret}/${singleSecret}`, `${singleTarget}/${singleTarget}`, `${SECRET.length} secrets + ${TARGET.length} targets`]); row(["per-step identity", STEP.length, `${perStepSecret}/${singleSecret}`, `${perStepTarget}/${singleTarget}`, `at most ${widest} entries`]); const secretless = total((a) => (a.secrets.length === 0 ? 1 : 0)); const blocked = STEP.filter((a) => a.secrets.length > declared(a).length).map((a) => a.name); console.log(""); console.log(`access dropped ${singleSecret + singleTarget - perStepSecret - perStepTarget}/${singleSecret + singleTarget} ` + `(${(100 * (1 - (perStepSecret + perStepTarget) / (singleSecret + singleTarget))).toFixed(0)}%); cost ` + `${STEP.length} identity definitions and ${perStepSecret + perStepTarget} binding entries`); console.log(`steps using no secret at all ${secretless}/${STEP.length} — ` + `still reach ${SECRET.length} secrets under a single identity`); console.log(`actual need ${total((a) => a.secrets.length)} secrets, declared ${perStepSecret}; ` + `${blocked.length} steps under-declared (${blocked.join(", ")}) -> ${blocked.length} legitimate work stalls`); ``` ``` layout identities secret access target access reach if a step is compromised single pipeline identity 1 108/108 54/54 12 secrets + 6 targets per-step identity 9 7/108 7/54 at most 5 entries access dropped 148/162 (91%); cost 9 identity definitions and 14 binding entries steps using no secret at all 3/9 — still reach 12 secrets under a single identity actual need 9 secrets, declared 7; 2 steps under-declared (inventory-generation, deployment) -> 2 legitimate work stalls ``` Under a single pipeline identity, the access count is **162**: each of the nine steps reaches all twelve secrets and all six targets. Under a per-step identity, the same pipeline runs with **14 accesses**; the 148 accesses that drop are **91% of the surface**, at a cost of nine identity definitions and fourteen binding entries. **Three of the nine steps use no secret at all** — format parsing, unit testing, contract testing — but under a single identity, those three also reached all twelve secrets. This is not a bug, it is a default: identity is granted to the pipeline, not to the step. The record count a compromised step can reach falls from 18 to **at most 5**. The last line is the gate's other face: actual need is nine secrets, declared is seven, two jobs stall. **The source of the wrongful hold is not the constraint, it is a list again** — the trigger filter in `pipeline-design/08` and the cache key in `pipeline-design/05` broke from the same place. #### What Gets Carried Between Two Jobs **RD33.** Four kinds of state can carry over on the runner: cache, environment variable, leftover file, credential memory; the leave and read draws come from seed `70413+i`. **RD34.** Baseline runtime is 120 seconds, a cache hit saves 45 seconds. ```js // measurement-network/isolation.mjs — state carried between jobs run back to back on the same runner (model) const generator = (seed) => () => (seed = (seed * 1103515245 + 12345) % 2147483648) / 2147483648; const KIND = ["cache", "environment variable", "leftover file", "credential memory"]; // Fictional job sequence: twelve jobs, four services, one comes from an external contribution. // Each job leaves some state behind and reads some; leave and read draws come from the seed. const JOB = Array.from({ length: 12 }, (_, i) => { const r = generator(70413 + i); return { number: i + 1, externalContribution: i === 6, leaves: KIND.filter((_, k) => k === 0 || r() < 0.45), // cache is always left behind reads: KIND.filter(() => r() < 0.5) }; }); // Three isolation levels: which kind of state is cleared between jobs. const LEVEL = [ ["shared runner", []], ["clean workspace per job", ["leftover file"]], ["new runner per job", KIND], ]; // Cost: runtime stretches out as hit rate falls. Baseline 120 s, a hit saves 45 s. const COLUMN = [26, 7, 5, 10, 12, 18, 6, 0]; const row = (s) => console.log(s.map((v, i) => String(v).padEnd(COLUMN[i])).join("")); row(["isolation", "cache", "env", "leftover", "credential", "total / external", "hits", "twelve jobs"]); for (const [name, cleared] of LEVEL) { const count = Object.fromEntries(KIND.map((t) => [t, 0])); let crossedOut = 0, hits = 0; const remaining = new Map(); // kind -> number of the job that left it for (const job of JOB) { for (const t of cleared) remaining.delete(t); // kinds cleared before the job starts for (const t of job.reads) { if (!remaining.has(t)) continue; count[t] += 1; // the previous job's state carried into this one if (t === "cache") hits += 1; if (job.externalContribution || remaining.get(t) === 7) crossedOut += 1; // to/from external contribution } for (const t of job.leaves) remaining.set(t, job.number); } const total = KIND.reduce((a, t) => a + count[t], 0); row([name, ...KIND.map((t) => count[t]), `${total} / ${crossedOut}`, `${hits}/${JOB.length}`, `${JOB.length * 120 - hits * 45} s`]); } ``` ``` isolation cache env leftover credential total / external hits twelve jobs shared runner 6 9 8 5 28 / 5 6/12 1170 s clean workspace per job 6 9 0 5 20 / 4 6/12 1170 s new runner per job 0 0 0 0 0 / 0 0/12 1440 s ``` On a shared runner, **28 carry-overs** occur among the twelve jobs; five of them have an externally contributed job on one end. Only six are cache — nine are environment variable, eight are leftover file, **five are credential memory**, meaning a token the previous job decrypted stays in memory. This is exactly where the "per-step identity" gain in the authority surface gets punctured: binding authority to a step does not stop the next job from reading what the step left behind. The middle row is the isolation most commonly applied: clearing the workspace **brings leftover files' eight down to zero**, but environment variable and credential memory remain — total 20, four crossing outward, **71% of the carry-over is still open.** A new runner per job closes all four at once: **28 of 28 carry-overs are zeroed out.** The cost is a single item — the hit rate falls from six to zero, and the twelve jobs take **1,440 seconds** instead of 1,170: 270 seconds, **23%.** The cost accounting for runner types was measured in `pipeline-design/06` and is not repeated here; the novelty here is that the "clean start" advantage measured there also reads as a **carry-over count**. #### A Known Set of Misuse Attempts **RD35.** The misuse set is fictional: 40 attempts across three classes and 60 legitimate jobs, seed `56092`. An attempt succeeds while at least one of its paths is open. **RD36.** The "authorized subject" path is the path no constraint closes. ```js // measurement-network/misuse.mjs — a known set of misuse attempts and the ledger of constraints (model) const generator = (seed) => () => (seed = (seed * 1103515245 + 12345) % 2147483648) / 2147483648; const r = generator(56092); // seed visible const CLASS = [ ["secret access from an external contribution", "external-contribution-secret", "carried-over-state", 0.40], ["running code through a dependency", "install-script", "carried-over-state", 0.30], ["changing the pipeline definition", "definition-change", "external-contribution-secret", 0.25], ]; // Fictional ledgers. An attempt succeeds when at least one of its paths is open. const ATTEMPT = Array.from({ length: 40 }, () => { const [name, path, second, p] = CLASS[Math.floor(r() * CLASS.length)]; const paths = [path]; if (r() < p) paths.push(second); if (r() < 0.15) paths.push("authorized-subject"); // the path no constraint closes return { class: name, paths }; }); const LEGITIMATE = Array.from({ length: 60 }, () => ({ externalContribution: r() < 0.35, wantsSecret: r() < 0.30, installScript: r() < 0.12, definitionChange: r() < 0.18, })); const CONSTRAINT = [ ["no secret is given to external contributions", "external-contribution-secret", (m) => m.externalContribution && m.wantsSecret], ["external-contribution jobs wait for manual approval", "external-contribution-secret", (m) => m.externalContribution], ["dependency install scripts are not run", "install-script", (m) => m.installScript], ["pipeline definition changes require separate approval", "definition-change", (m) => m.definitionChange], ["new runner per job", "carried-over-state", () => false], ]; const passes = (active) => ATTEMPT.filter((g) => g.paths.some((y) => !active.some(([, closes]) => closes === y))).length; const FULL = passes(CONSTRAINT); const COLUMN = [56, 8, 10, 0]; const row = (s) => console.log(s.map((v, i) => String(v).padEnd(COLUMN[i])).join("")); row(["constraint", "closes", "marginal", "legitimate work stopped"]); for (const k of CONSTRAINT) row([k[0], ATTEMPT.filter((g) => g.paths.includes(k[1])).length, passes(CONSTRAINT.filter((o) => o !== k)) - FULL, LEGITIMATE.filter(k[2]).length]); const zero = CONSTRAINT.filter((k) => passes(CONSTRAINT.filter((o) => o !== k)) === FULL); console.log(""); console.log(`attempts ${ATTEMPT.length} (${CLASS.map(([a]) => ATTEMPT.filter((g) => g.class === a).length).join("/")}), legitimate work ${LEGITIMATE.length}; seed 56092`); console.log(`passing with no constraints ${ATTEMPT.length}, with the full set ${FULL} ` + `— all through the authorized-subject path`); console.log(`zero-marginal constraints ${zero.length}: ${zero.map((k) => k[0]).join(" | ")}; ` + `removing both at once, passing ` + `${passes(CONSTRAINT.filter((o) => o[1] !== "external-contribution-secret"))}/${ATTEMPT.length}`); ``` ``` constraint closes marginal legitimate work stopped no secret is given to external contributions 16 0 7 external-contribution jobs wait for manual approval 16 0 16 dependency install scripts are not run 14 12 8 pipeline definition changes require separate approval 13 12 9 new runner per job 5 3 0 attempts 40 (13/14/13), legitimate work 60; seed 56092 passing with no constraints 40, with the full set 5 — all through the authorized-subject path zero-marginal constraints 2: no secret is given to external contributions | external-contribution jobs wait for manual approval; removing both at once, passing 19/40 ``` The full constraint set closes **thirty-five** of forty attempts. The five that get through, get through by the same path: **the authorized subject using its own authority.** No constraint closes this, because closing it would mean removing the authority; this is the counterpart of the "whoever has access to the key can sign" limit in `repository-based-deployment/05` — 12.5%. The marginal column separates the constraints: the install-script ban closes **12** attempts by itself, pipeline-definition approval closes **12**. New runner per job closes 5, 3 of which are marginal, and it **stops no legitimate work at all** — its cost is only 270 seconds. The last line gives **the constraint that is better off removed.** Two constraints have zero marginal value, because they close the same path; removing both at once, passing attempts rise to **19**, meaning the path must stay closed. The legitimate-work column tells us which one should close it: "no secret is given" stops 7 jobs, "waits for manual approval" stops **16**. The one that buys the same protection at twice the price is removed. #### Summary - Under a single pipeline identity, 162 of 162 accesses are open, under a per-step identity 14 are; the 148 accesses that drop are 91% of the surface, at a cost of 9 identity definitions and 14 binding entries. - Three of the nine steps use no secret at all but reach all twelve secrets under a single identity; the record count a compromised step can reach falls from 18 to at most 5. The 2 under-declared steps stall 2 legitimate jobs. - On a shared runner, 28 carry-overs occur (6/9/8/5), 5 with an external contribution on one end; workspace cleanup brings it to 20, a new runner to 0. The cost is 1,440 seconds instead of 1,170. - The full constraint set closes 35 of 40 attempts; all 5 that get through do so via the authorized-subject path, which no constraint closes. - Two zero-marginal constraints close the same path; removing both at once, passing attempts rise to 19. The path must stay closed — with the one that stops 7 legitimate jobs, not the one that stops 16. #### Course Wrap-Up | Lesson | Gate | Holds / misses / wrongly holds | Time added | |---|---|---|---| | `pipeline-design/01` | merge | 90 conflict / 10 mismatch / 2 | 23 min/change | | `pipeline-design/02` | manual approval | 5, marginal 2 / 38 / 26 | 226.5 min wait | | `pipeline-design/03` | contract test | 18, marginal 1 / 119 / 17 | 6,138 runner min | | `pipeline-design/04` | retention rule | 38, marginal 1 / 2 / 47 | 338 min recovery | | `pipeline-design/05` | integration test | 3, marginal 2 / 2 from cache / — | 768 s, eight runs | | `pipeline-design/06` | clean-starting runner | 8, marginal 2 / 0 / 0 | 40 s/run | | `pipeline-design/07` | schema validator | 4 errors / 2 errors / 0 | 80 s, twenty changes | | `pipeline-design/08` | path filter | 7 unnecessary runs / 0 / 10 | 3.1 hr delay | | `release-strategies/01` | integration | 34, marginal 24 / 8 / 2–54 | 490–1,638 gate min | | `release-strategies/02` | environment promotion | 7, marginal 7 and 1 / 7 / 5 | 3 hr 56 min wait | | `release-strategies/03` | percentage ramp | 200/200 distinct / 18–60% faint / 27–32% | 30–70 min staircase | | `release-strategies/04` | rollback | 18, single-round 11 / 7 releases, 491 rows / 178 | 40,320 replica-min | | `release-strategies/05` | flag condition | 0, 76 invoices / 4 paths / 1 pending | 18 read points | | `release-strategies/06` | precondition check | 5, marginal 2 / 1 / 2 | 12 rounds, 17-round cost | | `release-strategies/07` | compatibility check | 1, marginal 1 / 1 / 1 | 3 rounds, three schema steps | | `repository-based-deployment/01` | review | 24, marginal 13 / 11 / 7 | 796 hours, ninety days | | `repository-based-deployment/02` | reconciliation round | 48, marginal 17 / 6 / 9 | 8.5 min propagation | | `repository-based-deployment/03` | decrypt authority | 682 pairs, all marginal / 24 / 11 | 385 min escalation | | `repository-based-deployment/04` | output-tree audit | 14, all marginal / 0 / 3 | 12 files/change | | `repository-based-deployment/05` | verification at deploy | 18 forgeries, all marginal / 0 / 18 | digest and list lookup | | `repository-based-deployment/06` | least privilege, isolation | 35, marginal 12-12-3 / 5 subjects / 16 | 270 s, twelve jobs | The course's rule reads from the table: **a gate is worth exactly as much as the defect it alone catches; a gate cannot be defended unless what it holds, misses, and holds up are written together.** The count it holds does not carry the decision by itself — the format gate holds 51 and holds 51 of them by itself alone, the contract gate holds 18 and holds 1. Zero-marginal gates were measured in four places across the course: `pipeline-design/08`'s time trigger (84 empty runs, 0.0-second contribution to delay), `repository-based-deployment/04`'s gate that reads manifest files (holds zero undeclared components), `repository-based-deployment/05`'s pipeline-end verification (lets 18 of 18 forgeries through), and here, "waits for manual approval" (16 legitimate jobs). All four can be removed; the reason is not that they are weak, it is that **they hold only what someone else already holds.** The measure also works in reverse: `pipeline-design/07`'s review holds a single error in 714 minutes, but that error is a change that removes a gate. The pipeline now carries the application to production, and every step of the path is counted. The machines, network, and storage underneath it, though, are still built through another path: one that passes through no gate, keeps no record, and has no defined rollback unit. **M22/K05 Infrastructure as Code** asks this — can they too pass through the same gates, and when they do, what does this table's twenty-one rows show for them?