--- title: 'DevOps Culture and Fundamentals' source: 'https://academia.sh/en/courses/devops-fundamentals' language: en updated: '2026-08-23T17:27:01+00:00' license: 'CC BY-SA 4.0' lessons: 12 outcomes: - 'Reads the delivery metrics as a set that constrains itself' - 'Counts where the difference between environments hides' - "Ties a build artifact's immutability to a check" --- # DevOps Culture and Fundamentals Where the difference hides on the road from idea to production: the technical counterpart of shared responsibility, the delay and risk separate teams produce, mapping the value stream, deployment frequency, lead time, change failure rate and time to restore moving one another, building feedback loops, environment separation, the build once deploy everywhere principle, artifact repositories, separating configuration from secrets, portable application principles, and infrastructure provisioning models. ## Culture and Flow Measuring the path from idea to production: the technical counterpart of shared responsibility and who sees which signal, the delay and risk produced by separate teams, mapping the value stream step by step, deployment frequency, lead time, change failure rate, and time to restore moving one another, the overlapping ground of operating approaches, and the length of the feedback loop. ### What Is DevOps? Source: https://academia.sh/en/courses/devops-fundamentals/what-is-devops The technical and organizational side of shared responsibility: in the same 22 occurrences' separate observation scopes, 9 are visible only to operations, 1 only to development, and 1 in neither scope; 4 occurrences never reach their owner and reaching one averages 7.0 steps; sharing three surfaces brings the unreached count to 1 and the reach time to 5.0 steps; fully combining the scopes brings reach down to 1.6 steps while the development team's load rises from 95 to 107 steps, lead time rises from 25.8 to 34.8 steps, and deployment frequency falls from 5 to 4 releases. The previous lesson measured handover as a **contract**: twenty of the twenty-eight responsibility items were handed over, eight stayed in development for lack of information, tooling, or procedure. The contract wrote down who would do what. What it did not write down was who would **see** what. Two roles can be jointly responsible for the same service and still see the same occurrence at a different step, on a different surface, sometimes not at all. **Shared responsibility** holds the development and operations roles jointly accountable for a service's outcome. A handover contract divides *items*; shared responsibility divides *the outcome*, and the second requires, as a precondition, that both roles can see the same occurrence. This lesson measures that precondition: the same set of occurrences is dropped into two roles' observation scope, how many occurrences stay visible to only one side is counted, and the bill for combining scope is worked out. #### Observation Scope Is a Data Structure The example is this curriculum's fictional **regional measurement network**: software that collects readings from a municipality's water meters, verifies the readings, turns them into invoices, and opens work orders for field crews. The network and the teams are both fictional. **CF1:** there are three environments (development, test, production), two teams (development and operations), and one nightly batch job. **CF2:** there are eight observation surfaces; three sit in the development team's scope, four in the operations team's scope, and `invoice-statement` sits in no one's scope. **CF3:** twenty-two occurrences are born within one window; each occurrence has a class (error, delay, drift), a birth step, and the surfaces it appears on together with its delay on each surface. **CF4:** every occurrence has an owner who can close it — development, operations, or shared; a shared occurrence cannot be closed until it is visible in both scopes, and closing takes 4, 2, and 5 steps respectively. **CF5:** an occurrence visible in a scope writes 2 steps of triage for that role; an occurrence that has not reached its owner within 30 steps is surfaced from above, and because the context has to be rebuilt secondhand this adds 2 extra steps. **CF6:** the development team works toward a release during its idle steps, and a release needs 8 steps of work; every occurrence attaches to the last release before it, and a release with an attached occurrence that never reaches its owner counts as failed. The window is 140 steps. The occurrences, the surfaces, and the two roles' rotation are a **process-internal model**; there is no real alert stream or incident log. ```js // observation/scope.mjs — the fictional regional measurement network's occurrence and observation-scope // model (model): there is no real alert stream or incident log, steps are a data structure. No randomness. export const WINDOW = 140; // measured step window export const TRIAGE = 2; // cost to one role of triaging an occurrence (steps) export const RELEASE_WORK = 8; // work steps a release needs export const CLOSURE = { dev: 4, ops: 2, shared: 5 }; export const CEILING = 30; // an occurrence that never reaches its owner surfaces at this delay export const SECOND_HAND = 2; // extra context cost of an occurrence surfaced this way export const SURFACE = { // surface -> whose scope it sits in, in the separate arrangement "app-record": "dev", "error-tracking": "dev", "release-log": "dev", "work-schedule": "ops", "machine-metrics": "ops", "alert-stream": "ops", "support-record": "ops", "invoice-statement": "-", }; export const ROLE = ["dev", "ops"]; const table = (s, f) => s.trim().split("\n").map((r) => f(r.trim().split(/\s+/))); // occurrence: birth step, class, role that can close it, name, surfaces it appears on (surface:delay) export const OCCURRENCE = table(` 2 error dev reading-parser app-record:0 support-record:12 6 delay ops batch-overflow work-schedule:1 machine-metrics:2 11 drift dev untested-setting release-log:0 alert-stream:9 17 error ops queue-jam machine-metrics:0 alert-stream:1 21 delay dev query-slowdown app-record:3 machine-metrics:1 26 error dev duplicate-invoice support-record:16 31 drift ops disk-full machine-metrics:0 36 error dev blank-address-work-order error-tracking:2 support-record:9 42 delay ops network-latency machine-metrics:1 alert-stream:2 46 drift dev library-drift release-log:1 51 error shared nightly-job-incomplete work-schedule:1 app-record:4 57 error dev negative-consumption invoice-statement:0 support-record:14 61 delay ops backup-collision work-schedule:2 machine-metrics:3 66 drift shared timezone-difference app-record:6 work-schedule:3 71 error dev rounding-difference invoice-statement:0 77 delay dev cache-warmup error-tracking:2 machine-metrics:1 81 error ops connection-pool alert-stream:1 app-record:0 86 drift dev missing-env-var app-record:0 alert-stream:5 91 error dev meter-id-conflict error-tracking:2 support-record:10 96 delay ops job-rerun work-schedule:1 102 drift dev format-break invoice-statement:0 support-record:15 107 error shared duplicate-work-order app-record:3 support-record:8`, ([born, cls, owner, name, ...y]) => ({ born: +born, cls, owner, name, surface: y.map((p) => ({ name: p.split(":")[0], delay: +p.split(":")[1] })) })); // Arrangement: on top of the separate arrangement, which role additionally sees which surface (contrast run). export const ARRANGEMENT = { separate: { dev: [], ops: [] }, selected: { dev: ["support-record"], ops: ["release-log", "invoice-statement"] }, combined: { dev: Object.keys(SURFACE), ops: Object.keys(SURFACE) }, }; export const scope = (d) => Object.fromEntries(ROLE.map((r) => [r, new Set([...Object.keys(SURFACE).filter((y) => SURFACE[y] === r), ...ARRANGEMENT[d][r]])])); export function run(d) { const S = scope(d); const G = OCCURRENCE.map((o) => { // in which step each occurrence is seen by each role const g = Object.fromEntries(ROLE.map((r) => { const v = o.surface.filter((y) => S[r].has(y.name)).map((y) => y.delay); return [r, v.length ? Math.min(...v) : null]; })); const closeStep = o.owner === "shared" // a shared occurrence must be seen in both scopes ? (g.dev === null || g.ops === null ? null : Math.max(g.dev, g.ops)) : g[o.owner]; return { o, g, closeStep, ownerRole: o.owner === "shared" ? "dev" : o.owner }; }); const queue = Object.fromEntries(ROLE.map((r) => [r, []])); const remaining = Object.fromEntries(ROLE.map((r) => [r, 0])); const load = Object.fromEntries(ROLE.map((r) => [r, 0])); const closure = new Map(), releases = []; let accumulated = 0, start = 0; for (let t = 0; t <= WINDOW; t++) { for (const x of G) { for (const r of ROLE) // an occurrence seen in scope opens triage work if (x.g[r] !== null && t === x.o.born + x.g[r]) queue[r].push([TRIAGE, null]); if (x.closeStep !== null && t === x.o.born + x.closeStep) queue[x.ownerRole].push([CLOSURE[x.o.owner], x]); else if (x.closeStep === null && t === x.o.born + CEILING) // surfaced from above queue[x.ownerRole].push([CLOSURE[x.o.owner] + SECOND_HAND, x]); } for (const r of ROLE) { if (remaining[r] === 0 && queue[r].length) [remaining[r]] = queue[r][0]; if (remaining[r] > 0) { // role is busy with an occurrence this step remaining[r]--; load[r]++; if (remaining[r] === 0) { const [, x] = queue[r].shift(); if (x) closure.set(x.o.name, t + 1); } continue; } if (r !== "dev") continue; // idle step: the dev team works toward a release if (accumulated === 0) start = t; if (++accumulated === RELEASE_WORK) { releases.push({ t: t + 1, start, occurrence: [] }); accumulated = 0; } } } for (const x of G) { // an occurrence attaches to the release before it const y = [...releases].reverse().find((v) => v.t <= x.o.born); if (y) y.occurrence.push(x); } return { G, load, closure, releases }; } export const avg = (a) => (a.length ? (a.reduce((s, v) => s + v, 0) / a.length).toFixed(1) : "-"); export function measure(d) { const R = run(d), count = (f) => R.G.filter(f).length; const reach = R.G.map((x) => (x.closeStep === null ? CEILING : x.closeStep)); // ceiling for those never reached const duration = R.G.map((x) => (R.closure.get(x.o.name) ?? WINDOW) - x.o.born); const broken = R.releases.filter((y) => y.occurrence.some((x) => x.closeStep === null)).length; return { R, devOnly: count((x) => x.g.dev !== null && x.g.ops === null), opsOnly: count((x) => x.g.ops !== null && x.g.dev === null), both: count((x) => x.g.dev !== null && x.g.ops !== null), neither: count((x) => x.g.dev === null && x.g.ops === null), unreached: count((x) => x.closeStep === null), reachStep: avg(reach), noiseDev: count((x) => x.g.dev !== null && x.o.owner === "ops"), noiseOps: count((x) => x.g.ops !== null && x.o.owner === "dev"), loadDev: R.load.dev, loadOps: R.load.ops, recovery: avg(duration), releaseCount: R.releases.length, leadTime: avg(R.releases.map((y) => y.t - y.start)), changeFailureRate: R.releases.length ? (100 * broken / R.releases.length).toFixed(0) + "%" : "-" }; } ``` The four delivery metrics are read from the same run: **deployment frequency** is the number of releases in the window, **lead time** is the steps from when work starts on a release to when it goes out, **change failure rate** is the share of failed releases, **time to restore** is the steps from an occurrence's birth to its closure. ```js // observation/measure.mjs — how an occurrence falls into the two roles' observation scope, and what combining the scopes costs. import { OCCURRENCE, SURFACE, ROLE, measure, scope, CEILING } from "./scope.mjs"; const print = (g, ...s) => console.log(s.map((v, i) => (g[i] < 0 ? String(v).padEnd(-g[i]) : String(v).padStart(g[i]))).join("")); const D = ["separate", "selected", "combined"]; const R = Object.fromEntries(D.map((d) => [d, measure(d)])); console.log(`${OCCURRENCE.length} occurrences, ${Object.keys(SURFACE).length} observation surfaces; ` + ["error", "delay", "drift"].map((s) => `${s} ${OCCURRENCE.filter((o) => o.cls === s).length}`).join(", ")); for (const r of ROLE) console.log(` ${r}: ${[...scope("separate")[r]].join(", ")}`); console.log(` in no one's scope: ` + Object.keys(SURFACE).filter((y) => SURFACE[y] === "-").join(", ")); const A = [-33, 9, 9, 11]; console.log("\n1. same 22 occurrences, three observation arrangements"); print(A, "measure", "separate", "selected", "combined"); for (const [ad, k] of [["dev-only scope", "devOnly"], ["ops-only scope", "opsOnly"], ["both scopes", "both"], ["no scope", "neither"], ["never reaches owner", "unreached"], [`owner reach (avg. step)`, "reachStep"], ["noise to dev (occurrences)", "noiseDev"], ["noise to ops (occurrences)", "noiseOps"], ["dev load (steps)", "loadDev"], ["ops load (steps)", "loadOps"]]) print(A, ad, ...D.map((d) => R[d][k])); console.log("\n2. four delivery metrics, same window"); print(A, "metric", "separate", "selected", "combined"); for (const [ad, k] of [["deployment frequency (releases)", "releaseCount"], ["lead time (avg. step)", "leadTime"], ["change failure rate", "changeFailureRate"], ["time to restore (avg. step)", "recovery"]]) print(A, ad, ...D.map((d) => R[d][k])); const B = [-11, 12, 10, 10, 22]; console.log("\n3. in the separate arrangement, which scope an occurrence class falls into"); print(B, "class", "occurrence", "dev-only", "ops-only", "never reaches owner"); for (const s of ["error", "delay", "drift"]) { const g = R.separate.R.G.filter((x) => x.o.cls === s); print(B, s, g.length, g.filter((x) => x.g.dev !== null && x.g.ops === null).length, g.filter((x) => x.g.ops !== null && x.g.dev === null).length, g.filter((x) => x.closeStep === null).length); } console.log("\n4. occurrences that never reach their owner in the separate arrangement (ceiling " + CEILING + " steps)"); const C = [-26, -9, 10, 22]; print(C, "occurrence", "owner", "closure", "scope it appears in"); for (const x of R.separate.R.G.filter((x) => x.closeStep === null)) print(C, x.o.name, x.o.owner, R.separate.R.closure.get(x.o.name) - x.o.born, ROLE.filter((r) => x.g[r] !== null).join("+") || "neither"); ``` ``` 22 occurrences, 8 observation surfaces; error 10, delay 6, drift 6 dev: app-record, error-tracking, release-log ops: work-schedule, machine-metrics, alert-stream, support-record in no one's scope: invoice-statement 1. same 22 occurrences, three observation arrangements measure separate selected combined dev-only scope 1 0 0 ops-only scope 9 7 0 both scopes 11 15 22 no scope 1 0 0 never reaches owner 4 1 0 owner reach (avg. step) 7.0 5.0 1.6 noise to dev (occurrences) 1 1 7 noise to ops (occurrences) 10 12 12 dev load (steps) 95 95 107 ops load (steps) 54 58 58 2. four delivery metrics, same window metric separate selected combined deployment frequency (releases) 5 5 4 lead time (avg. step) 25.8 27.0 34.8 change failure rate 40% 20% 0% time to restore (avg. step) 13.5 11.9 10.2 3. in the separate arrangement, which scope an occurrence class falls into class occurrence dev-only ops-only never reaches owner error 10 0 3 3 delay 6 0 4 0 drift 6 1 2 1 4. occurrences that never reach their owner in the separate arrangement (ceiling 30 steps) occurrence owner closure scope it appears in duplicate-invoice dev 42 ops negative-consumption dev 42 ops rounding-difference dev 40 neither format-break dev 36 ops ``` The numbers are of the **measurement** kind; their inputs are the assumptions above. #### The Same Occurrence, Two Separate Scopes The first table's `separate` column is today's split. Eleven of the twenty-two occurrences appear in both scopes, nine only in operations' scope, one only in development's scope, and one appears in **neither scope** — `rounding-difference`, which falls on the `invoice-statement` surface, is on no one's rotation. The third table breaks this down by class: four of the six delays sit only in operations' scope, because delay falls on machine metrics and the work schedule; one of the six drifts sits only in development's scope, because only the development team reads the release log. The real number is in the bottom row: **four occurrences never reach their owner.** The fourth table lists them by name, and the owner of all four is the development team. Three of them appear in operations' scope — they fall on the support record, so the operations team sees them but cannot close them; one appears nowhere. Being surfaced from above takes 30 steps, and their closure falls between 36 and 42 steps. Averaged together with the occurrences that do reach their owner within the same window, **the average time for a signal to reach its owner is 7.0 steps.** The ten occurrences in operations' scope are occurrences operations cannot close; in other words, in the separate arrangement the operations team is already looking at a pile of occurrences it cannot close. **A split observation scope does not destroy the signal — it puts it on the wrong side.** #### The Bill for Combining Scope The `combined` column gives both roles all eight surfaces. Visibility becomes complete: all twenty-two occurrences appear in both scopes, zero never reach their owner, and the average reach time drops from 7.0 to **1.6 steps**. The cost is in the same column: the noise landing on the development team — occurrences it cannot close — rises from 1 to 7, and its triage load rises from 95 to 107 steps. Those twelve steps show up in the second table as a delivery metric. Deployment frequency falls from 5 to 4 releases, and lead time rises from 25.8 to 34.8 steps. In exchange, the change failure rate falls from 40% to 0%, and time to restore falls from 13.5 to 10.2 steps. **Two of the four metrics improve, and two get worse;** a sentence that looks at only one metric cannot carry this decision. The `selected` column is a third option: only three of the eight surfaces are shared — the development team sees the support record, and the operations team sees the release log and the invoice statement. Occurrences that never reach their owner fall from 4 to 1, average reach time falls from 7.0 to 5.0 steps, the change failure rate falls from 40% to 20%, and time to restore falls from 13.5 to 11.9 steps. Deployment frequency stays at 5 releases, and lead time rises by only 1.2 steps. The development team's load does not change at all: **95 steps stays 95 steps.** The triage work brought by the three surfaces exactly covers the secondhand context-building cost of the three occurrences that are no longer surfaced from above. #### Where the Difference Hides In this lesson, the difference hides in **the split of the observation scope**, and its number is this: seven of the eight surfaces sit in a single role, one sits in no role; ten of the twenty-two occurrences appear in a single scope, one in neither. The declared part of the difference is the surface split — which team looks at what is known. The undeclared part is which occurrence that split leaves on which side; that is only visible in the run. How many steps it takes the signal to reach whom also reads from here: in the separate arrangement it reaches its owner in an average of 7.0 steps, in 5.0 steps once three surfaces are shared, and in 1.6 steps once the scopes are combined. This is the technical side of shared responsibility — who the surfaces are opened to; the organizational side is whose flow the triage load of an opened surface is deducted from. #### Summary - Twenty-two occurrences are split across the separate observation scopes: 11 appear in both scopes, 9 only in operations' scope, 1 only in development's scope, 1 in neither; 4 occurrences never reach their owner, and the owner of all four is the development team. - The signal reaching its owner takes an average of 7.0 steps in the separate arrangement, 5.0 steps once three surfaces are shared, and 1.6 steps once the scopes are combined. - The cost of combining falls on the development team's flow: noise rises from 1 to 7 occurrences and load rises from 95 to 107 steps; deployment frequency falls from 5 to 4 releases and lead time stretches from 25.8 to 34.8 steps. - In exchange, the change failure rate falls from 40% to 0% and time to restore falls from 13.5 to 10.2 steps: two of the four metrics improve, and two get worse. - Selectively sharing three surfaces brings most of the improvement (1 unreached, 5.0-step reach, 20% change failure rate) and leaves the development team's load fixed at 95 steps. #### Next Step When observation scope is split, the signal arrives late or never — that is now measured. But the split is not only in observation: the work itself is split too, and a change gets **thrown** from team to team. The next lesson runs the same set of changes through two arrangements — separate teams and shared responsibility — and counts the wait per boundary, the changes that bounce back because context could not cross it, and the number of errors that surface in production. ### Throwing Over the Wall Source: https://academia.sh/en/courses/devops-fundamentals/throwing-over-the-wall The delay and risk produced by separate teams: of the 29 context items carried by the same 18 changes, 21 are not written down; in the separate arrangement, 14 items drop at the boundary, producing 10 boundary bounces and 4 production errors; under shared responsibility the dropped items fall to 5, bounces to 3, production errors to 2, and the change failure rate falls from 22% to 11% — but deployment frequency falls from 9.5 to 7.9, wait at the internal transition rises from 8.1 to 12.2 steps, and time to restore rises from 61.0 to 65.0 steps; for the 7 changes carrying silent context, both arrangements produce the same 3 bounces and 2 errors. The previous lesson measured the split in observation scope: four of the twenty-two occurrences never reached their owner, and sharing three surfaces brought the reach time down from 7.0 to 5.0 steps. Observation is not the only thing that is split. A change's path from idea to production is split too, and the change is **thrown** over that split. **Throwing over the wall** is handing a change across a team boundary before the context it carries has been written down. What gets thrown is not the change itself but what is known about it: which setting changed, which data format is expected, how to roll it back. This lesson runs the same set of changes through two arrangements and counts the context that drops at the boundary. The Process, Team and Delivery course counted handoff along a work item's path across teams; the scale here is different — what is counted is the team boundary on a change's path from idea to production, and the context that drops at that boundary. #### The Change's Path and the Context It Carries The example is again the fictional **regional measurement network**; the network and the teams are both fictional. **CF7:** a change's path is seven steps — definition, writing, review, test setup, acceptance test, production setup, production verification; in the separate arrangement the last four steps sit with the operations team. **CF8:** eighteen changes enter the flow eight steps apart, and each team runs a single change at a time. **CF9:** every change carries context items; where an item sits depends on its class — setting, data format, dependency, and rollback sit with the development team, scheduling and resource sit with the operations team. **CF10:** an item is in one of three states: **written** (in hand for both roles), **askable** (not written but can be asked of the other side), **silent** (not written and no one knows it is missing). **CF11:** if the role running the step is not the role the item sits with and the item is not written, then under shared responsibility an askable item passes at a cost of 2 steps of asking, and a silent item drops; in the separate arrangement, both drop. A dropped item is learned in that same run. **CF12:** if the dropped item is before production verification, the change returns to review and writes 3 steps of rework; if it is at production verification, the change becomes a production error, takes 4 steps, and the fix path runs from the start. A team boundary crossing writes 5 steps of wait, an internal one writes 1. Under shared responsibility, test setup and production verification pass to the development team. The path and the two arrangements are a **process-internal model**; there is no real pipeline or change record. ```js // wall/flow.mjs — a change's path from idea to production (model): there is no real // pipeline or change record, steps are a data structure. No randomness. export const BOUNDARY = 5, INTERNAL = 1, ASK = 2, REWORK = 3, REPAIR = 4, ARRIVAL = 8, LIMIT = 900; export const REVIEW_STEP = 2; // a change bouncing at the boundary returns to review export const ROLE = ["dev", "ops"]; export const STEP = [["definition", "dev", 2], ["writing", "dev", 0], ["review", "dev", 2], ["test-setup", "ops", 2], ["acceptance-test", "ops", 3], ["production-setup", "ops", 2], ["production-verification", "ops", 2]] .map(([name, role, duration]) => ({ name, role, duration })); export const SHARED = new Set(["test-setup", "production-verification"]); const SHORT = { k: "test-setup", s: "acceptance-test", u: "production-setup", d: "production-verification" }; // source of a context item's class: setting/data-format/dependency/rollback sit with dev, // scheduling and resource sit with ops. Item state: w written, a not written but askable, // s not written and silent (no one knows it is missing). export const OWNER = { setting: "dev", "data-format": "dev", dependency: "dev", rollback: "dev", scheduling: "ops", resource: "ops" }; // change: name, writing step size, context items it carries (class/step it is needed at/state) export const CHANGE = ` c01 5 setting/k/a data-format/s/w c02 3 scheduling/u/a c03 6 dependency/k/w resource/u/a c04 4 setting/s/a rollback/d/a c05 7 data-format/k/s c06 3 scheduling/d/s setting/u/w c07 5 resource/s/w dependency/u/a c08 4 rollback/d/a c09 6 setting/k/a scheduling/s/a c10 3 data-format/u/a c11 5 dependency/d/s resource/k/w c12 4 setting/s/w c13 6 scheduling/k/s rollback/u/a c14 3 resource/d/s c15 5 data-format/s/a dependency/k/w c16 4 setting/u/s scheduling/d/w c17 6 rollback/s/a c18 3 resource/k/s data-format/d/a` .trim().split("\n").map((r) => r.trim().split(/\s+/)).map(([name, size, ...k]) => ({ name, size: +size, item: k.map((p) => { const [cls, a, s] = p.split("/"); return { cls, step: SHORT[a], state: s, owner: OWNER[cls] }; }) })); export function run(arrangement) { const step = STEP.map((a) => ({ ...a, role: arrangement === "shared" && SHARED.has(a.name) ? "dev" : a.role })); const D = CHANGE.map((d, i) => ({ ...d, i, step: 0, ready: i * ARRIVAL, extra: 0, transition: null, known: new Set(d.item.filter((k) => k.state === "w").map((k) => k.cls)), lost: null, dropped: [], done: false, prodDone: null, errorAt: null, recovery: null, error: false, bounced: 0 })); const server = { dev: null, ops: null }; const S = { lost: 0, bounced: 0, prodError: 0, deploy: 0, ask: 0, rework: 0, boundaryCross: 0, boundaryWait: 0, internalCross: 0, internalWait: 0 }; const advance = (d, target, t, extra) => { const isBoundary = step[target].role !== step[d.step].role; if (isBoundary) { S.boundaryCross++; S.boundaryWait += BOUNDARY; d.transition = "boundary"; } else { S.internalCross++; S.internalWait += INTERNAL; d.transition = "internal"; } d.ready = t + (isBoundary ? BOUNDARY : INTERNAL); d.step = target; d.extra = extra; }; const start = (d, r) => { const a = step[d.step]; let duration = (a.name === "writing" ? d.size : a.duration) + d.extra; d.extra = 0; d.lost = null; for (const k of d.item.filter((k) => k.step === a.name && !d.known.has(k.cls))) { d.known.add(k.cls); // the item is learned here regardless of state if (k.owner === r) continue; // the role running this step already carries the item if (arrangement === "shared" && k.state === "a") { // it is asked of the other side, cost is a step duration += ASK; S.ask += ASK; continue; } S.lost++; d.lost = k; d.dropped.push(k); // dropped at the boundary } d.remaining = duration; }; const finish = (d, t) => { const a = step[d.step]; if (a.name === "production-setup") { S.deploy++; if (d.prodDone === null) d.prodDone = t; } if (d.lost) { const inProd = a.name === "production-verification"; if (inProd) { S.prodError++; d.error = true; d.errorAt = t; } else { S.bounced++; d.bounced++; } S.rework += inProd ? REPAIR : REWORK; return advance(d, REVIEW_STEP, t, inProd ? REPAIR : REWORK); } if (d.step < step.length - 1) return advance(d, d.step + 1, t, 0); if (d.error && d.recovery === null) d.recovery = t - d.errorAt; d.done = true; }; for (let t = 0; t < LIMIT; t++) for (const r of ROLE) { if (server[r] === null) { const candidate = D.filter((d) => !d.done && d !== server.dev && d !== server.ops && d.ready <= t && step[d.step].role === r).sort((a, b) => a.ready - b.ready || a.i - b.i)[0]; if (candidate) { if (candidate.transition === "boundary") S.boundaryWait += t - candidate.ready; else if (candidate.transition === "internal") S.internalWait += t - candidate.ready; start(candidate, r); server[r] = candidate; } } const d = server[r]; if (d && --d.remaining === 0) { server[r] = null; finish(d, t + 1); } } return { D, S, window: Math.max(...D.map((d) => d.prodDone ?? 0)) }; } export const avg = (a) => (a.length ? (a.reduce((s, v) => s + v, 0) / a.length).toFixed(1) : "-"); export function measure(arrangement) { const { D, S, window } = run(arrangement); const lead = D.map((d) => d.prodDone - d.i * ARRIVAL); return { D, S, window, boundaryCross: S.boundaryCross, waitPerBoundary: (S.boundaryWait / S.boundaryCross).toFixed(1), waitPerInternal: (S.internalWait / S.internalCross).toFixed(1), infoLoss: S.lost, ask: S.ask, bounced: S.bounced, prodErrors: S.prodError, rework: S.rework, leadTime: avg(lead), deployFrequency: (100 * S.deploy / window).toFixed(1), changeFailureRate: (100 * D.filter((d) => d.error).length / D.length).toFixed(0) + "%", recovery: avg(D.filter((d) => d.error).map((d) => d.recovery)), unfinished: D.filter((d) => !d.done).length }; } ``` The four delivery metrics are read from the same run: **deployment frequency** is production setups completed per 100 steps (including redeployments), **lead time** is the steps from a change entering the flow to its first production setup, **change failure rate** is the share of changes that produce an error in production, **time to restore** is the steps from an error surfacing to the fix being verified in production. ```js // wall/measure.mjs — the same 18 changes in two arrangements: boundary wait, information loss, production errors. import { CHANGE, STEP, SHARED, ARRIVAL, measure, avg } from "./flow.mjs"; const print = (g, ...s) => console.log(s.map((v, i) => (g[i] < 0 ? String(v).padEnd(-g[i]) : String(v).padStart(g[i]))).join("")); const item = CHANGE.flatMap((d) => d.item); const count = (f) => item.filter(f).length; const R = { separate: measure("separate"), shared: measure("shared") }; console.log(`${CHANGE.length} changes, ${STEP.length} steps, ${item.length} context items: ` + `${count((k) => k.state === "w")} written, ${count((k) => k.state === "a")} askable, ` + `${count((k) => k.state === "s")} silent`); console.log(` where items sit: ` + ["dev", "ops"].map((r) => `${r} ${count((k) => k.owner === r)}`).join(", ")); console.log(` separate: ${STEP.filter((a) => a.role === "ops").length} steps sit with ops; ` + `shared: ${[...SHARED].join(" and ")} pass to dev`); console.log(` changes unfinished in the window: separate ${R.separate.unfinished}, ` + `shared ${R.shared.unfinished}`); const A = [-38, 9, 13]; console.log("\n1. same set of changes in two arrangements"); print(A, "measure", "separate", "shared"); for (const [ad, k] of [["team boundary crossings", "boundaryCross"], ["wait per boundary (steps)", "waitPerBoundary"], ["wait per internal transition (steps)", "waitPerInternal"], ["information loss (dropped items)", "infoLoss"], ["asking cost (steps)", "ask"], ["changes bounced at the boundary", "bounced"], ["errors surfacing in production", "prodErrors"], ["rework (steps)", "rework"]]) print(A, ad, R.separate[k], R.shared[k]); console.log("\n2. four delivery metrics"); print(A, "metric", "separate", "shared"); for (const [ad, k] of [["deployment frequency (per 100 steps)", "deployFrequency"], ["lead time (avg. step)", "leadTime"], ["change failure rate", "changeFailureRate"], ["time to restore (avg. step)", "recovery"], ["last change's entry into production", "window"]]) print(A, ad, R.separate[k], R.shared[k]); const B = [-25, 7, 12, 9, 13, 16]; console.log("\n3. which step an item dropped at (dropped: separate / shared)"); print(B, "step", "items", "unwritten", "dev", "silent", "dropped"); for (const a of STEP.filter((a) => a.role === "ops")) { const g = item.filter((k) => k.step === a.name), h = g.filter((k) => k.state !== "w"); const d = (az) => R[az].D.flatMap((x) => x.dropped ?? []).filter((k) => k.step === a.name).length; print(B, a.name, g.length, h.length, h.filter((k) => k.owner === "dev").length, h.filter((k) => k.state === "s").length, `${d("separate")} / ${d("shared")}`); } const C = [-25, -10, 11, 18, 18]; console.log("\n4. path per change"); print(C, "change", "arrangement", "lead time", "boundary bounces", "production errors"); for (const [ad, f] of [["written only", (d) => d.item.every((k) => k.state === "w")], ["has askable", (d) => d.item.some((k) => k.state === "a") && !d.item.some((k) => k.state === "s")], ["has silent", (d) => d.item.some((k) => k.state === "s")]]) for (const az of ["separate", "shared"]) { const g = R[az].D.filter(f); if (!g.length) continue; print(C, az === "separate" ? `${ad} (${g.length})` : "", az, avg(g.map((d) => d.prodDone - d.i * ARRIVAL)), g.reduce((s, d) => s + d.bounced, 0), g.filter((d) => d.error).length); } ``` ``` 18 changes, 7 steps, 29 context items: 8 written, 14 askable, 7 silent where items sit: dev 19, ops 10 separate: 4 steps sit with ops; shared: test-setup and production-verification pass to dev changes unfinished in the window: separate 0, shared 0 1. same set of changes in two arrangements measure separate shared team boundary crossings 46 42 wait per boundary (steps) 11.2 11.0 wait per internal transition (steps) 8.1 12.2 information loss (dropped items) 14 5 asking cost (steps) 0 12 changes bounced at the boundary 10 3 errors surfacing in production 4 2 rework (steps) 46 17 2. four delivery metrics metric separate shared deployment frequency (per 100 steps) 9.5 7.9 lead time (avg. step) 85.6 87.2 change failure rate 22% 11% time to restore (avg. step) 61.0 65.0 last change's entry into production 275 267 3. which step an item dropped at (dropped: separate / shared) step items unwritten dev silent dropped test-setup 8 5 3 3 3 / 2 acceptance-test 7 4 3 0 3 / 0 production-setup 7 6 4 1 4 / 1 production-verification 7 6 4 3 4 / 2 4. path per change change arrangement lead time boundary bounces production errors written only (1) separate 93.0 0 0 shared 115.0 0 0 has askable (10) separate 77.2 7 2 shared 70.0 0 0 has silent (7) separate 96.4 3 2 shared 107.7 3 2 ``` The numbers are of the **measurement** kind; their inputs are the assumptions above. #### What Drops at the Boundary In the separate arrangement, fourteen items **drop** at the boundary: every unwritten item that sits with the development team but is needed at a step the operations team runs is lost. The result is ten boundary bounces, four production errors, and 46 steps of rework. Under shared responsibility, dropped items fall to 5, boundary bounces to 3, production errors to 2; in exchange, 12 steps of asking cost are paid. But the third table shows the real change is a **change in direction**. In the separate arrangement, what drops are items that sit with the development team: 3 at test setup, 3 at acceptance test, 4 at production setup, 4 at production verification. Under shared responsibility, test setup and production verification pass to the development team, so at those steps it is now items sitting with the operations team that drop — scheduling and resource. **Removing the wall does not eliminate information loss, it reverses its direction;** all five of the remaining losses are things operations knows and development does not. The fourth table sharpens the distinction. For the ten changes carrying askable context, shared responsibility brings boundary bounces from 7 to 0, production errors from 2 to 0, and lead time down from 77.2 to 70.0 steps. For the seven changes carrying silent context, **nothing changes**: both arrangements produce the same 3 bounces and the same 2 errors, and lead time even rises from 96.4 to 107.7 steps. Asking only works for something whose absence is known. #### The Cost of Removing the Wall The second table lays the four metrics side by side. The change failure rate falls from 22% to 11%; that is the only metric shared responsibility gains. Deployment frequency goes from 9.5 to 7.9, lead time from 85.6 to 87.2 steps, time to restore from 61.0 to 65.0 steps. **One of the four metrics improves, three get worse.** The reason is in the third row of the first table: wait per internal transition rises from 8.1 to 12.2 steps. Under shared responsibility the development team runs five of the seven steps, and the bottleneck moves there. Team boundary crossings per change rise from 1 to 2, yet the total falls from 46 to 42, because bounces have fallen; wait per boundary goes from 11.2 to 11.0 steps — in other words it barely changes at all. **Removing the wall does not make the boundary cheaper, it reduces how much work crosses it.** #### Where the Difference Hides In this lesson, the difference hides in **the context that is not written down**: twenty-one of the twenty-nine items are not written — fourteen are askable, seven are silent. The declared part is only eight items. The fourteen are only visible in the run, at the step where they drop; the seven are not visible even when they drop, because no one knows the item is needed at all. How many steps it takes the signal to reach whom also reads from here: if the dropped item is before production verification, the signal returns to the development team with one boundary crossing and an average wait of 11.2 steps; if it drops at production verification, the same signal arrives an average of 61.0 steps later, as an error that has already surfaced in production. **The same missing item is more than five times as expensive, depending on where it drops.** #### Summary - Of the 29 context items carried by the same 18 changes, 21 are not written down: 14 are askable, 7 are silent. - In the separate arrangement, 14 items drop at the boundary, producing 10 boundary bounces and 4 production errors and writing 46 steps of rework; under shared responsibility these become 5, 3, 2, and 17, in exchange for 12 steps of asking cost. - Loss does not disappear, it changes direction: under shared responsibility, all five dropped items are scheduling and resource items that sit with the operations team. - Of the four metrics, only the change failure rate improves (22% → 11%); deployment frequency goes from 9.5 to 7.9, lead time from 85.6 to 87.2 steps, time to restore from 61.0 to 65.0 steps — the bottleneck moves to the development team (wait per internal transition 8.1 → 12.2 steps). - For the 10 changes carrying askable context, shared responsibility brings bounces from 7 to 0; for the 7 changes carrying silent context, the same 3 bounces and 2 errors occur under both arrangements. #### Next Step Both arrangements were measured on the same path; the path itself was held fixed. But the real question is where along that path time is spent standing still. The next lesson builds the path as a **value stream map**: for every step, process time, wait time, and percent complete and accurate are measured; how much of the total lead time is value-added, where the longest wait sits, and why improving one step so often fails to show up in the total are all counted. ### Value Stream Source: https://academia.sh/en/courses/devops-fundamentals/value-stream Mapping the path from idea to production: in a nine-step flow, 20 work items' average 182.0-step lead time breaks down into 70.2 steps of process and 111.8 steps of wait (value-added share 38.5%); of the total 2237 steps of wait, 1485 sit in a single step — the release window; with the release window at 20 steps, no single-step improvement shortens lead time by more than 2 steps; once the window is cut to 5 steps, the same improvements pull apart, with writing dropping from 10 to 5 gaining 32 steps but acceptance approval dropping from 2 to 1 losing 4 steps; and the bottleneck shifts from the release window to writing, then to integration testing. The previous lesson ran the same path through two arrangements and counted the context that dropped at the boundary, but held the path itself fixed: seven steps, fixed durations. On a change's path from idea to production, the real question is not how many steps there are, but how long each of those steps spends **standing still**. A **value stream map** is a table that writes three numbers for every step on this path: process time, wait time, and percent complete and accurate. It measures the same object as the Process, Team and Delivery course's `work item` — the scale differs: there, what was measured was a work item's path across teams; here, what is measured is a change's path from idea to production. #### Building the Map The example is again the fictional **regional measurement network**; the network and the flow are both fictional. **CF13:** the path is nine steps — request queuing, analysis, writing, review, integration test, acceptance approval, release window, production setup, production verification. **CF14:** every step has a concurrent-executor count, a process time, and a percent complete and accurate; an item that is not complete and accurate the first time goes back to the step's bounce target and runs forward again from there. **CF15:** twenty work items enter the flow twelve steps apart. **Wait is not an input, it is measured:** it accumulates in the queue if the executor is busy, and in the window if the release window is closed. **CF16:** the release window opens once every twenty steps. **CF17:** randomness is written with a generator, the seed is 20260801; each item is given its own sequence of rolls, so when a step's duration changes an item's luck does not change and runs stay comparable. **CF18:** lead time is the steps from entering the flow to production verification finishing; value-added time is the total process time an item spends across the steps. The map and the flow are a **process-internal model**; there is no real flow record or release calendar. ```js // flow/map.mjs — the value stream map from idea to production (model): there is no real flow record or // release calendar, steps are a data structure. Randomness is written with a generator, the seed is visible. export const N = 20, ARRIVAL = 12, LIMIT = 4000, RELEASE_WINDOW = 20, SEED = 20260801; // step: name, concurrent executors, process time, percent complete and accurate, bounce target export const STEP = [ ["request-queuing", 1, 1, 1.00, 0], ["analysis", 1, 4, 0.85, 0], ["writing", 2, 10, 0.80, 1], ["review", 1, 3, 0.75, 2], ["integration-test", 1, 8, 0.70, 2], ["acceptance-approval", 1, 2, 0.90, 1], ["release-window", 1, 1, 1.00, 0], ["production-setup", 1, 3, 0.95, 2], ["production-verification", 1, 2, 0.90, 2], ].map(([name, cap, process, pca, bounceTo], i) => ({ name, cap, process, pca, bounceTo, i })); // linear congruential generator: each item gets its own sequence of rolls, so when a step's // process time changes an item's luck does not change, and runs stay comparable. export function rng(seed) { let x = seed >>> 0; return () => ((x = (x * 1664525 + 1013904223) >>> 0) / 2 ** 32); } export function run({ process = {}, window = RELEASE_WINDOW } = {}) { const step = STEP.map((a) => ({ ...a, process: process[a.name] ?? a.process })); const roll = rng(SEED); const K = Array.from({ length: N }, (_, i) => ({ i, arrival: i * ARRIVAL, step: 0, ready: i * ARRIVAL, running: false, remaining: 0, done: false, finishAt: null, processTotal: 0, wait: 0, rework: 0, error: false, errorAt: null, recovery: null, rolls: Array.from({ length: 80 }, roll), rollAt: 0 })); const busy = step.map(() => 0); const S = { wait: step.map(() => 0), process: step.map(() => 0), visits: step.map(() => 0), bounces: step.map(() => 0), deploys: 0 }; const finish = (d, t) => { const a = step[d.step]; busy[a.i]--; d.running = false; if (a.pca < 1 && d.rolls[d.rollAt++] >= a.pca) { // not complete and accurate the first time S.bounces[a.i]++; d.rework++; if (a.name === "production-verification" && !d.error) { d.error = true; d.errorAt = t; } d.step = a.bounceTo; d.ready = t; return; } if (a.name === "production-setup") S.deploys++; if (a.i === step.length - 1) { d.done = true; d.finishAt = t; if (d.error) d.recovery = t - d.errorAt; return; } d.step = a.i + 1; d.ready = t; }; for (let t = 0; t < LIMIT; t++) { for (const a of step) { if (a.name === "release-window" && t % window !== 0) continue; // batched release window const queue = K.filter((d) => !d.done && !d.running && d.step === a.i && d.ready <= t) .sort((x, y) => x.ready - y.ready || x.i - y.i); for (const d of queue) { if (busy[a.i] >= a.cap) break; S.wait[a.i] += t - d.ready; d.wait += t - d.ready; S.process[a.i] += a.process; S.visits[a.i]++; d.processTotal += a.process; d.running = true; d.remaining = a.process; busy[a.i]++; } } for (const d of K) if (d.running && --d.remaining === 0) finish(d, t + 1); } return { K, S, step }; } export const avg = (a) => (a.length ? (a.reduce((s, v) => s + v, 0) / a.length).toFixed(1) : "-"); export function measure(config) { const { K, S, step } = run(config); const lead = K.map((d) => d.finishAt - d.arrival); const windowEnd = Math.max(...K.map((d) => d.finishAt)); const valueAdded = K.reduce((s, d) => s + d.processTotal, 0), totalLead = lead.reduce((s, v) => s + v, 0); return { K, S, step, leadTime: avg(lead), valueAddedShare: (100 * valueAdded / totalLead).toFixed(1) + "%", longestWaitStep: step[S.wait.indexOf(Math.max(...S.wait))].name, rolledFpy: (100 * K.filter((d) => d.rework === 0).length / N).toFixed(0) + "%", bounces: K.reduce((s, d) => s + d.rework, 0), deployFrequency: (100 * S.deploys / windowEnd).toFixed(1), changeFailureRate: (100 * K.filter((d) => d.error).length / N).toFixed(0) + "%", recovery: avg(K.filter((d) => d.error).map((d) => d.recovery)), unfinished: K.filter((d) => !d.done).length }; } ``` ```js // flow/measure.mjs — reading the map: value-added share, longest wait, single-step improvement. import { STEP, N, RELEASE_WINDOW, SEED, measure, avg } from "./map.mjs"; const print = (g, ...s) => console.log(s.map((v, i) => (g[i] < 0 ? String(v).padEnd(-g[i]) : String(v).padStart(g[i]))).join("")); const T = measure({}), P = measure({ window: 5 }); console.log(`${N} work items, ${STEP.length} steps, release window ${RELEASE_WINDOW} steps, ` + `seed ${SEED}; unfinished ${T.unfinished}`); const A = [-25, 6, 10, 12, 9, 9, 9, 10]; console.log("\n1. value stream map (measured)"); print(A, "step", "cap", "process", "avg. wait", "visits", "%C&A", "bounces", "wait"); for (const a of T.step) { const visits = T.S.visits[a.i]; print(A, a.name, a.cap, a.process, (T.S.wait[a.i] / visits).toFixed(1), visits, a.pca < 1 ? `${(100 * (1 - T.S.bounces[a.i] / visits)).toFixed(0)}%` : "-", T.S.bounces[a.i], T.S.wait[a.i]); } const B = [-35, 16]; console.log("\n2. the map's totals (baseline)"); for (const [ad, v] of [["avg. lead time (step)", T.leadTime], ["process per item (step)", avg(T.K.map((d) => d.processTotal))], ["wait per item (step)", avg(T.K.map((d) => d.wait))], ["value-added share", T.valueAddedShare], ["step where the longest wait sits", T.longestWaitStep], ["rolled first-pass yield", T.rolledFpy], ["total bounces", T.bounces]]) print(B, ad, v); const C = [-25, 11, 12, 10, 12, 10]; console.log("\n3. if a single step's process time is halved (at two window settings)"); print(C, "improved step", "process", "window 20", "change", "window 5", "change"); print(C, "(baseline)", "-", T.leadTime, "-", P.leadTime, "-"); for (const a of STEP.filter((a) => a.process >= 2)) { const y = Math.floor(a.process / 2), i = { [a.name]: y }; const R = measure({ process: i }), Q = measure({ process: i, window: 5 }); print(C, a.name, `${a.process} -> ${y}`, R.leadTime, (R.leadTime - T.leadTime).toFixed(1), Q.leadTime, (Q.leadTime - P.leadTime).toFixed(1)); } const D = [-32, 10, 9, 16, 10, 12]; const K = { "baseline (window 20)": T, "bottleneck: writing 10->5": measure({ process: { writing: 5 } }), "non-bottleneck: analysis 4->2": measure({ process: { analysis: 2 } }), "window 20->5": P, "window 5 + writing 10->5": measure({ window: 5, process: { writing: 5 } }) }; console.log("\n4. four delivery metrics and value-added share"); print(D, "run", "lead time", "deploy", "change failure", "recovery", "value added"); for (const [ad, R] of Object.entries(K)) print(D, ad, R.leadTime, R.deployFrequency, R.changeFailureRate, R.recovery, R.valueAddedShare); const E = [-32, 16, 16]; console.log("\n5. where the wait sits"); print(E, "run", "longest wait", "total wait"); for (const [ad, R] of Object.entries(K)) print(E, ad, R.longestWaitStep, R.S.wait.reduce((s, v) => s + v, 0)); ``` ``` 20 work items, 9 steps, release window 20 steps, seed 20260801; unfinished 0 1. value stream map (measured) step cap process avg. wait visits %C&A bounces wait request-queuing 1 1 0.0 28 - 0 0 analysis 1 4 1.0 48 83% 8 50 writing 2 10 8.6 61 70% 18 524 review 1 3 0.0 43 74% 11 0 integration-test 1 8 5.6 32 78% 7 178 acceptance-approval 1 2 0.0 25 92% 2 0 release-window 1 1 64.6 23 - 0 1485 production-setup 1 3 0.0 23 100% 0 0 production-verification 1 2 0.0 23 87% 3 0 2. the map's totals (baseline) avg. lead time (step) 182.0 process per item (step) 70.2 wait per item (step) 111.8 value-added share 38.5% step where the longest wait sits release-window rolled first-pass yield 20% total bounces 49 3. if a single step's process time is halved (at two window settings) improved step process window 20 change window 5 change (baseline) - 182.0 - 118.0 - analysis 4 -> 2 182.0 0.0 114.0 -4.0 writing 10 -> 5 180.0 -2.0 86.0 -32.0 review 3 -> 1 182.0 0.0 115.5 -2.5 integration-test 8 -> 4 180.0 -2.0 106.0 -12.0 acceptance-approval 2 -> 1 182.0 0.0 122.0 4.0 production-setup 3 -> 1 180.0 -2.0 117.0 -1.0 production-verification 2 -> 1 181.0 -1.0 118.0 0.0 4. four delivery metrics and value-added share run lead time deploy change failure recovery value added baseline (window 20) 182.0 4.5 15% 213.3 38.5% bottleneck: writing 10->5 180.0 4.5 15% 213.3 30.5% non-bottleneck: analysis 4->2 182.0 4.5 15% 220.0 35.9% window 20->5 118.0 6.0 15% 130.0 59.4% window 5 + writing 10->5 86.0 6.7 15% 73.3 63.8% 5. where the wait sits run longest wait total wait baseline (window 20) release-window 2237 bottleneck: writing 10->5 release-window 2502 non-bottleneck: analysis 4->2 release-window 2333 window 20->5 writing 957 window 5 + writing 10->5 integration-test 622 ``` The numbers are of the **measurement** kind; their inputs are the assumptions above. The model defines value-added as "process," and rework counts as process too; that is why 38.5% is a ceiling value. #### What is Inside Lead Time Average lead time is 182.0 steps: 70.2 steps of process, 111.8 steps of wait. The value-added share is 38.5% — an item spends two-thirds of the path standing still. Rolled first-pass yield is more striking: only four of the twenty items, that is **20%**, get through the flow without ever bouncing back. The remaining sixteen items produce a total of 49 bounces. The first table distributes these 49 bounces across the steps: writing 18, review 11, analysis 8, integration test 7, production verification 3, acceptance approval 2. The visit counts are the result of this — twenty items visit the writing step 61 times. The distribution of wait, however, is not even. Five of the nine steps have zero wait. Of the total 2237 steps of wait, **1485 sit in a single step: the release window.** That is 64.6 steps per visit. The second-largest pileup is at the writing step (524 steps, 8.6 per visit). **Where the wait sits is not where the work gets done.** #### Why Improving a Step Does not Show The third table halves every step's process time and measures the change in total lead time. With the release window at 20 steps, the result is uniform: **no improvement gains more than 2 steps.** Cutting analysis from 4 to 2 gains 0.0 steps, cutting writing from 10 to 5 gains 2.0 steps. Halving a ten-step piece of work does not change even one percent of the 182-step lead time, because the time gained is waited back in the release window. Once the window is cut to 5 steps, the same improvements pull apart. Writing dropping from 10 to 5 shortens lead time by 32.0 steps; integration test dropping from 8 to 4 gains 12.0; analysis dropping from 4 to 2 gains only 4.0. Cutting acceptance approval from 2 to 1, on the other hand, **lengthens lead time by 4.0 steps** — items arrive at the bottleneck more often and more bunched up. **An improvement outside the bottleneck is not merely invisible; it can even flip sign.** #### The Bottleneck Shifts The fifth table tracks, run by run, where the longest wait sits. At baseline it is the release window. Once the window is cut to 5, it shifts to writing (total wait falls from 2237 to 957). Once writing is also halved, it shifts to integration test (622). Three runs, three separate steps. The fourth table gives the delivery-metric counterpart of this. The change failure rate is 15% in all five runs; since the percent complete and accurate values never change, this metric never moves. Deployment frequency climbs from 4.5 to 6.0, then to 6.7; time to restore falls from 213.3 to 130.0, then to 73.3 steps; the value-added share rises from 38.5% to 59.4%, then to 63.8%. Against that, **in the run that improves only the bottleneck, the value-added share falls from 38.5% to 30.5%:** process time got shorter, wait stayed exactly where it was. #### Where the Difference Hides In this lesson, the difference hides in **the distribution of wait**, and its number is this: five of the nine steps have zero wait, and one holds 66% of the total. The declared part is the process times — anyone looking at the map reads that writing takes 10 steps and integration test takes 8. The undeclared part is the wait; it is not an input, it is only born in the run, and it is what decides the improvement call. How many steps it takes the signal to reach whom also reads from the bounce targets. A drift found at integration test returns two steps back to writing, and 7 of the bounces come from there. If the same class of drift is found at production verification, it returns six steps back to writing; the time to restore for those 3 items is 213.3 steps. **The signal is the same; what is expensive is the length of the path back.** #### Summary - In a nine-step flow, average lead time is 182.0 steps: 70.2 steps of process, 111.8 steps of wait; the value-added share is 38.5% (since rework counts as process too, this is a ceiling value). - Only 20% of the twenty items get through without ever bouncing back; of the 49 bounces, 18 come from writing, 11 from review, 8 from analysis. - Of the total 2237 steps of wait, 1485 sit in a single step (the release window, 64.6 per visit); five of the nine steps have zero wait. - With the release window at 20 steps, no single-step improvement shortens lead time by more than 2 steps; once the window is cut to 5, writing gains 32.0 steps, while acceptance approval loses 4.0 steps. - The bottleneck shifts to three separate steps across three runs; deployment frequency goes 4.5 → 6.0 → 6.7 and time to restore goes 213.3 → 130.0 → 73.3 steps, while the change failure rate stays at 15% across all five runs. #### Next Step This lesson read the four metrics as a result: the map changed, and the metrics followed. But the metrics themselves are a set, and they move one another. The next lesson runs the four delivery metrics — deployment frequency, lead time, change failure rate, time to restore — as a **linked set**: the change each of the four measures for shrinking batch size, adding a quality gate, and automatic rollback is written down, and the decisions that improve one metric while breaking another are named. ### Delivery Metrics Source: https://academia.sh/en/courses/devops-fundamentals/delivery-metrics Deployment frequency, lead time, change failure rate, and time to restore as a linked set: while 12 of 60 changes are defective, shrinking batch size from 8 to 2 brings deployment frequency from 3.2 to 13.5, lead time from 34.6 down to 14.9 steps, and the release failure rate from 80% down to 31%, even as the number of errors surfacing in production stays fixed at 12; shrinking the batch to 1 pushes deployment frequency to 20.0 but carries lead time to 69.7 steps; a gate catches 8 of the defects, bringing the change failure rate from 20% down to 7% while pushing time to restore from 45.4 up to 54.8 steps; automatic rollback brings time to restore down to 7.0 steps while rolling back 48 innocent changes; and the three decisions together push lead time up to 109.3 steps. The previous lesson read the four metrics as a result of the map: the map changed, and the metrics followed. But the metrics are not independent results; they form **a set**, and the same decision moves all of them at once. This lesson reads that set from a single run. The four metrics are these. **Deployment frequency:** the number of releases reaching production per unit of time. **Lead time:** the time from a change entering the flow to taking permanent hold in production. **Change failure rate:** the share of changes that produce an error in production, that is, that fail. **Time to restore:** the time from an error surfacing in production to the service recovering. In the Process, Team and Delivery course, `delivery time` was the time a work item spent from start to finish in a team's flow; the **lead time** here waits for the change to take *permanent* hold in production — for a rolled-back change, the clock does not stop. #### Building the Set The example is again the fictional **regional measurement network**; the network and the release arrangement are both fictional. **CF19:** sixty changes enter the flow four steps apart. **CF20:** two unconditional dice are rolled for every change — whether it is defective (rate 0.20) and whether the defect would be caught at the gate (rate 0.60); the seed is 20260804 and the roll order does not change from run to run. **CF21:** changes are collected into a batch; prep starts once the batch is full or the first item has waited 30 steps. One release is prepared at a time, and prep takes 4 plus the batch size steps. **CF22:** if there is a gate, it adds 5 steps to prep and runs before release; a defect it catches is pulled from the batch, fixed in 6 steps, and re-enters the queue. **CF23:** if a release ships an uncaught defect, it is an error in production; without automatic rollback, sound changes stay in production, and the defective one returns to the queue after 12 steps of detection and 6 of fixing, with time to restore counted until the fix ships. **CF24:** with automatic rollback, the whole release is rolled back with 3 steps of detection and 4 of rollback; time to restore is 7 steps, but every change in the batch returns to the queue. **CF25:** a fixed change never becomes defective again. The release arrangement and the decisions are a **process-internal model**; there is no real release record. ```js // metric/set.mjs — the model of the four delivery metrics as a linked set (model): there is no real // release record, steps are a data structure. Randomness is written with a generator, the seed is visible. export const N = 60, ARRIVAL = 4, LIMIT = 4000, SEED = 20260804; export const DEFECT = 0.20, GATE_CATCH = 0.60; // defect rate and the gate's catch rate export const PREP0 = 4, PREP1 = 1, GATE = 5; // prep = PREP0 + PREP1 * batch (+ gate) export const DETECT = 12, FIX = 6; // manual detection and fix export const AUTO_DETECT = 3, ROLLBACK = 4; // automatic detection and rollback export const MAX_WAIT = 30; // a half-batch waiting this long still ships export function rng(seed) { // linear congruential generator let x = seed >>> 0; return () => ((x = (x * 1664525 + 1013904223) >>> 0) / 2 ** 32); } export function run({ batch = 8, gate = false, automatic = false } = {}) { const roll = rng(SEED); const D = Array.from({ length: N }, (_, i) => ({ i, arrival: i * ARRIVAL, ready: i * ARRIVAL, status: "in-transit", defective: roll() < DEFECT, catchable: roll() < GATE_CATCH, finishAt: null, errorAt: null, recovery: null, rolledBackCount: 0, queuedAt: 0 })); const S = { releases: 0, brokenReleases: 0, caughtAtGate: 0, rollbacks: 0, prodErrors: 0, batchTotal: 0, rolledBackDefective: 0, rolledBackInnocent: 0 }; let active = null; const releaseFinish = (rel, t) => { let group = rel.group; if (gate) { // a gate catches part of the defects before release const caught = group.filter((d) => d.defective && d.catchable); S.caughtAtGate += caught.length; for (const d of caught) { d.status = "in-transit"; d.ready = t + FIX; d.defective = false; } group = group.filter((d) => !caught.includes(d)); } S.releases++; S.batchTotal += group.length; const broken = group.filter((d) => d.defective); if (!broken.length) { for (const d of group) { d.status = "done"; d.finishAt = t; } return; } S.brokenReleases++; S.prodErrors += broken.length; if (!automatic) { // an innocent change stays in production for (const d of group) { if (!d.defective) { d.status = "done"; d.finishAt = t; continue; } d.errorAt = t; d.defective = false; d.status = "in-transit"; d.ready = t + DETECT + FIX; } return; } S.rollbacks++; // the whole release is rolled back, innocents too for (const d of group) { const wasDefective = d.defective; if (wasDefective) { d.errorAt = t; d.defective = false; d.recovery = AUTO_DETECT + ROLLBACK; S.rolledBackDefective++; } else S.rolledBackInnocent++; d.rolledBackCount++; d.status = "in-transit"; d.ready = t + AUTO_DETECT + ROLLBACK + (wasDefective ? FIX : 0); } }; for (let t = 0; t <= LIMIT; t++) { for (const d of D) if (d.status === "in-transit" && d.ready <= t) { d.status = "queued"; d.queuedAt = t; } if (active && active.finishAt === t) { releaseFinish(active, t); active = null; } if (active) continue; // one release prepared at a time const K = D.filter((d) => d.status === "queued").sort((a, b) => a.queuedAt - b.queuedAt || a.i - b.i); if (!K.length) continue; if (K.length < batch && t - K[0].queuedAt < MAX_WAIT) continue; const group = K.slice(0, batch); for (const d of group) d.status = "releasing"; active = { group, finishAt: t + PREP0 + PREP1 * group.length + (gate ? GATE : 0) }; } for (const d of D) if (d.errorAt !== null && d.recovery === null && d.finishAt !== null) d.recovery = d.finishAt - d.errorAt; // manual recovery: until the fix ships return { D, S }; } export const avg = (a) => (a.length ? (a.reduce((s, v) => s + v, 0) / a.length).toFixed(1) : "-"); export function measure(config) { const { D, S } = run(config); const windowEnd = Math.max(...D.map((d) => d.finishAt ?? 0)); const recoveries = D.filter((d) => d.recovery !== null).map((d) => d.recovery); return { D, S, windowEnd, deployFrequency: (100 * S.releases / windowEnd).toFixed(1), releases: S.releases, avgBatch: (S.batchTotal / S.releases).toFixed(1), leadTime: avg(D.map((d) => d.finishAt - d.arrival)), releaseFailureRate: (100 * S.brokenReleases / S.releases).toFixed(0) + "%", changeFailureRate: (100 * S.prodErrors / N).toFixed(0) + "%", recovery: avg(recoveries), prodErrors: S.prodErrors, caughtAtGate: S.caughtAtGate, rolledBack: D.reduce((s, d) => s + d.rolledBackCount, 0), rolledBackDefective: S.rolledBackDefective, rolledBackInnocent: S.rolledBackInnocent, rollbacks: S.rollbacks, unfinished: D.filter((d) => d.finishAt === null).length }; } ``` ```js // metric/measure.mjs — the change a decision measures across four metrics; the decision that improves one metric while breaking another. import { N, SEED, DEFECT, measure, avg } from "./set.mjs"; const print = (g, ...s) => console.log(s.map((v, i) => (g[i] < 0 ? String(v).padEnd(-g[i]) : String(v).padStart(g[i]))).join("")); const K = { "baseline (batch 8)": {}, "batch 8 -> 2": { batch: 2 }, "batch 8 -> 1": { batch: 1 }, "gate added": { gate: true }, "automatic rollback": { automatic: true }, "batch 2 + automatic": { batch: 2, automatic: true }, "batch 2 + gate + automatic": { batch: 2, gate: true, automatic: true }, }; const R = Object.fromEntries(Object.entries(K).map(([ad, a]) => [ad, measure(a)])); const T = R["baseline (batch 8)"]; console.log(`${N} changes, defect rate ${DEFECT}, seed ${SEED}; ` + `defective ${T.D.filter((d) => d.errorAt !== null).length}, ` + `unfinished ${Object.values(R).reduce((s, r) => s + r.unfinished, 0)}`); const A = [-27, 11, 11, 16, 16, 11]; console.log("\n1. four metrics as a linked set"); print(A, "run", "deploy", "lead time", "change failure", "change failure", "recovery"); print(A, "", "(100 steps)", "(steps)", "(release)", "(change)", "(steps)"); for (const [ad, r] of Object.entries(R)) print(A, ad, r.deployFrequency, r.leadTime, r.releaseFailureRate, r.changeFailureRate, r.recovery); const B = [-27, 9, 12, 13, 16, 14]; console.log("\n2. the decision's mechanics, in the same runs"); print(B, "run", "releases", "avg. batch", "prod errors", "caught at gate", "rolled back"); for (const [ad, r] of Object.entries(R)) print(B, ad, r.releases, r.avgBatch, r.prodErrors, r.caughtAtGate, r.rolledBack); const C = [-27, 11, 12, 24, 12]; console.log("\n3. raw difference from baseline (direction reads separately per metric)"); print(C, "decision", "deploy", "lead time", "change failure/release", "recovery"); const dev = (r, k) => { const d = +r[k].replace("%", "") - +T[k].replace("%", ""); return (d > 0 ? "+" : "") + d.toFixed(1); }; for (const ad of Object.keys(R).slice(1)) print(C, ad, dev(R[ad], "deployFrequency"), dev(R[ad], "leadTime"), dev(R[ad], "releaseFailureRate") + " points", dev(R[ad], "recovery")); const E = [-27, 9, 18, 12, 10, 26]; console.log("\n4. innocent changes in a rolled-back release"); print(E, "run", "rollbacks", "change-instances", "defective", "innocent", "innocent items' lead time"); for (const ad of ["automatic rollback", "batch 2 + automatic", "batch 2 + gate + automatic"]) { const r = R[ad], g = r.D.filter((d) => d.rolledBackCount > 0 && d.errorAt === null); print(E, ad, r.rollbacks, r.rolledBack, r.rolledBackDefective, r.rolledBackInnocent, `${avg(g.map((d) => d.finishAt - d.arrival))} (baseline ${T.leadTime})`); } ``` ``` 60 changes, defect rate 0.2, seed 20260804; defective 12, unfinished 0 1. four metrics as a linked set run deploy lead time change failure change failure recovery (100 steps) (steps) (release) (change) (steps) baseline (batch 8) 3.2 34.6 80% 20% 45.4 batch 8 -> 2 13.5 14.9 31% 20% 26.3 batch 8 -> 1 20.0 69.7 17% 20% 65.8 gate added 3.1 36.8 40% 7% 54.8 automatic rollback 5.1 50.6 50% 20% 7.0 batch 2 + automatic 15.4 24.1 27% 20% 7.0 batch 2 + gate + automatic 9.0 109.3 11% 7% 7.0 2. the decision's mechanics, in the same runs run releases avg. batch prod errors caught at gate rolled back baseline (batch 8) 10 7.2 12 0 0 batch 8 -> 2 36 2.0 12 0 0 batch 8 -> 1 72 1.0 12 0 0 gate added 10 6.4 4 8 0 automatic rollback 16 7.5 12 0 60 batch 2 + automatic 41 2.0 12 0 22 batch 2 + gate + automatic 38 1.8 4 8 8 3. raw difference from baseline (direction reads separately per metric) decision deploy lead time change failure/release recovery batch 8 -> 2 +10.3 -19.7 -49.0 points -19.1 batch 8 -> 1 +16.8 +35.1 -63.0 points +20.4 gate added -0.1 +2.2 -40.0 points +9.4 automatic rollback +1.9 +16.0 -30.0 points -38.4 batch 2 + automatic +12.2 -10.5 -53.0 points -38.4 batch 2 + gate + automatic +5.8 +74.7 -69.0 points -38.4 4. innocent changes in a rolled-back release run rollbacks change-instances defective innocent innocent items' lead time automatic rollback 8 60 12 48 64.5 (baseline 34.6) batch 2 + automatic 11 22 12 10 35.8 (baseline 34.6) batch 2 + gate + automatic 4 8 4 4 163.3 (baseline 34.6) ``` The numbers are of the **measurement** kind; their inputs are the assumptions above. #### Shrinking Batch Size: The Metric That Moves, the Ground That Does not At baseline the batch is eight: ten releases, an average of 7.2 changes, deployment frequency 3.2, lead time 34.6 steps, recovery 45.4 steps. The release failure rate is **80%** — eight of the ten releases carry a defect. Once the batch shrinks to two, all four metrics improve relative to baseline: deployment frequency goes to 13.5, lead time to 14.9 steps, recovery to 26.3 steps, the release failure rate to 31%. The second table underlines this: **the number of errors surfacing in production stays at 12.** The change failure rate is also 20% — the same as baseline. The release failure rate falling from 80% to 31% is not a quality gain, it is the same twelve errors spread across more releases. **A metric improves by forty-nine points while nothing on the ground has changed.** Shrinking the batch to one shows this limit. Deployment frequency climbs to 20.0, the number of releases to 72, and the release failure rate falls to 17% — the best value on three metrics. But lead time climbs from 34.6 to **69.7 steps** and recovery from 45.4 to 65.8 steps. The reason is the fixed share of prep: a fixed cost of 4 steps per release becomes 5 steps per change at batch one, and the arrival interval is 4 steps. The flow saturates. **The gain from shrinking batch size ends where the fixed release cost runs out.** #### Adding a Gate The gate is the only decision that genuinely reduces errors in production: eight of the twelve defects are caught before release, four get through, and the change failure rate falls from 20% to **7%**. This is the one improvement that does not come from the metric's own definition. The bill is in the others. Since 5 steps are added to every release, lead time goes from 34.6 to 36.8 steps and deployment frequency from 3.2 to 3.1. Time to restore also gets worse: from 45.4 to **54.8 steps**. The four defects that get past the gate have their fix pass through the same gate too. **Every gate placed on a release is also a gate on the fix.** #### Automatic Rollback and the Innocent Change Automatic rollback brings time to restore from 45.4 down to **7.0 steps** — the single largest gain across the four metrics (−38.4). Its cost is in the fourth table: across eight rollback events, sixty change-instances are rolled back, and **forty-eight of them are innocent.** The innocent changes' lead time is 64.5 steps; baseline is 34.6. Overall lead time climbs from 34.6 to 50.6 steps. The cost depends on batch size. Once the batch is shrunk to two and automatic rollback is turned on, the forty-eight rolled-back innocent changes fall to ten, the innocents' lead time pulls in to 35.8 steps, and the set balances out: deployment 15.4, lead time 24.1, recovery 7.0. **The size of the rollback unit is the cost of the rollback.** Applying all three decisions at once breaks this. Batch two, the gate, and automatic rollback together bring the release failure rate to 11% and recovery to 7.0 steps, but push lead time to **109.3 steps**: for two changes, 4 plus 2 plus 5, that is 11 steps of prep, comes to 5.5 steps per change, and the arrival interval is 4 steps. **The sum of three decisions that each improve a separate metric produces the worst lead time in the set.** #### Where the Difference Hides In this lesson, the difference hides in **the metric's own definition**, and its number is this: across the same seven runs, the release-based failure rate swings between 11% and 80%, the change-based rate takes only two values (20% and 7%), and the number of errors surfacing in production is 12 in five of the seven runs. The declared part is the rate itself; the undeclared part is the denominator. Seven runs and four metrics give twenty-eight measurements: in four of the six decisions at least one metric gets worse, in two (batch 2, and batch 2 plus automatic) all four improve — and in both of those two, the number of errors on the ground stays fixed at 12. How many steps it takes the signal to reach whom also changes with these decisions: with automatic rollback, a defect is detected in 3 steps and closes in 7; detected manually, it takes 12 steps and averages 45.4 steps until the fix ships. The Testing Process and Automation Infrastructure course's quality indicators measured a change's content; the four metrics here measure the delivery itself, and only two of them connect at all, and only through the defect rate. #### Summary - The four metrics are read from a single run and are linked: in none of the seven runs does a decision stand alone. - Shrinking batch size from 8 to 2 takes deployment frequency 3.2 → 13.5, lead time 34.6 → 14.9, recovery 45.4 → 26.3 steps, and the release failure rate 80% → 31%; errors surfacing in production stay at 12 and the change-based rate stays fixed at 20%. - Shrinking the batch to 1 pushes deployment frequency to 20.0, but the fixed release cost saturates the flow: lead time 69.7, recovery 65.8 steps. - The gate is the only decision that brings production errors from 12 to 4 (change failure rate 20% → 7%), and in exchange pushes lead time to 36.8 and time to restore to 54.8 steps — the fix passes through the same gate. - Automatic rollback brings recovery to 7.0 steps; at batch 8 it rolls back 48 innocent changes and carries their lead time to 64.5 steps, at batch 2 that same count falls to 10 and 35.8 steps. - The sum of the three decisions (batch 2, gate, automatic rollback) brings the release failure rate to 11% while pushing lead time to 109.3 steps. #### Next Step Every one of these decisions was made against a threshold, but the threshold itself was never written down: how much error is acceptable, and what stops when it becomes unacceptable. The next lesson ties that threshold to a number — given a target availability, the **error budget** minutes left in a period, what each incident takes from the budget, and the constraint placed on deployment frequency once the budget is exhausted are all measured; the budget rule's effect, measured across these four metrics, is counted. ### Site Reliability Engineering Source: https://academia.sh/en/courses/devops-fundamentals/site-reliability-engineering Tying the error budget to a number: in a 30-day period, a 99.9% target leaves 43.2 minutes of budget, and a single heavy incident takes 208% of it; without the budget rule, 12 periods measure 630 minutes of outage and 99.878% availability, while turning the rule on exhausts the budget in 8 of 12 periods and freezes for 166 days, yet cuts outage only to 620 minutes while lead time climbs from 3.0 to 12.6 days; pulling the target to 99.95% produces 236 frozen days and 60 unshipped changes, carrying lead time to 49.9 days; halving outage duration exhausts the rule in only 4 periods, availability rises to 99.939%, and lead time stays at 5.4 days. The previous lesson counted the bill of three decisions measured across four metrics, but every decision was made against an implicit threshold: how much error is acceptable, and what stops when it becomes unacceptable. That threshold was never written down anywhere. **Site reliability engineering** is the discipline that treats operational responsibility as an engineering problem and ties the reliability target to a number first. Where it overlaps with DevOps is shared responsibility — both approaches want development and operations held accountable for the same outcome. Where they diverge is the metric: the target is written down first, then an **error budget** is derived from that target, and delivery decisions are tied to that budget. This lesson ties the budget to a minute count and counts the rule's effect measured across four metrics. Recovery mechanics were measured in the Resilience and Reliability course; here, time to restore is taken as an **input**. #### Building the Budget The example is again the fictional **regional measurement network**; the network and the periods are both fictional. **CF26:** a period is 30 days, that is 43200 minutes, and twelve periods are run. **CF27:** twenty changes enter the flow at equal intervals in every period; the batch is four and prep takes one day. **CF28:** two unconditional dice are rolled for every change — whether it is defective (rate 0.10) and its incident class (light 10 minutes, share 0.60; moderate 30 minutes, share 0.30; heavy 90 minutes, share 0.10); the seed is 20260805. **CF29:** if a defective change ships, it becomes an incident in production, and the incident's outage minutes are deducted from the period's budget. The budget is derived at the start of the period from the target availability: period minutes times one minus the target. **CF30:** if the budget rule is on, the moment the remaining budget drops below zero, no new release ships until the period ends; queued changes wait, and the budget only renews in the new period. **CF31:** lead time is the days from a change entering the flow to being released, time to restore is the incident's outage minutes, availability is one minus the ratio of total outage to the total period. The periods, the incidents, and the budget rule are a **process-internal model**; there is no real outage record. ```js // budget/period.mjs — the period model of the error budget (model): there is no real outage record, // days and minutes are a data structure. Randomness is written with a generator, the seed is visible. export const PERIOD_DAYS = 30, PERIODS = 12, DAY_MIN = 1440, CHANGES = 20, BATCH = 4, PREP = 1; export const DEFECT = 0.10, SEED = 20260805; // incident class: outage minutes and its share export const CLASS = [["light", 10, 0.60], ["moderate", 30, 0.30], ["heavy", 90, 0.10]]; export function rng(seed) { // linear congruential generator let x = seed >>> 0; return () => ((x = (x * 1664525 + 1013904223) >>> 0) / 2 ** 32); } export const budgetMin = (target) => PERIOD_DAYS * DAY_MIN * (1 - target); export function run({ target = 0.999, rule = true, outageMultiplier = 1 } = {}) { const roll = rng(SEED), budget = budgetMin(target); const D = Array.from({ length: PERIODS * CHANGES }, (_, i) => { const p = Math.floor(i / CHANGES), k = i % CHANGES; const defective = roll() < DEFECT, r = roll(); let share = 0, min = 0; for (const [, m, sh] of CLASS) { share += sh; if (min === 0 && r < share) min = m; } return { i, arrival: p * PERIOD_DAYS + Math.floor(k * PERIOD_DAYS / CHANGES), defective, outage: min * outageMultiplier, status: "in-transit", releaseDay: null }; }); const S = { releases: 0, brokenReleases: 0, outage: 0, incidents: 0, frozenDays: 0, exhausted: 0, remainders: [], outages: [], periodReleases: Array(PERIODS).fill(0) }; let active = null, remaining = budget, period = -1, frozen = false; for (let t = 0; t < PERIODS * PERIOD_DAYS; t++) { const p = Math.floor(t / PERIOD_DAYS); if (p !== period) { // the budget renews at the start of the period if (period >= 0) S.remainders.push(remaining); period = p; remaining = budget; frozen = false; } for (const x of D) if (x.status === "in-transit" && x.arrival <= t) x.status = "queued"; if (active && active.finishAt === t) { // a release reaches production S.releases++; S.periodReleases[p]++; const broken = active.group.filter((x) => x.defective); if (broken.length) { S.brokenReleases++; for (const x of broken) { S.incidents++; S.outage += x.outage; S.outages.push(x.outage); remaining -= x.outage; } if (rule && remaining < 0 && !frozen) { frozen = true; S.exhausted++; } } for (const x of active.group) { x.status = "done"; x.releaseDay = t; } active = null; } if (frozen) { S.frozenDays++; continue; } // budget exhausted: no new release ships if (active) continue; const K = D.filter((x) => x.status === "queued"); if (K.length < BATCH) continue; const group = K.slice(0, BATCH); for (const x of group) x.status = "releasing"; active = { group, finishAt: t + PREP }; } S.remainders.push(remaining); return { D, S, budget }; } export const avg = (a) => (a.length ? (a.reduce((s, v) => s + v, 0) / a.length).toFixed(1) : "-"); export function measure(config) { const { D, S, budget } = run(config); const shipped = D.filter((x) => x.releaseDay !== null); const totalMin = PERIODS * PERIOD_DAYS * DAY_MIN; return { D, S, budget, deployFrequency: (S.releases / PERIODS).toFixed(1), leadTime: avg(shipped.map((x) => x.releaseDay - x.arrival)), changeFailureRate: (100 * S.brokenReleases / S.releases).toFixed(0) + "%", recovery: avg(S.outages), availability: (100 * (1 - S.outage / totalMin)).toFixed(3) + "%", outage: S.outage, incidents: S.incidents, releases: S.releases, exhausted: S.exhausted, frozenDays: S.frozenDays, avgRemaining: avg(S.remainders), unshipped: D.length - shipped.length }; } ``` ```js // budget/measure.mjs — tying the error budget to a number; the budget rule's effect across four metrics. import { PERIODS, PERIOD_DAYS, DAY_MIN, CLASS, SEED, DEFECT, budgetMin, measure } from "./period.mjs"; const print = (g, ...s) => console.log(s.map((v, i) => (g[i] < 0 ? String(v).padEnd(-g[i]) : String(v).padStart(g[i]))).join("")); console.log(`${PERIODS} periods x ${PERIOD_DAYS} days = ${PERIODS * PERIOD_DAYS * DAY_MIN} minutes; ` + `defect rate ${DEFECT}, seed ${SEED}`); const H = [0.995, 0.999, 0.9995]; const A = [-14, 16, 16, 18, 15]; console.log("\n1. how many minutes of budget a target availability leaves in a period"); print(A, "target", "budget (minutes)", "light incident", "moderate incident", "heavy incident"); for (const h of H) { const b = budgetMin(h); print(A, (100 * h).toFixed(2) + "%", b.toFixed(1), ...CLASS.map(([, dk]) => `${(100 * dk / b).toFixed(0)}%`)); } console.log(" (last three columns: the share a single incident takes from the budget)"); console.log(" incident classes: " + CLASS.map(([ad, dk, p]) => `${ad} ${dk} min (share ${p})`).join(", ")); const K = { "99.9% no rule": { target: 0.999, rule: false }, "99.9% rule on": { target: 0.999, rule: true }, "99.5% rule on": { target: 0.995, rule: true }, "99.95% rule on": { target: 0.9995, rule: true }, "99.9% outage halved": { target: 0.999, rule: true, outageMultiplier: 0.5 }, }; const R = Object.fromEntries(Object.entries(K).map(([ad, a]) => [ad, measure(a)])); const B = [-22, 15, 12, 17, 12]; console.log("\n2. the budget rule's effect measured across four delivery metrics"); print(B, "run", "deploy", "lead time", "change failure", "recovery"); print(B, "", "(period/release)", "(days)", "(release)", "(minutes)"); for (const [ad, r] of Object.entries(R)) print(B, ad, r.deployFrequency, r.leadTime, r.changeFailureRate, r.recovery); const C = [-22, 10, 11, 17, 20, 14]; console.log("\n3. the state of the budget in the same runs"); print(C, "run", "outage", "incidents", "availability", "periods exhausted", "frozen days"); for (const [ad, r] of Object.entries(R)) print(C, ad, r.outage, r.incidents, r.availability, `${r.exhausted}/${PERIODS}`, r.frozenDays); const E = [-22, 8, 24, 20, 13]; console.log("\n4. releases and the queue"); print(E, "run", "releases", "avg. remaining budget", "unshipped changes", "target held"); for (const [ad, r] of Object.entries(R)) { const h = K[ad].target; print(E, ad, r.releases, r.avgRemaining, r.unshipped, +r.availability.replace("%", "") >= 100 * h ? "yes" : "no"); } ``` ``` 12 periods x 30 days = 518400 minutes; defect rate 0.1, seed 20260805 1. how many minutes of budget a target availability leaves in a period target budget (minutes) light incident moderate incident heavy incident 99.50% 216.0 5% 14% 42% 99.90% 43.2 23% 69% 208% 99.95% 21.6 46% 139% 417% (last three columns: the share a single incident takes from the budget) incident classes: light 10 min (share 0.6), moderate 30 min (share 0.3), heavy 90 min (share 0.1) 2. the budget rule's effect measured across four delivery metrics run deploy lead time change failure recovery (period/release) (days) (release) (minutes) 99.9% no rule 5.0 3.0 35% 23.3 99.9% rule on 4.8 12.6 34% 23.8 99.5% rule on 5.0 3.0 35% 23.3 99.95% rule on 3.8 49.9 29% 27.5 99.9% outage halved 5.0 5.4 35% 11.7 3. the state of the budget in the same runs run outage incidents availability periods exhausted frozen days 99.9% no rule 630 27 99.878% 0/12 0 99.9% rule on 620 26 99.880% 8/12 166 99.5% rule on 630 27 99.878% 0/12 0 99.95% rule on 440 16 99.915% 9/12 236 99.9% outage halved 315 27 99.939% 4/12 65 4. releases and the queue run releases avg. remaining budget unshipped changes target held 99.9% no rule 60 -9.3 0 no 99.9% rule on 58 -8.5 8 no 99.5% rule on 60 163.5 0 yes 99.95% rule on 45 -15.1 60 no 99.9% outage halved 60 17.0 0 yes ``` The numbers are of the **measurement** kind; their inputs are the assumptions above. #### How Much of the Budget One Incident Takes The first table converts the target directly into minutes. In a thirty-day period, a 99.5% target leaves 216.0 minutes, a 99.9% target leaves 43.2 minutes, a 99.95% target leaves 21.6 minutes. The three columns on the right are the share a single incident takes from this budget, and the decision starts here: at the 99.5% target a heavy incident takes 42% of the budget, at 99.9% **208%**, at 99.95% 417%. **A target reads by how many heavy incidents it can withstand in a period:** the 99.5% target withstands two, the 99.9% target does not even withstand one. The same relationship holds in reverse too. The measured average outage is 23.3 minutes; the 99.9% budget is 43.2 minutes. So the budget **runs out after two incidents**, and an average of 2.3 incidents occur per period. The error budget is a ceiling set on the period total of time to restore. #### What the Rule Stops The second and third tables compare the same sequence of incidents with the rule off and on. With the rule off, twelve periods measure 27 incidents, 630 minutes of outage, and 99.878% availability — below the target. Once the rule is on, the budget runs out in eight of the twelve periods and **166 days** of freezing are applied. In exchange: outage falls from 630 to 620 minutes, incidents from 27 to 26, and availability rises from 99.878% to **99.880%**. The target still does not hold. Why so little? The fourth table tells us: the number of releases falls from 60 to only 58. Freezing does not eliminate a change, it **postpones** it; once the frozen period ends, the queue empties and the same defects ship in the same order. The cost is lead time: it climbs from 3.0 days to **12.6 days**, deployment frequency falls from 5.0 to 4.8, and eight changes never ship at all within the twelve periods. **The budget rule is not a prevention tool, it is a stopping tool; it runs after the incident, not before it.** The timing confirms this too. The budget runs out after an average of two incidents; by then, most of the period's five releases have already shipped. The releases the freeze blocks belong to the last days of the period, and they ship in the first days of the next period instead. Across the twelve periods combined, the number of blocked releases is two — against 166 days of freezing. The change failure rate falling from 35% to 34% has the same cause: the rule does not touch a release's **content**, only its **timing**. The same holds for time to restore; its rise from 23.3 to 23.8 minutes is nothing more than a small shift in the class distribution of the delayed releases. #### How Tight the Target Is and Where the Budget Comes From At the 99.5% target the rule never fires: the budget never runs out in any of the twelve periods, the average remaining at period end is 163.5 minutes, and every measured number looks as if the rule does not exist. **A rule tied to a loose target cannot be measured;** writing it down does not change behavior. At the 99.95% target the opposite happens. The budget runs out in nine of the twelve periods, 236 days freeze, releases fall from 60 to 45, and **sixty changes never ship at all**. Lead time jumps from 3.0 to 49.9 days. Outage falls from 630 to 440 minutes and availability comes to 99.915% — still below the target. A tight target stops the flow and still does not hit the target. The fifth run shows where the budget really comes from. Leaving the target at 99.9% and halving outage duration, the budget runs out in only four periods, freezing falls to 65 days, releases stay at 60, and lead time stays at 5.4 days; availability rises to **99.939%** and the target holds. Time to restore has fallen from 23.3 to 11.7 minutes. **It is not the freeze that determines the budget, it is time to restore.** #### Where the Difference Hides In this lesson, the difference hides in **the relationship between the target and the outage distribution**, and its number is this: the same rule produces three different budget draws across three targets — never exhausted at 216.0 minutes with 27 incidents, exhausted in eight periods at 43.2 minutes with 26, exhausted in nine periods at 21.6 minutes with 16. The declared part is the target itself; the undeclared part is the distribution of incident classes. The heavy incident's share is only 0.10, but it carries most of the total 630 minutes, and it is what exhausts the budget. How many steps it takes the signal to reach whom also changes with this rule. Without the budget rule, the remaining budget is never read at all; with the rule, every incident is deducted from the budget the same day, and the moment of exhaustion stops the release decision on **that same day**. The effect measured across the four metrics also reads together: once the rule is on, deployment frequency goes from 5.0 to 4.8, lead time from 3.0 to 12.6 days; the change failure rate goes from 35% to 34%, time to restore from 23.3 to 23.8 minutes — that is, the last two barely move at all. #### Summary - The error budget is derived from the target: in a 30-day period, a 99.5% target leaves 216.0, a 99.9% target 43.2, a 99.95% target 21.6 minutes; a single heavy incident (90 minutes) takes 42%, 208%, and 417% of these respectively. - With the rule off, 12 periods measure 27 incidents, 630 minutes of outage, and 99.878% availability. - Once the rule is on, the budget runs out in 8 periods and 166 days freeze, but since releases fall from 60 to only 58, outage falls only to 620 minutes; the cost is lead time (3.0 → 12.6 days), and 8 changes never ship at all. - If the target is loosened to 99.5%, the rule never fires (0/12 exhausted, average remaining 163.5 minutes); if it is tightened to 99.95%, 9/12 periods exhaust, 236 days freeze, 60 changes never ship, and lead time climbs to 49.9 days — the target still does not hold. - Once outage duration is halved, the budget runs out in only 4 periods, lead time stays at 5.4 days, and availability hits the target at 99.939%: it is time to restore that determines the budget. #### Next Step The budget rule saw the incident only after it happened, and stopped things only after that; that is why the measured effect was small. How late the signal arrives has come up somewhere in every lesson, but it was never counted directly. The next lesson takes on the loop itself: how many separate feedback loops there are, how long each one is, which error class it covers, and what moving an error class to an earlier loop costs and what it misses. ### Feedback Loops Source: https://academia.sh/en/courses/devops-fundamentals/feedback-loops The length and error-class coverage of four feedback loops: at cumulative lengths of 2, 30, 240, and 4320 minutes, 42 of 120 errors are caught while writing, 21 at integration, 25 at staging, and 32 fall into none of them and escape to production; the average delay per class ranges from 2.0 to 3810.0 minutes; moving the data-format class to the integration loop adds 12 minutes and brings total delay from 144954 down to 128520 minutes, but catches only 5 of the class's 12 errors early; moving the scheduling class costs 20 minutes for only 1 of 10 errors gained; and all three moves together bring the change failure rate from 53% down to 42% while pushing lead time from 351.9 up to 420.2 minutes. The previous lesson counted how the budget rule saw the incident only after it happened, and why the measured effect stayed small because of that. How late the signal arrives has come up somewhere in every lesson of this topic: 7.0 steps when observation scope was split, 61.0 steps for context dropped at the boundary, six steps back for a drift found at production verification. But the source of the delay — **the loop itself** — was never counted directly. A **feedback loop** is the path from the moment an error is born to the moment it reaches the role that can fix it. A flow does not have a single loop; it has several, each with its own length and its own **coverage**. Detection delay was measured in the Process, Team and Delivery course and the Testing Process and Automation Infrastructure course; what is measured here is not that delay but the loop itself: how many there are, how long each one runs, which error class it covers, and what it costs to move a class to an earlier loop. #### Building the Loops The example is again the fictional **regional measurement network**; the network and the loops are both fictional. **CF32:** there are four loops — writing, integration, staging, production; each has a length, and the lengths are cumulative, because the signal only returns to the development role at that link in the chain. **CF33:** there are eight error classes and 120 errors are born in the period; the counts per class are fixed. **CF34:** every loop has the classes it covers and a catch rate for each class; the production loop catches every class, because that is where the person using the error finds it. **CF35:** randomness is written with a generator, the seed is 20260806; one die per loop is rolled in advance for every error, so an error's luck does not change across move runs. **CF36:** 120 errors are distributed to 60 changes in order; a change's lead time is the staging cumulative plus the loop cumulative of every one of its errors caught before production. **CF37:** moving an error class to an earlier loop adds a check to that loop: the loop's length grows, the cumulative of the later loops shifts too, and the class is caught at that loop with a lower rate. The loops, the classes, and the move are a **process-internal model**; there is no real pipeline or error record. ```js // loop/loop.mjs — the model of the feedback loops (model): there is no real pipeline or error // record, minutes are a data structure. Randomness is written with a generator, the seed is visible. export const N_CHANGES = 60, SEED = 20260806; export const LOOP = [["writing", 2], ["integration", 28], ["staging", 210], ["production", 4080]]; // name and its increase over the previous loop // error class: how many are born in the period export const CLASS = { syntax: 24, type: 18, "unit-logic": 22, integration: 14, "data-format": 12, configuration: 12, scheduling: 10, resource: 8 }; // coverage: loop -> class -> catch rate. The production loop catches every class. export const COVERAGE = { writing: { syntax: 1.00, type: 0.90, "unit-logic": 0.20 }, integration: { type: 1.00, "unit-logic": 0.85, integration: 0.35 }, staging: { "unit-logic": 0.60, integration: 0.85, "data-format": 0.55, configuration: 0.35, scheduling: 0.20, resource: 0.15 }, production: Object.fromEntries(Object.keys(CLASS).map((s) => [s, 1.00])), }; export function rng(seed) { // linear congruential generator let x = seed >>> 0; return () => ((x = (x * 1664525 + 1013904223) >>> 0) / 2 ** 32); } // move: shifts an error class to an earlier loop; that loop's length grows by the extra. export function run(moves = []) { const increase = LOOP.map(([, a]) => a); const coverage = Object.fromEntries(LOOP.map(([ad]) => [ad, { ...COVERAGE[ad] }])); for (const { cls, loop, rate, extra } of moves) { coverage[loop][cls] = rate; increase[LOOP.findIndex(([ad]) => ad === loop)] += extra; } const cumulative = increase.map((_, i) => increase.slice(0, i + 1).reduce((s, v) => s + v, 0)); const roll = rng(SEED); const ERR = Object.entries(CLASS).flatMap(([cls, n]) => Array.from({ length: n }, () => ({ cls, rolls: LOOP.map(() => roll()) }))); ERR.forEach((h, i) => { h.change = i % N_CHANGES; }); // errors are assigned to changes in order for (const h of ERR) { // which loop catches the error h.loop = null; for (let i = 0; i < LOOP.length; i++) { const o = coverage[LOOP[i][0]][h.cls] ?? 0; if (h.rolls[i] < o) { h.loop = i; h.delay = cumulative[i]; break; } } } const CHANGE = Array.from({ length: N_CHANGES }, (_, i) => { const errors = ERR.filter((h) => h.change === i); const early = errors.filter((h) => h.loop < LOOP.length - 1); return { i, errors, escaped: errors.filter((h) => h.loop === LOOP.length - 1).length, lead: cumulative[LOOP.length - 2] + early.reduce((s, h) => s + cumulative[h.loop], 0) }; }); return { ERR, CHANGE, cumulative, increase, coverage }; } export const avg = (a) => (a.length ? (a.reduce((s, v) => s + v, 0) / a.length).toFixed(1) : "-"); export function measure(moves = []) { const { ERR, CHANGE, cumulative, increase, coverage } = run(moves); const escaped = ERR.filter((h) => h.loop === LOOP.length - 1); const leads = CHANGE.map((d) => d.lead); return { ERR, CHANGE, cumulative, increase, coverage, totalDelay: ERR.reduce((s, h) => s + h.delay, 0), avgDelay: avg(ERR.map((h) => h.delay)), escaped: escaped.length, leadTime: avg(leads), deployFrequency: (1440 / (leads.reduce((s, v) => s + v, 0) / CHANGE.length)).toFixed(2), changeFailureRate: (100 * CHANGE.filter((d) => d.escaped > 0).length / N_CHANGES).toFixed(0) + "%", recovery: cumulative[LOOP.length - 1] }; } ``` ```js // loop/measure.mjs — how many loops there are, how long they run, which error class they cover; the cost of moving one. import { LOOP, CLASS, COVERAGE, SEED, measure, avg } from "./loop.mjs"; const print = (g, ...s) => console.log(s.map((v, i) => (g[i] < 0 ? String(v).padEnd(-g[i]) : String(v).padStart(g[i]))).join("")); const T = measure(); const total = Object.values(CLASS).reduce((s, v) => s + v, 0); console.log(`${LOOP.length} feedback loops, ${Object.keys(CLASS).length} error classes, ` + `${total} errors, seed ${SEED}`); const A = [-15, 10, 12, 9, 11, 12]; console.log("\n1. the loops: length and coverage"); print(A, "loop", "length", "cumulative", "classes", "caught", "caught %"); LOOP.forEach(([ad], i) => { const g = T.ERR.filter((h) => h.loop === i); print(A, ad, T.increase[i], T.cumulative[i], Object.keys(COVERAGE[ad]).length, g.length, (100 * g.length / total).toFixed(0) + "%"); }); const B = [-14, 9, 18, 14, 23]; console.log("\n2. which loop an error class falls into"); print(B, "class", "errors", "first covered by", "avg. delay", "escaped to production"); for (const [s, n] of Object.entries(CLASS)) { const g = T.ERR.filter((h) => h.cls === s); const first = LOOP.findIndex(([ad]) => (COVERAGE[ad][s] ?? 0) > 0); print(B, s, n, LOOP[first][0], avg(g.map((h) => h.delay)), g.filter((h) => h.loop === LOOP.length - 1).length); } const K = { "baseline": [], "data-format to integration": [{ cls: "data-format", loop: "integration", rate: 0.45, extra: 12 }], "configuration to integration": [{ cls: "configuration", loop: "integration", rate: 0.30, extra: 8 }], "scheduling to integration": [{ cls: "scheduling", loop: "integration", rate: 0.10, extra: 20 }], "all three at once": [{ cls: "data-format", loop: "integration", rate: 0.45, extra: 12 }, { cls: "configuration", loop: "integration", rate: 0.30, extra: 8 }, { cls: "scheduling", loop: "integration", rate: 0.10, extra: 20 }], }; const R = Object.fromEntries(Object.entries(K).map(([ad, t]) => [ad, measure(t)])); const C = [-30, 13, 16, 14, 23]; console.log("\n3. the cost of moving an error class to an earlier loop"); print(C, "run", "integration", "total delay", "avg. delay", "escaped to production"); for (const [ad, r] of Object.entries(R)) print(C, ad, r.increase[1], r.totalDelay, r.avgDelay, r.escaped); const E = [-27, 17, 18, 17]; console.log("\n4. what happened to the moved class itself (baseline values in parentheses)"); print(E, "class (added time)", "caught early", "avg. delay", "still escapes"); for (const [ad, t] of Object.entries(K).slice(1, 4)) { const s = t[0].cls, r = R[ad], g = r.ERR.filter((h) => h.cls === s); const b = T.ERR.filter((h) => h.cls === s), last = LOOP.length - 1; print(E, `${s} (+${t[0].extra} min)`, `${g.filter((h) => h.loop === 1).length}/${g.length}`, `${avg(g.map((h) => h.delay))} (${avg(b.map((h) => h.delay))})`, `${g.filter((h) => h.loop === last).length} (${b.filter((h) => h.loop === last).length})`); } const F = [-30, 12, 12, 16, 12]; console.log("\n5. four delivery metrics"); print(F, "run", "deploy", "lead time", "change failure", "recovery"); print(F, "", "(chg./day)", "(minutes)", "(change)", "(minutes)"); for (const [ad, r] of Object.entries(R)) print(F, ad, r.deployFrequency, r.leadTime, r.changeFailureRate, r.recovery); ``` ``` 4 feedback loops, 8 error classes, 120 errors, seed 20260806 1. the loops: length and coverage loop length cumulative classes caught caught % writing 2 2 3 42 35% integration 28 30 3 21 18% staging 210 240 6 25 21% production 4080 4320 8 32 27% 2. which loop an error class falls into class errors first covered by avg. delay escaped to production syntax 24 writing 2.0 0 type 18 writing 5.1 0 unit-logic 22 writing 241.5 1 integration 14 integration 792.9 2 data-format 12 staging 2280.0 6 configuration 12 staging 3300.0 9 scheduling 10 staging 3096.0 7 resource 8 staging 3810.0 7 3. the cost of moving an error class to an earlier loop run integration total delay avg. delay escaped to production baseline 28 144954 1208.0 32 data-format to integration 40 128520 1071.0 28 configuration to integration 36 136578 1138.2 30 scheduling to integration 48 142224 1185.2 31 all three at once 68 117414 978.5 25 4. what happened to the moved class itself (baseline values in parentheses) class (added time) caught early avg. delay still escapes data-format (+12 min) 5/12 844.5 (2280.0) 2 (6) configuration (+8 min) 4/12 2558.0 (3300.0) 7 (9) scheduling (+20 min) 1/10 2687.0 (3096.0) 6 (7) 5. four delivery metrics run deploy lead time change failure recovery (chg./day) (minutes) (change) (minutes) baseline 4.09 351.9 53% 4320 data-format to integration 3.87 372.4 47% 4332 configuration to integration 4.00 360.3 50% 4328 scheduling to integration 3.71 388.1 52% 4340 all three at once 3.43 420.2 42% 4360 ``` The numbers are of the **measurement** kind; their inputs are the assumptions above. #### Four Loops, Four Lengths The first table lays the loops side by side. The lengths are 2, 28, 210, and 4080 minutes; cumulatively 2, 30, 240, and 4320. The ratio between the two ends is 2160. The distribution of catches, however, is not even: of 120 errors, 42 are caught while writing (35%), 21 at integration (18%), 25 at staging (21%), and **32, that is a quarter of them, fall into none of the first three loops.** The classes column explains why. The writing loop covers three classes, integration covers three, staging covers six; the production loop covers all eight, because there is no filtering rate there. If an error class is not in any early loop's coverage, the loops' lengths stop mattering at all. Total delay is 144954 minutes, and 138240 of that — thirty-two times 4320 — comes from errors that escape to production alone. **A quarter of the errors carry 95% of the delay.** #### Where Each Class Is Found The second table breaks the same number down by class, and the average delay stretches from 2.0 to 3810.0 minutes. Syntax and type errors finish in the first two loops (2.0 and 5.1 minutes) and none of them escape to production. Unit-logic is in the coverage of three separate loops, its average is 241.5 minutes, and only one of its 22 errors escapes. Of the 32 escaped errors, 29 come from the four classes first covered at **staging**: configuration 9, scheduling 7, resource 7, data format 6. These four classes' catch rates at staging are 0.35, 0.20, 0.15, and 0.55; in other words, staging is not a loop for these classes, it is a sieve. **The later a class is covered, the lower the catch rate where it is covered** — because if a class is covered late, it depends on something that does not show up in the earlier environment. #### The Cost of a Move and What It Misses The third table measures three moves. Moving the data-format class to the integration loop adds 12 minutes to that loop; total delay falls from 144954 to 128520 minutes (an 11% decrease), and errors escaping to production fall from 32 to 28. The fourth table shows the counterpart of this in the class itself: only **five** of the twelve errors are caught early, the class's average delay falls from 2280.0 to 844.5 minutes, and two errors still escape (six at baseline). The scheduling class is the limit of this decision. Moving it to the integration loop demands 20 minutes — the most expensive of the three moves — and gains only **one** of ten errors; the number that escapes falls from 7 to 6, and total delay drops by only 2730 minutes. The batch window does not exist in the integration environment, so the rate is 0.10. **The most expensive move is the one that gains the least, and the reason is written in the rate.** The fifth table gives the four metrics. All three moves carry the same sign: the change failure rate improves (from 53% to 47%, 50%, and 52%), and the other three get worse. When all three are applied together, the integration loop grows from 28 to 68 minutes; escaped errors fall from 32 to 25, the change failure rate falls to 42%, but deployment frequency falls from 4.09 to 3.43 changes/day, lead time climbs from 351.9 to 420.2 minutes, and time to restore goes from 4320 to 4360 minutes. **Lengthening the loop is a bill charged to every change that passes through it.** #### Where the Difference Hides In this lesson, the difference hides in **loop coverage**, and its number is this: four of the eight classes are not covered in either of the first two loops at all, and one is never covered above a 0.20 rate in any early loop. The declared part is the loop lengths — 2, 30, 240, 4320 minutes; anyone can read them. The undeclared part is the coverage: which loop catches which class at which rate is only visible in the run. How many steps it takes the signal to reach whom also reads from here: a syntax error reaches the development role in 2.0 minutes, a resource error in 3810.0 minutes. Same role, same flow, the same change within the same flow — a 1905-fold difference comes from the error class alone. #### Summary - The four loops' cumulative lengths are 2, 30, 240, and 4320 minutes; of 120 errors, 42 are caught while writing, 21 at integration, 25 at staging, 32 fall into none of them. - Of the 32 escaped errors, 29 come from the four classes first covered at staging (configuration 9, scheduling 7, resource 7, data format 6), and those classes' catch rates at staging range from 0.15 to 0.55. - The average delay per class spans from 2.0 to 3810.0 minutes; the difference does not come from loop length but from which loop covers the class. - Moving data format to integration gains 5 of 12 errors for 12 minutes and cuts total delay by 11%; moving scheduling gains only 1 of 10 errors for 20 minutes. - The three moves together bring escaped errors from 32 to 25 and the change failure rate from 53% to 42%, while carrying deployment frequency from 4.09 to 3.43 and lead time from 351.9 to 420.2 minutes. #### Next Step The loops are built, and how many steps it takes the signal to reach whom has been counted: 7.0 steps in observation scope, 61.0 steps for context dropped at the boundary, 2.0 to 3810.0 minutes in loop coverage. All six lessons measured the same thing — where the difference hides. But all of these measurements rested on a silent assumption: **that the same software is the same software in every environment.** An error not being caught at staging and surfacing in production instead was written, in the model above, as a *rate* — 0.35, 0.20, 0.15. Why that rate is not one was never asked. What runs on the developer's machine and what runs in production are not the same thing; there is a difference between them, and this topic never even named that difference. The next topic starts exactly there: what environments are, what difference is hidden at every link in the chain, and what it costs to skip a link. ## Fundamental Concepts Counting where the difference hides: the separation of development, test, staging, and production environments, the build once deploy everywhere principle and build artifact immutability, versioned binary and package storage, separating environment-varying values into configuration and secrets, portable application principles, and infrastructure models running from manual setup to declarative provisioning. ### Environments Source: https://academia.sh/en/courses/devops-fundamentals/environments Where the difference between development, test, staging, and production hides is counted: how many of ten dimensions diverge from production, how many are written in the environment manifest, which difference conceals which defect class, and how many defect classes skipping a link carries to production. The previous topic built feedback loops and counted how many steps an occurrence took to reach a role. Every one of those measurements carried a single silent assumption: the software running on a developer's machine and the software running in production at midnight are **the same software**. If the assumption held, a defect first seen in production would have to be seen on the development machine too. The difference is hiding somewhere, and it is not measured until it is named. This lesson takes up the first place the difference hides: the **environment** the software runs in. Here, an environment is not just where testing happens — it is a link in the chain leading to production, and the question this lesson asks is what each link can see. One fictional setup runs through the whole topic: a municipality's **regional measurement network**. The software collects readings from water meters, verifies the readings, converts them into invoices, and opens work orders for field crews. A nightly batch job processes the whole day's readings. It is fiction; the numbers below come from this fiction's model, not from a measured setup. **DC1.** The team that writes the network and the team that operates it are separate. The network runs three environments: development, test, and production. The fourth link, the staging environment, is not set up; what this lesson measures is exactly what that link would add. #### The Chain's Four Links An environment is a stop a change passes through on its way to production. The four stops are distinguished by purpose, not by name. The **development environment** is the writer's machine. A single process runs, every external dependency is faked, and the data is a few hundred hand-generated records. Its purpose is to see the shape of the change: does the code compile, does the function return the expected value. The **test environment** runs the change independently of its writer. Automated tests run here, the data has a realistic shape but is small, and some of the dependencies are real. Its purpose is to verify, independently of the writer's machine, whether the change is correct on its own. The **staging environment** is the link closest to production. External dependencies are real, resource limits are production's limits, versions are production's versions. Its purpose is not verification — it is seeing production-specific defect classes before they reach production. The **production environment** is where real meter readings are processed, real invoices are produced, and real work orders are opened for real field crews. The cost of a defect here is a wrong invoice. Which of these four stops can see what depends on how closely that stop resembles production. The measure of that resemblance is called **environment parity**; the name for it becoming unmeasurable is **configuration drift**: the links drift apart from one another over time, and where they drift apart goes unrecorded. #### What Parity Is Measured By For parity to be counted, its dimensions must be named. The fictional network's environment document declares seven dimensions: two package versions, one resource limit, one concurrency setting, one data volume, and whether two external dependencies are real or fake. **DC2.** The seven dimensions are written in the environment manifest, and both teams can read this document. **DC3.** Three more dimensions exist — timezone, the file system's case sensitivity, network latency — and none of these is written in any document; they can only be measured while the process is running. ```js // measurement-network/environments.mjs — the four environments' manifest and runtime probe (model, fictional data) // Seven dimensions written in the environment document. Values belong to the regional measurement network fiction. export const MANIFEST = { development: { resolverVersion: "3.2.1", rulePackageVersion: "9.1.0", memoryMB: 512, concurrentWorkers: 1, meterCount: 120, billingGateway: "fake", workOrderQueue: "fake" }, test: { resolverVersion: "3.2.1", rulePackageVersion: "9.1.0", memoryMB: 1024, concurrentWorkers: 2, meterCount: 5000, billingGateway: "fake", workOrderQueue: "real" }, staging: { resolverVersion: "3.4.0", rulePackageVersion: "9.1.0", memoryMB: 4096, concurrentWorkers: 8, meterCount: 180000, billingGateway: "real", workOrderQueue: "real" }, production: { resolverVersion: "3.4.0", rulePackageVersion: "9.1.0", memoryMB: 4096, concurrentWorkers: 8, meterCount: 1240000, billingGateway: "real", workOrderQueue: "real" }, }; // Three dimensions absent from the manifest, measurable only from inside the running process. export const PROBE = { development: { timezone: "+03:00", caseSensitiveFileSystem: false, networkLatencyMs: 0 }, test: { timezone: "+03:00", caseSensitiveFileSystem: true, networkLatencyMs: 2 }, staging: { timezone: "+00:00", caseSensitiveFileSystem: true, networkLatencyMs: 11 }, production: { timezone: "+00:00", caseSensitiveFileSystem: true, networkLatencyMs: 34 }, }; // The defect class each dimension conceals when it differs from production. export const CONCEALED = { resolverVersion: "resolution drift", rulePackageVersion: "amount error", memoryMB: "memory exhaustion", concurrentWorkers: "race condition", meterCount: "scale collapse", billingGateway: "contract drift", workOrderQueue: "contract drift", timezone: "day-boundary shift", caseSensitiveFileSystem: "path resolution error", networkLatencyMs: "timeout", }; export const DECLARED = Object.keys(MANIFEST.production); export const PROBED = Object.keys(PROBE.production); export const DIMENSIONS = [...DECLARED, ...PROBED]; export const full = (env) => ({ ...MANIFEST[env], ...PROBE[env] }); export const differences = (env) => DIMENSIONS.filter((d) => full(env)[d] !== full("production")[d]); ``` The measurement compares every environment against production and writes the difference dimension by dimension. ```js // measurement-network/parity.mjs — counts environment parity dimension by dimension import { DIMENSIONS, PROBED, CONCEALED, full, differences } from "./environments.mjs"; const RING = ["development", "test", "staging"]; const star = (d) => (PROBED.includes(d) ? d + " *" : d); console.log("parity against production (* = absent from manifest, visible only at runtime)"); console.log("dimension".padEnd(26) + RING.map((o) => o.padEnd(13)).join("") + "concealed defect class"); for (const d of DIMENSIONS) { const cell = RING.map((o) => (full(o)[d] === full("production")[d] ? "SAME" : "DIFF").padEnd(13)); console.log(star(d).padEnd(26) + cell.join("") + CONCEALED[d]); } const count = RING.map((o) => differences(o)); const total = count.flat().length; const atRuntime = count.flat().filter((d) => PROBED.includes(d)).length; console.log("\ndifference count: " + RING.map((o, i) => `${o} ${count[i].length}/${DIMENSIONS.length}`).join(", ")); console.log(`total ${total} differences; declared in manifest ${total - atRuntime}, visible only at runtime ${atRuntime}`); console.log("dimensions where staging diverges from production: " + differences("staging").join(", ")); console.log("dimensions with no environment at parity with production: " + DIMENSIONS.filter((d) => RING.every((o) => full(o)[d] !== full("production")[d])).join(", ")); ``` ``` parity against production (* = absent from manifest, visible only at runtime) dimension development test staging concealed defect class resolverVersion DIFF DIFF SAME resolution drift rulePackageVersion SAME SAME SAME amount error memoryMB DIFF DIFF SAME memory exhaustion concurrentWorkers DIFF DIFF SAME race condition meterCount DIFF DIFF DIFF scale collapse billingGateway DIFF DIFF SAME contract drift workOrderQueue DIFF SAME SAME contract drift timezone * DIFF DIFF SAME day-boundary shift caseSensitiveFileSystem * DIFF SAME SAME path resolution error networkLatencyMs * DIFF DIFF DIFF timeout difference count: development 9/10, test 7/10, staging 2/10 total 18 differences; declared in manifest 12, visible only at runtime 6 dimensions where staging diverges from production: meterCount, networkLatencyMs dimensions with no environment at parity with production: meterCount, networkLatencyMs ``` There are **18 differences** between the three environments and production. **12 of these are declared in the manifest**: they are visible to a reader, can be discussed, and can be closed. The remaining **6 are written in no document**; they can only be seen while the process is running in that environment. This distinction is the lesson's measurement axis: a declared difference is a matter for a decision, while a difference visible only at runtime is a surprise. None of these 18 differences was put there by a decision. The environments were set up at different times, upgraded at different moments, and maintained separately: when production's package version was upgraded, the development machine was left behind; when the test environment was connected to a real work order queue, its billing gateway was left fake. This is what configuration drift means — the difference accumulates wherever no one decided anything. And there is exactly one condition for drift to be measurable: the dimensions must be named and the manifest must be readable. The gradient is also readable: development diverges from production on nine of the ten dimensions, test on seven, staging on two. The number of differences drops as the links approach production — that is why the chain is ordered. Two dimensions — data volume and network latency — are also at parity with production in no environment. No link goes without carrying the gap between a meter count of one and a half million and one of a hundred twenty. #### The Defect Class the Difference Conceals The number of differences alone carries no decision; what the difference **conceals** does. The model below binds a defect to each dimension and computes which link in the chain will first see that defect. **DC4.** The model has one rule: a defect is visible if the environment carries the same value as production on the dimension it is bound to, and invisible otherwise. **DC5.** Each dimension carries exactly one defect class; ten dimensions means ten defects. **DC6.** The chain is walked in order, and a defect stops at the first link where it is seen. ```js // measurement-network/chain.mjs — counts which link in the chain first sees a defect import { CONCEALED, full } from "./environments.mjs"; // Model rule: a defect is visible if the environment carries the same value as production on its bound dimension. const DEFECTS = [ ["empty read field fails to parse", "resolverVersion"], ["new tariff bracket rounds incorrectly", "rulePackageVersion"], ["batch job loads every reading into memory", "memoryMB"], ["two invoice lines for the same meter", "concurrentWorkers"], ["verification query is quadratic in meter count", "meterCount"], ["billing gateway rejection code goes unhandled", "billingGateway"], ["same work order enters the queue twice", "workOrderQueue"], ["nightly job cuts the day boundary by local time", "timezone"], ["tariff file is looked up in uppercase", "caseSensitiveFileSystem"], ["aggregator response exceeds the timeout", "networkLatencyMs"], ]; const FOUR = ["development", "test", "staging", "production"]; const THREE = ["development", "test", "production"]; const firstToSee = (chain, dim) => chain.find((o) => full(o)[dim] === full("production")[dim]); console.log("defect".padEnd(49) + "four links".padEnd(13) + "three links"); for (const [name, dim] of DEFECTS) { console.log(name.padEnd(49) + firstToSee(FOUR, dim).padEnd(13) + firstToSee(THREE, dim)); } const tally = (chain) => { const d = Object.fromEntries(chain.map((o) => [o, 0])); for (const [, dim] of DEFECTS) d[firstToSee(chain, dim)] += 1; return chain.map((o) => `${o} ${d[o]}`).join(", "); }; console.log("\nfour-link chain: " + tally(FOUR)); console.log("three-link chain: " + tally(THREE)); const escaping = (chain) => DEFECTS.filter(([, d]) => firstToSee(chain, d) === "production"); const fourEscaping = escaping(FOUR).map(([, d]) => CONCEALED[d]); const threeEscaping = escaping(THREE).map(([, d]) => CONCEALED[d]); const added = threeEscaping.filter((s) => !fourEscaping.includes(s)); console.log(`escaping to production even with staging in place, ${fourEscaping.length} classes: ` + fourEscaping.join(", ")); console.log(`escaping when staging is skipped, ${threeEscaping.length} classes; added ${added.length}:`); for (let i = 0; i < added.length; i += 3) console.log(" " + added.slice(i, i + 3).join(", ")); console.log("\nchain length scanned (links added in order):"); let previous = 0; for (let n = 0; n <= 3; n++) { const chain = [...FOUR.slice(0, n), "production"]; const caught = DEFECTS.filter(([, d]) => firstToSee(chain, d) !== "production").length; console.log(`${n} links + production`.padEnd(22) + `caught before production ${caught}` + ` (last link's contribution ${caught - previous})`); previous = caught; } ``` ``` defect four links three links empty read field fails to parse staging production new tariff bracket rounds incorrectly development development batch job loads every reading into memory staging production two invoice lines for the same meter staging production verification query is quadratic in meter count production production billing gateway rejection code goes unhandled staging production same work order enters the queue twice test test nightly job cuts the day boundary by local time staging production tariff file is looked up in uppercase test test aggregator response exceeds the timeout production production four-link chain: development 1, test 2, staging 5, production 2 three-link chain: development 1, test 2, production 7 escaping to production even with staging in place, 2 classes: scale collapse, timeout escaping when staging is skipped, 7 classes; added 5: resolution drift, memory exhaustion, race condition contract drift, day-boundary shift chain length scanned (links added in order): 0 links + production caught before production 0 (last link's contribution 0) 1 links + production caught before production 1 (last link's contribution 1) 2 links + production caught before production 3 (last link's contribution 2) 3 links + production caught before production 8 (last link's contribution 5) ``` The table makes one rule visible: **an environment can only see the defects on dimensions where it is the same as production.** The development environment catches only one of ten defects, because it is the same as production on only a single dimension — the rule package version. Test catches two defects; both are bound to dimensions where test is the same as production: a real work order queue and a case-sensitive file system. Reading it the other way says more: the list of defect classes an environment **fails to catch** is the list of dimensions where that environment diverges from production. Staging diverged on two dimensions; the two defect classes that escape to production even with staging in place are bound to exactly those two dimensions. This is not an inference from the model — it is a consequence of its definition. But because it turns the definition into a number, it takes the environment debate out of opinion: the question "do we need a staging environment" is replaced by "which dimensions are we buying parity on." The last part of the output scans the chain length: as links are added in order, the number of classes caught before production is 0, 1, 3, and 8. The contribution per link is **not equal** — the first link adds one class, the second two, the third five. The ordering also explains why: each link closes the dimensions the previous one could not see, and the link closest to production closes the most dimensions. Adding a new environment to the front of the chain is investing where the contribution is smallest; the number of dimensions a link closes is read from its manifest, not from its name. #### The Cost of Skipping a Link The fictional network was not running a staging environment. The model's two runs count the cost of that decision. In the four-link chain, two defect classes escape to production: scale collapse and timeout. In the three-link chain, that climbs to seven. The staging link catches **five defect classes** ahead of production: resolution drift, memory exhaustion, race condition, contract drift, and day-boundary shift. As long as the network does not set up staging, these five are classes that first appear **while a real invoice is being produced**. What skipping gains can be counted too: the chain drops from four stops to three, which means one less wait per release and one less handoff. This gain is written to lead time; the cost is written to change failure rate and time to restore, because each of the five escaping classes produces a change to roll back and an outage to repair when it is seen in production. A sentence that looks at a single metric carries no decision here: the speed-up is taken from one place and set down in another. The two escaping classes stay in place even if staging is set up, because no link carries a data volume of one and a quarter million meters or a network latency of thirty-four milliseconds. There is a way to close these two dimensions — a production-scale-shaped data set and injected artificial latency — and each has a cost: the environment's setup cost and its run time. The measurement makes this trade-off concrete too: there are exactly two dimensions to close, both are named, and which defect class gets headed off once they are closed is written down. This, finally, is what the six differences between the manifest and the runtime mean. The twelve differences written in the manifest can come up in a meeting; the six that are not written cannot. Timezone diverging between development and production shows up as the nightly batch job cutting the day boundary in the wrong place, and it occurs to no one that the cause is an environment difference until the defect is found. Widening the environment manifest's scope — declaring all three of the runtime-measured dimensions too — makes all six of these differences discussable. #### Summary - An environment is a link in the chain leading to production; the four links are distinguished by purpose, and each link can only see the defects on dimensions where it is the same as production. - Environment parity is counted dimension by dimension. The fictional model has ten dimensions; 18 differences were measured between the three environments and production: development 9, test 7, staging 2. - 12 of these 18 differences are declared in the environment manifest, 6 are visible only while the process is running; an undeclared difference cannot be discussed and so cannot be a matter for a decision. - Every difference conceals a defect class, and the list of classes an environment fails to catch equals the list of dimensions where that environment diverges from production. - The staging link catches five defect classes ahead of production; skipping it carries those five to production. Data volume and network latency are at parity with production in no link, so two classes reach production regardless. #### Next Step This lesson's chain assumed one thing: the object that travels between links is always the same object. But the environment manifests do not confirm this — each link has its own package versions, its own file paths, its own environment variables. If the change is recompiled from source at every stop, then it is not a single object circulating through the chain but four separate objects, and which one is the same as which has not been measured. The next lesson names and measures this object: is output produced twice from the same source actually the same, and if not, which sources introduce the difference, and how many separate binaries does recompiling per environment produce? ### Build Artifacts and Immutability Source: https://academia.sh/en/courses/devops-fundamentals/build-artifacts-and-immutability The build once, deploy everywhere principle is measured: the digest of output produced twice from the same source is compared, the four sources that break reproducibility are closed one at a time, and the number of separate binaries produced by recompiling per environment is compared against a single-output layout. The previous lesson measured the four environments as links in the chain leading to production and counted how each link could only see the defects on dimensions where it matched production. That measurement assumed one thing: the object that travels between links is always the same object. If what is verified in test is not what runs in production, the assurance testing provides shrinks by exactly that much. This lesson names that object. A **build artifact** is the single object produced from source and deployed: an archive, a package, one executable file. The questions are measurable: is output produced twice from the same source actually the same, and if not, where does the difference come from, and how many separate objects does recompiling for each environment produce? **DC7.** The build step below is a model: three source files are read and merged into a single file. It is not what a real compiler does, but it carries the same fragilities with respect to reproducibility. **DC8.** Two machines are modeled with a manifest — `A` is the writing team's machine, `B` is the runner that executes the build — and the time is written in the manifest too, so the measurement can be repeated. `A1` and `A2` are two separate runs of the same machine. **DC9.** The digest is the first sixteen hexadecimal digits of the file's sha256 value; the truncation is only for readability. #### Build Once, Deploy Everywhere The immutability principle is one sentence: **a build artifact is produced once, is never changed after it is produced, and the same one is deployed to every environment.** Patching a file in place in production turns the object verified in test into an unknown object; if a fix is needed, a new artifact is produced and a new digest is born. Rollback works by the same rule: the previous artifact is redeployed, the current artifact is not edited backward. The violation is concrete too. Hand-fixing a single file in production creates a new object with no record anywhere in that environment: one object circulates through the chain while a second one sits in production, that second object's digest is written in no deployment record, and the environment manifest does not show it. The next deployment silently erases the fix; the reason the behavior disappeared is not found in any of the places anyone would look. This is the cost of patching in place: the difference is hidden somewhere entirely outside the measurement's reach. The principle working depends on exactly one condition: the artifact must have an **identity**. Identity is the content digest. Whether two artifacts are the same can only be said by comparing their digests; a file name, a production date, or a version tag cannot say it. That is why the measurement starts from the digest. ```js // measurement-network/build.mjs — writes the source to disk, produces a single-file build artifact (model) import { mkdirSync, writeFileSync, readFileSync, rmSync } from "node:fs"; import { createHash } from "node:crypto"; const SOURCE = "source"; export const writeSource = () => { rmSync(SOURCE, { recursive: true, force: true }); mkdirSync(SOURCE, { recursive: true }); writeFileSync(`${SOURCE}/reading-resolver.js`, "export const resolve = (s) => Number(s.slice(4));\n"); writeFileSync(`${SOURCE}/billing-rules.js`, "export const amount = (m3) => m3 * 7.4;\n"); writeFileSync(`${SOURCE}/work-order.js`, "export const openOrder = (no) => ({ no, type: 'onsite' });\n"); }; // Machine manifest: the time is written in the manifest too, so the measurement can be repeated. export const A1 = { name: "A1", time: "21:04:07", dir: "/home/writer/measurement-network", builtBy: "writer-team", readOrder: ["reading-resolver.js", "billing-rules.js", "work-order.js"] }; export const A2 = { ...A1, name: "A2", time: "21:19:52" }; export const B = { name: "B", time: "22:41:30", dir: "/opt/runner/job-1873", builtBy: "pipeline-runner", readOrder: ["work-order.js", "billing-rules.js", "reading-resolver.js"] }; export const FOUR_SOURCES = ["timestamp", "directoryOrder", "environmentVariable", "absolutePath"]; export const only = (name) => Object.fromEntries(FOUR_SOURCES.map((k) => [k, k === name])); export const build = (manifest, options, target, embedded = null) => { const files = options.directoryOrder ? manifest.readOrder : [...manifest.readOrder].sort(); const header = []; if (options.timestamp) header.push(`// build time: ${manifest.time}`); if (options.environmentVariable) header.push(`// built by: ${manifest.builtBy}`); if (options.absolutePath) header.push(`// source root: ${manifest.dir}/${SOURCE}`); if (embedded) header.push(`export const CONFIG = ${JSON.stringify(embedded)};`); const body = files.map((f) => readFileSync(`${SOURCE}/${f}`, "utf8")).join(""); mkdirSync("dist", { recursive: true }); writeFileSync(`dist/${target}`, header.join("\n") + (header.length ? "\n" : "") + body); return createHash("sha256").update(readFileSync(`dist/${target}`)).digest("hex").slice(0, 16); }; ``` #### The Four Sources That Break Reproducibility **DC10.** Four separate breaking sources are planted in the build step, and each can be closed independently: writing the build moment into the output (`timestamp`), merging source files in directory read order (`directoryOrder`), embedding the name of whoever ran the build (`environmentVariable`), and writing the source root's absolute path (`absolutePath`). All four are sources encountered in real build steps. The scan makes three runs — two moments on the same machine (`A1`, `A2`) and a second machine (`B`) — and counts how many distinct digests each option set produces. ```js // measurement-network/scan.mjs — scans the sources that break reproducibility, one at a time import { writeSource, build, only, FOUR_SOURCES, A1, A2, B } from "./build.mjs"; writeSource(); const all = Object.fromEntries(FOUR_SOURCES.map((k) => [k, true])); const none = Object.fromEntries(FOUR_SOURCES.map((k) => [k, false])); const digests = (options) => [A1, A2, B].map((k) => build(k, options, `package-${k.name}.js`)); const row = (label, options) => { const [a1, a2, b] = digests(options); const distinct = new Set([a1, a2, b]).size; console.log(label.padEnd(26) + (a1 === a2 ? "SAME " : "DIFF ").padEnd(12) + (a1 === b ? "SAME " : "DIFF ").padEnd(11) + distinct); }; console.log("active source".padEnd(26) + "A1-A2".padEnd(12) + "A1-B".padEnd(11) + "distinct digests"); row("all four", all); for (const k of FOUR_SOURCES) row("only " + k, only(k)); row("none", none); const [a1, a2, b] = digests(none); console.log(`\nwith all four sources closed, the digest of all three runs: ${a1}, ${a2}, ${b}`); console.log("same input, same output: " + (a1 === a2 && a2 === b)); ``` ``` active source A1-A2 A1-B distinct digests all four DIFF DIFF 3 only timestamp DIFF DIFF 3 only directoryOrder SAME DIFF 2 only environmentVariable SAME DIFF 2 only absolutePath SAME DIFF 2 none SAME SAME 1 with all four sources closed, the digest of all three runs: 0a690fcb84c2b166, 0a690fcb84c2b166, 0a690fcb84c2b166 same input, same output: true ``` The table separates two things. First: **each of the four sources alone is enough to break reproducibility.** Even with only one active, the digest of the output produced on the second machine differs; two objects produced from the same source cannot be said to be the same. Second, and more important: three of the four sources are **invisible across two runs on the same machine.** Directory read order, the environment variable, and the absolute path all stay fixed on a single machine, so they produce no difference between `A1` and `A2`. The writing team runs the build twice on its own machine, sees the same digest, and concludes the build is reproducible. The difference only surfaces the moment the output is produced on a second machine. The previous lesson's distinction holds here too: this is not a declared difference — it is a difference visible only at runtime, and only on a run on a different machine at that — and the defect class it conceals is named **unverified binary**: whether the object that passed testing is the object headed to production has not been measured. With all four closed, the three runs produce a single digest. This means the digest now names not the run but the **input**: same digest, same source. The way to close each source is known too, and all four live inside the build step — not writing the timestamp into the output, sorting files by name instead of directory read order, not embedding the identity of whoever ran the build, and keeping paths relative to the source root. What the digest says and what it does not say must also be kept separate. The digest states **byte identity**: if two outputs have the same digest, the two objects are identical, and one can be written in place of the other in a deployment record. There are two things it does not say. First, it does not say which source the output was produced from; the digest is computed from the output itself, not from the source's identity. This gap can close once the build step is reproducible: someone else running the build a second time from the same source gets the same digest, and the claim becomes testable — this is reproducibility's value beyond mere tidiness. Second, a digest difference does not say the behavior difference is **meaningful**; a single comment line added to the output changes the digest end to end. A different digest is not proof — it is a signal that needs investigating. #### Per-Environment Compilation Versus a Single Output Compiling separately for each link in the chain has a concrete counterpart: values that change by environment get written into the output at build time. **DC11.** In the fictional network, these values are four: the environment name, whether the billing gateway is real or fake, the log level, and the nightly batch job's threshold. **DC12.** In the single-output layout, the same four values are read from the environment at startup; how the read is done and which value counts as a secret is the next lesson's subject. ```js // measurement-network/per-environment.mjs — compares per-environment recompilation against a single output import { writeSource, build, FOUR_SOURCES, A1, A2, B } from "./build.mjs"; const CONFIG = { development: { envName: "development", billingGateway: "fake", log: "verbose", threshold: 500 }, test: { envName: "test", billingGateway: "fake", log: "info", threshold: 20000 }, production: { envName: "production", billingGateway: "real", log: "warn", threshold: 250000 }, }; const ENV = Object.keys(CONFIG); const none = Object.fromEntries(FOUR_SOURCES.map((k) => [k, false])); const all = Object.fromEntries(FOUR_SOURCES.map((k) => [k, true])); writeSource(); console.log("embedded value keys: " + Object.keys(CONFIG.production).join(", ")); const separate = ENV.map((o) => [o, build(A1, none, `package-${o}.js`, CONFIG[o])]); for (const [o, digest] of separate) { const diff = Object.keys(CONFIG[o]).filter((k) => CONFIG[o][k] !== CONFIG.production[k]); console.log(`${o.padEnd(11)} digest ${digest} values differing from the production pair ${diff.length}`); } console.log(`per-environment build: ${new Set(separate.map(([, h]) => h)).size} distinct binaries, ` + `embedded values across binaries ${ENV.length * Object.keys(CONFIG.production).length}`); const single = ENV.map((o) => build(A1, none, `package-single-${o}.js`)); console.log(`single output layout : ${new Set(single).size} distinct binary, digest ${single[0]}, embedded values 0`); console.log("\nif the per-environment build runs on different machines and at different times:"); const noise = [[A1, "test"], [B, "production"], [A2, "production"]] .map(([k, o]) => [`${o} @ ${k.name}`, build(k, all, `package-noise-${o}-${k.name}.js`, CONFIG[o])]); for (const [label, digest] of noise) console.log(" " + label.padEnd(17) + "digest " + digest); console.log(" same environment, two runs gave the same digest: " + (noise[1][1] === noise[2][1])); ``` ``` embedded value keys: envName, billingGateway, log, threshold development digest 9f8d94167519742a values differing from the production pair 4 test digest 41579ab330e96a22 values differing from the production pair 4 production digest 75cc8f6293d5a489 values differing from the production pair 0 per-environment build: 3 distinct binaries, embedded values across binaries 12 single output layout : 1 distinct binary, digest 0a690fcb84c2b166, embedded values 0 if the per-environment build runs on different machines and at different times: test @ A1 digest 78ffd2e872d44825 production @ B digest ca355c34a8a96b0c production @ A2 digest 68bb8cdccf606938 same environment, two runs gave the same digest: false ``` Per-environment compilation produces **three separate binaries**, and a total of **twelve embedded values** are carried inside those three binaries. The object test approves diverges from the object headed to production on four values: environment name, billing gateway, log level, and batch threshold. These four sit **inside** the output; someone looking from outside sees that all three files carry the same name, and only the digest tells the difference. The billing gateway being fake in the test binary and real in the production binary makes the previous lesson's contract-drift class invisible in every run of testing. In the single-output layout, the three environments **share a single binary**: the digest is the same across all three, and the embedded-value count is zero. The twelve values do not vanish — they change location, moving from inside the output to the environment manifest. In the previous lesson's terms: twelve undeclared, runtime-only differences turn into twelve declared differences. The amount of difference stays the same; where it is hidden changes — and that is what becomes measurable. The last three lines show the second cost of per-environment compilation. When builds run on different machines and at different moments, even two outputs produced for the same environment carry different digests. The digest no longer names the environment; it names the run. When two digests differ, whether the cause is an environment difference or build noise cannot be told apart, and the answer to "is what is running in production what test verified" becomes unmeasurable. Whether the output really sits on disk and whether the digest is that file's digest can also be verified directly. ```bash # verify.sh — is the output really on disk, does its digest match what node printed ls dist/package-single-*.js shasum -a 256 dist/package-single-production.js | cut -c1-16 cat dist/package-single-production.js ``` ``` dist/package-single-development.js dist/package-single-production.js dist/package-single-test.js 0a690fcb84c2b166 export const amount = (m3) => m3 * 7.4; export const resolve = (s) => Number(s.slice(4)); export const openOrder = (no) => ({ no, type: 'onsite' }); ``` The content of the files written for the three environments is identical, and the digest taken from the shell matches the `0a690fcb84c2b166` value `node` printed. Even though the file names differ, the object is one; this is the immutability principle in its measured form. It is also visible that the files are merged in name order — because directory read order is closed. #### How Many Objects Circulate Through the Chain The previous lesson's chain was a single object advancing through four stops. The measurement also counts what the two layouts give that chain. In per-environment compilation, three separate objects circulate through the chain and three separate build runs happen; each run is exposed all over again to the four breaking sources counted in the scan. In the single-output layout, the object is one, the run is one, and moving between links reduces to a copy operation. The difference shows up in how many questions can be asked. In the single-output layout, the question "is the object test verified the object headed to production" is answered by a single digest comparison: yes if the two values are equal, no otherwise. In per-environment compilation, the same question cannot even be asked, because the digests of two environments already differ by definition, and how much of the difference comes from environment settings versus build noise cannot be separated out. A measurable assurance turns into an unmeasurable habit. The distinction is also written into the delivery metrics. Getting a fix to production requires three build runs under per-environment compilation; all three have to succeed and reach the right environment, which adds three builds and three deployment preparations to lead time. On the change-failure-rate side, there is the unverified-binary class: because the object that passed testing does not go to production, some of the defect classes testing caught can be reborn in production. The single-output layout moves both metrics in the same direction — but its cost is that it forces the twelve environment-varying values to be kept somewhere outside the output. #### Summary - A build artifact is the single object produced from source and deployed; the immutability principle says this object is produced once, is never changed in place, and the same one goes to every environment. - Whether two artifacts are the same can only be said by their content digest; a file name, a date, and a version tag cannot say it. - Four sources that break reproducibility were measured — timestamp, directory read order, the environment variable, absolute path — and each alone was enough to produce a different digest on a second machine. With all four closed, three runs gave a single digest. - Three of these four sources are invisible across two runs on the same machine; the difference only surfaces once the build runs on a different machine. The defect class it conceals is the unverified binary. - Per-environment recompilation produced three separate binaries and twelve embedded values; in the single-output layout, the binary count dropped to one and the embedded-value count to zero. The values did not vanish — they moved from inside the output to the environment manifest. #### Next Step The single-output layout left one question unanswered: where does this one produced object sit, and how is it named? A digest is an identity, but it is not a name; hand-writing sixteen hexadecimal digits into deployment records is not a sustainable arrangement. And if the artifact is produced once and deployed everywhere, it has to sit somewhere between where it is produced and where it is deployed, and the previous version has to still be findable when a rollback is needed. The next lesson measures that storage place: how is a version tagged, in how many deployments does the difference between an immutable version and a movable tag show itself, and where does a retention rule close the rollback window? ### Build Artifact Repositories Source: https://academia.sh/en/courses/devops-fundamentals/build-artifact-repositories Versioned binary and package storage is measured: the distinction between an immutable version and a movable tag is built into a repository model, how many deployments a movable tag pointed to a different artifact is counted, and the retention policy is scanned through the rollback window, repository bytes, and the count of unreachable deployments. The previous lesson produced a single build artifact and measured that its identity is its content digest. A digest is an identity, but it is not a name: hand-carrying a string of hexadecimal digits into deployment records is not a sustainable arrangement. And if an artifact is produced once and deployed everywhere, it has to sit somewhere between the moment it is produced and the moment it is deployed, and the previous artifact has to still be findable when a rollback is needed. This lesson takes up that storage place: the **build artifact repository**. A repository is the component that stores versioned binaries and packages, names them, and returns them on request. It is treated here not as a product but as a type; what is measured is what the two naming forms a repository offers — the immutable version and the movable tag — produce in deployment. **DC13.** The repository model is limited to two functions: immutable version storage and name resolution. Access control, replication, and the network layer are outside the model. **DC14.** The release schedule is built with our own generator; the seed `20260311` is visible, and the schedule publishes 24 versions at an average of one every four days. **DC15.** Version sizes come from the generator in the 38–52 MB range; this is not a real measurement, it is the fiction's order of magnitude. #### What a Repository Stores A repository stores two separate things: **content** and **name**. The content is the object the previous lesson measured; the name is the word people write into a deployment record. The relationship between these two splits in two. An **immutable version** is a name that, once published, can never be bound to different content. If `1.5.0` has been published once, `1.5.0` is always the same output. A **movable tag** is a name that points at a version and can move to a different version over time: `stable`, `latest`, `release-candidate`, and so on. Both are names, but one corresponds to an identity, the other to a pointer. The model below contains both functions and the enforcement of the immutability rule. ```js // measurement-network/repository.mjs — versioned build artifact repository and release schedule (model) import { createHash } from "node:crypto"; // Our own generator: linear congruential. The seed belongs to the caller, output is repeatable. export const generator = (seed) => () => (seed = (seed * 1103515245 + 12345) % 2147483648) / 2147483648; export const digest = (text) => createHash("sha256").update(text).digest("hex").slice(0, 12); export const createRepository = () => { const versions = new Map(); // immutable: version -> { digest, bytes, day } const tags = new Map(); // movable: tag -> version const rejected = []; return { versions, tags, rejected, publish(version, dig, bytes, day) { const existing = versions.get(version); if (existing && existing.digest !== dig) { rejected.push(version); return "REJECTED"; } if (existing) return "SAME"; versions.set(version, { digest: dig, bytes, day }); return "PUBLISHED"; }, tag: (name, version) => tags.set(name, version), resolve(name) { const version = tags.get(name) ?? name; const record = versions.get(version); return record ? { version, digest: record.digest } : null; }, }; }; // n-version release schedule: day gaps and byte sizes come from the generator. export const schedule = (n, avgDays, seed) => { const r = generator(seed); const list = []; let day = 0; for (let i = 1; i <= n; i++) { day += 1 + Math.round(r() * (2 * avgDays - 2)); list.push({ version: `1.${i}.0`, digest: digest(`1.${i}.0-content`), bytes: 38_000_000 + Math.round(r() * 14_000_000), day, cleanRun: r() > 0.45, }); } return list; }; ``` The `publish` function has three responses, and these three are the whole of the repository's behavior: a new version is recorded, a republish with the same content is ignored, and the same name arriving with different content is rejected. The third is the immutability principle's repository-side counterpart — without the rejection, every digest comparison the previous lesson measured loses its meaning, because which object the name `1.5.0` points to becomes time-dependent. #### Immutable Version and Movable Tag The measurement publishes 24 versions on schedule and services the deployment requests that arrive in between. **DC16.** The `latest` tag moves on every publish; the `stable` tag moves only to versions where the nightly batch job ran clean. **DC17.** The deployment record stores only the **requested name**, not the resolved version — this is the source of the defect being measured. ```js // measurement-network/tag.mjs — counts how many separate artifacts a movable tag pointed to across deployments import { createRepository, schedule } from "./repository.mjs"; const VERSIONS = schedule(24, 4, 20260311); const repo = createRepository(); // Deployment requests: day and requested name. The name is either a movable tag or an immutable version. const REQUESTS = [ [6, "stable"], [11, "stable"], [17, "latest"], [23, "stable"], [29, "stable"], [34, "1.5.0"], [41, "stable"], [48, "latest"], [55, "stable"], [66, "1.13.0"], [70, "stable"], [79, "stable"], [86, "latest"], [92, "1.21.0"], ]; const records = []; let next = 0; for (const [day, name] of REQUESTS) { while (next < VERSIONS.length && VERSIONS[next].day <= day) { const s = VERSIONS[next++]; repo.publish(s.version, s.digest, s.bytes, s.day); repo.tag("latest", s.version); if (s.cleanRun) repo.tag("stable", s.version); } records.push({ day, name, ...repo.resolve(name) }); } console.log("day".padEnd(5) + "requested name".padEnd(17) + "resolved version".padEnd(19) + "digest"); for (const r of records) { console.log(String(r.day).padEnd(5) + r.name.padEnd(17) + r.version.padEnd(19) + r.digest); } const movable = records.filter((r) => r.name === "stable" || r.name === "latest"); const lastSeen = new Map(); let changes = 0; for (const r of movable) { if (lastSeen.has(r.name) && lastSeen.get(r.name) !== r.digest) changes += 1; lastSeen.set(r.name, r.digest); } const distinct = (name) => new Set(records.filter((r) => r.name === name).map((r) => r.digest)).size; console.log(`\ntotal deployments ${records.length}: by movable tag ${movable.length}, ` + `by immutable version ${records.length - movable.length}`); console.log(`"stable" was deployed ${records.filter((r) => r.name === "stable").length} times, ` + `pointed to ${distinct("stable")} distinct artifacts`); console.log(`"latest" was deployed ${records.filter((r) => r.name === "latest").length} times, ` + `pointed to ${distinct("latest")} distinct artifacts`); console.log(`same name, different artifact than the previous deployment: ${changes} deployments`); // Immutability check: attempting to republish a published version under a different digest. console.log("\nsame version, different digest: " + repo.publish("1.5.0", "ffffffffffff", 4e7, 99)); console.log("same version, same digest : " + repo.publish("1.5.0", repo.resolve("1.5.0").digest, 4e7, 99)); console.log("rejected republish attempts: " + repo.rejected.length); ``` ``` day requested name resolved version digest 6 stable 1.1.0 fdcdc5473178 11 stable 1.2.0 8f4a3006353c 17 latest 1.4.0 d8a7b48a4c8f 23 stable 1.2.0 8f4a3006353c 29 stable 1.6.0 7cd8d49ade60 34 1.5.0 1.5.0 3341780c3d35 41 stable 1.7.0 569d9c794b31 48 latest 1.10.0 4fc0137cb6ae 55 stable 1.10.0 4fc0137cb6ae 66 1.13.0 1.13.0 d85823359ee1 70 stable 1.16.0 cbeaeddcd652 79 stable 1.16.0 cbeaeddcd652 86 latest 1.19.0 337f943ca885 92 1.21.0 1.21.0 d07c4f23ad28 total deployments 14: by movable tag 11, by immutable version 3 "stable" was deployed 8 times, pointed to 6 distinct artifacts "latest" was deployed 3 times, pointed to 3 distinct artifacts same name, different artifact than the previous deployment: 7 deployments same version, different digest: REJECTED same version, same digest : SAME rejected republish attempts: 1 ``` The numbers collapse into one sentence: **eleven of fourteen deployments were made to a name, and that name carried a different artifact than before on seven of them.** The eight deployments made under the name `stable` carried six separate objects into production; the word written in the record is the same in all eight. Two situations in the table deserve a separate reading. Between day 11 and day 23, the `stable` tag stayed put: the versions published in between did not run the nightly batch job clean, so the tag did not move, and both deployments got the `1.2.0` artifact. So a movable tag does not always move; whether it moved is invisible from the side requesting the deployment. Between day 55 and day 79, the opposite happened: the same name carried the `1.16.0` artifact instead of `1.10.0`. This is rollback's measurable trap too. If a problem is seen on day 79 and someone asks to go back to the stable version, the object resolved is not the `1.10.0` deployed on day 55 — it is the current `1.16.0`. A rollback request produces a forward deployment. For rollback to work, the requested name must be an **immutable version**; three deployments in the table did that. **The difference is hidden here in name resolution.** The resolution happens at deployment time and is not recorded; the deployment record shows only `stable`. Someone looking back later cannot read, in eleven of the records, which deployment carried which object. The difference is not merely undeclared — it is not recorded at all. Closing it is a one-line fix too: the resolved version and digest are written alongside the requested name in the deployment record; then all eleven ambiguous records become readable. The output's last three lines show the immutability check. When someone tries to republish the already-published `1.5.0` under a different digest, the repository returns `REJECTED`; when it arrives with the same digest, it says `SAME` and does not rewrite anything. This second behavior keeps the build step from failing if it runs twice — the natural counterpart for a reproducible build. #### The Input Side and Metadata A repository does not store only a build step's output; a repository of the same kind also stores the packages that are the build step's **input**. The name-resolution problem being measured applies exactly the same on this side. If a build step requests its input package by a movable name instead of an immutable version — something like "newest compatible" — two runs can produce different output even with the previous lesson's four breaking sources closed, because the name has moved to a different object in the time between. This is the **fifth source** that breaks reproducibility, and unlike all the previous ones, it sits outside the build step, not inside it. As in the measured ratio: the object had changed on seven of the eleven requests resolved by name; that same uncertainty carries into the build step on the input side. Closing it means pinning input versions as versions, not as names. The second topic is metadata. The gap the previous lesson left was that the digest does not say which source the output was produced from. The repository is the place that can close this gap: three fields are added to each version record — the source version the output was produced from, the build run's identity, and the result of the tests that ran in that run. Once these three fields are written, the question "which source did the object running in production come from" is read from the repository; unwritten, the answer lives only in people's memory. It should be noted that all three fields are written **beside** the output, not **inside** it: writing them inside would change the digest on every run, exactly as the previous lesson measured, and would break reproducibility. #### What a Retention Policy Closes A repository is not unlimited. A retention policy says which versions are kept for how long. **DC18.** The rule is bound to a number: "keep the last K versions." As K changes, the scan measures three quantities at once — the repository's bytes, the rollback window, and how many of the previous measurement's deployments would have their target deleted. ```js // measurement-network/retention.mjs — scans the retention policy: window, bytes, and unreachable deployments import { schedule } from "./repository.mjs"; const S = schedule(24, 4, 20260311); const last = S[S.length - 1]; // Versions resolved for the 14 deployments in the previous measurement: const DEPLOYED = ["1.1.0", "1.2.0", "1.4.0", "1.2.0", "1.6.0", "1.5.0", "1.7.0", "1.10.0", "1.10.0", "1.13.0", "1.16.0", "1.16.0", "1.19.0", "1.21.0"]; const mb = (b) => (b / 1e6).toFixed(0) + " MB"; console.log("rule".padEnd(18) + "repo".padEnd(10) + "rollback window".padEnd(21) + "unreachable deployments"); for (const k of [3, 5, 8, 12, 18, 24]) { const retained = S.slice(-k); const bytes = retained.reduce((t, s) => t + s.bytes, 0); const names = new Set(retained.map((s) => s.version)); const unreachable = DEPLOYED.filter((v) => !names.has(v)).length; console.log(`last ${k} versions`.padEnd(18) + mb(bytes).padEnd(10) + `${last.day - retained[0].day} days`.padEnd(21) + `${unreachable}/${DEPLOYED.length}`); } console.log(""); for (const interval of [4, 2]) { const sched = schedule(24, interval, 20260311); console.log(`release interval ~${interval} days: 24 versions spread across ${sched[23].day} days, ` + `"last 8 versions" window ${sched[23].day - sched[16].day} days`); } const total = S.reduce((t, s) => t + s.bytes, 0); console.log(`\nwith no rule: ${S.length} versions ${mb(total)}; annual growth at this rate ` + mb((total / last.day) * 365)); ``` ``` rule repo rollback window unreachable deployments last 3 versions 129 MB 4 days 14/14 last 5 versions 214 MB 11 days 13/14 last 8 versions 352 MB 23 days 12/14 last 12 versions 535 MB 37 days 9/14 last 18 versions 805 MB 69 days 6/14 last 24 versions 1073 MB 97 days 0/14 release interval ~4 days: 24 versions spread across 99 days, "last 8 versions" window 23 days release interval ~2 days: 24 versions spread across 47 days, "last 8 versions" window 11 days with no rule: 24 versions 1073 MB; annual growth at this rate 3955 MB ``` The scan turns the retention policy into a trade-off table. The "last eight versions" rule keeps the repository at 352 MB but drops the rollback window to 23 days: wanting to go back to a version older than 23 days finds the object no longer there. Under that same rule, the artifact carried by twelve of the fourteen measured deployments has been deleted — so even if the answer to "which version were we running" is known, that version cannot be brought back. Extending the window to 69 days requires the repository to grow to 805 MB, more than double. A number-bound rule hides a dependency: **the window depends on release frequency.** The same "last eight versions" rule gives a 23-day window for a team releasing every four days and an 11-day window for a team releasing every two days. The rule's text has not changed; the rule's meaning has. Increasing deployment frequency — improving one delivery metric — cuts the rollback window in half without anyone changing anything; this is an example of a decision that improves one metric moving another, measured elsewhere. If the window is meant to be independent of frequency, the rule is bound to a duration ("versions older than ninety days are deleted") or the two conditions are written together. Even when no rule is set, there is a number: at this rate the repository grows by about 3955 MB a year. Unlimited retention is not a decision — it is a deferred one; and when it is deferred, deletion usually starts on the day there is no room left, which is exactly when the rollback window is needed most. #### Summary - A build artifact repository stores two things: content and name. The name comes in two forms — a version that never rebinds once it is bound, and a tag that can move to another version over time. - The immutability check is enforced in the repository: republishing the same version under different content is rejected, republishing under the same content is ignored. - In the measurement, 11 of 14 deployments were made to a movable name, and that name carried a different artifact than before on 7 of them; the 8 deployments made under the name `stable` deployed 6 separate objects. - The difference is hidden in name resolution at deployment time and cannot be read back later because it is not recorded. Writing the resolved version and digest into the record makes 11 ambiguous records readable; for rollback to work, the requested name has to be an immutable version. - A retention policy fixes three quantities at once: "last 8 versions" means a 352 MB repository, a 23-day rollback window, and 12 of 14 deployments having their target deleted. A number-bound rule's window shortens as release frequency rises; when frequency doubles, the window drops from 23 days to 11. #### Next Step The output sitting in the repository is now single and named, but the previous lesson left a debt at its end: the twelve environment-varying values were pulled out of the output and placed into "the environment manifest." Not all of these values are the same kind. The billing gateway's address and the key used to connect to that gateway are read from the same place, but one can be written to a log and the other cannot; one can appear in a deployment record and the other cannot. The next lesson draws and measures that boundary: how many of the values coming from the environment count as configuration, how many count as secret, how many separate ways can a secret leak out, and how is the leak's closure shown with a check? ### Configuration and Secret Separation Source: https://academia.sh/en/courses/devops-fundamentals/configuration-and-secret-separation The three environments' configuration files are written to disk, and a running checker counts how many keys actually change, how many sit in configuration without changing, and how many are undefined in an environment; the four surfaces a secret can leak through are scanned with a scanner, and caught, escaped, and false positives are written separately. What makes the same software the same software across three environments is not the code being identical — it is where the values the code reads sit. An application reading its configuration from the environment, keeping secret values separate from source, was built in the **Server-Side Fundamentals** course. This lesson does not repeat that setup. The unit here is not a single application — it is the **difference between environments**: how many values actually change from environment to environment, how many sit in configuration without changing, how many are undefined in some environment at all, and through how many separate paths do values counted as secrets leak out. The measurement is done on the **regional measurement network**: **fictional** software that collects readings from water meters, verifies the readings, converts them into invoices, and opens work orders for field crews. It has three environments and a nightly batch job. The files below are genuinely written to disk, the checkers genuinely run; every value they contain is fiction, and none of them is a real secret. **DC19.** The only distinction between environments is the configuration files; code, the build artifact, and operating steps are the same across the three environments. **DC20.** Configuration sits in plain files in `KEY=value` form; keys counted as secrets are identified by a name pattern, there is no separate secret store. **DC21.** There are four surfaces a secret can leak through, and all four are represented by a file on disk: build artifact, log, error message, and the file listing committed to the version repository. **DC22.** The leaks are deliberately planted before the scan, and their count is known; counting what escapes depends on that. #### The Three Environments' Files ```bash #!/usr/bin/env bash # Writes the fictional regional measurement network's three-environment configuration to disk. set -e mkdir -p config source cat > config/development.env <<'SON' ENVIRONMENT_LABEL=development MEASUREMENT_DATA_ADDRESS=127.0.0.1:5432 MEASUREMENT_DATA_POOL=4 MEASUREMENT_DATA_PASSWORD=fake-dev-p4t9 METER_READ_TIMEOUT_MS=2000 VERIFICATION_THRESHOLD_PERCENT=12 BATCH_JOB_START=02:30 BATCH_JOB_CONCURRENCY=2 FIELD_WORK_ORDER_ADDRESS=127.0.0.1:8081 INVOICE_CURRENCY=TRY LOG_LEVEL=detail MAX_RETRIES=3 SON cat > config/test.env <<'SON' ENVIRONMENT_LABEL=test MEASUREMENT_DATA_ADDRESS=measure-test.internal:5432 MEASUREMENT_DATA_POOL=8 MEASUREMENT_DATA_PASSWORD=fake-test-r7w2 METER_READ_TIMEOUT_MS=2000 VERIFICATION_THRESHOLD_PERCENT=12 BATCH_JOB_START=02:30 BATCH_JOB_CONCURRENCY=4 FIELD_WORK_ORDER_ADDRESS=work-order-test.internal:8081 INVOICE_CURRENCY=TRY LOG_LEVEL=detail MAX_RETRIES=3 TEST_SEED=4471 SON cat > config/production.env <<'SON' ENVIRONMENT_LABEL=production MEASUREMENT_DATA_ADDRESS=measure-prod.internal:5432 MEASUREMENT_DATA_POOL=32 MEASUREMENT_DATA_PASSWORD=fake-prod-h3m8 METER_READ_TIMEOUT_MS=2000 VERIFICATION_THRESHOLD_PERCENT=12 BATCH_JOB_START=02:30 BATCH_JOB_CONCURRENCY=16 INVOICE_CURRENCY=TRY LOG_LEVEL=summary MAX_RETRIES=3 INVOICE_SIGNING_KEY=fake-sig-9c1e SON ``` The application's read point is singular and contains no branching that looks at the environment's name. ```js // source/config.mjs — the fictional application's single settings point: every value is read from the environment. export const config = { env: process.env.ENVIRONMENT_LABEL, dataAddress: process.env.MEASUREMENT_DATA_ADDRESS, password: process.env.MEASUREMENT_DATA_PASSWORD, workOrderAddress: process.env.FIELD_WORK_ORDER_ADDRESS, signingKey: process.env.INVOICE_SIGNING_KEY, pool: Number(process.env.MEASUREMENT_DATA_POOL), timeoutMs: Number(process.env.METER_READ_TIMEOUT_MS ?? 2000), thresholdPercent: Number(process.env.VERIFICATION_THRESHOLD_PERCENT ?? 12), batchStart: process.env.BATCH_JOB_START ?? "02:30", concurrency: Number(process.env.BATCH_JOB_CONCURRENCY ?? 2), currency: process.env.INVOICE_CURRENCY ?? "TRY", logLevel: process.env.LOG_LEVEL ?? "summary", maxRetries: Number(process.env.MAX_RETRIES ?? 3), rereadDays: Number(process.env.REREAD_DAYS ?? 30), }; ``` #### Counting the Difference The checker reads the three files and the source's read points separately, then compares the two. ```js // check.mjs — compares the three environment files against the source's read points. import { readFileSync } from "node:fs"; const ENVIRONMENTS = ["development", "test", "production"]; const readEnvFile = (a) => Object.fromEntries(readFileSync(`config/${a}.env`, "utf8") .split("\n").filter(Boolean).map((s) => [s.slice(0, s.indexOf("=")), s.slice(s.indexOf("=") + 1)])); const files = Object.fromEntries(ENVIRONMENTS.map((a) => [a, readEnvFile(a)])); const keys = [...new Set(ENVIRONMENTS.flatMap((a) => Object.keys(files[a])))].sort(); const sourceText = readFileSync("source/config.mjs", "utf8"); // which keys are read, const sourceReads = new Map(); // how many have an embedded default for (const [, name, opt] of sourceText.matchAll(/process\.env\.([A-Z_]+)(\s*\?\?)?/g)) sourceReads.set(name, Boolean(opt)); const counts = { changing: [], fixed: [], missing: [] }; for (const k of keys) { const v = ENVIRONMENTS.map((a) => files[a][k]); if (v.some((x) => x === undefined)) counts.missing.push(k); else if (new Set(v).size === 1) counts.fixed.push(k); else counts.changing.push(k); } const hiddenDefault = [...sourceReads].filter(([k]) => !keys.includes(k)).map(([k]) => k); const required = [...sourceReads].filter(([, v]) => !v).map(([k]) => k); const print = (label, list) => { // print the count, wrap names three per line console.log(`${label.padEnd(37)}: ${list.length}`); for (let i = 0; i < list.length; i += 3) console.log(" " + list.slice(i, i + 3).join(" ")); }; console.log("total keys (across three files)".padEnd(37) + `: ${keys.length}`); print(" present in all, value differs", counts.changing); print(" present in all, value the same", counts.fixed); print(" missing in at least one env", counts.missing); console.log("keys the source reads".padEnd(37) + `: ${sourceReads.size}, without a default: ${required.length}`); print(" in no environment file", hiddenDefault); print(" in a file, source does not read it", keys.filter((k) => !sourceReads.has(k))); for (const a of ENVIRONMENTS) print(`fails at runtime / ${a}`, required.filter((k) => files[a][k] === undefined)); ``` ``` total keys (across three files) : 14 present in all, value differs : 6 BATCH_JOB_CONCURRENCY ENVIRONMENT_LABEL LOG_LEVEL MEASUREMENT_DATA_ADDRESS MEASUREMENT_DATA_PASSWORD MEASUREMENT_DATA_POOL present in all, value the same : 5 BATCH_JOB_START INVOICE_CURRENCY MAX_RETRIES METER_READ_TIMEOUT_MS VERIFICATION_THRESHOLD_PERCENT missing in at least one env : 3 FIELD_WORK_ORDER_ADDRESS INVOICE_SIGNING_KEY TEST_SEED keys the source reads : 14, without a default: 6 in no environment file : 1 REREAD_DAYS in a file, source does not read it : 1 TEST_SEED fails at runtime / development : 1 INVOICE_SIGNING_KEY fails at runtime / test : 1 INVOICE_SIGNING_KEY fails at runtime / production : 1 FIELD_WORK_ORDER_ADDRESS ``` Of fourteen keys, only six are defined across all three environments and actually differ in value. **This is where the difference is hidden — these six lines.** The remaining eight keys do not carry the difference; they carry its appearance. Five keys carry the same value in all three environments. These are not configuration — they are constants; sitting in the configuration file has a cost: they can be hand-changed in a single environment. When `VERIFICATION_THRESHOLD_PERCENT` is pulled from twelve to fifteen in production, verification behavior silently diverges, and this change falls under no code review, because the code did not change. This is where the declared difference and the difference visible only at runtime split apart. The declared kind is the six lines in the files: written, readable, comparable. The runtime-only kind splits in two. `REREAD_DAYS` is in no environment file; its default in the source is thirty. So **a fourth environment definition sits inside the source**, and all three environments use it. Changing its value takes changing code, not configuration; the principle that configuration lives in the environment breaks exactly here, and where it breaks is read from no file. `TEST_SEED` is this pattern's mirror image: it is written in the test file, but the source never reads it. A declared difference cannot be assumed to be effective; someone who thinks test runs with a fixed seed is looking at a test that runs seedless. The defect class the difference conceals shows up in how the missing keys are distributed. `FIELD_WORK_ORDER_ADDRESS` is missing only in production: the work-order-opening path runs fine in development and test, and in production it goes to an undefined address on its first work order. This defect class's name is **the path visible only in production**, and test cannot see it, because nothing is missing in test. `INVOICE_SIGNING_KEY` is the mirror case: because it is defined only in production, the signing path has never run outside production at all. Both gaps carry the same count — one per environment — but one blows up in production, and the other is tried there for the first time. #### When the Difference Shows Up The count reported one missing required key in each environment; it did not report **when** the gap would show up. The same number corresponds to two separate defect classes, and what determines the difference between them is whether the application reads that key at startup or on first use. ```js // start.mjs — validates the required keys at startup; the signing path runs at job time. import { config } from "./source/config.mjs"; const STARTUP_REQUIRED = ["env", "dataAddress", "password", "workOrderAddress"]; const missing = STARTUP_REQUIRED.filter((k) => config[k] === undefined); if (missing.length) { console.log(` startup rejected -> missing: ${missing.join(", ")}`); process.exit(1); } console.log(` started -> pool=${config.pool} concurrency=${config.concurrency} log=${config.logLevel}`); try { // nightly batch job's signing step if (!config.signingKey) throw new Error("signing key undefined"); console.log(` 03:00 batch job -> signed`); } catch (e) { console.log(` 03:00 batch job -> failed: ${e.message}`); } ``` ```bash #!/usr/bin/env bash # Loads each environment's file and starts the application separately. for o in development test production; do echo "$o:" ( set -a; . "config/$o.env"; set +a; node start.mjs ) || true done ``` ``` development: started -> pool=4 concurrency=2 log=detail 03:00 batch job -> failed: signing key undefined test: started -> pool=8 concurrency=4 log=detail 03:00 batch job -> failed: signing key undefined production: startup rejected -> missing: workOrderAddress ``` All three environments were missing a required key, and the count was one in all three. The run produced three separate outcomes. Production was rejected at startup: this is the cheapest form of the gap, because the failure is visible while the eyes of whoever is deploying are on the screen. Development and test started, and the failure came out at three in the morning, in the middle of the nightly batch job. The gap is the same; its cost is not — **the only thing that produces the difference is the length of the startup-validation list.** Because `signingKey` is not on that list, it fails late in all three environments. The output also shows two differences that are not the measurement's subject. The log level is detail in two environments and summary in production. The record of a path visible only in production is kept in a log that is throttled in that same environment: the difference throttles, in the same environment, the very signal that would let someone see the defect it produces. The nightly batch job's concurrency is two, four, and sixteen. A race condition that passes with four workers in test shows up on the first night in production with sixteen. This is a declared difference — it is written in the file — but being declared does not mean it is been tested. Who changes these three values is a separate question too. Pool size, concurrency, and log level are in the operating team's hands and usually change after an incident; the verification threshold and the payment form are the writing team's decision. Because both sit in the same file, the changes look identical, and which difference came from which team can only be told apart by looking at the file's history. #### The Secret's Four Ways Out A secret is also a value that changes from environment to environment, and it is included in the count up to this point. The reason it is counted separately is a single asymmetry: a wrong port gets fixed, a leaked secret cannot be fixed — it can only be rotated, and it stays valid until it is. The four surfaces it can leak through are written to disk below with fictional content. ```bash #!/usr/bin/env bash # Four surfaces: build artifact, log, error message, file listing committed to the repository. set -e mkdir -p build log error cat > build/package.mjs <<'SON' // Build artifact file (fictional): source bundled into a single file together with production values. const config = { address: "measure-prod.internal:5432", MEASUREMENT_DATA_PASSWORD: "fake-prod-h3m8", signature: "fake-sig-" + "9c1e", }; export const connect = () => config; SON cat > log/nightly.jsonl <<'SON' {"time":"03:12:04","event":"batch_job_started","env":"production","meterCount":48210} {"time":"03:12:04","event":"data_connection","connection":"measure:fake-prod-h3m8@measure-prod.internal:5432"} {"time":"03:12:05","event":"config_summary","MEASUREMENT_DATA_PASSWORD":"****","LOG_LEVEL":"summary"} {"time":"04:41:19","event":"invoice_signed","signatureB64":"ZmFrZS1zaWctOWMxZQ==","count":48210} SON cat > error/last-error.txt <<'SON' Uncaught error: invoice signature could not be verified (nightly batch job, production) request: POST /invoice/sign?signature=fake-sig-9c1e config : { env: 'production', MEASUREMENT_DATA_PASSWORD: 'fake-prod-h3m8', pool: 32 } stack : batch-job.mjs:118 -> sign.mjs:42 SON cat > repository-listing.txt <<'SON' source/config.mjs config/example.env config/production.env package.json SON ``` The scanner has two rules: a literal match on the secret's value, and the secret key's name pattern. ```js // scanner.mjs — searches four surfaces for secrets; rule=value is a literal match, rule=name is the name pattern. import { readFileSync } from "node:fs"; const SURFACES = { build: "build/package.mjs", log: "log/nightly.jsonl", error: "error/last-error.txt", repo: "repository-listing.txt" }; const NAME_PATTERN = /PASSWORD|KEY|\.env$/; const values = ["development", "test", "production"] // secret values across the three files .flatMap((a) => readFileSync(`config/${a}.env`, "utf8").split("\n").filter(Boolean)) .filter((s) => NAME_PATTERN.test(s.slice(0, s.indexOf("=")))) .map((s) => s.slice(s.indexOf("=") + 1)); const findings = []; for (const [name, path] of Object.entries(SURFACES)) readFileSync(path, "utf8").split("\n").filter(Boolean).forEach((line, i) => { const rule = [values.some((v) => line.includes(v)) && "value", NAME_PATTERN.test(line.trim()) && "name"].filter(Boolean); if (rule.length) findings.push({ name, lineNo: i + 1, line, rule: rule.join("+") }); }); // Planted leaks: surface name + that line's distinguishing trace. const PLANTED = [["build", "MEASUREMENT_DATA_PASSWORD:"], ["build", 'signature: "fake-sig-'], ["log", '"connection"'], ["log", '"signatureB64"'], ["error", "MEASUREMENT_DATA_PASSWORD:"], ["error", "signature=fake"], ["repo", "production.env"]]; const matched = (f) => PLANTED.some(([s, trace]) => s === f.name && f.line.includes(trace)); const escaped = PLANTED.filter(([s, trace]) => !findings.some((f) => f.name === s && f.line.includes(trace))); console.log(`secret values: ${values.length} surfaces: ${Object.keys(SURFACES).length} ` + `planted leaks: ${PLANTED.length} reported lines: ${findings.length}`); console.log(`caught: ${PLANTED.length - escaped.length} escaped: ${escaped.length} ` + `false positives: ${findings.filter((f) => !matched(f)).length}`); for (const f of findings) console.log(` [${matched(f) ? "real" : "FALSE"}] ${f.name}:${f.lineNo} rule=${f.rule}`); for (const [s, trace] of escaped) console.log(` [ESCAPED] ${s} trace=${trace}`); ``` ``` secret values: 4 surfaces: 4 planted leaks: 7 reported lines: 7 caught: 5 escaped: 2 false positives: 2 [real] build:4 rule=value+name [real] log:2 rule=value [FALSE] log:3 rule=name [real] error:2 rule=value [real] error:3 rule=value+name [FALSE] repo:2 rule=name [real] repo:3 rule=name [ESCAPED] build trace=signature: "fake-sig- [ESCAPED] log trace="signatureB64" ``` #### Caught, Escaped, False Positive The scanner reported seven lines, and seven leaks were planted. **The counts are equal — the sets are not.** This equality is a warning for every reading that judges coverage by looking at a scan report's total. What the two escaped leaks share is that the secret's value does not sit in the form being searched for. One is split in two by concatenation (`"fake-sig-" + "9c1e"`), the other is turned into base 64 (`ZmFrZS1zaWctOWMxZQ==`). Both carry the same secret, and both fall outside the value-matching rule. The general shape of this is: **a literal value search falls behind every step that transforms the value** — a build tool's line concatenation, a log writer's encoding, a message formatter's truncation. All five caught leaks are lines where the value sits unchanged. The way to raise the catch rate is known: transformed forms of the searched value are also generated — its encoded form, its truncated form, each of its parts — and all of them are searched separately. The cost of this is written directly to false positives: the shorter the value, the greater the chance of a random match, and a four-character fragment like `9c1e` turns up in ordinary text too. This is the relationship between the two rules; every widening that reduces escapes increases false positives, and unless the two counts are written separately, which one is rising stays invisible. Both false positives come from the name rule, and both **are correct behavior itself**: the masked log line was written to hide the secret, and the example configuration file is exactly what belongs in the version repository. Its cost is not numeric — it is behavioral. Once two false positives are on the list, whoever reads that list stops checking lines one by one on the next scan; the chance of finding the two escaped leaks drops right along with the false-positive count. The four surfaces also have separate owners: the build artifact belongs to the compile step, the log to operations, the error message to application code, the repository listing to version control. When a leak is closed on one surface, the others stay open, because whoever closes it only sees their own surface. The decision the number delivers here is this: secret separation is not a decision made in one place — it is a boundary maintained separately across four distinct surfaces. #### Summary - Six of the fourteen keys in the three environment files actually change; this is where the difference is hidden. Five keys carry the same value in all three environments, meaning they are constants, not configuration. - Three keys are missing in at least one environment; one of the six required keys stays undefined in every environment. The work order address missing in production blows up only in production, while the signing key defined only in production means the signing path has never run outside it. - In the run, the same count produced three separate outcomes: production was rejected at startup, the other two environments started and failed at three in the morning in the middle of the batch job. What produces the difference is the length of the startup-validation list. - The declared difference is the six lines in the files; the difference visible only at runtime is the fourth environment defined by a default embedded in the source (`REREAD_DAYS`) and the dead key no code reads (`TEST_SEED`). - Five of the seven leaks planted across four surfaces were caught, two escaped; both escapees were transformed forms of the value (concatenation and base 64). - The scanner reported seven lines, and there were seven leaks; two were false positives, and both flagged correct behavior. Equal totals are not proof of coverage. #### Next Step Configuration living in the environment is not a rule that stands alone — it is a member of an interdependent set; this lesson measured one member of that set and, while measuring, saw the rule silently break through a default embedded in the source. The full set was built with its names in the **Server-Side Fundamentals** course. Rather than repeat those names, the next lesson asks a single question: can each principle be turned into a check that runs against the files on disk? The ones that can are run against the fictional application's real files, with caught and escaped counted separately; for the ones that cannot, what replaces them is written down too. ### Twelve-Factor App Source: https://academia.sh/en/courses/devops-fundamentals/twelve-factor-app Eight of the twelve principles are turned into a runnable check that decides by looking at files on disk and is run against a fictional application's real files; how many of eight planted defects were caught, how many escaped, and how many false positives were raised is counted, and what replaces the four principles that cannot be translated is written down. The twelve factors name the contract between whoever writes an application and whoever runs it. What the factors say was built in the **Server-Side Fundamentals** course and is not repeated here. This lesson's question is different: can a factor be turned into a **runnable check** that looks at files on disk and says yes or no? If it can, what does the check catch, what does it miss, what does it raise a false alarm over? If it cannot, what replaces it? A factor carrying a decision does not depend on its sentence sounding true — it depends on someone seeing it when it is violated. A factor that cannot be turned into a check is only visible when someone thinks to look, and the look that has to be thought of is an unmeasured one. The measurement is done again on the **regional measurement network**: **fictional** software that collects water meter readings, verifies them, converts them into invoices, and opens field work orders. The file tree below is genuinely written to disk, the check genuinely runs; the files and the dependency names are all fiction. **DC23.** The checker sees only the file tree on disk; it does not see repository history, operating steps, or team habits. **DC24.** A factor counts as translatable if it can produce a binary decision on a single tree in a single run; a factor that needs a second environment, a second process, or human judgment for its decision counts as untranslatable. **DC25.** The defects are deliberately planted before the scan, and their count is known; counting what escapes depends on that. **DC26.** All the checks operate at the level of text and files; there is no syntax-tree parsing. The class of defect that escapes follows directly from this choice. #### The Eight Principles Turned Into Checks The table below does not write what the principle says — it writes what the principle **looks at on disk**. | Principle | What the check looks at | What decides | |---|---|---| | Dependencies | is there a lock file next to the declaration file | file existence | | Configuration | embedded default and embedded server address in the source | the `??` pattern, address pattern | | Port binding | does the source open its own socket | the `.listen(` call | | Logs | is the log written to a file instead of standard output | a file-write call | | Disposability | does every declared entry point handle the shutdown signal | the `SIGTERM` string | | Processes | is persistent state kept in process memory | a top-level mutable container | | Build, release, run | is an environment value embedded in the build artifact | address pattern | | Admin processes | does the one-off job use the application's code | import path | What the eight share is that the decision can be read from a single tree. The remaining four principles do not appear in this table; the reasons are counted after the measurement. The third column carries the table's real information. A principle is a sentence of intent; a check is a text pattern. The translation between them is a narrowing on every line: the sentence "logs are an event stream" is reduced to the rule "no file-write call should appear in the source directory"; the sentence "processes are stateless" to the rule "no top-level mutable container should be declared." The narrowing produces loss in both directions. Where the rule stays narrower than the principle, a defect escapes; where it reaches into a file the principle does not cover, a false alarm is born. The numbers below measure both losses separately, because a single "pass / fail" total does not say which direction the loss went in. #### The Fictional Application's Files ```bash #!/usr/bin/env bash # Writes the fictional measurement network application's file tree to disk. Dependency names are fictional too. set -e mkdir -p source test script build cat > package.json <<'SON' { "name": "measurement-network", "type": "module", "entrypoints": ["source/server.mjs", "script/nightly-fix.mjs"], "dependency": { "measurement-resolver": "2.4.0", "invoice-formatter": "1.1.3" } } SON cat > source/config.mjs <<'SON' export const config = { env: process.env.ENVIRONMENT_LABEL, dataAddress: process.env.MEASUREMENT_DATA_ADDRESS, port: Number(process.env.MEASUREMENT_PORT), rereadDays: Number(process.env.REREAD_DAYS ?? 30), }; SON cat > source/server.mjs <<'SON' import { createServer } from "node:http"; import { config } from "./config.mjs"; export const server = createServer((request, response) => response.end(config.env ?? "none")); server.listen(config.port); SON cat > source/log.mjs <<'SON' import { appendFileSync } from "node:fs"; export const log = (k) => appendFileSync("/var/log/measurement.log", JSON.stringify(k) + "\n"); SON cat > source/report.mjs <<'SON' import { writeFileSync } from "node:fs"; export const writeInvoice = (f) => writeFileSync(`output/invoice-${f.no}.json`, JSON.stringify(f)); SON cat > source/batch-job.mjs <<'SON' const WORK_ORDER_ADDRESS = "work-order-production.internal:8081"; export const queue = (() => { const pending = new Map(); return { add: (id, v) => pending.set(id, v), size: () => pending.size, WORK_ORDER_ADDRESS }; })(); SON cat > test/fake-measurement.mjs <<'SON' // In-process fake measurement store; the address does not go to a real destination. export const fakeAddress = "127.0.0.1:5432"; SON cat > script/nightly-fix.mjs <<'SON' const ADDRESS = process.env.MEASUREMENT_DATA_ADDRESS; console.log(`fix ran -> ${ADDRESS}`); SON cat > build/package.mjs <<'SON' // Build artifact file (fictional): the production address was embedded at compile time. export const TARGET = "measure-prod.internal:" + 5432; SON ``` Eight defects are planted in this tree. Because their count is written inside the checker, what escapes can be counted too; none of the checks reads that list. #### The Run ```js // audit.mjs — runs the translatable principles against the fictional application's real files. import { readFileSync, existsSync } from "node:fs"; const FILES = ["package.json", "source/config.mjs", "source/server.mjs", "source/log.mjs", "source/report.mjs", "source/batch-job.mjs", "test/fake-measurement.mjs", "script/nightly-fix.mjs", "build/package.mjs"]; const text = Object.fromEntries(FILES.map((y) => [y, readFileSync(y, "utf8")])); const under = (prefix) => FILES.filter((y) => y.startsWith(prefix)); const search = (scope, pattern) => scope .flatMap((y) => text[y].split("\n").map((s, i) => ({ path: y, lineNo: i + 1, line: s }))) .filter((b) => pattern.test(b.line)); const note = (path, line) => ({ path, lineNo: 0, line }); const ADDRESS = /["'][\w.-]+:\d{4,5}["']/; const CHECKS = [ ["dependencies", () => existsSync("package.lock") ? [] : [note("package.json", "no package.lock")]], ["configuration", () => [...search(under("source/"), /process\.env\.\w+\s*\?\?/), ...search([...under("source/"), ...under("test/"), ...under("script/")], ADDRESS)]], ["port-binding", () => search(under("source/"), /\.listen\(/).length ? [] : [note("source/", "no listen call")]], ["logs", () => search(under("source/"), /(appendFileSync|writeFileSync|createWriteStream)\(/)], ["disposability", () => JSON.parse(text["package.json"]).entrypoints .filter((y) => !text[y].includes("SIGTERM")).map((y) => note(y, "no SIGTERM handler"))], ["processes", () => search(under("source/"), /^(const|let|var)\s+\w+\s*=\s*new (Map|Set)\(/)], ["build-release-run", () => search(under("build/"), ADDRESS)], ["admin-processes", () => under("script/").filter((y) => !text[y].includes("source/")) .map((y) => note(y, "no import from source/"))], ]; const PLANTED = [ // defects planted before the scan ["package.json", "package.lock"], ["source/config.mjs", "?? 30"], ["source/batch-job.mjs", "work-order-production.internal"], ["source/log.mjs", "appendFileSync"], ["source/server.mjs", "SIGTERM"], ["source/batch-job.mjs", "new Map("], ["build/package.mjs", "measure-prod.internal:"], ["script/nightly-fix.mjs", "source/"], ]; const findings = CHECKS.flatMap(([name, f]) => f().map((b) => ({ ...b, check: name }))); const real = (b) => PLANTED.some(([y, trace]) => y === b.path && b.line.includes(trace)); const escaped = PLANTED.filter(([y, trace]) => !findings.some((b) => b.path === y && b.line.includes(trace))); const falsePositives = findings.filter((b) => !real(b)); const row = (a, b, c, d) => a.padEnd(24) + b.padStart(10) + c.padStart(8) + d.padStart(8); console.log(row("check", "findings", "real", "false")); for (const [name, f] of CHECKS) { const b = f(); console.log(row(name, `${b.length}`, `${b.filter(real).length}`, `${b.filter((x) => !real(x)).length}`)); } console.log(`translatable principles: ${CHECKS.length} untranslatable: ${12 - CHECKS.length}`); console.log(`planted defects: ${PLANTED.length} reported findings: ${findings.length}`); console.log(`caught: ${PLANTED.length - escaped.length} escaped: ${escaped.length} ` + `false positives: ${falsePositives.length}`); for (const [y, trace] of escaped) console.log(` [ESCAPED] ${y} trace=${trace}`); for (const b of falsePositives) console.log(` [FALSE] ${b.path}:${b.lineNo} check=${b.check}`); ``` ``` check findings real false dependencies 1 1 0 configuration 3 2 1 port-binding 0 0 0 logs 2 1 1 disposability 2 1 1 processes 0 0 0 build-release-run 0 0 0 admin-processes 1 1 0 translatable principles: 8 untranslatable: 4 planted defects: 8 reported findings: 9 caught: 6 escaped: 2 false positives: 3 [ESCAPED] source/batch-job.mjs trace=new Map( [ESCAPED] build/package.mjs trace=measure-prod.internal: [FALSE] test/fake-measurement.mjs:2 check=configuration [FALSE] source/report.mjs:2 check=logs [FALSE] script/nightly-fix.mjs:0 check=disposability ``` #### Caught, Escaped, False Positive Six of eight defects were caught, two escaped, and the checks raised three false positives. The rows that actually need reading in the table are the ones reporting zero. `port-binding` reported zero findings and was right: the source opens its own socket. `processes` and `build-release-run` also reported zero, and **both were wrong** — the defect was sitting right there in both. **Zero findings measures the rule's coverage, not the absence of defects**, and the table writes both situations the same way. The two escaped defects share a class. `batch-job.mjs` keeps a map in process memory, but the map sits inside a closure: the rule looks for a top-level declaration, and the container is two spaces in. The production address in the build artifact, meanwhile, was concatenated at compile time (`"measure-prod.internal:" + 5432`), so it does not sit in the form the address pattern searches for at all. Both are examples of a rule operating at the text level failing to see a fact that sits at the structural level; the secret scanner in the previous lesson missed exactly this same class. This is not carelessness in the checks — it is the direct consequence of DC26: the rule looks at text, the defect sits in structure. All three false positives flagged **correct behavior**. The fixed address in the test fixture points to a fake store that stays in-process; reading from configuration there would be meaningless. The call that writes the invoice file is not a log — it is the product itself. The nightly fix script is a one-off job and has no need to handle a shutdown signal. All three arise from the gap between the rule's **scope** and the principle's **subject**: the rule looks at the file, while the principle asks what that file is for. The cost of a false positive runs in the same direction as the cost of an escape. If three of a nine-line report are empty, whoever reads the report a second time stops opening lines one by one; the chance of finding the two escaped defects drops right at that point. There is no single number that evaluates a set of checks — unless caught, escaped, and false positive are written separately, which one is rising stays invisible. #### What the Escapes Look Like at Runtime In the check table, the two escaped defects were each just a gap. To see what that gap corresponds to, both are run in the same process. ```js // leaked.mjs — shows the two defects the audit could not see, in a run. import { queue } from "./source/batch-job.mjs"; import { TARGET } from "./build/package.mjs"; queue.add("meter-1041", 3120); // first reading packet queue.add("meter-2277", 8840); // second packet, same process console.log(`ENVIRONMENT_LABEL : ${process.env.ENVIRONMENT_LABEL}`); console.log(`records left in queue : ${queue.size()}`); console.log(`build artifact target : ${TARGET}`); ``` ```bash #!/usr/bin/env bash # Runs the same build artifact under two environment labels. for e in test production; do ENVIRONMENT_LABEL="$e" node leaked.mjs; done ``` ``` ENVIRONMENT_LABEL : test records left in queue : 2 build artifact target : measure-prod.internal:5432 ENVIRONMENT_LABEL : production records left in queue : 2 build artifact target : measure-prod.internal:5432 ``` Two records were left in the queue. The second reading packet found what the first left in process memory. When the process restarts, these records vanish; once scale changes, a second process's queue starts empty, and two processes run the same nightly job with two different states. The defect the check could not see shows up in the run as a number. The second line is sharper. Whether the environment label is test or production, the build artifact's target stays the same: the production data store's address. **The build artifact carries the environment; the environment does not carry the build artifact.** A nightly batch job running in the test environment looks at production data, and no configuration file declares this, because the difference is not in configuration — it is hidden in a decision made at compile time. The previous lesson's environment-difference count could never have seen this value; what it counted was configuration files, and this address appears in none of them. This is the course's rule in its narrowest form. The same software did not behave differently across two environments — **it behaved the same**, and that is exactly the problem. The environment label changed, the behavior did not; the difference is embedded somewhere the environment variable cannot reach, so the environment lost its authority to change it. The zero on the `build-release-run` row in the check table was what covered up this difference. The way to catch both defects is known: the rule looks not at text but at a parsed syntax tree; only at that level can it see a declaration inside a closure or a value concatenated at compile time. Its cost is removing DC26 — the check no longer just reads the file, it has to parse the language too. The eight checks' cheapness came from that choice; once the cheapness is lost, whether the check can still run frequently is up for debate too. #### What Replaces the Four Untranslatable Principles | Principle | Why it cannot be measured on a single tree | What replaces it | |---|---|---| | Codebase | how many deployments come out of a single codebase is a repository-level fact; the tree shows a single deployment | matching the source identity the build artifact carries against deployment records | | Backing services | that the address comes from configuration is checked, but that the source is swappable can only be seen by actually swapping it | swapping the address and repeating the run | | Concurrency | that it scales through the process model cannot be read from a single-process tree | running under load with two processes and output that does not change with the process count | | Dev/prod parity | requires comparing two environments; the tree is a single environment | the previous lesson's environment-difference count | What all four share is that their decision needs a **second observation**: a second deployment, a second address, a second process, a second environment. The boundary runs exactly here, and its name is **decidability from a single observation**. The eight checks decided on a single tree; the four needed a comparison. This distinction determines cost too. The eight checks run in seconds on every change; the remaining four need a separate environment, a separate run, or a load-generating harness. What is cheap runs often, what is expensive runs rarely — and a defect a rarely-run measurement catches has usually already reached production by the time it is caught. The previous lesson's environment-difference count was what replaced one of the four expensive principles, and none of this lesson's checks stands in for it. The cost difference brings an ownership difference too. The eight cheap checks are in the writing team's hands; they run the instant the code changes, and whoever changed it sees the result. The four expensive measurements fall into the domain of the operating team, which holds the second environment, the second process, or the load harness. Half of the same set of principles is measured by one team, half by the other; which team a violation becomes visible to depends on the kind of violation. The signal gap between the measurement network's two teams comes from exactly this split. #### Summary - Eight of twelve principles were turned into checks that decide on a single file tree, four could not be; the criterion that separates them is decidability from a single observation. - Six of eight planted defects were caught, two escaped; the checks reported nine findings and three were false positives. - One of the three checks reporting zero findings was right, two were wrong. Zero findings measures the rule's coverage, not the absence of defects. - Both escaped defects belong to the same class: the rule looks at text, the defect sits in structure — a state inside a closure and an address concatenated at compile time. Both showed up at runtime: two records were left in process memory, and the build artifact's target came out as the production address under both environment labels. - All three false positives flagged correct behavior; they arose from the gap between the rule's scope and the principle's subject. - What replaces the four untranslatable principles all needs a second observation; that is why they run rarely, and the defects they catch are caught late. #### Next Step All four untranslatable principles needed a second observation, and two of them named that observation directly: a second environment. Up to this point, environments were assumed to exist — development, test, and production existed, the difference between them was counted, checks were run against the same tree. **How** the environment itself comes to exist was never asked. Is it set up by hand, does a script set it up, or is it a declaration stating the desired end state? The next lesson compares these three forms and asks the same question: how many differences remain between two environments set up twice from the same declaration, and where is that difference hidden? ### Infrastructure Provisioning Models Source: https://academia.sh/en/courses/devops-fundamentals/infrastructure-provisioning-models The same environment is set up manually, with a script, and declaratively; the difference between two setups, the state a script leaves behind on its second run and when it is cut off midway, and the configuration drift accumulated over six periods are counted. A running reconciler reads the desired state from a file, the actual state from disk, closes the difference, and does zero work on its second run. The previous lesson turned twelve principles into checks and found that four needed a second observation to decide — often a second environment. The application itself is now checkable: its dependencies are declared, its configuration is external, its shutdown is testable. The machine it stands on is set up by hand, and how it is set up is inside the scope of no check. **Provisioning** is bringing an environment to its desired state: installing packages, writing settings, opening directories. It has three forms — **manual**, **scripted**, **declarative** — and what separates them is not speed, it is **how many separate results the same recipe produces**. This lesson sets up the same environment in all three and measures three numbers: the setup steps, the difference between two setups, and how much configuration drift grows over successive periods. The measurement is again on the **regional measurement network**: **fictional** software that collects water meter readings, verifies them, and opens field work orders. The files are genuinely written to disk, the scripts and the reconciler genuinely run. **DC27.** The environment is represented by seven items — two package versions, three settings, two directories. **DC28.** The manual setup's instructions are prose, and three of the seven steps give no number; the two hands' interpretations diverge on exactly these three steps. **A hand is a model**, its interpretation is written in advance. **DC29.** The script runs the seven steps in order, and no step reads the state first; the cutoff point is triggered by an environment variable. **DC30.** In the declarative model, the desired state is a file; the reconciler reads the actual state from disk and touches only the differing item. **DC31.** The drift scan runs six periods, and one manual intervention happens in each period; the interventions are written in advance, there is no randomness. #### Three Models, the Same Seven Items ```bash #!/usr/bin/env bash # Three separate recipes for the same environment: a declaration, a manual instruction sheet, and a provisioning script. set -e cat > desired.json <<'SON' { "package/resolver": "3.4.0", "package/rule-package": "9.1.0", "config/log-level": "summary", "config/concurrent-workers": "8", "config/batch-start": "02:30", "dir/data": null, "dir/log": null } SON cat > manual-recipe.txt <<'SON' 1. open the package directory, install resolver version 3.4.0 2. pin the rule package to version 9.1.0 3. pull the log level to a value suited for production 4. set the concurrent worker count to the machine's capacity 5. schedule the nightly batch job for a low-traffic hour 6. open the data directory 7. open the log directory SON cat > setup.sh <<'SON' #!/usr/bin/env bash # Scripted provisioning: seven steps run in order; no step reads the state first. set -e mkdir -p environment/package environment/config environment/dir echo "3.4.0" >> environment/package/resolver echo "9.1.0" >> environment/package/rule-package echo "summary" >> environment/config/log-level [ -z "$CUT" ] || exit 3 # the cutoff point echo "8" >> environment/config/concurrent-workers echo "02:30" >> environment/config/batch-start mkdir environment/dir/data environment/dir/log SON ``` All three recipes target the same seven items. The declaration counts items along with their values; the instructions give no number on three steps ("a value suited for," "to the machine's capacity," "a low-traffic hour"); the script runs the seven steps in order. The checker fixes nothing — it only compares the disk against the desired state and counts. ```js // check.mjs — compares the environment on disk against the desired state and counts; fixes nothing. import { readFileSync, existsSync, statSync } from "node:fs"; const root = process.argv[2] ?? "environment"; const desired = JSON.parse(readFileSync("desired.json", "utf8")); const state = (path) => { const t = `${root}/${path}`; if (!existsSync(t)) return "missing"; const g = statSync(t).isDirectory() ? "" : readFileSync(t, "utf8").trim(); return g === (desired[path] ?? "") ? "ok" : "drifted"; }; const s = Object.keys(desired).map(state); const count = (t) => s.filter((x) => x === t).length; const drift = count("drifted") + count("missing"); console.log(process.argv[3] === "-d" ? drift // -d: drift count only : ` items ${s.length} ok ${count("ok")} drifted ${count("drifted")}` + ` missing ${count("missing")} drift ${drift}`); ``` #### Same Recipe, How Many Separate Results ```bash #!/usr/bin/env bash # The scripted model's three scenarios and manual setup's two hands. run() { printf '%s\n' "$1"; shift; "$@" >/dev/null 2>&1; printf ' script exit code %s\n' "$?"; } rm -rf environment run "1st run (empty environment)" bash setup.sh node check.mjs run "2nd run (same script again)" bash setup.sh node check.mjs rm -rf environment run "3rd run (cut off midway)" env CUT=1 bash setup.sh node check.mjs printf ' records of where it stopped: %s files\n' "$(ls environment/*.state 2>/dev/null | wc -l | tr -d ' ')" hand() { # $1 dir, $2..$4 the three values the hand interpreted rm -rf "$1"; mkdir -p "$1/package" "$1/config" "$1/dir/data" "$1/dir/log" printf '3.4.0\n' > "$1/package/resolver"; printf '9.1.0\n' > "$1/package/rule-package" printf '%s\n' "$2" > "$1/config/log-level"; printf '%s\n' "$3" > "$1/config/concurrent-workers" printf '%s\n' "$4" > "$1/config/batch-start" } hand hand-1 summary 8 02:30 hand hand-2 detail 4 03:00 printf 'manual setup: two hands from the same instructions\n difference between the two setups: %s items\n' \ "$(diff -rq hand-1 hand-2 | wc -l | tr -d ' ')" node check.mjs hand-1 node check.mjs hand-2 ``` ``` 1st run (empty environment) script exit code 0 items 7 ok 7 drifted 0 missing 0 drift 0 2nd run (same script again) script exit code 1 items 7 ok 2 drifted 5 missing 0 drift 5 3rd run (cut off midway) script exit code 3 items 7 ok 3 drifted 0 missing 4 drift 4 records of where it stopped: 0 files manual setup: two hands from the same instructions difference between the two setups: 3 items items 7 ok 7 drifted 0 missing 0 drift 0 items 7 ok 4 drifted 3 missing 0 drift 3 ``` The script's first run puts all seven items in place. The same script's second run **breaks** the environment: five items drift, and the script falls over at the seventh step. The reason is two characters — the appending `>>` and the `mkdir` that takes no `-p`. Because the script never reads the state first, it cannot know whether the work was already done; an operation producing the same result whether applied once or many times is called **idempotence**, and this script does not carry it. The run cut off midway is more expensive: three items are in place, four are missing, and **the record of where it stopped is zero**. The environment is neither in its old state nor its new one; the only thing the script reports is an exit code, and that code does not say which step it stopped at. In manual setup, two hands read the same instructions and diverged on three items — exactly the three steps the instructions gave no number for. The difference here is not carelessness — it is **the recipe's unwritten part**: a human fills the gap between the recipe and the environment, and the value they fill it with is written down nowhere. The measure collapses into one line: in manual setup, the same recipe produced two separate results; in scripted provisioning, the second run produced a third result — and more broken than the first. #### Drift Being Countable ```js // reconcile.mjs — reads the desired state from a file and the actual state from disk; writes and closes the difference. import { readFileSync, writeFileSync, mkdirSync, existsSync, statSync } from "node:fs"; const desired = JSON.parse(readFileSync("desired.json", "utf8")); const actual = (path) => { const t = `environment/${path}`; if (!existsSync(t)) return null; return statSync(t).isDirectory() ? "" : readFileSync(t, "utf8").trim(); }; const diff = Object.entries(desired).map(([path, d]) => ({ path, d: d ?? "", a: actual(path) })) .filter((f) => f.a !== f.d) .map((f) => ({ ...f, kind: f.a === null ? "missing" : "drifted" })); for (const f of diff) { // reconciliation: only the differing item is touched const t = `environment/${f.path}`; if (desired[f.path] === null) mkdirSync(t, { recursive: true }); else { mkdirSync(t.slice(0, t.lastIndexOf("/")), { recursive: true }); writeFileSync(t, desired[f.path] + "\n"); } } console.log(` items ${Object.keys(desired).length} diff ${diff.length}` + ` (missing ${diff.filter((f) => f.kind === "missing").length},` + ` drifted ${diff.filter((f) => f.kind === "drifted").length}) work done ${diff.length}`); for (const f of diff) console.log(` ${f.kind.padEnd(8)} ${f.path.padEnd(27)}` + (desired[f.path] === null ? " (directory opened)" : ` "${f.a ?? ""}" -> "${f.d}"`)); ``` ```bash #!/usr/bin/env bash # Reconciler: empty environment, second run (idempotence), after manual intervention. rm -rf environment echo "1st run (empty environment)"; node reconcile.mjs echo "2nd run (right after)"; node reconcile.mjs echo "manual intervention: worker count set to 16, log directory deleted" echo 16 > environment/config/concurrent-workers; rm -rf environment/dir/log echo "3rd run (after the intervention)"; node reconcile.mjs drift() { case $1 in 1) echo 16 > environment/config/concurrent-workers ;; 2) echo detail > environment/config/log-level ;; 3) rm -rf environment/dir/log ;; 4) echo 3.5.0 > environment/package/resolver ;; 5) echo 03:15 > environment/config/batch-start ;; 6) echo 9.2.0 > environment/package/rule-package ;; esac; } echo "" echo "six periods, one manual intervention per period (drift accumulation)" for reconciliation in off on; do rm -rf environment; node reconcile.mjs > /dev/null printf ' reconciliation %-3s :' "$reconciliation" for d in 1 2 3 4 5 6; do drift "$d" [ "$reconciliation" = on ] && node reconcile.mjs > /dev/null printf ' %s' "$(node check.mjs environment -d)" done printf '\n' done echo "" printf 'recipe size (lines): instructions %s, script %s, declaration %s, reconciler %s\n' \ "$(wc -l < manual-recipe.txt)" "$(wc -l < setup.sh)" "$(wc -l < desired.json)" \ "$(wc -l < reconcile.mjs)" | tr -s ' ' ``` ``` 1st run (empty environment) items 7 diff 7 (missing 7, drifted 0) work done 7 missing package/resolver "" -> "3.4.0" missing package/rule-package "" -> "9.1.0" missing config/log-level "" -> "summary" missing config/concurrent-workers "" -> "8" missing config/batch-start "" -> "02:30" missing dir/data (directory opened) missing dir/log (directory opened) 2nd run (right after) items 7 diff 0 (missing 0, drifted 0) work done 0 manual intervention: worker count set to 16, log directory deleted 3rd run (after the intervention) items 7 diff 2 (missing 1, drifted 1) work done 2 drifted config/concurrent-workers "16" -> "8" missing dir/log (directory opened) six periods, one manual intervention per period (drift accumulation) reconciliation off : 1 2 3 4 5 6 reconciliation on : 0 0 0 0 0 0 recipe size (lines): instructions 7, script 11, declaration 3, reconciler 27 ``` The reconciler does three things: reads the desired state from a file, reads the actual state from disk, touches only the differing item. The first run finds all seven items missing and does seven jobs; the second run does **zero work**. The idempotence the script lacked is measured here, and the reason is a single change of order — read first, write second. The distinguishing part is in the third run. The manual intervention broke two items; the reconciler wrote both down by name, with their old and new values, and closed them. **Drift is a number here:** `diff 2`. In the other two models, there was nowhere to count drift, because the desired state was not readable — one sat in a person's interpretation, the other in a sequence of commands. A sequence of commands says **what to do**, not **what should be**; drift's countability is the difference between these two sentences. The six-period scan reduces this to two lines: when reconciliation does not run, interventions accumulate (1, 2, 3, 4, 5, 6); when it does, the series stays at zero. Drift does not disappear — it is born and closed every period. What does not accumulate is not drift — it is **unknown drift**. #### The Cost of the Three Models | Measure | Manual | Scripted | Declarative | |---|---|---|---| | recipe size (lines) | 7 | 11 | 3 plus 27 (written once) | | items in place on first setup | 7 and 4 (two hands) | 7 | 7 | | separate results from the same recipe | 2 | 2 | 1 | | if the recipe runs a second time | unmeasurable | 5 items drift, exit 1 | 0 work | | what is left if it is cut off midway | unrecorded | 3 items, 0 records | the difference is recomputed | | drift accumulated over 6 periods | 1 to 6 | 1 to 6 | 0 every period | | catching drift | none | none | the 7 items in the declaration | Writing cost looks lowest in the declarative model — three lines — but the number is misleading: the reconciler is twenty-seven lines, written **once** and reused for every environment. The script is eleven lines and grows with every new item; the instructions are seven lines, and grow more ambiguous too, because every sentence of prose is a point of interpretation. First-setup time is the same order of magnitude across all three models; the distinction does not show up there — it shows up on the second run and after an intervention. This is what the provisioning debate most often misses: on first setup alone, a script looks sufficient — there it produces the **same result** as the declarative model. The declarative model's limit is countable too. The reconciler sees only the seven items written in the declaration; when an eighth file is added to the environment by hand, it never shows up as a difference in any run. **What makes drift countable is the declaration's scope**, and the difference outside that scope accumulates in no written place — in a hand's interpretation, in the partial state of a run cut off midway. This is where the six differences visible only at runtime came from, in this topic's first lesson. #### Summary - In manual setup, two hands diverged from the same instructions on 3 items; the steps where they diverged were the three steps the instructions gave no number for. - The script's first run put 7 items in place; the second broke 5 items and fell over; the run cut off midway left 3 items in place, 4 missing, and 0 records of where it stopped. - The reconciler did 7 jobs on the first run, 0 on the second; after a manual intervention it wrote down and closed 2 drifted items by name. - Over six periods, drift accumulated from 1 to 6 without reconciliation and stayed at 0 with it; what does not accumulate is not drift — it is unknown drift. - The declarative model's limit is the declaration's scope: everything outside that scope falls back to manual setup's condition. #### Course Wrap-Up | Lesson | Measured difference or metric | Number | Where the difference is hidden | |---|---|---|---| | What Is DevOps? | occurrence that never reaches its owner, steps to reach | 4 of 22 occurrences; 7.0 → 1.6 steps | the split observation scope | | Throwing Over the Wall | context item dropped at the boundary | 21 of 29 items unwritten, 14 drop | tacit context | | Value Stream | wait piling up in a single step | 1485 of 2237 steps; value-added share 38.5% | the distribution of wait | | Delivery Metrics | the metric that moves when batch drops from 8 to 2 | deployment 3.2 → 13.5; errors fixed at 12 | the metric's denominator | | Site Reliability Engineering | the error budget and its exhaustion | 43.2 minutes; heavy incident 208%; 8/12 periods | between the target and the outage distribution | | Feedback Loops | the error that does not fall into an early loop | 32 of 120 errors, 95% of the delay | loop coverage | | Environments | environment parity, difference per dimension | 18 differences; 12 in the manifest, 6 at runtime | the dimension the manifest does not declare | | Build Artifacts and Immutability | the source that breaks reproducibility | 4 sources, 3 invisible on a single machine | the value embedded in the output | | Build Artifact Repositories | the separate artifact a tag carried | 11 deployments by name, 7 carried a different artifact | name resolution at deployment time | | Configuration and Secret Separation | the key that changes, the leak path | 6 of 14 keys; 5 of 7 leaks | the default embedded in the source | | Twelve-Factor App | the principle turned into a check, the defect caught | 8 of 12 principles; 6 of 8 defects | between the rule's scope and the principle's subject | | Infrastructure Provisioning Models | the separate result born from the same recipe | manual 3, scripted 5, declarative 0; drift 6 → 0 | the recipe's unwritten part | All twelve rows are examples of a single rule: **if the same software behaves differently across two environments, the difference is somewhere, and it is not measured until it is named.** No number in the table comes from the software itself — every one comes from where the difference is hidden. Whether an item is written down, what a name resolves to at deployment time, at which step an instruction gives no number — none of these lives inside the code, none of them shows on its own. Reading the last column, a clustering becomes visible: most of the places the difference hid were **the environment itself** — installed packages, library versions, file paths, timezone. Widening the manifest, pulling configuration out, and making provisioning declarative made this difference measurable, but they did not **remove the difference from the environment**; each time, what happened was moving the difference somewhere countable. One question remains, and **M22/K02 Containers** takes it up: is it possible to pull this difference inside the build artifact and remove it from the environment? If installed packages, versions, and file paths become part of the output, the only thing left for the environment to do is run that output, and nothing is left to provision. If it is possible, what is the cost — how much does the output grow, what happens to reproducibility, what is left that cannot be pulled out of the environment?