Pipeline Integration

The previous lesson baked the image, the one before that applied the setting. Neither asked one thing: who triggers these, and who approves them. All of this course’s measurements were run by hand, and in a hand-run flow a person stands between plan and apply and sees the two back to back. A pipeline separates the two: the plan is produced at one stage, stored, submitted for approval; the apply runs at another moment. M22/K04 measured what a gate caught and what it missed; that measure is not repeated, it is applied.

IV37. The pipeline runs the regional measurement network’s — the fictional software collecting water-meter readings’ — infrastructure record in two stages, and passes 18 changes over 30 days. IV38. An item holds 3 lines in the plan, 12 lines are read per minute, and the approver does not read a plan past 40 lines. IV39. The seeds 20260901–20260903 are visible and the numbers are an order of magnitude. IV40. The provider offers 9 resource types and 4 operations, the environment has 84 fictional resources, and the narrow permission is derived from the last 12 runs. IV41. 26 manual changes occur over 30 days; the touched item’s probability of being in the record’s scope is 0.62, an in-scope item’s probability of being protected is 0.24, and the urgent-fix rate is 0.35. IV42. The freshness gate regenerates the plan at apply time and compares it against the approved one, adding 0.5 minutes per change.

Plan, Approval, Apply

Approval is a signature; what can be measured is how much reading stands behind it, and how much time passes between the signature and the apply — a plan is a snapshot of a moment, and it goes stale.

// measurement-network/pipeline.mjs — plan stage, approval, apply stage (model; a kind, not a product)
export const generator = (seed) => () =>
  (seed = (seed * 1103515245 + 12345) % 2147483648) / 2147483648;

// plan lines per item; lines read per minute; the line count where the approver stops reading;
// minutes the freshness gate adds per change.
const LINE = 3, SPEED = 12, THRESHOLD = 40, REGEN = 0.5;

// 18 infrastructure changes pass through the pipeline over 30 days; quantities come from the generator.
export const changes = (seed = 20260901) => {
  const r = generator(seed);
  return Array.from({ length: 18 }, () => {
    const item = 1 + Math.floor(r() * 14);
    return { item, line: item * LINE + 6, wait: Math.round(r() * 470) / 10,
      changed: r() < 0.3, drifted: 1 + Math.floor(r() * 3) };
  });
};

// gate=true: at apply time the plan is regenerated and compared against the approved one.
export const run = (list, gate) => {
  const unread = list.filter((d) => d.line > THRESHOLD);
  const stale = list.filter((d) => d.changed);
  const drifted = stale.reduce((t, d) => t + d.drifted, 0);
  return {
    unread: unread.length, unreadItems: unread.reduce((t, d) => t + d.item, 0),
    minutes: list.reduce((t, d) =>
      t + (d.line <= THRESHOLD ? d.line / SPEED : 0.5) + (gate ? REGEN : 0), 0),
    stale: stale.length, unprotected: gate ? 0 : drifted, drifted,
    held: gate ? stale.length : 0, applied: 18 - (gate ? stale.length : 0),
  };
};

if (import.meta.url === `file://${process.argv[1]}`) {
  const list = changes(), sum = (a) => list.reduce((t, d) => t + a(d), 0);
  console.log(`18 changes, ${sum((d) => d.item)} items, ${sum((d) => d.line)} plan lines; ` +
    `average ${(sum((d) => d.wait) / 18).toFixed(1)} hours between plan and apply\n`);
  console.log("freshness gate".padEnd(16) + "approved".padEnd(11) + "unread".padEnd(11) +
    "unread items".padEnd(17) + "review time".padEnd(16) + "stale plan".padEnd(12) +
    "unprotected items".padEnd(19) + "applied");
  for (const gate of [false, true]) {
    const s = run(list, gate);
    console.log((gate ? "yes" : "no").padEnd(16) + "18/18".padEnd(11) +
      `${s.unread}/18`.padEnd(11) + `${s.unreadItems}/${sum((d) => d.item)}`.padEnd(17) +
      `${s.minutes.toFixed(1)} min`.padEnd(16) + `${s.stale}/18`.padEnd(12) +
      `${s.unprotected}/${s.drifted}`.padEnd(19) + `${s.applied}/18`);
  }
  const noGate = run(list, false);
  console.log(`\nthe approver does not read once the plan exceeds ${THRESHOLD} lines: ${noGate.unread} changes, ` +
    `${noGate.unreadItems} items approved without review`);
}

18 changes, 129 items, 495 plan lines; average 28.6 hours between plan and apply

freshness gate  approved   unread     unread items     review time     stale plan  unprotected items  applied
no              18/18      4/18       49/129           29.0 min        3/18        5/5                18/18
yes             18/18      4/18       49/129           38.0 min        3/18        0/5                15/18

the approver does not read once the plan exceeds 40 lines: 4 changes, 49 items approved without review

Eighteen of eighteen changes passed approval; the approval rate says nothing. What says something is the second column: 4 plans were approved without being read, and those four plans carried 49 of 129 items — thirty-eight percent of the items were applied unlooked-at. This is not an attention problem, it is a scale problem; the review load grows linearly, as already counted in declarative-provisioning/07.

The second measure is the time between plan and apply: an average of 28.6 hours, and reality does not stand still over that time. 3 of the eighteen plans were stale when applied, and 5 items that did not appear in the approved plan entered the apply; the approved document and the applied work were not the same thing.

One of this course’s gates only does its job here. Format checking, policy scanning, infrastructure testing, and cost estimation can all run outside a pipeline too; the freshness gate cannot. Regenerating the plan at apply time and comparing it against the approved plan requires the approved plan to be a stored document; a hand-run flow has nothing stored to compare against. The gate adds 9 minutes, holds 3 applies, and rescues 5 items.

The Pipeline’s Permission

The pipeline now runs the apply itself, and for that it has an identity on the infrastructure. If the permission is broad, the pipeline can do anything; if that identity falls, anything becomes possible.

// measurement-network/permission.mjs — the pipeline's apply permission over the infrastructure (model)
import { generator, changes } from "./pipeline.mjs";

const TYPE = ["machine", "disk", "network", "subnet", "security rule", "queue", "data store",
  "object store", "identity"];
const OPERATION = ["create", "read", "update", "delete"];
const RESOURCE = [14, 22, 3, 6, 19, 2, 4, 3, 11];   // fictional resources per type; 84 total
const WINDOW = 12;   // past runs the permission is derived from

// Each change touches a few (type, operation) pairs; the pair set comes from the generator.
const touches = (seed = 20260902) => {
  const r = generator(seed);
  return changes().map(() => {
    const c = new Set(), n = 1 + Math.floor(r() * 3);
    for (let i = 0; i < n; i++)
      c.add(`${TYPE[Math.floor(r() * 9)]}/${OPERATION[Math.floor(r() * 4)]}`);
    return [...c];
  });
};

const touch = touches();
const all = TYPE.flatMap((t) => OPERATION.map((i) => `${t}/${i}`));
const narrow = [...new Set(touch.slice(0, WINDOW).flat())];
const noDelete = narrow.filter((c) => !c.endsWith("/delete"));
const deletable = (y) => TYPE.reduce((t, x, i) => t + (y.includes(`${x}/delete`) ? RESOURCE[i] : 0), 0);

console.log(`${TYPE.length} resource types x ${OPERATION.length} operations = ${all.length} operations; ` +
  `${RESOURCE.reduce((a, b) => a + b)} fictional resources\n`);
console.log("permission".padEnd(23) + "operations".padEnd(12) + "passed".padEnd(14) +
  "left over".padEnd(12) + "deletable resources");
for (const [ad, y] of [["broad (all)", all], [`narrow (from ${WINDOW} runs)`, narrow],
  ["narrow, no delete", noDelete]]) {
  const passed = touch.filter((c) => c.every((x) => y.includes(x))).length;
  console.log(ad.padEnd(23) + `${y.length}/36`.padEnd(12) + `${passed}/18`.padEnd(14) +
    `${18 - passed}/18`.padEnd(12) + `${deletable(y)}/84`);
}
const newPairs = [...new Set(touch.slice(WINDOW).flat())].filter((c) => !narrow.includes(c));
console.log(`\nnarrow permission was derived by looking at the past; the next 6 runs produced ${newPairs.length} ` +
  `new operation pairs: ${newPairs.join(", ")}`);

9 resource types x 4 operations = 36 operations; 84 fictional resources

permission             operations  passed        left over   deletable resources
broad (all)            36/36       18/18         0/18        84/84
narrow (from 12 runs)  19/36       14/18         4/18        14/84
narrow, no delete      18/36       13/18         5/18        0/84

narrow permission was derived by looking at the past; the next 6 runs produced 5 new operation pairs: subnet/delete, queue/update, security rule/delete, subnet/create, disk/create

Broad permission finishes the table in one line: 36 of 36 operations, 18 of 18 changes, zero left on the table. The last column is the cost — if the pipeline’s identity is compromised, 84 of 84 resources can be deleted; the pipeline is a wider access than any record written in this course grants.

When permission is narrowed by looking at the past twelve runs, the operation count drops to 19, deletable resources to 14; the cost is that 4 of the 18 changes are left on the table. Once delete permission is removed too, deletable resources drop to zero, and what is left on the table rises to 5. The bottom line writes down why narrowing is not permanent: the next six runs produced 5 new operation pairs never seen before. Permission derived from the past leaves the future’s work on the table, and for what is left, the manual path counted in the first lesson carries on exactly as before.

When a Manual Change Hits the Pipeline

Machines still get touched by hand after a pipeline is set up; the question is what happens to that touch at the pipeline’s next run.

// measurement-network/collision.mjs — what a manual change becomes at the pipeline's next run (model)
import { generator } from "./pipeline.mjs";

const RUN = [1.4, 3.1, 4.0, 6.8, 8.2, 9.9, 12.5, 14.1, 15.0, 17.7,
  19.2, 21.4, 22.8, 24.3, 25.1, 27.6, 28.9, 29.5];   // day of each of the 18 pipeline runs (fictional)
const COVERAGE = 0.62;    // probability that a manually touched item is in the record's scope
const PROTECTION = 0.24;  // probability that an in-scope item is protected or recreates on change

// 26 manual changes occur over 30 days; each touches a single item.
const manual = (seed = 20260903) => {
  const r = generator(seed);
  return Array.from({ length: 26 }, () => {
    const day = Math.round(r() * 300) / 10;
    const inScope = r() < COVERAGE;
    return { day, inScope, protected: inScope && r() < PROTECTION, urgent: r() < 0.35 };
  });
};

const RESULT = { "silently reverted": [], "conflicting (pipeline halts)": [], "unnoticed": [] };
for (const o of manual()) {
  const next = RUN.find((g) => g > o.day);
  const outcome = !o.inScope ? "unnoticed"
    : o.protected ? "conflicting (pipeline halts)" : "silently reverted";
  // An out-of-record item is never seen on any run; it has no delay either.
  const delay = !o.inScope || next === undefined ? null : next - o.day;
  RESULT[outcome].push({ ...o, delay });
}

console.log("26 manual changes, 30 days, 18 pipeline runs\n");
console.log("outcome".padEnd(31) + "count".padEnd(9) + "urgent".padEnd(8) + "delay");
for (const [ad, list] of Object.entries(RESULT)) {
  const g = list.map((o) => o.delay).filter((v) => v !== null);
  console.log(ad.padEnd(31) + `${list.length}/26`.padEnd(9) +
    String(list.filter((o) => o.urgent).length).padEnd(8) +
    (g.length ? `${(g.reduce((a, b) => a + b, 0) / g.length).toFixed(1)} days` : "-"));
}
const reverted = RESULT["silently reverted"];
console.log(`\n${reverted.length} reverted changes produced no error, all of them were a single line in the plan; ` +
  `${reverted.filter((o) => o.urgent).length} of them were urgent fixes`);

26 manual changes, 30 days, 18 pipeline runs

outcome                        count    urgent  delay
silently reverted              15/26    4       1.2 days
conflicting (pipeline halts)   3/26     1       0.7 days
unnoticed                      8/26     2       -

15 reverted changes produced no error, all of them were a single line in the plan; 4 of them were urgent fixes

The three rows are three separate events, and only the middle one is visible. In 3 collisions the pipeline halts and a person makes the call; these count as proof the pipeline is working. 15 changes are silently reverted and vanish after an average of 1.2 days; none of them produce an error, because the pipeline was built to do exactly that. 4 of those fifteen were urgent fixes — because their records were never updated, they went back to the old value the record wanted. iac-fundamentals/05 already counted a rollback breaking a live fix; the pipeline makes this reversion faster and quieter.

Every reversion was a single line in the plan; given that four of the plans were approved without being read, the source of the silence is not the pipeline, it is the unread plan line.

The third row writes down the course’s rule one last time: 8 items are items the pipeline never sees at all. Because they are out of record, they neither revert nor collide; they remain in place while the pipeline runs flawlessly.

Summary

• Eighteen of eighteen changes passed approval, but 4 plans were approved without being read, and those four plans carried 49 of 129 items (38 percent); the approval rate is not a measure.
• An average of 28.6 hours passed between plan and apply, 3 of the 18 plans were stale, and 5 items not visible at approval time entered the apply.
• The freshness gate is meaningful only in a pipeline, because it requires the approved plan to be stored; it adds 9 minutes, holds 3 applies, and rescues 5 items.
• Under broad permission, if the identity falls, 84 of 84 resources can be deleted; once permission narrows to 19 operations, deletable resources drop to 14, work left on the table rises to 4 changes, and the next six runs produce 5 new operation pairs.
• Of 26 manual changes, 15 were silently reverted (4 urgent fixes), 3 collided, and 8 were never seen at all because they were out of record.

Course Wrap-Up

Lesson	What enters the record	Gap between record and reality	The record’s cost or limit
iac-fundamentals/01	8-line description	9 items diverge	5 items out of scope
iac-fundamentals/02	9-item target	6/9 against 2/9	5 guard lines
iac-fundamentals/03	image, 96 events	10/12 machines unique	3 fixes lost
iac-fundamentals/04	34 items, 35 definitions	6 items conflict	5 items in no record
iac-fundamentals/05	26 items	12 drifted, 6 close	12 items out of scope
declarative-provisioning/01	3 blocks, 14 assignments	8 defects at apply	9 fields blind to the type
declarative-provisioning/02	20 fields	36 fields on target	16 fields not in record
declarative-provisioning/03	8 resources, 5 data sources	scan 8/13 items	2 external fields invisible
declarative-provisioning/04	14 variables, 24 values	4 wrong inputs passed	48 hand-written values across 6 environments
declarative-provisioning/05	14 resources, 23 edges	requirements 27, seen 23	4 requirements not in record
declarative-provisioning/06	3 forms, 10 resources	1 versus 6 items	525 machine-seconds downtime
declarative-provisioning/07	plan, 11 items	8 of 13 addresses hold	39.6 minutes across 30 regions
declarative-provisioning/08	20 of 24 fields	first plan, 4 items	4 fields out of record
state-and-modules/01	56 fields, 80 in state	4 of 33 items out of record	3 items do not return on loss
state-and-modules/02	two concurrent applies	12 items lost without a lock	12 resources out of record
state-and-modules/03	620 items, 673 edges	66 edges cross the boundary	3 items out of record
state-and-modules/04	17 or 51 lines	23 versus 4 wrong environments	2 environments, 9 fields behind each
state-and-modules/05	module, 20 lines	0 of 4 items seen	1 resource out of record
state-and-modules/06	9 definitions, 12 versions	9/9 do not say what is running	25 versions behind
state-and-modules/07	7 trace points	14 copies, marking → 4	3 points stay open
iac-validation/01	11 resources, 59 lines	three checks 16/16	4 unseen rejections
iac-validation/02	62 lines, 7 policies	3 missed, 2 false alarms	3 computed values invisible
iac-validation/03	15 defects	three levels 12/15	11 leftovers, 0 in record
iac-validation/04	24 items, 12 changes	9 at 0.51x–1.45x	11 items not estimable
iac-validation/05	1700 record lines	1598 vs. 1530 hold	27/61 settings out of scope
iac-validation/06	content record 32 entries	52 entries in images	20 entries out of record
iac-validation/07	18 changes, 495 lines	3 stale plans, 5 items	8 items unseen

This is the course’s rule: infrastructure can only be managed as well as its record; every item the record does not cover is a source of drift. Three limits repeat in the last column. In nineteen rows, the limit is an item the record does not cover — an out-of-record resource, a provider default, an unseen requirement, a computed value, an indirect component. In five rows the limit is the record’s own operating overhead: a guard line, review minutes, a version left behind, a delayed apply. In three rows the limit is an unrecoverable loss: a deleted fix, a machine out of service, an item that does not return. Growing the record shrinks the first group and grows the second; this trade-off was never zeroed out in any lesson.

Infrastructure is now tied to a record: it is written, planned, checked, priced, approved, and becomes work a pipeline applies. But all of this course’s measurements belonged to the moment of change. No one yet sees what the infrastructure does while it runs: which machine carries how much load, which queue is backing up. M22/K06 Observability and Operations starts with that question — who decides what gets measured?