---
title: 'Infrastructure as Code'
source: 'https://academia.sh/en/courses/infrastructure-as-code'
language: en
updated: '2026-09-13T05:08:23+00:00'
license: 'CC BY-SA 4.0'
lessons: 27
outcomes:
    - 'Provisions and changes resources with a declarative infrastructure definition'
    - 'Sets up state management to suit team work'
    - 'Writes reusable infrastructure components with modules'
---

# Infrastructure as Code

Tying infrastructure to a record and measuring how well the record matches reality: reproducibility and traceability, the separation of the declarative from the imperative approach, replacement instead of in-place modification, provisioning and configuration management as separate layers, detecting a manual change as drift, the block and type structures of the configuration language, providers, resources and data sources, parameterization, dependency ordering, iteration, the plan and apply flow, resource replacement and import, the state file, remote state and locking, state splitting, workspaces, module authoring and registries, sensitive values, static checks, policy scanning, infrastructure testing, cost estimation, agent-based and agentless configuration management, server image production, and integration with the pipeline.

## Fundamental Concepts

What tying infrastructure to code buys: tying reproducibility and traceability to a number, the difference between a desired-state declaration and a step list, the cost and gain of a rebuild versus an in-place change, the separation between the provisioning layer and the configuration management layer, and detecting a manual change as drift.

### What Is Infrastructure as Code

Source: https://academia.sh/en/courses/infrastructure-as-code/what-is-infrastructure-as-code

The same staging environment is set up twice — once by hand from a prose request, once from a record — and the two setups are compared item by item. For a quarter's worth of changes, how many times the question of who, when, and what changed gets answered, how many items fall outside the record's scope, and the cost of tying to a record in lines and extra steps are counted.

The previous course tied the application's path to production to the repository: when a branch
merges, the pipeline runs, gates are passed, and deployment proceeds on its own. The application's
path is written from end to end. The machines it stands on, the queue, the data store, and the
network, though, are set up through a different route — a request is opened, someone reads that
request and sets up the environment, and the result of the setup lives only in the environment
itself.

**Infrastructure as code** is tying infrastructure to a readable **record**: what the environment
should be is written in a file, and the environment is set up from that file. This course's unit
of measure is the record, and every lesson counts three things — the difference between what the
record says and reality, the record's scope, and the record's cost. This lesson sets up the first:
the same environment is set up twice, once by hand from a prose request and once from a record,
then compared **item by item**. Then, for a quarter's worth of changes, it measures how many times
the question "who, when, what changed" gets answered, how many items fall outside the record's
scope, and the cost of tying to a record, in lines and extra steps.

The measurement runs on the **regional measurement network**: **fictional** software that collects
water-meter readings, verifies them, converts them into invoices, and opens field work orders. It
has three environments; only the staging environment is set up here. Infrastructure is not really
provisioned — resources and setups are modeled with `node`; the record and request files that feed
the model are genuinely written to disk.

**IF1.** The staging environment is represented by six resources — one network, two machine
groups, one queue, one data store, one object store. An item is one attribute of a resource; the
record writes twenty items.
**IF2.** The setup request is prose, and seven of its eleven items give no number. **A hand is a
model**; its interpretation is written in advance — there is no randomness.
**IF3.** The provider default is five items: settings the record does not write, that arise in the
environment regardless. They are born the same in both setups.
**IF4.** A quarter sees fourteen changes; each change asks three questions — who, when, what
changed. Each change's route is written in advance.
**IF5.** A change routed through the record answers all three questions; a change that falls into
a work ticket answers two (it does not record what changed); an untracked change answers none.
**IF6.** The record route has three extra steps per change: write the record, get it reviewed,
read the plan.

#### Same Environment, Two Recipes

```bash
#!/usr/bin/env bash
# Two recipes for the same environment: the record (configuration language) and a prose setup request.
set -e

cat > record.infra <<'SON'
# Our configuration language's format: resource <type> "<name>" { <key> = <value> }
# A value is a number, a "string", or true/false; a line starting with # is a comment.
resource network "measurement-network" { block = "10.40.0.0/16"  private = true  region = "north" }
resource machine "collector" { count = 3  size = "medium"  disk_gb = 40  image = "base-7" }
resource machine "verifier" { count = 2  size = "small"  disk_gb = 20  image = "base-7" }
resource queue "reading-queue" { retention_hours = 72  partitions = 6  dead_letter = true }
resource data_store "measurement-store" { size_gb = 200  backup_days = 14  version = "5.2" }
resource object_store "archive" { class = "sparse"  versioning = true  lifecycle_days = 365 }
SON

cat > request.txt <<'SON'
staging environment setup request
1. open the measurement network as private, address block 10.40.0.0/16, north region
2. open enough collector machines, size medium, image base-7
3. size the collector disk to the measurement volume
4. two verifier machines, size small, disk 20 GB, image base-7
5. open the reading queue, set retention close to production, dead letter on
6. set the queue partition count to the worker count
7. open the measurement store at 200 GB, version 5.2
8. set the store's backup window to the corporate standard
9. open an object store for the archive, versioning on
10. choose the archive's storage class by cost
11. write a reasonable value for the archive lifecycle day count
SON
```

The configuration language is defined here with its own small format: a `resource` line carries a
type, a name, and key-value pairs inside curly braces. Six lines count twenty items, and each item
is tied to a single value — two readers understand the same thing. The request targets the same
environment, but seven of its items give a criterion instead of a number: "enough", "to the
measurement volume", "close to production", "the corporate standard", "by cost", "a reasonable
value". These seven points are the request's **unwritten** part, and whoever builds it fills them
in.

```js
// setup.mjs — parses the record and builds the same environment two ways: from the record and by hand. Both are models.
import { readFileSync } from "node:fs";

const BLOCK = /^resource\s+(\w+)\s+"([\w-]+)"\s*\{(.*)\}$/;      // the language's one-line format
export const readRecord = (text) => {
  const k = new Map();
  for (const line of text.split("\n")) {
    const m = BLOCK.exec(line.trim());                        // comments and blank lines are skipped
    if (m) for (const x of m[3].matchAll(/(\w+)\s*=\s*("[^"]*"|[\w./-]+)/g))
      k.set(`${m[1]}.${m[2]}.${x[1]}`, x[2].replaceAll('"', ""));
  }
  return k;
};
export const record = readRecord(readFileSync("record.infra", "utf8"));

// Provider default: items the record does not write, that arise in the environment regardless.
const providerDefaults = new Map([["network.measurement-network.egress_path", "open"],
  ["machine.collector.access_rule", "management-port-open"],
  ["queue.reading-queue.encryption", "provider-key"],
  ["data_store.measurement-store.maintenance_window", "sunday-03:00"],
  ["object_store.archive.access_rule", "in-account"]]);

// The hand's pre-written interpretation: the seven points the request gives no number for (2, 3, 5, 6, 8, 10, 11).
const handInterpretation = new Map([["machine.collector.count", "2"], ["machine.collector.disk_gb", "40"],
  ["queue.reading-queue.retention_hours", "24"], ["queue.reading-queue.partitions", "6"],
  ["data_store.measurement-store.backup_days", "7"], ["object_store.archive.class", "dense"],
  ["object_store.archive.lifecycle_days", "90"]]);
const handSkipped = ["object_store.archive.versioning"];        // written in the request, skipped in practice
const handAdded = new Map([["network.measurement-network.access_rule", "office-block"],
  ["machine.collector.extra_disk", "attached"], ["machine.verifier.dns_record", "verifier.internal"]]);

export const buildFromRecord = () => new Map([...record, ...providerDefaults]);
export const buildManually = () => {
  const o = new Map([...record, ...handInterpretation, ...providerDefaults, ...handAdded]);
  for (const a of handSkipped) o.delete(a);
  return o;
};
export const interpretationPoints = handInterpretation.size, outOfRecordCount = providerDefaults.size;
```

#### The Difference, Item by Item

```js
// diff.mjs — compares the two setups item by item; fixes nothing.
import { record, buildFromRecord, buildManually, interpretationPoints, outOfRecordCount } from "./setup.mjs";

const a = buildFromRecord(), b = buildManually();
const all = [...new Set([...a.keys(), ...b.keys()])].sort();
const diff = all.filter((k) => a.get(k) !== b.get(k)).map((k) => ({ k,
  kind: !b.has(k) ? "missing" : !a.has(k) ? "extra" : "diverged" }));
const countBy = (t) => diff.filter((f) => f.kind === t).length;

console.log(`items  from record ${a.size}   by hand ${b.size}   union ${all.length}`);
console.log(`diff ${diff.length} items   (diverged ${countBy("diverged")}, missing ${countBy("missing")},` +
  ` extra ${countBy("extra")})`);
for (const f of diff)
  console.log(`  ${f.kind.padEnd(8)} ${f.k.padEnd(42)} ${(a.get(f.k) ?? "-").padEnd(20)}` +
    ` ${b.get(f.k) ?? "-"}`);
console.log(`items in the record ${record.size}   points the request gives no number for ${interpretationPoints}` +
  `   of these diverged ${diff.filter((f) => f.kind === "diverged").length}`);
console.log(`items outside the record's scope ${outOfRecordCount}   the same in both setups,` +
  ` written in neither`);
```

```
items  from record 25   by hand 27   union 28
diff 9 items   (diverged 5, missing 1, extra 3)
  diverged data_store.measurement-store.backup_days   14                   7
  diverged machine.collector.count                    3                    2
  extra    machine.collector.extra_disk               -                    attached
  extra    machine.verifier.dns_record                -                    verifier.internal
  extra    network.measurement-network.access_rule    -                    office-block
  diverged object_store.archive.class                 sparse               dense
  diverged object_store.archive.lifecycle_days        365                  90
  missing  object_store.archive.versioning            true                 -
  diverged queue.reading-queue.retention_hours        72                   24
items in the record 20   points the request gives no number for 7   of these diverged 5
items outside the record's scope 5   the same in both setups, written in neither
```

The two setups of the same environment diverged on nine items, and the difference comes from three
separate kinds. **Diverged** covers five items, five of the seven points the request gave no
number for: the remaining two are invisible because the hand's interpretation happened to land on
the same value as the record. These two are not a gain, they are a coincidence — the
interpretation-point count is still seven, only five of them diverged in this run. The one
**missing** item was written explicitly in the request and never applied; an item skipped in a
prose list leaves no trace anywhere. The three **extra** items never appear in the request at all:
an access rule, an extra disk, and a DNS record. They come from the routine of the person setting
up the environment and are written in no recipe — they exist once setup is done, but why they
exist lives only in one person's memory.

The direction of the difference is meaningful too. In four of the five diverged items, the hand's
value is **smaller** than the record's — two collectors instead of three, twenty-four hours instead
of seventy-two, seven days instead of fourteen, ninety days instead of three hundred sixty-five.
When a request without a criterion is read, the value chosen answers the load visible at that
moment; the value the record carries is the one decided once. The difference between the two
setups is therefore not a list of errors, it is a list of **two separate moments of decision**.

The last line is the second half of this lesson's measurement axis. The five items the record does
not cover are **the same in both setups**, because the provider gives rise to both: the egress
path, the management-port rule, the encryption key, the maintenance window, the in-account access
rule. Because the record says nothing about them, a comparison never shows them as a difference.
Writing the record does not zero out the difference; it makes the difference visible **only as far
as the record's scope reaches**.

#### Who, When, What Changed

```js
// track.mjs — how traceable a quarter's changes are in the two environments. Model.
import { record } from "./setup.mjs";

// Fourteen changes in a quarter; written in advance, no randomness.
const CHANGES = [["machine.collector.count", "record"], ["machine.collector.size", "record"],
  ["queue.reading-queue.partitions", "record"], ["data_store.measurement-store.size_gb", "record"],
  ["machine.collector.access_rule", "hand"], ["object_store.archive.lifecycle_days", "record"],
  ["network.measurement-network.block", "record"], ["machine.verifier.count", "record"],
  ["queue.reading-queue.retention_hours", "hand"], ["data_store.measurement-store.backup_days", "record"],
  ["machine.verifier.image", "record"], ["object_store.archive.class", "record"],
  ["network.measurement-network.egress_path", "hand"], ["machine.collector.disk_gb", "record"]];
const ticketFiled = [0, 3, 7, 10];                    // the four that get a ticket opened in the manual environment

// A change routed through the record answers all three questions; a work ticket answers two (it does not record what changed).
const answered = { record: 3, ticket: 2, untracked: 0 };
const A = CHANGES.map(([, y]) => (y === "record" ? "record" : "untracked"));
const B = CHANGES.map((_, i) => (ticketFiled.includes(i) ? "ticket" : "untracked"));
const sumAnswers = (x) => x.reduce((t, s) => t + answered[s], 0);
const percent = (n) => (100 * n / (CHANGES.length * 3)).toFixed(1);

console.log(`changes ${CHANGES.length}   questions ${CHANGES.length * 3} (who / when / what changed)`);
for (const [name, x] of [["environment set up from the record", A], ["manually set up environment", B]])
  console.log(`  ${name.padEnd(35)}: answered ${sumAnswers(x)}/${CHANGES.length * 3}` +
    ` (${percent(sumAnswers(x))}%)   untracked changes ${x.filter((s) => s === "untracked").length}`);

const handOnly = CHANGES.filter(([, y]) => y === "hand").map(([k]) => k);
console.log(`${handOnly.length} changes that skip the record in the record-built environment:`);
for (const k of handOnly)
  console.log(`  ${k.padEnd(41)} ${record.has(k) ? "in the record  -> shows up as drift"
    : "not in the record  -> shows up in no comparison"}`);
```

```
changes 14   questions 42 (who / when / what changed)
  environment set up from the record : answered 33/42 (78.6%)   untracked changes 3
  manually set up environment        : answered 8/42 (19.0%)   untracked changes 10
3 changes that skip the record in the record-built environment:
  machine.collector.access_rule             not in the record  -> shows up in no comparison
  queue.reading-queue.retention_hours       in the record  -> shows up as drift
  network.measurement-network.egress_path   not in the record  -> shows up in no comparison
```

Thirty-three of the forty-two questions are answered on the record route, eight on the manual
route. The difference is not one of discipline, it is one of **where things are kept**: on the
record route a change lives in a file and the file is versioned, so all three questions are read
from the same place. On the work-ticket route two questions are answered — who opened it, when it
closed — but the third, **what changed**, is left to prose, and prose carries no measurable value.

The real information is in the last three lines. Even in the environment set up from the record,
three changes skipped the record, and the three produced three different outcomes. When an item
written in the record is changed by hand, a comparison shows it as **drift**; when an item not
written in the record is changed by hand, it shows up in no comparison at all. Traceability comes
not from the record's existence but from its **scope**: for an item the record never touches, there
is no difference at all between the record route and the manual route.

#### The Record's Cost

```bash
#!/usr/bin/env bash
# The cost of tying to a record: lines written and extra steps per change.
printf 'lines written   request %s   record %s   parser and applier %s (once)\n' \
  "$(wc -l < request.txt | tr -d ' ')" "$(wc -l < record.infra | tr -d ' ')" \
  "$(wc -l < setup.mjs | tr -d ' ')"
printf 'record has %s item lines, %s items  -> items per line %s\n' \
  "$(grep -c '^resource' record.infra)" "20" "$(node -e 'console.log((20/6).toFixed(2))')"
printf 'extra step per change   hand route 0   record route 3 (write, get reviewed, read the plan)\n'
printf 'over a quarter (14 changes)  hand route 0 extra steps   record route %s extra steps\n' "$((14 * 3))"
```

```
lines written   request 12   record 8   parser and applier 38 (once)
record has 6 item lines, 20 items  -> items per line 3.33
extra step per change   hand route 0   record route 3 (write, get reviewed, read the plan)
over a quarter (14 changes)  hand route 0 extra steps   record route 42 extra steps
```

The record itself is shorter than the request: eight lines, six of them carrying items, 3.33 items
per line. The cost is not there. The thirty-eight-line parser and applier is written once and
reused for every environment — that is a one-time cost. The cost paid continuously is in the last
line: on the record route, every change asks for three extra steps, and over a quarter that comes
to forty-two extra steps. On the manual route these steps are zero; a person walks into the
environment and changes the value.

What is bought for the forty-two extra steps is countable too: twenty-five more questions answered
and twenty items made comparable. But what is not bought is just as clear. Five items stayed
outside the record, and the forty-two extra steps do not touch them; three changes skipped the
record, and the extra steps did not stop them either. A record does not produce an obligation, it
produces **common ground for comparison** — who uses that ground is a separate question, and the
rest of this course largely measures it.

#### Summary

- The two setups of the same environment diverged on 9 items: 5 diverged (5 of the 7 points the
  request gave no number for), 1 missing (written, skipped), 3 extra (written in no recipe).
- Two of the request's seven interpretation points did not show up because the hand's
  interpretation happened to land on the same value as the record; the interpretation-point count
  did not change, only five of them diverged in this run.
- The 5 items the record does not cover were born the same in both setups; a comparison never
  shows them as a difference.
- 33 of a quarter's 42 questions were answered on the record route, 8 on the manual route; the
  difference comes not from discipline but from where the change is kept.
- The record is 8 lines, the parser and applier 38 lines (once); the continuous cost is 3 extra
  steps per change, 42 extra steps over a quarter.

#### Next Step

In this lesson, the record was a list of items: each line counted a resource's attributes and
wrote what the environment **should be**. This is not the only form a record can take. The same
environment can also be written as a list of steps — first open the network, then open the
machines, then attach the queue — and that list also lives in a file, is versioned, is reviewed.
Both forms are records, and both ask for the same three extra steps.

The difference does not show up on the first run. The next lesson writes the same target in both
forms and runs both through **partial failure**: where the step list stops when the third step
errors, what the desired-state definition does; how much work each produces when rerun; and how
many lines each requires when the target changes.

### Declarative and Imperative Approaches

Source: https://academia.sh/en/courses/infrastructure-as-code/declarative-and-imperative-approaches

The same nine-item target is written in two forms — a four-line desired-state definition and a nine-line step list — and run so that the third step fails. The items that reach the target after partial failure, the no-op writes and errors produced on a rerun, and the lines written when the target changes are counted.

The previous lesson wrote the record as a list of items: each line said what a resource **should
be**. The same environment can also be written as a list of steps — first open the network, then
open the store, then attach the machine — and that list also lives in a file, is versioned, is
reviewed. Both forms are records, and both ask for the same extra steps.

The **declarative** recipe writes the desired state; the **imperative** recipe writes the steps
that lead there. The `Infrastructure Provisioning Models` lesson in the **DevOps Culture and
Fundamentals** course (M22/K01) compared manual, scripted, and declarative provisioning, and showed
why drift can only be counted in the declarative model; that comparison is not repeated here. This
lesson carries the measurement to a different axis: **partial failure**. The same target is written
in two forms, run so that the third step fails, and three things are counted — the items that reach
the target after the failure, the work produced when the same recipe is rerun, and the lines written
when the target changes.

The environment is again the **regional measurement network**: **fictional** software that collects
meter readings, verifies them, and converts them into invoices. Infrastructure is not provisioned;
the environment, the operations, and the quota are modeled with `node`. The recipe files are
genuinely written to disk.

**IF7.** The target is four resources and nine items; **the resource itself is an item too**, and
so are its attributes.
**IF8.** The imperative recipe is a nine-line step list, the declarative recipe a four-line
definition; both target the same nine items.
**IF9.** `create` errors on an existing resource; `set` written with the same value counts as a
**no-op write**. Whether idempotence holds is decided by these two rules.
**IF10.** On the first run the data store quota is zero and the third step fails; the quota is
raised to one before the second run. The error is written in advance, there is no randomness.
**IF11.** The declarative applier applies items in dependency order; an item whose dependency is
not met is **skipped**, not counted as an error.
**IF12.** The imperative runner has two modes: stop on the first error, or skip the error and
continue.

#### Same Target, Two Forms

```bash
#!/usr/bin/env bash
# Two forms of the same target: a declarative definition and an imperative step list. Second versions are written too.
set -e

cat > target.infra <<'SON'
resource network "measurement-network" { block = "10.40.0.0/16" }
resource data_store "measurement-store" { size_gb = 200 }
resource machine "collector" { count = 3  attached = "data_store.measurement-store" }
resource queue "reading-queue" { partitions = 6 }
SON

cat > steps.txt <<'SON'
create network.measurement-network
set network.measurement-network.block 10.40.0.0/16
create data_store.measurement-store
set data_store.measurement-store.size_gb 200
create machine.collector
set machine.collector.count 3
create queue.reading-queue
set queue.reading-queue.partitions 6
attach machine.collector data_store.measurement-store
SON

# The target changes: collector 3 -> 5, queue partitions 6 -> 12, an object store is added.
cat > target-v2.infra <<'SON'
resource network "measurement-network" { block = "10.40.0.0/16" }
resource data_store "measurement-store" { size_gb = 200 }
resource machine "collector" { count = 5  attached = "data_store.measurement-store" }
resource queue "reading-queue" { partitions = 12 }
resource object_store "archive" { class = "sparse" }
SON

cat > steps-v2.txt <<'SON'
skip-if-exists create network.measurement-network
set network.measurement-network.block 10.40.0.0/16
skip-if-exists create data_store.measurement-store
set data_store.measurement-store.size_gb 200
skip-if-exists create machine.collector
set machine.collector.count 5
skip-if-exists create queue.reading-queue
set queue.reading-queue.partitions 12
skip-if-exists attach machine.collector data_store.measurement-store
skip-if-exists create object_store.archive
set object_store.archive.class sparse
SON
```

The declarative definition is four lines, the step list nine, and both carry the same nine items.
The real difference is not size: in the definition, `attached` is an attribute, and it says the
machine **is attached** to the data store; in the list, `attach` is a step at the very end, and it
says **when** the attaching happens. The first declares a dependency, the second declares an order.

```js
// engine.mjs — environment model and two runners; both are models, no real infrastructure is provisioned.
import { readFileSync } from "node:fs";

export const newWorld = (quota) => ({ resource: new Set(), item: new Map(), quota,
  attempted: 0, applied: 0, noop: 0, error: 0, skipped: 0 });
export const copy = (d) => ({ ...d, resource: new Set(d.resource), item: new Map(d.item),
  attempted: 0, applied: 0, noop: 0, error: 0, skipped: 0 });

export const op = (d, name, target, value) => {           // create / set / attach
  d.attempted++;
  if (name === "create") {
    if (d.resource.has(target)) return { ok: false, reason: "already exists" };
    const kind = target.split(".")[0];
    const n = [...d.resource].filter((k) => k.startsWith(kind + ".")).length;
    if (n >= (d.quota[kind] ?? 9)) return { ok: false, reason: kind + " quota full" };
    d.resource.add(target); d.applied++; return { ok: true };
  }
  const key = name === "attach" ? target + ".attached" : target;
  const owner = key.split(".").slice(0, 2).join(".");
  if (!d.resource.has(owner)) return { ok: false, reason: "resource missing" };
  if (name === "attach" && !d.resource.has(value)) return { ok: false, reason: "target resource missing" };
  if (d.item.get(key) === value) { d.noop++; return { ok: true, noop: true }; }
  d.item.set(key, value); d.applied++; return { ok: true };
};

// Imperative: runs the step list in order. mode "stop" stops on the first error, "skip" skips the error.
export const imperative = (d, lines, mode) => {
  const log = [];
  for (const [i, raw] of lines.entries()) {
    const guarded = raw.startsWith("skip-if-exists ");
    const [name, target, value] = raw.replace("skip-if-exists ", "").split(" ");
    if (guarded && name === "create" && d.resource.has(target)) { d.skipped++; continue; }
    if (guarded && name === "attach" && d.item.get(target + ".attached") === value) { d.skipped++; continue; }
    const r = op(d, name, target, value);
    if (!r.ok) { d.error++; log.push(`step ${i + 1} (${name} ${target}): ${r.reason}`);
      if (mode === "stop") return { log, stoppedAt: i + 1, notAttempted: lines.length - i - 1 }; }
  }
  return { log, stoppedAt: 0, notAttempted: 0 };
};

const BLOCK = /^resource\s+(\w+)\s+"([\w-]+)"\s*\{(.*)\}$/;   // same as 01's configuration language
export const readTarget = (path) => {
  const k = new Map();
  for (const s of readFileSync(path, "utf8").split("\n")) {
    const m = BLOCK.exec(s.trim());
    if (!m) continue;
    k.set(`${m[1]}.${m[2]}`, "exists");                       // the resource itself is an item too
    for (const x of m[3].matchAll(/(\w+)\s*=\s*("[^"]*"|[\w./-]+)/g))
      k.set(`${m[1]}.${m[2]}.${x[1]}`, x[2].replaceAll('"', ""));
  }
  return k;
};

// Declarative: the diff is computed, an item whose dependency is unmet is skipped, order goes by depth.
export const declarative = (d, target) => {
  const log = [];
  const diff = [...target].filter(([k, v]) => (k.split(".").length === 2
    ? !d.resource.has(k) : d.item.get(k) !== v));
  for (const [k, v] of diff.sort((a, b) => a[0].split(".").length - b[0].split(".").length)) {
    const needs = [k.split(".").length > 2 ? k.split(".").slice(0, 2).join(".") : null,
      k.endsWith(".attached") ? v : null].filter(Boolean);
    if (needs.some((g) => !d.resource.has(g))) { d.skipped++; log.push(`${k}: dependency missing`); continue; }
    const [name, targetName] = k.split(".").length === 2 ? ["create", k]
      : k.endsWith(".attached") ? ["attach", k.split(".").slice(0, 2).join(".")] : ["set", k];
    const r = op(d, name, targetName, v);
    if (!r.ok) { d.error++; log.push(`${k}: ${r.reason}`); }
  }
  return { log, diff: diff.length };
};

export const report = (name, d, extra = "") =>
  `  ${name.padEnd(35)} attempted ${d.attempted}  applied ${d.applied}  no-op writes ${d.noop}` +
  `  errors ${d.error}  skipped ${d.skipped}${extra}`;
```

#### An Error at the Third Step

```js
// scenario.mjs — the same target in two forms: partial failure, a rerun, a third run.
import { readFileSync } from "node:fs";
import { newWorld, copy, imperative, declarative, readTarget, report } from "./engine.mjs";

const lines = (p) => readFileSync(p, "utf8").trim().split("\n");
const steps = lines("steps.txt"), target = readTarget("target.infra");
const narrow = { data_store: 0 }, wide = { data_store: 1 };   // third step fails on the narrow quota

console.log(`target ${target.size} items   step list ${steps.length} lines\n`);
console.log("A. partial failure (data store quota 0)");
const bA = newWorld(narrow), sA = imperative(bA, steps, "stop");
console.log(report("imperative (stops on first error)", bA,
  `  not attempted ${sA.notAttempted}`));
console.log(`     ${sA.log.join("; ")}   record of where it stopped: 0 files`);
const dA = newWorld(narrow), tA = declarative(dA, target);
console.log(report("declarative (applies the diff)", dA, `  diff ${tA.diff}`));
console.log(`     ${tA.log.join("; ")}`);
console.log(`  items reaching the target   imperative ${bA.resource.size + bA.item.size}/${target.size}` +
  `   declarative ${dA.resource.size + dA.item.size}/${target.size}\n`);

console.log("B. second run (quota raised, recipe unchanged)");
const bB = copy({ ...bA, quota: wide }), sB = imperative(bB, steps, "stop");
console.log(report("imperative (same list from scratch)", bB, `  not attempted ${sB.notAttempted}`));
console.log(`     ${sB.log.join("; ")}`);
const shortened = steps.filter((a) => a !== "create network.measurement-network");   // 1 line removed by hand
const bB2 = copy({ ...bA, quota: wide });
imperative(bB2, shortened, "stop");
console.log(report("imperative (1 line removed)", bB2, `  hand edit 1 line`));
const dB = copy({ ...dA, quota: wide }), tB = declarative(dB, target);
console.log(report("declarative (same recipe)", dB, `  diff ${tB.diff}, hand edit 0 lines`));
console.log(`  items reaching the target   imperative ${bB2.resource.size + bB2.item.size}/${target.size}` +
  `   declarative ${dB.resource.size + dB.item.size}/${target.size}\n`);

console.log("C. third run (target reached, recipe rerun)");
const bC = copy({ ...bB2, quota: wide });
imperative(bC, steps, "skip");
console.log(report("imperative (skipping the error)", bC));
const dC = copy({ ...dB, quota: wide }), tC = declarative(dC, target);
console.log(report("declarative", dC, `  diff ${tC.diff}`));

console.log("\nD. target changed (collector 3 -> 5, queue partitions 6 -> 12, object store added)");
const target2 = readTarget("target-v2.infra"), steps2 = lines("steps-v2.txt");
const bD = copy({ ...bC, quota: wide });
imperative(bD, steps2, "skip");
console.log(report("imperative v2 (guarded list)", bD, `  list ${steps2.length} lines`));
const dD = copy({ ...dC, quota: wide }), tD = declarative(dD, target2);
console.log(report("declarative v2", dD, `  diff ${tD.diff}, definition ${target2.size} items`));
```

```
target 9 items   step list 9 lines

A. partial failure (data store quota 0)
  imperative (stops on first error)   attempted 3  applied 2  no-op writes 0  errors 1  skipped 0  not attempted 6
     step 3 (create data_store.measurement-store): data_store quota full   record of where it stopped: 0 files
  declarative (applies the diff)      attempted 7  applied 6  no-op writes 0  errors 1  skipped 2  diff 9
     data_store.measurement-store: data_store quota full; data_store.measurement-store.size_gb: dependency missing; machine.collector.attached: dependency missing
  items reaching the target   imperative 2/9   declarative 6/9

B. second run (quota raised, recipe unchanged)
  imperative (same list from scratch) attempted 1  applied 0  no-op writes 0  errors 1  skipped 0  not attempted 8
     step 1 (create network.measurement-network): already exists
  imperative (1 line removed)         attempted 8  applied 7  no-op writes 1  errors 0  skipped 0  hand edit 1 line
  declarative (same recipe)           attempted 3  applied 3  no-op writes 0  errors 0  skipped 0  diff 3, hand edit 0 lines
  items reaching the target   imperative 9/9   declarative 9/9

C. third run (target reached, recipe rerun)
  imperative (skipping the error)     attempted 9  applied 0  no-op writes 5  errors 4  skipped 0
  declarative                         attempted 0  applied 0  no-op writes 0  errors 0  skipped 0  diff 0

D. target changed (collector 3 -> 5, queue partitions 6 -> 12, object store added)
  imperative v2 (guarded list)        attempted 6  applied 4  no-op writes 2  errors 0  skipped 5  list 11 lines
  declarative v2                      attempted 4  applied 4  no-op writes 0  errors 0  skipped 0  diff 4, definition 11 items
```

The error at the third step produced a very different cost in the two forms. When the imperative list
stopped, **two items** were in place and six steps had never been attempted — yet four of those six steps
have nothing to do with the data store. Because the list declares an **order**, an error in the sequence
stops everything after it too; the machine and the queue were left unopened even though the quota never
touched them. The declarative applier hit the same error and put **six items** in place; it only skipped
the two items that actually depend on the data store. The four-item difference between the two comes not
from the applier's speed but from what the recipe declares: if a dependency is declared, the error stops
only its own subtree; if an order is declared, the error stops the whole rest of the queue.

The **readability** of the state left behind also differs. The only information the imperative run
leaves behind is the step number it stopped at, and checking which items of the environment are in
place takes a separate look; there is no record of where things were left off. The declarative run,
on the other hand, recomputes the diff on the next run — the question "where were things left off"
is never asked, because the answer is produced from scratch on every run.

#### Work Produced on a Rerun

The second run shows the same divide from a different angle. After the quota is raised, the
imperative list, **run again from the start, fails at the first step**: the network already exists
and `create` errors on an existing resource. Eight steps are never attempted. Advancing the list
requires removing a line by hand; which line gets removed depends on the environment's current
state, and a different environment would need a different line removed. The declarative recipe, on
the other hand, runs **completely unedited**, finds a diff of three items, and applies all three.

The third run counts idempotence directly. In an environment that has already reached the target,
when the imperative list is run while skipping errors, four of the nine steps error and five
produce a **no-op write**: the same value is written to the same item again, the environment does
not change, but all nine operations are still attempted. The declarative recipe finds zero diff in
the same environment and **attempts no operation at all**. The difference between the two is not an
efficiency detail: four of the nine attempts show up as errors, and picking out the next real error
from among these four takes separate work.

#### Lines Written When the Target Changes

```bash
#!/usr/bin/env bash
# Lines written when the target changes: the first and second versions of both recipes are compared.
printf 'recipe size (lines)   declarative %s -> %s   imperative %s -> %s\n' \
  "$(wc -l < target.infra | tr -d ' ')" "$(wc -l < target-v2.infra | tr -d ' ')" \
  "$(wc -l < steps.txt | tr -d ' ')" "$(wc -l < steps-v2.txt | tr -d ' ')"
printf 'lines written for the target change   declarative %s   imperative %s\n' \
  "$(diff target.infra target-v2.infra | grep -c '^>')" \
  "$(diff steps.txt steps-v2.txt | grep -c '^>')"
printf 'of these, %s lines add a guard, %s lines change a value, %s lines are a new resource\n' \
  "$(diff steps.txt steps-v2.txt | grep '^> skip-if-exists' | grep -vc archive)" \
  "$(diff steps.txt steps-v2.txt | grep '^> set' | grep -vc archive)" \
  "$(diff steps.txt steps-v2.txt | grep '^>' | grep -c archive)"
```

```
recipe size (lines)   declarative 4 -> 5   imperative 9 -> 11
lines written for the target change   declarative 3   imperative 9
of these, 5 lines add a guard, 2 lines change a value, 2 lines are a new resource
```

For the same target change, three lines were written in the declarative definition, nine in the
step list. Of the nine, only four concern the target itself — two value changes and two lines for
the new resource. The remaining **five lines are a guard**: "skip if exists" is written in front of
every `create` step, or the list could not run a second time. These five lines do their job in
section D: the run finishes with zero errors, five steps are skipped, four items are applied. The
cost is two no-op writes — the guard reads whether the resource exists, not the value of the
attribute.

What is visible here is a transformation. The way to make the imperative list rerunnable is to put
a **state read** in front of every step; once these reads are complete, the list stops saying "what
to do" and starts saying "what should be." The boundary between the two forms dissolves at this
point, and the only difference left is whether those reads are written into the recipe by hand or
left to the applier. The gain of the declarative form is not a superiority, it is a
**non-repetition**: the same read is not written nine times across nine lines, it is written once
in the applier.

#### Summary

- After the error at the third step, the items reaching the target were 2/9 in the imperative form
  and 6/9 in the declarative form; 5 of the 6 steps never attempted in the imperative run had
  nothing to do with the error.
- The imperative run left behind no record beyond the step number; the declarative run recomputes
  the diff from scratch on the next run, so it never asks "where were things left off."
- On the second run the imperative list failed at step 1, and advancing it required removing 1 line
  by hand; the declarative recipe ran with 0 lines edited and applied 3 items.
- In an environment that had already reached the target, the imperative list attempted 9 operations
  (4 errors, 5 no-op writes); the declarative recipe attempted 0 operations.
- When the target changed, 3 lines were written in the declarative definition, 9 in the step list;
  5 of the 9 were only guard lines making the list rerunnable.

#### Next Step

Up to this point, both recipes did the same thing: they changed an existing resource **in place**.
When the queue's partition count went from six to twelve, it was still the same queue; when the
machine count went from three to five, the existing three machines were untouched. This is not the
only way to change infrastructure.

The next lesson makes the same update two ways — an in-place patch and a rebuild — and counts four
things: how many machines diverge from one another by the end of a period, the update duration, the
rollback round count, and what a rebuild **cannot carry across**. The case where a rebuild gains
nothing is shown with the same measure too.

### Immutable Infrastructure

Source: https://academia.sh/en/courses/infrastructure-as-code/immutable-infrastructure

A twelve-machine fleet is updated two ways over eight periods: an in-place patch and a rebuild from an image. The same event sequence is replayed on both paths; the number of servers that diverge from one another, the update duration, the rollback round count, the local state and connections a rebuild cannot carry across, and the case where a rebuild gains nothing are counted.

The previous two lessons changed infrastructure in place: the queue's partition count grew, the
machine count grew, existing resources were untouched. A machine's package version advances the
same way — the machine stays the same machine, the software on it changes. This is an **in-place
patch**, and every application of it adds a layer to the machine's history.

The second way never touches the machine at all: a new image is prepared, the old machine is shut
down, and a new machine is born from the image in its place. This is **immutable infrastructure** —
a running resource is not changed, a new one replaces it. This lesson runs the same update two ways
over eight periods and counts four things: the number of **diverged servers**, the update duration,
the rollback round count, and what a rebuild **cannot carry across**. At the end, the case where a
rebuild gains nothing is shown with the same measure too.

The fleet is the collector machines of the **regional measurement network**: **fictional** software
that takes meter readings from the field and writes them to the queue. Machines are not really
provisioned; the fleet, the events, and the durations are modeled with `node`, and the generator is
self-written.

**IF13.** The fleet is twelve machines, updated over eight periods. A machine's **fingerprint**
consists of its version and the traces it carries; if two machines carry the same fingerprint, they
have not diverged.
**IF14.** Events are generated once (seed 8140723, the generator is self-written) and replayed on
both paths **in the same order**; the difference comes from the path, not from the events.
**IF15.** In a given period, the probability of a patch failing per machine is 0.10; the probability
of a hotfix being applied is 0.10.
**IF16.** On the rebuild path, a machine is born from the current image every period. A hotfix
applied within a period is baked into the image on even periods, and **lost** on odd periods.
**IF17.** Durations: an in-place patch is 40 seconds, a rebuild is 360 seconds plus 90 seconds of
draining; a wave is four machines. A failed patch is retried twice more, a failed health check takes
one more round.
**IF18.** Rollback happens in periods 3 and 6. A reverse patch fails on a machine carrying traces,
and half of what remains is fixed on each round.

#### The Fleet and Eight Periods

```js
// fleet.mjs — eight periods of a twelve-machine collector fleet. Model; the generator is self-written.
export const SEED = 8140723, MACHINES = 12, PERIODS = 8;
const generator = (t) => () => (t = (t * 1103515245 + 12345) % 2147483648) / 2147483648;

// Events are generated once and replayed in the same order on both paths; the difference comes from the path.
const r = generator(SEED);
export const events = [];
for (let d = 1; d <= PERIODS; d++)
  for (let m = 0; m < MACHINES; m++)
    events.push({ d, m, patchFails: r() < 0.10, hotfix: r() < 0.10 });
const periodOf = (d) => events.filter((o) => o.d === d);
const fingerprint = (x) => `${x.version}|${[...x.traces].sort().join(",")}`;
export const diverged = (s) => new Set(s.map(fingerprint)).size;
const emptyFleet = () => Array.from({ length: MACHINES }, () => ({ version: "1.0.0", traces: new Set() }));

export const pathInPlace = () => {                        // in-place patch: traces accumulate, never wiped
  const s = emptyFleet(), series = [];
  let failed = 0, hotfix = 0;
  for (let d = 1; d <= PERIODS; d++) {
    for (const o of periodOf(d)) {
      if (o.patchFails) { s[o.m].traces.add(`half-patch-${o.d}`); failed++; }
      else s[o.m].version = `1.${d}.0`;
      if (o.hotfix) { s[o.m].traces.add(`hotfix-${d}`); hotfix++; }
    }
    series.push(diverged(s));
  }
  return { s, series, failed, hotfix, lost: 0 };
};

export const pathRebuild = () => {                        // rebuild: from the image, every period
  const s = emptyFleet(), series = [], imageTraces = new Set();
  let failed = 0, hotfix = 0, lost = 0;
  for (let d = 1; d <= PERIODS; d++) {
    for (let m = 0; m < MACHINES; m++) s[m] = { version: `1.${d}.0`, traces: new Set(imageTraces) };
    const toBake = [];
    for (const o of periodOf(d)) {
      if (o.patchFails) failed++;                          // health check fails: one more round
      if (o.hotfix) { hotfix++; s[o.m].traces.add(`hotfix-${d}`);  // applied to a single machine within the period
        if (d % 2 === 0) toBake.push(`hotfix-${d}`); else lost++; }
    }
    series.push(diverged(s));
    for (const x of toBake) imageTraces.add(x);            // the fix is baked into the image on even periods
  }
  return { s, series, failed, hotfix, lost };
};
```

#### Measurement

```js
// measure.mjs — divergence, update duration, rollback rounds, and what a rebuild cannot carry across.
import { SEED, MACHINES, PERIODS, events, diverged, pathInPlace, pathRebuild } from "./fleet.mjs";

const A = pathInPlace(), B = pathRebuild();
console.log(`seed ${SEED}   machines ${MACHINES}   periods ${PERIODS}   events ${events.length}`);
console.log(`diverged servers, period by period (distinct fingerprint)`);
console.log(`  in-place patch     ${A.series.join("  ")}   -> final ${diverged(A.s)}/${MACHINES}`);
console.log(`  rebuild            ${B.series.join("  ")}   -> final ${diverged(B.s)}/${MACHINES}`);
console.log(`  failed patch/health ${A.failed}   hotfixes applied ${A.hotfix}` +
  `   lost, never baked into image ${B.lost}\n`);

// Durations are in seconds; a wave is 4 machines. In-place patch 40 s, rebuild 360 s plus 90 s drain.
const WAVE = Math.ceil(MACHINES / 4), PATCH = 40, BUILD = 360, DRAIN = 90;
const periodFailed = (d) => events.filter((o) => o.d === d && o.patchFails).length;
let sA = 0, sB = 0;
for (let d = 1; d <= PERIODS; d++) {
  sA += WAVE * PATCH + periodFailed(d) * 2 * PATCH;          // a failed patch is retried twice more
  sB += WAVE * (DRAIN + BUILD) + periodFailed(d) * BUILD;    // a failed health check adds one more round
}
console.log(`update duration (${PERIODS}-period total, seconds)`);
console.log(`  in-place patch     ${sA}   period average ${(sA / PERIODS).toFixed(0)}`);
console.log(`  rebuild            ${sB}   period average ${(sB / PERIODS).toFixed(0)}` +
  `   ratio ${(sB / sA).toFixed(1)}x\n`);

// Rollback: the 3rd and 6th period's update is rolled back. A reverse patch fails on a machine carrying traces.
const rollback = (s) => {
  const traced = s.filter((x) => x.traces.size).length;
  let round = 1, remaining = traced;
  while (remaining > 0 && round < 3) { remaining = Math.floor(remaining / 2); round++; }
  return { round: remaining > 0 ? round + 1 : round, traced, manual: remaining };
};
const g = rollback(A.s);
console.log(`rollback (period 3 and period 6, twice)`);
console.log(`  in-place patch     round ${g.round} x 2 = ${g.round * 2}   machines the reverse patch fails on` +
  ` ${g.traced}   left for manual repair ${g.manual}`);
console.log(`  rebuild            round 1 x 2 = 2   the prior image tag is selected, fleet returns in` +
  ` ${WAVE * (DRAIN + BUILD)} seconds\n`);

// What a rebuild cannot carry across: local state, open connections, a warm cache.
const BUFFER = 1800, PROCESS_RATE = 20, CONNECTIONS = 40, WARMUP = 720, READS = 15, FACTOR = 3.4;
console.log(`what a rebuild cannot carry across (per period, ${MACHINES} machines)`);
console.log(`  queue buffer       reads dropped if not drained ${MACHINES * BUFFER}` +
  `   extra time if drained ${WAVE * DRAIN} s (buffer empties in ${BUFFER / PROCESS_RATE} s)`);
console.log(`  open connections   broken ${MACHINES * CONNECTIONS}   0 on in-place patch`);
console.log(`  warm cache         warmup ${WARMUP} s, data-store reads ${FACTOR}x` +
  `   extra reads ${MACHINES * WARMUP * READS * (FACTOR - 1)}\n`);

// What a rebuild does not gain: a single machine carrying state.
const DATA_GB = 400, SPEED_MBS = 180;
const transfer = Math.round(DATA_GB * 1024 / SPEED_MBS);
console.log(`a single state-carrying machine (measurement store, ${DATA_GB} GB local data)`);
console.log(`  in-place patch ${PATCH} s   rebuild ${transfer} s` +
  ` (${(transfer / 60).toFixed(1)} min, data copy at ${SPEED_MBS} MB/s)`);
console.log(`  diverged servers 1/1 -> 1/1   gain 0   cost ${(transfer / PATCH).toFixed(0)}x duration`);
```

```
seed 8140723   machines 12   periods 8   events 96
diverged servers, period by period (distinct fingerprint)
  in-place patch     3  6  6  6  7  7  8  10   -> final 10/12
  rebuild            2  2  1  1  1  2  2  2   -> final 2/12
  failed patch/health 7   hotfixes applied 9   lost, never baked into image 3

update duration (8-period total, seconds)
  in-place patch     1520   period average 190
  rebuild            13320   period average 1665   ratio 8.8x

rollback (period 3 and period 6, twice)
  in-place patch     round 4 x 2 = 8   machines the reverse patch fails on 10   left for manual repair 2
  rebuild            round 1 x 2 = 2   the prior image tag is selected, fleet returns in 1350 seconds

what a rebuild cannot carry across (per period, 12 machines)
  queue buffer       reads dropped if not drained 21600   extra time if drained 270 s (buffer empties in 90 s)
  open connections   broken 480   0 on in-place patch
  warm cache         warmup 720 s, data-store reads 3.4x   extra reads 311040

a single state-carrying machine (measurement store, 400 GB local data)
  in-place patch 40 s   rebuild 2276 s (37.9 min, data copy at 180 MB/s)
  diverged servers 1/1 -> 1/1   gain 0   cost 57x duration
```

#### Diverged Servers and Rollback

The same ninety-six events were replayed on both paths, and by the end of eight periods, ten of the
twelve machines on the in-place patch path had become **unique**: ten distinct fingerprints, meaning
groups of 1.2 machines on average. The series shows accumulation — three to ten, with no reversal.
On the rebuild path, the series oscillates between one and two and never accumulates. The difference
is not one of care: on both paths seven patches failed and nine hotfixes were applied. On the
in-place patch path, their trace **stays** on the machine; on the rebuild path, the trace is
**wiped** because the machine is reborn from scratch the next period.

Wiping has a cost, and it too is counted: three of the nine hotfixes were never written back into
the image and were lost. A rebuild makes writing the fix back into the image **mandatory**; if it is
not written back, the fix disappears the next period and the problem it fixed comes back. On the
in-place patch path there is no such requirement — the fix stays. This is exactly what produces
divergence: persistence.

The cost of divergence does not show up in the series itself; it shows up in a later operation. Ten
unique fingerprints mean the next change gets tried against ten separate starting states. A patch
working on one machine and not on another is no longer a possibility — it is the accumulated result
of eight periods. On the rebuild path, the same change gets tried against at most two starting
states, and one of the two is a single hotfix applied within the period — it too disappears once the
period closes.

The duration table runs the other way. Rebuild takes 13,320 seconds over eight periods, in-place
patching 1,520 seconds — **8.8 times as long**. The gap between 190 seconds and 1,665 seconds per
period is large enough to determine the size of a maintenance window. The sign flips for rollback:
in-place patching spent a total of eight rounds across two rollbacks, because the reverse patch
failed on the ten machines carrying traces and two machines ended up needing manual repair. Rebuild
finished in two rounds — the prior image tag is selected, and the fleet returns to that tag in 1,350
seconds. Rollback's **round count** and **duration** come apart here: a rebuild's round count is
fixed and predictable, an in-place patch's depends on the trace the environment has accumulated and
cannot be known in advance.

#### What a Rebuild Cannot Carry Across

The third table shows that a rebuild's cost is not just duration. When a machine is shut down, it
carries three things, and none of them are part of the image. Reads waiting in the queue buffer:
without draining, 21,600 reads are dropped per period; with draining, none are dropped, but ninety
seconds are added to every wave. Open connections: four hundred eighty long-lived connections from
the field break and are re-established every period; on in-place patching this number is zero. Warm
cache: a newly born machine's cache is empty, data-store reads run at 3.4 times their rate through a
twelve-minute warmup, and 311,040 extra reads are born per period.

All three have a way to reduce them, and all three write their cost somewhere else. Draining saves
the buffer and lengthens the duration. Broken connections become lossless if the field unit waits to
reconnect and holds its reads locally — but that means adding a buffer to the field unit. A warming
cache is hidden by warming the machine up before it takes traffic; that too is extra time per wave.
The measure does not change, it only changes column.

What these three items have in common is that none of them is **written in the record**. The image
version, the package list, and the settings go into the record; buffer occupancy, connection count,
and cache temperature do not. A rebuild reproduces exactly what is written in the record, and
**zeroes out, every time,** what is not written in it. The mechanism that prevents divergence and
the mechanism that destroys local state are the same mechanism; one cannot be separated from the
other.

#### Where It Gains Nothing

The last table shows where a rebuild does not help. The measurement store is a single machine and
carries four hundred gigabytes of local data. Being a single machine, there is nothing to diverge:
the fingerprint count is one both before and after the rebuild. The gain is zero. The cost is the
copy duration — 2,276 seconds, **fifty-seven times** the in-place patch — and the whole of that
duration goes to moving data, not to updating software.

The same calculation gives a different result on the measurement network's other components.
Verifier machines carry no state and number four; there, a rebuild produces a gain close to what it
does on the collectors. The queue and the object store are already resources the provider operates,
and they carry no operating environment to patch; the question itself is never asked for these two
components. The decision comes not from seeing the fleet as one piece, but from separating it **role
by role**.

The criterion that follows fits in one sentence: a rebuild's gain is proportional to the number of
machines sharing a role and to the trace accumulated on those machines. On twelve collectors, the
gain falls from ten to one; on a single state-carrying machine, there is no gain. Immutable
infrastructure is not decided as a principle, it is decided **role by role**, and the decision's
input is these two numbers.

#### Summary

- When the same 96 events were replayed on both paths, in-place patching made 10 of 12 machines
  unique by the end of 8 periods (series 3 to 10); on rebuild, the series stayed between 1 and 2.
- Both paths saw 7 failed patches and 9 applied hotfixes; what produced the difference was the
  persistence of the trace.
- Rebuild lost 3 of the 9 hotfixes; writing a fix back into the image becomes mandatory.
- Update duration is 8.8 times longer on rebuild (1,665 versus 190 seconds per period), but rollback
  finished in 2 rounds instead of 8, and the round count is known in advance.
- The three items a rebuild cannot carry across are 21,600 dropped reads, 480 broken connections,
  and 311,040 extra data-store reads per period; on a single state-carrying machine, gain is 0, cost
  is 57 times the duration.

#### Next Step

In this lesson, two jobs got mixed together. Which packages the image carries, what the settings on
the machine are — that is one job; how many machines there are, which network they attach to, which
disk they carry — that is another. In-place patching touches the first, rebuild touches the second —
and the hotfix getting lost happened exactly on the boundary between these two jobs.

The next lesson draws that boundary. An environment's items are split into two layers — the layer
that creates the resource and the layer that configures the inside of the resource — and where the
boundary is drawn gets counted: how many items are defined in both layers, how many in neither, how
many items get applied twice when the boundary shifts, and which layer wins when both write to the
same item.

### Provisioning and Configuration Management

Source: https://academia.sh/en/courses/infrastructure-as-code/provisioning-and-configuration-management

An environment's thirty-four items are split into two layers — the provisioning layer that creates the resource and the configuration layer that sets up the inside of the resource — and where the boundary is drawn gets counted. Overlapping items, the gap that falls into neither layer, which layer wins on a conflicting value, the drift window after reprovisioning, and the cost of three separate boundary lines are measured.

The previous lesson mixed two jobs together. Which packages the image carries and what the settings
on the machine are is one job; how many machines there are, which network they attach to, and which
disk they carry is another. The hotfix getting lost happened exactly on the boundary between these
two jobs.

**Provisioning** creates the resource: it opens the machine, sets up the network, attaches the disk,
creates the queue. **Configuration management** sets up the inside of the resource: it installs
packages, writes files, starts the service. Both have a record, and the two records are written in
different formats. This lesson splits an environment's items into two layers and counts **where the
boundary is drawn**: how many items are defined in both layers, how many in neither, which layer
wins on a conflicting item, and how many items get applied twice when the boundary shifts.

The environment is again the staging environment of the **regional measurement network**, and it is
**fictional**. The record files are genuinely written to disk; the split into layers, the
application order, and the drift window are modeled with `node`.

**IF19.** The staging environment is represented by thirty-four items, and each item's **natural
layer** is written in the inventory: `resource` or `content`.
**IF20.** The two layers' records are two separate formats — the provisioning record in blocks of
the configuration language, the config record in the configuration agent's line format.
**IF21.** If an item is defined in only one layer, that layer applies it; if it is defined in both,
**whichever runs last** wins.
**IF22.** The usual order is provisioning then config. When a reprovisioning is triggered, the order
reverses: config has already run, provisioning runs after.
**IF23.** The configuration agent runs every thirty minutes; there are fourteen machines in the
environment.
**IF24.** A quarter sees twenty-two changes — seven on resource items, fifteen on content items; six
of the content changes are on items baked into the image.

#### The Inventory and the Two Records' Formats

```bash
#!/usr/bin/env bash
# An environment's item inventory and the two layers' records: provisioning (creates the resource), config (inside the resource).
set -e

cat > inventory.txt <<'SON'
network.measurement-network.block                  resource
network.measurement-network.subnet_count           resource
network.measurement-network.egress_path            resource
machine.collector.count                            resource
machine.collector.size                             resource
machine.collector.disk_gb                          resource
machine.collector.image                            resource
machine.verifier.count                             resource
machine.verifier.size                              resource
queue.reading.partitions                           resource
queue.reading.retention_hours                      resource
data_store.measurement.size_gb                     resource
data_store.measurement.backup_days                 resource
object_store.archive.class                         resource
object_store.archive.lifecycle_days                resource
security.collector.ingress                         resource
security.collector.egress                          resource
dns_record.collector.internal                      resource
machine.collector.package.resolver                 content
machine.collector.package.rule_package             content
machine.collector.config.log_level                 content
machine.collector.config.concurrent_workers        content
machine.collector.config.batch_hour                content
machine.collector.dir.data                         content
machine.collector.dir.log                          content
machine.collector.service.autostart                content
machine.collector.timezone                         content
machine.collector.log_rotation                     content
machine.verifier.package.resolver                  content
machine.verifier.config.log_level                  content
machine.verifier.service.autostart                 content
data_store.measurement.user.report                 content
data_store.measurement.parameter.connection_count  content
object_store.archive.access_policy                 content
SON

cat > provisioning.infra <<'SON'
resource network "measurement-network" { block = "10.40.0.0/16"  subnet_count = 3 }
resource machine "collector" { count = 3  size = "medium"  disk_gb = 40  image = "base-7" }
resource machine "collector" { config.log_level = "summary"  timezone = "UTC" }
resource machine "collector" { service.autostart = "open"  package.resolver = "3.4.0" }
resource machine "verifier" { count = 2  size = "small" }
resource queue "reading" { partitions = 6  retention_hours = 72 }
resource data_store "measurement" { size_gb = 200  backup_days = 14 }
resource object_store "archive" { class = "sparse"  lifecycle_days = 365  access_policy = "in-account" }
resource security "collector" { ingress = "10.40.0.0/16" }
SON

cat > config.record <<'SON'
machine/collector       package.resolver = 3.5.0
machine/collector       package.rule_package = 9.1.0
machine/collector       config.log_level = summary
machine/collector       config.concurrent_workers = 8
machine/collector       config.batch_hour = 02:30
machine/collector       dir.data = /data/measurement
machine/collector       dir.log = /log/collector
machine/collector       service.autostart = open
machine/collector       timezone = UTC
machine/collector       disk_gb = 40
machine/verifier        package.resolver = 3.5.0
machine/verifier        config.log_level = detail
machine/verifier        service.autostart = open
data_store/measurement  user.report = read-only
object_store/archive    access_policy = in-account-plus-read
SON
```

The inventory counts the items that **genuinely exist** in the environment; it is independent of
what the records say. The two records fall on top of it. The provisioning record is written in
blocks, and a single block can hold both attributes belonging to the resource's shell (`count`,
`size`, `disk_gb`) and ones belonging to the inside of the resource (`package.resolver`,
`timezone`). The config record is in line format and writes only to the inside of the resource. The
boundary sits not between the two, but **at their intersection**.

#### Where the Boundary Falls

```js
// layer.mjs — matches the inventory against the two layers' records; counts overlap and gap. Model.
import { readFileSync } from "node:fs";

const BLOCK = /^resource\s+(\w+)\s+"([\w-]+)"\s*\{(.*)\}$/;
export const provisioning = new Map();
for (const s of readFileSync("provisioning.infra", "utf8").split("\n")) {
  const m = BLOCK.exec(s.trim());
  if (m) for (const x of m[3].matchAll(/([\w.]+)\s*=\s*("[^"]*"|[\w./:-]+)/g))
    provisioning.set(`${m[1]}.${m[2]}.${x[1]}`, x[2].replaceAll('"', ""));
}
export const config = new Map();                      // the configuration agent's record: a different format
for (const s of readFileSync("config.record", "utf8").split("\n")) {
  const m = /^(\w+)\/([\w-]+)\s+([\w.]+)\s*=\s*(\S+)$/.exec(s.trim());
  if (m) config.set(`${m[1]}.${m[2]}.${m[3]}`, m[4]);
}
export const inventory = readFileSync("inventory.txt", "utf8").trim().split("\n")
  .map((s) => s.trim().split(/\s+/)).map(([k, natural]) => ({ k, natural }));

export const split = (item, S, A) => {               // every item falls into exactly one box
  const g = { provisioningOnly: [], configOnly: [], overlap: [], gap: [] };
  for (const { k } of item) {
    const s = S.has(k), a = A.has(k);
    g[s && a ? "overlap" : s ? "provisioningOnly" : a ? "configOnly" : "gap"].push(k);
  }
  return g;
};

if (process.argv[1].endsWith("layer.mjs")) {       // writes when run directly, silent when imported
  const g = split(inventory, provisioning, config);
  console.log(`inventory ${inventory.length} items   provisioning record ${provisioning.size}` +
    `   config record ${config.size}`);
  console.log(`  provisioning only ${g.provisioningOnly.length}   config only ${g.configOnly.length}` +
    `   overlap ${g.overlap.length}   gap ${g.gap.length}`);
  console.log(`${g.overlap.length} overlapping items (defined in both layers):`);
  for (const k of g.overlap)
    console.log(`  ${k.padEnd(40)} provisioning "${provisioning.get(k)}"   config "${config.get(k)}"` +
      `   ${provisioning.get(k) === config.get(k) ? "same" : "CONFLICT"}`);
  console.log(`${g.gap.length} items in the gap (in neither layer):`);
  for (const k of g.gap)
    console.log(`  ${k.padEnd(51)} natural layer: ${inventory.find((e) => e.k === k).natural}`);
}
```

```
inventory 34 items   provisioning record 20   config record 15
  provisioning only 14   config only 9   overlap 6   gap 5
6 overlapping items (defined in both layers):
  machine.collector.disk_gb                provisioning "40"   config "40"   same
  machine.collector.package.resolver       provisioning "3.4.0"   config "3.5.0"   CONFLICT
  machine.collector.config.log_level       provisioning "summary"   config "summary"   same
  machine.collector.service.autostart      provisioning "open"   config "open"   same
  machine.collector.timezone               provisioning "UTC"   config "UTC"   same
  object_store.archive.access_policy       provisioning "in-account"   config "in-account-plus-read"   CONFLICT
5 items in the gap (in neither layer):
  network.measurement-network.egress_path             natural layer: resource
  security.collector.egress                           natural layer: resource
  dns_record.collector.internal                       natural layer: resource
  machine.collector.log_rotation                      natural layer: content
  data_store.measurement.parameter.connection_count   natural layer: content
```

Thirty-four items fell into four boxes: fourteen only in provisioning, nine only in config, six in
both, five in neither. The two records define thirty-five items in total, but the number of items
they cover is twenty-nine — the difference of six is the **overlap**, and a definition written a
second time gains no new item.

Four of the six overlapping items carry the same value. These four look harmless, and the only
condition under which they are harmless is that when one changes, the other is changed too; because
the two are written in separate files, in separate formats, and often by separate people, this
condition is not met automatically. The remaining two items are already **conflicting**: the
resolver version is 3.4.0 in one record, 3.5.0 in the other; the archive's access policy is narrow
in one record, wide in the other.

Where the overlap comes from is also readable. Five of the six items sit on the machine resource,
and all five have an image counterpart: because the provisioning layer opens the machine from an
image, it **indirectly** defines the package version, timezone, and service state that image
carries. The config layer defines the same values directly. The two records reach the same item by
two different routes, and neither knows the other writes it; the overlap is not carelessness, it is
the image carrying a definition that is invisible in the record.

The five items in the gap are the same measure as the last line of lesson 01, and here they appear
together with their natural layers: three are naturally resource, two naturally content. These
stayed unwritten not because no one wrote a record, but because they **fell between** the two
records — each layer's owner assumed the other one wrote it. The egress path, the security rule's
egress direction, the DNS record, log rotation, and the connection count: all five are items no one
claimed.

#### Which Layer Wins

```js
// boundary.mjs — application order, the post-reprovisioning drift window, and three boundary lines.
import { provisioning, config, inventory, split } from "./layer.mjs";

const g = split(inventory, provisioning, config);
const conflicting = g.overlap.filter((k) => provisioning.get(k) !== config.get(k));
console.log(`which layer wins on an overlapping item (last writer wins)`);
for (const [name, order] of [["provisioning then config (usual)", ["provisioning", "config"]],
  ["config then reprovisioning", ["config", "provisioning"]]]) {
  const final = (k) => (order[1] === "config" ? config : provisioning).get(k);
  console.log(`  ${name.padEnd(33)} conflicting ${conflicting.length} items -> ` +
    conflicting.map((k) => `${k.split(".").pop()}="${final(k)}"`).join("  "));
}

// Reprovisioning: the machine is reborn from the image, and the item the config layer wrote reverts to it.
const MACHINES = 14, AGENT_MIN = 30;
const machineItems = [...config.keys()].filter((k) => k.startsWith("machine."));
const reverted = machineItems.filter((k) => (provisioning.get(k) ?? "default") !== config.get(k));
console.log(`\ndrift window after reprovisioning (agent runs every ${AGENT_MIN} minutes)`);
console.log(`  items the config layer writes on the machine ${machineItems.length}` +
  `   at the wrong value after reprovisioning ${reverted.length}`);
console.log(`  ${MACHINES} machines -> ${reverted.length * MACHINES} items` +
  `   average ${AGENT_MIN / 2} min, worst ${AGENT_MIN} min` +
  `   -> ${reverted.length * MACHINES * AGENT_MIN / 2} item-minutes of drift`);

// Three boundary lines. Items baked into the image: package, service, and timezone.
const bakedIntoImage = (k) => /package\.|service\.|timezone/.test(k);
const boundary = {
  "C1 boundary at the resource shell": (k) => [provisioning.has(k), config.has(k)],
  "C2 boundary shifted to the image": (k) => [provisioning.has(k) || (config.has(k) && bakedIntoImage(k)),
    config.has(k) && !bakedIntoImage(k)],
  "C3 single layer (everything in provisioning)": (k) => [provisioning.has(k) || config.has(k), false],
};
const HEAVY = { "C1 boundary at the resource shell": 0, "C2 boundary shifted to the image": 6,
  "C3 single layer (everything in provisioning)": 15 };
console.log(`\nwhat happens depending on where the boundary is drawn (22 changes in a quarter: 7 resource, 15 content)`);
for (const [name, f] of Object.entries(boundary)) {
  const S = new Map(), A = new Map();
  for (const { k } of inventory) { const [s, a] = f(k); if (s) S.set(k, 1); if (a) A.set(k, 1); }
  const b = split(inventory, S, A);
  console.log(`  ${name.padEnd(46)} provisioning ${S.size}  config ${A.size}` +
    `  overlap ${b.overlap.length}  gap ${b.gap.length}` +
    `  applied twice ${b.overlap.length}  changes falling to the heavy path ${HEAVY[name]}`);
}
```

```
which layer wins on an overlapping item (last writer wins)
  provisioning then config (usual)  conflicting 2 items -> resolver="3.5.0"  access_policy="in-account-plus-read"
  config then reprovisioning        conflicting 2 items -> resolver="3.4.0"  access_policy="in-account"

drift window after reprovisioning (agent runs every 30 minutes)
  items the config layer writes on the machine 13   at the wrong value after reprovisioning 9
  14 machines -> 126 items   average 15 min, worst 30 min   -> 1890 item-minutes of drift

what happens depending on where the boundary is drawn (22 changes in a quarter: 7 resource, 15 content)
  C1 boundary at the resource shell              provisioning 20  config 15  overlap 6  gap 5  applied twice 6  changes falling to the heavy path 0
  C2 boundary shifted to the image               provisioning 23  config 9  overlap 3  gap 5  applied twice 3  changes falling to the heavy path 6
  C3 single layer (everything in provisioning)   provisioning 29  config 0  overlap 0  gap 5  applied twice 0  changes falling to the heavy path 15
```

The final value of the two conflicting items depends on the **run order**. In the usual order, the
config layer runs last, so it wins; the resolver becomes 3.5.0, the access policy becomes wide. When
a reprovisioning is triggered, the provisioning layer runs last, and the same two items revert to
the opposite value. The winning layer is not a decision, it is a **timing outcome** — and what
determines that timing is often an event nobody is watching: a machine dying, a scaling change, an
image update.

The second table gives its duration. After reprovisioning, nine of the thirteen items the config
layer writes on the machine revert to the value the image carries; because the agent runs every
thirty minutes, they stay at the wrong value for fifteen minutes on average. Across fourteen
machines that is 126 items and 1,890 item-minutes. This duration is not an outage, it is a **drift
window**: the environment is running, taking measurements, but nine items are not at the value the
record says.

#### When the Boundary Shifts

The last table compares the boundary drawn in three separate places. `C1` keeps the boundary at the
resource's shell: each layer writes its own record, overlap is six, items applied twice are six, but
no change falls onto the heavy path — all fifteen content changes finish with an agent run. `C2`
shifts the boundary to the image: package, service, and timezone items get baked into the image, the
config layer drops from fifteen to nine, overlap falls from six to three — but six changes now
require baking an image and rebuilding the machines. `C3` removes the boundary entirely: twenty-nine
items in a single layer, overlap zero, items applied twice zero — and all fifteen content changes
fall onto the heavy path.

The provisioning record's item count also climbs across the three drawings, from twenty to
twenty-three to twenty-nine. Dropping to a single layer does not simplify the record; it piles the
sum of the two records into one file and confronts everyone who reads that file with items that do
not concern them.

The one number that does not change across all three drawings is the **gap**: five. Moving the
boundary reduces or increases overlap, but it never brings into scope an item no layer claims. The
gap has nothing to do with where the boundary is; it has to do with **the part of the inventory that
falls outside the record**, and the only way to close it is to write an owner for the item.

The overlap decreasing is not free either. Between `C1` and `C3`, as overlap falls from six to zero,
the changes falling onto the heavy path climb from zero to fifteen. The two numbers move in opposite
directions, and the decision is made between them: in a quarter, is an inconsistency six overlaps
might miss more expensive, or is the duration of fifteen rebuilds more expensive. The question has
no single right answer, but both answers are countable.

#### Summary

- When 34 items were split across two records, 14 came out only in provisioning, 9 only in config, 6
  in both, 5 in neither; 35 definitions cover only 29 items.
- 4 of the 6 overlapping items carry the same value, and their being harmless depends on the two
  records being updated together; 2 are already conflicting.
- On a conflicting item, the winning layer is whichever ran last: config in the usual order,
  provisioning on reprovisioning; the winner is not a decision, it is a timing outcome.
- After reprovisioning, 9 items per machine revert to the image's value and stay wrong for an average
  of 15 minutes until the agent runs; across 14 machines that is 1,890 item-minutes of drift.
- When the boundary was drawn in three separate places, overlap was 6, 3, and 0; changes falling onto
  the heavy path climbed to 0, 6, and 15; the gap stayed at 5 across all three drawings.

#### Next Step

This lesson's last two numbers say the same thing from two places. After reprovisioning, nine items
were not at the value the record said, and stayed that way for fifteen minutes; five items are
always that way, because they fall inside no record's scope. Both share the same name: the record
says one thing, reality says another.

The next lesson closes this topic and measures that difference directly. A drift detector is written
— the record is read, the actual state is read, the difference is worked out item by item — and four
things are counted: the drift accumulated over a period, the **detection delay** by scan frequency,
the source of the drift, and the cost of fixing it. The last is at least as important as the other
three: undoing a drift sometimes breaks something that works.

### Drift

Source: https://academia.sh/en/courses/infrastructure-as-code/drift

A drift detector is written: the record is read, the actual state is read from disk, the difference is worked out item by item, and each difference's source is separated. The drift accumulated over a thirty-day period, the detection delay by scan frequency, the drift that never closes because it falls outside the record's scope, and the case where a rollback breaks something that works are counted.

The previous four lessons said the same thing from four places. The two setups diverged on nine
items; after partial failure, the record and the environment disagreed on three items; a rebuild
wiped hotfixes; after reprovisioning, nine items stayed for fifteen minutes at a value the record
never said. All of them share the same name.

**Drift** is the difference between what the record says and reality. This lesson measures it
directly: a detector is written — the record is read, the actual state is read **from disk**, the
difference is worked out item by item, and each difference's source is separated. Then a
thirty-day period is run, and four things are counted: the drift that accumulates, the **detection
delay** by scan frequency, the drift that never closes because it falls outside the record's scope,
and the cost of fixing it. The last is at least as important as the other three: rolling back a
drift sometimes breaks something that works.

One distinction is set up from the start. The reconciliation loop measured while **Cluster
Architecture** was covered in M22/K03 ran **continuously**, and a difference closed the moment it
was born; here, an apply is **triggered**, and drift accumulates between two triggers. Every number
in this lesson is born in that gap.

The environment is again the staging environment of the **regional measurement network**, and it is
**fictional**. The record and the actual state are genuinely written to disk and the detector
genuinely runs; the thirty-day period is modeled with `node`, and the generator is self-written.

**IF25.** The record writes twenty items. The actual state is a directory tree and twenty-five
files; the detector reads it not from the record but from **disk**.
**IF26.** The provider's documented defaults are in a separate file. If an item not in the record is
on that list, its source is a **provider default**; if not, it is an **out-of-record resource**.
**IF27.** Over the thirty-day period, events are produced by the generator (seed 4471903, the
generator is self-written); the daily event count is zero, one, or two.
**IF28.** An event's source is a manual change, an out-of-record resource, or a provider default.
Only a **manual change** falls within the record's scope; the others are not closed by an apply.
**IF29.** Apply is triggered: on days 1, 12, and 25. **A scan is separate from apply** — it detects,
it does not close.
**IF30.** Some of the manual changes are **live fixes** made to close an incident; when rolled back,
the problem they closed comes back.

#### The Record and Reality

```bash
#!/usr/bin/env bash
# Record and actual environment: the record is one file, the actual state is a directory tree. Both are written to disk.
set -e
rm -rf actual

cat > record.infra <<'SON'
resource network "measurement-network" { block = "10.40.0.0/16"  subnet_count = 3  private = "true" }
resource machine "collector" { count = 3  size = "medium"  disk_gb = 40  image = "base-7" }
resource machine "verifier" { count = 2  size = "small"  disk_gb = 20  image = "base-7" }
resource queue "reading" { partitions = 6  retention_hours = 72  dead_letter = "open" }
resource data_store "measurement" { size_gb = 200  backup_days = 14  version = "5.2" }
resource object_store "archive" { class = "sparse"  versioning = "open"  lifecycle_days = 365 }
SON

# The provider's documented defaults: fields the record does not write, that arise on their own.
cat > defaults.txt <<'SON'
network.measurement-network.egress_path
queue.reading.encryption
data_store.measurement.maintenance_window
object_store.archive.access_rule
SON

write() { mkdir -p "actual/$(dirname "$1")"; printf '%s\n' "$2" > "actual/$1"; }
# Fourteen items that match the record
write network/measurement-network/block 10.40.0.0/16;    write network/measurement-network/private true
write machine/collector/size medium;                     write machine/collector/image base-7
write machine/verifier/count 2;                           write machine/verifier/size small
write machine/verifier/disk_gb 20;                          write machine/verifier/image base-7
write queue/reading/partitions 6;                            write queue/reading/dead_letter open
write data_store/measurement/size_gb 200;                     write data_store/measurement/version 5.2
write object_store/archive/versioning open;                    write object_store/archive/lifecycle_days 365
# Five items changed by hand (different from the record's value)
write network/measurement-network/subnet_count 4;               write machine/collector/count 5
write machine/collector/disk_gb 80;                                write queue/reading/retention_hours 24
write data_store/measurement/backup_days 7
# object_store/archive/class deleted: in the record, not in reality
# Provider default (three items) and out-of-record resource (three items)
write network/measurement-network/egress_path open;                 write queue/reading/encryption provider-key
write data_store/measurement/maintenance_window sunday-03:00
write security/temporary/ingress 0.0.0.0-8443;                       write dns_record/collector/internal collector.internal
write machine/diagnostics/count 1
printf 'record %s lines   actual %s files\n' "$(wc -l < record.infra | tr -d ' ')" \
  "$(find actual -type f | wc -l | tr -d ' ')"
```

```
record 6 lines   actual 25 files
```

#### The Difference, Item by Item

```js
// drift.mjs — reads the record, reads the actual state from disk, works out the difference item by item. Fixes nothing.
import { readFileSync, readdirSync, statSync } from "node:fs";

const BLOCK = /^resource\s+(\w+)\s+"([\w-]+)"\s*\{(.*)\}$/;
const record = new Map();
for (const s of readFileSync("record.infra", "utf8").split("\n")) {
  const m = BLOCK.exec(s.trim());
  if (m) for (const x of m[3].matchAll(/(\w+)\s*=\s*("[^"]*"|[\w./-]+)/g))
    record.set(`${m[1]}.${m[2]}.${x[1]}`, x[2].replaceAll('"', ""));
}
const providerDefault = new Set(readFileSync("defaults.txt", "utf8").trim().split("\n"));

const actual = new Map();                       // actual state is read from disk, not from the record
const walk = (dir, prefix = []) => {
  for (const name of readdirSync(dir).sort()) {
    const path = `${dir}/${name}`;
    if (statSync(path).isDirectory()) walk(path, [...prefix, name]);
    else actual.set([...prefix, name].join("."), readFileSync(path, "utf8").trim());
  }
};
walk("actual");

const all = [...new Set([...record.keys(), ...actual.keys()])].sort();
const drift = all.filter((k) => record.get(k) !== actual.get(k)).map((k) => ({ k,
  source: !actual.has(k) ? "missing item" : !record.has(k)
    ? (providerDefault.has(k) ? "provider default" : "out-of-record resource") : "manual change" }));
const countBy = (t) => drift.filter((f) => f.source === t).length;

console.log(`record ${record.size} items   actual ${actual.size} items   union ${all.length}`);
console.log(`drift ${drift.length} items` +
  `   manual ${countBy("manual change")}   missing ${countBy("missing item")}` +
  `   out-of-record ${countBy("out-of-record resource")}   default ${countBy("provider default")}`);
for (const f of drift)
  console.log(`  ${f.source.padEnd(22)} ${f.k.padEnd(43)} record "${record.get(f.k) ?? "-"}"` +
    `  actual "${actual.get(f.k) ?? "-"}"`);
const inScope = drift.filter((f) => record.has(f.k)).length;
console.log(`drift in the record's scope, i.e. closable by an apply ${inScope}` +
  `   out of scope ${drift.length - inScope}`);
```

```
record 20 items   actual 25 items   union 26
drift 12 items   manual 5   missing 1   out-of-record 3   default 3
  manual change          data_store.measurement.backup_days          record "14"  actual "7"
  provider default       data_store.measurement.maintenance_window   record "-"  actual "sunday-03:00"
  out-of-record resource dns_record.collector.internal               record "-"  actual "collector.internal"
  manual change          machine.collector.count                     record "3"  actual "5"
  manual change          machine.collector.disk_gb                   record "40"  actual "80"
  out-of-record resource machine.diagnostics.count                   record "-"  actual "1"
  provider default       network.measurement-network.egress_path     record "-"  actual "open"
  manual change          network.measurement-network.subnet_count    record "3"  actual "4"
  missing item           object_store.archive.class                  record "sparse"  actual "-"
  provider default       queue.reading.encryption                    record "-"  actual "provider-key"
  manual change          queue.reading.retention_hours               record "72"  actual "24"
  out-of-record resource security.temporary.ingress                  record "-"  actual "0.0.0.0-8443"
drift in the record's scope, i.e. closable by an apply 6   out of scope 6
```

Twelve items drifted, and the last line shows where the measure actually sits: **six** of the
twelve fall within the record's scope. For the remaining six items, running an apply changes
nothing, because the record says nothing about them — three are fields the provider opens on its
own, three are resources created without a record ever being written. The only reason these are
visible at all is that the detector is **two-directional**: it does not just scan the items in the
record, it walks the whole actual state and writes down the ones with no counterpart in the record
too.

The source distinction is not a labeling exercise, it is an **action distinction**. A manual change
is closed by an apply. A missing item comes back with an apply. An out-of-record resource is only
closed by being brought into the record or being deleted; a provider default neither closes nor
should it — all that needs to be known is that the record is silent about it. Four separate kinds of
difference, four separate decisions.

#### Over a Period

```js
// period.mjs — 30-day drift accumulation, detection delay, and the cost of fixing it. Model.
const SEED = 4471903, DAYS = 30, APPLY = [1, 12, 25];   // apply is triggered, it does not run continuously
const generator = (t) => () => (t = (t * 1103515245 + 12345) % 2147483648) / 2147483648;
const r = generator(SEED);

const events = [];
for (let d = 1; d <= DAYS; d++)
  for (let n = (r() < 0.55) + (r() < 0.18); n > 0; n--) {
    const p = r();
    const source = p < 0.45 ? "manual" : p < 0.72 ? "out-of-record" : "default";
    events.push({ d, source, inScope: source === "manual", live: source === "manual" && r() < 0.35 });
  }
const countBy = (t) => events.filter((o) => o.source === t).length;
const inScope = events.filter((o) => o.inScope);
console.log(`seed ${SEED}   days ${DAYS}   events ${events.length}   apply days` +
  ` ${APPLY.join(", ")}`);
console.log(`source of the drift   manual ${countBy("manual")}   out-of-record ${countBy("out-of-record")}` +
  `   provider default ${countBy("default")}   in scope ${inScope.length}`);

// Two regimes: apply is triggered (three days) and continuous reconciliation (every day). Both close only what is in scope.
const regime = (days) => {
  const total = [], scope = [];
  let closed = 0;
  for (let d = 1; d <= DAYS; d++) {
    if (days.includes(d)) closed = inScope.filter((o) => o.d <= d).length;
    scope.push(inScope.filter((o) => o.d <= d).length - closed);
    total.push(events.filter((o) => o.d <= d).length - closed);
  }
  const area = (x) => x.reduce((a, b) => a + b, 0);
  return { total, scope, areaT: area(total), areaS: area(scope) };
};
const A = regime(APPLY), B = regime([...Array(DAYS)].map((_, i) => i + 1));
console.log(`\nopen drift (end of day, ${DAYS} days)`);
console.log(`  triggered apply     total   ${A.total.join(" ")}`);
console.log(`  triggered apply     scope   ${A.scope.join(" ")}`);
console.log(`  continuous reconcile total  ${B.total.join(" ")}`);
console.log(`  day-item area   in-scope drift: triggered ${A.areaS}   continuous ${B.areaS}`);
console.log(`  day-item area   all drift:      triggered ${A.areaT}   continuous ${B.areaT}` +
  `   difference ${A.areaT - B.areaT}`);

console.log(`\ndetection delay (only the ${inScope.length} in-scope drift; a scan detects, it does not close)`);
for (const n of [1, 3, 7, 30]) {
  const delay = inScope
    .map((o) => { for (let d = o.d; d <= DAYS; d++) if (d % n === 0) return d - o.d; return null; })
    .filter((x) => x !== null);
  console.log(`  scan every ${String(n).padStart(2)} days   average` +
    ` ${(delay.reduce((a, b) => a + b, 0) / delay.length).toFixed(1)} days` +
    `   worst ${Math.max(...delay)} days   detected ${delay.length}/${inScope.length}`);
}

const rolledBack = inScope.filter((o) => o.d <= APPLY[APPLY.length - 1]);
const broken = rolledBack.filter((o) => o.live);
console.log(`\nthe cost of fixing it`);
console.log(`  drift rolled back ${rolledBack.length}   of these a live fix` +
  ` ${broken.length}   rolling it back breaks something that works`);
console.log(`  closed for good ${rolledBack.length - broken.length}` +
  `   came back ${broken.length} (${(100 * broken.length / rolledBack.length).toFixed(0)}%)`);
console.log(`  out-of-scope ${events.length - inScope.length} drift was closed by no apply;` +
  ` all of it is still standing at the end of the period`);
```

```
seed 4471903   days 30   events 21   apply days 1, 12, 25
source of the drift   manual 9   out-of-record 6   provider default 6   in scope 9

open drift (end of day, 30 days)
  triggered apply     total   1 1 2 2 2 3 3 3 4 5 6 3 5 6 7 8 8 8 9 9 10 11 12 13 9 10 10 11 11 13
  triggered apply     scope   0 0 1 1 1 1 1 1 2 3 3 0 1 2 2 2 2 2 2 2 2 2 3 4 0 1 1 1 1 1
  continuous reconcile total  1 1 1 1 1 2 2 2 2 2 3 3 4 4 5 6 6 6 7 7 8 9 9 9 9 9 9 10 10 12
  day-item area   in-scope drift: triggered 45   continuous 0
  day-item area   all drift:      triggered 205   continuous 160   difference 45

detection delay (only the 9 in-scope drift; a scan detects, it does not close)
  scan every  1 days   average 0.0 days   worst 0 days   detected 9/9
  scan every  3 days   average 1.0 days   worst 2 days   detected 9/9
  scan every  7 days   average 3.1 days   worst 5 days   detected 9/9
  scan every 30 days   average 13.7 days   worst 27 days   detected 9/9

the cost of fixing it
  drift rolled back 8   of these a live fix 2   rolling it back breaks something that works
  closed for good 6   came back 2 (25%)
  out-of-scope 12 drift was closed by no apply; all of it is still standing at the end of the period
```

Twenty-one drift events were born over thirty days, and nine of them fell within the record's
scope. The difference between the two series reduces what a triggered apply means to a single line:
in-scope drift stayed open for forty-five day-items in the triggered regime, zero in continuous
reconciliation. The three trigger days show up clearly in the series — on days 1, 12, and 25, the
scope line drops to zero, then climbs again. Drift does not disappear, it is reborn between every
trigger; what accumulates is not drift itself, it is **unclosed** drift.

The total line, on the other hand, keeps climbing in both regimes and stops at thirteen and twelve
on day thirty. The difference between them is only forty-five day-items, and that comes from the
nine in-scope items. The remaining twelve drift events closed under neither regime: running apply
continuously does not make them visible, because the reason they stay invisible is not frequency, it
is **scope**. This shows how far the thing seen as five items in the topic's first lesson has grown
over the course of a period.

#### Detection Delay and the Cost of Fixing It

The scan table measures a separate decision. A scan is independent of apply: it makes a drift
visible **without closing** it. A daily scan zeroes out the delay, a scan every three days raises it
to an average of one day, a weekly scan to 3.1 days, a monthly scan to 13.7 days — and on a monthly
scan the worst delay is twenty-seven days. Twenty-seven days is how long an item runs at a value the
record never said, with nobody knowing. Choosing a scan frequency is choosing this number.

The last table carries this lesson's most counterintuitive number. By day twenty-five, eight drift
events were rolled back, and two were **live fixes**: changes made by hand to close an incident,
that genuinely resolved the problem. When rolled back, the problem they closed came back, and both
were redone on later days. The drift rollback closed for good is six, the drift that came back is
two — twenty-five percent.

These two items show that drift cannot be read in only one direction. Drift does not mean "the
record is right, reality is wrong"; it only means the two have **diverged**. Divergence has two
interpretations, and which one is correct cannot be read from the difference itself: either the
record should be updated and the fix written back into it, or reality should be rolled back. An
automated apply always picks the second, because it has no other option. The cost of the record
staying silent about an item was invisibility; the cost of the record staying **stale** about an
item is breaking something that works.

#### Summary

- The detector compared 26 items and found 12 drifted: 5 manual changes, 1 missing item, 3
  out-of-record resources, 3 provider defaults; only 6 of these can be closed by an apply.
- 21 drift events were born over thirty days; 9 fell within the record's scope, 12 were still open
  at the end of the period and closed under no regime.
- Triggered apply left in-scope drift open for 45 day-items, continuous reconciliation 0; the
  difference comes from frequency, the 12 items that never close come from scope.
- Scan frequency determines the delay directly: 0.0 days daily, 3.1 days weekly, 13.7 days monthly
  on average, and 27 days worst case.
- 2 of the 8 rolled-back drift events were live fixes, and rolling them back broke something that
  works; the rollback closed 6 for good, 2 came back.

#### Next Step

At this point, drift is something countable. The record's scope is known, the item outside that
scope can be written down by name, the source of the difference can be split into four, the
detection delay can be tied to scan frequency, and fixing it has been measured to carry a cost. The
whole measurement leans on a single assumption: that a readable record exists somewhere.

The record itself was never addressed. Across five lessons, the record was always found ready-made
— a six-line file, a list counting its items by their values. How it is written, what structures it
is built from, how one resource attaches to another, how the same definition gets written for three
environments without repeating itself, where a value comes from, and what it will show before a
change is applied — none of this was asked. The next topic starts by writing the record itself.

## Declarative Provisioning

Writing and applying the desired infrastructure: the configuration language's block, expression, and type structures; providers and resources that connect to the target system; data sources that read existing infrastructure; parameterization with variables, locals, and outputs; implicit and explicit dependency ordering; count- and set-based iteration; the plan workflow that shows a change in advance; and bringing an existing resource under management.

### Configuration Language

Source: https://academia.sh/en/courses/infrastructure-as-code/configuration-language

We define our own small configuration language and write its parser; the declaration, expression, and type structures the language carries are counted separately. It measures how many of eleven defects in a record without type declarations show up only at apply, how many get caught at parse time once types are added, and how line count and read hops move in opposite directions across three writings of the same target.

The previous topic made drift measurable; the one thing that stayed unmeasured was the record
itself. This lesson defines our own small **configuration language**, writes a **parser** for it
with `node`, and measures three things: the declaration, expression, and type structures the
language carries; where an untyped record and a typed record catch the same defects; and the cost
of three ways of writing the same target. The measurement runs on the west environment of the
**regional measurement network** and is **fictional**; the language, the parser, and the target
system are a **model** built with `node`.

**DP1.** The fictional record carries two resources for the west environment: a machine cluster
and a queue.
**DP2.** The type declaration is a structure of the language, not of the target system; it runs
before apply.
**DP3.** Type checking does not compute an expression's result; it only tests a value written as a
constant.
**DP4.** The target system model writes the record's value as-is; it only rejects a non-number
where a number is expected.
**DP5.** If the value written to the target differs from what the record intends and no stage
reports it, the defect counts as **silent**.
**DP6.** A hop is the number of block references followed to find a field's value.

#### The Language's Three Structures

```js
// lang.mjs — our language's tokenizer and parser.
const TYPE = ["string", "number", "name", "punct", "comment", "unknown"];
const MARK = /"([^"]*)"|(-?\d+)|([a-z_][a-z0-9_]*)|([{}=,.()])|(#[^\n]*)|(\S)/g;

export function tokenize(s) {
  const t = [];
  for (const m of s.matchAll(MARK)) {
    const i = m.slice(1).findIndex((x) => x !== undefined);
    if (i === 5) throw new Error(`unresolved mark "${m[0]}"`);
    if (i !== 4) t.push({ type: TYPE[i], value: m[i + 1] });
  }
  return t;
}

export function parse(source) {
  const t = tokenize(source); let i = 0;
  const peek = () => t[i], next = () => t[i++];
  const expect = (d) => { const x = next();
    if (!x || x.value !== d) throw new Error(`expected "${d}", found "${x ? x.value : "end"}"`); };
  const body = () => { const f = {};                 // block body and map take the same path
    while (peek().value !== "}") { const a = next().value; expect("="); f[a] = expression();
      if (peek().value === ",") next(); }
    next(); return f; };
  function expression() {
    const x = next();
    if (!x) throw new Error("record ended before expression finished");
    if (x.type === "number") return { d: "number", value: Number(x.value) };
    if (x.type === "string") return { d: "string", value: x.value };
    if (x.value === "{") return { d: "map", fields: body() };
    if (x.type !== "name") throw new Error(`unexpected "${x.value}"`);
    if (peek() && peek().value === "(") { next(); const arg = [];
      while (peek().value !== ")") { arg.push(expression()); if (peek().value === ",") next(); }
      next(); return { d: "call", name: x.value, arg }; }
    const path = [x.value];
    while (peek() && peek().value === ".") { next(); path.push(next().value); }
    return { d: "path", path };
  }
  const b = [];
  while (i < t.length) {
    const type = next().value, tag = [];
    while (peek() && peek().type !== "punct") tag.push(next().value);
    expect("{");
    b.push({ type, tag, fields: body(), key: type === "resource" ? tag.join(".") : type });
  }
  return b;
}
```

```js
// record.mjs — the fictional record of the west environment.
export const ENVIRONMENT = `environment "west" { region = "west"  collectors = 6  threshold_ms = 900  retention = 30 }`;
export const BODY = `
resource machine "collector" {
  name        = concat(environment.region, "-collector")
  count       = environment.collectors
  cores       = 4
  memory_gb   = 8
  tags        = { role = "meter-reading", region = environment.region }
}
resource queue "meter" {
  name           = concat(environment.region, "-meter-queue")
  partitions     = multiply(environment.collectors, 2)
  retention_days = environment.retention
  format         = "binary"
  threshold_ms   = environment.threshold_ms
}`;
export const TYPES = `
type environment { region = string  collectors = number  threshold_ms = number  retention = number }
type machine { name = string  count = range(1, 64)  cores = range(1, 64)
             memory_gb = range(1, 256)  tags = map }
type queue { name = string  partitions = range(1, 128)  retention_days = range(1, 365)
             format = set("binary", "text")  threshold_ms = number }`;
```

The language has a single declaration form: a type word, tags, and field assignments inside
braces. The right-hand side can be a **constant**, a reference to another block's field, a
**call**, or a **map**; there are no operators, and even multiplication is a call. This is the
smallest form that can carry the measurement. The type declaration is written with the same
syntax.

#### Untyped Record, Typed Record

```js
// check.mjs — name resolution, type checking, apply model.
import { parse } from "./lang.mjs";

const FN = { concat: (...a) => a.join(""), multiply: (a, b) => a * b, select: (m, k) => m[k] };
const children = (e) => e.d === "call" ? e.arg : e.d === "map" ? Object.values(e.fields) : [];
export const print = (w, ...s) => console.log(s.map((x) => String(x).padEnd(w)).join("").trimEnd());
export const nodes = (e) => [e, ...children(e).flatMap(nodes)];
export const depth = (e) => 1 + Math.max(0, ...children(e).map(depth));
export const isConstant = (e) => e.d === "number" || e.d === "string";
export const kind = (b) => b.type === "resource" ? b.tag[0] : b.type;
export const indexBy = (b) => new Map(b.map((x) => [x.key, x]));

export const extractSchema = (m) => Object.fromEntries(parse(m).map((b) => [b.tag[0],
  Object.fromEntries(Object.entries(b.fields).map(([a, e]) => [a, e.d === "path" ? { base: e.path[0] }
    : e.name === "range" ? { base: "number", range: e.arg.map((x) => x.value) }
    : { base: "string", set: e.arg.map((x) => x.value) }]))]));

export function resolve(path, idx) {
  for (let n = path.length - 1; n >= 1; n--) {
    const b = idx.get(path.slice(0, n).join(".")), a = path.slice(n).join(".");
    if (!b) continue;
    if (!(a in b.fields)) throw new Error(`${b.key} has no ${a}`);
    return { block: b, field: a };
  }
  throw new Error(`${path.join(".")} does not match any block`);
}

export function checkTypes(blocks, schema) {          // only runs on the typed record
  const errs = [];
  for (const b of blocks) {
    const t = schema[kind(b)];
    for (const a of Object.keys(t)) if (!(a in b.fields)) errs.push(`${b.key}.${a} missing`);
    for (const [a, e] of Object.entries(b.fields)) {
      const k = t[a], y = `${b.key}.${a}`;
      if (!k) errs.push(`${y} untyped`);
      else if (!isConstant(e)) continue;               // a computed value cannot be read from the record
      else if (e.d !== k.base) errs.push(`${y} not ${k.base}`);
      else if (k.range && (e.value < k.range[0] || e.value > k.range[1])) errs.push(`${y} out of range`);
      else if (k.set && !k.set.includes(e.value)) errs.push(`${y} out of set`);
    }
  }
  return errs;
}

export function evaluate(e, idx) {
  if (isConstant(e)) return e.value;
  if (e.d === "map") return Object.fromEntries(Object.entries(e.fields).map(([a, v]) => [a, evaluate(v, idx)]));
  if (e.d === "call") return FN[e.name](...e.arg.map((x) => evaluate(x, idx)));
  const c = resolve(e.path, idx);
  return evaluate(c.block.fields[c.field], idx);
}

export function apply(blocks, schema) {                // target system model
  const idx = indexBy(blocks);
  return Object.fromEntries(blocks.filter((x) => x.type === "resource").flatMap((b) =>
    Object.entries(b.fields).map(([a, e]) => {
      const v = evaluate(e, idx), y = `${b.key}.${a}`;
      if (schema[kind(b)][a]?.base === "number" && !Number.isFinite(v)) throw new Error(`${y} not a number`);
      return [y, typeof v === "object" ? JSON.stringify(v) : v];
    })));
}
```

```js
// defects.mjs — three structures and eleven defects are counted.
import { parse } from "./lang.mjs";
import { ENVIRONMENT, BODY, TYPES } from "./record.mjs";
import { nodes, isConstant, indexBy, resolve, checkTypes, apply, extractSchema, print } from "./check.mjs";

const SCHEMA = extractSchema(TYPES), RECORD = ENVIRONMENT + BODY;
const blocks = parse(RECORD), BASE = apply(blocks, SCHEMA);
const assignments = blocks.flatMap((x) => Object.values(x.fields));
const allNodes = assignments.flatMap(nodes), typeFields = Object.values(SCHEMA).flatMap(Object.values);
const count = (t) => `${allNodes.filter((e) => e.d === t).length} ${t}`;
console.log(`declaration ${blocks.length} blocks, ${assignments.length} assignments, ${assignments.filter((e) => !isConstant(e)).length} computed`);
console.log(`expression   ${allNodes.length} nodes = ` + ["number", "string", "path", "call", "map"].map(count).join(" + "));
console.log(`type         ${Object.keys(SCHEMA).length} kinds, ${typeFields.length} field declarations, ` +
  `${typeFields.filter((k) => k.range || k.set).length} constrained\n`);

const corrupt = (field, next) => next === null            // a one-line defect
  ? RECORD.replace(new RegExp(`\n *${field} *=[^\n]*`), "")
  : RECORD.replace(new RegExp(`(\n *)${field}( *=)[^\n]*`),
    next[0] === "@" ? `$1${next.slice(1)}` : `$1${field}$2 ${next}`);
const DEFECTS = [
  ["type mismatch", "cores", '"4"'], ["missing field", "memory_gb", null],
  ["wrong field name", "retention_days", "@retention = 30"],
  ["out-of-set value", "format", '"raw"'],
  ["out-of-range constant", "cores", "128"], ["reference typo", "count", "environment.collector"],
  ["nonexistent field", "threshold_ms", "machine.collector.core_count"],
  ["syntax", "format", 'binary"'], ["reference to wrong field", "count", "environment.threshold_ms"],
  ["computed overshoot", "retention_days", "multiply(environment.retention, 20)"],
  ["computed zero", "threshold_ms", "multiply(environment.threshold_ms, 0)"],
];

function stage(text, typed) {                        // which stage the defect is caught at
  let b;
  try { b = parse(text); const idx = indexBy(b);
    for (const x of b) for (const e of Object.values(x.fields))
      nodes(e).filter((y) => y.d === "path").forEach((y) => resolve(y.path, idx));
  } catch (e) { return ["parse", e.message]; }
  if (typed) { const t = checkTypes(b, SCHEMA); if (t.length) return ["type", t[0]]; }
  let g; try { g = apply(b, SCHEMA); } catch (e) { return ["apply: error", e.message]; }
  const f = Object.keys(BASE).filter((k) => BASE[k] !== g[k]);
  return f.length ? ["apply: silent", `${f[0]} ${BASE[f[0]]} instead of ${g[f[0]]}`] : ["passed", "-"];
}

console.log(`base record ${stage(RECORD, true)[0]}, ${Object.keys(BASE).length} fields written to target`);
print(27, "defect", "untyped record", "typed record", "typed record's mark in apply");
const tally = { A: {}, B: {} };
for (const [name, field, next] of DEFECTS) {
  const m = corrupt(field, next), [a] = stage(m, false), [b, mark] = stage(m, true);
  tally.A[a] = (tally.A[a] || 0) + 1; tally.B[b] = (tally.B[b] || 0) + 1;
  print(27, name, a, b, b[0] === "a" ? mark : "");
}
console.log();
print(27, "stage", "untyped", "typed");
for (const a of ["parse", "type", "apply: error", "apply: silent"])
  print(27, a, tally.A[a] || 0, tally.B[a] || 0);
```

```
declaration 3 blocks, 14 assignments, 7 computed
expression   22 nodes = 6 number + 5 string + 7 path + 3 call + 1 map
type         3 kinds, 14 field declarations, 6 constrained

base record passed, 10 fields written to target
defect                     untyped record             typed record               typed record's mark in apply
type mismatch              apply: error               type
missing field              apply: silent              type
wrong field name           apply: silent              type
out-of-set value           apply: silent              type
out-of-range constant      apply: silent              type
reference typo             parse                      parse
nonexistent field          parse                      parse
syntax                     parse                      parse
reference to wrong field   apply: silent              apply: silent              machine.collector.count 6 instead of 900
computed overshoot         apply: silent              apply: silent              queue.meter.retention_days 30 instead of 600
computed zero              apply: silent              apply: silent              queue.meter.threshold_ms 900 instead of 0

stage                      untyped                    typed
parse                      3                          3
type                       0                          5
apply: error               1                          0
apply: silent              7                          3
```

Three blocks carry fourteen assignments; seven are computed expressions, and eleven of the
twenty-two expression nodes are constants. The type side writes fourteen field declarations for
three kinds — if ten resources of the same kind had been written, those same fourteen
declarations would check a hundred and forty assignments. **The type's cost is paid per kind, not
per resource.**

Three defects show up independent of typing: broken syntax, a reference that matches no block, and
a nonexistent field on a block that does exist; name resolution alone catches these. **The
remaining eight defects fall through to apply in the untyped record**, and seven of them write a
different value to the target without raising an error. The type declaration pulls five of those
back to parse time; the defects that fall through to apply drop to three, and the silent defects
drop to three. The gain is distance: a defect caught at parse time never touches a single resource.

In the three defects left standing, the value is not a constant but an **expression**: nine
hundred machines instead of six, six hundred days of retention instead of thirty, a zero threshold
instead of nine hundred milliseconds. The type gets the base right for all three, because they are
all numbers; it cannot test the range, because the value does not exist yet while the record is
being read. **This is what the record does not cover.**

#### Expressive Power versus Readability

```js
// style.mjs — the same target measured across three forms.
import { parse } from "./lang.mjs";
import { BODY, TYPES } from "./record.mjs";
import { extractSchema, nodes, depth, isConstant, resolve, indexBy, kind, apply, print } from "./check.mjs";

const SCHEMA = extractSchema(TYPES), REGIONS = ["west", "east", "central"], COUNTS = { west: 6, east: 6, central: 4 };
const MID = BODY.split("\n}\n")[0] + "\n}";          // the machine block from the record, unchanged
const midEnvironment = (o) => `environment "${o}" { region = "${o}"  collectors = ${COUNTS[o]} }`;
const tightEnvironment = (o) => `environment "${o}" { region = "${o}" }`;
const FLAT = `
resource machine "collector" {
  name      = "west-collector"
  count     = 6
  cores     = 4
  memory_gb = 8
  tags      = { role = "meter-reading", region = "west" }
}`;
const TIGHT = `
local "scale" {
  base   = 2
  factor = { west = 3, east = 3, central = 2 }
  cores  = { west = 4, east = 4, central = 4 }
}
resource machine "collector" {
  name      = concat(environment.region, "-collector")
  count     = multiply(local.base, select(local.factor, environment.region))
  cores     = select(local.cores, environment.region)
  memory_gb = multiply(local.base, multiply(select(local.cores, environment.region), 1))
  tags      = { role = "meter-reading", region = environment.region }
}`;

const lineCount = (m) => m.split("\n").filter((s) => s.trim()).length;
const hops = (e, idx) => nodes(e).filter((x) => x.d === "path")
  .reduce((t, x) => { const c = resolve(x.path, idx); return t + 1 + hops(c.block.fields[c.field], idx); }, 0);
const signature = (m) => JSON.stringify(apply(parse(m), SCHEMA));

const measure = (name, texts, lines, perEnv) => {
  const a = texts.flatMap((m) => { const b = parse(m), idx = indexBy(b);
    return b.filter((x) => x.type === "resource").flatMap((k) => Object.entries(k.fields).map(([n, e]) =>
      [depth(e), hops(e, idx), (SCHEMA[kind(k)][n].range || SCHEMA[kind(k)][n].set) && !isConstant(e) ? 1 : 0])); });
  const maxOf = (i) => Math.max(...a.map((x) => x[i]));
  print(14, name, lines, perEnv, maxOf(0), maxOf(1), a.reduce((s, x) => s + x[2], 0));
};

console.log(`three forms produce the same target: ${signature(FLAT) === signature(midEnvironment("west") + MID) &&
  REGIONS.every((o) => signature(tightEnvironment(o) + TIGHT) === signature(midEnvironment(o) + MID)) ? "yes" : "no"}\n`);
print(14, "form", "lines", "per env", "deepest", "hops", "type blind to");
measure("flat", [FLAT], lineCount(FLAT) * 3, lineCount(FLAT));
measure("mid", REGIONS.map((o) => midEnvironment(o) + MID), lineCount(MID) + 3, 1);
measure("tight", REGIONS.map((o) => tightEnvironment(o) + TIGHT), lineCount(TIGHT) + 3, 1);
```

```
three forms produce the same target: yes

form          lines         per env       deepest       hops          type blind to
flat          21            7             2             0             0
mid           10            1             2             1             3
tight         15            1             4             3             9
```

The three forms produce the same fifteen fields with the same values across three environments.
The flat form has no hops, and the field the type is blind to is zero; its cost is twenty-one
lines for three environments and **seven lines for every new environment**: eighty-four across
twelve environments.

The mid form brings the same target down to ten lines and drops the cost of a new environment to
one; nineteen lines across twelve environments. In return it costs one hop and three fields the
type is blind to. The tight form goes the other way: the line count **rises** to fifteen, because
it also writes the extra block that carries the maps; the deepest expression rises to four, hops
to three, and the fields the type is blind to rise to nine — nine of the fifteen fields cannot be
tested while the record is being read. **Expressive power reduces repetition, not lines, and it
reduces auditability outright.**

#### Summary

- The fictional record carried 3 blocks, 14 assignments, and 22 expression nodes; 7 assignments
  were computed, and the type wrote 14 field declarations for 3 kinds.
- In the untyped record, 3 of 11 defects showed up at parse time, and **8 fell through to apply**;
  7 of them wrote a different value to the target without raising an error.
- The type declaration pulled 5 of those 8 back to parse time; the defects falling through to
  apply dropped to 3, and silent defects dropped to 3.
- The value in the remaining 3 defects was an expression; 900 was written instead of 6, 600 days
  instead of 30, and a 900 ms threshold became 0.
- The three forms took 21, 10, and 15 lines; hops were 0, 1, and 3, and the fields the type is
  blind to were 0, 3, and 9; the cost of a new environment was 7 lines versus 1.

#### Next Step

The target system here was a five-line model. In reality, creating, reading, updating, and
deleting a resource are separate behaviors, and the layer that ties them to the record is called
the **provider**. The next lesson models the provider as an interface: which field, when it
changes, keeps the resource from updating in place and forces it to be **recreated**, and how many
fields' defaults shift when the provider version changes.

### Providers and Resources

Source: https://academia.sh/en/courses/infrastructure-as-code/providers-and-resources

The provider is modeled as a create-read-update-delete interface, and five resource types implement it; the field the record writes and the field that actually stands on the target are counted. It measures how many items in a change set the plan predicts, which fields force a resource to be recreated, and how many fields drift with the record completely unchanged when the provider version changes.

The previous lesson tied the record to a language, but the target system facing the record was a
five-line function: it took a value, wrote it, or rejected it. In reality a resource is created,
read, updated, and deleted; the layer that knows these four operations, translates the record's
field names into the target system's counterparts, and knows which field can be changed in place
is called the **provider**. This lesson models the provider as an interface and counts three
things: the gap between the field the record writes and the field that actually stands on the
target, how well a change's plan predicts the apply, and how much shifts while the record stays
completely unchanged when the provider version changes.

The measurement runs on the west environment of the **regional measurement network** and is
**fictional**; the provider, the resource type catalog, and the target store are a **model** built
with `node`. No real product or target system is called.

**DP7.** The provider defines five resource types: machine, queue, data store, network, object
store.
**DP8.** Every field has three properties: its type, whether it can be updated in place, and its
default. A field with no default is required.
**DP9.** If a field cannot be updated in place, the resource is deleted and recreated when that
field changes, and this carries a fixed downtime in minutes per type.
**DP10.** The plan phase knows only the in-place-updatability flag in the catalog.
**DP11.** The target system refuses to grow a numeric field in place if it is grown to more than
double; this rule is not written in the catalog.
**DP12.** The provider fills fields not written in the record with its own default, and the value
it fills in appears nowhere in the record.

#### The Provider Interface and Resource Types

```js
// provider.mjs — fictional provider: resource type catalog, create/read/update/delete interface (model).
const A = (type, inPlace, default_) => ({ type, inPlace, default: default_ });
export const CATALOG = {                            // provider version 1
  machine: { region: A("string", false), count: A("number", true), cores: A("number", false),
    memory_gb: A("number", true), disk_gb: A("number", true, 100), image: A("string", false, "base-2"),
    monitoring: A("string", true, "on") },
  queue: { region: A("string", false), partitions: A("number", false), retention_days: A("number", true, 7),
    format: A("string", false, "binary"), encryption: A("string", true, "off") },
  data_store: { region: A("string", false), size_gb: A("number", true), version: A("string", false, "3"),
    backup_days: A("number", true, 1), encryption: A("string", false, "off") },
  network: { region: A("string", false), subnet: A("number", false), gateways: A("number", true, 1) },
  object_store: { region: A("string", false), class: A("string", true, "hot"),
    versioning: A("string", false, "off"), retention_days: A("number", true, 365) },
};
export const DOWNTIME = { machine: 4, queue: 9, data_store: 25, network: 12, object_store: 30 };

export const RECORD = {                              // fictional record of the west environment
  "machine.collector": { region: "west", count: 6, cores: 4, memory_gb: 8 },
  "machine.verifier": { region: "west", count: 2, cores: 8, memory_gb: 16, disk_gb: 200, image: "base-2" },
  "queue.meter": { region: "west", partitions: 12 },
  "queue.alert": { region: "west", partitions: 3, format: "text" },
  "data_store.metrics": { region: "west", size_gb: 300 },
  "network.backbone": { region: "west", subnet: 3 },
  "object_store.archive": { region: "west" },
};

export const defaults = (t) => Object.fromEntries(Object.entries(t)
  .filter(([, a]) => a.default !== undefined).map(([f, a]) => [f, a.default]));

export function makeProvider(catalog) {             // interface: create / read / update / delete
  const store = new Map(), calls = { read: 0, create: 0, update: 0, delete: 0 };
  return { catalog, calls, store,
    read: (t, a) => (calls.read++, store.get(`${t}.${a}`) ?? null),
    create: (t, a, fields) => (calls.create++, store.set(`${t}.${a}`, { ...defaults(catalog[t]), ...fields })),
    update: (t, a, fields) => (calls.update++, Object.assign(store.get(`${t}.${a}`), fields)),
    delete: (t, a) => (calls.delete++, store.delete(`${t}.${a}`)) };
}

const OVERSIZED = (a, old, next) => a.type === "number" && next > old * 2;   // growth that cannot happen in place
export function decide(p, key, desired, plan) {  // the plan phase does not know the OVERSIZED rule
  const [t, a] = key.split(".");
  const actual = p.read(t, a);
  if (!actual) return ["create", []];
  const diff = Object.keys(desired).filter((f) => actual[f] !== desired[f]);
  const forcing = diff.filter((f) => !p.catalog[t][f].inPlace ||
    (!plan && OVERSIZED(p.catalog[t][f], actual[f], desired[f])));
  return forcing.length ? ["recreate", forcing]
    : diff.length ? ["update", diff] : ["unchanged", []];
}

export const PAST_TENSE = { create: "created", recreate: "recreated",
  update: "updated", unchanged: "unchanged", delete: "deleted" };

export function apply(p, record) {
  const result = [];
  for (const [k, desired] of Object.entries(record)) {
    const [t, a] = k.split("."), [decision, fields] = decide(p, k, desired, false);
    if (decision === "create") p.create(t, a, desired);
    else if (decision === "recreate") { p.delete(t, a); p.create(t, a, desired); }
    else if (decision === "update") p.update(t, a, Object.fromEntries(fields.map((f) => [f, desired[f]])));
    result.push([k, PAST_TENSE[decision], fields]);
  }
  for (const k of [...p.store.keys()]) if (!(k in record)) { const [t, a] = k.split("."); p.delete(t, a);
    result.push([k, "deleted", []]); }
  return result;
}
```

The provider consists of four operations, and all of them are counted; the record's contact with
the target system is the sum of these four calls. A resource type, in turn, is a field table:
every field's type, whether it can be updated in place, and its default are written down. Of these
three properties, the second sets the record's cost, and the third sets the record's silence. The
decision function uses both: if even one of the changed fields cannot be updated in place, the
resource is deleted and recreated; otherwise a single update call is enough.

#### The Gap Between Plan and Apply

```js
// change.mjs — the catalog is counted, the record applied, then a change set is planned and applied.
import { CATALOG, DOWNTIME, RECORD, makeProvider, decide, apply, PAST_TENSE } from "./provider.mjs";

const allFields = Object.values(CATALOG).flatMap(Object.values);
const print = (...s) => console.log(s.map((x, i) => String(x).padEnd(i ? 22 : 23)).join("").trimEnd());

console.log(`provider v1: ${Object.keys(CATALOG).length} resource types, ${allFields.length} fields ` +
  `(${(allFields.length / Object.keys(CATALOG).length).toFixed(1)} per type); ` +
  `${allFields.filter((a) => a.default === undefined).length} required, ` +
  `${allFields.filter((a) => !a.inPlace).length} fields cannot update in place`);

const narrow = (...s) => console.log(s.map((x) => String(x).padEnd(14)).join("").trimEnd());
narrow("type", "fields", "required", "not in place", "downtime");
for (const [t, f] of Object.entries(CATALOG)) { const v = Object.values(f);
  narrow(t, v.length, v.filter((a) => a.default === undefined).length,
    v.filter((a) => !a.inPlace).length, `${DOWNTIME[t]} min`); }
console.log();

const p = makeProvider(CATALOG);
apply(p, RECORD);
const written = Object.values(RECORD).reduce((t, r) => t + Object.keys(r).length, 0);
const actualFields = [...p.store.values()].reduce((t, r) => t + Object.keys(r).length, 0);
console.log(`record writes ${written} fields for ${Object.keys(RECORD).length} resources; ` +
  `${actualFields} fields exist on target, ${actualFields - written} of them filled by provider defaults`);
console.log(`first apply calls: ${JSON.stringify(p.calls)}\n`);

const RECORD2 = structuredClone(RECORD);              // the change set
RECORD2["machine.collector"].count = 10;
RECORD2["machine.collector"].cores = 8;
RECORD2["machine.verifier"].memory_gb = 48;
RECORD2["queue.meter"].partitions = 16;
RECORD2["data_store.metrics"].size_gb = 600;
delete RECORD2["queue.alert"];
RECORD2["machine.archiver"] = { region: "west", count: 1, cores: 2, memory_gb: 4 };

Object.assign(p.calls, { read: 0, create: 0, update: 0, delete: 0 });
const plan = Object.entries(RECORD2).map(([k, r]) => [k, ...decide(p, k, r, true)])
  .concat([...p.store.keys()].filter((k) => !(k in RECORD2)).map((k) => [k, "delete", []]));
const planReads = p.calls.read;
const result = apply(p, RECORD2);

print("resource", "plan", "apply", "forcing field");
let drift = 0;
for (const [k, decision, fields] of plan) {
  const g = result.find((x) => x[0] === k);
  if (PAST_TENSE[decision] !== g[1]) drift++;
  print(k, decision, g[1], (g[2].length ? g[2] : fields).join(", "));
}
const recreated = (l) => l.filter((x) => x[1].startsWith("recreate")).map((x) => DOWNTIME[x[0].split(".")[0]]);
const rp = recreated(plan), ru = recreated(result);
console.log(`\nplan predicted ${rp.length} recreation, apply did ${ru.length}; ` +
  `${plan.length - drift}/${plan.length} items held`);
console.log(`downtime: ${rp.reduce((a, b) => a + b, 0)} min in plan, ${ru.reduce((a, b) => a + b, 0)} min in apply`);
console.log(`plan did ${planReads} reads; change calls ${JSON.stringify(p.calls)}`);
```

```
provider v1: 5 resource types, 24 fields (4.8 per type); 11 required, 13 fields cannot update in place
type          fields        required      not in place  downtime
machine       7             4             3             4 min
queue         5             2             3             9 min
data_store    5             2             3             25 min
network       3             2             2             12 min
object_store  4             1             2             30 min

record writes 20 fields for 7 resources; 36 fields exist on target, 16 of them filled by provider defaults
first apply calls: {"read":7,"create":7,"update":0,"delete":0}

resource               plan                  apply                 forcing field
machine.collector      recreate              recreated             cores
machine.verifier       update                recreated             memory_gb
queue.meter            recreate              recreated             partitions
data_store.metrics     update                updated               size_gb
network.backbone       unchanged             unchanged
object_store.archive   unchanged             unchanged
machine.archiver       create                created
queue.alert            delete                deleted

plan predicted 2 recreation, apply did 3; 7/8 items held
downtime: 13 min in plan, 17 min in apply
plan did 7 reads; change calls {"read":14,"create":4,"update":1,"delete":4}
```

The first table gives the record's scope. Five types define twenty-four fields in total; eleven
are required, and thirteen cannot be updated in place. The split varies by type, and so does its
cost: three of the machine's seven fields and two of the network's three fields cannot be updated
in place, but recreating a machine takes four minutes and recreating an object store takes thirty.
**The share of fields that cannot be updated in place and the downtime in minutes are two separate
measures, and they need to be read together:** the network's share is high but its resource is
cheap; the object store's share is low, and its resource is expensive.

The second line gives the gap between the record and reality. The record writes twenty fields for
seven resources, but after apply thirty-six fields stand on the target. The provider has filled
the sixteen fields in between with its own defaults, and these values are written nowhere in the
record. Someone reading the record cannot learn from it which image the machines were built with,
whether the queue is encrypted, or how many days the data store is backed up for. The
infrastructure the record does not cover starts here: nearly half the fields of the resources the
same record built run on values outside the record.

The third measure is the plan's predictive accuracy. In an eight-item change set, the plan gets
seven items right and misses one. The item it misses is this: the verifier machines' memory is
raised from sixteen to forty-eight, and the catalog has marked this field as updatable in place, so
the plan says "will update." The target system, however, will not accept growth of more than double
in place, so it deletes the resource and recreates it. The result: the plan predicts two
recreations, apply does three; predicted downtime is thirteen minutes, actual downtime is
seventeen. **Where the plan gets it wrong is where the catalog falls short** — and the catalog
belongs to the provider, not the record.

The last line is the call load. For eight items, the provider takes fourteen reads, four creates,
one update, and four deletes. Half the reads belong to the plan itself: producing a plan means
reading the target system once, start to finish; as the resource count grows, the plan's cost
grows right along with it. The three recreated resources alone account for three of the four
deletes and three of the four creates. **Three fields that cannot be updated in place turn into six
calls and seventeen minutes of downtime.**

#### When the Provider Version Changes

```js
// version.mjs — when the provider version changes: shifting defaults, protected fields, changing decisions.
import { CATALOG, DOWNTIME, RECORD, makeProvider, decide, apply } from "./provider.mjs";

const CATALOG2 = structuredClone(CATALOG);          // provider version 2
CATALOG2.machine.image.default = "base-3";
CATALOG2.queue.encryption.default = "on";
CATALOG2.data_store.backup_days.default = 7;
CATALOG2.object_store.class.default = "cold";
CATALOG2.queue.retention_days.inPlace = false;         // in-place updatability changed too
CATALOG2.object_store.access = { type: "string", inPlace: true, default: "private" };

const setUp = (k) => { const p = makeProvider(k); apply(p, RECORD); return p; };
const a = setUp(CATALOG), b = setUp(CATALOG2), diff = [];
for (const [k, r] of a.store) for (const f of new Set([...Object.keys(r), ...Object.keys(b.store.get(k))]))
  if (r[f] !== b.store.get(k)[f]) diff.push([k, f, r[f], b.store.get(k)[f]]);
console.log(`same record set up with v1 and v2: ${diff.length} fields differ, ` +
  `${new Set(diff.map((x) => x[0])).size}/${a.store.size} resources affected`);
for (const [k, f, x, y] of diff) console.log(`  ${k.padEnd(23)}${f.padEnd(13)}${x} -> ${y}`);

const protectedFields = Object.entries(RECORD).flatMap(([k, r]) => Object.keys(r)
  .filter((f) => CATALOG[k.split(".")[0]][f].default !== CATALOG2[k.split(".")[0]][f].default)
  .map((f) => `${k}.${f}`));
console.log(`fields that do not shift because the record writes them: ${protectedFields.length} (${protectedFields.join(", ")})\n`);

const CHANGE = structuredClone(RECORD);                 // a one-line change in the record
CHANGE["queue.meter"].retention_days = 30;
for (const [name, cat] of [["v1", CATALOG], ["v2", CATALOG2]]) {
  const p = setUp(cat), pl = Object.entries(CHANGE).map(([k, r]) => [k, ...decide(p, k, r, true)]);
  const y = pl.filter((x) => x[1].startsWith("recreate"));
  console.log(`${name}: same change -> ${pl.filter((x) => x[1] === "update").length} update, ` +
    `${y.length} recreate, ${y.reduce((t, x) => t + DOWNTIME[x[0].split(".")[0]], 0)} min downtime`);
}
```

```
same record set up with v1 and v2: 6 fields differ, 5/7 resources affected
  machine.collector      image        base-2 -> base-3
  queue.meter            encryption   off -> on
  queue.alert            encryption   off -> on
  data_store.metrics     backup_days  1 -> 7
  object_store.archive   class        hot -> cold
  object_store.archive   access       undefined -> private
fields that do not shift because the record writes them: 1 (machine.verifier.image)

v1: same change -> 1 update, 0 recreate, 0 min downtime
v2: same change -> 0 update, 1 recreate, 9 min downtime
```

A version change is the record's quietest risk, because nothing in the record changes at all.
When the same seven-resource record is set up with two versions of the provider, the state on the
target diverges in six fields, and five of the seven resources are affected: the machine's image,
the encryption on two queues, the data store's backup frequency, the object store's storage class,
and a brand-new access field the second version adds. The last one also shows that a version
does not just change existing defaults — it can add new fields that have no counterpart anywhere in
the record.

The one field that does not shift is the one explicitly written in the record: the verifier
machines write their own image, so they are not affected by the version change. The rule follows
from this: if the record writes a field, that field is protected; if it does not, it is left to the
provider's version. A record that leaves sixteen fields at their defaults has left sixteen fields
to the provider's release notes. This is the second face of the sixteen from the first section:
the same number is a readability gain in one place and a drift surface in another.

The version also changes the decision itself. The one-line change that raises the queue's
retention from seven days to thirty ends, in the first version, with one update call and zero
minutes of downtime; in the second version, the same line recreates the resource and produces nine
minutes of downtime. The record is the same, the change is the same, the outcome is not. Whether a
field can be updated in place is knowledge that belongs to the provider, not the record; that
knowledge changes with the version, and the only place that reports the change is the plan's
output — a change applied without reading the plan learns this difference only during the
downtime.

#### Summary

- Provider v1's 5 resource types defined 24 fields; 11 were required, 13 could not be updated in
  place. Recreation cost ranged from 4 to 30 minutes by type.
- The record wrote 20 fields for 7 resources, and 36 fields ended up on the target: the provider
  default filled 16 fields, and these values never appeared in the record.
- In an 8-item change, the plan got 7 items right; 2 recreations were predicted, 3 happened, and
  downtime came out to 17 minutes instead of 13. The plan knows only what is written in the catalog.
- Call load came to 14 reads, 4 creates, 1 update, 4 deletes; 7 of the reads belonged to the plan
  itself.
- When the provider version changed, 6 fields drifted with the record completely unchanged, and 5
  of 7 resources were affected; the 1 field explicitly written in the record was protected.
- The same one-line change produced 1 update and 0 minutes in v1, and 1 recreation and 9 minutes of
  downtime in v2.

#### Next Step

Up to this point, the record built every resource on the target. Real environments do not work that
way: part of the network, an identity definition, or a store another team built stands outside the
record, and the record can connect to them only by reading. The next lesson measures that
connection: how many reads the **data sources** that read existing infrastructure make, how often,
how many resources get recomputed when a value they read changes, and why the items the record
does not cover never show up in drift detection.

### Data Sources

Source: https://academia.sh/en/courses/infrastructure-as-code/data-sources

It counts the cost of connecting to infrastructure standing outside the record by reading: how many data sources, how many reads a day, how long they take, and how many resources get recomputed versus recreated when a value they read changes. It measures how the drift scan stays limited to the items the record manages, and why a change outside the record never shows up in that scan.

The previous lesson assumed the record built every resource on the target. In a real environment
that is not true: the network backbone, the base image, identity definitions, certificates, and
regional quotas are mostly in other teams' hands. The record does not create them, delete them, or
update them — it only **reads** them. The structure that reads is called a **data source**, and
its cost accumulates in three separate places: the number and duration of reads made on every plan
run, the recomputation triggered when a value it reads changes, and the drift scan never seeing
these items at all.

The measurement runs on the west environment of the **regional measurement network** and is
**fictional**; the infrastructure outside the record, the data sources, the read durations, and
the dependency propagation are a **model** built with `node`.

**DP13.** The record manages eight resources; five data sources read items outside the record, and
other teams own these items.
**DP14.** When a data source is read, it brings back all of its fields; individual fields cannot be
selected.
**DP15.** Every plan run reads every data source once, and the result is not cached.
**DP16.** The pipeline runs six times a day, and a manual plan runs four.
**DP17.** When a read value changes, every resource that takes it as an input is recomputed; if the
value is in that resource's forcing list, the resource is recreated, and the recreation propagates
downstream.
**DP18.** The drift scan compares only the resources the record manages against their counterparts
on the target.

#### Infrastructure Outside the Record

```js
// external.mjs — fictional infrastructure standing outside the record, and the data sources that read it (model).
export const EXTERNAL = {                                // items other teams manage
  network: { subnet: "subnet-7", mtu: 1500, gateway: "gateway-a" },
  image: { version: "base-2", age_days: 34 },
  identity: { permissions: 12 },
  certificate: { expiry_days: 41 },
  quota: { machine_cap: 24, queue_cap: 16 },
};
export const READ_MS = { network: 120, image: 340, identity: 90, certificate: 210, quota: 60 };
export const OWNER = { network: "network team", image: "image team", identity: "security team",
  certificate: "security team", quota: "cloud ops" };

export function makeReader() {                      // every read is counted, its time totaled
  const counts = {}, elapsed = { ms: 0 };
  return { counts, elapsed,
    read(name) { counts[name] = (counts[name] || 0) + 1; elapsed.ms += READ_MS[name]; return { ...EXTERNAL[name] }; },
    total: () => Object.values(counts).reduce((a, b) => a + b, 0) };
}
```

```js
// record.mjs — resources the record manages; the input list says what they read, forcing
// says which input's change forces the resource to be recreated.
export const RESOURCE = {
  "machine.collector": { input: ["network.subnet", "image.version", "quota.machine_cap"],
    forcing: ["network.subnet", "image.version"] },
  "machine.verifier": { input: ["network.subnet", "image.version"], forcing: ["network.subnet", "image.version"] },
  "queue.meter": { input: ["quota.queue_cap", "network.subnet"], forcing: ["network.subnet"] },
  "queue.alert": { input: ["quota.queue_cap"], forcing: [] },
  "data_store.metrics": { input: ["network.subnet", "machine.collector"], forcing: ["network.subnet"] },
  "network_rule.collector": { input: ["network.gateway", "machine.collector"],
    forcing: ["network.gateway", "machine.collector"] },
  "endpoint.metrics": { input: ["certificate.expiry_days", "data_store.metrics"], forcing: ["data_store.metrics"] },
  "identity_binding.collector": { input: ["identity.permissions", "machine.collector"], forcing: ["machine.collector"] },
};

export function propagate(root) {                        // who gets recomputed when a value changes
  const recomputed = new Set(), recreated = new Set(), worklist = [[root, true]];
  while (worklist.length) {
    const [x, xNew] = worklist.shift();
    for (const [k, r] of Object.entries(RESOURCE)) {
      if (!r.input.includes(x) || recomputed.has(k)) continue;
      recomputed.add(k);
      const kNew = xNew && r.forcing.includes(x);
      if (kNew) recreated.add(k);
      worklist.push([k, kNew]);
    }
  }
  return { recomputed, recreated };
}
```

The two files define two separate worlds. The first holds the items the record does not manage and
each one's owner; the second holds the resources the record manages and which external value each
one reads. The only link between them runs in the direction of reading: the record takes values
from the outside world, it never writes a value to it. This one-way direction is the source of
every measure in this lesson.

#### The Cost and Propagation of Reading

```js
// measure.mjs — the cost of reading, how an external value propagates, and the drift scan's coverage.
import { EXTERNAL, READ_MS, OWNER, makeReader } from "./external.mjs";
import { RESOURCE, propagate } from "./record.mjs";

const externalFields = new Set(Object.entries(EXTERNAL).flatMap(([a, v]) => Object.keys(v).map((f) => `${a}.${f}`)));
const used = new Set(Object.values(RESOURCE).flatMap((r) => r.input).filter((g) => externalFields.has(g)));
const print = (...s) => console.log(s.map((x, i) => String(x).padEnd(i ? 20 : 26)).join("").trimEnd());

console.log(`record manages ${Object.keys(RESOURCE).length} resources; ` +
  `${Object.keys(EXTERNAL).length} data sources bring in ${externalFields.size} external fields, ` +
  `${used.size} of them actually used`);
print("data source", "fields", "used", "read ms", "blocked resources", "owner");
for (const [a, v] of Object.entries(EXTERNAL)) {
  const fields = Object.keys(v).filter((f) => used.has(`${a}.${f}`));
  const blocked = new Set(fields.flatMap((f) => [...propagate(`${a}.${f}`).recomputed]));
  print(a, Object.keys(v).length, fields.length, READ_MS[a], blocked.size, OWNER[a]);
}

const PLANS_PER_DAY = 10;                             // 6 pipeline runs a day + 4 manual plans
const reader = makeReader();
for (let i = 0; i < PLANS_PER_DAY; i++) for (const a of Object.keys(EXTERNAL)) reader.read(a);
console.log(`\n${PLANS_PER_DAY} plans a day: ${reader.total()} external reads, ${(reader.elapsed.ms / 1000).toFixed(1)} s; ` +
  `${Object.keys(EXTERNAL).length} external + ${Object.keys(RESOURCE).length} resource reads per plan`);
console.log(`${reader.total() * 365} external reads a year, ${((reader.elapsed.ms * 365) / 60000).toFixed(0)} minutes`);
const onceMs = Object.values(READ_MS).reduce((a, b) => a + b, 0);
console.log(`cached once a day: ${Object.keys(EXTERNAL).length} reads, ${(onceMs / 1000).toFixed(2)} s; ` +
  `expected time for an external change to surface is ${(24 / PLANS_PER_DAY).toFixed(1)} hours instead of 24`);

console.log(`\n${"changed external value".padEnd(27)}${"reading".padStart(4)}${"recomputed".padStart(18)}${"recreated".padStart(20)}`);
for (const field of [...used].sort()) {
  const { recomputed, recreated } = propagate(field);
  const direct = Object.values(RESOURCE).filter((r) => r.input.includes(field)).length;
  console.log(`  ${field.padEnd(25)}${String(direct).padStart(4)}` +
    `${String(recomputed.size).padStart(18)}${String(recreated.size).padStart(20)}`);
}

const DRIFTED = ["network.subnet", "quota.machine_cap"];     // someone changed the external item
const impact = DRIFTED.reduce((s, a) => { const { recomputed, recreated } = propagate(a);
  recomputed.forEach((x) => s.recomputed.add(x)); recreated.forEach((x) => s.recreated.add(x)); return s; },
  { recomputed: new Set(), recreated: new Set() });
const items = Object.keys(RESOURCE).length + Object.keys(EXTERNAL).length;
console.log(`\ndrift scan compares ${Object.keys(RESOURCE).length}/${items} items ` +
  `(coverage ${Math.round((Object.keys(RESOURCE).length / items) * 100)}%); ` +
  `${Object.keys(EXTERNAL).length} data sources are never compared`);
console.log(`${DRIFTED.length} external fields changed -> the scan reports "no drift"; ` +
  `the next plan recomputes ${impact.recomputed.size} resources, recreates ${impact.recreated.size}`);
```

```
record manages 8 resources; 5 data sources bring in 9 external fields, 7 of them actually used
data source               fields              used                read ms             blocked resources   owner
network                   3                   2                   120                 7                   network team
image                     2                   1                   340                 6                   image team
identity                  1                   1                   90                  1                   security team
certificate               1                   1                   210                 1                   security team
quota                     2                   2                   60                  7                   cloud ops

10 plans a day: 50 external reads, 8.2 s; 5 external + 8 resource reads per plan
18250 external reads a year, 50 minutes
cached once a day: 5 reads, 0.82 s; expected time for an external change to surface is 2.4 hours instead of 24

changed external value     reading        recomputed           recreated
  certificate.expiry_days     1                 1                   0
  identity.permissions        1                 1                   0
  image.version               2                 6                   4
  network.gateway             1                 1                   1
  network.subnet              4                 7                   7
  quota.machine_cap           1                 5                   0
  quota.queue_cap             2                 2                   0

drift scan compares 8/13 items (coverage 62%); 5 data sources are never compared
2 external fields changed -> the scan reports "no drift"; the next plan recomputes 7 resources, recreates 7
```

The first table gives the width of the connection. Five data sources bring back nine external
fields in total, but the record actually uses seven of them: the network's maximum packet size and
the image's age get read but never feed into a resource. This comes from the data source's
inability to select fields — reading happens per item, not per field. Read times are not equal
either: the image source takes three hundred forty milliseconds, the quota source sixty; the most
expensive source is the one that brings back the least-used field.

The second measure is repetition frequency. Ten plan runs a day come to fifty external reads and
8.2 seconds; eighteen thousand two hundred fifty reads and fifty minutes a year. This
duration is small on its own, but it hides two things. First, every plan also reads the eight
managed resources at the same time; the cost of producing a plan grows not with the resource count
but with the resource count plus the data source count. Second, cutting the cost of reading costs
freshness: cached once a day, reads drop from fifty to five and the duration from 8.2
seconds to eight hundred milliseconds, but the expected time for an external change to be noticed
rises from two and a half hours to twenty-four. **Read frequency is directly a freshness measure,
and lowering it lengthens drift's invisibility window.**

The third table gives propagation, and this lesson's harshest number is here. Four resources read
the subnet identifier directly, but when that value changes, seven resources are recomputed and
all seven are recreated — nearly the entire eight-resource record. The reason for the propagation
is a chain: when the subnet changes, the collector machines are recreated; when their identity
changes, the network rule and the identity binding are recreated; when the data store is
recreated, the endpoint is recreated too. The image version recomputes six resources and recreates
four. The quota cap, by contrast, recomputes five resources but recreates none of them; the
certificate and the identity permissions each stay confined to one resource. The same table shows
whether a read value is cheap or expensive: what decides it is not how many resources read it, but
whether the field is on any of their forcing lists.

The fourth column gives outage risk. If the network or quota source becomes unreadable, seven
resources each cannot be planned; six for the image source, one each for identity and certificate.
The record depends on the availability of five items it does not manage, and it does not own that
dependency. The owner column names this directly: network team, image team, security team, cloud
ops. The team authorized to change the record and the team authorized to change the value the
record reads are not the same team.

#### Where the Scan Does not Look

The last two lines count the record's scope. The drift scan compares eight of thirteen items;
coverage is sixty-two percent. The five items it does not compare are the ones the record reads but
does not manage, and the reason they are not compared is simple: the scan compares what the record
says against reality, and the record says nothing at all about these five items. If there is no
desired value in the record, there is no way to say reality has drifted from it either.

The consequence of this is in the last line. The network team changes the subnet identifier, and
cloud ops changes the machine cap. The drift scan runs and reports no drift; all eight resources it
manages match the record, because none of them has changed. At the same moment, the next plan run
is about to recompute seven resources and recreate all seven of them. Two sentences are true for
the same environment at the same time: there is no drift, and seven items of infrastructure are
about to change. The difference is that drift is defined within the record's scope — **a change
the record does not cover is not drift, it is a surprise.**

The practical payoff of this measure is knowing the number of external items and their propagation
radius. Two of the seven used external fields are on their own capable of recreating seven
resources; three trigger no recreation at all. Every new external value that gets read lowers the
scan's coverage and grows the surprise surface; every external value that does not get read has to
be copied into the record by hand instead, and this time it is the copy itself that drifts.

#### Read or Copy

```js
// copy.mjs — reading an external value versus copying it into the record: a fictional 90-day calendar (model).
import { EXTERNAL } from "./external.mjs";
import { RESOURCE, propagate } from "./record.mjs";

const CALENDAR = [                                    // who changed which external item, and when
  [12, "image.version"], [30, "quota.machine_cap"], [40, "network.subnet"],
  [55, "image.version"], [70, "quota.machine_cap"], [78, "network.gateway"],
];
const REFRESH = [30, 60, 90], DAYS = 90;            // days for a manual review of the copy
const externalFields = new Set(Object.entries(EXTERNAL).flatMap(([a, v]) => Object.keys(v).map((f) => `${a}.${f}`)));
const copied = new Set(Object.values(RESOURCE).flatMap((r) => r.input).filter((g) => externalFields.has(g)));

let readRecreations = 0, copyRecreations = 0, mismatchDays = 0, delay = 0;
for (const [day, field] of CALENDAR) {
  const { recomputed, recreated } = propagate(field), t = REFRESH.find((g) => g >= day) ?? DAYS;
  readRecreations += recreated.size; copyRecreations += recreated.size;
  delay += t - day; mismatchDays += (t - day) * recomputed.size;
}
console.log(`${CALENDAR.length} external changes in ${DAYS} days; fields to copy ${copied.size}`);
console.log(`by reading   : 0 days delay, ${readRecreations} recreations, 0 resource-days record-reality mismatch`);
console.log(`by copying   : ${(delay / CALENDAR.length).toFixed(1)} days average delay, ` +
  `${copyRecreations} recreations, ${mismatchDays} resource-days mismatched`);
console.log(`in both approaches the drift scan compares ${Object.keys(RESOURCE).length} items; ` +
  `${Object.keys(EXTERNAL).length} external items stay outside it either way`);
```

```
6 external changes in 90 days; fields to copy 7
by reading   : 0 days delay, 16 recreations, 0 resource-days record-reality mismatch
by copying   : 12.5 days average delay, 16 recreations, 390 resource-days mismatched
in both approaches the drift scan compares 8 items; 5 external items stay outside it either way
```

The alternative to reading is writing external values into the record by hand. The fictional
calendar holds six external changes across ninety days, and both approaches produce the same
sixteen recreations in the end — if the outside world changed, the infrastructure will change
sooner or later too. The difference is in timing. A record that reads picks up the change on the
next plan run; a record that copies depends on a monthly review and lags twelve and a half days
behind on average. The cost of this delay is three hundred ninety resource-days: the total of how
many resources were affected for how long the record misrepresented reality.

The visible gain of copying is zeroing out the read load; its invisible cost is keeping a frozen
copy of seven fields in the record. This copy does enter the drift scan, but what the scan compares
is the record's own copy against the resource built from it — the external item's real value is
never compared in either approach. **Copying does not widen the scan's coverage, it only renames the
drift:** a surprise in the record that reads, a stale record in the one that copies.

#### Summary

- The record managed 8 resources; 5 data sources brought back 9 external fields, and only 7 of them
  were used — 2 fields were read on every plan run but never fed into a resource.
- 10 plan runs a day took 50 external reads and 8.2 seconds; 18,250 reads and 50 minutes a year.
  Caching once a day cuts reads to 5 and raises the invisibility window from 2.4 hours to 24.
- When the subnet identifier changed, 7 of 8 resources were recomputed and 7 were recreated; the
  image version recomputed 6 resources and recreated 4. The quota cap recomputed 5 resources but
  recreated none: what decides it is not the number of readers but whether the field is on the
  forcing list.
- If the network or quota source cannot be read, 7 resources each cannot be planned; the record
  depends on the availability of 5 items owned by other teams.
- The drift scan compared 8 of 13 items (62% coverage). When 2 external fields changed, the scan
  said "no drift" while the next plan was, at the same moment, about to recreate 7 resources.
- 6 external changes across 90 days produced 16 recreations in both approaches; the copying record
  lagged 12.5 days behind on average and misrepresented reality for 390 resource-days.

#### Next Step

Up to this point, the record described a single environment, and the values it read came from
outside. When the same record has to run across three environments, the question changes: which
value should vary by environment, which should stay the same in every one, and who writes that
distinction. The next lesson measures parameterization: when the same definition is applied to
three environments, how many values actually vary by environment, how many stand able to vary when
they should not have to, how the odds of giving a wrong value grow as the input surface grows, and
how many dependencies outputs create in other definitions.

### Variables, Locals, and Outputs

Source: https://academia.sh/en/courses/infrastructure-as-code/variables-locals-and-outputs

The same definition is applied to three environments and parameterization's balance sheet is counted: how many values actually vary by environment, how many are written the same in every environment, how many variables never get used at all. As the input surface grows, the odds of giving a wrong value are measured by how many mistakes validation catches versus misses, and the dependencies outputs create in other definitions are counted.

The previous lessons dealt with the record of a single environment. When the same definition has
to run in three environments, two questions arise: which value should change by environment, and
which should stay the same in every one. The structure that writes this has three parts —
**variables** that take a value from outside, **locals** derived from variables, and **outputs**
the definition hands to the outside. This lesson counts all three: how much of the input surface
is genuinely environment-specific, where the odds of giving a wrong value go as the surface grows,
and how many dependencies outputs create in other definitions.

The measurement runs on three environments of the **regional measurement network** and is
**fictional**; the definition, the validation, and the consuming definitions are a **model** built
with `node`.

**DP19.** The definition carries fourteen variables, three locals, four resources, and five
outputs.
**DP20.** A variable with no default is required; if an environment does not give a value, the
default is used silently.
**DP21.** Validation tests only the type and the value range or set membership; it does not test
whether the value is correct for that environment.
**DP22.** A variable counts as used if it is read directly or through a local; a variable never read
is **dead**.
**DP23.** Ten wrong inputs are chosen in advance; each is a typo or a unit mix-up that actually
happens often.
**DP24.** If an output changes, every definition that reads it gets replanned.

#### The Definition and Three Environments

```js
// definition.mjs — a fictional definition applied to three environments: variables, locals, resources, outputs.
const D = (type, validate, default_) => ({ type, validate, default: default_ });
const oneOf = (...k) => (v) => k.includes(v);
const between = (a, b) => (v) => v >= a && v <= b;

export const VARIABLE = {                           // a variable with no default is required
  region: D("string", oneOf("west", "east", "central")),
  collectors: D("number", between(1, 64)),
  queue_partitions: D("number", between(1, 128)),
  disk_gb: D("number", between(20, 4000)),
  retention_days: D("number", between(1, 365), 30),
  threshold_ms: D("number", between(50, 5000), 900),
  cores: D("number", between(1, 64), 4),
  memory_gb: D("number", between(1, 256), 8),
  format: D("string", oneOf("binary", "text"), "binary"),
  monitoring: D("string", oneOf("on", "off"), "on"),
  backup_days: D("number", between(0, 35), 1),
  tag_owner: D("string", oneOf("metrics-team", "network-team"), "metrics-team"),
  certificate_days: D("number", between(1, 365), 90),
  legacy_flag: D("string", oneOf("on", "off"), "off"),
};

export const LOCAL = {                              // intermediate values derived from variables
  name_prefix: { input: ["region"], compute: (d) => `${d.region}-metrics` },
  total_cores: { input: ["collectors", "cores"], compute: (d) => d.collectors * d.cores },
  queue_name: { input: ["region"], compute: (d) => `${d.region}-meter-queue` },
};

export const RESOURCE = {                           // the variable and local each resource reads
  "machine.collector": ["region", "collectors", "cores", "memory_gb", "disk_gb", "monitoring",
    "tag_owner", "local.total_cores"],
  "queue.meter": ["queue_partitions", "retention_days", "format", "tag_owner", "local.queue_name"],
  "data_store.metrics": ["threshold_ms", "backup_days", "disk_gb", "local.name_prefix"],
  "object_store.archive": ["retention_days", "local.name_prefix"],
};

export const ENVIRONMENT = {                        // the values the three environments give
  west: { region: "west", collectors: 6, queue_partitions: 12, disk_gb: 200, retention_days: 30,
    threshold_ms: 900, cores: 4, memory_gb: 8 },
  east: { region: "east", collectors: 6, queue_partitions: 12, disk_gb: 200, retention_days: 30,
    threshold_ms: 900, cores: 4, memory_gb: 8 },
  central: { region: "central", collectors: 2, queue_partitions: 4, disk_gb: 60, retention_days: 7,
    threshold_ms: 900, cores: 4, memory_gb: 8 },
};

export const OUTPUT = {                             // values other definitions read
  queue_name: "local.queue_name", store_prefix: "local.name_prefix",
  total_cores: "local.total_cores", owner: "tag_owner", archive_days: "retention_days",
};
export const CONSUMER = {                           // which definition reads which output
  "pipeline-definition": ["queue_name", "store_prefix"],
  "alert-definition": ["queue_name", "archive_days"],
  "report-definition": ["store_prefix", "owner"],
};

export function resolve(given) {                    // effective value: given or default
  const d = {};
  for (const [name, t] of Object.entries(VARIABLE)) d[name] = name in given ? given[name] : t.default;
  for (const [name, l] of Object.entries(LOCAL)) d[`local.${name}`] = l.compute(d);
  return d;
}
```

A variable declaration carries three things: type, validation, and default. A local is a
computation, and it names its input by name; resources read both variables and locals. An output,
in turn, is just a routing table — it opens a value inside the definition to the outside by naming
it.

#### The Input Surface's Balance Sheet

```js
// measure.mjs — input surface, values that vary vs. stay fixed by environment, dead variables, output dependencies.
import { VARIABLE, LOCAL, RESOURCE, ENVIRONMENT, OUTPUT, CONSUMER, resolve } from "./definition.mjs";

const ENVS = Object.keys(ENVIRONMENT), resolved = Object.fromEntries(ENVS.map((o) => [o, resolve(ENVIRONMENT[o])]));
const required = Object.entries(VARIABLE).filter(([, t]) => t.default === undefined).map(([a]) => a);
const used = new Set([...Object.values(RESOURCE).flat(),
  ...Object.values(LOCAL).flatMap((l) => l.input)]);
const dead = Object.keys(VARIABLE).filter((a) => !used.has(a));
const given = Object.keys(VARIABLE).filter((a) => ENVS.every((o) => a in ENVIRONMENT[o]));
const varying = given.filter((a) => new Set(ENVS.map((o) => ENVIRONMENT[o][a])).size > 1);

console.log(`variables ${Object.keys(VARIABLE).length}: ${required.length} required, ` +
  `${Object.keys(VARIABLE).length - required.length} defaulted, ${dead.length} dead (${dead.join(", ")})`);
console.log(`three environments write ${given.length * ENVS.length} values: ` +
  `${varying.length * ENVS.length} vary by environment (${varying.join(", ")}), ` +
  `${(given.length - varying.length) * ENVS.length} are the same across all three ` +
  `(${given.filter((a) => !varying.includes(a)).join(", ")})`);
console.log(`variables never given, left at their default: ` +
  `${Object.keys(VARIABLE).length - given.length}\n`);

const print = (...s) => console.log(s.map((x, i) => String(x).padEnd(i ? 17 : 23)).join("").trimEnd());
print("resource", "input", "by environment", "always same");
for (const [k, inputs] of Object.entries(RESOURCE)) {
  const varies = inputs.filter((g) => new Set(ENVS.map((o) => resolved[o][g])).size > 1);
  print(k, inputs.length, varies.length, inputs.length - varies.length);
}

const inputs = [...new Set(Object.values(RESOURCE).flat())];
const varyingInputs = inputs.filter((g) => new Set(ENVS.map((o) => resolved[o][g])).size > 1);
console.log(`  ${inputs.length} unique inputs total, ${varyingInputs.length} of them vary by environment`);

const missing = { ...ENVIRONMENT.central };                  // if the central environment forgets a value
delete missing.retention_days;
const resolved2 = resolve(missing), shifted = Object.keys(resolved.central).filter((k) => resolved.central[k] !== resolved2[k]);
console.log(`\nif central does not give retention_days, the default ${VARIABLE.retention_days.default} takes hold ` +
  `(the environment's value is ${ENVIRONMENT.central.retention_days}): ${shifted.length} values shift, ` +
  `${Object.entries(RESOURCE).filter(([, g]) => g.some((x) => shifted.includes(x))).length} resources affected, ` +
  `validation says nothing at all`);

console.log(`\noutputs ${Object.keys(OUTPUT).length}, consuming definitions ${Object.keys(CONSUMER).length}`);
for (const c of Object.keys(OUTPUT)) {
  const t = Object.entries(CONSUMER).filter(([, l]) => l.includes(c)).map(([a]) => a);
  console.log(`  ${c.padEnd(17)}${t.length} consumers  ${t.join(", ") || "(dead output)"}`);
}
const linked = Object.values(CONSUMER).flat().length;
console.log(`total dependencies ${linked}; if an output's value changes, on average ` +
  `${(linked / Object.keys(OUTPUT).length).toFixed(1)} definitions get replanned`);
```

```
variables 14: 4 required, 10 defaulted, 2 dead (certificate_days, legacy_flag)
three environments write 24 values: 15 vary by environment (region, collectors, queue_partitions, disk_gb, retention_days), 9 are the same across all three (threshold_ms, cores, memory_gb)
variables never given, left at their default: 6

resource               input            by environment   always same
machine.collector      8                4                4
queue.meter            5                3                2
data_store.metrics     4                2                2
object_store.archive   2                2                0
  15 unique inputs total, 8 of them vary by environment

if central does not give retention_days, the default 30 takes hold (the environment's value is 7): 1 values shift, 2 resources affected, validation says nothing at all

outputs 5, consuming definitions 3
  queue_name       2 consumers  pipeline-definition, alert-definition
  store_prefix     2 consumers  pipeline-definition, report-definition
  total_cores      0 consumers  (dead output)
  owner            1 consumers  report-definition
  archive_days     1 consumers  alert-definition
total dependencies 6; if an output's value changes, on average 1.2 definitions get replanned
```

The first line gives parameterization's quiet cost. Four of fourteen variables are required, ten
have defaults; two are **dead** — no resource and no local reads them. A dead variable looks
harmless, but it is part of the input surface: it sits in the documentation, gets read during
review, gets argued over when a new environment is added about whether to fill it in, and changes
nothing.

The second line is this lesson's main measure. Three environments write twenty-four values in
total. Fifteen of these genuinely differ by environment — region, collector count, queue
partitions, disk size, retention. The remaining nine are the same across all three: threshold time,
cores, and memory are written with the same value in every environment. These nine values do not
have to be variables; they could stand inside the definition as a constant or a default. The
difference between them shows up in one thing: each has been hand-written three times, and all
three carry the chance of being written wrong. Six variables, meanwhile, are never given in any
environment and stay at their default; these six are already constants, but they show up in the
record as variables.

The third table gives the load's distribution across resources. Four of the collector machine's
eight inputs vary by environment, four always stay the same; both of the archive object store's
two inputs vary by environment. How much a resource diverges across environments depends on how
many of the inputs it reads actually change — not on the number of resources. In total, four
resources read fifteen unique inputs, and eight of these vary by environment; so the entire
difference between three environments is concentrated at eight points. This is parameterization's
real width, not fourteen variables.

The next line shows the default's silent side. If the central environment forgets to give
retention, the value does not stay empty: the definition's default of thirty days quietly takes
effect, even though that environment wants seven. One value shifts, two resources are set up
wrong, and validation says nothing, because thirty days is flawless on type and range. Rather than
turning a missing input into an error, the default turns it into a silent decision; giving a
default to a variable that should vary by environment is, for this reason, the same as making that
variable no longer required.

The last section counts outputs. Four of five outputs are consumed, one is a **dead output**: no
definition reads the total core count. Three consuming definitions establish six dependencies in
total; the queue name and the store prefix are each read by two definitions. What this means is: a
change to the queue name forces not just its own definition but two other definitions to be
replanned as well. An output becomes a promise the definition makes to the outside, and the cost of
that promise is however many definitions read it.

#### Giving a Wrong Value

```js
// input.mjs — as the input surface grows, giving a wrong value: ten fictional mistakes, how many validation catches.
import { VARIABLE, LOCAL, RESOURCE, ENVIRONMENT } from "./definition.mjs";

const WRONG = [                                    // chosen in advance, not random
  ["region typo", "region", "wes"], ["number instead of string", "collectors", "6"],
  ["extra zero", "collectors", 600], ["hours instead of days", "retention_days", 720],
  ["missing digit", "queue_partitions", 2], ["missing zero", "threshold_ms", 90],
  ["unit mix-up", "disk_gb", 20], ["extra digit", "cores", 40],
  ["out-of-set format", "format", "raw"], ["sign", "memory_gb", -8],
];

const validate = (name, v) => {                        // type and value validation
  const t = VARIABLE[name];
  if (typeof v !== (t.type === "number" ? "number" : "string")) return `${t.type} expected`;
  return t.validate(v) ? null : "value not accepted";
};
const affected = (name) => Object.entries(RESOURCE).filter(([, g]) => g.includes(name) ||
  g.some((x) => x.startsWith("local.") && LOCAL[x.slice(6)].input.includes(name))).map(([k]) => k);

console.log(`input surface: ${Object.keys(ENVIRONMENT).length} environments x ` +
  `${Object.keys(ENVIRONMENT.west).length} given variables = ` +
  `${Object.keys(ENVIRONMENT).length * Object.keys(ENVIRONMENT.west).length} hand-written values\n`);
console.log(`${"mistake".padEnd(29)}${"variable".padEnd(18)}${"value".padEnd(9)}${"validation".padEnd(23)}affected resource`);
let passed = 0;
for (const [name, variable, value] of WRONG) {
  const h = validate(variable, value);
  if (!h) passed++;
  console.log(`  ${name.padEnd(27)}${variable.padEnd(18)}${String(value).padEnd(9)}` +
    `${(h ?? "passed").padEnd(23)}${h ? "" : affected(variable).length}`);
}
console.log(`\n${WRONG.length - passed} of ${WRONG.length} mistakes were caught by validation, ` +
  `${passed} fell through to apply`);
const required = Object.entries(VARIABLE).filter(([, t]) => t.default === undefined).length;
console.log(`everything that is not caught is fine on type and range; the only way to tell it is ` +
  `wrong is from the environment's expectation`);
console.log(`if the input surface grows from ${Object.keys(ENVIRONMENT).length} environments to ${Object.keys(ENVIRONMENT).length * 2}, ` +
  `hand-written values go from ${Object.keys(ENVIRONMENT.west).length * Object.keys(ENVIRONMENT).length} to ` +
  `${Object.keys(ENVIRONMENT.west).length * Object.keys(ENVIRONMENT).length * 2}, and required values ` +
  `from ${required * Object.keys(ENVIRONMENT).length} to ${required * Object.keys(ENVIRONMENT).length * 2}`);
```

```
input surface: 3 environments x 8 given variables = 24 hand-written values

mistake                      variable          value    validation             affected resource
  region typo                region            wes      value not accepted     
  number instead of string   collectors        6        number expected        
  extra zero                 collectors        600      value not accepted     
  hours instead of days      retention_days    720      value not accepted     
  missing digit              queue_partitions  2        passed                 1
  missing zero               threshold_ms      90       passed                 1
  unit mix-up                disk_gb           20       passed                 2
  extra digit                cores             40       passed                 1
  out-of-set format          format            raw      value not accepted     
  sign                       memory_gb         -8       value not accepted     

6 of 10 mistakes were caught by validation, 4 fell through to apply
everything that is not caught is fine on type and range; the only way to tell it is wrong is from the environment's expectation
if the input surface grows from 3 environments to 6, hand-written values go from 24 to 48, and required values from 12 to 24
```

The input surface is twenty-four hand-written values for three environments. Six of ten fictional
mistakes get caught by validation: a misspelled region name, a string given instead of a number,
six hundred written instead of six, hours entered instead of days, an out-of-set format, negative
memory. Four of them, though, **pass** — two written instead of twelve for queue partitions, ninety
instead of nine hundred for the threshold, twenty instead of two hundred for disk, forty instead of
four for cores. What the four have in common is that they are flawless on type and range. Validation
tests a value's **form**, it cannot test whether it is **correct** for that environment; because the
standard of correctness lives not inside the definition but in the environment's expectation.

The impact of the four that pass is also counted: three set up one resource wrong, one sets up two.
How many resources an input touches depends on the resources that read it directly and the
resources that the locals reading it pass through — since disk size feeds into two resources at
once, a single wrong digit shifts both of them together.

The last line shows how the surface grows. If the environment count rises from three to six, hand-written
values go from twenty-four to forty-eight, and required values from twelve to twenty-four. The opportunity
for a wrong entry grows linearly with the environment count, and three of every eight of that growth come
from values written the same across all three environments — that is, values sitting on the surface for no
reason. The cheapest way to shrink the input surface is not writing a new validation, it is taking a value
that does not vary by environment out of being a variable at all.

#### Summary

- The definition carried 14 variables: 4 required, 10 defaulted, **2 dead**. 6 variables were never
  given in any environment and stayed at their default.
- Three environments wrote 24 values; **15 genuinely differed by environment**, 9 were the same
  across all three. Those 9 values did not have to be variables, but they were hand-written three
  times.
- Four resources read 15 unique inputs; the entire difference between three environments was
  concentrated in **8** of these inputs. When the central environment forgot to give a single
  value, the default silently kicked in and set up 2 resources wrong.
- 6 of 10 wrong inputs were caught by validation, **4 passed**; every one of the four was flawless
  on type and range and set up 3 resources with a wrong value in 5 places.
- 4 of 5 outputs were consumed, 1 was a **dead output**; 3 consuming definitions established 6
  dependencies, and two outputs were each read by two definitions.
- If the environment count rises from 3 to 6, hand-written values go from 24 to 48, and required
  values from 12 to 24.

#### Next Step

Variables gave the definition values from outside, locals derived them, outputs opened the result
to other definitions. But these three structures never said one thing: which resource has to be
created before which. If the queue name depends on a machine, the endpoint on a data store, the
network rule on a subnet, apply cannot try them in a random order. The next lesson takes on this
ordering: the difference between the **implicit dependency** born from one resource reading another
resource's value and the hand-written **explicit dependency**, how the ordering gets resolved, and
where apply stalls when the order is wrong.

### Dependencies and Lifecycle

Source: https://academia.sh/en/courses/infrastructure-as-code/dependencies-and-lifecycle

A dependency graph is extracted from the record and a topological order is resolved: how many edges are implicit, how many exist only through an explicit declaration, and how many races a requirement the record never sees produces. Once lifecycle rules are added, the plan's item count, apply's step count, and downtime in seconds are counted separately.

The previous lesson parameterized the definition: the same record became applicable to three
environments, each with its own set of variables. The record now says which resource will exist
with which values. There is one thing it does not say, and that thing gets asked in apply's first
second: in what **order** will these fourteen resources be created.

A declarative record does not write down order. Not writing it does not mean order does not exist — a
subnet cannot exist before its own network. The order is **inferred** from within the record: if a
resource's field uses another resource's output, there is an edge between them. What this lesson
measures is the scope of that inference: how many of the real requirements the record sees, what
happens when it cannot see them, and what every hand-written line to make it see costs.

The measurement again runs on the **regional measurement network**'s record and is **fictional**;
what runs is a **model** built with `node`, and no call goes to a real provider.

**DP25.** A region's record consists of fourteen resources: a network, subnet, security group,
role, grant, object store, config object, data store, queue, two collector machines, load
balancer, health check, and DNS record.
**DP26.** The configuration language's syntax is this lesson's own small form: `resource <type>
"<name>" { field = expression }`. If a field's expression contains `<type>.<name>.<output>`, an
**implicit dependency** is born; a `linked = [...]` list is an **explicit declaration**.
**DP27.** Resources with resolved dependencies are created in the same **round**, i.e. in parallel;
the next round does not start until one round finishes.
**DP28.** The machine pulls its config file from the object store on boot and needs a grant for it.
These two requirements appear in the record only in free text; neither enters an expression.
**DP29.** Durations: creating a machine is 90 seconds, destroying is 10, adding to and removing
from the load balancer is 5. Two machines are full capacity, one machine is half.
**DP30.** Recreating the data store is irreversible: the 41 days of meter readings inside it are
deleted.

#### The Order the Record Builds on Its Own

```js
// graph.mjs — builds a dependency graph from the record text and resolves a topological order. Model;
// no real infrastructure is created. The configuration language's syntax is this lesson's own form.

export const RECORD = `
resource network "metrics" {
  region = "north"
  block  = "10.20.0.0/16"
}
resource object_store "config" {
  name = "metrics-config"
}
resource role "collector" {
  name = "collector-role"
}
resource grant "machine_read" {
  target = object_store.config.name
  to     = role.collector.id
}
resource subnet "collector" {
  network = network.metrics.id
  block   = "10.20.1.0/24"
}
resource security_group "collector" {
  network = network.metrics.id
  ingress = "10.20.1.0/24"
}
resource data_store "meter" {
  subnet   = subnet.collector.id
  security = security_group.collector.id
  size     = "medium"
}
resource queue "batch" {
  subnet = subnet.collector.id
}
resource object "collector_config" {
  store         = object_store.config.name
  data_address  = data_store.meter.address
  queue_address = queue.batch.address
}
resource machine "collector_1" {
  subnet   = subnet.collector.id
  security = security_group.collector.id
  role     = role.collector.id
  boot     = "pull config from the object store"
}
resource machine "collector_2" {
  subnet   = subnet.collector.id
  security = security_group.collector.id
  role     = role.collector.id
  boot     = "pull config from the object store"
}
resource load_balancer "collector" {
  subnet  = subnet.collector.id
  targets = [machine.collector_1.id, machine.collector_2.id]
  linked  = [security_group.collector]
}
resource health_check "collector" {
  target = load_balancer.collector.id
}
resource dns_record "collector" {
  address = load_balancer.collector.address
  linked  = [health_check.collector]
}
`;

// DP28: four requirements the record never sees. The machine pulls its config from the object
// store on boot and needs a grant for it; both live only in the "boot" text, not in an expression.
export const HIDDEN = [
  ["machine.collector_1", "object.collector_config"],
  ["machine.collector_1", "grant.machine_read"],
  ["machine.collector_2", "object.collector_config"],
  ["machine.collector_2", "grant.machine_read"],
];

export function parse(text) {
  const nodes = [], edges = new Map();                    // edge: "a->b" -> {implicit, explicit}
  const block = /resource\s+(\w+)\s+"(\w+)"\s*\{([\s\S]*?)\n\}/g;
  let m;
  while ((m = block.exec(text)) !== null) {
    const id = `${m[1]}.${m[2]}`;
    nodes.push(id);
    const explicit = m[3].match(/linked\s*=\s*\[([^\]]*)\]/);
    const rest = m[3].replace(/linked\s*=\s*\[[^\]]*\]/, "");
    const addEdge = (target, kind) => {
      const a = `${id}->${target}`;
      if (!edges.has(a)) edges.set(a, { implicit: false, explicit: false });
      edges.get(a)[kind] = true;
    };
    for (const r of rest.matchAll(/\b([a-z_]+)\.([a-z_0-9]+)\.[a-z_]+\b/g))
      addEdge(`${r[1]}.${r[2]}`, "implicit");                   // an expression uses another resource's output
    if (explicit)
      for (const r of explicit[1].matchAll(/\b([a-z_]+)\.([a-z_0-9]+)\b/g))
        addEdge(`${r[1]}.${r[2]}`, "explicit");                  // a hand-written declaration
  }
  return { nodes, edges };
}

// Resources with resolved dependencies are created in the same round, i.e. in parallel.
export function levelize({ nodes, edges }) {
  const deps = new Map(nodes.map((d) => [d, []]));
  for (const a of edges.keys()) {
    const [k, h] = a.split("->");
    if (deps.has(k) && deps.has(h)) deps.get(k).push(h);
  }
  const level = new Map();
  let remaining = [...nodes], round = 0;
  while (remaining.length) {
    const ready = remaining.filter((d) => deps.get(d).every((h) => level.has(h)));
    if (!ready.length) return { cycle: remaining, level };
    ready.forEach((d) => level.set(d, round));
    remaining = remaining.filter((d) => !ready.includes(d));
    round++;
  }
  return { cycle: null, level };
}

export function measure(text, hidden = HIDDEN) {
  const g = parse(text), d = levelize(g);
  const counts = { implicit: 0, explicitOnly: 0, both: 0 };
  for (const v of g.edges.values())
    v.implicit && v.explicit ? counts.both++ : v.implicit ? counts.implicit++ : counts.explicitOnly++;
  if (d.cycle) return { g, d, counts, cycle: d.cycle };
  const race = hidden.filter(([a, b]) => d.level.get(a) <= d.level.get(b));
  const round = Math.max(...d.level.values()) + 1;
  const width = Math.max(...[...Array(round).keys()].map((t) =>
    [...d.level.values()].filter((x) => x === t).length));
  return { g, d, counts, race, round, width, cycle: null, lines: text.trim().split("\n").length };
}

if (import.meta.url === `file://${process.argv[1]}`) {
  const print = (title, r) => {
    console.log(`\n${title}`);
    console.log(`  resources ${r.g.nodes.length}, edges ${r.g.edges.size}, record ${r.lines} lines`);
    console.log(`  implicit (from expression) ${r.counts.implicit} | explicit only ${r.counts.explicitOnly}` +
      ` | both ${r.counts.both}`);
    console.log(`  ${r.round} rounds, widest round ${r.width} resources`);
    for (const [a, b] of HIDDEN)
      console.log(`  ${a.padEnd(18)} <- ${b.padEnd(21)} round ${r.d.level.get(a)} vs ` +
        `${r.d.level.get(b)}  ${r.d.level.get(a) <= r.d.level.get(b) ? "RACE" : "order correct"}`);
    console.log(`  race: ${r.race.length}/${HIDDEN.length}`);
  };
  print("A) record as-is — order comes only from implicit dependencies", measure(RECORD));

  const WITH_EXPLICIT = RECORD.replace(/ {2}boot     = "pull config from the object store"/g,
    '  boot     = "pull config from the object store"\n' +
    "  linked   = [object.collector_config, grant.machine_read]");
  print("B) explicit declaration added to both machines (2 lines)", measure(WITH_EXPLICIT));

  const WITH_CYCLE = WITH_EXPLICIT.replace(/ {2}data_address  = data_store.meter.address/,
    "  data_address  = data_store.meter.address\n  linked        = [machine.collector_1]");
  const c = measure(WITH_CYCLE);
  console.log("\nC) 'linked = [machine.collector_1]' added to the config object");
  const mutual = [...c.g.edges.keys()]                 // pair waiting on each other
    .filter((a) => c.g.edges.has(a.split("->").reverse().join("->")));
  console.log(`  still unresolved: ${c.cycle.length} resources`);
  console.log(`  cycle: ${mutual[0].split("->").join(" and ")} are waiting on each other`);
}
```

```
A) record as-is — order comes only from implicit dependencies
  resources 14, edges 23, record 59 lines
  implicit (from expression) 21 | explicit only 2 | both 0
  6 rounds, widest round 4 resources
  machine.collector_1 <- object.collector_config round 2 vs 3  RACE
  machine.collector_1 <- grant.machine_read    round 2 vs 1  order correct
  machine.collector_2 <- object.collector_config round 2 vs 3  RACE
  machine.collector_2 <- grant.machine_read    round 2 vs 1  order correct
  race: 2/4

B) explicit declaration added to both machines (2 lines)
  resources 14, edges 27, record 61 lines
  implicit (from expression) 21 | explicit only 6 | both 0
  8 rounds, widest round 3 resources
  machine.collector_1 <- object.collector_config round 4 vs 3  order correct
  machine.collector_1 <- grant.machine_read    round 4 vs 1  order correct
  machine.collector_2 <- object.collector_config round 4 vs 3  order correct
  machine.collector_2 <- grant.machine_read    round 4 vs 1  order correct
  race: 0/4

C) 'linked = [machine.collector_1]' added to the config object
  still unresolved: 6 resources
  cycle: object.collector_config and machine.collector_1 are waiting on each other
```

The first line gives the ratio: the record found twenty-one of twenty-three edges without anyone
writing them. The subnet writing `network.metrics.id` is enough to say the network has to be
created before the subnet; nobody said "set up the network first." The main source of ordering is
the declaration itself, and it is **free** — a byproduct of an expression that had to be written
anyway.

The two edges that exist only through explicit declaration are meaningful too. The load balancer
does not use the security group's output in any of its fields, but a load balancer opened before
the security rule exists drops traffic; the DNS record does not use the health check's output
either, but if the name is published before the check is set up, requests go to a broken node.
Both are links with **no data flow but a time relationship**, and the only way to write them is by
hand. This is exactly the blind spot of implicit inference.

#### The Requirement the Record Does not See

The next section gives the other side of the ratio. The record knows fourteen resources and
twenty-three edges; in reality there are twenty-seven requirements. Four never entered the record,
because they are written in free text, not in an expression: the machine pulls its config file on
boot and needs a grant for it. The record sees 23 of 27 of reality — eighty-five percent.

Two of the four requirements it does not see resolved into the correct order by coincidence. The
grant is created in round one because it derives from the object store and the role; the machines
are in round two. Nobody wanted this, it just worked out that way. The other two did not work out:
the config object drops into round three because it uses the data store's and the queue's
addresses, while the machines boot in round two. Both machines try to pull a file that does not
exist yet. This is not a bug, it is a **race** — the record left the order of two resources free, and
the provider created them without an order.

The fix is two lines, and section B shows its cost: the race was zeroed out, the edge count rose
from 23 to 27, but the round count went from six to eight, and the widest round's parallelism
dropped from four to three. Because the machines now wait on the config object, the graph's longest
path grew by two rounds. The price of the correct order is a longer apply and narrower
parallelism.

Section C counts explicit declaration's other risk. When a single `linked` line is written in the
wrong direction on the config object, a cycle closes: the config object waits on the machine, the
machine waits on the config object. The resolver cannot produce a single round, and six resources —
the two in the cycle and the four resources depending on them — never get created at all. An
implicit dependency cannot form a cycle, because an expression cannot read an output that does not
exist yet; a cycle can only be built with a hand-written declaration. Explicit declaration closes
what implicit inference cannot see, and in return it opens a new class of failure.

#### How Lifecycle Rules Change the Plan

```js
// lifecycle.mjs — how lifecycle rules change the plan and the apply. Model.
const OP = { remove: 5, destroy: 10, create: 90, add: 5 };   // DP29: seconds

function scenario({ createFirst, parallel }) {
  const steps = [], service = [], assets = [];
  let cursor = 0;
  for (const m of ["collector_1", "collector_2"]) {
    let t = parallel ? 0 : cursor;
    if (createFirst) {
      steps.push([m, "create (new name)", t, OP.create]); assets.push([t, +1]); t += OP.create;
      steps.push([m, "add to target", t, OP.add]); service.push([t + OP.add, +1]); t += OP.add;
      steps.push([m, "remove (old)", t, OP.remove]); service.push([t + OP.remove, -1]); t += OP.remove;
      steps.push([m, "destroy (old)", t, OP.destroy]); assets.push([t + OP.destroy, -1]); t += OP.destroy;
    } else {
      steps.push([m, "remove from target", t, OP.remove]); service.push([t + OP.remove, -1]); t += OP.remove;
      steps.push([m, "destroy", t, OP.destroy]); assets.push([t, -1]); t += OP.destroy;
      steps.push([m, "create", t, OP.create]); assets.push([t, +1]); t += OP.create;
      steps.push([m, "add to target", t, OP.add]); service.push([t + OP.add, +1]); t += OP.add;
    }
    cursor = t;
  }
  const duration = Math.max(...steps.map(([, , t, s]) => t + s));
  let zero = 0, half = 0, mostAssets = 2;
  for (let t = 0; t <= duration; t++) {
    const h = 2 + service.filter(([a]) => a <= t).reduce((s, [, d]) => s + d, 0);
    const v = 2 + assets.filter(([a]) => a <= t).reduce((s, [, d]) => s + d, 0);
    if (h === 0) zero++; else if (h === 1) half++;
    mostAssets = Math.max(mostAssets, v);
  }
  return { steps: steps.length, duration, zero, half, mostAssets };
}

console.log("two collector machines are being recreated (machine type changed)");
console.log("rule                                 steps  time   zero capacity   half  most machines");
for (const [name, k] of [
  ["default, sequential", { createFirst: false, parallel: false }],
  ["default, parallel", { createFirst: false, parallel: true }],
  ["create_first, sequential", { createFirst: true, parallel: false }],
  ["create_first, parallel", { createFirst: true, parallel: true }],
]) {
  const r = scenario(k);
  console.log(`${name.padEnd(35)}${String(r.steps).padStart(7)}${String(r.duration).padStart(6)}` +
    `${String(r.zero).padStart(16)}${String(r.half).padStart(7)}${String(r.mostAssets).padStart(15)}`);
}

// DP30: the second change — the data store's region changed in the record; cannot update in place.
const CHANGE = [
  ["network.metrics", "update", false],
  ["data_store.meter", "recreate", true],        // carries prevent_destroy
  ["machine.collector_1", "recreate", false],
  ["machine.collector_2", "recreate", false],
  ["load_balancer.collector", "update", false],
  ["dns_record.collector", "update", false],
];

const blocked = CHANGE.filter(([, k, e]) => e && k === "recreate");
console.log(`\nplan: ${CHANGE.length} items — ` +
  `${CHANGE.filter(([, k]) => k === "recreate").length} recreate, ` +
  `${CHANGE.filter(([, k]) => k === "update").length} update`);
console.log(`items carrying the prevent_destroy rule: ${blocked.length} ` +
  `(${blocked.map(([a]) => a).join(", ")})`);
console.log(`applied without the rule: ${CHANGE.length}/${CHANGE.length} items, ` +
  `1 unrecoverable (41 days of meter readings)`);
console.log(`applied with the rule: 0/${CHANGE.length} items — the plan is rejected in full; ` +
  `${CHANGE.length - blocked.length} harmless items also wait`);
```

```
two collector machines are being recreated (machine type changed)
rule                                 steps  time   zero capacity   half  most machines
default, sequential                      8   220               0    210              2
default, parallel                        8   110             105      0              2
create_first, sequential                 8   220               0      0              3
create_first, parallel                   8   110               0      0              4

plan: 6 items — 3 recreate, 3 update
items carrying the prevent_destroy rule: 1 (data_store.meter)
applied without the rule: 6/6 items, 1 unrecoverable (41 days of meter readings)
applied with the rule: 0/6 items — the plan is rejected in full; 5 harmless items also wait
```

This table's real finding is the distance between what the plan shows and what apply does. The
plan writes two items: two machines will be recreated. Apply runs **eight steps**, and in the
first row it runs at half capacity for **210 seconds**. Neither the eight steps nor the 210 seconds
show up in the plan; the plan states an item's decision, not its cost.

The four rows scan two axes at once. Parallelism cuts the duration in half — from 220 seconds to
110 — but combined with the default rule it drops service entirely: 105 seconds of zero capacity,
meaning nobody collects meter readings. The `create_first` rule rescues both: the old machine
is not removed until the new one is ready and in the target, and the zero- and half-capacity columns
stay empty. Its bill is in the last column: the number of machines standing at the same time rises
from two to three, and to four under parallel apply. The price of recreating without downtime is
temporarily doubled resources — and the new machine's name has to be generated so its resource
name does not collide.

The lower block measures the second rule. When the data store's region changes in the record, the
plan produces six items; three are recreations, one of them the data store itself. If that item is
applied, the 41 days of meter readings are deleted, never to return. The `prevent_destroy` rule
stops this, but not the one item — **the entire plan**: zero of the six items apply, and the five
harmless items wait too. The record's protection is deducted from the record's fluidity.

#### Summary

- The record knows 14 resources and 23 edges; 21 of the 23 edges are implicit — dependency that is
  a byproduct of an expression, that nobody wrote. The main source of ordering is free.
- The 2 edges that exist only through explicit declaration are time relationships with no data
  flow; this is implicit inference's blind spot, and the only way to write them is by hand.
- Real requirements: 27; what the record sees: 23 — 85 percent. 2 of the 4 unseen requirements
  landed in the correct order by coincidence; 2 produced a race — machines pulled a config file
  that did not exist yet.
- 2 lines of explicit declaration zeroed out the race; in return the round count went from 6 to 8,
  and the widest round's parallelism from 4 to 3. A single declaration written in the wrong
  direction built a cycle, and 6 resources never got created.
- The plan says 2 items, apply runs 8 steps and runs at half capacity for 210 seconds; neither
  shows up in the plan. `create_first` zeroes out the downtime; its cost is 3 machines standing at
  once (4 under parallel).
- `prevent_destroy` saves the 41 days of data but rejects the plan in full: 0 of 6 items apply, and
  the 5 harmless items wait too.

#### Next Step

This lesson measured order and lifecycle, but looking at the record itself, one thing stands out:
the two collector machines were written as two separate blocks, four lines identical letter for
letter. When the region grows and the machine count reaches ten, will this block get copied ten
times? The next lesson writes the same ten resources in three separate forms — by hand, with
**count-based iteration**, with **set-based iteration** — and compares the three on the same
items: how many lines, how many resources get recreated when an element is removed from the
middle, and what readability loses.

### Iteration Constructs

Source: https://academia.sh/en/courses/infrastructure-as-code/iteration-constructs

The same ten resources are written in three record forms: ten blocks by hand, count-based iteration, set-based iteration. Line count, the cost of an exception, and address stability are counted; it measures that the count-based form produces a six-item plan when a subregion is removed from the middle, and that five shifted addresses correspond to thirty-five missed meter-reading batches on the measurement network.

The previous lesson's record had two collector machines, written separately as two blocks, four
lines identical letter for letter. The region has grown: the measurement network now reads meters
in ten separate subregions. Copying the same block ten times is one option; telling the record
itself "do this ten times" is another. This lesson adds a third option and compares all three on
the same items.

The comparison's starting point is this: **all three forms produce the same reality.** Ten
machines, the same subnets, the same sizes, the same tags. The difference between them is not in
reality, it is in the record and in the **address** the record gives a resource. That address is the
only thing that determines which real resource the record means on the next apply — and every
number in this lesson comes from there.

The measurement runs on the **regional measurement network** and is **fictional**; what runs is a
**model** built with `node`.

**DP31.** The measurement network reads meters in ten subregions; the subregion names are in the
order written in the list.
**DP32.** All three record forms define the same ten machines; their attributes are identical.
**DP33.** The address is the key under which a resource sits in state: a block name in the manual
form, an index in the count-based form, a map key in the set-based form.
**DP34.** One subregion carries twice the meters and needs a large machine; this is the only
deviation placed here to measure how the forms carry an exception.
**DP35.** The `subnet` and `tag` fields cannot be updated in place; if they change, the resource is
recreated. `size` updates in place.
**DP36.** Every collector pulls a meter-reading batch every 15 seconds. A recreated machine, with
the previous lesson's durations, is out of service for 105 seconds.

#### The Same Ten Resources, Three Records

```js
// iteration.mjs — ten resources written in three forms; lines, address stability, and the plan are counted. Model.
export const SUBREGION = ["north_1", "north_2", "north_3", "east_1", "east_2",
  "east_3", "west_1", "west_2", "west_3", "west_4"];
const EXCEPTION = { east_2: "large" };            // DP34: one subregion carries twice the meters
const attributes = (b, exc) => ({ subnet: `subnet.${b}`, size: exc[b] ?? "medium",
  tag: `collector-${b}` });

export function manual(subregions, exc = EXCEPTION) {
  const text = subregions.map((b) => `resource machine "collector_${b}" {\n` +
    `  subnet = subnet.${b}.id\n  size   = "${exc[b] ?? "medium"}"\n` +
    `  tag    = "collector-${b}"\n}`).join("\n");
  const resolved = new Map(subregions.map((b) => [`machine.collector_${b}`, attributes(b, exc)]));
  return { text, resolved, branches: 0 };               // value in its own block, no condition
}

export function countBased(subregions, exc = EXCEPTION) {
  const branches = Object.entries(exc).map(([k, v]) =>
    `subregions[count.index] == "${k}" ? "${v}" : `).join("");
  const text = `local subregions = [${subregions.map((b) => `"${b}"`).join(", ")}]\n` +
    `resource machine "collector" {\n  count  = ${subregions.length}\n` +
    `  subnet = subnet[subregions[count.index]].id\n  size   = ${branches}"medium"\n` +
    `  tag    = "collector-" + subregions[count.index]\n}`;
  const resolved = new Map(subregions.map((b, i) => [`machine.collector[${i}]`, attributes(b, exc)]));
  return { text, resolved, branches: Object.keys(exc).length };
}

export function setBased(subregions, exc = EXCEPTION) {
  const text = "local subregions = {\n" +
    subregions.map((b) => `  ${b} = "${exc[b] ?? "medium"}"`).join("\n") + "\n}\n" +
    `resource machine "collector" {\n  set    = subregions\n` +
    "  subnet = subnet[set.key].id\n  size   = set.value\n" +
    '  tag    = "collector-" + set.key\n}';
  const resolved = new Map(subregions.map((b) => [`machine.collector["${b}"]`, attributes(b, exc)]));
  return { text, resolved, branches: 0 };               // exception lives in the map's value
}

const FORCES_RECREATE = ["subnet", "tag"];            // DP35: these fields cannot update in place

export function plan(before, after) {
  const p = { add: 0, update: 0, recreate: 0, delete: 0 };
  for (const [address, y] of after.resolved) {
    const e = before.resolved.get(address);
    if (!e) { p.add++; continue; }
    const diff = Object.keys(y).filter((k) => e[k] !== y[k]);
    if (diff.length) diff.some((k) => FORCES_RECREATE.includes(k)) ? p.recreate++ : p.update++;
  }
  for (const address of before.resolved.keys()) if (!after.resolved.has(address)) p.delete++;
  return p;
}

export const FORM = [["manual, ten blocks", manual], ["count-based", countBased],
  ["set-based", setBased]];
const lineCount = (m) => m.split("\n").length;
const THREE = { east_2: "large", west_1: "large", north_3: "small" };

if (import.meta.url === `file://${process.argv[1]}`) {
  // Do all three forms produce the same ten real resources: compare attributes, not the address.
  const signature = (c) => [...c.values()].map((v) => JSON.stringify(v)).sort().join("|");
  const signatures = FORM.map(([, f]) => signature(f(SUBREGION).resolved));
  console.log("do all three forms produce the same reality: " +
    `${signatures.every((x) => x === signatures[0]) ? "yes" : "no"} ` +
    `(${manual(SUBREGION).resolved.size} resources, same attributes)`);

  console.log("\nthe record's cost (10 resources)");
  console.log(`${"form".padEnd(21)}${"lines".padStart(7)}${"cond branch".padStart(13)}${"lines/branch at 3 exc".padStart(24)}${"subregion in address".padStart(23)}`);
  for (const [name, f] of FORM) {
    const r = f(SUBREGION), u = f(SUBREGION, THREE);
    const addressed = [...r.resolved.keys()].some((a) => a.includes("east_2"));
    console.log(`${name.padEnd(21)}${String(lineCount(r.text)).padStart(7)}${String(r.branches).padStart(13)}` +
      `${`${lineCount(u.text)}/${u.branches}`.padStart(24)}${(addressed ? "yes" : "no").padStart(23)}`);
  }

  const SCENARIO = [
    ["a subregion drops out of the middle (east_2 closes)", SUBREGION.filter((b) => b !== "east_2")],
    ["a subregion is inserted in the middle (east_1b opens)", [...SUBREGION.slice(0, 4), "east_1b", ...SUBREGION.slice(4)]],
    ["a subregion is appended at the end (west_5 opens)", [...SUBREGION, "west_5"]],
  ];

  for (const [name, next] of SCENARIO) {
    console.log(`\n${name}`);
    console.log(`${"form".padEnd(21)}${"add".padStart(5)}${"update".padStart(8)}${"recreate".padStart(10)}${"delete".padStart(8)}${"total items".padStart(13)}${"shifted address".padStart(18)}`);
    for (const [formName, f] of FORM) {
      const before = f(SUBREGION), after = f(next), p = plan(before, after);
      const shifted = [...after.resolved].filter(([a, v]) =>
        before.resolved.has(a) && JSON.stringify(before.resolved.get(a)) !== JSON.stringify(v)).length;
      console.log(`${formName.padEnd(21)}${String(p.add).padStart(5)}${String(p.update).padStart(8)}` +
        `${String(p.recreate).padStart(10)}${String(p.delete).padStart(8)}` +
        `${String(p.add + p.update + p.recreate + p.delete).padStart(13)}${String(shifted).padStart(18)}`);
    }
  }
}
```

```
do all three forms produce the same reality: yes (10 resources, same attributes)

the record's cost (10 resources)
form                   lines  cond branch   lines/branch at 3 exc   subregion in address
manual, ten blocks        50            0                    50/0                    yes
count-based                7            1                     7/3                     no
set-based                 18            0                    18/0                    yes

a subregion drops out of the middle (east_2 closes)
form                   add  update  recreate  delete  total items   shifted address
manual, ten blocks       0       0         0       1            1                 0
count-based              0       0         5       1            6                 5
set-based                0       0         0       1            1                 0

a subregion is inserted in the middle (east_1b opens)
form                   add  update  recreate  delete  total items   shifted address
manual, ten blocks       1       0         0       0            1                 0
count-based              1       0         6       0            7                 6
set-based                1       0         0       0            1                 0

a subregion is appended at the end (west_5 opens)
form                   add  update  recreate  delete  total items   shifted address
manual, ten blocks       1       0         0       0            1                 0
count-based              1       0         0       0            1                 0
set-based                1       0         0       0            1                 0
```

The first line is the comparison's foundation: all three forms produce the ten resources with the
same attributes. The record's form does not change reality.

The lines column gives the first cost. The manual form is 50 lines, set-based is 18, count-based
is 7. 40 of the manual form's 50 lines are repetition: the same three fields ten times, only the
subregion name changing. The count-based form is shortest because it packs the list into a single
line. The set-based form sits in between, and its extra length is not an accident — every line of
the map writes a subregion by its name and its value.

The conditional branch column gives the exception's price. A subregion wanting a large machine is,
in the manual form, a value written on that block's own line; in the set-based form, the value at
that key in the map; both carry zero conditions. In the count-based form, though, a branch enters
the expression. Looking from the last column to the one before it shows how this grows: when the
exception count rises to three, the line and branch counts do not change in the manual and
set-based forms, but in the count-based form, three nested conditions pile up in a single line. In
the count-based form, the place that carries the exception is not the record's data, it is its
expression.

The last column, though, is this lesson's real item. In the manual and set-based forms, the address
carries the subregion name: `machine.collector_east_2` or `machine.collector["east_2"]`. In the
count-based form, the address is `machine.collector[4]`, and which subregion the fourth position
belongs to is not written in the address — that information lives in the list's state **at apply
time**. To look at a resource in state and say which subregion it belongs to, you have to find that
day's version of the record.

What the address is looks like a detail, but in this course's measure it is a central item. When the
record is applied, every resource's real identity is written under an address; on the next apply,
that address is the only thing that matches the record to reality. If the address carries the
resource's identity, the match can be read from the record itself; if it carries an index, the
match depends on the record's order **that day**. In the second case, the link between record and
reality rests on something not written in the record at all — the list's past state.

#### When an Element Drops Out of the Middle

Three scenarios are subsections of the same output, and reading them together makes the rule
plain.

When subregion `east_2` closes, the manual and set-based forms produce a **single-item
plan**: one resource is deleted, the other nine stay in place untouched. The count-based form
produces **six items** for the same real change: five recreations and one delete. The reason is
arithmetic. When the list loses its fourth element, the fifth element shifts to the fourth
position, the sixth to the fifth; the address `machine.collector[4]` pointed to `east_2` yesterday,
today it points to `east_3`. The record says "the fourth machine's subnet and tag changed"; since
these two fields cannot be updated in place, the decision is recreation. The shifted address column
counts this directly: five addresses now point to a different real resource.

Inserting a subregion in the middle is even more expensive: seven items, six shifted addresses.
Every address after the insertion point shifts by one.

The third scenario draws the boundary. When a subregion is appended to the end of the list, all
three forms produce a single item; the count-based form is clean too, because no address shifts.
So count-based iteration is not wrong — it is the correct form for sets whose order carries no
meaning and that only grow from the end. The subregion list is not that kind of set: subregions
close, and new ones get inserted in the middle.

Here, the plan's predictive accuracy asks an inverse question. The plan is not wrong — in the
count-based form it says "five resources will be recreated," and apply recreates exactly five, a
ratio of 5/5. What is wrong is not the plan, it is the expectation of the person reading it: one
subregion closed, one machine should be gone. The expectation is one item, the plan is six. The
five items in between are born not from reality but from the **record's form**.

That the record cannot tell this apart on its own is also a measurable limit. The record cannot see
the difference between "the fifth machine's subnet changed" and "position five is now a different
subregion," because all it has is an address and attributes. Both look like the same thing: an
address, changed attributes. Only the person who wrote the record knows the difference, and even
that person might mistake the five items for a real change while reading the plan. The count-based
form's risk is not hidden — it is written plainly in the plan — but telling apart the plan's six
items from reality's one item is left to the reader.

#### The Shifted Address's Counterpart on the Measurement Network

```js
// cost.mjs — counts a plan item in the measurement network's own unit: missed meter-reading batches. Model.
import { SUBREGION, FORM, plan } from "./iteration.mjs";
const OUTAGE = 105, PERIOD = 15, RECREATE_S = 110, DELETE_S = 10;   // DP36, previous lesson's durations
const next = SUBREGION.filter((b) => b !== "east_2");

console.log("east_2 closed — same real change, three record forms");
console.log(`${"form".padEnd(21)}${"recreate".padStart(9)}${"out-of-service machine-s".padStart(27)}${"missed batch".padStart(15)}${"apply s".padStart(10)}`);
for (const [name, f] of FORM) {
  const p = plan(f(SUBREGION), f(next));
  const s = p.recreate * OUTAGE;
  console.log(`${name.padEnd(21)}${String(p.recreate).padStart(9)}${String(s).padStart(27)}` +
    `${String(p.recreate * Math.floor(OUTAGE / PERIOD)).padStart(15)}` +
    `${String(p.recreate * RECREATE_S + p.delete * DELETE_S).padStart(10)}`);
}
console.log("\nthe closing subregion's own batches were never going to be collected anyway;");
console.log("the loss above belongs to the nine sound subregions.");
```

```
east_2 closed — same real change, three record forms
form                  recreate   out-of-service machine-s   missed batch   apply s
manual, ten blocks           0                          0              0        10
count-based                  5                        525             35       560
set-based                    0                          0              0        10

the closing subregion's own batches were never going to be collected anyway;
the loss above belongs to the nine sound subregions.
```

When the numbers are read in the measurement network's unit, the decision becomes clear. The
closing subregion's machine was going away regardless; the loss above belongs to the nine
subregions standing sound. In the count-based form, five sound machines each go out of service for
105 seconds: 525 machine-seconds and **35 missed meter-reading batches**. In the manual and
set-based forms, these items are zero. Apply time is 560 seconds against 10.

A missed batch does not come back. The meters keep reading, but because nobody pulls during that
window, a 105-second gap opens in those five subregions of the measurement network — and there is
no trace of that gap anywhere in the record. The record is correct, the plan is correct, apply is
correct; the only thing lost is **data**.

This section should close by naming an item the record does not cover: the subregion list itself.
Ten subregion names are written in the record, but which subregion has meters is not something the
record knows — it was hand-copied from a list the field team keeps. When a subregion closes, the
record does not learn this on its own; someone goes and deletes it from the list. The trigger for
this change — the record's most expensive item — sits entirely outside the record.

#### The Record Form Decision

| Form | Lines (10 resources) | Items on removal from the middle | Shifted address | Exception branch | When |
|---|---|---|---|---|---|
| Manual, ten blocks | 50 | 1 | 0 | 0 | Resources genuinely differ from one another |
| Count-based | 7 | 6 | 5 | 1 per exception | Order is meaningless, a set that only grows from the end |
| Set-based | 18 | 1 | 0 | 0 | A set whose elements have identity |

The table is summarized by a single measure: **the price of saving lines is address stability.**
The count-based form brings the manual form's 50 lines down to 7 — sevenfold — and in return
produces five to six needless recreations on every change in the middle. The set-based form comes
down to a third of the manual form at 18 lines and never loses address stability; the price it pays
is that subregion names get written a second time in the record, as map keys.

The decision is not between three forms, it is between two questions. First: are these ten resources
copies of one another, or does each have its own identity? On the measurement network, the second
is true — `east_2` is not an index, it is a subregion standing in the field. Second: does this set
change from the middle? The subregion list changes through closings and openings, so it does. The
two answers together point to the set-based form; the count-based form is not cheap enough here to
make its 7 lines worth choosing over 18.

#### Summary

- All three record forms produce the same 10 real resources with the same attributes; the
  difference is not in reality, it is in the address the record gives a resource.
- Lines: manual 50, set-based 18, count-based 7. When the exception count rises to 3, the
  count-based form piles up 3 nested conditions in a single expression, while the branch count
  stays 0 in the other two.
- When east_2 closed, the manual and set-based forms produced 1 item, the count-based form
  produced 6; 5 addresses shifted to a different real resource. Inserting a subregion in the
  middle came to 1 versus 7 and 6.
- Appending at the end came to 1 item in all three forms: count-based iteration is the right form
  for sets whose order carries no meaning and that only grow from the end.
- The plan was not wrong — it said "5 will be recreated" and 5 were recreated, a ratio of 5/5. The
  5-item gap between expectation and plan was born from the record's form.
- In the measurement network's unit: 5 sound machines out of service for 105 seconds each, 525
  machine-seconds, 35 missed meter-reading batches, a 560-second apply — in the set-based form,
  all of that is 0 and 10 seconds.

#### Next Step

This lesson counted plan items but always took the plan as a given: the items came from somewhere.
The next lesson builds that somewhere — it writes the **planner** itself, the thing that compares
the record against state and produces an add, update, recreate, or delete decision item by item.
What gets measured is the plan's own accuracy: how many items are not in the plan but happen at
apply, how many items are in the plan but do not happen, and how many items marked "will be
recreated" are actually recreated.

### Plan and Apply Workflow

Source: https://academia.sh/en/courses/infrastructure-as-code/plan-and-apply-workflow

A planner is written that compares the record against state and produces an add, update, recreate, or delete decision item by item, and then the plan itself is measured: how many items are not in the plan but happen at apply, how many are in the plan but do not happen, how many items marked for recreation actually get recreated, and how many lines and minutes it takes to review the plan.

The previous lesson counted plan items according to the record's form, but it always took the plan
itself as a given: the items came from somewhere. This lesson builds that somewhere. The
**planner** is declarative provisioning's most concrete promise — a list that shows what will
happen before a change is applied. The way to measure that promise is clear too: produce the list,
then apply, then compare the two.

Earlier, a continuously running **reconciliation loop** was measured: a controller that compares
the desired state against the real state every round and closes the gap on its own. The flow here
is its opposite. Apply is **triggered**; in the time between two triggers, reality changes on its
own, without asking anyone. The plan gets produced right in that gap, and every deviation in this
lesson comes out of that gap.

The measurement runs on the **regional measurement network**'s record and is **fictional**; what
runs is a **planner model** written with `node`, and no call goes to any provider.

**DP37.** The planner compares the record against **state**, not reality. State is the ledger the
last apply wrote; reality is what is standing at the provider right now. The two are separate
sources.
**DP38.** State carries thirteen resources and is left over from the last apply.
**DP39.** The record diverges from state at eleven points: one new resource, one deleted resource,
three image upgrades, and six field changes.
**DP40.** Reality has drifted from state at three points; all three are out-of-record, hand-made
changes.
**DP41.** The planner's schema and the running provider's behavior diverge at three points: the
planner counts a `size` change as recreation, but the provider updates that field in place; the
provider stores strings without case sensitivity; and it adds a field to a newly created machine
that is not in the record. A computed field's value is not known before apply; it can only be
predicted from whether the resource it looks at is being recreated.
**DP42.** A plan item writes one title line; an add, delete, or recreate item lists the resource's
entire field set, an update item lists only the changed fields. Careful reading speed is 25 lines a
minute.

#### The Planner

```js
// plan.mjs — a planner that compares the record against state and produces a plan, and a
// measurement that compares the plan against the real apply. Model; no call goes to a provider.

const REF = (resource) => ({ ref: resource });      // a computed field: its value is not known before apply

// DP38: state — the record the last apply wrote. Thirteen resources.
const STATE = {
  "network.metrics": { block: "10.20.0.0/16", region: "north" },
  "subnet.collector": { network: "network-01", block: "10.20.1.0/24" },
  "security_group.collector": { network: "network-01", ingress: "10.20.1.0/24" },
  "data_store.meter": { subnet: "subnet-01", size: "medium", version: "4.2" },
  "queue.batch": { subnet: "subnet-01", retention: 3 },
  "queue.old_batch": { subnet: "subnet-01", retention: 3 },
  "machine.collector_north_1": { subnet: "subnet-01", image: "base-7", size: "medium" },
  "machine.collector_north_2": { subnet: "subnet-01", image: "base-7", size: "medium" },
  "machine.collector_east_1": { subnet: "subnet-01", image: "base-7", size: "medium" },
  "object.collector_config": { store: "metrics-config", data_address: "meter-01.data" },
  "load_balancer.collector": { subnet: "subnet-01", algorithm: "round_robin" },
  "health_check.collector": { target: "load_balancer-01", interval: 30 },
  "dns_record.collector": { address: "load_balancer-01.address", ttl: 300 },
};

// DP39: record — desired. Diverges from state at eleven points.
const RECORD = {
  ...STATE,
  "data_store.meter": { subnet: "subnet-01", size: "large", version: "4.2" },
  "queue.batch": { subnet: "subnet-01", retention: 7 },
  "machine.collector_north_1": { subnet: "subnet-01", image: "base-9", size: "medium" },
  "machine.collector_north_2": { subnet: "subnet-01", image: "base-9", size: "medium" },
  "machine.collector_east_1": { subnet: "subnet-01", image: "base-9", size: "medium" },
  "machine.collector_east_2": { subnet: "subnet-01", image: "base-9", size: "medium" },
  "object.collector_config": { store: "metrics-config", data_address: REF("data_store.meter") },
  "load_balancer.collector": { subnet: "subnet-01", algorithm: "ROUND_ROBIN" },
  "health_check.collector": { target: "load_balancer-01", interval: 10 },
  "dns_record.collector": { address: "load_balancer-01.address", ttl: 60 },
};
delete RECORD["queue.old_batch"];

// DP40: actual — what is really sitting at the provider. Diverges from state at three points; the plan never sees these.
const ACTUAL = {
  ...STATE,
  "network.metrics": { block: "10.20.0.0/17", region: "north" },          // manually narrowed
  "security_group.collector": { network: "network-01", ingress: "0.0.0.0/0" },  // manually opened
  "health_check.collector": { target: "load_balancer-01", interval: 5 },  // manually lowered
};

const RECREATE_PLAN = ["image", "size", "subnet"];   // DP41: the schema the planner has
const RECREATE_ACTUAL = ["image", "subnet"];          // the provider updates size in place
const NORMALIZE = (v) => (typeof v === "string" ? v.toLowerCase() : v);
const DEFAULT = { machine: { monitoring: "on" } };  // the provider adds this to a new resource on its own
const K = (v) => JSON.stringify(v);

// Shared comparator. real=false looks through the planner's eyes, true through apply's.
function compare(record, baseline, { real }) {
  const forcing = real ? RECREATE_ACTUAL : RECREATE_PLAN;
  const equal = real ? (a, b) => K(NORMALIZE(a)) === K(NORMALIZE(b)) : (a, b) => K(a) === K(b);
  const items = new Map();
  for (const [address, y] of Object.entries(record)) {
    const e = baseline[address];
    const total = Object.keys(y).length;
    if (!e) { items.set(address, { decision: "add", fields: Object.keys(y), total }); continue; }
    const fields = Object.keys(y).filter((k) => !y[k]?.ref && !equal(y[k], e[k]));
    const decision = fields.length
      ? (fields.some((k) => forcing.includes(k)) ? "recreate" : "update") : null;
    items.set(address, { decision, fields, total });
  }
  for (const address of Object.keys(baseline))
    if (!record[address]) items.set(address, { decision: "delete", fields: [], total: Object.keys(baseline[address]).length });
  // A computed field: its value changes if the resource it looks at is being recreated.
  for (const [address, y] of Object.entries(record))
    for (const [k, v] of Object.entries(y))
      if (v?.ref && items.get(v.ref)?.decision === "recreate") {
        const it = items.get(address);
        it.fields.push(k);
        it.decision = it.fields.some((a) => forcing.includes(a)) ? "recreate" : "update";
      }
  for (const [address, it] of items) if (!it.decision) items.delete(address);
  return items;
}

const PLAN = compare(RECORD, STATE, { real: false });
const APPLY = compare(RECORD, ACTUAL, { real: true });
// Provider default: an out-of-plan field gets added when a new resource is created.
let defaultFields = 0;
for (const [address, it] of APPLY)
  if (it.decision === "add") defaultFields += Object.keys(DEFAULT[address.split(".")[0]] ?? {}).length;

const count = (m, k) => [...m.values()].filter((v) => v.decision === k).length;
console.log("decision       plan  apply");
const print = (name, a, b) =>
  console.log(`${name.padEnd(15)}${String(a).padStart(4)}${String(b).padStart(10)}`);
for (const k of ["add", "update", "recreate", "delete"])
  print(k, count(PLAN, k), count(APPLY, k));
print("total", PLAN.size, APPLY.size);

const notInPlan = [...APPLY.keys()].filter((a) => !PLAN.has(a));
const notInApply = [...PLAN.keys()].filter((a) => !APPLY.has(a));
const common = [...PLAN.keys()].filter((a) => APPLY.has(a));
const decisionMismatch = common.filter((a) => PLAN.get(a).decision !== APPLY.get(a).decision);
const planRecreate = [...PLAN].filter(([, v]) => v.decision === "recreate").map(([a]) => a);
const actualRecreate = planRecreate.filter((a) => APPLY.get(a)?.decision === "recreate");
const staleValueWrong = common.filter((a) => STATE[a] && ACTUAL[a] && K(STATE[a]) !== K(ACTUAL[a]));

console.log("\nthe plan's accuracy at predicting the apply");
console.log(`  not in plan, happened in apply  : ${notInPlan.length}  (${notInPlan.join(", ")})`);
console.log(`  in plan, did not happen in apply: ${notInApply.length}  (${notInApply.join(", ")})`);
console.log(`  in both                         : ${common.length}, decision mismatch on ` +
  `${decisionMismatch.length} (${decisionMismatch.join(", ")})`);
console.log(`  recreation hit rate             : ${actualRecreate.length}/${planRecreate.length}`);
console.log(`  plan's stated old value is wrong: ${staleValueWrong.length} item ` +
  `(${staleValueWrong.join(", ")})`);
console.log(`  field never shown in the plan   : ${defaultFields} (provider default)`);
const intersection = common.length - decisionMismatch.length;
const union = new Set([...PLAN.keys(), ...APPLY.keys()]).size;
console.log(`  exact hit ${intersection}/${union} = ${Math.round((intersection / union) * 100)}%`);

// DP42: a plan item writes 1 title line; the add/delete/recreate item lists the resource's
// full field set, the update item lists only the changed fields. Careful reading is 25 lines/minute.
const FULL_LISTING = ["add", "delete", "recreate"];
const lineCount = (m) => [...m.values()].reduce((s, v) =>
  s + 1 + (FULL_LISTING.includes(v.decision) ? v.total : v.fields.length), 0);
const decisionLines = (m) => [...m.values()]
  .filter((v) => v.decision === "recreate" || v.decision === "delete")
  .reduce((s, v) => s + 1 + v.total, 0);

// The same record repeated across N regions: addresses and computed-field references get prefixed.
const scaleOut = (obj, n) => Object.fromEntries(Array.from({ length: n }, (_, i) =>
  Object.entries(obj).map(([address, y]) => [`r${i}_${address}`, Object.fromEntries(
    Object.entries(y).map(([k, v]) => [k, v?.ref ? REF(`r${i}_${v.ref}`) : v]))])).flat());

console.log("\nthe cost of reading the plan");
console.log("region  items  lines  minutes  decision lines   %");
for (const n of [1, 3, 10, 30]) {
  const pl = compare(scaleOut(RECORD, n), scaleOut(STATE, n), { real: false });
  const t = lineCount(pl), kr = decisionLines(pl);
  console.log(`${String(n).padStart(6)}${String(pl.size).padStart(7)}${String(t).padStart(7)}` +
    `${(t / 25).toFixed(1).padStart(9)}${String(kr).padStart(17)}` +
    `${String(Math.round((kr / t) * 100)).padStart(4)}`);
}
```

```
decision       plan  apply
add               1         1
update            5         6
recreate          4         3
delete            1         1
total            11        11

the plan's accuracy at predicting the apply
  not in plan, happened in apply  : 2  (network.metrics, security_group.collector)
  in plan, did not happen in apply: 2  (object.collector_config, load_balancer.collector)
  in both                         : 9, decision mismatch on 1 (data_store.meter)
  recreation hit rate             : 3/4
  plan's stated old value is wrong: 1 item (health_check.collector)
  field never shown in the plan   : 1 (provider default)
  exact hit 8/13 = 62%

the cost of reading the plan
region  items  lines  minutes  decision lines   %
     1     11     33      1.3               19  58
     3     33     99      4.0               57  58
    10    110    330     13.2              190  58
    30    330    990     39.6              570  58
```

The top table is reassuring at first glance: the plan has eleven items, apply has eleven items.
The numbers hold. The section below removes that comfort — **two lists of the same size are not
the same list.** Plan and apply agree on both item and decision for eight of thirteen distinct
addresses. The exact hit rate is 62 percent.

The source of the deviation is a single structural choice: the planner compares the record against
state, while apply touches reality. Every difference between state and reality cannot enter the
plan, **by definition**. Three items on the measurement network are like this: the network block
has been narrowed by someone, the security group's ingress rule has been opened all the way, the
health check's interval has been manually lowered. The plan writes none of them. Apply silently
reverts the first two — two items the plan never showed — and for the third it shows the right item
but the **wrong old value**: the plan says "30 to 10," reality goes from 5 to 10.

This third type is especially insidious, because it does not throw off the item count. The person
reading the plan sees the right number of items, sees the right decisions, and is only misled about
where each item **starts from**. The security group's being opened all the way also gets reverted
this way: it is corrected without anyone asking, closed without anyone noticing, and how long that
rule stayed open is written nowhere.

#### Items in the Plan That Do not Happen

The two items in the opposite direction come from a different mechanism. `load_balancer.collector`
is in the plan because the record writes the algorithm value in uppercase, and the planner compares
strings literally. Because the provider stores the same value without case sensitivity, nothing
happens at apply. The plan promised a change, apply did zero work.

This item looks harmless, but it has a cost: it comes back on every plan run. A gap between the
record and state that never closes becomes a permanent, unreconciled item, and the person reading
the plan has to review it every single time. Over time, items like this get learned as "always
shows up, ignore it" — and that habit dismisses a **real** item in the plan with the same reflex.

The second item runs deeper. `object.collector_config`'s record carries the data store's address,
and this address is a **computed field**: its value is known only after apply. The planner cannot
predict this field; it knows only one thing — if the resource it looks at is going to be recreated,
the address will change too. The data store is a recreation item in the plan, so the config object
enters the plan too. At apply, the data store is not recreated, its address does not change, and the
config object is untouched. Here the planner did not write wrong data — it wrote the **consequence
of a wrong assumption**.

#### Items Marked for Recreation

The recreation row is this course's most expensive decision: destroying and recreating is
irreversible and, as measured in the previous lesson, produces downtime. The plan's hit rate on
this item is three of four.

The missed item is the data store, and the reason is not a record error. The schema in the
planner's hands says the `size` field requires recreation; the running provider, though, can update
that field in place. This single-line gap between schema and behavior showed the person looking at
the plan that forty-one days of meter data would be deleted. The direction of the error is lucky —
the plan promised something worse than what actually happened — but direction is not a guarantee; a
schema gap in the opposite direction turns an item the plan marks "will change in place" into a
recreation at apply.

This single item's ripple effect enters the count too: the **only** reason the config object
entered the plan was the mistaken belief that the data store would be recreated. One wrong
recreation decision produced at least one more item beyond itself. This multiplier grows as the
number of resources looking at a computed field grows.

There is also an item that never shows up at all. The provider adds a field to the newly created
machine that is not written in the record. The plan says it will create a resource with three
fields; what comes out is a resource with four. The fourth field does not enter the plan on the next
apply either, because the record does not know about it at all. The infrastructure the record
does not cover is, here, exactly one field wide, and nobody manages that field.

#### The Cost of Reading the Plan

The bottom table gives the plan's own cost. For a single region, the plan is 33 lines and 1.3
minutes with careful reading. This is text that can be reviewed. When the measurement network grows
to thirty regions, the same plan rises to 990 lines and 39.6 minutes — and this is the plan for a
**single image upgrade**.

It is worth noting that the ratio stays fixed: the lines requiring a decision — that is, recreation
and delete items — are 58 percent of the plan at every scale. As the plan grows, the signal ratio
does not drop, but the **absolute review load** grows linearly. In a thirty-region plan, 570 lines
carry an irreversible decision, and every one of them scrolls by on the same screen, in the same
format, in the same color.

The operating rule that follows from this rests on the number. Nobody reads a forty-minute plan
line by line; and a plan that is not read does not deliver its promise. For the plan to stay
reviewable, either the record's unit of apply gets shrunk — thirty regions applied region by
region, not all at once — or only the recreation and delete items get filtered out of the plan and
read first. Both are about **splitting the record**, and that topic is outside this lesson's
boundary.

Finally, the plan's predictive accuracy and the cost of reading it should be read together. A list
with 62 percent accuracy does not describe the whole of apply even when read for thirty minutes.
**The plan's value is not certainty, it is decidability**: showing the irreversible items before
they are applied. It does that job with three-of-four accuracy, and the reason for the quarter it
misses is not in the record — it is in the gap between the planner's schema and the provider's
behavior.

#### Summary

- The planner compares the record against state, apply touches reality; every difference between
  state and reality cannot enter the plan by definition. This single structural choice is the
  source of most of the deviations.
- The plan has 11 items, apply has 11 items, but item and decision agree on only 8 of 13
  addresses: exact hit rate 62 percent.
- Not in the plan, happened in apply: 2 items (a manually narrowed network block, an ingress rule
  opened all the way). 1 more item looked correct but showed the wrong old value: the plan says
  30, reality is 5.
- In the plan, not in apply: 2 items — one from case normalization (comes back on every run), one
  from a computed field (the ripple effect of a wrong recreation assumption).
- Recreation hit rate: 3/4. The missed item came from a one-field gap between the planner's schema
  and the provider's behavior, and it alone produced one more item.
- The plan is 33 lines and 1.3 minutes for 1 region, 990 lines and 39.6 minutes for 30; the share
  of lines requiring a decision stays at 58 percent at every scale, meaning the review load grows
  linearly.

#### Next Step

This lesson measured the plan for resources written in the record. On the measurement network,
though, there are also resources never written in the record at all: an object store opened by
hand at initial setup, a DNS zone, two old machines. They never enter the plan, because the planner
only compares what the record and state know. The next lesson brings these resources under
management — **import** — and counts its cost: how many fields get hand-matched per item, how many
items the first plan after import shows as needless changes, and what the difference is between
leaving a resource out of the record and bringing it under management.

### Resource Modification and Import

Source: https://academia.sh/en/courses/infrastructure-as-code/resource-modification-and-import

Seven out-of-record resources are brought under management and import's cost is counted: how many fields get hand-matched per item, how many fields stay out of the record, how many needless changes the first plan after import shows. Resource modification's downtime and the fields lost to recreation under partial import are measured.

The previous lesson measured the plan for resources written in the record. On the measurement
network, though, there are also resources never written in the record at all: an object store
opened by hand at initial setup, a DNS zone, two old collector machines, a default security group,
an old data store, and a queue. These seven resources never enter a plan, their drift never shows,
and they never pass through a review. The planner does not see them, because the planner only
compares what the record and state know.

This lesson's job is bringing those seven under management. The operation has two parts, and both
are measured separately: matching a resource's real identity to an address, and hand-writing that
resource's counterpart in the record. The first is mechanical. The second is a **copying** job, and
the copying job's error rate is this lesson's central number.

The measurement runs on the **regional measurement network** and is **fictional**; what runs is a
**model** built with `node`.

**DP43.** The seven resources sitting at the provider are resources no record knows about.
**DP44.** The record is written by hand; values are read from the provider interface. It is wrong in
four places: two values were misread, one value was assumed to be the default, one value was
written in a different letter case.
**DP45.** Import writes the entirety of the resource's real fields to state; only the hand-written
fields enter the record. The gap between the two is the fields left out of the record.
**DP46.** The planner and the provider carry over the previous lesson's rules: `image`, `size`, and
`subnet` require recreation, and the provider stores strings without case sensitivity.
**DP47.** Recreation downtime is 105 seconds for a machine, 900 for a data store; a meter-reading
batch is missed every 15 seconds.
**DP48.** A recreated resource loses the fields not written in the record and reverts to the
provider default. Computed fields the provider generates on its own are outside this.

#### Bringing Seven Resources Under Management

```js
// import.mjs — bringing out-of-record resources under management, and measuring the first plan
// after import. Model; no call goes to a real provider.

// DP43: seven resources sitting at the provider that no record knows about. Opened by hand at initial setup.
const ACTUAL = {
  "object_store.old_metrics": { name: "old-metrics", region: "north",
    version_protection: "on", encryption: "provider" },
  "dns_zone.metrics": { name: "metrics.internal", ttl: 3600, record_count: 41 },
  "machine.old_collector_1": { subnet: "subnet-01", image: "base-4", size: "large",
    tag: "collector-old-1" },
  "machine.old_collector_2": { subnet: "subnet-01", image: "base-4", size: "large",
    tag: "collector-old-2" },
  "security_group.default": { network: "network-01", ingress: "10.20.0.0/16", egress: "0.0.0.0/0" },
  "data_store.old_meter": { subnet: "subnet-01", size: "medium", version: "3.9", backup: "daily" },
  "queue.old_stream": { subnet: "subnet-01", retention: 3, type: "fifo" },
};

// DP44: the record is written by hand; values are read from the provider interface. Wrong in four places.
const MANUAL = {
  "object_store.old_metrics": { name: "old-metrics", region: "north" },
  "dns_zone.metrics": { name: "metrics.internal", ttl: 300 },                    // assumed to be the default
  "machine.old_collector_1": { subnet: "subnet-01", image: "base-4-x86", size: "large",
    tag: "collector-old-1" },                                       // misread
  "machine.old_collector_2": { subnet: "subnet-01", image: "base-4", size: "large",
    tag: "collector-old-2" },
  "security_group.default": { network: "network-01", ingress: "10.20.1.0/24" }, // misread
  "data_store.old_meter": { subnet: "subnet-01", size: "MEDIUM", version: "3.9" },  // letter case
  "queue.old_stream": { subnet: "subnet-01", retention: 3, type: "fifo" },
};

const COMPUTED = ["record_count"];            // the provider produces this, it is not written to the record
const FORCES_RECREATE = ["image", "size", "subnet"];     // DP46: previous lesson's schema rules
const DOWNTIME = { machine: 105, data_store: 900 };  // DP47: recreation downtime, seconds
const NORMALIZE = (v) => (typeof v === "string" ? v.toLowerCase() : v);
const K = (v) => JSON.stringify(v);

// DP45: import writes the entirety of reality's fields to state; only what is written enters the record.
const STATE = structuredClone(ACTUAL);

function plan(record, baseline, { normalize }) {
  const equal = normalize ? (a, b) => K(NORMALIZE(a)) === K(NORMALIZE(b)) : (a, b) => K(a) === K(b);
  const items = new Map();
  for (const [address, y] of Object.entries(record)) {
    const fields = Object.keys(y).filter((k) => !equal(y[k], baseline[address][k]));
    if (fields.length)
      items.set(address, { fields,
        decision: fields.some((k) => FORCES_RECREATE.includes(k)) ? "recreate" : "update" });
  }
  return items;
}

const written = (address) => Object.keys(MANUAL[address]).length;
const manageable = (address) =>
  Object.keys(ACTUAL[address]).filter((k) => !COMPUTED.includes(k)).length;
const uncovered = (address) => manageable(address) - written(address);
const addresses = Object.keys(ACTUAL);

console.log("imported resources");
console.log("address                      real fields  hand-written  uncovered  first plan");
const PLAN = plan(MANUAL, STATE, { normalize: false });
const APPLY = plan(MANUAL, ACTUAL, { normalize: true });
for (const address of addresses)
  console.log(`${address.padEnd(28)}${String(manageable(address)).padStart(11)}` +
    `${String(written(address)).padStart(14)}${String(uncovered(address)).padStart(13)}` +
    `  ${PLAN.get(address) ? `${PLAN.get(address).decision} (${PLAN.get(address).fields.join(",")})` : "-"}`);

const total = (f) => addresses.reduce((s, a) => s + f(a), 0);
console.log(`\ntotal: ${addresses.length} items, ${total(manageable)} manageable fields, ` +
  `${total(written)} hand-written, ${total(uncovered)} fields stayed out of the record`);

console.log(`\nfirst plan after import: ${PLAN.size} items ` +
  `(${[...PLAN.values()].filter((v) => v.decision === "recreate").length} recreate)`);
const phantom = [...PLAN.keys()].filter((a) => !APPLY.has(a));
console.log(`  happened at apply: ${APPLY.size} items; did not happen ${phantom.length}` +
  ` (${phantom.join(", ")})`);
const recreations = (m) => [...m].filter(([, v]) => v.decision === "recreate").map(([a]) => a);
const downtime = (m) => recreations(m).reduce((s, a) => s + (DOWNTIME[a.split(".")[0]] ?? 0), 0);
console.log(`  recreation: ${recreations(PLAN).length} in plan, ` +
  `${recreations(APPLY).length} at apply; downtime ${downtime(PLAN)} s in plan, ` +
  `${downtime(APPLY)} s at apply (${(downtime(APPLY) / 15).toFixed(0)} missed meter-reading batches)`);
const lost = (m) => recreations(m).reduce((s, a) => s + uncovered(a), 0);
console.log(`  field reverting to default on recreation: ${lost(PLAN)} in plan, ` +
  `${lost(APPLY)} at apply`);

// Correction round: read every field the plan shows back from the provider and write it into the record.
const CORRECTED = structuredClone(MANUAL);
let corrected = 0;
for (const [address, v] of PLAN)
  for (const k of v.fields) { CORRECTED[address][k] = ACTUAL[address][k]; corrected++; }
const SECOND = plan(CORRECTED, STATE, { normalize: false });
console.log(`\ncorrection round: ${corrected} fields read back -> ` +
  `second plan ${SECOND.size} items`);

console.log("\nthe record's coverage: three options");
console.log("option               entered in record (drift visible)  unmanaged  " +
  "first plan  lost on recreation");
const option = [
  ["leave out of record", 0, total(manageable), 0, 0],
  ["partial import", total(written), total(uncovered), PLAN.size, total(uncovered)],
  ["full import", total(manageable), 0, 0, 0],
];
for (const [name, ...s] of option)
  console.log(name.padEnd(21) + s.map((x, i) => String(x).padStart([29, 14, 10, 27][i])).join(""));
```

```
imported resources
address                      real fields  hand-written  uncovered  first plan
object_store.old_metrics              4             2            2  -
dns_zone.metrics                      2             2            0  update (ttl)
machine.old_collector_1               4             4            0  recreate (image)
machine.old_collector_2               4             4            0  -
security_group.default                3             2            1  update (ingress)
data_store.old_meter                  4             3            1  recreate (size)
queue.old_stream                      3             3            0  -

total: 7 items, 24 manageable fields, 20 hand-written, 4 fields stayed out of the record

first plan after import: 4 items (2 recreate)
  happened at apply: 3 items; did not happen 1 (data_store.old_meter)
  recreation: 2 in plan, 1 at apply; downtime 1005 s in plan, 105 s at apply (7 missed meter-reading batches)
  field reverting to default on recreation: 1 in plan, 0 at apply

correction round: 4 fields read back -> second plan 0 items

the record's coverage: three options
option               entered in record (drift visible)  unmanaged  first plan  lost on recreation
leave out of record                              0            24         0                          0
partial import                                  20             4         4                          4
full import                                     24             0         0                          0
```

The top table is the copying job's balance sheet. Seven resources have twenty-four manageable
fields; twenty of them were hand-written. Four fields were never written at all: the object store's
version protection and encryption settings, the security group's egress rule, the data store's
backup arrangement. These were not forgotten — the person writing the record either did not see them
in the provider interface or passed over them assuming they were the default. The record's coverage
here is exactly eighty-three percent, and the remaining seventeen percent is infrastructure the
record never knew about.

The coverage ratio varies from resource to resource. Two machines were written with all four of their four
fields and the queue with all three of its three; the object store with two of four. The difference
depends on how visible the fields are: the machine's subnet and image are on the interface's first screen,
the object store's encryption setting is on its third tab. **The record's coverage is determined by how
far down the person writing it scrolled.**

Import itself does exactly one thing: it binds the real identity at the provider to an address in
the record, and writes all of that resource's fields to state. What it does not do matters more — it
does not write the record. State knows reality exactly; the record only knows what the human wrote.
For the first time in this course, record and state start from the same moment and are born with
**separate contents**, and the gap between them shows up immediately in the first plan.

The chosen address is a decision too, and it connects to this topic's earlier measure. The seven
resources here were addressed by their own names; if they had been addressed by an index, every
future addition and removal would shift the addresses, and the imported resources would start
pointing at different real resources. Import is where the price of address stability is highest: a
wrong address means not just a needless recreation, but the recreation of the **wrong resource**.

#### The First Plan After Import

After import finishes, nothing is expected to have changed: the resources are the same, state is an
exact copy of reality. Despite this, the first plan shows **four items**, and every one of them
comes from the record not matching reality — not one of them is a wanted change.

Three of the four happen at apply. The item that does not is the data store: because the record
writes `size` in uppercase, the planner sees a difference, but because the provider does not
distinguish case, nothing happens at apply. This item will keep coming back on every plan run and
will stay permanent noise unless someone touches it.

The danger in these four items is not their size, it is that they run counter to expectation. The
plan is expected to come out empty after import; when it does not, two readings are possible —
either the record describes reality wrong, or a change is genuinely wanted. Both look the same in
the plan. The review load measured in the previous lesson is at its worst here: the person reading
the plan of a freshly imported resource has nothing in hand to tell which kind an item is. The only
way is to read every shown value back from the provider — that is, to do import's copying job one
more time.

The real number is in the recreation row. The plan says two resources will be recreated; at apply,
one does. The gap between them is large in the measurement network's unit: the plan's promised
downtime is 1005 seconds, the actual is 105. Nine hundred of the thousand five belong to the data
store, and that item was born from nothing more than a letter-case difference. Because the person
writing the record wrote `MEDIUM` instead of `medium`, the plan showed that a data store carrying
forty-one days of meter data would be recreated.

Even the 105 seconds that did happen did not come free. One of the old collector machines' image had
been misread; the plan mistook this for a change and recreated the machine, the machine was out of
service for 105 seconds, and seven meter-reading batches were missed in that subregion. The
resource had never changed; the only thing that changed was its description in the record.
**Import's downtime comes from the copy's error, not the resource's.**

The correction round closes this section. When the four fields the plan shows are read back from
the provider and written into the record, the second plan comes out to zero items. So import
does not finish in a single step: once the address is bound, settling the record against reality
needs at least one more round, and that round's input is the plan itself. Here the plan works not
as a warning, but as a **correction list**.

#### Fields Lost on Recreation

The four fields left out of the record look harmless right after import. The plan does not show
them, apply does not touch them, the resource keeps working as it is. Their harmlessness rests on a
single condition: the resource not being recreated.

Recreation removes that condition. When a resource is destroyed and recreated, the new resource is
built from the record, and every field not written in the record reverts to the provider default.
In this measurement, that is one field in the two planned recreations, zero in the one that actually
happened — a small number, but the field at risk of loss is the data store's **backup
arrangement**. A data store with daily backups could have ended up without backups, with nobody
wanting it and no plan ever showing it.

The shape of the danger here is worth noting. An out-of-record field does not show up as drift
either, because drift can only be measured on fields that can be compared against the record. A
field the record does not know about does not know its own drift either. Four of the seven resources'
twenty-four fields enter none of the differences this course measures — not in the plan, not in the
drift count, not in review.

#### The Record's Coverage: Three Options

The last table opens the decision between three options, and the first column gives away the
misleading one. Leaving the resource out of the record asks for zero lines of record, produces a
zero-item plan, loses zero fields on recreation. Every column is zero — but the first column is
zero too: **no field's drift is visible.** None of the twenty-four fields is ever compared. An
out-of-record resource is not cheap, it is **unmeasured**.

Partial import makes twenty fields visible and leaves four outside, and in return asks for a
four-item first plan and a correction round. Full import writes all twenty-four fields; its first
plan is zero and no field is lost on recreation, but the number of fields to write rises by twenty
percent, and each of those fields is a line that will need maintenance going forward.

There is a fourth option not shown in the table: import it, then remove it from management. The
resource stays in the record, but apply never touches it. This option combines the first three's
downsides — the record needs maintenance, drift is still invisible — and its only gain is that the
resource's existence is written down. On the measurement network, its one legitimate use is the old
data store, which is dangerous to destroy.

Put in this course's measure: the wider the record, the more measurable the gap between it and
reality, and the more measurable the gap, the more manageable it is. An unmeasured item is not
failing to be fixed — nobody knows it is broken. Four of the seven resources' fields are correct
today; when someone changes one tomorrow, there is no plan anywhere that will show it.

#### Summary

- 20 of seven out-of-record resources' 24 manageable fields were hand-written, 4 stayed out of the
  record: coverage 83 percent. What determines coverage is how visible the field is in the
  provider interface.
- The first plan after import showed 4 items, and none of them was a wanted change; 3 happened at
  apply, and 1 will keep coming back on every run because it was born from a letter-case
  difference.
- The plan promised 2 recreations, apply did 1: downtime was 1005 seconds in the plan, 105 at
  apply, with 7 missed meter-reading batches. The cause of the downtime was not the resource
  changing, it was the copy being misread.
- The correction round read 4 fields back from the provider and brought the second plan down to 0
  items; import does not finish in a single step, and the correction's input is the plan itself.
- The 4 fields left out of the record are harmless only as long as the resource is not recreated;
  once it is, they revert to the default. One of the fields at risk of loss was the data store's
  backup arrangement.
- When the three options are compared, leaving a resource out of the record makes every column
  zero — including the column for fields whose drift is visible. An out-of-record resource is not
  cheap, it is unmeasured.

#### Next Step

This topic measured the record itself: order, iteration, the plan, and now the record's coverage.
In all four lessons, the same thing surfaced once more — the gap between plan and apply was
measured, but the **state** the plan rests on was always taken as a given. Yet state is a file: it
sits somewhere, someone writes it, someone reads it. This lesson's import wrote to it directly too.

The next topic opens that file. What gets measured is clear: where state sits and who can access
it, what is written inside it — **sensitive values** included — and what happens when two people
apply at the same time. This last one is the most expensive question, because two applies
overwriting each other never shows up in any plan.

## State and Modules

Where the record stands and how it is split: why the state file is needed and what it holds, remote state and locking preventing team collisions, splitting large infrastructure with state splitting, workspaces for multiple instances of the same definition, module authoring with its input, output and version, shared module registries, and the visibility of sensitive values in state and output.

### State File

Source: https://academia.sh/en/courses/infrastructure-as-code/state-file

Where the record ties to reality: the read cost of mapping without a state file and the items that can never be mapped, the information the state keeps per item, the distinction between a deleted resource and a resource the record does not cover, and the cost of losing state.

The gap between the plan and the result of the apply was measured: the plan showed one item, the
apply produced a different outcome. That comparison rested on a silent assumption. To be able to
say "this resource already exists" or "this resource does not exist," the plan assumed that
reality was written down somewhere. Where does that written form live?

This lesson actually writes that file, actually reads it, and counts what cannot be done without it.

#### Definition, Apply and Two Outputs

A fictional **regional measurement network** will be used: three environments, in each environment
two machines collecting meter readings, a queue, a data store, an object store, a subnet and
network rules. Both the network and the provider are models; no real infrastructure is touched.

An apply produces two things: **resources** on the provider side, and a **state file** on its own
side. The script below writes all three to disk.

```js
// apply.mjs — the fictional "regional measurement network": builds the definition, models the apply,
// and writes the provider-side reality and the state file to disk. Both are models.
import { writeFileSync } from "node:fs";

const environments = ["test", "staging", "production"];
const definition = [];
const add = (type, name, env, fields, deps = []) =>
  definition.push({ name, type, env, fields, deps });

environments.forEach((e, i) => {
  add("subnet", `subnet_${e}`, e, { block: `10.${i}.0.0/16` });
  add("queue", `read_queue_${e}`, e, { retention_hours: 24, partitions: 4 });
  add("data_store", `measurement_store_${e}`, e, { size_gb: 200, backup: true });
  add("object_store", `archive_${e}`, e, { class: "infrequent" });
  for (const n of [1, 2])
    add("machine", `collector_${e}_${n}`, e, { cores: 2, memory_gb: 8 },
      [`subnet_${e}`, `read_queue_${e}`]);
  for (const port of [443, 5671, 9100])
    add("security_rule", null, e, { direction: "inbound", port }, [`subnet_${e}`]);
  add("route_table_entry", null, e, { destination: "0.0.0.0/0", gateway: "edge" }, [`subnet_${e}`]);
});
// Second port-9100 rule for the backup collector in production: the same item appears twice in the definition.
add("security_rule", null, "production", { direction: "inbound", port: 9100 }, ["subnet_production"]);

// Values the provider fills in on its own for fields the definition does not give.
const defaults = {
  machine: { disk_gb: 40, monitoring: "on" }, queue: { encryption: "on" },
  data_store: { maintenance_window: "sunday" }, object_store: { version_retention: false },
  subnet: { ipv6: false }, security_rule: {}, route_table_entry: {},
};
const unnamed = new Set(["security_rule", "route_table_entry"]);

const ids = definition.map((_, i) => `res-${String(i + 1).padStart(4, "0")}`);
const nameIndex = new Map(definition.map((k, i) => [k.name, ids[i]]));

const actual = [], state = [];
definition.forEach((k, i) => {
  const fields = { ...defaults[k.type], ...k.fields };
  const parentName = k.deps.find((b) => b.startsWith("subnet")) ?? null;
  actual.push({
    id: ids[i], type: k.type, fields,
    tag: unnamed.has(k.type) ? {} : { name: k.name, env: k.env },
    parent: parentName ? nameIndex.get(parentName) : null,
  });
  state.push({
    record_name: k.name ?? `${k.type}.${k.env}[${i}]`, type: k.type, env: k.env,
    id: ids[i], fields, deps: k.deps, provider_version: "3.4.0",
  });
});

writeFileSync("definition.json", JSON.stringify(definition, null, 1));
writeFileSync("provider.json", JSON.stringify(actual, null, 1));
writeFileSync("state.json", JSON.stringify(
  { format_version: 4, serial: 1, items: state }, null, 1));

const fieldCount = state.reduce((t, k) => t + Object.keys(k.fields).length, 0);
const definitionFields = definition.reduce((t, k) => t + Object.keys(k.fields).length, 0);
const edges = state.reduce((t, k) => t + k.deps.length, 0);
console.log(`definition: ${definition.length} items, ${definitionFields} field values`);
console.log(`state.json: ${state.length} items, ${fieldCount} field values, ` +
  `${edges} dependency edges, ${fieldCount - definitionFields} fields state-only`);
console.log(`record per item: id 1, fields ${(fieldCount / state.length).toFixed(1)}, ` +
  `deps ${(edges / state.length).toFixed(1)}, provider version 1`);
console.log(`sample item: ${JSON.stringify(state[4])}`);
```

**SM1.** The provider assigns ids in apply order; in the model, ids are assigned in sequence
starting from `res-0001`. **SM2.** The security rule and the route table entry are types that
**carry no tag**; the provider gives these items only an id. **SM3.** The provider fills fields
the definition does not give with defaults, and these values come back in the apply's response.

```sh
node apply.mjs
```

```
definition: 31 items, 56 field values
state.json: 31 items, 80 field values, 25 dependency edges, 24 fields state-only
record per item: id 1, fields 2.6, deps 0.8, provider version 1
sample item: {"record_name":"collector_test_1","type":"machine","env":"test","id":"res-0005","fields":{"disk_gb":40,"monitoring":"on","cores":2,"memory_gb":8},"deps":["subnet_test","read_queue_test"],"provider_version":"3.4.0"}
```

The first pair of numbers is interesting: the definition writes 56 field values, the state holds
80. In between are **24 fields that exist only in state**; the definition did not write them, the
provider set them. What the record "wants" and what reality "is" already diverge in the apply's
first second, and the state file is the only place that keeps that distinction.

#### After a Manual Intervention

Time passed since the apply. During a night shift, someone touched the infrastructure through the
provider's interface.

```js
// intervention.mjs — models a manual touch on the provider-side reality: two items deleted,
// four items opened outside the record, one field changed. The state file is not touched.
import { readFileSync, writeFileSync } from "node:fs";
const g = JSON.parse(readFileSync("provider.json", "utf8"));

const idOf = (name) => g.find((k) => k.tag.name === name).id;
const subnetStaging = idOf("subnet_staging"), subnetProduction = idOf("subnet_production");
const remove = (cond) => { const i = g.findIndex(cond); const [k] = g.splice(i, 1); return k.id; };
const del1 = remove((k) => k.tag.name === "archive_test");
const del2 = remove((k) => k.type === "route_table_entry" && k.parent === subnetStaging);

g.push({ id: "res-9001", type: "machine", fields: { cores: 4, memory_gb: 16, disk_gb: 40 },
  tag: { name: "temp_collector", env: "production" }, parent: subnetProduction });
g.push({ id: "res-9002", type: "object_store", fields: { class: "hot", version_retention: false },
  tag: { name: "manual_archive", env: "production" }, parent: null });
g.push({ id: "res-9003", type: "security_rule", fields: { direction: "inbound", port: 22 },
  tag: {}, parent: subnetProduction });
g.push({ id: "res-9004", type: "security_rule", fields: { direction: "inbound", port: 443 },
  tag: {}, parent: subnetProduction });

const m = g.find((k) => k.tag.name === "collector_production_1");
m.fields.memory_gb = 16;

writeFileSync("provider.json", JSON.stringify(g, null, 1));
console.log(`deleted: ${del1}, ${del2} | opened outside record: 4 | field manually changed: 1`);
console.log(`item count on provider: ${g.length}`);
```

```sh
node apply.mjs > /dev/null   # setup rebuilt from scratch
node intervention.mjs
```

```
deleted: res-0004, res-0020 | opened outside record: 4 | field manually changed: 1
item count on provider: 33
```

#### Matching Without State

The question now is: if there had never been a state file at all, what would it take to map the
record to reality? The only way is to scan the provider and match logical names against tags.

```js
// matching.mjs — tries to map the record to reality without looking at the state file.
import { readFileSync } from "node:fs";
const definition = JSON.parse(readFileSync("definition.json", "utf8"));
const actual = JSON.parse(readFileSync("provider.json", "utf8"));
const unnamed = new Set(["security_rule", "route_table_entry"]);

const types = [...new Set(actual.map((k) => k.type))];
let listings = types.length;                     // one listing per type
let detail = actual.filter((k) => unnamed.has(k.type)).length;  // every unnamed candidate is read

// Matching by name: the listing response carries the tag, so no extra read is needed.
const nameIndex = new Map(actual.filter((k) => k.tag.name).map((k) => [k.tag.name, k]));
const used = new Set();
let byName = 0, byNameMissing = 0;
for (const k of definition.filter((k) => k.name)) {
  const found = nameIndex.get(k.name);
  if (found) { byName++; used.add(found.id); } else byNameMissing++;
}

// Unnamed items: distinguishable only by content and the subnet they are attached to.
const signature = (type, parent, fields) =>
  `${type}|${parent}|${Object.keys(fields).sort().map((a) => `${a}=${fields[a]}`).join(",")}`;
const actualSig = new Map();
for (const k of actual.filter((k) => unnamed.has(k.type)))
  actualSig.set(signature(k.type, k.parent, k.fields), (actualSig.get(signature(k.type, k.parent, k.fields)) ?? 0) + 1);
const defSig = new Map();
for (const k of definition.filter((k) => unnamed.has(k.type))) {
  const parent = nameIndex.get(k.deps.find((b) => b.startsWith("subnet")))?.id ?? null;
  const i = signature(k.type, parent, k.fields);
  defSig.set(i, (defSig.get(i) ?? 0) + 1);
}
let unique = 0, ambiguous = 0, unnamedMissing = 0;
for (const [i, count] of defSig) {
  const g = actualSig.get(i) ?? 0;
  if (g === 0) unnamedMissing += count;
  else if (count === 1 && g === 1) unique++;
  else ambiguous += count;
}
const unowned = actual.filter((k) => !used.has(k.id) && !unnamed.has(k.type)).length;
const unownedUnnamed = [...actualSig].reduce((t, [i, g]) => t + Math.max(0, g - (defSig.get(i) ?? 0)), 0);

console.log(`matching without state — reads: ${listings} listings + ${detail} detail = ${listings + detail}`);
console.log(`  matched by name: ${byName}/${definition.filter((k) => k.name).length}`);
console.log(`  matched uniquely by content: ${unique}/${definition.filter((k) => unnamed.has(k.type)).length}`);
console.log(`  never matched (unnamed, signature collides): ${ambiguous}`);
console.log(`  in record but not in reality: ${byNameMissing + unnamedMissing} -> deleted or never created: indistinguishable`);
console.log(`  in reality but not in record: ${unowned} named + ${unownedUnnamed} unnamed`);
```

**SM4.** One listing per type counts as one read; the tag of a named item comes back with the
listing, while an unnamed item's fields are only visible through a detail read.

```sh
node matching.mjs
```

```
matching without state — reads: 7 listings + 14 detail = 21
  matched by name: 17/18
  matched uniquely by content: 9/13
  never matched (unnamed, signature collides): 3
  in record but not in reality: 2 -> deleted or never created: indistinguishable
  in reality but not in record: 2 named + 2 unnamed
```

After twenty-one reads, the table left in hand is incomplete. 26 of the thirty-one items matched,
**3 items could never be matched**. All three are unnamed resources: the two port-9100 rules in
production are identical twins, and the port-443 rule has two counterparts on the provider — one
from the definition, one by hand. Between two items with identical content, there is no
information to say which id belongs to which record line.

The second loss is heavier: for the two missing items, the distinction between "deleted" and
"never created" cannot be made. The record sees both the same way and makes the same decision for
both — create. It is never learned that the deleted resource was deleted; it is only recreated.

#### What the State File Holds

The sample item shows that state keeps four separate pieces of information per item: **id**
(`res-0005`), **field values** (four; two from the definition, two from the provider default),
**dependencies** (two record names), and **provider version** (`3.4.0`). Totals for thirty-one
items: 31 ids, 80 field values, 25 dependency edges, 31 version stamps.

Each of these four pieces does a separate job. The id makes matching unique. Field values are the
comparison baseline for drift — because the default the provider set is recorded too, a
difference becomes visible if that default changes with the provider version. The dependency edge
gives the deletion order; it is read from the applied state, not from the text of the record file.
The version stamp makes it possible to detect that the same definition produces a different
result on a different provider version.

#### Comparing With State

```js
// state-read.mjs — compares the ids in the state file against reality on the provider.
import { readFileSync } from "node:fs";
const st = JSON.parse(readFileSync("state.json", "utf8"));
const actual = new Map(JSON.parse(readFileSync("provider.json", "utf8")).map((k) => [k.id, k]));

let reads = 1, matched = 0;                      // 1: the state file itself
const deleted = [], drifted = [];
for (const k of st.items) {
  reads++;                                       // one item read by id
  const g = actual.get(k.id);
  if (!g) { deleted.push(`${k.record_name} (${k.id})`); continue; }
  matched++;
  for (const [field, value] of Object.entries(k.fields))
    if (g.fields[field] !== value) drifted.push(`${k.record_name}.${field} ${value} -> ${g.fields[field]}`);
}
const stateIds = new Set(st.items.map((k) => k.id));
const outOfRecord = [...actual.values()].filter((g) => !stateIds.has(g.id));

console.log(`comparing with state — reads: 1 state file + ${reads - 1} ids = ${reads}`);
console.log(`  matched: ${matched}/${st.items.length}, never matched: 0`);
console.log(`  deleted (id in state, not on provider): ${deleted.length} -> ${deleted.join("; ")}`);
console.log(`  drifted field: ${drifted.length} -> ${drifted.join("; ")}`);
console.log(`  not covered by the record (on provider, not in state): ${outOfRecord.length} -> ` +
  outOfRecord.map((g) => g.tag.name ?? `${g.type}/${g.id}`).join(", "));
```

```sh
node state-read.mjs
```

```
comparing with state — reads: 1 state file + 31 ids = 32
  matched: 29/31, never matched: 0
  deleted (id in state, not on provider): 2 -> archive_test (res-0004); route_table_entry.staging[19] (res-0020)
  drifted field: 1 -> collector_production_1.memory_gb 8 -> 16
  not covered by the record (on provider, not in state): 4 -> temp_collector, manual_archive, security_rule/res-9003, security_rule/res-9004
```

The read count went from 21 to 32 — state does not **cheapen** matching, it makes it certain.
Three numbers were gained: unmatched items dropped from 3 to 0, two deleted items were found by
name and id, and one drifted field (`memory_gb` 8 instead of 16) surfaced. None of these three
were visible in matching without state.

The fourth number gives the record's **coverage**: 4 of the 33 items on the provider are inside no
record at all. Two machines and an object store stand by name, two security rules stand only by
id. The record neither plans, nor deletes, nor counts drift for these; this part of the
infrastructure is outside the record, and it is known to be there only as long as no one deletes
it without looking.

#### The Loss of State

The state file is a record as critical as the infrastructure itself. The cost of losing it can be
counted.

```js
// loss.mjs — counts what can be recovered if the state file is lost.
import { readFileSync, rmSync, existsSync } from "node:fs";
const d = JSON.parse(readFileSync("state.json", "utf8"));
const lost = { id: d.items.length, edge: 0, version: 0, field: 0 };
for (const k of d.items) {
  lost.edge += k.deps.length;
  lost.version += k.provider_version ? 1 : 0;
  lost.field += Object.keys(k.fields).length;
}
rmSync("state.json");
console.log(`state.json deleted; does the file still exist: ${existsSync("state.json")}`);
console.log(`lost information: ${lost.id} ids, ${lost.field} field values, ` +
  `${lost.edge} dependency edges, ${lost.version} provider versions`);
console.log(`the only path left: matching without state`);
```

```sh
node apply.mjs > /dev/null && node intervention.mjs > /dev/null   # setup rebuilt from scratch
node loss.mjs
```

```
state.json deleted; does the file still exist: false
lost information: 31 ids, 80 field values, 25 dependency edges, 31 provider versions
the only path left: matching without state
```

When the file is gone, the infrastructure is not gone; its **manageability** is. The only path
left is the matching-without-state above, and its ceiling is known: 26 items can be recovered by
importing them one by one, 3 items can never be matched by any means. The information that the
two deleted items were deleted has also gone with the file. Until the record is rebuilt, every
apply plans to recreate resources that already exist.

**SM5.** The state file sits in a single working directory, as plain text. This is the second
face of the record's own risk: the file holds all 80 field values as they are. Even if a
connection string or access credential given through a variable was hidden in the definition, it
is written here in its post-apply form, and anyone who can read the file reads it. **SM6.** The
only thing that writes the file is the apply; editing it by hand genuinely pulls the state away
from reality, because it has no counterpart on the provider side.

#### Summary

- The apply produced two outputs: 31 resources on the provider and the state file that records
  them; the definition wrote 56 field values while state held 80, with 24 fields set by the
  provider.
- Matching without state took 21 reads and 3 items were never matched; all three were unnamed
  resources whose signatures were identical.
- Without state, "deleted" cannot be separated from "never created"; with state, two deleted
  items were found by id and one drifted field surfaced.
- State keeps four pieces of information per item: id, field values, dependencies, provider
  version — for 31 items, 31 ids, 80 fields, 25 edges, 31 version stamps.
- 4 of the 33 items on the provider fell outside the record; the record neither plans them nor
  counts them.
- When state is lost, 26 items come back by importing, 3 do not; the loss does not destroy the
  infrastructure, it makes it unmanageable.

#### Next Step

This whole lesson assumed a single person working in a single directory. The state file sat on
local disk, and only one process touched it at a time. That assumption falls the moment two
people on a team start an apply against the same environment at the same time: both read the same
state, both change it, and the second overwrites what the first wrote. The next lesson actually
runs two concurrent applies, counts the write that is lost without a lock, measures the waiting
rounds a lock produces, and deals with the stale lock a process leaves behind when it dies without
releasing it.

### Remote State and Locking

Source: https://academia.sh/en/courses/infrastructure-as-code/remote-state-and-locking

Two applies writing the same state at the same time: the lost write and the out-of-record resource without a lock, the waiting rounds a lock produces, the queue that lock hold time creates, the stale lock a process leaves behind when it dies without releasing it, and the coverage gap between a local file and a remote store.

The previous lesson left the state file in a single directory, in the hands of a single process.
That assumption holds for a one-person team. It collapses the moment a second person arrives: two
applies read the same state, both create resources, both write to the same file. This lesson
actually produces that collision, counts its cost, and measures how much of it a lock resolves.

The optimistic and pessimistic solutions for concurrent updates were measured in The Data Access
Layer and Business Logic course; the mechanics there are not repeated here. The difference is
granular: there, what was locked was a row, and two clients writing to different rows never met;
here, what is locked is **the entire state file**. If three environments sit in a single file, the
person adding a machine to the test environment waits on the person growing the production queue;
the concurrency ceiling drops to one.

#### Remote State Store and Two Appliers

The script below models a single apply: it reads the state, runs for the duration of the apply,
adds its own items, and writes the state back. **SM7.** The remote state store is modeled with a
directory; both processes access the same `store/state.json` file. **SM8.** Apply duration is
modeled with a busy loop; in reality that time passes in calls made to the provider.

```js
// applier.mjs — models a single apply against a remote state store.
// modes: setup | apply <name> <count> <lock|nolock> <duration_ms> [wait_deadline_ms] | summary
import { readFileSync, writeFileSync, appendFileSync, mkdirSync, rmSync,
  openSync, closeSync, unlinkSync, existsSync } from "node:fs";

const STORE = "store", STATE = "store/state.json", LOCK = "store/lock.json";
const mode = process.argv[2];
const wait = (ms) => { const b = Date.now() + ms; while (Date.now() < b); }; // model of apply duration

if (mode === "setup") {
  rmSync(STORE, { recursive: true, force: true });
  mkdirSync(STORE);
  writeFileSync(STATE, JSON.stringify({ serial: 1, items: [] }));
  process.exit(0);
}

if (mode === "summary") {
  const d = JSON.parse(readFileSync(STATE, "utf8"));
  const owner = {};
  for (const k of d.items) owner[k[0]] = (owner[k[0]] ?? 0) + 1;
  const distribution = Object.keys(owner).sort().map((a) => `${a}=${owner[a]}`).join(" ") || "none";
  const g = existsSync(`${STORE}/log.txt`) ? readFileSync(`${STORE}/log.txt`, "utf8") : "";
  for (const s of g.trim().split("\n").sort()) console.log(s);
  const created = [...g.matchAll(/(\d+) items written/g)].reduce((t, m) => t + Number(m[1]), 0);
  console.log(`state: ${d.items.length} items (${distribution}), serial ${d.serial}, ` +
    `lock ${existsSync(LOCK) ? "held" : "none"}`);
  console.log(`actually created ${created} resources, visible in record ${d.items.length} ` +
    `-> out of record ${created - d.items.length}`);
  process.exit(0);
}

if (mode !== "apply") {
  console.log("usage: setup | apply <name> <count> <lock|nolock> <duration_ms> | summary");
  process.exit(0);
}

const [name, count, lockMode, durationMs] =
  [process.argv[3], Number(process.argv[4]), process.argv[5], Number(process.argv[6])];
let round = 0;
if (lockMode === "lock") {
  const deadline = Date.now() + Number(process.argv[7] ?? 10000);
  for (;;) {
    try {                                        // "wx": errors if the file exists, atomic
      const f = openSync(LOCK, "wx");
      writeFileSync(f, JSON.stringify({ owner: name, at: "fictional-timestamp" }));
      closeSync(f); break;
    } catch {
      round++;
      if (Date.now() > deadline) { appendFileSync(`${STORE}/log.txt`,
        `${name}: lock not acquired, ${round} waiting rounds\n`); process.exit(1); }
      wait(25);
    }
  }
}
const d = JSON.parse(readFileSync(STATE, "utf8"));
writeFileSync(`${STORE}/${name}-read`, "");
if (lockMode === "nolock")                       // let both applies read before either writes
  while (!existsSync(`${STORE}/a-read`) || !existsSync(`${STORE}/b-read`)) wait(5);
wait(durationMs);
for (let i = 1; i <= count; i++) d.items.push(`${name}-item-${i}`);
d.serial += 1;
writeFileSync(STATE, JSON.stringify(d));
if (lockMode === "lock") unlinkSync(LOCK);
appendFileSync(`${STORE}/log.txt`,
  `${name}: ${count} items written, read serial ${d.serial - 1}, waiting rounds ${round}\n`);
```

Two people are adding resources to the fictional measurement network's staging environment at the
same time: `a` adds twelve items, `b` adds twelve items. **SM9.** They are synchronized with
marker files to make sure both read before either writes; because `b`'s apply duration is sixty
milliseconds longer, it is the one that writes last.

```sh
# nolock.sh — two applies change the same remote state at the same time, no lock
node applier.mjs setup
node applier.mjs apply a 12 nolock 0 &
node applier.mjs apply b 12 nolock 60 &
wait
node applier.mjs summary
```

```
a: 12 items written, read serial 1, waiting rounds 0
b: 12 items written, read serial 1, waiting rounds 0
state: 12 items (b=12), serial 2, lock none
actually created 24 resources, visible in record 12 -> out of record 12
```

Three numbers show three separate harms. The first is the **lost write**: both applies wrote
twelve items each, but twelve items are left in the file; `a`'s write was overwritten by `b`. The
second is the **gap between record and reality**: `a` really did create its resources, there are
twenty-four resources on the provider, and the record says twelve — twelve resources fell outside
the record and will now neither be planned nor deleted. The third is the **inconsistent state**:
even though two applies ran, the serial number went up by one; both say "read serial 1." The
record stands as if only a single apply had run.

#### The Lock

The solution is to acquire the right to write before reading the state. The lock is acquired
through the filesystem's atomic "error if it exists" creation: even if two processes try at the
same instant, only one can create the file. The other catches the error, waits twenty-five
milliseconds, and tries again.

```sh
# lock.sh — the same two applies, this time with the state lock
node applier.mjs setup
node applier.mjs apply a 12 lock 250 &
until [ -f store/lock.json ]; do :; done   # a took the lock
node applier.mjs apply b 12 lock 250 &
wait
node applier.mjs summary
```

```
a: 12 items written, read serial 1, waiting rounds 0
b: 12 items written, read serial 2, waiting rounds 10
state: 24 items (a=12 b=12), serial 3, lock none
actually created 24 resources, visible in record 24 -> out of record 0
```

Twenty-four items, serial three, zero out of record. `b` spent ten waiting rounds — the
two-hundred-fifty-millisecond hold divided by the twenty-five-millisecond poll — and read the
**current** state: "read serial 2." The trade-off between the lost write and waiting is plain —
the only cost the lock produced is `b`'s two hundred fifty milliseconds, and what it bought is
twelve resources entering the record.

The lock file is deleted after the apply: the lock is not part of the state, it is the gate to
accessing the state.

Acquiring the lock **before** the read is not an arbitrary order, either. When `b` acquires the
lock, it rereads the state and sees serial two; if it had read first and then tried to acquire the
lock, it would have been left holding serial one and would have erased `a`'s twelve items the
moment it wrote. The result from the lockless measurement comes back if the lock is acquired in
the wrong place. There is one order: lock, read, apply, write, release.

#### Scanning Lock Hold Time

Waiting lasts exactly as long as the lock is held. If the apply takes longer, everyone behind it
waits that much longer.

Measurement rule: the measured duration is not printed, it is only run through a band check —
between 0.7 and 2.0 times the expected value. The lower bound confirms the queue actually forms;
the upper bound is left wide because the twenty-five-millisecond poll grain and the
process-startup overhead can approach a multiple of the expected value at short hold times.

```js
// scan.mjs — how the queue for four applies grows as lock hold time grows.
// Everything printed is run-independent: the expected values come from queue arithmetic,
// the measured duration is only run through a band check and printed as yes/no.
import { spawn } from "node:child_process";
import { readFileSync } from "node:fs";

const run = (...a) => new Promise((c) =>
  spawn("node", ["applier.mjs", ...a], { stdio: "ignore" }).on("exit", c));
const N = 4, POLL = 25, LOW = 0.7, HIGH = 2.0;
console.log(`queue arithmetic (${N} applies, run-independent): ` +
  `total wait = ${(N * (N - 1)) / 2} x hold, longest = ${N - 1} x hold`);
console.log(`band: measured must fall between ${LOW}-${HIGH}x expected`);

const accuracy = [];
for (const hold of [50, 200, 800]) {
  await run("setup");
  await Promise.all(["a", "b", "c", "d"].map(
    (name) => run("apply", name, "12", "lock", String(hold))));
  const rounds = readFileSync("store/log.txt", "utf8").trim().split("\n")
    .map((s) => Number(/waiting rounds (\d+)/.exec(s)[1]));
  const d = JSON.parse(readFileSync("store/state.json", "utf8"));
  accuracy.push(`${d.items.length} items, serial ${d.serial}`);

  const expTotal = ((N * (N - 1)) / 2) * hold, expLongest = (N - 1) * hold;
  const inBand = (measured, expected) =>
    measured >= LOW * expected && measured <= HIGH * expected ? "yes" : "no";
  console.log(`hold ${String(hold).padStart(3)} ms | expected total ` +
    `${String(expTotal).padStart(4)} ms, longest ${String(expLongest).padStart(4)} ms | ` +
    `measured in band: ${inBand(rounds.reduce((x, y) => x + y, 0) * POLL, expTotal)} / ` +
    `${inBand(Math.max(...rounds) * POLL, expLongest)}`);
}
console.log(`accuracy (run-independent): ` +
  `${new Set(accuracy).size === 1 ? `${accuracy[0]} at all three hold times` : accuracy.join(" | ")}`);
```

```
queue arithmetic (4 applies, run-independent): total wait = 6 x hold, longest = 3 x hold
band: measured must fall between 0.7-2x expected
hold  50 ms | expected total  300 ms, longest  150 ms | measured in band: yes / yes
hold 200 ms | expected total 1200 ms, longest  600 ms | measured in band: yes / yes
hold 800 ms | expected total 4800 ms, longest 2400 ms | measured in band: yes / yes
accuracy (run-independent): 48 items, serial 5 at all three hold times
```

The first line comes not from measurement but from the queue itself: when four applies line up,
the first waits not at all, the second waits one, the third two, the fourth three lock hold times.
The total is six, the longest is three hold times. The lesson's claim rests on this arithmetic —
queue length is directly proportional to apply duration — and the total wait, three hundred
milliseconds at a fifty-millisecond hold, climbs to four thousand eight hundred at eight hundred:
sixteen times the hold, sixteen times the wait.

The measurement only confirms this: **yes** at all three hold times. The source of deviation is a
fixed overhead — process startup and the poll grain — and its share shrinks as the hold grows; in
this run, the eight-hundred-millisecond row measured about two percent above expected.

The last line is also run-independent: forty-eight items, serial five. The lock does not give
correctness for free, it gives it in exchange for **a queue**; the length of that queue can be
calculated before it is ever measured.

From this comes an operating rule: long applies keep an entire team waiting. The way to shorten
the queue is not to make the lock faster, but to shrink the area the lock covers.

#### The Stale Lock

If the process holding the lock dies, the lock file is left behind. **SM10.** The setup below
builds this directly: the lock file is written, and its owner disappears without ever writing.

```js
// stale-lock.mjs — the lock a process leaves behind when it dies without releasing it, and opening it by hand.
import { writeFileSync, readFileSync, existsSync, unlinkSync, rmSync, mkdirSync } from "node:fs";
import { spawnSync } from "node:child_process";

rmSync("store", { recursive: true, force: true });
mkdirSync("store");
writeFileSync("store/state.json", JSON.stringify({ serial: 1, items: [] }));
// a took the lock and died before writing; the lock file was left behind.
writeFileSync("store/lock.json", JSON.stringify({ owner: "a", at: "fictional-timestamp" }));

const lastLine = () => readFileSync("store/log.txt", "utf8").trim().split("\n").at(-1);
const tryB = () => spawnSync("node",
  ["applier.mjs", "apply", "b", "12", "lock", "0", "1000"]).status;

console.log(`stale lock: ${readFileSync("store/lock.json", "utf8")}`);
console.log(`b tried (exit ${tryB()}) -> ${lastLine()}`);
console.log(`state: ${JSON.parse(readFileSync("store/state.json", "utf8")).items.length} items`);

unlinkSync("store/lock.json");                   // opening it by hand
console.log(`lock opened by hand; does the file still exist: ${existsSync("store/lock.json")}`);
console.log(`b tried again (exit ${tryB()}) -> ${lastLine()}`);
console.log(`state: ${JSON.parse(readFileSync("store/state.json", "utf8")).items.length} items`);
console.log(`fields in the lock record: 2 (owner, at) | field saying whether the owner is still running: 0`);
```

```
stale lock: {"owner":"a","at":"fictional-timestamp"}
b tried (exit 1) -> b: lock not acquired, 42 waiting rounds
state: 0 items
lock opened by hand; does the file still exist: false
b tried again (exit 0) -> b: 12 items written, read serial 1, waiting rounds 0
state: 12 items
fields in the lock record: 2 (owner, at) | field saying whether the owner is still running: 0
```

`b` waited forty-two rounds, filled the one-second deadline, and managed to write zero items. The
lock does not open on its own; the process that would release it no longer exists. From here on,
this is the record's **own risk**: even though the infrastructure was never touched, it is
unmanageable, and the only way out is to open the lock by hand.

Opening it by hand takes two steps, but the person making the call has only two fields to go on:
the owner's name and a timestamp. The number of fields that say whether the owner is **still
running** is zero. If the call is wrong — if the owner is in the middle of a long apply — the lock
loses its meaning and the measurement from the first section comes back: twelve items are lost,
twelve resources fall out of record.

#### The Difference Between a Local File and a Remote Store

For the lock to work, everyone has to be looking at **the same** state. If state sits on each
applier's own disk, there is nothing shared to lock.

```js
// local.mjs — state sitting on each applier's own disk: two records, one reality.
import { writeFileSync, readFileSync, rmSync, mkdirSync } from "node:fs";

rmSync("local", { recursive: true, force: true });
mkdirSync("local");
const actual = [];
for (const name of ["a", "b"]) {                 // two people worked on their own copy
  const own = [];
  for (let i = 1; i <= 12; i++) { own.push(`${name}-item-${i}`); actual.push(`${name}-item-${i}`); }
  writeFileSync(`local/state-${name}.json`, JSON.stringify({ serial: 2, items: own }));
}
const read = (name) => JSON.parse(readFileSync(`local/state-${name}.json`, "utf8")).items;
const a = read("a"), b = read("b");
const pct = (n) => `${Math.round((100 * n) / actual.length)}%`;

console.log(`actual resources on provider: ${actual.length}`);
console.log(`local file: 2 -> state-a ${a.length} items (coverage ${pct(a.length)}), ` +
  `state-b ${b.length} items (coverage ${pct(b.length)})`);
console.log(`shared items: ${a.filter((k) => b.includes(k)).length}, ` +
  `file that knows all of it: 0, sum of both files: ${a.length + b.length}`);
console.log(`if a drift scan runs with state-a, appears out of record: ` +
  `${actual.filter((k) => !a.includes(k)).length} items (all false)`);
console.log(`with a remote store: 1 file, ${actual.length} items, coverage 100%, false out-of-record 0`);
```

```
actual resources on provider: 24
local file: 2 -> state-a 12 items (coverage 50%), state-b 12 items (coverage 50%)
shared items: 0, file that knows all of it: 0, sum of both files: 24
if a drift scan runs with state-a, appears out of record: 12 items (all false)
with a remote store: 1 file, 24 items, coverage 100%, false out-of-record 0
```

**SM11.** Two people open twelve resources each with no communication between them. The result is
two separate records: each is internally consistent, and neither knows more than half of reality.
There is no lost write — because there is no shared file — but the gap is worse: because both
records are correct within their own scope, no one notices that anything is wrong. If a drift scan
runs against `state-a`, it declares twelve items out of record; all twelve are false alarms, the
resources do have an owner, the owner is just another file.

This is the one thing a remote state store gives: **a single record**. Coverage rises to one
hundred percent, an object exists to lock, and the false-out-of-record count drops to zero.
**SM12.** The store assumes a service that can make file creation atomic; without that guarantee,
there is no lock either.

The gain does not come for free, and its cost is the record's own risk. Two local files were two
separate points of failure; one remote record is one point of failure. If a write that broke off
halfway corrupts the file, the previous lesson's numbers come back: twenty-six items are recovered
by importing, three cannot be. If the store keeps no versions, there is no copy to fall back to
either. The second risk is access: the file no longer sits on one person's disk, it sits somewhere
the whole team can read, and all eighty field values counted in the previous lesson are there.

#### Summary

- Two lockless concurrent applies wrote twelve items each, and twelve items are left in the
  record: one write was lost, twelve resources fell out of record, and the serial went up by one
  despite two applies.
- A lock built with atomic file creation produced twenty-four items, serial three, and zero
  out-of-record on the same measurement; its cost is the second apply's ten waiting rounds.
- The queue for four applies waits six lock hold times regardless of the run, with the longest
  wait at three; the measurement stayed inside the band at all three hold times, and correctness
  never changed.
- A process that died without releasing the lock produced forty-two waiting rounds that filled
  the one-second deadline and zero items written; the lock record has no field saying whether the
  owner is alive.
- Keeping state in a local file produced two records, both stuck at fifty percent coverage, and a
  drift scan produced twelve false out-of-record results; the remote store brought coverage to
  one hundred percent.

#### Next Step

The lock rescued correctness but brought a cost: if the area the lock covers is a single file,
everyone writing to that file gets in line. If the entire measurement network sits in one state,
someone who wants to change a security rule waits on the apply of someone scaling the data store.
The next lesson splits this single file: it measures plan duration and how many items a single
change locks in infrastructure that runs on one state, repeats the same measurements in the split
form, and counts the cost of splitting — the dependencies between the pieces.

### State Splitting

Source: https://academia.sh/en/courses/infrastructure-as-code/state-splitting

The plan duration and blast radius of infrastructure growing in a single state, the same measurements in split form, the count of cross-part dependencies that is the cost of splitting, the difference between reading an output and querying a data source, and how many parts reapply when one part changes.

The lock rescued correctness but also built a queue: everyone writing to the same state file
waits. The only way to shrink the area a lock covers is to split the state. This lesson puts the
numbers from before and after splitting side by side, then works out the bill for splitting.

#### Infrastructure Growing in a Single State

The fictional measurement network grew over the course of a year. **SM13.** The model builds a
structure that can be split into thirteen parts: network, data, compute and observability layers
in each of the three environments, plus a regional backbone shared across all three.

```js
// network-gen.mjs — the grown regional measurement network: thirteen parts, items and dependency edges.
// Output is written in two forms: a single state file, and one state file per part.
import { writeFileSync, mkdirSync, rmSync } from "node:fs";

const environments = ["test", "staging", "production"];
const layers = ["network", "data", "compute", "observability"];
const parts = [{ name: "shared", layer: "shared", count: 20 }];
for (const e of environments)
  for (const l of layers) parts.push({ name: `${e}_${l}`, env: e, layer: l, count: 50 });

// How many items a layer pulls information from other layers through.
const crossDeps = {
  network: [["shared", 2]], data: [["network", 3]],
  compute: [["network", 6], ["data", 4]], observability: [["compute", 5], ["data", 2]],
};
const targetPart = (p, layer) => (layer === "shared" ? "shared" : `${p.env}_${layer}`);

const items = [], innerEdges = [], crossEdges = [];
for (const p of parts) {
  for (let i = 0; i < p.count; i++) {
    items.push({ name: `${p.name}/i${i}`, part: p.name });
    if (i > 0) innerEdges.push([`${p.name}/i${i}`, `${p.name}/i${i - 1}`]);
  }
  for (const [layer, count] of crossDeps[p.layer] ?? [])
    for (let i = 0; i < count; i++)
      crossEdges.push([`${p.name}/i${i}`, `${targetPart(p, layer)}/i${i}`]);
}

rmSync("split", { recursive: true, force: true });
mkdirSync("split");
const item = (k) => ({ ...k, deps: [...innerEdges, ...crossEdges].filter((e) => e[0] === k.name).map((e) => e[1]) });
writeFileSync("single-state.json", JSON.stringify({ serial: 1, items: items.map(item) }));
for (const p of parts)
  writeFileSync(`split/${p.name}.json`, JSON.stringify(
    { serial: 1, items: items.filter((k) => k.part === p.name).map(item) }));

// Reality on the provider: the items in the record, plus three out-of-record resources in the production compute layer.
const provider = [...items.map((k) => k.name),
  "production_compute/manual-1", "production_compute/manual-2", "production_compute/manual-3"];
writeFileSync("provider.json", JSON.stringify(provider));

console.log(`parts: ${parts.length}, items: ${items.length}, ` +
  `inner edges: ${innerEdges.length}, cross-part edges: ${crossEdges.length}`);
console.log(`single state file: 1 x ${items.length} items | split: ` +
  `${parts.length} files, largest ${Math.max(...parts.map((p) => p.count))} items`);
console.log(`${provider.length} resources on provider, ${items.length} in record ` +
  `-> out of record ${provider.length - items.length}`);
```

```
parts: 13, items: 620, inner edges: 607, cross-part edges: 66
single state file: 1 x 620 items | split: 13 files, largest 50 items
623 resources on provider, 620 in record -> out of record 3
```

Six hundred twenty items, six hundred seventy-three edges. Only sixty-six of the edges cross a
part boundary — the rest stay inside their parts. This ratio says why splitting is possible at
all: ninety percent of the links are already local.

The last line sets up a difference that stays constant through the rest of the lesson: there are
six hundred twenty-three resources on the provider, six hundred twenty in the record. Three
resources are in no part's state at all. Splitting does not change this number — infrastructure
the record does not cover stays out of scope no matter how many files the record is split into.
The only effect it will feel below is which connection style accidentally pulls those three items
back in.

#### Plan Duration and Blast Radius

A plan refreshes every item in the record from the provider and compares it against the
definition. **SM14.** One refresh read per item takes one millisecond; the real duration depends
on the provider and network latency, but it stays linear in the number of items.

The measurement rule is the same as the previous lesson's: the measured duration is not printed,
only whether it falls between 0.7 and 2.0 times the expected value.

```js
// plan.mjs — reads the state of the given scope, refreshes each item, and measures plan duration.
// Everything printed is run-independent: the refresh count and the expected duration derived
// from it, plus a band check on the measured duration. Raw milliseconds are not printed.
// usage: node plan.mjs single | node plan.mjs part <name>
import { readFileSync } from "node:fs";
const REFRESH_MS = 1, LOW = 0.7, HIGH = 2.0;     // modeled cost of a refresh read, and the band
const wait = (ms) => { const b = Date.now() + ms; while (Date.now() < b); };
const [scope, name] = [process.argv[2] ?? "single", process.argv[3]];
const file = scope === "single" ? "single-state.json" : `split/${name}.json`;

const t0 = Date.now();
const d = JSON.parse(readFileSync(file, "utf8"));
for (const k of d.items) wait(REFRESH_MS);       // one refresh read per item
const measured = Date.now() - t0;
const expected = d.items.length * REFRESH_MS;

console.log(`${(scope === "single" ? "single state" : name).padEnd(24)} | ` +
  `${String(d.items.length).padStart(3)} refreshes, expected plan ${String(expected).padStart(3)} ms | ` +
  `measured in band: ${measured >= LOW * expected && measured <= HIGH * expected ? "yes" : "no"} | ` +
  `items locked ${String(d.items.length).padStart(3)}`);
```

The work done is the same in both scopes: changing a single item in the observability layer.

```sh
node plan.mjs single
node plan.mjs part production_observability
```

```
single state             | 620 refreshes, expected plan 620 ms | measured in band: yes | items locked 620
production_observability |  50 refreshes, expected plan  50 ms | measured in band: yes | items locked  50
```

The carrier of the measurement is the **refresh count**; that number depends not on the run but
on the scope's item count. Changing one item in the single state requires six hundred twenty
refreshes, fifty in the split form. Against one changed item, the wasted-read ratio is above
ninety-nine percent. Expected plan duration is derived from this number — six hundred twenty
milliseconds against fifty — and the measurement stayed inside the band in both scopes; duration
tracks the refresh count directly.

**Blast radius** is sharper still, and does not depend on time at all: the lock covers all six
hundred twenty items. Someone adding a threshold to the observability layer holds all three
environments' network, data store and archive locked for the duration of the plan.

In split form, the same work refreshes fifty items and locks fifty items. Plan duration drops to
one-twelfth, blast radius drops from one hundred percent to eight percent. Everyone touching the
other twelve parts can work at the same time; the concurrency ceiling that fell to one in the
previous lesson rises to thirteen.

The difference accumulates in daily work. If the team triggers an apply twenty times a day, total
plan duration in the single state climbs to twelve seconds, and all of that time passes under
lock; in split form, most of the same twenty applies land on different parts, so each one is both
shorter and does not wait on the others. The real meaning of the measurement is not duration, it
is **how many people are waiting**: in the single state, every apply keeps the whole team waiting;
in split form, only the people touching the same part.

Another effect is the readability of the plan output. Finding the one changed item in a plan that
refreshed six hundred twenty items is harder than finding it in a plan of fifty; drift gets lost
in the same pile. As scope shrinks, the odds that the plan actually gets read go up, and a read
plan is the one thing that can still be stopped before it is applied.

#### The Cost of Splitting

Splitting does not eliminate dependencies, it makes them visible. Each of the sixty-six edges that
cross a part boundary now has to be built between two separate state files.

```js
// dependency.mjs — counts cross-part edges, how they are resolved, and reapply propagation.
import { readFileSync, readdirSync } from "node:fs";

const parts = readdirSync("split").map((f) => f.replace(".json", ""));
const state = {}, itemPart = new Map();
for (const p of parts) {
  state[p] = JSON.parse(readFileSync(`split/${p}.json`, "utf8")).items;
  for (const k of state[p]) itemPart.set(k.name, p);
}
// Parts that publish outputs: they broadcast their values by name. Others can only be queried.
const outputParts = new Set(parts.filter((p) => /(^shared$|_network$|_data$)/.test(p)));

const edges = [];
for (const p of parts)
  for (const k of state[p])
    for (const b of k.deps) { const h = itemPart.get(b); if (h !== p) edges.push([p, h]); }
const byOutput = edges.filter(([, h]) => outputParts.has(h)).length;

console.log(`cross-part dependencies: ${edges.length} edges, ` +
  `${new Set(edges.map((e) => e.join(">"))).size} distinct part pairs`);
console.log(`  resolved by reading an output: ${byOutput}, ` +
  `resolved by data source: ${edges.length - byOutput}`);
const incoming = {};
for (const [, h] of edges) incoming[h] = (incoming[h] ?? 0) + 1;
console.log(`  most connected: ` + Object.entries(incoming).sort((a, b) => b[1] - a[1])
  .slice(0, 3).map(([p, n]) => `${p} ${n} edges`).join(", "));

// A data source returns everything matching the query: the record's uncovered resource too.
const provider = JSON.parse(readFileSync("provider.json", "utf8"));
const returned = provider.filter((a) => a.startsWith("production_compute/")).length;
const recorded = state["production_compute"].length;
console.log(`  data source query (production_compute): ${returned} items returned, ` +
  `recorded ${recorded}, out of record ${returned - recorded}`);
console.log(`  same link built by reading an output: ${recorded} items, out of record 0`);

const dependents = {};                           // target part -> parts that depend on it
for (const [k, h] of edges) (dependents[h] ??= new Set()).add(k);
const propagate = (p) => {
  const seen = new Set([p]), stack = [p];
  while (stack.length) {
    const x = stack.pop();
    for (const y of dependents[x] ?? []) if (!seen.has(y)) { seen.add(y); stack.push(y); }
  }
  return seen;
};
for (const p of ["production_observability", "production_data", "production_network", "shared"]) {
  const g = propagate(p);
  const items = [...g].reduce((t, x) => t + state[x].length, 0);
  console.log(`  ${p.padEnd(24)} changes: ${String(g.size).padStart(2)} parts reapplied, ` +
    `${String(items).padStart(3)} items refreshed, ${String(g.size).padStart(2)} locks taken`);
}
```

**SM15.** The network and data layers, along with the regional backbone, publish their values by
name as **outputs**; the compute layer does not publish, and connecting to it requires a **data
source** query. **SM16.** Three resources the record does not cover sit in the production compute
layer.

```
cross-part dependencies: 66 edges, 18 distinct part pairs
  resolved by reading an output: 51, resolved by data source: 15
  most connected: production_network 9 edges, staging_network 9 edges, test_network 9 edges
  data source query (production_compute): 53 items returned, recorded 50, out of record 3
  same link built by reading an output: 50 items, out of record 0
  production_observability changes:  1 parts reapplied,  50 items refreshed,  1 locks taken
  production_data          changes:  3 parts reapplied, 150 items refreshed,  3 locks taken
  production_network       changes:  4 parts reapplied, 200 items refreshed,  4 locks taken
  shared                   changes: 13 parts reapplied, 620 items refreshed, 13 locks taken
```

Sixty-six edges spread across eighteen part pairs. Fifty-one are resolved by **reading an
output**: the target part publishes a value by name, the connecting part reads that name. The
link is recorded and both sides are known; its cost is a version bond — if the target part removes
an output, every edge that depends on it breaks at the same time, and the break is only learned at
apply time.

The distribution of edges says something too: the three most connected parts are the network
layers of the three environments, and each carries nine edges. These three parts are the split's
breaking point. A naming change in the network layer's output list is written in a single part,
felt across nine edges, and forces three parts to reapply. Output names are therefore the parts'
**interface**; resource names inside a part can change freely, output names cannot.

The remaining fifteen are resolved by **data source**: the target part has no idea it is being
connected to, the connecting side queries the provider instead. The measurement shows the cost of
this path: the query returned fifty-three items, and fifty are in the record. Three items are
resources the record does not cover, and the data source pulls them in too. If the same link had
been built by reading an output, the returned item count would be fifty, out of record zero. A
query reads **reality**, an output reads **the record**; the two diverge by exactly these three
items.

#### Reapply Propagation

The last four lines are the real bill for splitting. **SM17.** When a part's output changes, the
parts that depend on it reapply, and this is transitive; whoever depends on the dependent gets in
line too. Changing the observability part forces one part to reapply; changing the network part
forces four; changing the regional backbone forces all thirteen.

The last line deserves a careful read: when the shared part changes, six hundred twenty items
refresh — the exact same number as the single state file — but this time thirteen separate locks
are taken and thirteen applies are put in a queue. Splitting does not cheapen a wide change; it
**makes it more expensive**. In the single state, one apply did everything in the right order; in
split form, the person doing the work is now the one who has to build that order.

From this comes a rule: where to split is chosen by **frequency of change**. A layer that changes
often should stay in its own part so blast radius stays small; a layer that changes rarely and
that everyone connects to should be collected into a single part so its propagation is not
triggered often. Split in the wrong place and two harms combine: the count of cross-part edges
grows, and every change propagates to many parts.

#### The Multiplication of the Record

Splitting multiplies the record too, and that is the record's own risk. One file became thirteen:
thirteen locks, thirteen access points, thirteen separate loss scenarios. The previous lesson's
measurement applies to each of them separately. Losing one part's state only makes that part
unmanageable — which is a gain — but the loss becomes harder to notice, because the other twelve
parts keep applying without a problem.

The second risk is the boundary itself. The part boundary is written down in no state file; which
item belongs to which part is visible only in how the files are distributed. If an item ends up in
two parts' definitions by mistake, two records claim ownership of the same resource, and the two
applies take turns undoing each other's work. The lock does not prevent this: two different files
have two different locks.

The third risk is **changing** the boundary. Splitting is not a one-time act; over time an item
turns out to sit in the wrong part and has to be moved. Moving is not copying the definition from
one file to another. The moment the item is removed from the old part's definition, that part's
plan shows it as **to be deleted**; the new part's plan shows it as **to be created**. Both plans
are correct on their own, but if they are applied one after the other, the resource is really
deleted and recreated. The right way is to remove the item's record from the old state by id and
add it to the new state by id — the reverse and continuation of the import from the first lesson.
Because this touches two files at once, it needs both locks at the same time, and if it is caught
in between, the item shows up in no record at all: the count of resources the record does not
cover rises from three to four.

#### Summary

- Of the six-hundred-twenty-item network's six hundred seventy-three edges, only sixty-six cross
  a part boundary; ninety percent of the links are already local.
- In the single state, a one-item change refreshed six hundred twenty items and locked six
  hundred twenty items; in split form, the same work refreshed fifty items and locked fifty
  items.
- Of the sixty-six cross-part edges, fifty-one are resolved by reading an output, fifteen by
  data source; the data source query returned fifty-three items, three of which were outside the
  record.
- The number of parts that reapply when one part changes ranges from one to thirteen; changing
  the regional backbone requires thirteen locks and six hundred twenty refreshes.
- Splitting shrinks the lock, grows coordination; where to split is chosen by frequency of
  change.
- Thirteen files mean thirteen loss scenarios, and the part boundary is written down in no state
  file.

#### Next Step

This lesson's split was made by layer: network, data, compute, observability. Environment was
also in the part names, but how the environment distinction itself is built was never examined.
If the definition of the three environments is the same, does the same text have to be written
three times, or does a single definition keep three instances? Both setups have a cost, and it
collects in the same place: whether the person starting the apply knows which environment they
are in. The next lesson builds the three environments in two separate forms, measures the risk of
applying to the wrong environment by step count, counts environment-specific differences, and
compares how many environments are forced to change together when the definition changes.

### Workspaces and Environments

Source: https://academia.sh/en/courses/infrastructure-as-code/workspaces-and-environments

Two setups for three environments: a single definition plus workspace versus separate directories, the count of environment-specific differences, the number of signals that say which environment you are in and applying to the wrong one, and how many environments change together when the definition changes, along with the divergence.

The previous lesson split state into parts, and environment appeared in the part names. How the
environment distinction itself is built was never examined. If the definition of the three
environments is largely the same, there are two paths: write a single definition and pick the
environment with a setting, or keep a separate directory for each environment. This lesson
actually builds both and measures the difference between them with three numbers.

How similar the environments are to each other — **environment parity** — was measured earlier
and is not repeated here. The question here is not similarity, it is **where that similarity is
written**.

#### Three Instances of the Same Definition

**SM19.** The configuration language below is the small form defined for this lesson; the
`environment(...)` expression picks a value according to the selected environment. The script
writes both setups to disk: a single definition plus workspace, and three separate directories.

```js
// setup.mjs — the three environments are built two ways: single definition + workspace, and separate directories.
// The configuration language is the small form defined for this lesson; the environment(...) expression picks a value per environment.
import { writeFileSync, mkdirSync, rmSync } from "node:fs";

const environments = ["test", "staging", "production"];
const DEFINITION = `resource subnet backbone {
  block = environment(test="10.0.0.0/16", staging="10.1.0.0/16", production="10.2.0.0/16")
}
resource machine collector {
  count      = environment(test=1, staging=2, production=6)
  cores      = environment(test=2, staging=2, production=8)
  memory_gb  = environment(test=8, staging=8, production=32)
  image      = "measurement-collector-4.2"
}
resource queue read {
  retention_hours = environment(test=6, staging=24, production=168)
  partitions      = environment(test=1, staging=4, production=12)
}
resource data_store measurement {
  size_gb = environment(test=20, staging=100, production=2000)
  backup  = environment(test=false, staging=false, production=true)
}`;

// Resolves the environment(...) expression for the given environment; a non-conditional line is left as is.
const resolve = (text, env) => text.replace(/environment\(([^)]*)\)/g, (_, ic) => {
  const match = new RegExp(`${env}=([^,]+)`).exec(ic);
  return match[1].trim();
});

rmSync("workspace", { recursive: true, force: true });
rmSync("separate-dir", { recursive: true, force: true });
mkdirSync("workspace/state", { recursive: true });
writeFileSync("workspace/definition.txt", DEFINITION);
writeFileSync("workspace/.selected", "test");
for (const e of environments)
  writeFileSync(`workspace/state/${e}.json`, JSON.stringify({ env: e, items: [] }));
for (const e of environments) {
  mkdirSync(`separate-dir/${e}`, { recursive: true });
  writeFileSync(`separate-dir/${e}/definition.txt`, resolve(DEFINITION, e));
  writeFileSync(`separate-dir/${e}/state.json`, JSON.stringify({ env: e, items: [] }));
}

const lines = DEFINITION.split("\n");
const conditional = lines.filter((s) => s.includes("environment(")).length;
const fields = lines.filter((s) => s.includes("=")).length;
const separateLines = environments.reduce((t, e) => t + resolve(DEFINITION, e).split("\n").length, 0);
const sharedFields = fields - conditional;

console.log(`workspace: 1 definition file, ${lines.length} lines, ` +
  `${fields} fields (${conditional} environment-specific, ${sharedFields} shared), 3 state files`);
console.log(`separate dir: 3 definition files, ${separateLines} lines, ` +
  `${fields * 3} fields (${conditional * 3} written separately, ${sharedFields * 3} duplicated), 3 state files`);
console.log(`environment-specific difference: ${conditional} fields; identical across all three: ${sharedFields} fields`);
```

```
workspace: 1 definition file, 17 lines, 9 fields (8 environment-specific, 1 shared), 3 state files
separate dir: 3 definition files, 51 lines, 27 fields (24 written separately, 3 duplicated), 3 state files
environment-specific difference: 8 fields; identical across all three: 1 fields
```

Both setups describe the same infrastructure, keep the same three state files, and produce the
same environment distinction. The difference is in the amount of text written, and in **where the
difference stands**.

In the workspace form, the eight environment-specific differences sit in a single file, inside
eight conditional expressions. In the separate-directory form, the same eight differences spread
across twenty-four separate lines — each environment carries its own value in its own file — and
the one field identical across all three environments is duplicated three times.

What this first pair of numbers says is this: collecting the difference in one place makes it
**visible**. To learn where the three environments diverge, looking at one file is enough in the
workspace form; in the separate-directory form, three files have to be compared. Against that, in
the separate-directory form each environment's definition can be read on its own: understanding
what production is does not require resolving conditionals in your head.

On the state side, there is no difference at all between the two setups: both keep one state file
per environment, three in total. The numbers from the previous two lessons therefore apply the
same way to both — the lock is taken per environment, the three environments can apply without
waiting on each other, and losing one environment's state only makes that environment
unmanageable. The workspace shares the **definition**; the one thing it does not share is state,
and this distinction comes back at the end of the lesson.

#### Which Environment Am I In

The second measurement belongs to operations. Before pressing the button, how many signals can the
person starting the apply read to know which environment they are in?

**SM20.** In the workspace form, the selected environment sits in a hidden file and is changed
with a separate command; it leaves no trace on the command line. In the separate-directory form,
the choice is the directory the command runs in, and it shows up in the path. **SM21.** The
probability of skipping the selection step is tied to the number of signals: fifteen percent
without a signal, three percent with one. **SM22.** A step that prints the environment name to the
screen and asks for confirmation before the apply catches nine out of ten skipped selections. The
generator is written inside the lesson, and the seed is visible.

```js
// env-choice.mjs — the step of knowing which environment you are in, and a model of applying to the wrong one.
import { readFileSync, writeFileSync } from "node:fs";

let x = 20260801 >>> 0;                          // seed is visible
const rand = () => { x = (Math.imul(x, 1664525) + 1013904223) >>> 0; return x / 4294967296; };
const environments = ["test", "staging", "production"];

// How many signals can the person starting the apply read to know which environment they are in?
console.log("which environment am I in: workspace -> 0 signals on screen, 1 extra command (selected file)");
console.log("                           separate dir -> 1 signal in the path, 0 extra commands");

const run = (label, forgetProb, confirmCatch) => {
  writeFileSync("workspace/.selected", "test");
  let wrong = 0, fellToProd = 0;
  for (let i = 0; i < 200; i++) {
    const target = environments[Math.floor(rand() * 3)];
    const current = readFileSync("workspace/.selected", "utf8");
    let picked = target;
    if (rand() < forgetProb) {                   // selection step skipped: previous environment stands
      picked = current;
      if (rand() < confirmCatch) picked = target; // confirmation step caught it
    }
    writeFileSync("workspace/.selected", picked);
    if (picked !== target) { wrong++; if (picked === "production") fellToProd++; }
  }
  console.log(`${label.padEnd(29)}: 200 applies, ${String(wrong).padStart(2)} wrong environment, ` +
    `${fellToProd} fell to production`);
};

run("workspace", 0.15, 0);
run("separate dir", 0.03, 0);
run("workspace + mandatory confirm", 0.15, 0.9);
```

```
which environment am I in: workspace -> 0 signals on screen, 1 extra command (selected file)
                           separate dir -> 1 signal in the path, 0 extra commands
workspace                    : 200 applies, 23 wrong environment, 8 fell to production
separate dir                 : 200 applies,  4 wrong environment, 1 fell to production
workspace + mandatory confirm: 200 applies,  2 wrong environment, 1 fell to production
```

Out of two hundred applies, the workspace form went to the wrong environment twenty-three times,
eight of which fell to production. In the separate-directory form, the count drops to four, and
the fall to production drops to one. The source of the difference is one single thing: whether
there is a signal on screen saying which environment this is.

The third row matters more. When a confirmation step that prints the environment name before the
apply was added to the workspace form, wrong applies dropped from twenty-three to two — better
than the separate-directory form. The result says that neither setup is inherently safe: what is
safe is the environment being **visible**. Separate directories give this for free because the
path is already on screen; in the workspace form, the same visibility has to be built separately.

The cost of applying to the wrong environment ties back to the measurement in this course's first
lesson. Each of the eight applies that fell to production writes test values into production's
state: machine count drops from six to one, the data store from two thousand gigabytes to twenty.
The gap between record and reality here is produced not by drift but by the record itself — and
the moment the apply finishes, both the record and reality are wrong.

#### How Many Environments Change When the Definition Changes

The third measurement is about time. When a field is added to the definition, how many
environments are forced to change together? **SM23.** Eight of twelve changes concern all three
environments, four are environment-specific; in the separate-directory form, each manual move is
skipped with a one-in-four probability. **SM24.** The apply was only run in the test environment.

```js
// change.mjs — how many environments are forced to change together when the definition changes, and the divergence.
import { readFileSync, writeFileSync, appendFileSync } from "node:fs";

let x = 991 >>> 0;                               // seed is visible
const rand = () => { x = (Math.imul(x, 1664525) + 1013904223) >>> 0; return x / 4294967296; };
const environments = ["test", "staging", "production"];

// Twelve changes: eight concern all three environments, four are environment-specific.
const changes = [];
for (let i = 1; i <= 8; i++) changes.push({ no: i, scope: "all" });
for (let i = 9; i <= 12; i++) changes.push({ no: i, scope: environments[(i - 9) % 3] });

let newConditional = 0, moveNeeded = 0, moveSkipped = 0;
for (const d of changes) {
  if (d.scope === "all") {                       // single definition: one line, all three environments
    appendFileSync("workspace/definition.txt", `\n  new_field_${d.no} = "d${d.no}"`);
    for (const e of environments) {               // separate dir: moved by hand into each directory
      moveNeeded++;
      if (rand() < 0.25) { moveSkipped++; continue; }
      appendFileSync(`separate-dir/${e}/definition.txt`, `\n  new_field_${d.no} = "d${d.no}"`);
    }
  } else {                                        // environment-specific: needs a conditional expression
    newConditional++;
    const inner = environments.map((e) => `${e}=${e === d.scope ? `"d${d.no}"` : '""'}`).join(", ");
    appendFileSync("workspace/definition.txt", `\n  new_field_${d.no} = environment(${inner})`);
    appendFileSync(`separate-dir/${d.scope}/definition.txt`, `\n  new_field_${d.no} = "d${d.no}"`);
  }
}

const conditional = (s) => s.split("\n").filter((l) => l.includes("environment(")).length;
const fields = (file) => new Set(readFileSync(file, "utf8").split("\n")
  .filter((l) => l.includes("new_field_")).map((l) => /new_field_\d+/.exec(l)[0]));
const expected = (e) => changes.filter((d) => d.scope === "all" || d.scope === e).length;
const missing = environments.map((e) => expected(e) - fields(`separate-dir/${e}/definition.txt`).size);

console.log(`workspace: 1 definition file, ${changes.length} changes processed`);
console.log(`  conditional expressions 8 -> ${conditional(readFileSync("workspace/definition.txt", "utf8"))} ` +
  `(each environment-specific change adds one conditional)`);
console.log(`  shared change: forced to change together 3/3 environments, diverged fields: 0`);
console.log(`separate dir: 3 definition files, ${changes.length} changes processed`);
console.log(`  shared change required ${moveNeeded} moves, ${moveSkipped} moves skipped`);
console.log(`  shared change: forced to change together 1/3 environments, remaining 2 by hand`);
console.log(`  diverged fields: ${missing.reduce((a, b) => a + b, 0)} ` +
  `(${environments.map((e, i) => `${e} ${missing[i]}`).join(", ")})`);

// The definition changed, the apply only ran in the test environment.
const behind = (form) => environments.filter((e) => e !== "test")
  .map((e) => `${e} ${expected(e) - (form === "separate" ? missing[environments.indexOf(e)] : 0)}`).join(", ");
console.log(`record vs. reality (only test applied), fields waiting to be applied:`);
console.log(`  workspace -> ${behind("workspace")} | separate dir -> ${behind("separate")}`);
```

```
workspace: 1 definition file, 12 changes processed
  conditional expressions 8 -> 12 (each environment-specific change adds one conditional)
  shared change: forced to change together 3/3 environments, diverged fields: 0
separate dir: 3 definition files, 12 changes processed
  shared change required 24 moves, 2 moves skipped
  shared change: forced to change together 1/3 environments, remaining 2 by hand
  diverged fields: 2 (test 0, staging 2, production 0)
record vs. reality (only test applied), fields waiting to be applied:
  workspace -> staging 9, production 9 | separate dir -> staging 7, production 9
```

In the workspace form, a shared change is a single line and changes the definition of all three
environments **forced together**: 3/3. Diverged fields are zero, because there is no second file
to diverge. The cost is paid at the other end: four environment-specific changes gave birth to
four new conditional expressions; the conditional count rose from eight to twelve, and the file
became a little less readable with every environment difference.

In the separate-directory form, the same eight shared changes required twenty-four manual moves,
and two were skipped. The result is a two-field divergence in the staging environment. No one
reports this divergence; both definitions are valid on their own, and both applies run without
error. The difference is visible only when the three files are compared.

The skip rate itself is a measurement too. A one-in-four probability produced a two-field
divergence from twelve changes in one cycle; if the rate stays the same, the gap grows a little
bigger every cycle, and because no apply reports it, it accumulates. That the divergence lands in
the staging environment in this run is an accident of the model — which environment will diverge
is not known in advance, only that one will. This also means the separate-directory form needs a
step that **checks** the moves: without a check that compares the three definitions and reports
differences that are not environment-specific, divergence only surfaces as a difference in
behavior in production.

The last two numbers show the **record's own risk**. The definition changed in both forms, the
apply only ran in the test environment. In the workspace form, staging and production are each
left nine fields behind; in the separate-directory form, staging is seven, production is nine.
The record has changed for all three environments, reality for only one. A single definition does
not prevent the difference from **accumulating**; it only ensures it is written in one place. The
gap between environments is born not in the definition, but in **apply lag**.

#### The Trade-off

The three measurements point in three different directions, and there is no single winner. When
environment-specific difference is small, the workspace form wins: one file, zero divergence,
eight conditionals. As environment-specific difference grows, the conditional expression count
swallows the definition, and separate directories stay readable. The risk of applying to the wrong
environment is five times higher in the workspace form, but a confirmation step brings it down to
two — meaning this difference is not the setup's doing, it is the doing of the guard placed around
the setup.

The one thing worth dwelling on is what stays the same in both forms: the state of the three
environments is three separate files. The workspace shares only the single definition; it does not
share state. So a change applied in one environment does not happen automatically in the others,
and the record being the same for all three environments does not mean reality is the same.

#### Summary

- The same infrastructure was built two ways: a single definition file, seventeen lines and
  eight conditional expressions; three separate directories, fifty-one lines, twenty-four fields
  written separately and three duplicated.
- Environment-specific difference is eight fields, one field stays identical across all three
  environments; the difference is collected in one file in the workspace form, and spread across
  three files in the separate-directory form.
- Out of two hundred applies, the count of going to the wrong environment is twenty-three in the
  workspace form, four in the separate-directory form; a confirmation step printing the
  environment name brought the count down to two.
- A shared change in the workspace form changes all three environments together and produces
  zero divergence; in the separate-directory form, two of twenty-four manual moves were skipped
  and two fields diverged.
- Four environment-specific changes raised the conditional count from eight to twelve in the
  workspace form; in the separate-directory form, each stayed a single line in a single file.
- The definition changed for three environments, the apply ran in one: in the workspace form two
  environments were left nine fields behind, and in the separate-directory form staging seven and
  production nine. The gap between environments is born not in the definition but in
  apply lag.

#### Next Step

Both setups in this lesson tried to solve one repetition: the same infrastructure being told
three times. The workspace turned the repetition into a conditional expression; the separate
directory repeated it outright. Neither eliminated the repetition, they only moved where it
lives. When ten teams, instead of three environments, want to set up the same queue, the same
collector machine, the same archive, neither form is enough; what is needed is packaging the
definition itself, and naming and versioning its inputs. The remaining lessons in this topic take
up the infrastructure module: input surface, version pinning, and how a breaking change
propagates to every dependent user.

### Module Authoring

Source: https://academia.sh/en/courses/infrastructure-as-code/module-authoring

The same record, written separately at three collection points, is extracted into a single module; input surface, line gain and the cost of flexibility growing across six versions are counted. The four items the module's author never saw are also measured.

The previous lesson ran the same definition across multiple environments: the workspace split
state, and the environments were produced from the same text without touching each other's
resources. The definition itself was still one single piece, though. The regional measurement
network's three collection points contain the same three resources, and each carries its own
record block start to finish; nothing shared sits in a shared place.

A **module** is a piece of configuration that names a set of resources created and deleted
together, whose interface is made of **inputs** and **outputs**. The word is the same as `module`
in software packages; the difference in context is this: there, the interface was callable
functions and what it produced was behavior; here, the interface is a list of variables, and what
it produces is the resource itself.

The measurement is again on the **regional measurement network**: fictional software that collects
water meter readings. No real infrastructure is provisioned; what is measured is a **model** built
with `node`.

**SM25.** There are three collection points; each consists of three resources, and the three
records were written separately by hand.
**SM26.** The configuration language is defined for this lesson: `resource <type> <name> { field =
value }`, `input <name> [= default]`, `output <name> = <value>`, `module <type> <name> { field =
value }`. `$name` is bound to the call's name, a field whose value is `none` is not produced, and
a resource carrying `if` is conditional.
**SM27.** The module is **written by hand**; the author decides which field surfaces on the
interface. The code tests the module against the three records and compares the interface against
the fields that change in the record.
**SM28.** Only required inputs and fields that diverge from the default are written into the call.
**SM29.** Four items were actually changed by hand and appear in no record; all four are written
out, there is no randomness.
**SM30.** The module grows to six versions across five new uses; each version's change is written
in advance, and one of the eight opened uses is later closed.

#### The Same Record in Three Places

```js
// records.mjs — the regional measurement network's three collection points; three separately hand-written records.
export const RECORD = `# point north
resource virtual-machine-cluster north-collector {
  count = 4
  image = collector-4.2.0
  region = north-1
}
resource disk north-buffer {
  size-gb = 200
  attached = north-collector
}
resource network-rule north-inbound {
  source-block = 10.20.0.0/16
  port = 8080
  direction = inbound
}
# point west
resource virtual-machine-cluster west-collector {
  count = 6
  image = collector-4.2.0
  region = west-1
}
resource disk west-buffer {
  size-gb = 500
  attached = west-collector
}
resource network-rule west-inbound {
  source-block = 10.30.0.0/16
  port = 8080
  direction = inbound
}
# point east
resource virtual-machine-cluster east-collector {
  count = 6
  image = collector-4.2.0
  region = east-1
}
resource disk east-buffer {
  size-gb = 500
  attached = east-collector
}
resource network-rule east-inbound {
  source-block = 10.40.0.0/16
  port = 8443
  direction = inbound
}`;

export const POINT = ["north", "west", "east"];
export const MANUAL = [                                // exists in reality, in no record
  ["west", "disk", "size-gb", "750"],                  // disk manually enlarged
  ["north", "virtual-machine-cluster", "count", "6"],  // cluster manually scaled
  ["east", "network-rule", "extra-port", "9443"],      // field not in the record added by hand
  ["north", "object-store", "*", "north-archive"]];    // out-of-record resource opened by hand
```

The record is forty-two lines, and each point has eight field slots. The `MANUAL` list is the
second reality standing next to the record: four items that exist in the field but are written in
no record. How many of these four the module's author sees — that is what gets measured.

#### The Module's Input Surface

```js
// module.mjs — a hand-written module tested against three records; input surface, line gain and record-reality gap.
import { RECORD, POINT, MANUAL } from "./records.mjs";

export const lineCount = (t) => t.split("\n").filter((s) => s.trim()).length;
export const TEXT = Object.fromEntries(RECORD.split(/^# point /m).filter(Boolean)
  .map((p) => [p.split("\n")[0], p.split("\n").slice(1).join("\n").trim()]));
export const K = Object.fromEntries(POINT.map((n) => [n, TEXT[n].split("\n").reduce((b, s) => {
  const k = s.match(/^resource (\S+) (\S+) \{$/), f = s.match(/^ {2}([\w-]+) = (.+)$/);
  if (k) b.push({ type: k[1], name: k[2], fields: [] }); else if (f) b.at(-1).fields.push([f[1], f[2]]);
  return b;
}, [])]));
const D = Object.fromEntries(POINT.map((n) => [n, Object.fromEntries(K[n].flatMap((b) => b.fields))]));

export const MODULE = `input source-block
input count = 6
input size-gb = 500
input port = 8080
resource virtual-machine-cluster $name-collector {
  count = $count
  image = collector-4.2.0
  region = $name-1
}
resource disk $name-buffer {
  size-gb = $size-gb
  attached = $name-collector
}
resource network-rule $name-inbound {
  source-block = $source-block
  port = $port
  direction = inbound
}
output buffer-name = $name-buffer
output inbound-port = $port`;                          // written by hand; the choice of output is not in the record

export const generate = (mod, name, arg) => {          // $name is the call's name; a field whose value is `none` is not produced
  const names = [...mod.matchAll(/^input ([\w-]+)/gm)].map((m) => m[1]).sort((a, b) => b.length - a.length);
  const lines = []; let skip = false;
  for (const s of mod.split("\n")) {
    if (/^(input|output) /.test(s)) continue;
    let t = s.replaceAll("$name", name);
    for (const i of names) t = t.replaceAll("$" + i, arg[i]);
    const k = t.match(/ if (\S+) \{$/);
    if (k) { skip = k[1] === "no"; t = t.replace(/ if \S+ \{$/, " {"); }
    else if (t === "}" && skip) { skip = false; continue; }
    if (!skip && !/ = none$/.test(t)) lines.push(t);
  }
  return lines.join("\n");
};
export const required = (m) => [...m.matchAll(/^input ([\w-]+)$/gm)].map((x) => x[1]);
export const defaults = (m) => Object.fromEntries([...m.matchAll(/^input ([\w-]+) = (.+)$/gm)].map((x) => [x[1], x[2]]));
export const passed = (m, n) => {                      // fields that diverge from required and default are passed to the call
  const v = defaults(m);
  return Object.fromEntries([...required(m), ...Object.keys(v)].filter((a) => D[n][a] !== v[a]).map((a) => [a, D[n][a]]));
};
export const callLines = (m, n) => Object.keys(passed(m, n)).length + 2;   // line count of the call

if (import.meta.main) {                                // does not run when version.mjs imports this
  const fields = Object.keys(D[POINT[0]]), v = defaults(MODULE);
  const changing = fields.filter((a) => new Set(POINT.map((n) => D[n][a])).size > 1);
  const iface = [...required(MODULE), ...Object.keys(v)], outside = changing.filter((a) => !iface.includes(a));
  const same = POINT.filter((n) => generate(MODULE, n, { ...v, ...passed(MODULE, n) }) === TEXT[n]).length;
  const byHand = POINT.reduce((t, n) => t + lineCount(TEXT[n]), 0), calls = POINT.map((n) => callLines(MODULE, n));
  const total = calls.reduce((x, y) => x + y);
  console.log(`field slots ${fields.length} | changing ${changing} | fixed ${fields.length - changing.length}` +
    ` | input surface ${iface.length} = required ${required(MODULE)} + defaulted ${Object.keys(v)}`);
  console.log(`changing but outside interface ${outside.length} (${outside}) | module ${lineCount(MODULE)} = ${iface.length}+` +
    `${lineCount(MODULE) - iface.length - 2}+2 | by hand ${byHand} | module path ${lineCount(MODULE)}+${total}=${lineCount(MODULE) + total}` +
    ` | gain ${byHand - lineCount(MODULE) - total} | calls ${calls} | same as original ${same}/3`);
  let covered = 0;
  for (const [n, type, a, d] of MANUAL) {              // the gap between record and reality
    const b = K[n].find((x) => x.type === type), y = b?.fields.find(([x]) => x === a);
    console.log(`  ${n.padEnd(6)} ` + (!b ? `out-of-record resource ${d}` : !y ? `field not in record ${a} = ${d}`
      : `value gap ${a}: record ${y[1]}, reality ${d}`).padEnd(44) + (b ? "in scope" : "out of scope"));
    if (b) covered++;
  }
  console.log(`items not written in record ${MANUAL.length} | seen by the module's author 0 | to be reverted ${covered}` +
    ` | left unrecorded ${MANUAL.length - covered}`);
}
```

```
field slots 8 | changing count,region,size-gb,attached,source-block,port | fixed 2 | input surface 4 = required source-block + defaulted count,size-gb,port
changing but outside interface 2 (region,attached) | module 20 = 4+14+2 | by hand 42 | module path 20+12=32 | gain 10 | calls 5,3,4 | same as original 3/3
  west   value gap size-gb: record 500, reality 750  in scope
  north  value gap count: record 4, reality 6        in scope
  east   field not in record extra-port = 9443       in scope
  north  out-of-record resource north-archive        out of scope
items not written in record 4 | seen by the module's author 0 | to be reverted 3 | left unrecorded 1
```

The input surface is four: one required, three defaulted. The module is twenty lines; four are
interface declarations, fourteen are body, two are output.

The first row counts what the interface is missing. Six of the eight field slots vary between the
three records, but only four are on the interface; the **two fields that vary but are not on the
interface** are `region` and `attached`. The measurement cannot tell them apart, because both are
written through `$name`. But `attached` belongs outside, rightfully: it is a dependency name,
pointing at a resource the module produces itself. `region` is not the same — the region value
starting with the point's name is not a rule, it is a pattern that happens to hold across three
examples. The module's author trusted the pattern and kept the field off the interface; the cost
is paid in the next section.

In the opposite direction stands `output`. Which resource's name gets exposed outward is written
in none of the three records. Input can be inferred from what differs in the record, **output
cannot be inferred.**

The call lines are five, three, and four. West is called with three lines because all three
defaults match west's record; choosing a default rewards the majority, and costs whoever is in the
minority an extra line. Line gain goes from forty-two to thirty-two, that is twenty-four percent —
not a number that would justify writing a module for three uses on its own.

#### The Four Items the Record Does Not Cover

The module reproduced the three records exactly, but saw zero of the four items that actually
exist: what it looked at was the record. Two are **value gaps** — west's disk is seven hundred
fifty gigabytes instead of five hundred, north's machine cluster is six instead of four. One is a
**field that is in no record at all**: a second port was manually opened on east's network rule.
The last is an **out-of-record resource**: no record says anything about the `north-archive`
object store up north.

The module's scope splits these four in two. Three sit inside the blocks the module produces; the
moment it applies, the disk reverts to five hundred, the machine count to four, and the extra port
is deleted — modularizing never saw these three items, but the apply will **silently revert**
them. The fourth is a resource type the module knows nothing about; the module neither produces it
nor deletes it. Modularizing does not reduce drift: it makes what it covers revertible from a
single place, and leaves what it does not cover exactly where it is.

#### The Cost of Flexibility

```js
// version.mjs — the module grows to six versions across five new uses; input surface and the cost of flexibility are counted.
import { MODULE, generate, lineCount, required, defaults } from "./module.mjs";

const append = (m, ...s) => m.replace(/^(input .*)$(?![\s\S]*^input )/m, ["$1", ...s].join("\n"));
const VERSION = [                                       // each version's change is written in advance
  ["v1", "written by hand", ["north", "west", "east"], (m) => m],
  ["v2", "region name pattern broken", ["south"], (m) =>
    append(m.replace("  region = $name-1", "  region = $region"), "input region")],
  ["v3", "usage that wants no disk", ["test-site"], (m) =>
    append(m.replace("e disk $name-buffer {", "e disk $name-buffer if $has-disk {"), "input has-disk = yes")],
  ["v4", "outbound direction and a second port", ["field-lab"], (m) => append(m.replace("  direction = inbound", "  direction = $direction")
    .replace("  port = $port\n", "  port = $port\n  extra-port = $extra-port\n"), "input direction = inbound", "input extra-port = none")],
  ["v5", "usage that wants an old image", ["archive-point"], (m) =>
    append(m.replace("  image = collector-4.2.0", "  image = $image"), "input image = collector-4.2.0")],
  ["v6", "machine tag", ["central-anatolia"], (m) =>
    append(m.replace("  count = $count\n", "  count = $count\n  tag = $tag\n"), "input tag = none")]];

const RAW = {                                           // the values each use passes into the module
  north: "source-block=10.20.0.0/16 region=north-1 count=4 size-gb=200",
  west: "source-block=10.30.0.0/16 region=west-1",
  east: "source-block=10.40.0.0/16 region=east-1 port=8443",
  south: "source-block=10.50.0.0/16 region=south-2",
  "test-site": "source-block=10.60.0.0/16 region=test-1 has-disk=no count=1",
  "field-lab": "source-block=10.70.0.0/16 region=field-1 direction=outbound extra-port=9443 count=1",
  "archive-point": "source-block=10.80.0.0/16 region=archive-1 image=collector-3.9.0 size-gb=2000",
  "central-anatolia": "source-block=10.90.0.0/16 region=central-1 tag=pilot count=2" };
const ARGS = Object.fromEntries(Object.entries(RAW).map(([k, v]) =>
  [k, Object.fromEntries(v.split(" ").map((x) => x.split("=")))]));

let m = MODULE, open = [], prevRequired = 1;
console.log("version   change                                 uses      input (required/defaulted)   module lines   breaking fix");
for (const [s, desc, newUses, transform] of VERSION) {
  m = transform(m);
  const req = required(m).length, total = req + Object.keys(defaults(m)).length;
  const breaking = req > prevRequired ? open.length : 0;   // a new required input breaks old calls
  open = [...open, ...newUses];
  console.log(`${s.padEnd(9)} ${desc.padEnd(38)} ${String(open.length).padEnd(9)} ${`${total} (${req}/${total - req})`.padEnd(28)} ` +
    `${String(lineCount(m)).padEnd(14)} ${breaking ? `${breaking} definitions, ${breaking} lines` : "-"}`);
  prevRequired = req;
}
open = open.filter((a) => a !== "archive-point");       // one use closed, the input it opened stayed in the module
const dv = defaults(m), varNames = Object.keys(dv);
const usedBy = (a) => open.filter((k) => a in ARGS[k]).length;
const dead = varNames.filter((a) => usedBy(a) === 0), single = varNames.filter((a) => usedBy(a) === 1);
const byHand = open.reduce((t, a) => t + lineCount(generate(m, a, { ...dv, ...ARGS[a] })), 0);
const cs = open.reduce((t, a) => t + Object.keys(ARGS[a]).length + 2, 0);

console.log(`opened 8, closed 1, open ${open.length} | met without changing the module 0/5 | input surface 4 -> ` +
  `${varNames.length + 2}, per use ${((varNames.length - 2) / 5).toFixed(1)} | by hand ${byHand} | module path ` +
  `${lineCount(m)}+${cs}=${lineCount(m) + cs} | gain ${byHand - lineCount(m) - cs}`);
console.log(`defaulted inputs ${varNames.length}: two or more ${varNames.length - single.length - dead.length}` +
  `, single use ${single.length} (${single}), dead ${dead.length} (${dead})`);
```

```
version   change                                 uses      input (required/defaulted)   module lines   breaking fix
v1        written by hand                        3         4 (1/3)                      20             -
v2        region name pattern broken             4         5 (2/3)                      21             3 definitions, 3 lines
v3        usage that wants no disk               5         6 (2/4)                      22             -
v4        outbound direction and a second port   6         8 (2/6)                      25             -
v5        usage that wants an old image          7         9 (2/7)                      26             -
v6        machine tag                            8         10 (2/8)                     28             -
opened 8, closed 1, open 7 | met without changing the module 0/5 | input surface 4 -> 10, per use 1.2 | by hand 96 | module path 28+38=66 | gain 30
defaulted inputs 8: two or more 1, single use 6 (size-gb,port,has-disk,direction,extra-port,tag), dead 1 (image)
```

The cost of flexibility reads in one line: **zero of the five new uses were met without changing
the module.** Every new use forced open a new input on the interface; the input surface went from
four to ten, and the module's text grew from twenty to twenty-eight lines. The measured rate is
1.2 inputs per use.

The `breaking fix` column at the second version is the cost's most visible form. When the
`region` field's naming pattern broke at the fourth point, the field surfaced as a required input,
and **all three** of the three definitions that existed at that moment became invalid. The burden
of opening a new required input falls not on the module's author, but on its users.

The last line counts who flexibility actually serves. Of the eight defaulted inputs, only one is
passed by two or more uses; six exist for a single use, one for none. The dead input, `image`, was
opened at the fifth version because the archive point wanted an old image; the archive point
closed, the input stayed in the module. Removing an input concerns every definition that uses it,
so no one takes it on.

The gain still grew, though. If the seven open uses had been written by hand, they would run
ninety-six lines; the module path is sixty-six lines, and the gain is thirty-one percent. Gain
grows with the number of uses, and so does the interface; the two do not grow at the same rate.

#### Summary

- The hand-written module's input surface is 4: 1 required, 3 defaulted; of 20 lines, 4 are
  interface declarations, 2 are output, and the choice of output cannot be read from the record.
- 6 of the 8 field slots vary across the three records, but only 4 are on the interface; `region`
  was left off because it relied on a naming pattern.
- Line gain went from 42 to 32 (24 percent) at three uses, and from 96 to 66 (31 percent) at
  seven uses.
- The module reproduced the three records exactly (3/3) but saw 0 of the 4 items that actually
  exist; 3 are silently reverted on apply, and the out-of-record `north-archive` stays unrecorded.
- 0 of the 5 new uses were met without changing the module; the input surface went from 4 to 10,
  and of 8 defaulted inputs, 6 serve a single use and 1 serves none.
- Opening a new required input is breaking: the second version required a one-line fix in all 3
  of the 3 definitions that existed at the time; this burden falls on the users.

#### Next Step

All six versions of this module lived in the same repository, under the same hand; the author and
the user were the same side. Once modules start being pulled from a shared place, that
relationship breaks: the upgrade decision is made in one place and felt in others. The next
lesson runs two setups, version-pinned and unpinned, over one period, and counts this: how many
applies produce a different result while the definition file never changed, how many users an
upgrade touches, and how many lines of fixing a breaking change demands across how many
definitions. The cost of pinning is measured in the same place: how many modules stay on an old
version, and how many fixes they never get.

### Module Registries

Source: https://academia.sh/en/courses/infrastructure-as-code/module-registries

Three modules pulled from a shared module registry, nine definitions and nine releases run a ten-week period in two setups; the version-pinned and unpinned setups are compared by unexpected change, broken definitions and fix-line count. The cost of pinning itself is also counted.

The previous lesson kept the module under a single hand: the author and the user were the same
side, and a version upgrade was a single file edit. A **module registry** cuts that relationship.
The registry is a shared repository that keeps versioned module text outside the definitions;
here it is referred to not as a product or a service, but **as a type**. The module's author
publishes a release, the user pulls it, and the only link between them is a name written in the
definition.

Whether to write a version next to that name is this lesson's pair of measurements. In the
**unpinned** setup, the definition writes only the module's name, and the newest version in the
registry is pulled at apply time. In the **pinned** setup, the definition writes the version too,
and that version stands until someone changes it by hand. The same release flow and the same
apply calendar are run in both setups; what gets measured is the gap between them.

The measurement is again on the **regional measurement network**: fictional software that collects
water meter readings. No real infrastructure is provisioned, and no real registry service is
connected to; what is measured is a **model** built with `node`.

**SM31.** The registry holds three modules and twelve versions; each version is written on one
line, with its required inputs, defaulted inputs, and body.
**SM32.** The period is ten weeks, and nine releases are written in advance: four patches, two
minors, three breaking. There is no randomness.
**SM33.** Nine definitions each call one module; apply weeks are written in advance, and eighteen
applies run over the period.
**SM34.** The unpinned setup pulls the newest version at apply time, the pinned setup pulls the
version written in the definition.
**SM35.** A breaking release stops the apply; the missing required input is written into the
definition on the spot, and the line is counted. The fix rule is written in `FIX`.
**SM36.** An **unexpected change** is the output being different from the previous one even though
the definition text was not touched in that apply. In the pinned setup, two definitions are
upgraded by hand during the period.

#### The Registry, the Release Flow and the Definitions

```js
// registry.mjs — module registry (as a type): three modules, twelve versions, nine releases and nine consumer definitions.
export const MODULE = {                                // version -> "required | defaulted | body"
  "collection-point": {
    "1.0.0": "source-block | count=6 | cluster count=$count; network block=$source-block",
    "1.1.0": "source-block | count=6,monitoring=on | cluster count=$count; network block=$source-block; monitoring status=$monitoring",
    "1.1.1": "source-block | count=6,monitoring=on | cluster count=$count; network block=$source-block direction=inbound; monitoring status=$monitoring",
    "2.0.0": "network-block | count=6,monitoring=on | cluster count=$count; network block=$network-block direction=inbound; monitoring status=$monitoring",
    "2.0.1": "network-block | count=6,monitoring=on | cluster count=$count type=collector; network block=$network-block direction=inbound; monitoring status=$monitoring" },
  "measurement-queue": {
    "1.0.0": "name | wait=30 | queue name=$name wait=$wait",
    "1.0.1": "name | wait=30 | queue name=$name wait=$wait; dead-letter name=dead-$name",
    "1.1.0": "name | wait=30,cap=50000 | queue name=$name wait=$wait cap=$cap; dead-letter name=dead-$name",
    "2.0.0": "name,region | wait=30,cap=50000 | queue name=$name wait=$wait cap=$cap region=$region; dead-letter name=dead-$name" },
  "field-network": {
    "1.0.0": "port |  | network-rule port=$port",
    "2.0.0": "port,direction |  | network-rule port=$port direction=$direction",
    "2.0.1": "port,direction |  | network-rule port=$port direction=$direction logging=on" } };

export const RELEASE = ("1 measurement-queue 1.0.1 patch|2 collection-point 1.1.0 minor|3 field-network 2.0.0 breaking|" +
  "4 collection-point 1.1.1 patch|5 measurement-queue 1.1.0 minor|6 collection-point 2.0.0 breaking|" +
  "7 measurement-queue 2.0.0 breaking|8 collection-point 2.0.1 patch|9 field-network 2.0.1 patch")
  .split("|").map((s) => s.split(" ")).map(([h, m, s, t]) => [+h, m, s, t]);

export const DEFINITION = {                            // "module | passed | pinned version | apply weeks"
  "north-point": "collection-point | source-block=10.20.0.0/16 count=4 | 1.0.0 | 2,6,9",
  "west-point": "collection-point | source-block=10.30.0.0/16 | 1.0.0 | 3,7",
  "east-point": "collection-point | source-block=10.40.0.0/16 count=8 | 1.0.0 | 1,5,10",
  "south-point": "collection-point | source-block=10.50.0.0/16 | 1.0.0 | 8",
  "north-queue": "measurement-queue | name=north-read | 1.0.0 | 2,8",
  "west-queue": "measurement-queue | name=west-read wait=60 | 1.0.0 | 4,9",
  "central-queue": "measurement-queue | name=central-read | 1.0.0 | 6,10",
  "field-inbound": "field-network | port=8080 | 1.0.0 | 4,10",
  "lab-inbound": "field-network | port=9443 | 1.0.0 | 5" };
export const UPGRADE = [[7, "west-point", "1.1.1"], [9, "north-queue", "1.1.0"]];   // manual, pinned setup
export const FIX = {                                   // the line written into the definition after a breaking release
  "network-block": (a) => a["source-block"], direction: () => "inbound", region: (a) => a.name.split("-")[0] };
```

The three modules' twelve versions are read together with the release flow. `collection-point`
renames its required input to `network-block` in the sixth week, `field-network` demands a new
required input called `direction` in the third week, `measurement-queue` demands `region` in the
seventh week. All three are **breaking** releases, and the cost of all three shows up in the
definitions that use the registry.

#### Two Setups, the Same Ten Weeks

```js
// period.mjs — the same release flow and the same apply calendar are run in two setups; the gap is counted.
import { MODULE, RELEASE, DEFINITION, UPGRADE, FIX } from "./registry.mjs";

const kv = (s) => Object.fromEntries(s.trim() ? s.trim().split(/[ ,]+/).map((x) => x.split("=")) : []);
const DEF = Object.fromEntries(Object.entries(DEFINITION).map(([t, s]) => {
  const [m, g, p, h] = s.split(" | ");
  return [t, { m, g: kv(g), p, h: h.split(",").map(Number) }];
}));
const resolve = (s) => { const [z, v, b] = s.split(" | ");
  return { req: z.trim() ? z.trim().split(",") : [], vars: kv(v), body: b.split("; ") }; };
const latest = (m, h) => (RELEASE.filter((y) => y[1] === m && y[0] <= h).at(-1) ?? [, , "1.0.0"])[2];
const render = (mv, a) => mv.body.map((l) => l.replace(/\$([\w-]+)/g, (_, x) => a[x])).join("; ");
const diff = (a, b) => {                               // words added/changed between two outputs
  const x = a.split(/[ ;]+/), y = b.split(/[ ;]+/);
  return y.filter((w) => !x.includes(w)).join(" ") || "(removed)";
};

const run = (pinned) => {
  const args = {}, pin = {}, prevOut = {}, ran = {}, last = {}, prevVer = {}, unexpected = [];
  let unexpectedCount = 0, brokenDefs = new Set(), brokenLines = 0, changed = 0;
  for (const t in DEF) { args[t] = { ...DEF[t].g }; pin[t] = DEF[t].p; ran[t] = new Set(); }
  for (let h = 1; h <= 10; h++) {
    if (pinned) for (const [hy, t, s] of UPGRADE) if (hy === h) { pin[t] = s; args[t].$fixed = 1; }
    for (const t in DEF) {
      if (!DEF[t].h.includes(h)) continue;
      const s = pinned ? pin[t] : latest(DEF[t].m, h);
      const mv = resolve(MODULE[DEF[t].m][s]), full = { ...mv.vars, ...args[t] };
      const missing = mv.req.filter((a) => !(a in full));
      let fixed = args[t].$fixed;
      if (missing.length) {                            // breaking release: the definition is fixed on the spot
        for (const e of missing) { args[t][e] = FIX[e](full); full[e] = args[t][e]; }
        brokenDefs.add(t); brokenLines += missing.length; fixed = 1;
      }
      const c = render(mv, full);
      ran[t].add(s);
      if (prevOut[t] !== undefined && prevOut[t] !== c) {
        changed++;
        if (!fixed) { unexpectedCount++; unexpected.push([h, t, prevVer[t], s, diff(prevOut[t], c)]); }
      }
      prevOut[t] = c; last[t] = s; prevVer[t] = s; delete args[t].$fixed;
    }
  }
  return { unexpectedCount, brokenDefs, brokenLines, changed, pin, ran, last, unexpected };
};

const UNPINNED = run(false), PINNED = run(true), total = Object.keys(DEF).length;
const versionsOf = (m) => RELEASE.filter((y) => y[1] === m);
console.log(`period 10 weeks | releases ${RELEASE.length}: ` + ["patch", "minor", "breaking"]
  .map((k) => `${k} ${RELEASE.filter((y) => y[3] === k).length}`).join(", ") +
  ` | definitions ${total} | applies ${Object.values(DEF).reduce((a, b) => a + b.h.length, 0)}`);
const mismatched = Object.keys(DEF).filter((t) => PINNED.pin[t] !== PINNED.last[t]).length;   // pinned version is not what is running
for (const [label, r, y] of [["unpinned ", UNPINNED, total], ["pinned   ", PINNED, mismatched]])
  console.log(`${label} | output changes ${r.changed} | unexpected changes ${r.unexpectedCount}` +
    ` | broken definitions ${r.brokenDefs.size} | fix lines ${r.brokenLines} | distinct versions run ` +
    `${Object.values(r.ran).reduce((a, b) => a + b.size, 0)} | definitions the record does not say what is running for ${y}/${total}`);

for (const [h, t, e, s, d] of UNPINNED.unexpected)     // outputs that changed with the definition untouched
  console.log(`  unexpected   h${String(h).padEnd(2)} ${t.padEnd(14)} ${e} -> ${s}  added: ${d}`);
for (const [h, m, s, kind] of RELEASE) {               // how many users a release reached
  const u = Object.keys(DEF).filter((t) => DEF[t].m === m && UNPINNED.ran[t].has(s));
  const b = Object.keys(DEF).filter((t) => DEF[t].m === m && PINNED.ran[t].has(s));
  console.log(`  h${String(h).padEnd(2)} ${m.padEnd(17)} ${s} ${kind.padEnd(9)} definitions using it ${Object.keys(DEF)
    .filter((t) => DEF[t].m === m).length} | reached: unpinned ${u.length}, pinned ${b.length}`);
}

let behind = 0, missedPatches = 0, stale = 0;          // the cost of pinning: at the end of the period
for (const t in DEF) {
  const l = versionsOf(DEF[t].m), after = l.slice(l.findIndex((y) => y[2] === PINNED.pin[t]) + 1);
  behind += after.length; missedPatches += after.filter((y) => y[3] === "patch").length;
  if (after.length) stale++;
  console.log(`  ${t.padEnd(18)} pinned ${PINNED.pin[t]}  ran ${PINNED.last[t]}  newest ${l.at(-1)[2]}` +
    `  behind ${after.length}  missed patches ${after.filter((y) => y[3] === "patch").length}`);
}
const used = new Set(Object.keys(DEF).map((t) => `${DEF[t].m} ${PINNED.pin[t]}`));
const outsideLines = [...used].reduce((a, k) => {
  const [m, s] = k.split(" "), v = resolve(MODULE[m][s]);
  return a + v.req.length + Object.keys(v.vars).length + v.body.length;
}, 0);
console.log(`definitions left on an old version ${stale}/${total} | total behind ${behind} versions | missed patches ${missedPatches}` +
  ` | definitions in the repo ${Object.values(DEF).reduce((a, d) => a + 3 + Object.keys(d.g).length, 0)} lines` +
  ` | depends on ${used.size} module versions, ${outsideLines} lines outside the repo`);
```

```
period 10 weeks | releases 9: patch 4, minor 2, breaking 3 | definitions 9 | applies 18
unpinned  | output changes 9 | unexpected changes 3 | broken definitions 9 | fix lines 9 | distinct versions run 18 | definitions the record does not say what is running for 9/9
pinned    | output changes 1 | unexpected changes 0 | broken definitions 0 | fix lines 0 | distinct versions run 10 | definitions the record does not say what is running for 1/9
  unexpected   h5  east-point     1.0.0 -> 1.1.1  added: direction=inbound monitoring status=on
  unexpected   h9  north-point    2.0.0 -> 2.0.1  added: type=collector
  unexpected   h10 field-inbound  2.0.0 -> 2.0.1  added: logging=on
  h1  measurement-queue 1.0.1 patch     definitions using it 3 | reached: unpinned 2, pinned 0
  h2  collection-point  1.1.0 minor     definitions using it 4 | reached: unpinned 2, pinned 0
  h3  field-network     2.0.0 breaking  definitions using it 2 | reached: unpinned 2, pinned 0
  h4  collection-point  1.1.1 patch     definitions using it 4 | reached: unpinned 1, pinned 1
  h5  measurement-queue 1.1.0 minor     definitions using it 3 | reached: unpinned 1, pinned 0
  h6  collection-point  2.0.0 breaking  definitions using it 4 | reached: unpinned 2, pinned 0
  h7  measurement-queue 2.0.0 breaking  definitions using it 3 | reached: unpinned 3, pinned 0
  h8  collection-point  2.0.1 patch     definitions using it 4 | reached: unpinned 3, pinned 0
  h9  field-network     2.0.1 patch     definitions using it 2 | reached: unpinned 1, pinned 0
  north-point        pinned 1.0.0  ran 1.0.0  newest 2.0.1  behind 4  missed patches 2
  west-point         pinned 1.1.1  ran 1.1.1  newest 2.0.1  behind 2  missed patches 1
  east-point         pinned 1.0.0  ran 1.0.0  newest 2.0.1  behind 4  missed patches 2
  south-point        pinned 1.0.0  ran 1.0.0  newest 2.0.1  behind 4  missed patches 2
  north-queue        pinned 1.1.0  ran 1.0.0  newest 2.0.0  behind 1  missed patches 0
  west-queue         pinned 1.0.0  ran 1.0.0  newest 2.0.0  behind 3  missed patches 1
  central-queue      pinned 1.0.0  ran 1.0.0  newest 2.0.0  behind 3  missed patches 1
  field-inbound      pinned 1.0.0  ran 1.0.0  newest 2.0.1  behind 2  missed patches 1
  lab-inbound        pinned 1.0.0  ran 1.0.0  newest 2.0.1  behind 2  missed patches 1
definitions left on an old version 9/9 | total behind 25 versions | missed patches 11 | definitions in the repo 39 lines | depends on 5 module versions, 20 lines outside the repo
```

All eighteen of the eighteen applies ran in both setups; the difference is in what ran. In the
unpinned setup, output changed nine times; in the pinned setup, once. That one time is also a
manual upgrade; no output changed without someone wanting it.

**Three of the nine changes are unexpected changes**: the definition text was never touched in
that apply, yet the produced resource came out different from the previous one. The remaining six
came right after breaking releases, and the definition had already been fixed by that point. The
breakdown of the three unexpected ones shows what they have in common: `east-point` jumped two
versions at once in the fifth week and gained a network field plus a monitoring line, `north-point`
picked up `type=collector` on its machine cluster in the ninth week, `field-inbound` picked up
`logging=on` on its network rule in the tenth week. All three are the result of a release that
shows up nowhere in the record, all three applied silently, and **none of them raised an error**.

The spread of breaking releases is the second number. Three breaking releases broke **nine of nine
definitions** in the unpinned setup and demanded nine lines of fixing. Three releases, nine fixes:
the cost of one breaking release is paid once on the publishing side, and once per user on the
consuming side. In the pinned setup, the same three releases broke no definition and demanded
zero lines of fixing — but that does not mean the cost was erased, it means it was deferred.

The per-release breakdown gives how many users each release reached. None of the nine releases
reached every definition using that module: the widest reach is three of three, the narrowest is
one of four. Who a release reaches is decided not by the publishing side, but by **the consuming
side's apply calendar**. In the pinned setup, eight of the same nine releases reached no user at
all; the one release that did reach someone is the patch the manually upgraded definition pulled.
In the unpinned setup, nine definitions ran eighteen distinct versions over the period; in the
pinned setup, ten: two per definition against 1.1.

#### The Cost of Pinning

At the end of the period, the table flips. In the pinned setup, **nine of nine definitions are on
an old version**; they are twenty-five versions behind in total, and **eleven patches** never
reached them at all. A patch is a bug fix; pinning does not only stop unwanted change, it stops
wanted fixes too.

The distribution shows how the deferral accumulates. Five of the seven definitions that were never
upgraded are three to four versions behind, two are two versions behind; `west-point`, upgraded once
during the period, is two versions behind and missing one patch. An upgrade pulls a definition forward,
but because the release flow does not stop, the distance reopens right away. The cost of pinning is not a
one-time delay, it is **a gap that keeps growing**.

The interest on that deferral is countable too: the nine lines of fixing not paid today are not
erased; if those nine definitions are ever upgraded, they will meet all three breaking releases at
once.

#### What the Record Does Not Say

The third number concerns the record itself. In the unpinned setup, the definition writes no
version at all; **in nine of nine definitions, the record does not describe what is running.**
Reading the definition file cannot tell you what resource will be produced, because part of the
text lives somewhere else, and that place can change.

Pinning brings this down to one, but not to zero. The `north-queue` row shows the gap: pinned
version 1.1.0, running version 1.0.0. The version was upgraded in the ninth week, but that
definition had no other apply during the period. **The record says the new version, the field
still holds the old one.** Pinning does not eliminate the gap between record and reality; it moves
the gap from release time to **apply time**.

The last line gives the size of the dependency. The repository's nine definitions are thirty-nine
lines, and they depend on five separate module versions, **twenty lines of text outside the repo**
in total. In the pinned setup, which version of those twenty lines is in effect is written in the
definition; in the unpinned setup it is not written anywhere, and it can change shape without a
single record in the repository changing.

#### Summary

- The same 18 applies ran in both setups: output changed 9 times in the unpinned setup, 1 time in
  the pinned setup; that 1 change is also a manual upgrade.
- 3 of the 9 changes are **unexpected changes** — the definition text was untouched, the output
  changed, and none of them raised an error.
- 3 breaking releases broke 9 definitions in the unpinned setup and demanded 9 lines of fixing;
  in the pinned setup, 0 definitions broke, but the cost was not erased, it was deferred.
- The cost of pinning is seen at the end of the period: 9 of 9 definitions are on an old version,
  25 versions behind in total, and 11 patches never arrived.
- In the unpinned setup, 9 of 9 definitions have a record that does not say what is running; in
  the pinned setup, 1 does not — because the written version was never applied, record and
  reality diverged again.

#### Next Step

Text coming from a module registry now behaves like part of the definition: it takes inputs,
produces resources, gives outputs. In this lesson, all of those inputs were harmless — a network
block, a port number, a queue name. What would happen if one of them were a data store password?
A value passes from definition to module, from module to resource, from resource to the state
file; along the way it also passes through the plan output, the log, and the pipeline record. The
next lesson sends a fictional sensitive value down that path and counts, with a trace scanner
written in `node`, how many places it shows up; then that value is stamped **sensitive** and the
scan is run again — counting how many places it is now hidden, and how many places it still
stands.

### Sensitive Values

Source: https://academia.sh/en/courses/infrastructure-as-code/sensitive-values

A fictional password is passed through a definition, and a trace scanner counts how many copies it leaves at seven trace points; then fields are marked sensitive and the scan runs again. The points marking closes and cannot close, the residue left after rotation, and exposure are each measured separately.

The previous lesson counted the inputs of text pulled from the module registry, and all of them
were harmless — a network block, a port number, a queue name. In this lesson, one of the inputs is
a data store password. The value did not change, the path did not change, what got measured
changed — the question asked is no longer "how many lines," it is **"how many places does it show
up."**

A **secret** is the value itself that has to be kept. A **sensitive value** is the mark that value
carries throughout the record; a field stamped "do not display this." The gap between them is what
this lesson measures: the mark is placed on the **field**, not the value, and the moment a value is
copied from one field to another, the mark does not travel with it.

Secret leakage was measured twice in this module's earlier courses: one ran a repo-wide leak scan
and dealt with encrypted secret storage, the other counted values showing up in pipeline records.
Those scans are not repeated here. What is measured is narrower: **the trace a single value leaves
in a single apply.**

The measurement is again on the **regional measurement network**, and the network is **fictional**.
The value itself is fictional too; no real secret, key, or credential is written. No real provider
is connected to; the trace points are a **model** built with `node`.

**SM37.** The sensitive value is a fictional string: `FICTIONAL-password-7f31`.
**SM38.** The value enters from a variable file in the repo, becomes an input to the module, and
is written into two resource fields: directly into the `password` field, and into a connection
string the value is embedded in.
**SM39.** There are seven trace points: the variable file, the plan output, the apply log, the
state file, the module output, the provider error message, and the pipeline record. The pipeline
record carries a copy of the others.
**SM40.** Marking is done on the field; a marked field is masked in the plan, the log, and the
module output.
**SM41.** The state file is unencrypted and unaffected by marking; the provider error message does
not know the masking rule.
**SM42.** Three passes are run: unmarked, `password` only marked, `password` and `connection`
marked. Then the value is rotated, and how many places the old value stays is counted.
**SM43.** Exposure is a **modeled number**: copies × readers × retention days. The place, day, and
reader counts are written inside the lesson.

#### The Value's Path and the Trace Scanner

```js
// trace.mjs — a fictional sensitive value passes through a definition, trace points are produced, a trace scanner runs.
const SECRET = "FICTIONAL-password-7f31";              // fictional string; not a real secret
const NEW = "FICTIONAL-password-b204";                 // fictional value after rotation

const generate = (d, marked) => {                       // marked: marked fields; masked in display
  const fields = { password: d, connection: `measurement:${d}@store.measurement-network:5432` };
  const mask = (a) => (marked.has(a) ? "(sensitive value)" : fields[a]);
  const plan = `+ data-store measurement-store\n    password = ${mask("password")}\n` +
    `+ connection measurement-connection\n    string = ${mask("connection")}`;
  const log = `applied data-store measurement-store password=${mask("password")}\n` +
    `applied connection measurement-connection string=${mask("connection")}`;
  const state = JSON.stringify({ "data-store": { password: fields.password },   // state is unencrypted
    connection: { string: fields.connection } });                               // marking does not affect it
  const output = `output connection-string = ${mask("connection")}`;
  const error = `provider error 400: invalid request, body {"password":"${fields.password}"}`;
  return { values: `password = ${fields.password}`, plan, log, state, output, error,
    pipeline: [plan, log, output].join("\n") };          // the pipeline record carries a copy of the others
};
const scan = (trace, d) => Object.entries(trace).map(([a, m]) => [a, m.split(d).length - 1]);

const POINT = ["values", "plan", "log", "state", "output", "error", "pipeline"];
const PASS = [["unmarked", new Set()], ["password marked", new Set(["password"])],
  ["password+connection", new Set(["password", "connection"])]];
console.log("pass                 " + POINT.map((a) => a.padStart(10)).join("") + "   point  copies");
const results = {};
for (const [label, marked] of PASS) {
  const s = scan(generate(SECRET, marked), SECRET);
  results[label] = s;
  console.log(label.padEnd(21) + s.map(([, n]) => String(n).padStart(10)).join("") +
    `   ${s.filter(([, n]) => n).length}/${POINT.length}` +
    `${String(s.reduce((a, [, n]) => a + n, 0)).padStart(7)}`);
}
const first = results["unmarked"], last = results["password+connection"];
const closed = first.filter(([a, n], i) => n && !last[i][1]).map(([a]) => a);
const open = last.filter(([, n]) => n).map(([a]) => a);
console.log(`after marking both, closed ${closed.length}/${POINT.length} (${closed})` +
  ` | still open ${open.length}/${POINT.length} (${open})`);

const RESIDUE = { "state version -1": generate(SECRET, new Set(["password", "connection"])).state,
  "archived pipeline record": generate(SECRET, new Set()).pipeline, "repo history": generate(SECRET, new Set()).values };
const newRun = scan(generate(NEW, new Set(["password", "connection"])), SECRET);
const res = scan(RESIDUE, SECRET);
console.log(`after the value is rotated, the old value in the new run: ${newRun.reduce((a, [, n]) => a + n, 0)} copies` +
  ` | in the residue: ${res.filter(([, n]) => n).length} places, ${res.reduce((a, [, n]) => a + n, 0)} copies: ` +
  res.map(([a, n]) => `${a} ${n}`).join(", "));
console.log(`the state file carried ${first[3][1]} copies in every pass; marking never reduced it in any pass`);

const LOCATION = {                                      // trace point: [where it lives, retention days, readers]
  values: ["repo", 3650, 24], plan: ["pipeline record", 90, 31], log: ["pipeline record", 90, 31],
  state: ["remote state store", 3650, 6], output: ["other definitions", 3650, 12],
  error: ["support record", 365, 40], pipeline: ["pipeline archive", 90, 31] };
const exposure = (s) => s.reduce((a, [ad, n]) => a + n * LOCATION[ad][1] * LOCATION[ad][2], 0);
console.log("point     where               days  readers  copies(first/last)  exposure(first/last)");
for (const [i, [ad, n]] of first.entries()) {
  const [y, g, o] = LOCATION[ad];
  console.log(`${ad.padEnd(10)}${y.padEnd(20)}${String(g).padStart(4)}${String(o).padStart(8)}` +
    `${`${n}/${last[i][1]}`.padStart(16)}${`${n * g * o}/${last[i][1] * g * o}`.padStart(20)}`);
}
const totalCopies = (s) => s.reduce((a, [, n]) => a + n, 0);
console.log(`copies ${totalCopies(first)} -> ${totalCopies(last)} (${Math.round((1 - totalCopies(last) / totalCopies(first)) * 100)}% down)` +
  ` | exposure ${exposure(first)} -> ${exposure(last)} (${Math.round((1 - exposure(last) / exposure(first)) * 100)}% down)`);

const CONSUMERS = ["field-service", "reporting", "validator"];  // definitions that take in the module's output
const downstream = Object.fromEntries(CONSUMERS.map((t) =>       // each writes to its own state
  [`state:${t}`, JSON.stringify({ [t]: { connection: `measurement:${SECRET}@store.measurement-network:5432` } })]));
const down = scan(downstream, SECRET), downExposure = totalCopies(down) * LOCATION.state[1] * LOCATION.state[2];
console.log(`definitions consuming the output ${CONSUMERS.length}, each its own state file | ` +
  down.map(([ad, n]) => `${ad} ${n}`).join(" ") + ` | open points with split state ` +
  `${open.length}+${down.filter(([, n]) => n).length}=${open.length + down.length}` +
  ` | copies ${totalCopies(last)}+${totalCopies(down)}=${totalCopies(last) + totalCopies(down)}` +
  ` | exposure ${exposure(last)}+${downExposure}=${exposure(last) + downExposure} (down ` +
  `${Math.round((1 - (exposure(last) + downExposure) / exposure(first)) * 100)}% from the unmarked pass)`);
```

```
pass                     values      plan       log     state    output     error  pipeline   point  copies
unmarked                      1         2         2         2         1         1         5   7/7     14
password marked               1         1         1         2         1         1         3   7/7     10
password+connection           1         0         0         2         0         1         0   3/7      4
after marking both, closed 4/7 (plan,log,output,pipeline) | still open 3/7 (values,state,error)
after the value is rotated, the old value in the new run: 0 copies | in the residue: 3 places, 8 copies: state version -1 2, archived pipeline record 5, repo history 1
the state file carried 2 copies in every pass; marking never reduced it in any pass
point     where               days  readers  copies(first/last)  exposure(first/last)
values    repo                3650      24             1/1         87600/87600
plan      pipeline record       90      31             2/0              5580/0
log       pipeline record       90      31             2/0              5580/0
state     remote state store  3650       6             2/2         43800/43800
output    other definitions   3650      12             1/0             43800/0
error     support record       365      40             1/1         14600/14600
pipeline  pipeline archive      90      31             5/0             13950/0
copies 14 -> 4 (71% down) | exposure 214910 -> 146000 (32% down)
definitions consuming the output 3, each its own state file | state:field-service 1 state:reporting 1 state:validator 1 | open points with split state 3+3=6 | copies 4+3=7 | exposure 146000+65700=211700 (down 1% from the unmarked pass)
```

#### Three Passes

In the unmarked pass, the value shows up in **seven of the seven trace points** and leaves
fourteen copies in total. Fourteen copies for a single input in a single apply. Most of that
number comes from duplication: the pipeline record collects the plan, the log, and the output as
they are, so it carries five copies on its own. None of these five copies were born from a new
decision; no one said "write the password into the pipeline record." A record was read, its output
was written somewhere, and that place was also kept. **Most copies are not a decision, they are a
byproduct**, and that is why none of them show up in a review.

The second pass shows how narrow marking really is. The `password` field was marked; the plan and
the log masked that field, and the copy count dropped from fourteen to ten — but **the visible
point count is still seven**. The reason was written in the first section: the value had been
copied from one field into another. The `connection` string is a separate field because it embeds
the password, and it was not marked. Masking recognizes the field, not the value. What marking
misses is not a rule, it is a **copy**.

The third pass marks the second field too, and that is when the real drop comes: copies fall to
four, visible points to three. The four points that close are the plan, the log, the module
output, and the pipeline record. The three points that stay open are the variable file, the state
file, and the provider error message — and they stay open for three separate reasons.

The variable file stays open because marking is a **display** rule; it has no say over where the
value is written. The mark says what will be shown where the record is read, not what is written
in the record itself. The provider error message stays open because the masking rule belongs to
the side that writes the record, and the side that produces the error does not know it; the error
writes the sent request's body back exactly as it was. What these two have in common is that both
are **outside the record's control**: one comes before the record, the other after it. The state
file is unlike either of them; it sits right in the middle of the record, which is why it matters
most.

#### State, Residue and Exposure

The state file carried two copies in all three passes, and marking never reduced it in any pass.
The reason is what state is: state is the record of the applied resources' actual values.
Stamping a field "do not display" does not remove that field's **value**; the value has to stand
somewhere for the next plan to be able to compare record against reality. This course's rule holds
here too: infrastructure is only as manageable as its record, and the record has to hold the
value.

Rotation shows this once more. After the value is replaced with a new one, the old value leaves
**zero copies** in the new run — clean going forward. Not going backward: three places and eight
copies stand in the residue. Two in the old state version, five in the archived pipeline record,
one in the repo's history. Five of the eight copies come from a single archive taken while marking
did not exist yet; a mark placed today does not fix yesterday's record. Rotation invalidates the
value but **does not erase it from the record**; what now needs erasing is not the value, it is
the record itself.

The final table measures the value of marking one more time, weighted this time. Exposure
multiplies the copy count by the retention time of where it sits and the number of people who can
read that place; it is a **modeled number**, not a real measurement. The result is this: the two
markings erased **seventy-one percent of the copies** but only **thirty-two percent of the
exposure**. The gap comes from every closed point being short-lived — the plan, the log, and the
pipeline record are kept ninety days, while two of the three open points are permanent.

The distribution of remaining exposure is also different from what you would expect. The largest
share is not in the state file, it is in the repo's variable file: eighty-seven thousand six
hundred of one hundred forty-six thousand, that is sixty percent. As long as the value is written
in the repo, the twenty-four people who can read it outnumber the six who can read the remote
state store. Marking never reduces this share; what would reduce it is a different decision —
taking the value out of the record and pulling it in from outside at apply time.

#### Multiplying in Split State

The count so far measured a single definition's single state. State was split earlier in this
topic: each team got its own definition, its own state, and its own apply calendar. Split state
behaves differently with a sensitive value.

The module gives an output called `connection-string`, and three downstream definitions take that
output as an input. The output was masked in display — in the third pass, the `output` point
showed zero copies. But masking is display; what goes to the consuming side is the value itself.
All three definitions write the value into their own resource, and the resource lands in their own
state file. The last line counts this: **three new state files, three new copies.**

The total therefore flips. After marking, four copies were left at three points; once split state
is counted in, that becomes seven copies at six points. Exposure is harsher still: one hundred
forty-six thousand climbs by sixty-five thousand seven hundred to two hundred eleven thousand
seven hundred. Compared against the unmarked pass's two hundred fourteen thousand nine hundred
ten, the drop is **one percent**. Two fields were marked, four trace points closed, and total
exposure barely changed at all.

The reason is not state splitting itself, it is the limit of marking. The mark is placed on one
definition's display; a value that crosses a definition boundary leaves the mark behind. A module
output is one boundary, a value one definition hands to another is another boundary, and the same
thing happens at every boundary — the value crosses, the mark does not. Splitting state made
record management easier, but as far as sensitive values go, what it did was **multiply the copy
count by the number of splits**.

#### Summary

- A fictional password shows up in 7 of 7 trace points in a single apply and leaves 14 copies;
  the pipeline record alone carries 5 copies because it duplicates the plan, the log, and the
  output.
- Marking only `password` brought copies from 14 to 10, but the visible point count stayed at 7 —
  the value had been copied into a connection string; marking both fields brought copies to 4,
  points to 3, and closed the plan, the log, the module output, and the pipeline record.
- The 3 points that stay open are the variable file, the state file, and the provider error
  message; the state file carried 2 copies in all three passes, because the record has to hold
  the value to be able to compare.
- After rotation, the old value leaves 0 copies in the new run; 8 copies stand in the residue at
  3 places: the old state version, the archived pipeline record, and the repo history.
- Marking erased 71 percent of the copies but only 32 percent of the exposure; 60 percent of the
  remaining exposure is in the repo's variable file.
- When 3 definitions consuming the module output wrote the value into their own state files, open
  points went from 3 to 6, copies from 4 to 7, and the drop in exposure fell from 32 percent to 1
  percent: the value crosses the definition boundary, its mark does not.

#### Next Step

Across this topic, the record was taken up four times. First the state file and the lock were
built, then state was split, then a repeated definition was extracted into a module and pulled
versioned from a shared registry, and finally the sensitive values passing through it were marked.
At every step, the record's scope, its cost, and its own risk were counted.

One thing was never done: the record was **never tested for correctness before being applied**.
Every record written in this topic was tried by applying it instead of being tested — the plan was
looked at, it was applied, the result was seen. A misspelled field, a resource that violates a
policy, or a change more expensive than expected only surfaced during or after the apply. The next
topic takes up this gap: checks that run on the infrastructure definition before the apply are
written, and this is counted — how many of a definition's defects can be caught by looking at the
text, how many are only visible once applied, and how many false alarms the catching check
produces.

## Validation and Configuration Management

Testing an infrastructure change before it is applied: format and validation checks, policy scanning that catches rule violations early, unit, contract, and end-to-end infrastructure testing, estimating a change's effect on the budget, agent-based and agentless configuration management models, golden server image production, and integrating an approved automated apply flow into a pipeline.

### Format and Validation Checks

Source: https://academia.sh/en/courses/infrastructure-as-code/format-and-validation-checks

Testing the infrastructure record before it is applied: actually running format, syntax, and type/reference checks over configuration definition files, measuring what each check catches, catches alone, and misses across sixteen known defects, counting run cost in files read and comparisons, and naming the defect class static checking cannot see.

The record is written: the infrastructure is defined in a configuration language, split across
files, and the repeated parts are turned into modules. What has been measured so far is the
record's **structure**; what has never been measured is its **correctness**. When these files are
applied, does the desired infrastructure come out, or does the apply stop halfway? If the question
is not asked before applying, the apply itself answers it, and the cost of that answer is a
resource that has already been created.

**Static checking** reads the record before it is applied: it touches no resource, connects to no
provider, and reads only the text on disk. This lesson writes three static checks and runs them
over fictional definition files — format, syntax, and type-and-reference consistency. For each
check, this lesson counts what it **catches and catches alone** in the known defect set, what it
**misses**, and its **run cost**; at the end, the defect class static checking cannot see is
measured against the apply. The through-line is the **regional measurement network**, and it is
fictional.

#### Record and Known Defect Set

**IV1 — the production environment of the measurement network is held in four definition files
across eleven resources: three networks, five machines, one queue, one data store, one object
store (fictional).** **IV2 — the configuration language's format is our own small format**: a
block header at column 0, below it two-space-indented fields, `=` a plain value, `->` a reference.
**IV3 — the known defect set is sixteen defects, hand-marked**; each defect is injected into a
specific line of the clean record, and which defect a finding corresponds to is read from the
line number.

```js
// setup.mjs — writes the measurement network's record, injects known defects into a copy (fictional)
import { mkdirSync, writeFileSync } from "node:fs";

// format: "resource <kind>/<name>:" at column 0, below it two-space-indented field lines;
// "=" a plain value, "->" a reference to another resource, "#" a comment, one blank line between blocks
const block = (kind, name, fields) => `resource ${kind}/${name}:\n` +
  fields.map(([a, y, d]) => `  ${a.padEnd(10)}${y} ${d}`).join("\n") + "\n";
const MACHINE = [["west-1", "west", "medium", 40], ["west-2", "west", "medium", 40],
  ["west-3", "west", "large", 80], ["east-1", "east", "medium", 40], ["east-2", "east", "large", 80]];
const FILE = {
  "backbone.def": ["west", "east", "north"].map((b, i) => block("network", `backbone-${b}`,
    [["region", "=", b], ["block", "=", `10.2${i}.0.0/16`], ["private", "=", "yes"]])).join("\n"),
  "machine.def": MACHINE.map(([n, b, sz, d]) => block("machine", `collector-${n}`,
    [["region", "=", b], ["size", "=", sz], ["disk_gb", "=", d],
      ["network", "->", `network/backbone-${b}`]])).join("\n"),
  "flow.def": block("queue", "measurement-queue", [["region", "=", "west"], ["retention", "=", 72],
    ["network", "->", "network/backbone-west"]]) + "\n" + block("data_store", "measurement-store",
    [["region", "=", "west"], ["size", "=", "large"], ["disk_gb", "=", 500], ["backup", "=", 14],
      ["network", "->", "network/backbone-west"]]),
  "hold.def": block("object_store", "measurement-archive", [["region", "=", "west"],
    ["versioned", "=", "yes"], ["class", "=", "cold"]]),
};

// known defect set: [code, file, line in the clean record, replacement, defect's location]
const DEFECT = [
  ["D01", "backbone.def", 2, "  region    = west  "],
  ["D02", "backbone.def", 9, "  private   = open"],
  ["D03", "machine.def", 3, "   size      = medium"],
  ["D04", "machine.def", 5, "  network   -> object_store/measurement-archive"],
  ["D05", "machine.def", 10, "  disk_gb=40"],
  ["D06", "machine.def", 16, "  disk_gb   = eighty"],
  ["D07", "machine.def", 18, "@delete", 19],
  ["D08", "machine.def", 22, "  disk_gb   = 40\n  disk_gb   = 60"],
  ["D09", "machine.def", 23, "  network   -> network/backbone-south"],
  ["D10", "machine.def", 25, "resource machine/collector-east-1:"],
  ["D11", "backbone.def", 11, "resource network/backbone-north"],
  ["D12", "flow.def", 2, "  region    = west\n  regions   = west,east"],
  ["D13", "flow.def", 3, "\tretention = 72"],
  ["D14", "flow.def", 8, "  size      ="],
  ["D15", "flow.def", 11, "  # network connection added later", 6],
  ["D16", "hold.def", 4, "  class     = cold\n  label measurement archive"],
];

mkdirSync("record", { recursive: true });
mkdirSync("draft", { recursive: true });
const location = [];
for (const [file, content] of Object.entries(FILE)) {
  writeFileSync(`record/${file}`, content);
  const patch = new Map(DEFECT.filter((k) => k[1] === file).map((k) => [k[2], k[3]]));
  const newLines = [], lineMap = {};
  content.split("\n").forEach((s, i) => { // a defect can delete a line or add lines
    const d = patch.get(i + 1);
    if (d === "@delete") return;
    newLines.push(...(d ?? s).split("\n"));
    lineMap[i + 1] = newLines.length; // clean-record line -> its final counterpart in the draft
  });
  writeFileSync(`draft/${file}`, newLines.join("\n"));
  for (const [code, , line, , k] of DEFECT.filter((x) => x[1] === file))
    location.push({ code, file, line: lineMap[k ?? line] });
}
writeFileSync("defects.json", JSON.stringify(location));

const text = Object.values(FILE).join("");
console.log(`record: ${Object.keys(FILE).length} files, ${text.match(/^resource /gm).length} resources, ` +
  `${text.split("\n").length} lines; draft: same record + ${DEFECT.length} known defects\n`);
console.log(FILE["flow.def"].split("\n").slice(0, 4).join("\n"));
```

```
record: 4 files, 11 resources, 59 lines; draft: same record + 16 known defects

resource queue/measurement-queue:
  region    = west
  retention = 72
  network   -> network/backbone-west
```

#### Three Static Checks

**IV4 — the format rules are six in number and team-accepted.** **IV5 — every resource kind's
fields, types, and requirements live in a schema kept alongside the record.** All three produce
findings in the same shape — file, line, rule — but their contexts differ: format looks at a
single line, syntax looks at the line and its block, type/reference wants the whole record,
because a machine's `network` field points to a resource in another file.

```js
// check.mjs — same record, three static checks: format, syntax, type/reference
import { readFileSync, readdirSync } from "node:fs";

const SCHEMA = { // kind -> "field:type" list; "!" required, "@t" a reference to a resource of kind t
  network: "region:text! block:text! private:boolean!",
  machine: "region:text! size:text! disk_gb:number! network:@network!",
  queue: "region:text! retention:number! network:@network!",
  data_store: "region:text! size:text! disk_gb:number! backup:number! network:@network!",
  object_store: "region:text! versioned:boolean! class:text!",
};
const schemaFields = (kind) => Object.fromEntries(SCHEMA[kind].split(" ").map((x) => x.split(":")));
const HEADER = /^resource ([a-z_]+)\/([a-z0-9-]+):$/, FIELD = /^\s+([a-z_]+)\s*(=|->)\s*(.*)$/;
const read = (dir) => readdirSync(dir).filter((d) => d.endsWith(".def"))
  .map((d) => [d, readFileSync(`${dir}/${d}`, "utf8").split("\n")]);
const counts = { format: 0, syntax: 0, type: 0 };

// 1) format: six line-local rules; reads nothing but the single line
const R = [
  ["R1", (s) => s.length && s !== s.trimEnd()],                          // trailing whitespace
  ["R2", (s) => /^ *\t/.test(s)],                                        // tab in the indent
  ["R3", (s) => /^ +\S/.test(s) && !/^ {2}\S/.test(s)],                  // indent is not two spaces
  ["R4", (s) => /=|->/.test(s) && !/^\s*#/.test(s) && !/ (=|->)( |$)/.test(s)],  // no space around separator
  ["R5a", (s) => /^[^\s#]/.test(s) && !s.endsWith(":")],                 // header does not end with a colon
  ["R5b", (s, i, L) => /^[^\s#]/.test(s) && i > 0 && L[i - 1] !== ""],   // no blank line before block
];
function format(files) {
  const b = [];
  for (const [d, L] of files) L.forEach((s, i) => R.forEach(([k, f]) => {
    counts.format += 1;
    if (f(s, i, L)) b.push({ file: d, line: i + 1, rule: k });
  }));
  return b;
}

// 2) syntax: line- and block-local; never looks outside its own file
function syntax(files) {
  const b = [], seen = new Map();
  for (const [d, L] of files) {
    let keys = null;
    L.forEach((s, i) => {
      const E = (k) => b.push({ file: d, line: i + 1, rule: k });
      counts.syntax += 3;
      if (s === "" || /^\s*#/.test(s)) return;
      if (/^\S/.test(s)) {
        const m = HEADER.exec(s);
        keys = null;
        if (!m) return E("P1 header pattern does not match");
        keys = new Set();
        const path = `${m[1]}/${m[2]}`;
        if (seen.has(path)) E(`P4 duplicate resource name (${seen.get(path)})`);
        else seen.set(path, `${d}:${i + 1}`);
        return;
      }
      const m = FIELD.exec(s);
      if (!m) return E("P2 unrecognized line");
      if (m[3].trim() === "") return E("P2 no value");
      if (keys?.has(m[1])) E(`P3 duplicate key (${m[1]})`);
      keys?.add(m[1]);
    });
  }
  return b;
}

// 3) type/reference: wants the whole record, because references cross file boundaries
function typeRef(files) {
  const resources = [], b = [];
  for (const [d, L] of files) {
    let current = null;
    L.forEach((s, i) => {
      const m = /^\S/.test(s) ? HEADER.exec(s) : FIELD.exec(s);
      if (/^\S/.test(s)) current = m ? { kind: m[1], name: m[2], file: d, line: i + 1, field: {} } : null;
      else if (current && m) current.field[m[1]] = { value: m[3].trim(), line: i + 1 };
      else return;
      if (/^\S/.test(s) && current) resources.push(current);
    });
  }
  const literalType = (v) => (["yes", "no"].includes(v) ? "boolean" : /^-?\d+$/.test(v) ? "number" : "text");
  for (const k of resources) {
    const schema = schemaFields(k.kind);
    for (const [a, v] of Object.entries(k.field)) {
      const E = (rule) => b.push({ file: k.file, line: v.line, rule });
      counts.type += 2;
      if (!schema[a]) { E(`C1 not in schema (${a})`); continue; }
      const t = schema[a].replace("!", "");
      if (!t.startsWith("@")) {
        if (v.value === "" || literalType(v.value) !== t) E(`C2 not ${t} ("${v.value}")`);
        continue;
      }
      let h = null;
      for (const x of resources) { counts.type += 1; if (`${x.kind}/${x.name}` === v.value) { h = x; break; } }
      if (!h) E(`C4 target does not exist (${v.value})`);
      else if (h.kind !== t.slice(1)) E(`C5 wrong kind (${h.kind})`);
    }
    for (const [a, t] of Object.entries(schema)) {
      counts.type += 1;
      if (t.endsWith("!") && !k.field[a])
        b.push({ file: k.file, line: k.line, rule: `C3 required field missing (${a})` });
    }
  }
  return b;
}

const draft = read("draft"), clean = read("record");
const LEVEL = [["format", format, 1], ["syntax", syntax, 1], ["type/reference", typeRef, 4]];
const finding = LEVEL.map(([ad, f]) => [ad, f(draft)]);
const totals = [counts.format, counts.syntax, counts.type]; // counters are read only from the draft run
const DEFECT = JSON.parse(readFileSync("defects.json", "utf8"));
const code = (f) => DEFECT.find((x) => x.file === f.file && x.line === f.line)?.code ?? "??";

console.log(`${"check".padEnd(16)}${"code".padEnd(6)}${"location".padEnd(20)}rule`);
for (const [ad, l] of finding) for (const f of l)
  console.log(`${ad.padEnd(16)}${code(f).padEnd(6)}${`${f.file}:${f.line}`.padEnd(20)}${f.rule}`);

const caught = finding.map(([, l]) => new Set(l.map(code)));
console.log(`\n${"check".padEnd(16)}${"findings".padStart(10)}${"caught".padStart(9)}` +
  `${"catches alone".padStart(16)}${"missed".padStart(11)}${"comparisons".padStart(15)}` +
  `${"files read".padStart(16)}`);
finding.forEach(([ad, l], j) => {
  const y = [...caught[j]].filter((k) => caught.every((s, o) => o === j || !s.has(k))).length;
  console.log(`${ad.padEnd(16)}${String(l.length).padStart(10)}${String(caught[j].size).padStart(9)}` +
    `${String(y).padStart(16)}${String(DEFECT.length - caught[j].size).padStart(11)}` +
    `${String(totals[j]).padStart(15)}${String(LEVEL[j][2]).padStart(16)}`);
});
console.log(`\nthe three checks together catch ${DEFECT.length} of the known defects, ` +
  `${new Set(caught.flatMap((s) => [...s])).size}; findings matching no defect ` +
  `${finding.flatMap(([, l]) => l).filter((f) => code(f) === "??").length}; ` +
  `total findings on the clean record ${LEVEL.reduce((s, [, f]) => s + f(clean).length, 0)}`);
```

```
check           code  location            rule
format          D01   backbone.def:2      R1
format          D11   backbone.def:11     R5a
format          D13   flow.def:4          R2
format          D03   machine.def:3       R3
format          D05   machine.def:10      R4
format          D07   machine.def:18      R5b
syntax          D11   backbone.def:11     P1 header pattern does not match
syntax          D14   flow.def:9          P2 no value
syntax          D16   hold.def:5          P2 unrecognized line
syntax          D08   machine.def:22      P3 duplicate key (disk_gb)
syntax          D10   machine.def:25      P4 duplicate resource name (machine.def:18)
type/reference  D02   backbone.def:9      C2 not boolean ("open")
type/reference  D12   flow.def:3          C1 not in schema (regions)
type/reference  D14   flow.def:9          C2 not text ("")
type/reference  D15   flow.def:7          C3 required field missing (network)
type/reference  D04   machine.def:5       C5 wrong kind (object_store)
type/reference  D06   machine.def:16      C2 not number ("eighty")
type/reference  D09   machine.def:23      C4 target does not exist (network/backbone-south)

check             findings   caught   catches alone     missed    comparisons      files read
format                   6        6               5         10            384               1
syntax                   5        5               3         11            192               1
type/reference           7        7               6          9            131               4

the three checks together catch 16 of the known defects, 16; findings matching no defect 0; total findings on the clean record 0
```

#### Coverage, Overlap, and Cost

The three checks together catch all sixteen of the sixteen defects, but none is sufficient alone:
even type/reference, the broadest in coverage, misses nine defects. Overlap happens in two places
— the missing colon in a header (D11) trips both format and syntax, and a value left empty (D14)
trips both syntax and type. The remaining fourteen defects are each one check's own catch; the
"catches alone" column reads 5, 3, and 6.

The comparison counts say the opposite of what intuition suggests: **the check that makes the most
comparisons wants the least context.** Format makes 384 comparisons (64 lines times six rules),
type/reference only 131. The cost is not in the comparisons but in the **scope**: when a single
file changes, the first two checks can read that file and decide; type/reference has to read all
four. What is more, because reference resolution scans the resource list from the start, the type
check's comparisons grow quadratically with the number of resources. Two quiet numbers sit in the
table as well: findings matching no defect, and total findings on the clean record, are both zero.

#### The Class Static Checking Cannot See

The common name for the defects none of the three checks can see is **defects that depend on
provider truth**: the state that decides is not in the record — the region's quota, the machine
sizes available in the region, a name held in the name pool, permission to create that resource
kind. None of it is written in the text.

**IV6 — the provider's quota, region-size catalog, name pool, permissions, and resource-creation
durations are fictional.** The block below applies the clean record — the one static checking
scores zero findings on — to this model; items are processed in the order the record is read, and
a rejected item is skipped.

```js
// apply.mjs — the clean record is applied to a provider model; the provider and durations are fictional
import { readFileSync, readdirSync } from "node:fs";

const PROVIDER = {
  quota: { west: 2, east: 4 },                             // machine ceiling per region
  catalog: { west: ["medium", "large"], east: ["medium"] }, // sizes available in the region
  reservedName: ["measurement-archive"],                    // name is globally unique
  permission: { network: 1, machine: 1, queue: 1, data_store: 0, object_store: 1 },
};
const DURATION = { network: 20, machine: 45, queue: 30, data_store: 180, object_store: 15, reject: 5 };

const resource = [];
for (const d of readdirSync("record")) for (const s of readFileSync(`record/${d}`, "utf8").split("\n")) {
  const h = /^resource ([a-z_]+)\/([a-z0-9-]+):$/.exec(s), a = /^\s+([a-z_]+)\s*=\s*(.+)$/.exec(s);
  if (h) resource.push({ kind: h[1], name: h[2], field: {} });
  else if (a) resource.at(-1).field[a[1]] = a[2].trim();
}

let seconds = 0, created = 0;
const rejected = [], counted = {};
for (const k of resource) {
  let error = null;
  if (!PROVIDER.permission[k.kind]) error = "permission denied";
  else if (PROVIDER.reservedName.includes(k.name)) error = "name reserved";
  else if (k.kind === "machine") {
    counted[k.field.region] = (counted[k.field.region] ?? 0) + 1;
    if (counted[k.field.region] > PROVIDER.quota[k.field.region]) error = "quota exceeded";
    else if (!PROVIDER.catalog[k.field.region].includes(k.field.size)) error = "not in catalog";
  }
  if (error) rejected.push({ name: `${k.kind}/${k.name}`, error, seconds, created });
  else created += 1;
  seconds += error ? DURATION.reject : DURATION[k.kind];
}

console.log(`${resource.length} items applied: ${created} created, ${rejected.length} rejected, ` +
  `${seconds} seconds total\n`);
console.log(`${"rejected item".padEnd(35)}${"reason".padEnd(19)}${"at second".padStart(17)}` +
  `${"created so far".padStart(21)}`);
for (const r of rejected) console.log(`${r.name.padEnd(35)}${r.error.padEnd(19)}` +
  `${String(r.seconds).padStart(17)}${String(r.created).padStart(21)}`);
```

```
11 items applied: 7 created, 4 rejected, 245 seconds total

rejected item                      reason                     at second       created so far
data_store/measurement-store       permission denied                 90                    4
object_store/measurement-archive   name reserved                     95                    4
machine/collector-west-3           quota exceeded                   190                    6
machine/collector-east-2           not in catalog                   240                    7
```

All four defects are in the record that static checking scored as clean; the difference is in time
and cost. The three checks scan the 64 lines with 707 comparisons and deliver their result without
creating a single resource; the first provider-dependent defect shows up at second 90, the last at
second 240, and by those moments four and seven resources have already been created. By the time
the run ends, the infrastructure is half-built: the queue exists, the data store that feeds it does
not. The only way to move this class into static checking is to bring provider state into the
record — a new record, a new source of drift.

#### Summary

- The fictional measurement network's record holds eleven resources across four files; sixteen
  known defects were injected into a copy, and three static checks were run over the real files.
- Format produced 6 findings (5 catches-alone), syntax 5 (3), type/reference 7 (6); together the
  three caught 16 of 16 defects, and the broadest check alone still missed 9.
- Overlap sits in two defects: a missing colon trips both format and syntax, an empty value trips
  both syntax and type.
- The cost is in scope, not comparisons: format makes 384 comparisons, syntax 192,
  type/reference 131, but when a single file changes the first two read 1 file and
  type/reference reads 4.
- The class it cannot see is **defects that depend on provider truth** (quota, region catalog, a
  reserved name, permission); on a record that passed with zero findings, four of them showed up
  during apply (the first at second 90, the last at second 240) and left behind a half-built
  infrastructure with 7 created resources.

#### Next Step

This lesson's checks tested the record's **own rules**. None of them tested a rule like "this
resource must not face the open network"; rules like that come from outside and concern not the
record's correctness but its **acceptability**. The next lesson runs a policy set over the same
record: should a policy run on the **definition**, or on the **plan**? The two scans are compared
on the same set of violations — which one sees computed values, and how many violations an
exemption opens up.

### Policy and Compliance Scanning

Source: https://academia.sh/en/courses/infrastructure-as-code/policy-and-compliance-scanning

Running externally sourced rules over the infrastructure record: the same policy set run once over the definition text and once over the resolved plan, comparing the two scans' caught, missed, and false-alarm counts, how computed values escape the definition scan, the trade-off between earliness and accuracy, and counting the violations an exemption record drops from the report.

The previous lesson tested the record's **correctness**: does the format hold, is the syntax
valid, do references resolve. All of that was the record's own rules. But a flawless record can
still be unacceptable — an archive with encryption turned off, a machine facing the open network,
or a data store with a three-day backup are all flawless as far as syntax is concerned.

**Policy scanning** runs externally sourced rules over the record. The caught/missed/false-alarm
triple was already established when the **fitness function** was measured over architectural
rules; since its mechanics were measured there, they are not repeated here — what changes is the
object being measured. This lesson's question is: should a policy run on the **definition**, or on
the **plan**? The definition is early and never connects to a provider, but it sees only what is
written; the plan is late and needs a provider call, but it sees the resolved value. The two scans
run the same policy set over the same record, and four things are counted: caught, missed, false
alarms, and the opening an exemption creates. The through-line is the **regional measurement
network**, and it is fictional.

#### Written Value vs. Resolved Value

Not every value written in a record reads the way it is written. Text and reality split apart in
four places: a **variable** (the value is defined on another line), a **provider default** (the
field is left out, and the provider supplies the value), **set-based iteration** (a single block
expands into several instances), and a **condition** (the block is never created in a given
environment). The question of which layer a policy should run on exists precisely because of
these four.

**IV7 — the measurement network's production environment is held in a single record file and
contains all four (fictional).** **IV8 — the provider defaults are fictional**: if a machine's
public-network field is left out, `yes` comes back; if an object store's encryption field is left
out, `no` comes back. **IV9 — values assigned at apply time stand in the record as `@pending` and
are not resolved in the plan either.**

```js
// record.mjs — writes the measurement network's production-environment record; contains variables,
// provider defaults, count-based iteration, and conditions (fictional)
import { writeFileSync } from "node:fs";

const RECORD = `variable environment = production
variable owner = measurement-team
variable backup = 3

resource network/backbone-west:
  region = west
  block  = 10.20.0.0/16
  owner = $owner

resource network/backbone-external:
  region = west
  block  = 198.51.100.0/24
  owner = $owner

resource machine/collector:
  iterate      = west,east,north
  region       = $iterate
  public_ip    = yes
  public_address = @pending
  owner        = $owner

resource machine/gateway:
  region       = west
  public_ip    = yes
  public_address = @pending
  owner        = $owner
  exemption    = P2:2026-03-31

resource machine/archiver:
  region       = west
  public_ip    = @default
  public_address = @pending
  owner        = $owner

resource machine/trial:
  condition    = environment=staging
  region       = west
  public_ip    = yes
  public_address = @pending
  owner        = $owner

resource data_store/measurement-store:
  region    = west
  backup    = $backup
  owner     = $owner
  exemption = *:2026-06-30

resource queue/measurement-queue:
  region   = west
  retention = 720

resource queue/trial-queue:
  condition = environment=staging
  region   = west
  retention = 720
  owner    = $owner

resource object_store/measurement-archive:
  region    = west
  encrypted = @default
  owner     = $owner
  exemption = P1:2025-11-30
`;

writeFileSync("record.def", RECORD);
const count = (k) => (RECORD.match(k) ?? []).length;
console.log(`record.def: ${RECORD.split("\n").length - 1} lines, ${count(/^variable /gm)} variables, ` +
  `${count(/^resource /gm)} blocks\n`);
console.log(`values to resolve: ${count(/= \$/gm)} variable references, ` +
  `${count(/@default/g)} provider defaults, ${count(/@pending/g)} assigned at apply; ` +
  `${count(/^  iterate /gm)} block using iteration, ${count(/^  condition /gm)} conditional blocks, ` +
  `${count(/^  exemption /gm)} exemption records`);
```

```
record.def: 62 lines, 3 variables, 10 blocks

values to resolve: 11 variable references, 2 provider defaults, 4 assigned at apply; 1 block using iteration, 2 conditional blocks, 3 exemption records
```

Ten blocks are written but ten blocks do not enter the plan: one expands into three instances
through iteration, two never enter because their condition does not hold. At the definition layer,
seventeen values still sit unresolved.

#### Two Scans

**IV10 — the policy set is seven rules and is team-accepted (fictional).** P1: an object store
must be encrypted, P2: a machine must not face the open network, P3: a data store's backup must be
at least seven days, P4: a queue's retention must be at most 168 hours, P5: every resource's owner
must be written, P6: a network's block must be in a private range, P7: a machine's public address
must be in a registered range. **IV11 — an exemption record is tied to a policy code (or the `*`
wildcard) and an expiry date; today is taken to be 2026-02-15 (fictional).** **IV12 — plan
generation makes one provider call per kind; a call is four seconds, plan computation is one
second per instance (fictional).**

Every policy declares which field it reads. If a scan cannot resolve that field's value — the
value starts with `$` or `@` — the rule stays **undecided** and counts as neither a violation nor
compliant. The plan scan sees reality; the definition scan is compared against it.

```js
// scan.mjs — the same policy set runs first on the definition, then on the plan (model)
import { readFileSync } from "node:fs";

const TODAY = "2026-02-15";
const DEFAULT = { "machine.public_ip": "yes", "object_store.encrypted": "no" }; // fictional
// [code, rule, field the rule reads (null: does not look at a value)]
const POLICY = [
  ["P1", (t, a) => t !== "object_store" || a.encrypted === "yes", "encrypted"],
  ["P2", (t, a) => t !== "machine" || a.public_ip === "no", "public_ip"],
  ["P3", (t, a) => t !== "data_store" || Number(a.backup) >= 7, "backup"],
  ["P4", (t, a) => t !== "queue" || Number(a.retention) <= 168, "retention"],
  ["P5", (t, a) => a.owner !== undefined, null],
  ["P6", (t, a) => t !== "network" || a.block.startsWith("10."), "block"],
  ["P7", (t, a) => t !== "machine" || a.public_address.startsWith("10."), "public_address"],
];

const VARIABLE = {}, BLOCK = [];
let current = null;
for (const s of readFileSync("record.def", "utf8").split("\n")) {
  const d = /^variable (\w+) = (.+)$/.exec(s), h = /^resource ([a-z_]+)\/([a-z0-9-]+):$/.exec(s);
  const f = /^\s+(\w+)\s*=\s*(.+)$/.exec(s);
  if (d) VARIABLE[d[1]] = d[2];
  else if (h) BLOCK.push((current = { kind: h[1], name: h[2], field: {} }));
  else if (f && current) current.field[f[1]] = f[2].trim();
}

// plan: the condition is evaluated, iteration is expanded, variables and provider defaults are
// resolved; "@pending" is not resolved, because its value is assigned at apply time
const PLAN = [];
for (const b of BLOCK) {
  const [k, v] = (b.field.condition ?? "=").split("=");
  if (b.field.condition && VARIABLE[k] !== v) continue;
  for (const item of b.field.iterate ? b.field.iterate.split(",") : [null]) {
    const field = {};
    for (const [fieldName, fieldValue] of Object.entries(b.field)) {
      if (["iterate", "condition", "exemption"].includes(fieldName)) continue;
      field[fieldName] = fieldValue === "@default" ? DEFAULT[`${b.kind}.${fieldName}`]
        : fieldValue === "$iterate" ? item
          : fieldValue.startsWith("$") ? VARIABLE[fieldValue.slice(1)] : fieldValue;
    }
    PLAN.push({ kind: b.kind, block: b.name, name: item ? `${b.name}-${item}` : b.name, field,
      exemption: b.field.exemption });
  }
}

const unresolved = (v) => v !== undefined && /^[$@]/.test(v);
const isExempt = (n, code) => {
  const [p, date] = (n.exemption ?? ":").split(":");
  return (p === "*" || p === code) && date >= TODAY;
};
function scan(items) {
  const finding = [];
  let undecided = 0;
  for (const n of items) for (const [code, rule, field] of POLICY) {
    if (field && unresolved(n.field[field])) { undecided += 1; continue; }
    if (!rule(n.kind, n.field)) finding.push({ name: n.name, block: n.block, code, exempt: isExempt(n, code) });
  }
  return { finding, undecided };
}

const definitionObjects = BLOCK.map((b) => ({ kind: b.kind, name: b.name, block: b.name, field: b.field,
  exemption: b.field.exemption }));
const defScan = scan(definitionObjects), planScan = scan(PLAN);
const sameFinding = (x, y) => x.block === y.block && x.code === y.code;
const caught = planScan.finding.filter((p) => defScan.finding.some((t) => sameFinding(t, p)));
const falseAlarm = defScan.finding.filter((t) => !planScan.finding.some((p) => sameFinding(t, p)));
const DURATION = new Set(BLOCK.map((b) => b.kind)).size * 4 + PLAN.length; // IV12: fictional seconds

console.log(`${"scan".padEnd(12)}${"value seen".padEnd(15)}${"findings".padStart(10)}` +
  `${"caught".padStart(11)}${"missed".padStart(8)}${"false alarms".padStart(14)}` +
  `${"exempt".padStart(8)}${"reported".padStart(12)}${"undecided".padStart(11)}${"ready".padStart(8)}`);
for (const [name, value, S, yak, yan, dur] of [
  ["definition", "written", defScan, caught.length, falseAlarm.length, 0],
  ["plan", "resolved", planScan, planScan.finding.length, 0, DURATION]]) {
  const exempt = S.finding.filter((b) => b.exempt).length;
  console.log(`${name.padEnd(12)}${value.padEnd(15)}${String(S.finding.length).padStart(10)}` +
    `${`${yak}/${planScan.finding.length}`.padStart(11)}${String(planScan.finding.length - yak).padStart(8)}` +
    `${String(yan).padStart(14)}${String(exempt).padStart(8)}` +
    `${String(S.finding.length - exempt).padStart(12)}${String(S.undecided).padStart(11)}` +
    `${`${dur} s`.padStart(8)}`);
}

console.log(`\nviolations only the plan sees:`);
for (const p of planScan.finding.filter((p) => !defScan.finding.some((t) => sameFinding(t, p)))) {
  const field = POLICY.find(([k]) => k === p.code)[2];
  console.log(`  ${p.code}  ${p.name.padEnd(21)}${field.padEnd(12)}definition has ` +
    `"${BLOCK.find((b) => b.name === p.block).field[field]}", plan has "${PLAN.find((n) => n.name === p.name).field[field]}"`);
}

const EXEMPTION = BLOCK.filter((b) => b.field.exemption).map((b) => ({ name: `${b.kind}/${b.name}`,
  record: b.field.exemption, inEffect: b.field.exemption.split(":")[1] >= TODAY }));
console.log(`\n${"exemption".padEnd(35)}${"record".padEnd(16)}${"status".padEnd(14)}` +
  `${"policies closed".padStart(19)}${"violations dropped".padStart(21)}`);
for (const e of EXEMPTION) {
  const dropped = planScan.finding.filter((b) => b.exempt && `${PLAN.find((n) => n.name === b.name).kind}/${b.block}` === e.name);
  const coverage = e.inEffect ? (e.record.startsWith("*") ? POLICY.length : 1) : 0;
  console.log(`${e.name.padEnd(35)}${e.record.padEnd(16)}` +
    `${(e.inEffect ? "in effect" : "expired").padEnd(14)}` +
    `${String(coverage).padStart(19)}${String(dropped.length).padStart(21)}`);
}
console.log(`\ndefinition scan's false alarms: ` +
  falseAlarm.map((y) => `${y.code} ${y.name}`).join(", ") + ` (condition is false, never enters the plan); ` +
  `the policy neither scan can decide is P7 — public_address is assigned at apply time`);
```

```
scan        value seen       findings     caught  missed  false alarms  exempt    reported  undecided   ready
definition  written                 7       7/10       3             2       1           6          7     0 s
plan        resolved               10      10/10       0             0       2           8          5    30 s

violations only the plan sees:
  P2  archiver             public_ip   definition has "@default", plan has "yes"
  P3  measurement-store    backup      definition has "$backup", plan has "3"
  P1  measurement-archive  encrypted   definition has "@default", plan has "no"

exemption                          record          status            policies closed   violations dropped
machine/gateway                    P2:2026-03-31   in effect                       1                    1
data_store/measurement-store       *:2026-06-30    in effect                       7                    1
object_store/measurement-archive   P1:2025-11-30   expired                         0                    0

definition scan's false alarms: P2 trial, P4 trial-queue (condition is false, never enters the plan); the policy neither scan can decide is P7 — public_address is assigned at apply time
```

#### Who Sees What

The plan scan finds ten violations; the definition scan catches seven of them, misses three, and
adds two false alarms on top. What the missed three have in common is spelled out one by one in
the second part of the output: all three are **computed values**. The archiver machine's
public-network field reads `@default` in the definition and becomes `yes` in the plan; the data
store's backup reads `$backup` in the definition and becomes `3` in the plan; the archive's
encryption field reads `@default` in the definition and becomes `no` in the plan. In all three, the
record text does not **hide** the violation — it only fails to show it; the value lives elsewhere
in the record, or at the provider.

The reason for the two false alarms is the other end of the same asymmetry: two blocks whose
condition does not hold never enter the plan at all, but they still sit in the definition text with
their violating values. The definition scan reports them, and two of the six reported findings —
one in three — belong to resources that will never be created.

Running the two scans together is not free either. Four of the definition report's six lines and
six of the plan report's eight lines describe the same four defects; the team sees the same
violation in two separate reports and has to keep a third record just to track which one has been
closed. For a pipeline that has to put only one of them at the gate, the question is not "which one
is more correct" but **which failure mode is more expensive**: a false alarm burns human time and
ends with the rule being waved through, while a missed violation ships to production.

A third difference is hidden in the numbers. The definition scan produces seven findings, but only
five of them are real violations; those five findings cover the plan's seven violations, because
the one block using iteration expands into three instances. The same defect is reported once in
the definition and three times in the plan. The fix is a single line either way; what differs is
the report's length and how the count should be read.

#### Earliness vs. Accuracy

The table's last two columns are this lesson's trade-off. The definition scan is ready in zero
seconds: it reads a single file, connects to no provider, and can run every time the record
changes. The plan scan waits thirty seconds — one provider call per resource kind, five of them,
then computation for ten plan instances — and only runs in an environment that can produce a plan;
it cannot run at all on a contribution pipeline with no such permission.

In practice, the two also have different places to live. The definition scan can run where the
record is written, before the contribution is even merged; a false alarm is cheapest there, because
the person reading the report is the same person who knows the block is conditional. The plan scan,
by contrast, sits at the gate where the plan is generated, and the decision it makes there is
binding.

What it gets in exchange is accuracy: zero missed, zero false alarms. What follows from this is not
"which one is better" but that the two are **different failure modes**. The definition scan
produces false alarms and misses computed values; the plan scan arrives late and depends on a
provider connection. The undecided column shows the shared boundary of both: the definition cannot
decide on seven rules, the plan on five, and the reason neither can decide on P7 is the same — the
public address is assigned at apply time and is not written in either layer.

#### Exemption's Opening

There are three exemption records, and the three show three separate behaviors. The gateway's
exemption is tied to a single policy and is in effect: it drops one violation from the report, and
the gateway's other six policies stay open. The data store's exemption is a wildcard: today it
still drops only one violation, but the number of policies it closes is seven — every violation
later added to that resource passes silently too, until the exemption expires. The archive's
exemption has expired and drops zero violations; encryption being turned off has come back into
the report.

The exemption itself is a record too, and no one owns its upkeep. One of the three records sits
expired; no one has deleted it. An expired exemption is visible — it brings the violation back into
the report and announces itself. The dangerous end is the other one: an exemption that has not
expired and has been forgotten is silent, shortens the report, and leaves the violation in place.
The only thing that audits an exemption record is the date; no rule audits a wildcard record's
scope.

As a number: the plan scan found ten violations, two were dropped by exemption, eight were
reported. The opening an exemption creates is **two violations** wide, and one of those is the
opening the wildcard record creates. An exemption's cost is measured not by the number of
violations it drops but by **the number of policies it closes**: an exemption tied to one policy
opens one gate, a wildcard exemption opens seven at once.

#### Summary

- The same seven-rule policy set was run over two layers of the fictional measurement network's
  record: the written definition text and the resolved plan.
- The plan scan found 10 violations; the definition scan caught 7, missed 3, and produced 2 false
  alarms — 2 of the 6 findings it reported belonged to resources that would never be created.
- All three missed violations are **computed values**: two are provider defaults (`@default` →
  `yes` and `no`), one is a variable (`$backup` → `3`); the definition text does not show them.
- Earliness is traded for accuracy: the definition scan is ready in 0 seconds without connecting to
  a provider, the plan scan takes 30 seconds and needs five provider calls; the value of P7, which
  neither can decide, is assigned at apply time (7 rules undecided in the definition, 5 in the
  plan).
- Two of the three exemption records are in effect and drop 2 of the 10 violations from the report;
  the wildcard exemption alone closes 7 policies at once while the policy-tied exemption closes 1,
  and the expired exemption brings its violation back into the report.

#### Next Step

Both scans read the record — one as it is written, the other as it is resolved. What neither can
see is the resource's **running state**: P7 stayed undecided because the public address is
assigned at apply time, and no rule in the policy set asks "does this machine actually come up,
does the message written to the queue reach the data store." The next lesson asks that question at
three levels: unit validation that looks at the infrastructure definition, contract validation that
tests the agreement between two resources, and end-to-end validation that creates real resources.
Over the same defect set, what each of the three catches, its duration, and its cost are all
counted — and the common gap all three miss together is named.

### Infrastructure Testing

Source: https://academia.sh/en/courses/infrastructure-as-code/infrastructure-testing

Validating an infrastructure change at three levels: running unit, contract, and end-to-end validation over the same set of fifteen defects, measuring what each level catches, catches alone, its duration, the resources it creates, and its cost in fictional units, naming the common gap all three levels miss, and counting the leftover resources and leaked cost left behind by test resources that are never cleaned up.

Policy scanning tested the record's **acceptability** and left behind one rule that stayed
undecided in both layers: a field whose value is assigned at apply time. That indecision is not an
accident, it is a boundary — the scan reads the record, it does not see the infrastructure running.
Does the resource actually come up, does its disk mount, does the message written to the queue
reach the data store: none of that is written in the text.

**Infrastructure testing** asks these questions and carries the same names as the test levels on
the software side — unit, contract, end-to-end — but the alignment is not in the names, it is in
**what gets created**: here the top level opens a real resource, runs a real charge, and leaves a
trace behind it. This lesson runs the three levels over the same defect set and counts three things
for each: the defects it catches, its duration, and its cost. Then two more numbers follow: the
common gap all three levels miss together, and the uncleaned resource testing leaves behind. The
through-line is the **regional measurement network**, and it is fictional.

#### Defect Set and Three Levels

**IV13 — the known defect set is fifteen defects, hand-marked, and each defect is tied to a
layer**: `definition` (shows up in a single resource's definition), `relation` (shows up only when
two resources are compared), `runtime` (shows up only while a resource is running), `change` (shows
up only when applied over infrastructure that already exists). **IV14 — unit validation tests a
single resource's or module's definition without creating a resource; contract validation runs on
the plan and looks only at interface fields.** **IV15 — end-to-end validation creates real
resources but only runs the main flow; a defect that stays on a side path is invisible to it.**

```js
// levels.mjs — the same defect set is run at three validation levels (model)

// [code, what, layer (where it shows up), field, path (main or side, in the end-to-end flow)]
const DEFECT = [
  ["D01", "machine disk is 10 GB, the collector is writing", "definition", "disk_gb", "main"],
  ["D02", "queue retention is 0 hours", "definition", "retention", "side"],
  ["D03", "archive class left as hot", "definition", "class", "side"],
  ["D04", "collector size is small", "definition", "size", "main"],
  ["D05", "owner tag not written", "definition", "owner", "side"],
  ["D06", "machine region does not match the network's region", "relation", "region", "main"],
  ["D07", "queue and data store are in different regions", "relation", "region", "side"],
  ["D08", "module output name is not what the consumer expects", "relation", "output", "main"],
  ["D09", "backup window overlaps the maintenance window", "relation", "window", "side"],
  ["D10", "security rule cuts off collector traffic", "runtime", "rule", "main"],
  ["D11", "disk does not mount, no mount point", "runtime", "mount", "main"],
  ["D12", "identity lacks permission, cannot write to the store", "runtime", "permission", "main"],
  ["D13", "field change recreates the resource, data is lost", "change", "disk_gb", "main"],
  ["D14", "in-place update causes an outage", "change", "size", "main"],
  ["D15", "an out-of-record manual change gets overwritten", "change", "rule", "main"],
].map(([code, what, layer, field, path]) => ({ code, what, layer, field, path }));

// IV16: end-to-end validation creates real resources; count, creation time, and hourly rate are fictional
const RESOURCE = [["network", 2, 20, 0], ["machine", 3, 45, 4], ["queue", 1, 30, 1],
  ["data_store", 1, 180, 12], ["object_store", 1, 15, 1]];
const COUNT = RESOURCE.reduce((s, [, n]) => s + n, 0);
const CREATE = RESOURCE.reduce((s, [, n, y]) => s + n * y, 0);
const HOURLY = RESOURCE.reduce((s, [, n, , u]) => s + n * u, 0);
const END_TO_END_DURATION = CREATE + 60 + CREATE / 2; // creation + flow validation + teardown

// IV17: contract validation looks only at interface fields
const CONTRACT_FIELD = ["region", "output", "disk_gb"];
const LEVEL = [
  ["unit", (k) => k.layer === "definition", 12, 0, 0],
  ["contract", (k) => k.layer === "relation" ||
    (k.layer === "definition" && CONTRACT_FIELD.includes(k.field)), 45, 0, 0],
  ["end-to-end", (k) => k.layer !== "change" && k.path === "main",
    END_TO_END_DURATION, COUNT, (HOURLY * END_TO_END_DURATION) / 3600],
];

const caught = LEVEL.map(([, f]) => DEFECT.filter(f).map((k) => k.code));
console.log(`${"level".padEnd(12)}${"caught".padStart(9)}${"catches alone".padStart(16)}` +
  `${"missed".padStart(11)}${"resources created".padStart(20)}${"duration".padStart(10)}` +
  `${"cost (fictional units)".padStart(25)}`);
LEVEL.forEach(([ad, , dur, resource, cost], j) => {
  const alone = caught[j].filter((k) => caught.every((t, o) => o === j || !t.includes(k)));
  console.log(`${ad.padEnd(12)}${`${caught[j].length}/${DEFECT.length}`.padStart(9)}` +
    `${String(alone.length).padStart(16)}${String(DEFECT.length - caught[j].length).padStart(11)}` +
    `${String(resource).padStart(20)}${`${dur} s`.padStart(10)}${cost.toFixed(2).padStart(25)}`);
});

const all = new Set(caught.flat());
const missed = DEFECT.filter((k) => !all.has(k.code));
console.log(`\nthe three levels together catch ${all.size} of the ${DEFECT.length} defects; ` +
  `total duration ${LEVEL.reduce((s, [, , t]) => s + t, 0)} s, ` +
  `cost ${LEVEL.reduce((s, [, , , , u]) => s + u, 0).toFixed(2)} fictional units`);
console.log(`\n${missed.length} defects all three levels miss (layer: ${missed[0].layer}):`);
for (const k of missed) console.log(`  ${k.code}  ${k.what}`);
console.log(`\ncost per defect at each level: ` + LEVEL.map(([ad, , dur, , cost], j) =>
  `${ad} ${(dur / caught[j].length).toFixed(1)} s/defect` +
  (cost ? ` and ${(cost / caught[j].length).toFixed(2)} units/defect` : "")).join("; "));
```

```
level          caught   catches alone     missed   resources created  duration   cost (fictional units)
unit             5/15               3         10                   0      12 s                     0.00
contract         5/15               2         10                   0      45 s                     0.00
end-to-end       7/15               3          8                   8     660 s                     4.77

the three levels together catch 12 of the 15 defects; total duration 717 s, cost 4.77 fictional units

3 defects all three levels miss (layer: change):
  D13  field change recreates the resource, data is lost
  D14  in-place update causes an outage
  D15  an out-of-record manual change gets overwritten

cost per defect at each level: unit 2.4 s/defect; contract 9.0 s/defect; end-to-end 94.3 s/defect and 0.68 units/defect
```

#### Who Catches What

The three levels together catch twelve of the fifteen defects. Looked at alone, none of them
clears half: unit five, contract five, end-to-end seven. The real information is in the "catches
alone" column, because that is where the cost of dropping a level from the pipeline is written. If
unit validation is removed, three defects go unowned — the queue's zero retention, the archive's
wrong class, the unwritten owner tag. All three sit in plain sight in a single resource's
definition, and all three are on a side path; end-to-end validation creates the real resource but
never travels those paths at all.

Contract validation's two defects-of-its-own are there for the same reason: the queue and the data
store being in different regions, and the backup window overlapping the maintenance window, only
show up when two resources are set side by side, and neither is on the main flow. Contract
validation's value is in the **comparison**, not the run; that is why, without creating a resource,
it sees what an end-to-end test cannot.

Overlap can be counted too. One of the twelve caught defects — the machine's ten-gigabyte disk —
trips all three levels at once: it is written in the definition, it is one of the contract fields,
and it shows itself while running on the main flow too. All three report the same defect, and the
team reads the same thing three times. Three defects trip two levels, the remaining eight are each
a single level's work. It is exactly this distribution that decides a pipeline's gate order.

End-to-end validation's three defects, by contrast, exist only at the runtime layer: a security
rule that cuts traffic, a disk that does not mount, a missing permission. None of these is written
wrong in the record — the record is correct, reality differs. This is the clearest form of this
course's measurement axis: the gap between text and reality can only be measured once reality is
actually run.

#### Duration and Cost

The last three columns are the price of that same coverage. Unit validation finishes in twelve
seconds, creates no resource, and costs nothing. Contract validation takes forty-five seconds — it
waits for plan generation — but still creates nothing. End-to-end validation opens eight real
resources, takes six hundred sixty seconds, and costs 4.77 fictional units; nine-tenths of its
duration is creation and teardown, validation itself is sixty seconds.

Looked at per defect, the difference is sharper still: unit 2.4 seconds/defect, contract 9.0
seconds/defect, end-to-end 94.3 seconds and 0.68 units/defect. End-to-end validation catches a
defect in forty times the time unit validation takes. That does not mean "end-to-end is bad" — no
other level can catch the three defects it catches. What it says is that **each level is cheapest
in its own layer**: testing the same defect at two levels means paying for it a second time at the
expensive one.

The order is also read off this table. Unit validation weeds out five defects in twelve seconds;
end-to-end validation, running after it, never looks for those five again. The cost of the reverse
order can also be computed: if end-to-end ran first, catching a defect that unit validation would
have found in twelve seconds would instead have cost eight real resources created, six hundred
sixty seconds waited, and 4.77 units paid. The same defect's price varies by up to forty times
depending on which gate catches it.

#### The Three Levels' Common Gap

The common name for the three missed defects is written in the output: the **change layer**. All
three levels validate in a clean environment built from scratch. A real change, though, is not
applied to an environment built from scratch — it is applied **on top of infrastructure that
already exists** — and that is exactly where these defects are born: a field changing recreates the
resource and the data inside it is gone; an in-place update produces an outage; an out-of-record
manual change gets overwritten.

Testing all three with a test built in an empty environment is impossible, because what needs
testing is **history itself**: the data a resource already carries, the traffic it is already
taking, the out-of-record change already made to it. What closes this gap is not a test level but
the plan read before applying — it is there that which item will be recreated gets written. A
manual change the record does not cover shows up in neither the test nor the plan; its only
counterpart is measuring drift separately.

#### Uncleaned Resources

The 4.77-unit cost of end-to-end validation in the table assumes the teardown step succeeds on
every run. Deletion is an apply step just like creation, and it can fail too: the pipeline times
out, a resource cannot be deleted because it is tied to another resource, the run gets cut off
halfway.

```js
// leftover.mjs — the leftover resources and leaked cost end-to-end validation leaves behind (model)

// IV18: one end-to-end run per day for thirty days; the teardown step fails on one run in five,
// and on a failed run each resource stays up with even odds (fictional)
const SEED = 20260215;
let state = SEED;
const random = () => (state = (state * 48271) % 2147483647) / 2147483647; // product stays under 2^53

const RESOURCE = [["network", 2, 0], ["machine", 3, 4], ["queue", 1, 1], ["data_store", 1, 12],
  ["object_store", 1, 1]]; // kind, count, fictional hourly cost
const INSTANCE = RESOURCE.flatMap(([t, n, u]) => Array.from({ length: n }, (_, i) => [`${t}-${i + 1}`, u]));
const DAY = 30, FAILURE = 0.2, RUN_COST = 4.77; // 4.77: run cost computed in the previous block

const leftover = [];
let failed = 0, leaked = 0;
for (let g = 1; g <= DAY; g += 1) {
  if (random() < FAILURE) {
    failed += 1;
    for (const [name, u] of INSTANCE) if (random() < 0.5) leftover.push({ name: `${name}#k${g}`, u, day: g });
  }
  leaked += leftover.reduce((s, k) => s + k.u, 0) * 24; // accumulated leftovers run a full day's cost
}

const intended = DAY * RUN_COST;
console.log(`${DAY} days, ${DAY} runs, seed ${SEED}: teardown failed on ${failed} runs, ` +
  `${leftover.length} leftover resources accumulated`);
console.log(`\n${"leftover resource".padEnd(20)}${"day left behind".padStart(17)}` +
  `${"hourly cost".padStart(14)}${"cost through month end".padStart(25)}`);
for (const k of leftover) {
  console.log(`${k.name.padEnd(20)}${String(k.day).padStart(17)}${String(k.u).padStart(14)}` +
    `${String(k.u * 24 * (DAY - k.day + 1)).padStart(25)}`);
}
console.log(`\nintended cost (${DAY} runs x ${RUN_COST}) ${intended.toFixed(2)} units; ` +
  `leaked cost ${leaked.toFixed(2)} units; ratio ${(leaked / intended).toFixed(1)}x`);
const priciest = [...leftover].sort((a, b) => b.u - a.u)[0];
console.log(`hourly cost of leftovers at month end ${leftover.reduce((s, k) => s + k.u, 0)} units; ` +
  `priciest leftover ${priciest.name} (${priciest.u} units/hour); ` +
  `leftover resources visible in the record 0`);
```

```
30 days, 30 runs, seed 20260215: teardown failed on 4 runs, 11 leftover resources accumulated

leftover resource     day left behind   hourly cost   cost through month end
network-1#k3                        3             0                        0
queue-1#k3                          3             1                      672
data_store-1#k3                     3            12                     8064
object_store-1#k3                   3             1                      672
network-2#k12                      12             0                        0
machine-3#k12                      12             4                     1824
data_store-1#k12                   12            12                     5472
machine-1#k24                      24             4                      672
network-1#k29                      29             0                        0
queue-1#k29                        29             1                       48
data_store-1#k29                   29            12                      576

intended cost (30 runs x 4.77) 143.10 units; leaked cost 18000.00 units; ratio 125.8x
hourly cost of leftovers at month end 47 units; priciest leftover data_store-1#k3 (12 units/hour); leftover resources visible in the record 0
```

Teardown failed on four of the thirty runs, and eleven resources stayed up. The intended cost of
the thirty runs was 143 fictional units; the leaked cost was 18,000 units, or **one hundred
twenty-six times as much**. What inflates the ratio this far is not the failure count but the
duration: a run takes eleven minutes, but a leftover resource sits for days before it is noticed.
The third day's data store alone ran up 8,064 units.

There is a second thing to read in the table. Seven of the eleven leftovers were left behind on the
third and twelfth days, and they produce almost all of the leaked cost: the third day's four
leftovers run up 9,408 units, the twelfth day's three run up 7,296, while the twenty-ninth day's
three run up only 624. Teardown failure's cost is not constant, it is multiplied by **how long it
takes to notice**; the way to cut down leftover resources is therefore not to make teardown one
hundred percent reliable, it is to find the leftover early.

The last line ties directly back to this course's measurement axis: **the number of leftover
resources visible in the record is zero.** The record did not create these resources, testing did;
the record's state does not contain them, and no one looking at the record can see them. The
leftover testing leaves behind is the quietest form of infrastructure the record does not cover —
and the only way to find it is to compare the provider against the record.

#### Summary

- The fifteen-defect set was run at three validation levels: unit caught 5, contract 5, end-to-end
  7; together the three caught 12, and none cleared half on its own.
- The counts of defects caught alone are 3, 2, and 3; contract validation catches two relation
  defects without creating a resource, and end-to-end validation catches three runtime defects no
  other level can see.
- Cost grows with the layer: unit 12 s and 0 resources, contract 45 s and 0 resources, end-to-end
  660 s, 8 real resources, and 4.77 fictional units; per defect, 2.4 / 9.0 / 94.3 seconds.
- The three levels' common gap is the **change layer**: all three run in a clean environment built
  from scratch, while a real change is applied on top of infrastructure that already exists
  (recreation and data loss, in-place update outages, an out-of-record manual change getting
  overwritten).
- Across thirty runs, teardown failed 4 times and 11 resources were left behind; against an
  intended 143 units, 18,000 units leaked (126 times as much), and the number of these resources
  visible in the record is zero.

#### Next Step

This lesson's last two numbers look at the same place: end-to-end validation's run cost and the
leaked cost of leftover resources. Both are the **monetary** face of a change, and both were
counted only once things had already happened. Yet the same question can be asked before applying:
if this change is applied, how much does the monthly bill grow? The next lesson produces an
estimate per change from the plan and compares it against what actually happens — which item can be
estimated because it has a fixed unit price, which cannot because it depends on usage, how the
estimation error is distributed, and how many changes a threshold rule of the form "stop if the
estimate exceeds this much" holds up for nothing over a known set of changes.

### Cost Estimation

Source: https://academia.sh/en/courses/infrastructure-as-code/cost-estimation

Counting an infrastructure change's effect on the budget before it is applied: producing a monthly estimate per change from the plan diff, separating fixed-unit-price items from usage-dependent items, where the error comes from and how it is distributed when compared against what actually happens, how overall accuracy hides the deviation in individual changes, and how many changes a threshold rule holds for nothing versus how many costly changes it lets through.

The previous lesson counted a validation run's own cost and the leaked cost of the leftovers it
left behind. Both were counted after the fact — once the money had already been spent. The same
question can be asked before applying too, because the plan is already in hand: which resource
will be added, which will be grown, which will be deleted is written right there. A plan diff is
at the same time a **bill diff**.

Capacity and cost awareness were measured at system scale earlier; the unit here is **the change
itself**: if this contribution is merged, how much does the monthly cost grow? This lesson produces
an estimate per change from the plan, compares it against what actually happens, and counts three
things: which item can be estimated, which cannot, how the error is distributed. Then a threshold
rule — "stop if the estimate exceeds this much" — is run over a known set of changes, and the
rule's two failure modes are both counted. The through-line is the **regional measurement
network**; the price list and the amounts are both fictional.

#### Estimable and Non-Estimable Items

A cost item is one of two kinds. A **fixed-unit-price** item's amount is read directly from the
plan: three machines, 120 gigabytes of disk, 500 gigabytes of storage. For these items, the
estimate is a multiplication and does not change over the month. A **usage-dependent** item's
amount, though, is not in the plan; the plan says the resource will exist but does not say how many
requests it will take, how many gigabytes of data it will send out, how many messages it will
carry. For these items, the estimate rests on the **expected usage** written in the record — that
is, it is an estimate built on top of an estimate.

```js
// estimate.mjs — produces a monthly cost estimate per change from the plan (model)
import { writeFileSync } from "node:fs";

// IV19: the price list is fictional and has nothing to do with any provider's pricing; unit is "fu" (fictional unit)
const FIXED = { "machine-medium": 30, "machine-large": 60, "disk-gb": 0.5, "store-gb": 0.8,
  "archive-gb": 0.1 };                       // monthly fixed unit price per item
// usage-dependent item: [monthly unit price, uncertainty multiplier r] — real amount is between 1/r and r
const USAGE = { "request-million": [4, 4], "egress-gb": [0.9, 4], "message-million": [2, 2],
  "read-million": [3, 2] };

// IV20: twelve changes are read from the plan; each item's amount is the plan's resource
// difference, and for usage-dependent items it is the expected usage written in the record (fictional)
const CHANGE = [
  ["G01", "three collector machines are added", [["machine-medium", 3], ["disk-gb", 120],
    ["egress-gb", 60]]],
  ["G02", "data store is grown by 500 GB", [["store-gb", 500], ["read-million", 20]]],
  ["G03", "archive stream is added", [["archive-gb", 2000], ["read-million", 30]]],
  ["G04", "measurement queue is added", [["message-million", 120]]],
  ["G05", "collector size is made large", [["machine-large", 3], ["machine-medium", -3]]],
  ["G06", "new region: network and two machines", [["machine-medium", 2], ["disk-gb", 80], ["egress-gb", 300]]],
  ["G07", "collector disk goes from 40 to 200 GB", [["disk-gb", 480], ["read-million", 10]]],
  ["G08", "backup window goes from 7 days to 30", [["store-gb", 1400]]],
  ["G09", "external endpoint is opened", [["request-million", 20], ["egress-gb", 800]]],
  ["G10", "archive class is moved to cold", [["archive-gb", -1500]]],
  ["G11", "collector count goes from 3 to 9", [["machine-medium", 6], ["disk-gb", 240],
    ["egress-gb", 120]]],
  ["G12", "collection frequency doubles", [["message-million", 40], ["read-million", 60]]],
];

const EST = CHANGE.map(([code, what, items]) => {
  let fixed = 0;
  const usage = [];
  for (const [item, amount] of items) {
    if (FIXED[item] !== undefined) fixed += FIXED[item] * amount;
    else usage.push([item, USAGE[item][0] * amount, USAGE[item][1]]);
  }
  const u = usage.reduce((s, [, t]) => s + t, 0);
  return { code, what, fixed, usage, estimate: fixed + u, fixedItems: items.length - usage.length };
});
writeFileSync("estimate.json", JSON.stringify(EST));

console.log(`${"code".padEnd(6)}${"change".padEnd(40)}${"fixed items".padStart(13)}` +
  `${"usage items".padStart(16)}${"fixed fu".padStart(10)}${"usage fu".padStart(13)}` +
  `${"estimate fu".padStart(13)}`);
for (const d of EST) {
  const u = d.estimate - d.fixed;
  console.log(`${d.code.padEnd(6)}${d.what.padEnd(40)}${String(d.fixedItems).padStart(13)}` +
    `${String(d.usage.length).padStart(16)}${d.fixed.toFixed(1).padStart(10)}` +
    `${u.toFixed(1).padStart(13)}${d.estimate.toFixed(1).padStart(13)}`);
}
const fixedTotal = EST.reduce((s, d) => s + d.fixed, 0);
const usageTotal = EST.reduce((s, d) => s + (d.estimate - d.fixed), 0);
const itemCount = EST.reduce((s, d) => s + d.fixedItems + d.usage.length, 0);
console.log(`\n${EST.length} changes, ${itemCount} items: ` +
  `${EST.reduce((s, d) => s + d.fixedItems, 0)} fixed-unit-price (estimable), ` +
  `${EST.reduce((s, d) => s + d.usage.length, 0)} usage-dependent (not estimable)`);
console.log(`total estimate ${(fixedTotal + usageTotal).toFixed(1)} fu; ` +
  `fixed share ${fixedTotal.toFixed(1)} fu (${(100 * fixedTotal / (fixedTotal + usageTotal)).toFixed(0)}%), ` +
  `usage share ${usageTotal.toFixed(1)} fu (${(100 * usageTotal / (fixedTotal + usageTotal)).toFixed(0)}%)`);
console.log(`changes carrying a usage-dependent item: ${EST.filter((d) => d.usage.length).length}/${EST.length}`);
```

```
code  change                                    fixed items     usage items  fixed fu     usage fu  estimate fu
G01   three collector machines are added                  2               1     150.0         54.0        204.0
G02   data store is grown by 500 GB                       1               1     400.0         60.0        460.0
G03   archive stream is added                             1               1     200.0         90.0        290.0
G04   measurement queue is added                          0               1       0.0        240.0        240.0
G05   collector size is made large                        2               0      90.0          0.0         90.0
G06   new region: network and two machines                2               1     100.0        270.0        370.0
G07   collector disk goes from 40 to 200 GB               1               1     240.0         30.0        270.0
G08   backup window goes from 7 days to 30                1               0    1120.0          0.0       1120.0
G09   external endpoint is opened                         0               2       0.0        800.0        800.0
G10   archive class is moved to cold                      1               0    -150.0          0.0       -150.0
G11   collector count goes from 3 to 9                    2               1     300.0        108.0        408.0
G12   collection frequency doubles                        0               2       0.0        260.0        260.0

12 changes, 24 items: 13 fixed-unit-price (estimable), 11 usage-dependent (not estimable)
total estimate 4362.0 fu; fixed share 2450.0 fu (56%), usage share 1912.0 fu (44%)
changes carrying a usage-dependent item: 9/12
```

Thirteen of the twenty-four items are fixed-unit-price, eleven are usage-dependent. By item count,
the majority looks estimable, but the split is different by amount: fifty-six percent of the total
is fixed, forty-four percent usage-dependent. More important is the distribution — nine of the
twelve changes carry at least one usage-dependent item. Three changes (G05, G08, G10) consist only
of fixed items, and their estimate is not a calculation but a **certainty**.

#### Comparing Against Actuals

**IV21 — a fixed-unit-price item does not change over the month, its error is exactly zero; a
usage-dependent item's actual amount is between `1/r` and `r` times the expected amount.** **IV22 —
twelve changes are applied within the same month, and the actual amount is read at month end
(fictional).** The uncertainty multiplier varies by item: external requests and egress data are
the most volatile items (`r = 4`), queue messages and store reads are calmer (`r = 2`).

```js
// actual.mjs — the actual cost, the distribution of estimate error, and the threshold rule's result (model)
import { readFileSync } from "node:fs";

// IV21: a fixed-unit-price item does not change over the month, its error is exactly zero; a
// usage-dependent item's actual amount is between 1/r and r times the expected amount (r is the item's multiplier)
const SEED = 20260401;
let state = SEED;
const random = () => (state = (state * 48271) % 2147483647) / 2147483647;

const ACTUAL = JSON.parse(readFileSync("estimate.json", "utf8")).map((d) => {
  const actual = d.fixed + d.usage.reduce((s, [, amount, r]) =>
    s + amount * Math.pow(r, 2 * random() - 1), 0);
  return { ...d, actual, error: actual - d.estimate };
});

console.log(`${"code".padEnd(6)}${"estimate fu".padStart(13)}${"actual fu".padStart(16)}` +
  `${"error fu".padStart(10)}${"error %".padStart(9)}${"source".padStart(20)}`);
for (const d of ACTUAL) {
  const pct = d.estimate === 0 ? 0 : (100 * d.error) / Math.abs(d.estimate);
  console.log(`${d.code.padEnd(6)}${d.estimate.toFixed(1).padStart(13)}${d.actual.toFixed(1).padStart(16)}` +
    `${d.error.toFixed(1).padStart(10)}${pct.toFixed(0).padStart(9)}` +
    `${(d.usage.length ? `${d.usage.length} usage item${d.usage.length === 1 ? "" : "s"}` : "fixed only").padStart(20)}`);
}

const exact = ACTUAL.filter((d) => d.error === 0);
const off = ACTUAL.filter((d) => d.error !== 0);
const ratio = (d) => d.actual / d.estimate;
console.log(`\n${exact.length} changes have exactly zero error (they carry only fixed-unit-price ` +
  `items); ${off.length} changes are off`);
console.log(`deviation ranges from ${Math.min(...off.map(ratio)).toFixed(2)}x to ` +
  `${Math.max(...off.map(ratio)).toFixed(2)}x; ` +
  `total estimate ${ACTUAL.reduce((s, d) => s + d.estimate, 0).toFixed(1)} fu, ` +
  `actual ${ACTUAL.reduce((s, d) => s + d.actual, 0).toFixed(1)} fu ` +
  `(${(100 * (ACTUAL.reduce((s, d) => s + d.error, 0) / ACTUAL.reduce((s, d) => s + d.estimate, 0))).toFixed(0)}% overall)`);

// IV23: the threshold rule looks at the plan's estimate: if the estimate exceeds the threshold, the change is held and goes to manual approval
console.log(`\n${"threshold fu".padStart(14)}${"held".padStart(7)}${"held for nothing".padStart(18)}` +
  `${"costly miss".padStart(14)}${"correctly held".padStart(16)}${"budget to approval".padStart(20)}`);
for (const E of [200, 300, 500, 800]) {
  const held = ACTUAL.filter((d) => d.estimate > E);
  const forNothing = held.filter((d) => d.actual <= E);
  const missed = ACTUAL.filter((d) => d.estimate <= E && d.actual > E);
  console.log(`${String(E).padStart(14)}${String(held.length).padStart(7)}` +
    `${String(forNothing.length).padStart(18)}${String(missed.length).padStart(14)}` +
    `${String(held.length - forNothing.length).padStart(16)}` +
    `${held.reduce((s, d) => s + d.estimate, 0).toFixed(0).padStart(20)}`);
}
for (const E of [200, 300]) {
  const ad = (f) => ACTUAL.filter(f).map((d) => d.code).join(", ") || "none";
  console.log(`\nthreshold ${E} fu: held ${ad((d) => d.estimate > E)}; ` +
    `held for nothing ${ad((d) => d.estimate > E && d.actual <= E)}; ` +
    `costly miss ${ad((d) => d.estimate <= E && d.actual > E)}`);
}
```

```
code    estimate fu       actual fu  error fu  error %              source
G01           204.0           192.3     -11.7       -6        1 usage item
G02           460.0           461.5       1.5        0        1 usage item
G03           290.0           362.5      72.5       25        1 usage item
G04           240.0           121.6    -118.4      -49        1 usage item
G05            90.0            90.0       0.0        0          fixed only
G06           370.0           535.2     165.2       45        1 usage item
G07           270.0           288.3      18.3        7        1 usage item
G08          1120.0          1120.0       0.0        0          fixed only
G09           800.0           958.4     158.4       20       2 usage items
G10          -150.0          -150.0       0.0        0          fixed only
G11           408.0           368.8     -39.2      -10        1 usage item
G12           260.0           316.2      56.2       22       2 usage items

3 changes have exactly zero error (they carry only fixed-unit-price items); 9 changes are off
deviation ranges from 0.51x to 1.45x; total estimate 4362.0 fu, actual 4664.7 fu (7% overall)

  threshold fu   held  held for nothing   costly miss  correctly held  budget to approval
           200     10                 2             0               8                4422
           300      5                 0             2               5                3158
           500      2                 0             1               2                1920
           800      1                 0             1               1                1120

threshold 200 fu: held G01, G02, G03, G04, G06, G07, G08, G09, G11, G12; held for nothing G01, G04; costly miss none

threshold 300 fu: held G02, G06, G08, G09, G11; held for nothing none; costly miss G03, G12
```

The first reading of the error table is misleading. In total, the estimate is 4,362, the actual is
4,664.7 fictional units — a seven percent deviation, which looks acceptable. But that seven percent
comes from individual changes' deviations canceling each other out: G04 comes in about half low
(0.51x), G06 comes in forty-five percent high (1.45x), G03 twenty-five percent high. **Overall
accuracy gives the decision-maker nothing**, because decisions are made per change. Seven percent
is a good number for a budget report; for a gate, the deviation's 0.51x–1.45x range is the number
that matters.

The second reading shows where the error comes from. Three changes have **exactly zero** error, and
that is not luck but a structural outcome: those three carry only fixed-unit-price items. All nine
of the rest are off, and the size of the deviation grows with the item's uncertainty multiplier.
The way to bring down estimate error is not a better estimation tool; it is **knowing the share
that is usage-dependent**. If a change's estimate does not come with the number "what percent of
this amount is usage-dependent," the estimate alone does not say how reliable it is.

The one negative item in the table follows the same rule: moving the archive class to cold is
estimated as a monthly saving of 150 units, and it saves exactly 150 units. In a fixed-unit-price
item, savings are as certain as spend; in a usage-dependent item, a promised reduction carries the
same uncertainty and may not materialize.

#### Threshold Rule

The moment an estimate is tied to a gate, it starts producing decisions: if the estimate exceeds
the threshold, the change is held and goes to manual approval. The rule has two failure modes, and
both are counted in the table. **Held for nothing** is a change whose estimate exceeds the
threshold but whose actual does not; its cost is a waiting contribution and spent approval time.
**Costly miss** is a change whose estimate stays under the threshold but whose actual comes in
above it; its cost is a surprise in the budget.

A two-hundred-unit threshold holds ten of the twelve changes. At this threshold, no costly change
slips through — but two of the held changes (G01 and G04) actually stay under the threshold, and
the budget going to approval climbs to 4,422 units, nearly the entire total. At this point the gate
stops being a gate: everything is held, and approval turns into a ceremony. At a three-hundred-unit
threshold, the held count drops to five, and all five of those actually exceed the threshold — but
this time two changes (G03 and G12) pass because their estimate stays under it, and they actually
come in above it.

At an eight-hundred-unit threshold, only one change is held and one slips through. No threshold
zeroes out both at once, because a threshold looks at a single number, and that number is itself
uncertain. The way to improve the rule is not to choose a better threshold; it is **to change the
number the threshold looks at**: if the threshold looks at the estimate's usage-dependent share
instead of the estimate itself, a change like G08, which carries only fixed items, can pass
automatically despite its high amount, because its estimate is certain.

There is one more limit. These estimates' input is the plan, and the plan does not fully predict
the apply either: an item that never appears in the plan does not enter the estimate either, and an
item that appears in the plan but does not materialize inflates the estimate. All the numbers here
assume the plan is correct; the gap between the record and reality is a second error stacked on
top of the estimate error.

#### Summary

- The estimate for twelve changes was produced from the plan diff: 13 of 24 items are
  fixed-unit-price (estimable), 11 are usage-dependent (not estimable); 56% of the amount is fixed,
  44% usage-dependent, and 9 of 12 changes carry at least one usage-dependent item.
- The 3 changes that carry only fixed items have exactly zero error; the remaining 9 deviate
  between 0.51x and 1.45x.
- The overall deviation is 7%, and that number is deceptive: individual deviations cancel each
  other out, while decisions are made per change.
- The threshold rule's two failure modes work in opposite directions: a 200-unit threshold holds 10
  of 12 changes (2 for nothing, budget to approval 4,422 units) and misses no costly change; at 300
  units, 5 are held and 2 costly changes pass; at 800 units, 1 is held and 1 costly change passes.
- The way to bring down estimate error is not a better tool, it is reporting the estimate's
  **usage-dependent share**; if the threshold also looks at that share, a high-amount change with a
  certain estimate is not held for nothing.

#### Next Step

This topic's lessons so far have checked the record, run it through policy, tested it, and priced
it. All four looked at the same thing: **the resource itself** — machine, disk, network, queue,
store. But when a machine is created, its inside is empty; which package will be installed on it,
which file will sit with which permission, which service will stay open is not written in the
provisioning layer's record. The next lesson takes up the far side of that boundary: the
**configuration management** layer that takes over once the machine is created, the difference
between agent-based and agentless approaches, and where that layer's own record overlaps with the
provisioning record and where it diverges.

### Configuration Management Tools

Source: https://academia.sh/en/courses/infrastructure-as-code/configuration-management-tools

Agent-based and agentless configuration management are measured as kinds: the same thirty-four-item setting set is rolled out to fifty machines under two models; the rollout's rounds and duration, the number of machines that diverge from the record on partial failure, how many minutes drift lives, and the model's own cost — the agent process, the connection surface, authorized identity — are all counted.

The previous lesson priced a change before it was applied: a unit cost was tied to plan items, how
far the estimate deviated was counted, the non-estimable item was named. But what got priced was
the resource itself — how many machines, at what size, how many disks. Once a resource is created,
the **setting that goes into it** is outside that record: which package version is installed, which
file sits with which permission, which service starts at boot. These settings are still entered by
hand.

Provisioning creates the resource; configuration management ties the resource's inside to a record.
There are two models, and both are named as a kind, not a product: in the **agent-based** model, a
process running continuously on the machine pulls the record itself and applies it; in the
**agentless** model, a runner at the center connects to the machines and pushes the setting.
M22/K01 compared manual, scripted, and declarative provisioning; that comparison is not repeated
here. The question here is the **path that carries the record to the machine** itself.

**IV25.** The fleet is the 50 machines running the collectors of the regional measurement network,
the fictional software that gathers water-meter readings. **IV26.** The record carries 34 items;
a machine can change 61 settings. **IV27.** All numbers come from our own generator; the
seeds `20260731`, `20260802`, and `20260803` are visible, and the numbers are an order of
magnitude, not a measurement. **IV28.** The agentless runner connects to 12 machines at a time; a
connection takes 1 minute, applying 4; the agent polls every 30 minutes on its own offset.
**IV29.** At a run, 8 percent of machines are cut off from the network and on 4 percent the agent
process has stopped; the agent tries its own version daily, succeeding 94 percent of the time, and
35 percent of failures kill it. **IV30.** Over 30
days a machine sees 0.6 manual changes on average, the agentless model adds three record-change
runs to the weekly schedule, the agent holds 46 MB of memory, and an item is 790 bytes in the
record.

#### Same Setting Set, Two Paths

The same 34 items are applied to 50 machines under two models. In the agentless model, the fleet is
split into rounds because the number of concurrent connections is capped; in the agent-based model
there are no rounds, each agent pulls independently on its own offset.

```js
// measurement-network/model.mjs — the same setting set rolled out under two models (model; a kind, not a product)
export const generator = (seed) => () =>
  (seed = (seed * 1103515245 + 12345) % 2147483648) / 2147483648;

export const ITEM = 34;    // configuration items in the record
export const WINDOW = 30;  // agent-based model: poll interval (min)
const LANE = 12, APPLY = 4, CONNECT = 1;  // agentless: concurrent lanes, apply, connect (min)

// Fleet: poll offset, current reachability, and agent status come from the generator.
export const fleet = (n, seed = 20260731) =>
  Array.from({ length: n }, ((r) => () => ({
    offset: Math.floor(r() * WINDOW), reachable: r() > 0.08, agentRunning: r() > 0.04,
  }))(generator(seed)));

const percentile = (d, p) => d.slice().sort((a, b) => a - b)[Math.floor((p / 100) * d.length)];

// Agentless: the center connects in rounds; a machine it cannot reach is skipped this run.
// Agent-based: each agent pulls on its own offset; there is no center queue, a stalled agent stays out.
export const run = (m, model) => {
  const agentless = model === "agentless";
  const target = m.filter((x) => (agentless ? x.reachable : x.agentRunning));
  const duration = agentless
    ? target.map((_, i) => (Math.floor(i / LANE) + 1) * (CONNECT + APPLY))
    : target.map((x) => x.offset + APPLY);
  return {
    model, round: agentless ? Math.ceil(target.length / LANE) : 1, reached: target.length,
    diverged: m.length - target.length, p50: percentile(duration, 50), p95: percentile(duration, 95),
    max: Math.max(...duration),
  };
};

if (import.meta.url === `file://${process.argv[1]}`) {
  const m = fleet(50);
  console.log(`50 machines, ${ITEM} items in the record, total in the record: ${50 * ITEM} lines\n`);
  console.log("model".padEnd(13) + "round".padEnd(7) + "p50".padEnd(8) + "p95".padEnd(8) +
    "max".padEnd(8) + "reached".padEnd(9) + "diverged".padEnd(10) + "lines held");
  for (const s of ["agentless", "agent-based"].map((x) => run(m, x)))
    console.log(s.model.padEnd(13) + String(s.round).padEnd(7) + `${s.p50} min`.padEnd(8) +
      `${s.p95} min`.padEnd(8) + `${s.max} min`.padEnd(8) + `${s.reached}/50`.padEnd(9) +
      `${s.diverged}/50`.padEnd(10) + `${s.reached * ITEM}/${m.length * ITEM}`);
  console.log("\nfull rollout as the fleet grows:");
  console.log("machines".padEnd(10) + "agentless rounds".padEnd(18) + "agentless max".padEnd(15) +
    "agent-based rounds".padEnd(20) + "agent-based max");
  for (const n of [50, 200, 800]) {
    const f = fleet(n), a = run(f, "agentless"), b = run(f, "agent-based");
    console.log(String(n).padEnd(10) + String(a.round).padEnd(18) + `${a.max} min`.padEnd(15) +
      String(b.round).padEnd(20) + `${b.max} min`);
  }
}
```

```
50 machines, 34 items in the record, total in the record: 1700 lines

model        round  p50     p95     max     reached  diverged  lines held
agentless    4      10 min  20 min  20 min  47/50    3/50      1598/1700
agent-based  1      21 min  32 min  33 min  45/50    5/50      1530/1700

full rollout as the fleet grows:
machines  agentless rounds  agentless max  agent-based rounds  agent-based max
50        4                 20 min         1                   33 min
200       16                80 min         1                   33 min
800       63                315 min        1                   33 min
```

At 50 machines, the agentless model is faster: it finishes in four rounds, in 20 minutes; the
agent-based model takes 33 minutes, because the last agent to poll sits at the end of the interval.
The second table upsets that ranking. In the agentless model, the round count grows in direct
proportion to the fleet — at 800 machines, 63 rounds, 315 minutes. In the agent-based model,
**33 minutes is constant**: the parallelism lives at the machines, not at the center.

The last column is this course's measure. The record claims 1,700 lines; after one run 1,598 hold
in the agentless model, 1,530 in the agent-based model. The difference comes from **machines that
diverge from the record**: 3 cut off from the network are skipped, 5 with a stopped process never
look. The failure mode differs too. Agentless failure is **seen from the center** — the runner
reports that it could not connect. Agent-based failure is **silent**: an agent that never looks
produces no error, and the center does not notice unless it checks the last report-in.

#### How Long Drift Lives

Rollout is a one-time event. The real difference shows up once a manual change is made on a machine
after applying. The agent-based model watches continuously; the agentless model is triggered, and
drift accumulates in the interval between. This course's `iac-fundamentals` topic already
established the difference between continuous reconciliation and triggered apply; here that
difference is counted **in minutes**.

```js
// measurement-network/drift.mjs — how many minutes drift lives under each model (model)
import { generator, fleet, ITEM, WINDOW } from "./model.mjs";

const DAY = 1440, END = 30 * DAY;
const SETTING = 61;    // settings a machine can change; 34 of these are in the record
const RATE = 0.6;      // manual change per machine over 30 days (fictional)
// Agentless model is triggered: weekly schedule + three record-change runs (fictional days).
const TRIGGER = [0, 3.2, 7, 11.6, 14, 19.4, 21, 28].map((g) => Math.round(g * DAY));

const events = (m, seed = 20260802) => {
  const r = generator(seed);
  return m.flatMap((x) => (r() < RATE
    ? [{ machine: x, at: Math.floor(r() * END), inRecord: r() * SETTING < ITEM }] : []))
    .sort((a, b) => a.at - b.at);
};

// Agent-based: each agent compares against the record once every WINDOW minutes, on its own offset.
const agentBased = (o) => o.machine.agentRunning
  ? ((((o.machine.offset - o.at) % WINDOW) + WINDOW) % WINDOW) || WINDOW : null;

// Agentless: lives until the next trigger; if the machine is unreachable at that moment, one more trigger.
const agentless = (o) => {
  const k = TRIGGER.filter((t) => t > o.at);
  const t = o.machine.reachable ? k[0] : k[1];
  return t === undefined ? null : t - o.at;
};

const summarize = (list, lifespan) => {
  const onRecord = list.filter((o) => o.inRecord).map((o) => [o, lifespan(o)]);
  const closed = onRecord.filter(([, v]) => v !== null).map(([, v]) => v);
  // Drift that stays open lives until the end of the window; it enters the total that way.
  const open = onRecord.filter(([, v]) => v === null);
  const openDuration = open.reduce((t, [o]) => t + (END - o.at), 0);
  const closedDuration = closed.reduce((t, v) => t + v, 0);
  return [onRecord.length, closed.length, open.length, Math.round(closedDuration / closed.length),
    Math.max(...closed), closed.filter((v) => v <= 60).length,
    closedDuration + openDuration, openDuration / (closedDuration + openDuration)];
};

const list = events(fleet(50));
console.log(`30 days, 50 machines, ${list.length} manual changes; ` +
  `the record covers ${ITEM}/${SETTING} settings\n`);
console.log("model".padEnd(13) + "trigger".padEnd(11) + "seen".padEnd(10) +
  "closed".padEnd(10) + "open".padEnd(7) + "avg lifespan".padEnd(14) + "longest".padEnd(11) +
  "within 1h".padEnd(11) + "drift-hours");
for (const [ad, trigger, fn] of [["agent-based", `${DAY / WINDOW}/day`, agentBased],
  ["agentless", `${TRIGGER.length}/30 days`, agentless]]) {
  const [event, closed, open, avg, longest, within, total, share] = summarize(list, fn);
  console.log(ad.padEnd(13) + trigger.padEnd(11) + String(event).padEnd(10) +
    String(closed).padEnd(10) + String(open).padEnd(7) + `${avg} min`.padEnd(14) +
    `${(longest / 60).toFixed(1)} hr`.padEnd(11) + `${within}/${closed}`.padEnd(11) +
    (total / 60).toFixed(0) + ` (${(100 * share).toFixed(1)}% of what stayed open)`);
}
console.log(`\ndrift born in an out-of-record setting: ${list.filter((o) => !o.inRecord).length}` +
  ` events - invisible to both models`);
console.log(`settings left out of the record: ${SETTING - ITEM}/${SETTING} (per machine)`);
```

```
30 days, 50 machines, 27 manual changes; the record covers 34/61 settings

model        trigger    seen      closed    open   avg lifespan  longest    within 1h  drift-hours
agent-based  48/day     18        15        3      20 min        0.5 hr     15/15      1251 (99.6% of what stayed open)
agentless    8/30 days  18        16        2      3873 min      159.5 hr   0/16       1320 (21.7% of what stayed open)

drift born in an out-of-record setting: 9 events - invisible to both models
settings left out of the record: 27/61 (per machine)
```

There are two orders of magnitude of difference per event. In the agent-based model, a closed drift
lives an average of **20 minutes**, and all fifteen close within an hour. In the agentless model
the average lifespan is **3,873 minutes** — more than two and a half days — and none of the sixteen
close within an hour; the longest drift stays up 159.5 hours, 6.6 days. The difference is
the ratio between the poll interval and the trigger interval.

The last column takes this advantage back. Thirty-day total drift duration is 1,251 hours in the
agent-based model, 1,320 in the agentless model — six percent apart. The reason is in the table:
**99.6 percent** of the agent-based total comes from three drifts that never close. On a machine
whose agent has stopped, drift lives without limit because nothing is watching; the model's speed
is defeated by its own failure mode. In the agentless model the share that stays open is 21.7
percent — failure is visible to the center and gets fixed on the next trigger.

The last two lines write down what neither model covers. A machine can change 61 settings, and the
record holds 34 of them; of the 27 changes that occur over thirty days, **9 are in out-of-record
settings** and are invisible to both models. How fast drift closes is a tooling question; which
drift gets seen is a **record coverage** question, and choosing a tool does not change that.

#### The Model's Own Cost

Both models add something to the machine: the agent-based model a process, the agentless model a
connection point. The cost is counted separately.

```js
// measurement-network/cost.mjs — the model's own cost: process, connection surface, identity (model)
import { generator, fleet, ITEM, WINDOW } from "./model.mjs";

const MEMORY = 46e6;   // memory the agent process holds on the machine (fictional)
const BYTES = 790;     // a configuration item's bytes in the record (fictional)
const N = fleet(50).length;
const pulls = (1440 / WINDOW) * N;          // agent-based: daily record pulls
const connections = Math.round((8 / 30) * N); // agentless: daily connections (8 runs in 30 days)
const mb = (b) => (b / 1e6).toFixed(1) + " MB";

const ROW = [
  ["process running on the machine", `${N}`, "0"],
  ["memory held on the machine", `${(N * MEMORY / 1e9).toFixed(2)} GB`, "0.00 GB"],
  ["listening endpoint on the machine", "0", `${N}`],
  ["listening endpoint at the center", "1", "0"],
  ["authorized identity", `${N} (open to a single machine)`, `1 (open to ${N} machines)`],
  ["if one identity falls", "1 machine", `${N} machines`],
  ["daily connections to the center", `${pulls}`, `${connections}`],
  ["daily traffic from the center", mb(pulls * ITEM * BYTES), mb(connections * ITEM * BYTES)],
];

// The agent's own version has to roll out too: one attempt per day, and failure can kill the agent.
const rollout = (seed = 20260803) => {
  const r = generator(seed), d = Array(N).fill("old"), trace = [];
  for (let g = 1; g <= 14; g++) {
    for (let i = 0; i < N; i++)
      if (d[i] === "old") d[i] = r() < 0.94 ? "new" : r() < 0.35 ? "dead" : "old";
    if ([1, 3, 7, 14].includes(g))
      trace.push([g, ...["new", "old", "dead"].map((v) => d.filter((x) => x === v).length)]);
  }
  return trace;
};

console.log("item".padEnd(36) + "agent-based".padEnd(32) + "agentless");
for (const [a, b, c] of ROW) console.log(a.padEnd(36) + b.padEnd(32) + c);
console.log(`\nas the agent's own version rolls out to ${N} machines:`);
console.log("day".padEnd(7) + "new version".padEnd(13) + "old version".padEnd(13) + "dead agent");
const trace = rollout();
for (const [g, newV, oldV, dead] of trace)
  console.log(`${g}.`.padEnd(7) + `${newV}/${N}`.padEnd(13) +
    `${oldV}/${N}`.padEnd(13) + `${dead}/${N}`);
console.log(`\n${trace.at(-1)[3]} machine with a dead agent cannot recover: it can only be reached ` +
  `the agentless way; at 12 versions a year that touches ${trace.at(-1)[3] * 12} machines`);
```

```
item                                agent-based                     agentless
process running on the machine      50                              0
memory held on the machine          2.30 GB                         0.00 GB
listening endpoint on the machine   0                               50
listening endpoint at the center    1                               0
authorized identity                 50 (open to a single machine)   1 (open to 50 machines)
if one identity falls               1 machine                       50 machines
daily connections to the center     2400                            13
daily traffic from the center       64.5 MB                         0.3 MB

as the agent's own version rolls out to 50 machines:
day    new version  old version  dead agent
1.     46/50        3/50         1/50
3.     49/50        0/50         1/50
7.     49/50        0/50         1/50
14.    49/50        0/50         1/50

1 machine with a dead agent cannot recover: it can only be reached the agentless way; at 12 versions a year that touches 12 machines
```

The first two rows are direct cost: in the agent-based model, the fleet holds **2.30 GB of memory**
without taking a single measurement reading. The third and fourth rows write the direction: in the
agent-based model no machine listens, the agent connects outward and a single endpoint listens at
the center; in the agentless model **50 machines** listen for remote access, and the center never
listens at all.

The fifth and sixth rows are the real distinction. In the agent-based model, every machine has its
own identity, and that identity can only write to its own machine; if one falls, one machine falls.
In the agentless model, a single identity is authorized on all fifty machines; if it falls,
**fifty machines** fall at once. The number of identities is fifty times higher in the agent-based
model, but a single identity's cost is fifty times lower.

The last two rows count the load on the center: 2,400 pulls and 64.5 MB a day in the agent-based
model, 13 connections and 0.3 MB in the agentless model. The 200-fold gap between them is the price
of drift closing in 20 minutes — watching continuously means asking continuously.

The table below is the agent-based model's unclosed loop. The agent itself is software too, and its
new version has to roll out as well. Over fourteen days, 49 machines get through, **the agent dies
on 1 machine**, and that machine cannot recover on its own: because the agent is not running, it
cannot pull the new version. At twelve versions a year, 12 machines get touched, and the
agent-based model requires keeping the agentless path around as an escape hatch.

#### Summary

- At fifty machines agentless rolls out in 4 rounds and 20 minutes, agent-based in one round and
  33; at 800 machines the ranking flips — agentless takes 63 rounds and 315 minutes, agent-based
  is still 33.
- After one run, 1,598 of the record's 1,700 lines hold in agentless, 1,530 in agent-based; the
  difference comes from 3 machines cut off from the network and 5 with a stopped agent, and the
  second failure mode is **silent**.
- A closed drift lives 20 minutes on average in agent-based, 3,873 in agentless; the longest
  agentless drift stays up 6.6 days.
- Total drift duration is nonetheless 1,251 hours against 1,320: 99.6 percent of the agent-based
  total comes from three drifts that never close.
- The 9 drifts born in the 27/61 settings the record does not cover are invisible to both; how
  fast drift closes is a tooling question, which drift gets seen is a coverage question.
- The cost: agent-based, 50 processes, 2.30 GB of memory, 2,400 daily pulls and an agent that can
  die; agentless, 50 listening endpoints and one identity authorized on fifty machines at once.

#### Next Step

Both models share the same assumption: a machine boots empty first, then gets configured. The
entire rollout duration comes from that ordering — both 20 minutes and 33 minutes are spent
bringing an already-booted machine up to the record. Partial failure comes from the same place too:
the machine has started taking work before it matches the record.

The ordering can be reversed. Instead of carrying the setting to the machine after boot, it can be
baked into the image the machine boots from; if the machine is **born** matching the record, the
rollout duration is zero. M22/K02 measured the base image's size and update overhead; that is not
repeated. The next lesson counts one layer up: with boot-time configuration and a pre-baked image
side by side, how long does boot take, how many outbound calls does it make, and how many images
get rebaked for a security fix?

### Server Image Building

Source: https://academia.sh/en/courses/infrastructure-as-code/server-image-building

Configuring at every boot is measured side by side with the golden image: boot duration, the number of outbound calls at boot, and how many machines fail to boot when a dependency drops are counted; how many images get rebaked when a security fix lands, how many days the rollout window takes, and how many components the image's content record misses are measured.

The previous lesson measured two ways of carrying the record to a running machine. Both assumed
the same ordering: the machine boots empty first, then gets configured. Both the rollout duration
and the partial failure came from that ordering — the machine had already started taking work
while it did not yet match the record.

The ordering can be reversed: instead of the setting being carried to the machine after boot, it
can be **placed in advance** inside the image the machine boots from. This lesson sets three paths
side by side as kinds — **configure at every boot**, **golden image**, and **hybrid** (a baked base
plus a small setting applied at boot). M22/K02 measured the base image's bytes and its update
overhead; that measurement is not repeated. What is measured here is not the image's size, it is
the **image build flow**.

**IV31.** The fleet is 50 machines running the regional measurement network's — the fictional
software collecting water-meter readings' — collectors, across 6 roles and 3 environments.
**IV32.** The three paths' boot steps and seconds are fictional; each step's dependency and number
of calls is written down. **IV33.** Dependencies' probability of being up at boot time is between
0.995 and 0.999, and a sudden spike in demand boots 40 machines at once. **IV34.** The numbers come
from our own generator; the seeds `20260811`–`20260814` are visible and are an order of magnitude.
**IV35.** An image bakes in 7–19 minutes, plus 6 minutes of testing on top; the image builder bakes
3 images at a time, and the fleet replaces 12 machines a day. **IV36.** The image's content record
writes only direct components, not the indirect ones that come along; the full rebake cycle is 30
days.

#### What Happens at Boot

All three paths boot the same service. The configure-at-every-boot path downloads and installs
packages at boot time; the golden-image path finds them already in the image; the hybrid path
finds the image ready but pulls the environment-varying setting at boot.

```js
// measurement-network/boot.mjs — the same service's three boot paths (model; a kind, not a product)
export const generator = (seed) => () =>
  (seed = (seed * 1103515245 + 12345) % 2147483648) / 2147483648;

// External dependency kinds and each one's probability of being up at boot time (fictional).
export const UP = {
  "name resolver": 0.999, "package repository": 0.995, "configuration server": 0.995,
  "secret store": 0.998, "image store": 0.997,
};

// Step: [name, seconds, dependency or null, number of calls made in that step].
export const PATH = {
  "every boot": [
    ["kernel boot", 12, null, 0], ["name resolution", 2, "name resolver", 1],
    ["package listing", 9, "package repository", 1], ["14 package installs", 128, "package repository", 14],
    ["config pull", 11, "configuration server", 1], ["secret pull", 6, "secret store", 1],
    ["service start", 8, null, 0],
  ],
  "golden image": [
    ["image pull", 19, "image store", 1], ["kernel boot", 12, null, 0],
    ["secret pull", 6, "secret store", 1], ["service start", 8, null, 0],
  ],
  "hybrid": [
    ["image pull", 19, "image store", 1], ["kernel boot", 12, null, 0],
    ["environment setting pull", 7, "configuration server", 1], ["secret pull", 6, "secret store", 1],
    ["service start", 8, null, 0],
  ],
};

export const measure = (ad) => {
  const step = PATH[ad];
  const calls = step.reduce((t, [, , , c]) => t + c, 0);
  const success = step.reduce((t, [, , b, c]) => t * (b ? UP[b] ** c : 1), 1);
  const dependency = [...new Set(step.filter(([, , b]) => b).map(([, , b]) => b))];
  return { ad, duration: step.reduce((t, [, s]) => t + s, 0), calls, success, dependency };
};

if (import.meta.url === `file://${process.argv[1]}`) {
  const NEW_MACHINES = 40;
  console.log("path".padEnd(14) + "boot".padEnd(9) + "ext calls".padEnd(11) +
    "dependency".padEnd(12) + "boot success".padEnd(17) + `not booting at ${NEW_MACHINES} machines`);
  for (const ad of Object.keys(PATH)) {
    const o = measure(ad);
    console.log(o.ad.padEnd(14) + `${o.duration} s`.padEnd(9) + String(o.calls).padEnd(11) +
      `${o.dependency.length} kinds`.padEnd(12) + `${(100 * o.success).toFixed(1)}%`.padEnd(17) +
      (NEW_MACHINES * (1 - o.success)).toFixed(1));
  }
  console.log("\nmachines that fail to boot when one dependency drops (out of 40):");
  const columns = Object.keys(PATH);
  console.log("dropped".padEnd(24) + columns.map((a) => a.padEnd(14)).join("").trimEnd());
  for (const b of Object.keys(UP))
    console.log((b.padEnd(24) + columns.map((ad) =>
      (measure(ad).dependency.includes(b) ? "0/40" : "40/40").padEnd(14)).join("")).trimEnd());
}
```

```
path          boot     ext calls  dependency  boot success     not booting at 40 machines
every boot    176 s    18         4 kinds     92.0%            3.2
golden image  45 s     2          2 kinds     99.5%            0.2
hybrid        52 s     3          3 kinds     99.0%            0.4

machines that fail to boot when one dependency drops (out of 40):
dropped                 every boot    golden image  hybrid
name resolver           0/40          40/40         40/40
package repository      0/40          40/40         40/40
configuration server    0/40          40/40         0/40
secret store            0/40          0/40          0/40
image store             40/40         0/40          0/40
```

Boot duration drops from 176 seconds to 45 seconds — roughly a fourfold cut. But the real
difference is in the second column: **18 outbound calls drop to 2**. This is a quantity independent
of duration; every call is a separate point of failure, and under a demand spike, all of them
happen at once. At a forty-machine boot, the expected number of machines that fail to boot drops
from 3.2 to 0.2; the golden-image path does less to speed up boot than to **make boot independent**.

The second table writes down the limit of this. On the secret-store row, all three paths are zero: a
sensitive value cannot be baked into an image — if it were, anyone who can read the image can read the
secret too. The golden path removes seventeen of eighteen calls; it cannot remove the eighteenth. The
image-store row is the other side of the trade-off: as the golden path cuts four dependencies down to two,
it **picks up a new one**. That is the configure-at-every-boot path's one advantage; it keeps booting even
if the image store drops.

The gap between the record and reality exists here too. In the configure-at-every-boot path, the
record says "these packages will be installed"; what actually gets installed is whatever the
package repository serves **that day**, and two machines booting on two different days do not get
the same content. In the golden path, the record is turned into reality once, and freezes into the
image.

#### How Many Images Get Rebaked When a Fix Lands

The cost of frozen content is that changing it now requires a build step. When a security fix
lands, the configure-at-every-boot path bakes nothing; the golden path rebakes every affected
image. How many images get baked depends on where the environment is written.

```js
// measurement-network/bake.mjs — how many images get rebaked when a security fix lands (model)
import { generator } from "./boot.mjs";

export const ROLE = 6, ENVIRONMENT = 3;    // the measurement network's six roles, three environments (fictional)
const LANE = 3, TEST = 6;                  // concurrent baking lanes; per-image test time (min)
const MACHINE = 50, REPLACE = 12;          // fleet size and daily machine-replacement capacity

// On the pure golden-image path, each role is baked separately for each environment; on the hybrid
// path the environment setting is applied at boot, so one image per role is enough.
export const IMAGE_COUNT = { "every boot": 0, "golden image": ROLE * ENVIRONMENT, "hybrid": ROLE };

// A fix touches the roles that carry the component; a base component is in every role.
const fixes = (seed = 20260811) => {
  const r = generator(seed);
  return ["core library", "encryption library", "name resolver plugin",
    "shell", "compression library"].map((ad) =>
      ({ ad, role: r() < 0.4 ? ROLE : 1 + Math.floor(r() * (ROLE - 1)) }));
};

// Window = baking wall-clock time + fleet renewal (REPLACE machines a day).
export const window = (n, seed = 20260812) => {
  const r = generator(seed);
  const single = Array.from({ length: n }, () => 7 + Math.round(r() * 12) + TEST);
  let wall = 0;
  for (let i = 0; i < n; i += LANE) wall += Math.max(...single.slice(i, i + LANE));
  return { wall, total: single.reduce((t, v) => t + v, 0), round: Math.ceil(n / LANE),
    days: wall / 1440 + MACHINE / REPLACE };
};

if (import.meta.url === `file://${process.argv[1]}`) {
  console.log(`${ROLE} roles, ${ENVIRONMENT} environments, ${MACHINE} machines; ` +
    `image count: golden ${IMAGE_COUNT["golden image"]}, hybrid ${IMAGE_COUNT["hybrid"]}\n`);
  console.log("fix".padEnd(26) + "roles touched".padEnd(15) +
    "golden image".padEnd(14) + "hybrid".padEnd(10) + "every boot");
  let golden = 0, hybrid = 0;
  for (const f of fixes()) {
    golden += f.role * ENVIRONMENT; hybrid += f.role;
    console.log(f.ad.padEnd(26) + `${f.role}/${ROLE}`.padEnd(15) +
      `${f.role * ENVIRONMENT} images`.padEnd(14) + `${f.role} image${f.role === 1 ? "" : "s"}`.padEnd(10) + "0 images");
  }
  console.log("total".padEnd(41) + `${golden} images`.padEnd(14) + `${hybrid} images`.padEnd(10) + "0 images");

  console.log("\nrollout window for the widest fix (every role):");
  console.log("path".padEnd(14) + "images baked".padEnd(14) + "round".padEnd(7) +
    "wall hours".padEnd(13) + "builder minutes".padEnd(17) + "window");
  for (const [ad, n] of Object.entries(IMAGE_COUNT)) {
    const p = window(n);
    console.log(ad.padEnd(14) + String(n).padEnd(14) + String(p.round).padEnd(7) +
      `${(p.wall / 60).toFixed(1)} hr`.padEnd(13) + String(p.total).padEnd(17) +
      `${p.days.toFixed(2)} days`);
  }
}
```

```
6 roles, 3 environments, 50 machines; image count: golden 18, hybrid 6

fix                       roles touched  golden image  hybrid    every boot
core library              6/6            18 images     6 images  0 images
encryption library        3/6            9 images      3 images  0 images
name resolver plugin      4/6            12 images     4 images  0 images
shell                     6/6            18 images     6 images  0 images
compression library       6/6            18 images     6 images  0 images
total                                    75 images     25 images 0 images

rollout window for the widest fix (every role):
path          images baked  round  wall hours   builder minutes  window
every boot    0             0      0.0 hr       0                4.17 days
golden image  18            6      2.1 hr       334              4.26 days
hybrid        6             2      0.7 hr       117              4.20 days
```

The first table shows the cost of where the environment gets written. If the environment is baked
into the image, each role gets baked three times: five fixes produce **75 image bakes**. If the
environment is applied at boot, the same five fixes need only **25 bakes**. The threefold gap
between the two is the price of the single extra call the hybrid path pays at boot.

The second table corrects a common expectation. A golden image's freshness is assumed to lag; once
counted, the lag is **0.09 days** — 4.26 days against 4.17 days. Baking wall-clock time is 2.1
hours, and holds about two percent of the window. What decides the window is not baking, it is
**the fleet's renewal**: since only 12 of 50 machines can be replaced a day, a fix cannot reach
every machine before four days pass, and that is the cost all three paths share. The discipline of
immutable infrastructure requires replacing a machine instead of patching it in place; the window
is the price of that rule.

The builder-minutes column is a repeated cost: the same fix spends 334 builder-minutes on the pure
golden path, 117 on the hybrid path. The image builder is itself an infrastructure component that
needs watching; if its capacity drops, the window stretches directly.

#### What is Inside the Image

A record decides which images get rebaked: the image's content record writes down which image
carries which component. If this record is incomplete, the rebake set is incomplete too, and no
one sees the gap, because baking finishes successfully.

```js
// measurement-network/freshness.mjs — the gap between an image's content record and its real content (model)
import { generator } from "./boot.mjs";
import { ROLE, ENVIRONMENT } from "./bake.mjs";

const COMPONENT = ["core library", "encryption library", "compression library",
  "name resolver plugin", "certificate store", "timezone data", "shell",
  "localization data", "math library", "version compatibility layer",
  "logging library", "text parser", "network client", "scheduler"];
const CYCLE = 30;                          // full rebake cycle (days)
const ROLE_MACHINE = [5, 6, 7, 9, 11, 12]; // machines per role; 50 total

// Each role image: direct components written in the record + indirect components the record misses.
const catalog = (seed = 20260813) => {
  const r = generator(seed);
  return Array.from({ length: ROLE }, (_, i) => {
    const recorded = COMPONENT.filter(() => r() < 0.4);
    const indirect = COMPONENT.filter((b) => !recorded.includes(b) && r() < 0.35);
    return { recorded, actual: [...recorded, ...indirect], machine: ROLE_MACHINE[i] };
  });
};

// Five separate security fixes: components are chosen without replacement, the day comes from the generator.
const fix = (seed = 20260814) => {
  const r = generator(seed), pool = [...COMPONENT];
  return Array.from({ length: 5 }, () => ({
    component: pool.splice(Math.floor(r() * pool.length), 1)[0],
    day: 1 + Math.floor(r() * CYCLE),
  }));
};

const roleImages = catalog(), fixList = fix();
const recordedCount = roleImages.reduce((t, i) => t + i.recorded.length, 0);
const actualCount = roleImages.reduce((t, i) => t + i.actual.length, 0);
console.log(`${ROLE} role images, ${COMPONENT.length} component kinds; the content record writes ` +
  `${recordedCount} entries, the images have ${actualCount} entries: ${actualCount - recordedCount} entries ` +
  `out of record (${(100 * (actualCount - recordedCount) / actualCount).toFixed(0)}%)\n`);
console.log("fix".padEnd(26) + "in record".padEnd(11) + "in reality".padEnd(12) +
  "missed".padEnd(11) + "unpatched machines".padEnd(20) + "machine-days");
let totalDays = 0, totalMissed = 0;
for (const f of fixList) {
  const inRecord = roleImages.filter((i) => i.recorded.includes(f.component));
  const inActual = roleImages.filter((i) => i.actual.includes(f.component));
  const missed = inActual.filter((i) => !inRecord.includes(i));
  const machines = missed.reduce((t, i) => t + i.machine, 0);
  const days = machines * (CYCLE - f.day);
  totalDays += days; totalMissed += missed.length;
  console.log(f.component.padEnd(26) + `${inRecord.length} image${inRecord.length === 1 ? "" : "s"}`.padEnd(11) +
    `${inActual.length} image${inActual.length === 1 ? "" : "s"}`.padEnd(12) + `${missed.length} image${missed.length === 1 ? "" : "s"}`.padEnd(11) +
    `${machines}/50`.padEnd(20) + String(days));
}
console.log(`\nimages missed across five fixes: ${totalMissed} (${totalMissed * ENVIRONMENT} on the ` +
  `pure golden-image path); unpatched machine-days: ${totalDays}`);
console.log(`missed images close only at the next ${CYCLE}-day full rebake`);
```

```
6 role images, 14 component kinds; the content record writes 32 entries, the images have 52 entries: 20 entries out of record (38%)

fix                       in record  in reality  missed     unpatched machines  machine-days
math library              6 images   6 images    0 images   0/50                0
shell                     3 images   5 images    2 images   23/50               322
certificate store         2 images   2 images    0 images   0/50                0
encryption library        1 image    3 images    2 images   14/50               126
timezone data             0 images   0 images    0 images   0/50                0

images missed across five fixes: 4 (12 on the pure golden-image path); unpatched machine-days: 448
missed images close only at the next 30-day full rebake
```

The header row is this lesson's number. The content record writes 32 component entries for six
images; the images have 52 entries inside them. **20 entries, 38 percent, are out of record** —
components that ride along with another component. The record is not wrong; it is incomplete.

The cost of that incompleteness shows up in the third and fourth columns. On the `shell` fix, the
record says 3 images are affected, in reality 5 images are affected; **2 images do not get
rebaked**, and 23 machines booted from those images stay unpatched. On the `encryption library`
fix, the record says 1 image, the actual is 3. Across five fixes, 4 images are missed in total, and
**448 unpatched machine-days** accumulate. These machines sit exposed until the next 30-day full
rebake, because no one is looking for them.

The last line also has an environment dimension: the same gap misses not 4 but **12 images** on
the pure golden path. Baking the environment into the image does not just triple the baking load,
it triples every gap in the record too.

This completes the three paths' measure. In the configure-at-every-boot path, there is no such
thing as a rebake set, so nothing can even be computed wrong; the moment a fix lands in the package
repository, it arrives at every machine's next boot. The golden image makes boot independent and
freezes the content — in exchange it **turns inventory into a record**, and that record's gaps are
a silent security hole. The 20 entries the record does not cover are what this course has been
counting from the start.

#### Summary

- Boot duration drops from 176 seconds to 45, outbound calls from 18 to 2; the expected number of
  machines that fail to boot in a forty-machine boot drops from 3.2 to 0.2.
- No path escapes the secret-store call — a sensitive value cannot be baked into an image; the
  golden path cuts the dependency count from four to two while picking up the image store as a
  new dependency.
- If the environment is baked into the image, five fixes need 75 image bakes; if applied at boot,
  25 bakes; builder-minutes are 334 against 117.
- The rollout window is 4.26 days against 4.17: baking's contribution is 0.09 days, and what
  decides the window is the fleet's ability to replace only 12 machines a day.
- The content record writes 32 entries, the images have 52; the 38 percent gap leaves 4 images
  (12 on the pure golden path) out of the rebake set across five fixes and accumulates 448
  unpatched machine-days.
- A golden image removes drift from boot but turns inventory into a record; that record's gaps
  are silent, because baking finishes successfully.

#### Next Step

The flow this lesson measured is itself a build step now: an image is compiled, tested, versioned,
and shipped to environments. The same is true of an infrastructure record — a plan is produced, it
passes through static checking and policy scanning, gets priced, gets applied. This topic's lessons
counted these steps one by one, but every one of them was run by hand.

M22/K04 ran an application change through gates and measured what the gate caught and what it
missed; that measure is not repeated, it is applied. The next lesson runs an infrastructure change
through the same pipeline and asks: when this course's checks are lined up in a pipeline, how many
changes stop at which gate, how many changes pass every gate and still produce drift, and how many
minutes does the pipeline itself add?

### Pipeline Integration

Source: https://academia.sh/en/courses/infrastructure-as-code/pipeline-integration

An infrastructure change is run through a pipeline: the plan stage and the apply stage are separated, and approval stands in between. The plan the approver approves without reading, the reality that goes stale between plan and apply, the breadth of the pipeline's apply permission, and a manual change being silently reverted, colliding with, or never seen by the pipeline's next run are all counted.

The previous lesson baked the image, the one before that applied the setting. Neither asked one
thing: who triggers these, and who approves them. All of this course's measurements were run by
hand, and in a hand-run flow a person stands between plan and apply and sees the two back to back.
A pipeline separates the two: the plan is produced at one stage, stored, submitted for approval;
the apply runs at another moment. M22/K04 measured what a gate caught and what it missed; that
measure is not repeated, it is applied.

**IV37.** The pipeline runs the regional measurement network's — the fictional software collecting
water-meter readings' — infrastructure record in two stages, and passes 18 changes over 30 days.
**IV38.** An item holds 3 lines in the plan, 12 lines are read per minute, and the approver does
not read a plan past 40 lines. **IV39.** The seeds `20260901`–`20260903` are visible and the
numbers are an order of magnitude. **IV40.** The provider offers 9 resource types and 4 operations,
the environment has 84 fictional resources, and the narrow permission is derived from the last 12
runs. **IV41.** 26 manual changes occur over 30 days; the touched item's probability of being in
the record's scope is 0.62, an in-scope item's probability of being protected is 0.24, and the
urgent-fix rate is 0.35. **IV42.** The freshness gate regenerates the plan at apply time and
compares it against the approved one, adding 0.5 minutes per change.

#### Plan, Approval, Apply

Approval is a signature; what can be measured is how much reading stands behind it, and how much
time passes between the signature and the apply — a plan is a snapshot of a moment, and it goes
stale.

```js
// measurement-network/pipeline.mjs — plan stage, approval, apply stage (model; a kind, not a product)
export const generator = (seed) => () =>
  (seed = (seed * 1103515245 + 12345) % 2147483648) / 2147483648;

// plan lines per item; lines read per minute; the line count where the approver stops reading;
// minutes the freshness gate adds per change.
const LINE = 3, SPEED = 12, THRESHOLD = 40, REGEN = 0.5;

// 18 infrastructure changes pass through the pipeline over 30 days; quantities come from the generator.
export const changes = (seed = 20260901) => {
  const r = generator(seed);
  return Array.from({ length: 18 }, () => {
    const item = 1 + Math.floor(r() * 14);
    return { item, line: item * LINE + 6, wait: Math.round(r() * 470) / 10,
      changed: r() < 0.3, drifted: 1 + Math.floor(r() * 3) };
  });
};

// gate=true: at apply time the plan is regenerated and compared against the approved one.
export const run = (list, gate) => {
  const unread = list.filter((d) => d.line > THRESHOLD);
  const stale = list.filter((d) => d.changed);
  const drifted = stale.reduce((t, d) => t + d.drifted, 0);
  return {
    unread: unread.length, unreadItems: unread.reduce((t, d) => t + d.item, 0),
    minutes: list.reduce((t, d) =>
      t + (d.line <= THRESHOLD ? d.line / SPEED : 0.5) + (gate ? REGEN : 0), 0),
    stale: stale.length, unprotected: gate ? 0 : drifted, drifted,
    held: gate ? stale.length : 0, applied: 18 - (gate ? stale.length : 0),
  };
};

if (import.meta.url === `file://${process.argv[1]}`) {
  const list = changes(), sum = (a) => list.reduce((t, d) => t + a(d), 0);
  console.log(`18 changes, ${sum((d) => d.item)} items, ${sum((d) => d.line)} plan lines; ` +
    `average ${(sum((d) => d.wait) / 18).toFixed(1)} hours between plan and apply\n`);
  console.log("freshness gate".padEnd(16) + "approved".padEnd(11) + "unread".padEnd(11) +
    "unread items".padEnd(17) + "review time".padEnd(16) + "stale plan".padEnd(12) +
    "unprotected items".padEnd(19) + "applied");
  for (const gate of [false, true]) {
    const s = run(list, gate);
    console.log((gate ? "yes" : "no").padEnd(16) + "18/18".padEnd(11) +
      `${s.unread}/18`.padEnd(11) + `${s.unreadItems}/${sum((d) => d.item)}`.padEnd(17) +
      `${s.minutes.toFixed(1)} min`.padEnd(16) + `${s.stale}/18`.padEnd(12) +
      `${s.unprotected}/${s.drifted}`.padEnd(19) + `${s.applied}/18`);
  }
  const noGate = run(list, false);
  console.log(`\nthe approver does not read once the plan exceeds ${THRESHOLD} lines: ${noGate.unread} changes, ` +
    `${noGate.unreadItems} items approved without review`);
}
```

```
18 changes, 129 items, 495 plan lines; average 28.6 hours between plan and apply

freshness gate  approved   unread     unread items     review time     stale plan  unprotected items  applied
no              18/18      4/18       49/129           29.0 min        3/18        5/5                18/18
yes             18/18      4/18       49/129           38.0 min        3/18        0/5                15/18

the approver does not read once the plan exceeds 40 lines: 4 changes, 49 items approved without review
```

Eighteen of eighteen changes passed approval; the approval rate says nothing. What says something
is the second column: **4 plans were approved without being read**, and those four plans carried
49 of 129 items — thirty-eight percent of the items were applied unlooked-at. This is not an
attention problem, it is a scale problem; the review load grows linearly, as already counted in
`declarative-provisioning/07`.

The second measure is the time between plan and apply: an average of **28.6 hours**, and reality
does not stand still over that time. **3 of the eighteen plans were stale when applied**, and 5
items that did not appear in the approved plan entered the apply; the approved document and the
applied work were not the same thing.

**One of this course's gates only does its job here.** Format checking, policy scanning,
infrastructure testing, and cost estimation can all run outside a pipeline too; the freshness gate
cannot. Regenerating the plan at apply time and comparing it against **the approved plan** requires
the approved plan to be a stored document; a hand-run flow has nothing stored to compare against.
The gate adds 9 minutes, holds 3 applies, and rescues 5 items.

#### The Pipeline's Permission

The pipeline now runs the apply itself, and for that it has an identity on the infrastructure. If
the permission is broad, the pipeline can do anything; if that identity falls, anything becomes
possible.

```js
// measurement-network/permission.mjs — the pipeline's apply permission over the infrastructure (model)
import { generator, changes } from "./pipeline.mjs";

const TYPE = ["machine", "disk", "network", "subnet", "security rule", "queue", "data store",
  "object store", "identity"];
const OPERATION = ["create", "read", "update", "delete"];
const RESOURCE = [14, 22, 3, 6, 19, 2, 4, 3, 11];   // fictional resources per type; 84 total
const WINDOW = 12;   // past runs the permission is derived from

// Each change touches a few (type, operation) pairs; the pair set comes from the generator.
const touches = (seed = 20260902) => {
  const r = generator(seed);
  return changes().map(() => {
    const c = new Set(), n = 1 + Math.floor(r() * 3);
    for (let i = 0; i < n; i++)
      c.add(`${TYPE[Math.floor(r() * 9)]}/${OPERATION[Math.floor(r() * 4)]}`);
    return [...c];
  });
};

const touch = touches();
const all = TYPE.flatMap((t) => OPERATION.map((i) => `${t}/${i}`));
const narrow = [...new Set(touch.slice(0, WINDOW).flat())];
const noDelete = narrow.filter((c) => !c.endsWith("/delete"));
const deletable = (y) => TYPE.reduce((t, x, i) => t + (y.includes(`${x}/delete`) ? RESOURCE[i] : 0), 0);

console.log(`${TYPE.length} resource types x ${OPERATION.length} operations = ${all.length} operations; ` +
  `${RESOURCE.reduce((a, b) => a + b)} fictional resources\n`);
console.log("permission".padEnd(23) + "operations".padEnd(12) + "passed".padEnd(14) +
  "left over".padEnd(12) + "deletable resources");
for (const [ad, y] of [["broad (all)", all], [`narrow (from ${WINDOW} runs)`, narrow],
  ["narrow, no delete", noDelete]]) {
  const passed = touch.filter((c) => c.every((x) => y.includes(x))).length;
  console.log(ad.padEnd(23) + `${y.length}/36`.padEnd(12) + `${passed}/18`.padEnd(14) +
    `${18 - passed}/18`.padEnd(12) + `${deletable(y)}/84`);
}
const newPairs = [...new Set(touch.slice(WINDOW).flat())].filter((c) => !narrow.includes(c));
console.log(`\nnarrow permission was derived by looking at the past; the next 6 runs produced ${newPairs.length} ` +
  `new operation pairs: ${newPairs.join(", ")}`);
```

```
9 resource types x 4 operations = 36 operations; 84 fictional resources

permission             operations  passed        left over   deletable resources
broad (all)            36/36       18/18         0/18        84/84
narrow (from 12 runs)  19/36       14/18         4/18        14/84
narrow, no delete      18/36       13/18         5/18        0/84

narrow permission was derived by looking at the past; the next 6 runs produced 5 new operation pairs: subnet/delete, queue/update, security rule/delete, subnet/create, disk/create
```

Broad permission finishes the table in one line: 36 of 36 operations, 18 of 18 changes, zero left
on the table. The last column is the cost — if the pipeline's identity is compromised, **84 of 84
resources** can be deleted; the pipeline is a wider access than any record written in this course
grants.

When permission is narrowed by looking at the past twelve runs, the operation count drops to
**19**, deletable resources to **14**; the cost is that **4 of the 18 changes** are left on the
table. Once delete permission is removed too, deletable resources drop to zero, and what is left on
the table rises to 5. The bottom line writes down why narrowing is not permanent: the next six runs
produced **5 new operation pairs** never seen before. Permission derived from the past leaves the
future's work on the table, and for what is left, the manual path counted in the first lesson
carries on exactly as before.

#### When a Manual Change Hits the Pipeline

Machines still get touched by hand after a pipeline is set up; the question is what happens to
that touch at the pipeline's next run.

```js
// measurement-network/collision.mjs — what a manual change becomes at the pipeline's next run (model)
import { generator } from "./pipeline.mjs";

const RUN = [1.4, 3.1, 4.0, 6.8, 8.2, 9.9, 12.5, 14.1, 15.0, 17.7,
  19.2, 21.4, 22.8, 24.3, 25.1, 27.6, 28.9, 29.5];   // day of each of the 18 pipeline runs (fictional)
const COVERAGE = 0.62;    // probability that a manually touched item is in the record's scope
const PROTECTION = 0.24;  // probability that an in-scope item is protected or recreates on change

// 26 manual changes occur over 30 days; each touches a single item.
const manual = (seed = 20260903) => {
  const r = generator(seed);
  return Array.from({ length: 26 }, () => {
    const day = Math.round(r() * 300) / 10;
    const inScope = r() < COVERAGE;
    return { day, inScope, protected: inScope && r() < PROTECTION, urgent: r() < 0.35 };
  });
};

const RESULT = { "silently reverted": [], "conflicting (pipeline halts)": [], "unnoticed": [] };
for (const o of manual()) {
  const next = RUN.find((g) => g > o.day);
  const outcome = !o.inScope ? "unnoticed"
    : o.protected ? "conflicting (pipeline halts)" : "silently reverted";
  // An out-of-record item is never seen on any run; it has no delay either.
  const delay = !o.inScope || next === undefined ? null : next - o.day;
  RESULT[outcome].push({ ...o, delay });
}

console.log("26 manual changes, 30 days, 18 pipeline runs\n");
console.log("outcome".padEnd(31) + "count".padEnd(9) + "urgent".padEnd(8) + "delay");
for (const [ad, list] of Object.entries(RESULT)) {
  const g = list.map((o) => o.delay).filter((v) => v !== null);
  console.log(ad.padEnd(31) + `${list.length}/26`.padEnd(9) +
    String(list.filter((o) => o.urgent).length).padEnd(8) +
    (g.length ? `${(g.reduce((a, b) => a + b, 0) / g.length).toFixed(1)} days` : "-"));
}
const reverted = RESULT["silently reverted"];
console.log(`\n${reverted.length} reverted changes produced no error, all of them were a single line in the plan; ` +
  `${reverted.filter((o) => o.urgent).length} of them were urgent fixes`);
```

```
26 manual changes, 30 days, 18 pipeline runs

outcome                        count    urgent  delay
silently reverted              15/26    4       1.2 days
conflicting (pipeline halts)   3/26     1       0.7 days
unnoticed                      8/26     2       -

15 reverted changes produced no error, all of them were a single line in the plan; 4 of them were urgent fixes
```

The three rows are three separate events, and only the middle one is visible. In **3 collisions**
the pipeline halts and a person makes the call; these count as proof the pipeline is working.
**15 changes are silently reverted** and vanish after an average of 1.2 days; none of them produce
an error, because the pipeline was built to do exactly that. **4 of those fifteen were urgent
fixes** — because their records were never updated, they went back to the old value the record
wanted. `iac-fundamentals/05` already counted a rollback breaking a live fix; the pipeline makes
this reversion faster and quieter.

Every reversion was a single line in the plan; given that four of the plans were approved without
being read, the source of the silence is not the pipeline, it is the unread plan line.

The third row writes down the course's rule one last time: **8 items are items the pipeline never
sees at all.** Because they are out of record, they neither revert nor collide; they remain in
place while the pipeline runs flawlessly.

#### Summary

- Eighteen of eighteen changes passed approval, but 4 plans were approved without being read, and
  those four plans carried 49 of 129 items (38 percent); the approval rate is not a measure.
- An average of 28.6 hours passed between plan and apply, 3 of the 18 plans were stale, and 5 items
  not visible at approval time entered the apply.
- The freshness gate is meaningful only in a pipeline, because it requires the approved plan to be
  stored; it adds 9 minutes, holds 3 applies, and rescues 5 items.
- Under broad permission, if the identity falls, 84 of 84 resources can be deleted; once permission
  narrows to 19 operations, deletable resources drop to 14, work left on the table rises to 4
  changes, and the next six runs produce 5 new operation pairs.
- Of 26 manual changes, 15 were silently reverted (4 urgent fixes), 3 collided, and 8 were never
  seen at all because they were out of record.

#### Course Wrap-Up

| Lesson | What enters the record | Gap between record and reality | The record's cost or limit |
|---|---|---|---|
| `iac-fundamentals/01` | 8-line description | 9 items diverge | 5 items out of scope |
| `iac-fundamentals/02` | 9-item target | 6/9 against 2/9 | 5 guard lines |
| `iac-fundamentals/03` | image, 96 events | 10/12 machines unique | 3 fixes lost |
| `iac-fundamentals/04` | 34 items, 35 definitions | 6 items conflict | 5 items in no record |
| `iac-fundamentals/05` | 26 items | 12 drifted, 6 close | 12 items out of scope |
| `declarative-provisioning/01` | 3 blocks, 14 assignments | 8 defects at apply | 9 fields blind to the type |
| `declarative-provisioning/02` | 20 fields | 36 fields on target | 16 fields not in record |
| `declarative-provisioning/03` | 8 resources, 5 data sources | scan 8/13 items | 2 external fields invisible |
| `declarative-provisioning/04` | 14 variables, 24 values | 4 wrong inputs passed | 48 hand-written values across 6 environments |
| `declarative-provisioning/05` | 14 resources, 23 edges | requirements 27, seen 23 | 4 requirements not in record |
| `declarative-provisioning/06` | 3 forms, 10 resources | 1 versus 6 items | 525 machine-seconds downtime |
| `declarative-provisioning/07` | plan, 11 items | 8 of 13 addresses hold | 39.6 minutes across 30 regions |
| `declarative-provisioning/08` | 20 of 24 fields | first plan, 4 items | 4 fields out of record |
| `state-and-modules/01` | 56 fields, 80 in state | 4 of 33 items out of record | 3 items do not return on loss |
| `state-and-modules/02` | two concurrent applies | 12 items lost without a lock | 12 resources out of record |
| `state-and-modules/03` | 620 items, 673 edges | 66 edges cross the boundary | 3 items out of record |
| `state-and-modules/04` | 17 or 51 lines | 23 versus 4 wrong environments | 2 environments, 9 fields behind each |
| `state-and-modules/05` | module, 20 lines | 0 of 4 items seen | 1 resource out of record |
| `state-and-modules/06` | 9 definitions, 12 versions | 9/9 do not say what is running | 25 versions behind |
| `state-and-modules/07` | 7 trace points | 14 copies, marking → 4 | 3 points stay open |
| `iac-validation/01` | 11 resources, 59 lines | three checks 16/16 | 4 unseen rejections |
| `iac-validation/02` | 62 lines, 7 policies | 3 missed, 2 false alarms | 3 computed values invisible |
| `iac-validation/03` | 15 defects | three levels 12/15 | 11 leftovers, 0 in record |
| `iac-validation/04` | 24 items, 12 changes | 9 at 0.51x–1.45x | 11 items not estimable |
| `iac-validation/05` | 1700 record lines | 1598 vs. 1530 hold | 27/61 settings out of scope |
| `iac-validation/06` | content record 32 entries | 52 entries in images | 20 entries out of record |
| `iac-validation/07` | 18 changes, 495 lines | 3 stale plans, 5 items | 8 items unseen |

This is the course's rule: **infrastructure can only be managed as well as its record; every item
the record does not cover is a source of drift.** Three limits repeat in the last column.
**In nineteen rows, the limit is an item the record does not cover** — an out-of-record resource, a
provider default, an unseen requirement, a computed value, an indirect component. **In five rows**
the limit is the record's own operating overhead: a guard line, review minutes, a version left
behind, a delayed apply. **In three rows** the limit is an unrecoverable loss: a deleted fix, a
machine out of service, an item that does not return. Growing the record shrinks the first group
and grows the second; this trade-off was never zeroed out in any lesson.

Infrastructure is now tied to a record: it is written, planned, checked, priced, approved, and
becomes work a pipeline applies. But all of this course's measurements belonged to **the moment of
change**. No one yet sees what the infrastructure does while it runs: which machine carries how
much load, which queue is backing up. **M22/K06 Observability and Operations** starts with that
question — who decides what gets measured?