Infrastructure Provisioning Models

The previous lesson turned twelve principles into checks and found that four needed a second observation to decide — often a second environment. The application itself is now checkable: its dependencies are declared, its configuration is external, its shutdown is testable. The machine it stands on is set up by hand, and how it is set up is inside the scope of no check.

Provisioning is bringing an environment to its desired state: installing packages, writing settings, opening directories. It has three forms — manual, scripted, declarative — and what separates them is not speed, it is how many separate results the same recipe produces. This lesson sets up the same environment in all three and measures three numbers: the setup steps, the difference between two setups, and how much configuration drift grows over successive periods.

The measurement is again on the regional measurement network: fictional software that collects water meter readings, verifies them, and opens field work orders. The files are genuinely written to disk, the scripts and the reconciler genuinely run.

DC27. The environment is represented by seven items — two package versions, three settings, two directories. DC28. The manual setup’s instructions are prose, and three of the seven steps give no number; the two hands’ interpretations diverge on exactly these three steps. A hand is a model, its interpretation is written in advance. DC29. The script runs the seven steps in order, and no step reads the state first; the cutoff point is triggered by an environment variable. DC30. In the declarative model, the desired state is a file; the reconciler reads the actual state from disk and touches only the differing item. DC31. The drift scan runs six periods, and one manual intervention happens in each period; the interventions are written in advance, there is no randomness.

Three Models, the Same Seven Items

#!/usr/bin/env bash
# Three separate recipes for the same environment: a declaration, a manual instruction sheet, and a provisioning script.
set -e

cat > desired.json <<'SON'
{ "package/resolver": "3.4.0", "package/rule-package": "9.1.0",
  "config/log-level": "summary", "config/concurrent-workers": "8",
  "config/batch-start": "02:30", "dir/data": null, "dir/log": null }
SON

cat > manual-recipe.txt <<'SON'
1. open the package directory, install resolver version 3.4.0
2. pin the rule package to version 9.1.0
3. pull the log level to a value suited for production
4. set the concurrent worker count to the machine's capacity
5. schedule the nightly batch job for a low-traffic hour
6. open the data directory
7. open the log directory
SON

cat > setup.sh <<'SON'
#!/usr/bin/env bash
# Scripted provisioning: seven steps run in order; no step reads the state first.
set -e
mkdir -p environment/package environment/config environment/dir
echo "3.4.0" >> environment/package/resolver
echo "9.1.0" >> environment/package/rule-package
echo "summary" >> environment/config/log-level
[ -z "$CUT" ] || exit 3                       # the cutoff point
echo "8"     >> environment/config/concurrent-workers
echo "02:30" >> environment/config/batch-start
mkdir environment/dir/data environment/dir/log
SON

All three recipes target the same seven items. The declaration counts items along with their values; the instructions give no number on three steps (“a value suited for,” “to the machine’s capacity,” “a low-traffic hour”); the script runs the seven steps in order. The checker fixes nothing — it only compares the disk against the desired state and counts.

// check.mjs — compares the environment on disk against the desired state and counts; fixes nothing.
import { readFileSync, existsSync, statSync } from "node:fs";

const root = process.argv[2] ?? "environment";
const desired = JSON.parse(readFileSync("desired.json", "utf8"));
const state = (path) => {
  const t = `${root}/${path}`;
  if (!existsSync(t)) return "missing";
  const g = statSync(t).isDirectory() ? "" : readFileSync(t, "utf8").trim();
  return g === (desired[path] ?? "") ? "ok" : "drifted";
};
const s = Object.keys(desired).map(state);
const count = (t) => s.filter((x) => x === t).length;
const drift = count("drifted") + count("missing");
console.log(process.argv[3] === "-d" ? drift                       // -d: drift count only
  : `  items ${s.length}   ok ${count("ok")}   drifted ${count("drifted")}` +
    `   missing ${count("missing")}   drift ${drift}`);

Same Recipe, How Many Separate Results

#!/usr/bin/env bash
# The scripted model's three scenarios and manual setup's two hands.
run() { printf '%s\n' "$1"; shift; "$@" >/dev/null 2>&1; printf '  script exit code %s\n' "$?"; }

rm -rf environment
run "1st run (empty environment)"       bash setup.sh
node check.mjs
run "2nd run (same script again)"       bash setup.sh
node check.mjs

rm -rf environment
run "3rd run (cut off midway)"          env CUT=1 bash setup.sh
node check.mjs
printf '  records of where it stopped: %s files\n' "$(ls environment/*.state 2>/dev/null | wc -l | tr -d ' ')"

hand() {                                     # $1 dir, $2..$4 the three values the hand interpreted
  rm -rf "$1"; mkdir -p "$1/package" "$1/config" "$1/dir/data" "$1/dir/log"
  printf '3.4.0\n' > "$1/package/resolver"; printf '9.1.0\n' > "$1/package/rule-package"
  printf '%s\n' "$2" > "$1/config/log-level"; printf '%s\n' "$3" > "$1/config/concurrent-workers"
  printf '%s\n' "$4" > "$1/config/batch-start"
}
hand hand-1 summary 8 02:30
hand hand-2 detail 4 03:00
printf 'manual setup: two hands from the same instructions\n  difference between the two setups: %s items\n' \
  "$(diff -rq hand-1 hand-2 | wc -l | tr -d ' ')"
node check.mjs hand-1
node check.mjs hand-2

1st run (empty environment)
  script exit code 0
  items 7   ok 7   drifted 0   missing 0   drift 0
2nd run (same script again)
  script exit code 1
  items 7   ok 2   drifted 5   missing 0   drift 5
3rd run (cut off midway)
  script exit code 3
  items 7   ok 3   drifted 0   missing 4   drift 4
  records of where it stopped: 0 files
manual setup: two hands from the same instructions
  difference between the two setups: 3 items
  items 7   ok 7   drifted 0   missing 0   drift 0
  items 7   ok 4   drifted 3   missing 0   drift 3

The script’s first run puts all seven items in place. The same script’s second run breaks the environment: five items drift, and the script falls over at the seventh step. The reason is two characters — the appending >> and the mkdir that takes no -p. Because the script never reads the state first, it cannot know whether the work was already done; an operation producing the same result whether applied once or many times is called idempotence, and this script does not carry it.

The run cut off midway is more expensive: three items are in place, four are missing, and the record of where it stopped is zero. The environment is neither in its old state nor its new one; the only thing the script reports is an exit code, and that code does not say which step it stopped at.

In manual setup, two hands read the same instructions and diverged on three items — exactly the three steps the instructions gave no number for. The difference here is not carelessness — it is the recipe’s unwritten part: a human fills the gap between the recipe and the environment, and the value they fill it with is written down nowhere. The measure collapses into one line: in manual setup, the same recipe produced two separate results; in scripted provisioning, the second run produced a third result — and more broken than the first.

Drift Being Countable

// reconcile.mjs — reads the desired state from a file and the actual state from disk; writes and closes the difference.
import { readFileSync, writeFileSync, mkdirSync, existsSync, statSync } from "node:fs";

const desired = JSON.parse(readFileSync("desired.json", "utf8"));
const actual = (path) => {
  const t = `environment/${path}`;
  if (!existsSync(t)) return null;
  return statSync(t).isDirectory() ? "" : readFileSync(t, "utf8").trim();
};

const diff = Object.entries(desired).map(([path, d]) => ({ path, d: d ?? "", a: actual(path) }))
  .filter((f) => f.a !== f.d)
  .map((f) => ({ ...f, kind: f.a === null ? "missing" : "drifted" }));

for (const f of diff) {                        // reconciliation: only the differing item is touched
  const t = `environment/${f.path}`;
  if (desired[f.path] === null) mkdirSync(t, { recursive: true });
  else {
    mkdirSync(t.slice(0, t.lastIndexOf("/")), { recursive: true });
    writeFileSync(t, desired[f.path] + "\n");
  }
}
console.log(`  items ${Object.keys(desired).length}   diff ${diff.length}` +
  `   (missing ${diff.filter((f) => f.kind === "missing").length},` +
  ` drifted ${diff.filter((f) => f.kind === "drifted").length})   work done ${diff.length}`);
for (const f of diff) console.log(`    ${f.kind.padEnd(8)} ${f.path.padEnd(27)}` +
  (desired[f.path] === null ? " (directory opened)" : ` "${f.a ?? ""}" -> "${f.d}"`));

#!/usr/bin/env bash
# Reconciler: empty environment, second run (idempotence), after manual intervention.
rm -rf environment
echo "1st run (empty environment)";     node reconcile.mjs
echo "2nd run (right after)";           node reconcile.mjs
echo "manual intervention: worker count set to 16, log directory deleted"
echo 16 > environment/config/concurrent-workers; rm -rf environment/dir/log
echo "3rd run (after the intervention)"; node reconcile.mjs

drift() { case $1 in
  1) echo 16 > environment/config/concurrent-workers ;;   2) echo detail > environment/config/log-level ;;
  3) rm -rf environment/dir/log ;;                        4) echo 3.5.0 > environment/package/resolver ;;
  5) echo 03:15 > environment/config/batch-start ;;       6) echo 9.2.0 > environment/package/rule-package ;;
esac; }

echo ""
echo "six periods, one manual intervention per period (drift accumulation)"
for reconciliation in off on; do
  rm -rf environment; node reconcile.mjs > /dev/null
  printf '  reconciliation %-3s :' "$reconciliation"
  for d in 1 2 3 4 5 6; do
    drift "$d"
    [ "$reconciliation" = on ] && node reconcile.mjs > /dev/null
    printf ' %s' "$(node check.mjs environment -d)"
  done
  printf '\n'
done

echo ""
printf 'recipe size (lines): instructions %s, script %s, declaration %s, reconciler %s\n' \
  "$(wc -l < manual-recipe.txt)" "$(wc -l < setup.sh)" "$(wc -l < desired.json)" \
  "$(wc -l < reconcile.mjs)" | tr -s ' '

1st run (empty environment)
  items 7   diff 7   (missing 7, drifted 0)   work done 7
    missing  package/resolver            "" -> "3.4.0"
    missing  package/rule-package        "" -> "9.1.0"
    missing  config/log-level            "" -> "summary"
    missing  config/concurrent-workers   "" -> "8"
    missing  config/batch-start          "" -> "02:30"
    missing  dir/data                    (directory opened)
    missing  dir/log                     (directory opened)
2nd run (right after)
  items 7   diff 0   (missing 0, drifted 0)   work done 0
manual intervention: worker count set to 16, log directory deleted
3rd run (after the intervention)
  items 7   diff 2   (missing 1, drifted 1)   work done 2
    drifted  config/concurrent-workers   "16" -> "8"
    missing  dir/log                     (directory opened)

six periods, one manual intervention per period (drift accumulation)
  reconciliation off : 1 2 3 4 5 6
  reconciliation on  : 0 0 0 0 0 0

recipe size (lines): instructions 7, script 11, declaration 3, reconciler 27

The reconciler does three things: reads the desired state from a file, reads the actual state from disk, touches only the differing item. The first run finds all seven items missing and does seven jobs; the second run does zero work. The idempotence the script lacked is measured here, and the reason is a single change of order — read first, write second.

The distinguishing part is in the third run. The manual intervention broke two items; the reconciler wrote both down by name, with their old and new values, and closed them. Drift is a number here: diff 2. In the other two models, there was nowhere to count drift, because the desired state was not readable — one sat in a person’s interpretation, the other in a sequence of commands. A sequence of commands says what to do, not what should be; drift’s countability is the difference between these two sentences.

The six-period scan reduces this to two lines: when reconciliation does not run, interventions accumulate (1, 2, 3, 4, 5, 6); when it does, the series stays at zero. Drift does not disappear — it is born and closed every period. What does not accumulate is not drift — it is unknown drift.

The Cost of the Three Models

Measure	Manual	Scripted	Declarative
recipe size (lines)	7	11	3 plus 27 (written once)
items in place on first setup	7 and 4 (two hands)	7	7
separate results from the same recipe	2	2	1
if the recipe runs a second time	unmeasurable	5 items drift, exit 1	0 work
what is left if it is cut off midway	unrecorded	3 items, 0 records	the difference is recomputed
drift accumulated over 6 periods	1 to 6	1 to 6	0 every period
catching drift	none	none	the 7 items in the declaration

Writing cost looks lowest in the declarative model — three lines — but the number is misleading: the reconciler is twenty-seven lines, written once and reused for every environment. The script is eleven lines and grows with every new item; the instructions are seven lines, and grow more ambiguous too, because every sentence of prose is a point of interpretation.

First-setup time is the same order of magnitude across all three models; the distinction does not show up there — it shows up on the second run and after an intervention. This is what the provisioning debate most often misses: on first setup alone, a script looks sufficient — there it produces the same result as the declarative model.

The declarative model’s limit is countable too. The reconciler sees only the seven items written in the declaration; when an eighth file is added to the environment by hand, it never shows up as a difference in any run. What makes drift countable is the declaration’s scope, and the difference outside that scope accumulates in no written place — in a hand’s interpretation, in the partial state of a run cut off midway. This is where the six differences visible only at runtime came from, in this topic’s first lesson.

Summary

• In manual setup, two hands diverged from the same instructions on 3 items; the steps where they diverged were the three steps the instructions gave no number for.
• The script’s first run put 7 items in place; the second broke 5 items and fell over; the run cut off midway left 3 items in place, 4 missing, and 0 records of where it stopped.
• The reconciler did 7 jobs on the first run, 0 on the second; after a manual intervention it wrote down and closed 2 drifted items by name.
• Over six periods, drift accumulated from 1 to 6 without reconciliation and stayed at 0 with it; what does not accumulate is not drift — it is unknown drift.
• The declarative model’s limit is the declaration’s scope: everything outside that scope falls back to manual setup’s condition.

Course Wrap-Up

Lesson	Measured difference or metric	Number	Where the difference is hidden
What Is DevOps?	occurrence that never reaches its owner, steps to reach	4 of 22 occurrences; 7.0 → 1.6 steps	the split observation scope
Throwing Over the Wall	context item dropped at the boundary	21 of 29 items unwritten, 14 drop	tacit context
Value Stream	wait piling up in a single step	1485 of 2237 steps; value-added share 38.5%	the distribution of wait
Delivery Metrics	the metric that moves when batch drops from 8 to 2	deployment 3.2 → 13.5; errors fixed at 12	the metric’s denominator
Site Reliability Engineering	the error budget and its exhaustion	43.2 minutes; heavy incident 208%; 8/12 periods	between the target and the outage distribution
Feedback Loops	the error that does not fall into an early loop	32 of 120 errors, 95% of the delay	loop coverage
Environments	environment parity, difference per dimension	18 differences; 12 in the manifest, 6 at runtime	the dimension the manifest does not declare
Build Artifacts and Immutability	the source that breaks reproducibility	4 sources, 3 invisible on a single machine	the value embedded in the output
Build Artifact Repositories	the separate artifact a tag carried	11 deployments by name, 7 carried a different artifact	name resolution at deployment time
Configuration and Secret Separation	the key that changes, the leak path	6 of 14 keys; 5 of 7 leaks	the default embedded in the source
Twelve-Factor App	the principle turned into a check, the defect caught	8 of 12 principles; 6 of 8 defects	between the rule’s scope and the principle’s subject
Infrastructure Provisioning Models	the separate result born from the same recipe	manual 3, scripted 5, declarative 0; drift 6 → 0	the recipe’s unwritten part

All twelve rows are examples of a single rule: if the same software behaves differently across two environments, the difference is somewhere, and it is not measured until it is named. No number in the table comes from the software itself — every one comes from where the difference is hidden. Whether an item is written down, what a name resolves to at deployment time, at which step an instruction gives no number — none of these lives inside the code, none of them shows on its own.

Reading the last column, a clustering becomes visible: most of the places the difference hid were the environment itself — installed packages, library versions, file paths, timezone. Widening the manifest, pulling configuration out, and making provisioning declarative made this difference measurable, but they did not remove the difference from the environment; each time, what happened was moving the difference somewhere countable.

One question remains, and M22/K02 Containers takes it up: is it possible to pull this difference inside the build artifact and remove it from the environment? If installed packages, versions, and file paths become part of the output, the only thing left for the environment to do is run that output, and nothing is left to provision. If it is possible, what is the cost — how much does the output grow, what happens to reproducibility, what is left that cannot be pulled out of the environment?