Provisioning and Configuration Management

The previous lesson mixed two jobs together. Which packages the image carries and what the settings on the machine are is one job; how many machines there are, which network they attach to, and which disk they carry is another. The hotfix getting lost happened exactly on the boundary between these two jobs.

Provisioning creates the resource: it opens the machine, sets up the network, attaches the disk, creates the queue. Configuration management sets up the inside of the resource: it installs packages, writes files, starts the service. Both have a record, and the two records are written in different formats. This lesson splits an environment’s items into two layers and counts where the boundary is drawn: how many items are defined in both layers, how many in neither, which layer wins on a conflicting item, and how many items get applied twice when the boundary shifts.

The environment is again the staging environment of the regional measurement network, and it is fictional. The record files are genuinely written to disk; the split into layers, the application order, and the drift window are modeled with node.

IF19. The staging environment is represented by thirty-four items, and each item’s natural layer is written in the inventory: resource or content. IF20. The two layers’ records are two separate formats — the provisioning record in blocks of the configuration language, the config record in the configuration agent’s line format. IF21. If an item is defined in only one layer, that layer applies it; if it is defined in both, whichever runs last wins. IF22. The usual order is provisioning then config. When a reprovisioning is triggered, the order reverses: config has already run, provisioning runs after. IF23. The configuration agent runs every thirty minutes; there are fourteen machines in the environment. IF24. A quarter sees twenty-two changes — seven on resource items, fifteen on content items; six of the content changes are on items baked into the image.

The Inventory and the Two Records’ Formats

#!/usr/bin/env bash
# An environment's item inventory and the two layers' records: provisioning (creates the resource), config (inside the resource).
set -e

cat > inventory.txt <<'SON'
network.measurement-network.block                  resource
network.measurement-network.subnet_count           resource
network.measurement-network.egress_path            resource
machine.collector.count                            resource
machine.collector.size                             resource
machine.collector.disk_gb                          resource
machine.collector.image                            resource
machine.verifier.count                             resource
machine.verifier.size                              resource
queue.reading.partitions                           resource
queue.reading.retention_hours                      resource
data_store.measurement.size_gb                     resource
data_store.measurement.backup_days                 resource
object_store.archive.class                         resource
object_store.archive.lifecycle_days                resource
security.collector.ingress                         resource
security.collector.egress                          resource
dns_record.collector.internal                      resource
machine.collector.package.resolver                 content
machine.collector.package.rule_package             content
machine.collector.config.log_level                 content
machine.collector.config.concurrent_workers        content
machine.collector.config.batch_hour                content
machine.collector.dir.data                         content
machine.collector.dir.log                          content
machine.collector.service.autostart                content
machine.collector.timezone                         content
machine.collector.log_rotation                     content
machine.verifier.package.resolver                  content
machine.verifier.config.log_level                  content
machine.verifier.service.autostart                 content
data_store.measurement.user.report                 content
data_store.measurement.parameter.connection_count  content
object_store.archive.access_policy                 content
SON

cat > provisioning.infra <<'SON'
resource network "measurement-network" { block = "10.40.0.0/16"  subnet_count = 3 }
resource machine "collector" { count = 3  size = "medium"  disk_gb = 40  image = "base-7" }
resource machine "collector" { config.log_level = "summary"  timezone = "UTC" }
resource machine "collector" { service.autostart = "open"  package.resolver = "3.4.0" }
resource machine "verifier" { count = 2  size = "small" }
resource queue "reading" { partitions = 6  retention_hours = 72 }
resource data_store "measurement" { size_gb = 200  backup_days = 14 }
resource object_store "archive" { class = "sparse"  lifecycle_days = 365  access_policy = "in-account" }
resource security "collector" { ingress = "10.40.0.0/16" }
SON

cat > config.record <<'SON'
machine/collector       package.resolver = 3.5.0
machine/collector       package.rule_package = 9.1.0
machine/collector       config.log_level = summary
machine/collector       config.concurrent_workers = 8
machine/collector       config.batch_hour = 02:30
machine/collector       dir.data = /data/measurement
machine/collector       dir.log = /log/collector
machine/collector       service.autostart = open
machine/collector       timezone = UTC
machine/collector       disk_gb = 40
machine/verifier        package.resolver = 3.5.0
machine/verifier        config.log_level = detail
machine/verifier        service.autostart = open
data_store/measurement  user.report = read-only
object_store/archive    access_policy = in-account-plus-read
SON

The inventory counts the items that genuinely exist in the environment; it is independent of what the records say. The two records fall on top of it. The provisioning record is written in blocks, and a single block can hold both attributes belonging to the resource’s shell (count, size, disk_gb) and ones belonging to the inside of the resource (package.resolver, timezone). The config record is in line format and writes only to the inside of the resource. The boundary sits not between the two, but at their intersection.

Where the Boundary Falls

// layer.mjs — matches the inventory against the two layers' records; counts overlap and gap. Model.
import { readFileSync } from "node:fs";

const BLOCK = /^resource\s+(\w+)\s+"([\w-]+)"\s*\{(.*)\}$/;
export const provisioning = new Map();
for (const s of readFileSync("provisioning.infra", "utf8").split("\n")) {
  const m = BLOCK.exec(s.trim());
  if (m) for (const x of m[3].matchAll(/([\w.]+)\s*=\s*("[^"]*"|[\w./:-]+)/g))
    provisioning.set(`${m[1]}.${m[2]}.${x[1]}`, x[2].replaceAll('"', ""));
}
export const config = new Map();                      // the configuration agent's record: a different format
for (const s of readFileSync("config.record", "utf8").split("\n")) {
  const m = /^(\w+)\/([\w-]+)\s+([\w.]+)\s*=\s*(\S+)$/.exec(s.trim());
  if (m) config.set(`${m[1]}.${m[2]}.${m[3]}`, m[4]);
}
export const inventory = readFileSync("inventory.txt", "utf8").trim().split("\n")
  .map((s) => s.trim().split(/\s+/)).map(([k, natural]) => ({ k, natural }));

export const split = (item, S, A) => {               // every item falls into exactly one box
  const g = { provisioningOnly: [], configOnly: [], overlap: [], gap: [] };
  for (const { k } of item) {
    const s = S.has(k), a = A.has(k);
    g[s && a ? "overlap" : s ? "provisioningOnly" : a ? "configOnly" : "gap"].push(k);
  }
  return g;
};

if (process.argv[1].endsWith("layer.mjs")) {       // writes when run directly, silent when imported
  const g = split(inventory, provisioning, config);
  console.log(`inventory ${inventory.length} items   provisioning record ${provisioning.size}` +
    `   config record ${config.size}`);
  console.log(`  provisioning only ${g.provisioningOnly.length}   config only ${g.configOnly.length}` +
    `   overlap ${g.overlap.length}   gap ${g.gap.length}`);
  console.log(`${g.overlap.length} overlapping items (defined in both layers):`);
  for (const k of g.overlap)
    console.log(`  ${k.padEnd(40)} provisioning "${provisioning.get(k)}"   config "${config.get(k)}"` +
      `   ${provisioning.get(k) === config.get(k) ? "same" : "CONFLICT"}`);
  console.log(`${g.gap.length} items in the gap (in neither layer):`);
  for (const k of g.gap)
    console.log(`  ${k.padEnd(51)} natural layer: ${inventory.find((e) => e.k === k).natural}`);
}

inventory 34 items   provisioning record 20   config record 15
  provisioning only 14   config only 9   overlap 6   gap 5
6 overlapping items (defined in both layers):
  machine.collector.disk_gb                provisioning "40"   config "40"   same
  machine.collector.package.resolver       provisioning "3.4.0"   config "3.5.0"   CONFLICT
  machine.collector.config.log_level       provisioning "summary"   config "summary"   same
  machine.collector.service.autostart      provisioning "open"   config "open"   same
  machine.collector.timezone               provisioning "UTC"   config "UTC"   same
  object_store.archive.access_policy       provisioning "in-account"   config "in-account-plus-read"   CONFLICT
5 items in the gap (in neither layer):
  network.measurement-network.egress_path             natural layer: resource
  security.collector.egress                           natural layer: resource
  dns_record.collector.internal                       natural layer: resource
  machine.collector.log_rotation                      natural layer: content
  data_store.measurement.parameter.connection_count   natural layer: content

Thirty-four items fell into four boxes: fourteen only in provisioning, nine only in config, six in both, five in neither. The two records define thirty-five items in total, but the number of items they cover is twenty-nine — the difference of six is the overlap, and a definition written a second time gains no new item.

Four of the six overlapping items carry the same value. These four look harmless, and the only condition under which they are harmless is that when one changes, the other is changed too; because the two are written in separate files, in separate formats, and often by separate people, this condition is not met automatically. The remaining two items are already conflicting: the resolver version is 3.4.0 in one record, 3.5.0 in the other; the archive’s access policy is narrow in one record, wide in the other.

Where the overlap comes from is also readable. Five of the six items sit on the machine resource, and all five have an image counterpart: because the provisioning layer opens the machine from an image, it indirectly defines the package version, timezone, and service state that image carries. The config layer defines the same values directly. The two records reach the same item by two different routes, and neither knows the other writes it; the overlap is not carelessness, it is the image carrying a definition that is invisible in the record.

The five items in the gap are the same measure as the last line of lesson 01, and here they appear together with their natural layers: three are naturally resource, two naturally content. These stayed unwritten not because no one wrote a record, but because they fell between the two records — each layer’s owner assumed the other one wrote it. The egress path, the security rule’s egress direction, the DNS record, log rotation, and the connection count: all five are items no one claimed.

Which Layer Wins

// boundary.mjs — application order, the post-reprovisioning drift window, and three boundary lines.
import { provisioning, config, inventory, split } from "./layer.mjs";

const g = split(inventory, provisioning, config);
const conflicting = g.overlap.filter((k) => provisioning.get(k) !== config.get(k));
console.log(`which layer wins on an overlapping item (last writer wins)`);
for (const [name, order] of [["provisioning then config (usual)", ["provisioning", "config"]],
  ["config then reprovisioning", ["config", "provisioning"]]]) {
  const final = (k) => (order[1] === "config" ? config : provisioning).get(k);
  console.log(`  ${name.padEnd(33)} conflicting ${conflicting.length} items -> ` +
    conflicting.map((k) => `${k.split(".").pop()}="${final(k)}"`).join("  "));
}

// Reprovisioning: the machine is reborn from the image, and the item the config layer wrote reverts to it.
const MACHINES = 14, AGENT_MIN = 30;
const machineItems = [...config.keys()].filter((k) => k.startsWith("machine."));
const reverted = machineItems.filter((k) => (provisioning.get(k) ?? "default") !== config.get(k));
console.log(`\ndrift window after reprovisioning (agent runs every ${AGENT_MIN} minutes)`);
console.log(`  items the config layer writes on the machine ${machineItems.length}` +
  `   at the wrong value after reprovisioning ${reverted.length}`);
console.log(`  ${MACHINES} machines -> ${reverted.length * MACHINES} items` +
  `   average ${AGENT_MIN / 2} min, worst ${AGENT_MIN} min` +
  `   -> ${reverted.length * MACHINES * AGENT_MIN / 2} item-minutes of drift`);

// Three boundary lines. Items baked into the image: package, service, and timezone.
const bakedIntoImage = (k) => /package\.|service\.|timezone/.test(k);
const boundary = {
  "C1 boundary at the resource shell": (k) => [provisioning.has(k), config.has(k)],
  "C2 boundary shifted to the image": (k) => [provisioning.has(k) || (config.has(k) && bakedIntoImage(k)),
    config.has(k) && !bakedIntoImage(k)],
  "C3 single layer (everything in provisioning)": (k) => [provisioning.has(k) || config.has(k), false],
};
const HEAVY = { "C1 boundary at the resource shell": 0, "C2 boundary shifted to the image": 6,
  "C3 single layer (everything in provisioning)": 15 };
console.log(`\nwhat happens depending on where the boundary is drawn (22 changes in a quarter: 7 resource, 15 content)`);
for (const [name, f] of Object.entries(boundary)) {
  const S = new Map(), A = new Map();
  for (const { k } of inventory) { const [s, a] = f(k); if (s) S.set(k, 1); if (a) A.set(k, 1); }
  const b = split(inventory, S, A);
  console.log(`  ${name.padEnd(46)} provisioning ${S.size}  config ${A.size}` +
    `  overlap ${b.overlap.length}  gap ${b.gap.length}` +
    `  applied twice ${b.overlap.length}  changes falling to the heavy path ${HEAVY[name]}`);
}

which layer wins on an overlapping item (last writer wins)
  provisioning then config (usual)  conflicting 2 items -> resolver="3.5.0"  access_policy="in-account-plus-read"
  config then reprovisioning        conflicting 2 items -> resolver="3.4.0"  access_policy="in-account"

drift window after reprovisioning (agent runs every 30 minutes)
  items the config layer writes on the machine 13   at the wrong value after reprovisioning 9
  14 machines -> 126 items   average 15 min, worst 30 min   -> 1890 item-minutes of drift

what happens depending on where the boundary is drawn (22 changes in a quarter: 7 resource, 15 content)
  C1 boundary at the resource shell              provisioning 20  config 15  overlap 6  gap 5  applied twice 6  changes falling to the heavy path 0
  C2 boundary shifted to the image               provisioning 23  config 9  overlap 3  gap 5  applied twice 3  changes falling to the heavy path 6
  C3 single layer (everything in provisioning)   provisioning 29  config 0  overlap 0  gap 5  applied twice 0  changes falling to the heavy path 15

The final value of the two conflicting items depends on the run order. In the usual order, the config layer runs last, so it wins; the resolver becomes 3.5.0, the access policy becomes wide. When a reprovisioning is triggered, the provisioning layer runs last, and the same two items revert to the opposite value. The winning layer is not a decision, it is a timing outcome — and what determines that timing is often an event nobody is watching: a machine dying, a scaling change, an image update.

The second table gives its duration. After reprovisioning, nine of the thirteen items the config layer writes on the machine revert to the value the image carries; because the agent runs every thirty minutes, they stay at the wrong value for fifteen minutes on average. Across fourteen machines that is 126 items and 1,890 item-minutes. This duration is not an outage, it is a drift window: the environment is running, taking measurements, but nine items are not at the value the record says.

When the Boundary Shifts

The last table compares the boundary drawn in three separate places. C1 keeps the boundary at the resource’s shell: each layer writes its own record, overlap is six, items applied twice are six, but no change falls onto the heavy path — all fifteen content changes finish with an agent run. C2 shifts the boundary to the image: package, service, and timezone items get baked into the image, the config layer drops from fifteen to nine, overlap falls from six to three — but six changes now require baking an image and rebuilding the machines. C3 removes the boundary entirely: twenty-nine items in a single layer, overlap zero, items applied twice zero — and all fifteen content changes fall onto the heavy path.

The provisioning record’s item count also climbs across the three drawings, from twenty to twenty-three to twenty-nine. Dropping to a single layer does not simplify the record; it piles the sum of the two records into one file and confronts everyone who reads that file with items that do not concern them.

The one number that does not change across all three drawings is the gap: five. Moving the boundary reduces or increases overlap, but it never brings into scope an item no layer claims. The gap has nothing to do with where the boundary is; it has to do with the part of the inventory that falls outside the record, and the only way to close it is to write an owner for the item.

The overlap decreasing is not free either. Between C1 and C3, as overlap falls from six to zero, the changes falling onto the heavy path climb from zero to fifteen. The two numbers move in opposite directions, and the decision is made between them: in a quarter, is an inconsistency six overlaps might miss more expensive, or is the duration of fifteen rebuilds more expensive. The question has no single right answer, but both answers are countable.

Summary

• When 34 items were split across two records, 14 came out only in provisioning, 9 only in config, 6 in both, 5 in neither; 35 definitions cover only 29 items.
• 4 of the 6 overlapping items carry the same value, and their being harmless depends on the two records being updated together; 2 are already conflicting.
• On a conflicting item, the winning layer is whichever ran last: config in the usual order, provisioning on reprovisioning; the winner is not a decision, it is a timing outcome.
• After reprovisioning, 9 items per machine revert to the image’s value and stay wrong for an average of 15 minutes until the agent runs; across 14 machines that is 1,890 item-minutes of drift.
• When the boundary was drawn in three separate places, overlap was 6, 3, and 0; changes falling onto the heavy path climbed to 0, 6, and 15; the gap stayed at 5 across all three drawings.

Next Step

This lesson’s last two numbers say the same thing from two places. After reprovisioning, nine items were not at the value the record said, and stayed that way for fifteen minutes; five items are always that way, because they fall inside no record’s scope. Both share the same name: the record says one thing, reality says another.

The next lesson closes this topic and measures that difference directly. A drift detector is written — the record is read, the actual state is read, the difference is worked out item by item — and four things are counted: the drift accumulated over a period, the detection delay by scan frequency, the source of the drift, and the cost of fixing it. The last is at least as important as the other three: undoing a drift sometimes breaks something that works.