Configuration Management Tools

The previous lesson priced a change before it was applied: a unit cost was tied to plan items, how far the estimate deviated was counted, the non-estimable item was named. But what got priced was the resource itself — how many machines, at what size, how many disks. Once a resource is created, the setting that goes into it is outside that record: which package version is installed, which file sits with which permission, which service starts at boot. These settings are still entered by hand.

Provisioning creates the resource; configuration management ties the resource’s inside to a record. There are two models, and both are named as a kind, not a product: in the agent-based model, a process running continuously on the machine pulls the record itself and applies it; in the agentless model, a runner at the center connects to the machines and pushes the setting. M22/K01 compared manual, scripted, and declarative provisioning; that comparison is not repeated here. The question here is the path that carries the record to the machine itself.

IV25. The fleet is the 50 machines running the collectors of the regional measurement network, the fictional software that gathers water-meter readings. IV26. The record carries 34 items; a machine can change 61 settings. IV27. All numbers come from our own generator; the seeds 20260731, 20260802, and 20260803 are visible, and the numbers are an order of magnitude, not a measurement. IV28. The agentless runner connects to 12 machines at a time; a connection takes 1 minute, applying 4; the agent polls every 30 minutes on its own offset. IV29. At a run, 8 percent of machines are cut off from the network and on 4 percent the agent process has stopped; the agent tries its own version daily, succeeding 94 percent of the time, and 35 percent of failures kill it. IV30. Over 30 days a machine sees 0.6 manual changes on average, the agentless model adds three record-change runs to the weekly schedule, the agent holds 46 MB of memory, and an item is 790 bytes in the record.

Same Setting Set, Two Paths

The same 34 items are applied to 50 machines under two models. In the agentless model, the fleet is split into rounds because the number of concurrent connections is capped; in the agent-based model there are no rounds, each agent pulls independently on its own offset.

// measurement-network/model.mjs — the same setting set rolled out under two models (model; a kind, not a product)
export const generator = (seed) => () =>
  (seed = (seed * 1103515245 + 12345) % 2147483648) / 2147483648;

export const ITEM = 34;    // configuration items in the record
export const WINDOW = 30;  // agent-based model: poll interval (min)
const LANE = 12, APPLY = 4, CONNECT = 1;  // agentless: concurrent lanes, apply, connect (min)

// Fleet: poll offset, current reachability, and agent status come from the generator.
export const fleet = (n, seed = 20260731) =>
  Array.from({ length: n }, ((r) => () => ({
    offset: Math.floor(r() * WINDOW), reachable: r() > 0.08, agentRunning: r() > 0.04,
  }))(generator(seed)));

const percentile = (d, p) => d.slice().sort((a, b) => a - b)[Math.floor((p / 100) * d.length)];

// Agentless: the center connects in rounds; a machine it cannot reach is skipped this run.
// Agent-based: each agent pulls on its own offset; there is no center queue, a stalled agent stays out.
export const run = (m, model) => {
  const agentless = model === "agentless";
  const target = m.filter((x) => (agentless ? x.reachable : x.agentRunning));
  const duration = agentless
    ? target.map((_, i) => (Math.floor(i / LANE) + 1) * (CONNECT + APPLY))
    : target.map((x) => x.offset + APPLY);
  return {
    model, round: agentless ? Math.ceil(target.length / LANE) : 1, reached: target.length,
    diverged: m.length - target.length, p50: percentile(duration, 50), p95: percentile(duration, 95),
    max: Math.max(...duration),
  };
};

if (import.meta.url === `file://${process.argv[1]}`) {
  const m = fleet(50);
  console.log(`50 machines, ${ITEM} items in the record, total in the record: ${50 * ITEM} lines\n`);
  console.log("model".padEnd(13) + "round".padEnd(7) + "p50".padEnd(8) + "p95".padEnd(8) +
    "max".padEnd(8) + "reached".padEnd(9) + "diverged".padEnd(10) + "lines held");
  for (const s of ["agentless", "agent-based"].map((x) => run(m, x)))
    console.log(s.model.padEnd(13) + String(s.round).padEnd(7) + `${s.p50} min`.padEnd(8) +
      `${s.p95} min`.padEnd(8) + `${s.max} min`.padEnd(8) + `${s.reached}/50`.padEnd(9) +
      `${s.diverged}/50`.padEnd(10) + `${s.reached * ITEM}/${m.length * ITEM}`);
  console.log("\nfull rollout as the fleet grows:");
  console.log("machines".padEnd(10) + "agentless rounds".padEnd(18) + "agentless max".padEnd(15) +
    "agent-based rounds".padEnd(20) + "agent-based max");
  for (const n of [50, 200, 800]) {
    const f = fleet(n), a = run(f, "agentless"), b = run(f, "agent-based");
    console.log(String(n).padEnd(10) + String(a.round).padEnd(18) + `${a.max} min`.padEnd(15) +
      String(b.round).padEnd(20) + `${b.max} min`);
  }
}

50 machines, 34 items in the record, total in the record: 1700 lines

model        round  p50     p95     max     reached  diverged  lines held
agentless    4      10 min  20 min  20 min  47/50    3/50      1598/1700
agent-based  1      21 min  32 min  33 min  45/50    5/50      1530/1700

full rollout as the fleet grows:
machines  agentless rounds  agentless max  agent-based rounds  agent-based max
50        4                 20 min         1                   33 min
200       16                80 min         1                   33 min
800       63                315 min        1                   33 min

At 50 machines, the agentless model is faster: it finishes in four rounds, in 20 minutes; the agent-based model takes 33 minutes, because the last agent to poll sits at the end of the interval. The second table upsets that ranking. In the agentless model, the round count grows in direct proportion to the fleet — at 800 machines, 63 rounds, 315 minutes. In the agent-based model, 33 minutes is constant: the parallelism lives at the machines, not at the center.

The last column is this course’s measure. The record claims 1,700 lines; after one run 1,598 hold in the agentless model, 1,530 in the agent-based model. The difference comes from machines that diverge from the record: 3 cut off from the network are skipped, 5 with a stopped process never look. The failure mode differs too. Agentless failure is seen from the center — the runner reports that it could not connect. Agent-based failure is silent: an agent that never looks produces no error, and the center does not notice unless it checks the last report-in.

How Long Drift Lives

Rollout is a one-time event. The real difference shows up once a manual change is made on a machine after applying. The agent-based model watches continuously; the agentless model is triggered, and drift accumulates in the interval between. This course’s iac-fundamentals topic already established the difference between continuous reconciliation and triggered apply; here that difference is counted in minutes.

// measurement-network/drift.mjs — how many minutes drift lives under each model (model)
import { generator, fleet, ITEM, WINDOW } from "./model.mjs";

const DAY = 1440, END = 30 * DAY;
const SETTING = 61;    // settings a machine can change; 34 of these are in the record
const RATE = 0.6;      // manual change per machine over 30 days (fictional)
// Agentless model is triggered: weekly schedule + three record-change runs (fictional days).
const TRIGGER = [0, 3.2, 7, 11.6, 14, 19.4, 21, 28].map((g) => Math.round(g * DAY));

const events = (m, seed = 20260802) => {
  const r = generator(seed);
  return m.flatMap((x) => (r() < RATE
    ? [{ machine: x, at: Math.floor(r() * END), inRecord: r() * SETTING < ITEM }] : []))
    .sort((a, b) => a.at - b.at);
};

// Agent-based: each agent compares against the record once every WINDOW minutes, on its own offset.
const agentBased = (o) => o.machine.agentRunning
  ? ((((o.machine.offset - o.at) % WINDOW) + WINDOW) % WINDOW) || WINDOW : null;

// Agentless: lives until the next trigger; if the machine is unreachable at that moment, one more trigger.
const agentless = (o) => {
  const k = TRIGGER.filter((t) => t > o.at);
  const t = o.machine.reachable ? k[0] : k[1];
  return t === undefined ? null : t - o.at;
};

const summarize = (list, lifespan) => {
  const onRecord = list.filter((o) => o.inRecord).map((o) => [o, lifespan(o)]);
  const closed = onRecord.filter(([, v]) => v !== null).map(([, v]) => v);
  // Drift that stays open lives until the end of the window; it enters the total that way.
  const open = onRecord.filter(([, v]) => v === null);
  const openDuration = open.reduce((t, [o]) => t + (END - o.at), 0);
  const closedDuration = closed.reduce((t, v) => t + v, 0);
  return [onRecord.length, closed.length, open.length, Math.round(closedDuration / closed.length),
    Math.max(...closed), closed.filter((v) => v <= 60).length,
    closedDuration + openDuration, openDuration / (closedDuration + openDuration)];
};

const list = events(fleet(50));
console.log(`30 days, 50 machines, ${list.length} manual changes; ` +
  `the record covers ${ITEM}/${SETTING} settings\n`);
console.log("model".padEnd(13) + "trigger".padEnd(11) + "seen".padEnd(10) +
  "closed".padEnd(10) + "open".padEnd(7) + "avg lifespan".padEnd(14) + "longest".padEnd(11) +
  "within 1h".padEnd(11) + "drift-hours");
for (const [ad, trigger, fn] of [["agent-based", `${DAY / WINDOW}/day`, agentBased],
  ["agentless", `${TRIGGER.length}/30 days`, agentless]]) {
  const [event, closed, open, avg, longest, within, total, share] = summarize(list, fn);
  console.log(ad.padEnd(13) + trigger.padEnd(11) + String(event).padEnd(10) +
    String(closed).padEnd(10) + String(open).padEnd(7) + `${avg} min`.padEnd(14) +
    `${(longest / 60).toFixed(1)} hr`.padEnd(11) + `${within}/${closed}`.padEnd(11) +
    (total / 60).toFixed(0) + ` (${(100 * share).toFixed(1)}% of what stayed open)`);
}
console.log(`\ndrift born in an out-of-record setting: ${list.filter((o) => !o.inRecord).length}` +
  ` events - invisible to both models`);
console.log(`settings left out of the record: ${SETTING - ITEM}/${SETTING} (per machine)`);

30 days, 50 machines, 27 manual changes; the record covers 34/61 settings

model        trigger    seen      closed    open   avg lifespan  longest    within 1h  drift-hours
agent-based  48/day     18        15        3      20 min        0.5 hr     15/15      1251 (99.6% of what stayed open)
agentless    8/30 days  18        16        2      3873 min      159.5 hr   0/16       1320 (21.7% of what stayed open)

drift born in an out-of-record setting: 9 events - invisible to both models
settings left out of the record: 27/61 (per machine)

There are two orders of magnitude of difference per event. In the agent-based model, a closed drift lives an average of 20 minutes, and all fifteen close within an hour. In the agentless model the average lifespan is 3,873 minutes — more than two and a half days — and none of the sixteen close within an hour; the longest drift stays up 159.5 hours, 6.6 days. The difference is the ratio between the poll interval and the trigger interval.

The last column takes this advantage back. Thirty-day total drift duration is 1,251 hours in the agent-based model, 1,320 in the agentless model — six percent apart. The reason is in the table: 99.6 percent of the agent-based total comes from three drifts that never close. On a machine whose agent has stopped, drift lives without limit because nothing is watching; the model’s speed is defeated by its own failure mode. In the agentless model the share that stays open is 21.7 percent — failure is visible to the center and gets fixed on the next trigger.

The last two lines write down what neither model covers. A machine can change 61 settings, and the record holds 34 of them; of the 27 changes that occur over thirty days, 9 are in out-of-record settings and are invisible to both models. How fast drift closes is a tooling question; which drift gets seen is a record coverage question, and choosing a tool does not change that.

The Model’s Own Cost

Both models add something to the machine: the agent-based model a process, the agentless model a connection point. The cost is counted separately.

// measurement-network/cost.mjs — the model's own cost: process, connection surface, identity (model)
import { generator, fleet, ITEM, WINDOW } from "./model.mjs";

const MEMORY = 46e6;   // memory the agent process holds on the machine (fictional)
const BYTES = 790;     // a configuration item's bytes in the record (fictional)
const N = fleet(50).length;
const pulls = (1440 / WINDOW) * N;          // agent-based: daily record pulls
const connections = Math.round((8 / 30) * N); // agentless: daily connections (8 runs in 30 days)
const mb = (b) => (b / 1e6).toFixed(1) + " MB";

const ROW = [
  ["process running on the machine", `${N}`, "0"],
  ["memory held on the machine", `${(N * MEMORY / 1e9).toFixed(2)} GB`, "0.00 GB"],
  ["listening endpoint on the machine", "0", `${N}`],
  ["listening endpoint at the center", "1", "0"],
  ["authorized identity", `${N} (open to a single machine)`, `1 (open to ${N} machines)`],
  ["if one identity falls", "1 machine", `${N} machines`],
  ["daily connections to the center", `${pulls}`, `${connections}`],
  ["daily traffic from the center", mb(pulls * ITEM * BYTES), mb(connections * ITEM * BYTES)],
];

// The agent's own version has to roll out too: one attempt per day, and failure can kill the agent.
const rollout = (seed = 20260803) => {
  const r = generator(seed), d = Array(N).fill("old"), trace = [];
  for (let g = 1; g <= 14; g++) {
    for (let i = 0; i < N; i++)
      if (d[i] === "old") d[i] = r() < 0.94 ? "new" : r() < 0.35 ? "dead" : "old";
    if ([1, 3, 7, 14].includes(g))
      trace.push([g, ...["new", "old", "dead"].map((v) => d.filter((x) => x === v).length)]);
  }
  return trace;
};

console.log("item".padEnd(36) + "agent-based".padEnd(32) + "agentless");
for (const [a, b, c] of ROW) console.log(a.padEnd(36) + b.padEnd(32) + c);
console.log(`\nas the agent's own version rolls out to ${N} machines:`);
console.log("day".padEnd(7) + "new version".padEnd(13) + "old version".padEnd(13) + "dead agent");
const trace = rollout();
for (const [g, newV, oldV, dead] of trace)
  console.log(`${g}.`.padEnd(7) + `${newV}/${N}`.padEnd(13) +
    `${oldV}/${N}`.padEnd(13) + `${dead}/${N}`);
console.log(`\n${trace.at(-1)[3]} machine with a dead agent cannot recover: it can only be reached ` +
  `the agentless way; at 12 versions a year that touches ${trace.at(-1)[3] * 12} machines`);

item                                agent-based                     agentless
process running on the machine      50                              0
memory held on the machine          2.30 GB                         0.00 GB
listening endpoint on the machine   0                               50
listening endpoint at the center    1                               0
authorized identity                 50 (open to a single machine)   1 (open to 50 machines)
if one identity falls               1 machine                       50 machines
daily connections to the center     2400                            13
daily traffic from the center       64.5 MB                         0.3 MB

as the agent's own version rolls out to 50 machines:
day    new version  old version  dead agent
1.     46/50        3/50         1/50
3.     49/50        0/50         1/50
7.     49/50        0/50         1/50
14.    49/50        0/50         1/50

1 machine with a dead agent cannot recover: it can only be reached the agentless way; at 12 versions a year that touches 12 machines

The first two rows are direct cost: in the agent-based model, the fleet holds 2.30 GB of memory without taking a single measurement reading. The third and fourth rows write the direction: in the agent-based model no machine listens, the agent connects outward and a single endpoint listens at the center; in the agentless model 50 machines listen for remote access, and the center never listens at all.

The fifth and sixth rows are the real distinction. In the agent-based model, every machine has its own identity, and that identity can only write to its own machine; if one falls, one machine falls. In the agentless model, a single identity is authorized on all fifty machines; if it falls, fifty machines fall at once. The number of identities is fifty times higher in the agent-based model, but a single identity’s cost is fifty times lower.

The last two rows count the load on the center: 2,400 pulls and 64.5 MB a day in the agent-based model, 13 connections and 0.3 MB in the agentless model. The 200-fold gap between them is the price of drift closing in 20 minutes — watching continuously means asking continuously.

The table below is the agent-based model’s unclosed loop. The agent itself is software too, and its new version has to roll out as well. Over fourteen days, 49 machines get through, the agent dies on 1 machine, and that machine cannot recover on its own: because the agent is not running, it cannot pull the new version. At twelve versions a year, 12 machines get touched, and the agent-based model requires keeping the agentless path around as an escape hatch.

Summary

• At fifty machines agentless rolls out in 4 rounds and 20 minutes, agent-based in one round and 33; at 800 machines the ranking flips — agentless takes 63 rounds and 315 minutes, agent-based is still 33.
• After one run, 1,598 of the record’s 1,700 lines hold in agentless, 1,530 in agent-based; the difference comes from 3 machines cut off from the network and 5 with a stopped agent, and the second failure mode is silent.
• A closed drift lives 20 minutes on average in agent-based, 3,873 in agentless; the longest agentless drift stays up 6.6 days.
• Total drift duration is nonetheless 1,251 hours against 1,320: 99.6 percent of the agent-based total comes from three drifts that never close.
• The 9 drifts born in the 27/61 settings the record does not cover are invisible to both; how fast drift closes is a tooling question, which drift gets seen is a coverage question.
• The cost: agent-based, 50 processes, 2.30 GB of memory, 2,400 daily pulls and an agent that can die; agentless, 50 listening endpoints and one identity authorized on fifty machines at once.

Next Step

Both models share the same assumption: a machine boots empty first, then gets configured. The entire rollout duration comes from that ordering — both 20 minutes and 33 minutes are spent bringing an already-booted machine up to the record. Partial failure comes from the same place too: the machine has started taking work before it matches the record.

The ordering can be reversed. Instead of carrying the setting to the machine after boot, it can be baked into the image the machine boots from; if the machine is born matching the record, the rollout duration is zero. M22/K02 measured the base image’s size and update overhead; that is not repeated. The next lesson counts one layer up: with boot-time configuration and a pre-baked image side by side, how long does boot take, how many outbound calls does it make, and how many images get rebaked for a security fix?