Drift

The previous four lessons said the same thing from four places. The two setups diverged on nine items; after partial failure, the record and the environment disagreed on three items; a rebuild wiped hotfixes; after reprovisioning, nine items stayed for fifteen minutes at a value the record never said. All of them share the same name.

Drift is the difference between what the record says and reality. This lesson measures it directly: a detector is written — the record is read, the actual state is read from disk, the difference is worked out item by item, and each difference’s source is separated. Then a thirty-day period is run, and four things are counted: the drift that accumulates, the detection delay by scan frequency, the drift that never closes because it falls outside the record’s scope, and the cost of fixing it. The last is at least as important as the other three: rolling back a drift sometimes breaks something that works.

One distinction is set up from the start. The reconciliation loop measured while Cluster Architecture was covered in M22/K03 ran continuously, and a difference closed the moment it was born; here, an apply is triggered, and drift accumulates between two triggers. Every number in this lesson is born in that gap.

The environment is again the staging environment of the regional measurement network, and it is fictional. The record and the actual state are genuinely written to disk and the detector genuinely runs; the thirty-day period is modeled with node, and the generator is self-written.

IF25. The record writes twenty items. The actual state is a directory tree and twenty-five files; the detector reads it not from the record but from disk. IF26. The provider’s documented defaults are in a separate file. If an item not in the record is on that list, its source is a provider default; if not, it is an out-of-record resource. IF27. Over the thirty-day period, events are produced by the generator (seed 4471903, the generator is self-written); the daily event count is zero, one, or two. IF28. An event’s source is a manual change, an out-of-record resource, or a provider default. Only a manual change falls within the record’s scope; the others are not closed by an apply. IF29. Apply is triggered: on days 1, 12, and 25. A scan is separate from apply — it detects, it does not close. IF30. Some of the manual changes are live fixes made to close an incident; when rolled back, the problem they closed comes back.

The Record and Reality

#!/usr/bin/env bash
# Record and actual environment: the record is one file, the actual state is a directory tree. Both are written to disk.
set -e
rm -rf actual

cat > record.infra <<'SON'
resource network "measurement-network" { block = "10.40.0.0/16"  subnet_count = 3  private = "true" }
resource machine "collector" { count = 3  size = "medium"  disk_gb = 40  image = "base-7" }
resource machine "verifier" { count = 2  size = "small"  disk_gb = 20  image = "base-7" }
resource queue "reading" { partitions = 6  retention_hours = 72  dead_letter = "open" }
resource data_store "measurement" { size_gb = 200  backup_days = 14  version = "5.2" }
resource object_store "archive" { class = "sparse"  versioning = "open"  lifecycle_days = 365 }
SON

# The provider's documented defaults: fields the record does not write, that arise on their own.
cat > defaults.txt <<'SON'
network.measurement-network.egress_path
queue.reading.encryption
data_store.measurement.maintenance_window
object_store.archive.access_rule
SON

write() { mkdir -p "actual/$(dirname "$1")"; printf '%s\n' "$2" > "actual/$1"; }
# Fourteen items that match the record
write network/measurement-network/block 10.40.0.0/16;    write network/measurement-network/private true
write machine/collector/size medium;                     write machine/collector/image base-7
write machine/verifier/count 2;                           write machine/verifier/size small
write machine/verifier/disk_gb 20;                          write machine/verifier/image base-7
write queue/reading/partitions 6;                            write queue/reading/dead_letter open
write data_store/measurement/size_gb 200;                     write data_store/measurement/version 5.2
write object_store/archive/versioning open;                    write object_store/archive/lifecycle_days 365
# Five items changed by hand (different from the record's value)
write network/measurement-network/subnet_count 4;               write machine/collector/count 5
write machine/collector/disk_gb 80;                                write queue/reading/retention_hours 24
write data_store/measurement/backup_days 7
# object_store/archive/class deleted: in the record, not in reality
# Provider default (three items) and out-of-record resource (three items)
write network/measurement-network/egress_path open;                 write queue/reading/encryption provider-key
write data_store/measurement/maintenance_window sunday-03:00
write security/temporary/ingress 0.0.0.0-8443;                       write dns_record/collector/internal collector.internal
write machine/diagnostics/count 1
printf 'record %s lines   actual %s files\n' "$(wc -l < record.infra | tr -d ' ')" \
  "$(find actual -type f | wc -l | tr -d ' ')"

record 6 lines   actual 25 files

The Difference, Item by Item

// drift.mjs — reads the record, reads the actual state from disk, works out the difference item by item. Fixes nothing.
import { readFileSync, readdirSync, statSync } from "node:fs";

const BLOCK = /^resource\s+(\w+)\s+"([\w-]+)"\s*\{(.*)\}$/;
const record = new Map();
for (const s of readFileSync("record.infra", "utf8").split("\n")) {
  const m = BLOCK.exec(s.trim());
  if (m) for (const x of m[3].matchAll(/(\w+)\s*=\s*("[^"]*"|[\w./-]+)/g))
    record.set(`${m[1]}.${m[2]}.${x[1]}`, x[2].replaceAll('"', ""));
}
const providerDefault = new Set(readFileSync("defaults.txt", "utf8").trim().split("\n"));

const actual = new Map();                       // actual state is read from disk, not from the record
const walk = (dir, prefix = []) => {
  for (const name of readdirSync(dir).sort()) {
    const path = `${dir}/${name}`;
    if (statSync(path).isDirectory()) walk(path, [...prefix, name]);
    else actual.set([...prefix, name].join("."), readFileSync(path, "utf8").trim());
  }
};
walk("actual");

const all = [...new Set([...record.keys(), ...actual.keys()])].sort();
const drift = all.filter((k) => record.get(k) !== actual.get(k)).map((k) => ({ k,
  source: !actual.has(k) ? "missing item" : !record.has(k)
    ? (providerDefault.has(k) ? "provider default" : "out-of-record resource") : "manual change" }));
const countBy = (t) => drift.filter((f) => f.source === t).length;

console.log(`record ${record.size} items   actual ${actual.size} items   union ${all.length}`);
console.log(`drift ${drift.length} items` +
  `   manual ${countBy("manual change")}   missing ${countBy("missing item")}` +
  `   out-of-record ${countBy("out-of-record resource")}   default ${countBy("provider default")}`);
for (const f of drift)
  console.log(`  ${f.source.padEnd(22)} ${f.k.padEnd(43)} record "${record.get(f.k) ?? "-"}"` +
    `  actual "${actual.get(f.k) ?? "-"}"`);
const inScope = drift.filter((f) => record.has(f.k)).length;
console.log(`drift in the record's scope, i.e. closable by an apply ${inScope}` +
  `   out of scope ${drift.length - inScope}`);

record 20 items   actual 25 items   union 26
drift 12 items   manual 5   missing 1   out-of-record 3   default 3
  manual change          data_store.measurement.backup_days          record "14"  actual "7"
  provider default       data_store.measurement.maintenance_window   record "-"  actual "sunday-03:00"
  out-of-record resource dns_record.collector.internal               record "-"  actual "collector.internal"
  manual change          machine.collector.count                     record "3"  actual "5"
  manual change          machine.collector.disk_gb                   record "40"  actual "80"
  out-of-record resource machine.diagnostics.count                   record "-"  actual "1"
  provider default       network.measurement-network.egress_path     record "-"  actual "open"
  manual change          network.measurement-network.subnet_count    record "3"  actual "4"
  missing item           object_store.archive.class                  record "sparse"  actual "-"
  provider default       queue.reading.encryption                    record "-"  actual "provider-key"
  manual change          queue.reading.retention_hours               record "72"  actual "24"
  out-of-record resource security.temporary.ingress                  record "-"  actual "0.0.0.0-8443"
drift in the record's scope, i.e. closable by an apply 6   out of scope 6

Twelve items drifted, and the last line shows where the measure actually sits: six of the twelve fall within the record’s scope. For the remaining six items, running an apply changes nothing, because the record says nothing about them — three are fields the provider opens on its own, three are resources created without a record ever being written. The only reason these are visible at all is that the detector is two-directional: it does not just scan the items in the record, it walks the whole actual state and writes down the ones with no counterpart in the record too.

The source distinction is not a labeling exercise, it is an action distinction. A manual change is closed by an apply. A missing item comes back with an apply. An out-of-record resource is only closed by being brought into the record or being deleted; a provider default neither closes nor should it — all that needs to be known is that the record is silent about it. Four separate kinds of difference, four separate decisions.

Over a Period

// period.mjs — 30-day drift accumulation, detection delay, and the cost of fixing it. Model.
const SEED = 4471903, DAYS = 30, APPLY = [1, 12, 25];   // apply is triggered, it does not run continuously
const generator = (t) => () => (t = (t * 1103515245 + 12345) % 2147483648) / 2147483648;
const r = generator(SEED);

const events = [];
for (let d = 1; d <= DAYS; d++)
  for (let n = (r() < 0.55) + (r() < 0.18); n > 0; n--) {
    const p = r();
    const source = p < 0.45 ? "manual" : p < 0.72 ? "out-of-record" : "default";
    events.push({ d, source, inScope: source === "manual", live: source === "manual" && r() < 0.35 });
  }
const countBy = (t) => events.filter((o) => o.source === t).length;
const inScope = events.filter((o) => o.inScope);
console.log(`seed ${SEED}   days ${DAYS}   events ${events.length}   apply days` +
  ` ${APPLY.join(", ")}`);
console.log(`source of the drift   manual ${countBy("manual")}   out-of-record ${countBy("out-of-record")}` +
  `   provider default ${countBy("default")}   in scope ${inScope.length}`);

// Two regimes: apply is triggered (three days) and continuous reconciliation (every day). Both close only what is in scope.
const regime = (days) => {
  const total = [], scope = [];
  let closed = 0;
  for (let d = 1; d <= DAYS; d++) {
    if (days.includes(d)) closed = inScope.filter((o) => o.d <= d).length;
    scope.push(inScope.filter((o) => o.d <= d).length - closed);
    total.push(events.filter((o) => o.d <= d).length - closed);
  }
  const area = (x) => x.reduce((a, b) => a + b, 0);
  return { total, scope, areaT: area(total), areaS: area(scope) };
};
const A = regime(APPLY), B = regime([...Array(DAYS)].map((_, i) => i + 1));
console.log(`\nopen drift (end of day, ${DAYS} days)`);
console.log(`  triggered apply     total   ${A.total.join(" ")}`);
console.log(`  triggered apply     scope   ${A.scope.join(" ")}`);
console.log(`  continuous reconcile total  ${B.total.join(" ")}`);
console.log(`  day-item area   in-scope drift: triggered ${A.areaS}   continuous ${B.areaS}`);
console.log(`  day-item area   all drift:      triggered ${A.areaT}   continuous ${B.areaT}` +
  `   difference ${A.areaT - B.areaT}`);

console.log(`\ndetection delay (only the ${inScope.length} in-scope drift; a scan detects, it does not close)`);
for (const n of [1, 3, 7, 30]) {
  const delay = inScope
    .map((o) => { for (let d = o.d; d <= DAYS; d++) if (d % n === 0) return d - o.d; return null; })
    .filter((x) => x !== null);
  console.log(`  scan every ${String(n).padStart(2)} days   average` +
    ` ${(delay.reduce((a, b) => a + b, 0) / delay.length).toFixed(1)} days` +
    `   worst ${Math.max(...delay)} days   detected ${delay.length}/${inScope.length}`);
}

const rolledBack = inScope.filter((o) => o.d <= APPLY[APPLY.length - 1]);
const broken = rolledBack.filter((o) => o.live);
console.log(`\nthe cost of fixing it`);
console.log(`  drift rolled back ${rolledBack.length}   of these a live fix` +
  ` ${broken.length}   rolling it back breaks something that works`);
console.log(`  closed for good ${rolledBack.length - broken.length}` +
  `   came back ${broken.length} (${(100 * broken.length / rolledBack.length).toFixed(0)}%)`);
console.log(`  out-of-scope ${events.length - inScope.length} drift was closed by no apply;` +
  ` all of it is still standing at the end of the period`);

seed 4471903   days 30   events 21   apply days 1, 12, 25
source of the drift   manual 9   out-of-record 6   provider default 6   in scope 9

open drift (end of day, 30 days)
  triggered apply     total   1 1 2 2 2 3 3 3 4 5 6 3 5 6 7 8 8 8 9 9 10 11 12 13 9 10 10 11 11 13
  triggered apply     scope   0 0 1 1 1 1 1 1 2 3 3 0 1 2 2 2 2 2 2 2 2 2 3 4 0 1 1 1 1 1
  continuous reconcile total  1 1 1 1 1 2 2 2 2 2 3 3 4 4 5 6 6 6 7 7 8 9 9 9 9 9 9 10 10 12
  day-item area   in-scope drift: triggered 45   continuous 0
  day-item area   all drift:      triggered 205   continuous 160   difference 45

detection delay (only the 9 in-scope drift; a scan detects, it does not close)
  scan every  1 days   average 0.0 days   worst 0 days   detected 9/9
  scan every  3 days   average 1.0 days   worst 2 days   detected 9/9
  scan every  7 days   average 3.1 days   worst 5 days   detected 9/9
  scan every 30 days   average 13.7 days   worst 27 days   detected 9/9

the cost of fixing it
  drift rolled back 8   of these a live fix 2   rolling it back breaks something that works
  closed for good 6   came back 2 (25%)
  out-of-scope 12 drift was closed by no apply; all of it is still standing at the end of the period

Twenty-one drift events were born over thirty days, and nine of them fell within the record’s scope. The difference between the two series reduces what a triggered apply means to a single line: in-scope drift stayed open for forty-five day-items in the triggered regime, zero in continuous reconciliation. The three trigger days show up clearly in the series — on days 1, 12, and 25, the scope line drops to zero, then climbs again. Drift does not disappear, it is reborn between every trigger; what accumulates is not drift itself, it is unclosed drift.

The total line, on the other hand, keeps climbing in both regimes and stops at thirteen and twelve on day thirty. The difference between them is only forty-five day-items, and that comes from the nine in-scope items. The remaining twelve drift events closed under neither regime: running apply continuously does not make them visible, because the reason they stay invisible is not frequency, it is scope. This shows how far the thing seen as five items in the topic’s first lesson has grown over the course of a period.

Detection Delay and the Cost of Fixing It

The scan table measures a separate decision. A scan is independent of apply: it makes a drift visible without closing it. A daily scan zeroes out the delay, a scan every three days raises it to an average of one day, a weekly scan to 3.1 days, a monthly scan to 13.7 days — and on a monthly scan the worst delay is twenty-seven days. Twenty-seven days is how long an item runs at a value the record never said, with nobody knowing. Choosing a scan frequency is choosing this number.

The last table carries this lesson’s most counterintuitive number. By day twenty-five, eight drift events were rolled back, and two were live fixes: changes made by hand to close an incident, that genuinely resolved the problem. When rolled back, the problem they closed came back, and both were redone on later days. The drift rollback closed for good is six, the drift that came back is two — twenty-five percent.

These two items show that drift cannot be read in only one direction. Drift does not mean “the record is right, reality is wrong”; it only means the two have diverged. Divergence has two interpretations, and which one is correct cannot be read from the difference itself: either the record should be updated and the fix written back into it, or reality should be rolled back. An automated apply always picks the second, because it has no other option. The cost of the record staying silent about an item was invisibility; the cost of the record staying stale about an item is breaking something that works.

Summary

• The detector compared 26 items and found 12 drifted: 5 manual changes, 1 missing item, 3 out-of-record resources, 3 provider defaults; only 6 of these can be closed by an apply.
• 21 drift events were born over thirty days; 9 fell within the record’s scope, 12 were still open at the end of the period and closed under no regime.
• Triggered apply left in-scope drift open for 45 day-items, continuous reconciliation 0; the difference comes from frequency, the 12 items that never close come from scope.
• Scan frequency determines the delay directly: 0.0 days daily, 3.1 days weekly, 13.7 days monthly on average, and 27 days worst case.
• 2 of the 8 rolled-back drift events were live fixes, and rolling them back broke something that works; the rollback closed 6 for good, 2 came back.

Next Step

At this point, drift is something countable. The record’s scope is known, the item outside that scope can be written down by name, the source of the difference can be split into four, the detection delay can be tied to scan frequency, and fixing it has been measured to carry a cost. The whole measurement leans on a single assumption: that a readable record exists somewhere.

The record itself was never addressed. Across five lessons, the record was always found ready-made — a six-line file, a list counting its items by their values. How it is written, what structures it is built from, how one resource attaches to another, how the same definition gets written for three environments without repeating itself, where a value comes from, and what it will show before a change is applied — none of this was asked. The next topic starts by writing the record itself.