Data Sources

The previous lesson assumed the record built every resource on the target. In a real environment that is not true: the network backbone, the base image, identity definitions, certificates, and regional quotas are mostly in other teams’ hands. The record does not create them, delete them, or update them — it only reads them. The structure that reads is called a data source, and its cost accumulates in three separate places: the number and duration of reads made on every plan run, the recomputation triggered when a value it reads changes, and the drift scan never seeing these items at all.

The measurement runs on the west environment of the regional measurement network and is fictional; the infrastructure outside the record, the data sources, the read durations, and the dependency propagation are a model built with node.

DP13. The record manages eight resources; five data sources read items outside the record, and other teams own these items. DP14. When a data source is read, it brings back all of its fields; individual fields cannot be selected. DP15. Every plan run reads every data source once, and the result is not cached. DP16. The pipeline runs six times a day, and a manual plan runs four. DP17. When a read value changes, every resource that takes it as an input is recomputed; if the value is in that resource’s forcing list, the resource is recreated, and the recreation propagates downstream. DP18. The drift scan compares only the resources the record manages against their counterparts on the target.

Infrastructure Outside the Record

// external.mjs — fictional infrastructure standing outside the record, and the data sources that read it (model).
export const EXTERNAL = {                                // items other teams manage
  network: { subnet: "subnet-7", mtu: 1500, gateway: "gateway-a" },
  image: { version: "base-2", age_days: 34 },
  identity: { permissions: 12 },
  certificate: { expiry_days: 41 },
  quota: { machine_cap: 24, queue_cap: 16 },
};
export const READ_MS = { network: 120, image: 340, identity: 90, certificate: 210, quota: 60 };
export const OWNER = { network: "network team", image: "image team", identity: "security team",
  certificate: "security team", quota: "cloud ops" };

export function makeReader() {                      // every read is counted, its time totaled
  const counts = {}, elapsed = { ms: 0 };
  return { counts, elapsed,
    read(name) { counts[name] = (counts[name] || 0) + 1; elapsed.ms += READ_MS[name]; return { ...EXTERNAL[name] }; },
    total: () => Object.values(counts).reduce((a, b) => a + b, 0) };
}

// record.mjs — resources the record manages; the input list says what they read, forcing
// says which input's change forces the resource to be recreated.
export const RESOURCE = {
  "machine.collector": { input: ["network.subnet", "image.version", "quota.machine_cap"],
    forcing: ["network.subnet", "image.version"] },
  "machine.verifier": { input: ["network.subnet", "image.version"], forcing: ["network.subnet", "image.version"] },
  "queue.meter": { input: ["quota.queue_cap", "network.subnet"], forcing: ["network.subnet"] },
  "queue.alert": { input: ["quota.queue_cap"], forcing: [] },
  "data_store.metrics": { input: ["network.subnet", "machine.collector"], forcing: ["network.subnet"] },
  "network_rule.collector": { input: ["network.gateway", "machine.collector"],
    forcing: ["network.gateway", "machine.collector"] },
  "endpoint.metrics": { input: ["certificate.expiry_days", "data_store.metrics"], forcing: ["data_store.metrics"] },
  "identity_binding.collector": { input: ["identity.permissions", "machine.collector"], forcing: ["machine.collector"] },
};

export function propagate(root) {                        // who gets recomputed when a value changes
  const recomputed = new Set(), recreated = new Set(), worklist = [[root, true]];
  while (worklist.length) {
    const [x, xNew] = worklist.shift();
    for (const [k, r] of Object.entries(RESOURCE)) {
      if (!r.input.includes(x) || recomputed.has(k)) continue;
      recomputed.add(k);
      const kNew = xNew && r.forcing.includes(x);
      if (kNew) recreated.add(k);
      worklist.push([k, kNew]);
    }
  }
  return { recomputed, recreated };
}

The two files define two separate worlds. The first holds the items the record does not manage and each one’s owner; the second holds the resources the record manages and which external value each one reads. The only link between them runs in the direction of reading: the record takes values from the outside world, it never writes a value to it. This one-way direction is the source of every measure in this lesson.

The Cost and Propagation of Reading

// measure.mjs — the cost of reading, how an external value propagates, and the drift scan's coverage.
import { EXTERNAL, READ_MS, OWNER, makeReader } from "./external.mjs";
import { RESOURCE, propagate } from "./record.mjs";

const externalFields = new Set(Object.entries(EXTERNAL).flatMap(([a, v]) => Object.keys(v).map((f) => `${a}.${f}`)));
const used = new Set(Object.values(RESOURCE).flatMap((r) => r.input).filter((g) => externalFields.has(g)));
const print = (...s) => console.log(s.map((x, i) => String(x).padEnd(i ? 20 : 26)).join("").trimEnd());

console.log(`record manages ${Object.keys(RESOURCE).length} resources; ` +
  `${Object.keys(EXTERNAL).length} data sources bring in ${externalFields.size} external fields, ` +
  `${used.size} of them actually used`);
print("data source", "fields", "used", "read ms", "blocked resources", "owner");
for (const [a, v] of Object.entries(EXTERNAL)) {
  const fields = Object.keys(v).filter((f) => used.has(`${a}.${f}`));
  const blocked = new Set(fields.flatMap((f) => [...propagate(`${a}.${f}`).recomputed]));
  print(a, Object.keys(v).length, fields.length, READ_MS[a], blocked.size, OWNER[a]);
}

const PLANS_PER_DAY = 10;                             // 6 pipeline runs a day + 4 manual plans
const reader = makeReader();
for (let i = 0; i < PLANS_PER_DAY; i++) for (const a of Object.keys(EXTERNAL)) reader.read(a);
console.log(`\n${PLANS_PER_DAY} plans a day: ${reader.total()} external reads, ${(reader.elapsed.ms / 1000).toFixed(1)} s; ` +
  `${Object.keys(EXTERNAL).length} external + ${Object.keys(RESOURCE).length} resource reads per plan`);
console.log(`${reader.total() * 365} external reads a year, ${((reader.elapsed.ms * 365) / 60000).toFixed(0)} minutes`);
const onceMs = Object.values(READ_MS).reduce((a, b) => a + b, 0);
console.log(`cached once a day: ${Object.keys(EXTERNAL).length} reads, ${(onceMs / 1000).toFixed(2)} s; ` +
  `expected time for an external change to surface is ${(24 / PLANS_PER_DAY).toFixed(1)} hours instead of 24`);

console.log(`\n${"changed external value".padEnd(27)}${"reading".padStart(4)}${"recomputed".padStart(18)}${"recreated".padStart(20)}`);
for (const field of [...used].sort()) {
  const { recomputed, recreated } = propagate(field);
  const direct = Object.values(RESOURCE).filter((r) => r.input.includes(field)).length;
  console.log(`  ${field.padEnd(25)}${String(direct).padStart(4)}` +
    `${String(recomputed.size).padStart(18)}${String(recreated.size).padStart(20)}`);
}

const DRIFTED = ["network.subnet", "quota.machine_cap"];     // someone changed the external item
const impact = DRIFTED.reduce((s, a) => { const { recomputed, recreated } = propagate(a);
  recomputed.forEach((x) => s.recomputed.add(x)); recreated.forEach((x) => s.recreated.add(x)); return s; },
  { recomputed: new Set(), recreated: new Set() });
const items = Object.keys(RESOURCE).length + Object.keys(EXTERNAL).length;
console.log(`\ndrift scan compares ${Object.keys(RESOURCE).length}/${items} items ` +
  `(coverage ${Math.round((Object.keys(RESOURCE).length / items) * 100)}%); ` +
  `${Object.keys(EXTERNAL).length} data sources are never compared`);
console.log(`${DRIFTED.length} external fields changed -> the scan reports "no drift"; ` +
  `the next plan recomputes ${impact.recomputed.size} resources, recreates ${impact.recreated.size}`);

record manages 8 resources; 5 data sources bring in 9 external fields, 7 of them actually used
data source               fields              used                read ms             blocked resources   owner
network                   3                   2                   120                 7                   network team
image                     2                   1                   340                 6                   image team
identity                  1                   1                   90                  1                   security team
certificate               1                   1                   210                 1                   security team
quota                     2                   2                   60                  7                   cloud ops

10 plans a day: 50 external reads, 8.2 s; 5 external + 8 resource reads per plan
18250 external reads a year, 50 minutes
cached once a day: 5 reads, 0.82 s; expected time for an external change to surface is 2.4 hours instead of 24

changed external value     reading        recomputed           recreated
  certificate.expiry_days     1                 1                   0
  identity.permissions        1                 1                   0
  image.version               2                 6                   4
  network.gateway             1                 1                   1
  network.subnet              4                 7                   7
  quota.machine_cap           1                 5                   0
  quota.queue_cap             2                 2                   0

drift scan compares 8/13 items (coverage 62%); 5 data sources are never compared
2 external fields changed -> the scan reports "no drift"; the next plan recomputes 7 resources, recreates 7

The first table gives the width of the connection. Five data sources bring back nine external fields in total, but the record actually uses seven of them: the network’s maximum packet size and the image’s age get read but never feed into a resource. This comes from the data source’s inability to select fields — reading happens per item, not per field. Read times are not equal either: the image source takes three hundred forty milliseconds, the quota source sixty; the most expensive source is the one that brings back the least-used field.

The second measure is repetition frequency. Ten plan runs a day come to fifty external reads and 8.2 seconds; eighteen thousand two hundred fifty reads and fifty minutes a year. This duration is small on its own, but it hides two things. First, every plan also reads the eight managed resources at the same time; the cost of producing a plan grows not with the resource count but with the resource count plus the data source count. Second, cutting the cost of reading costs freshness: cached once a day, reads drop from fifty to five and the duration from 8.2 seconds to eight hundred milliseconds, but the expected time for an external change to be noticed rises from two and a half hours to twenty-four. Read frequency is directly a freshness measure, and lowering it lengthens drift’s invisibility window.

The third table gives propagation, and this lesson’s harshest number is here. Four resources read the subnet identifier directly, but when that value changes, seven resources are recomputed and all seven are recreated — nearly the entire eight-resource record. The reason for the propagation is a chain: when the subnet changes, the collector machines are recreated; when their identity changes, the network rule and the identity binding are recreated; when the data store is recreated, the endpoint is recreated too. The image version recomputes six resources and recreates four. The quota cap, by contrast, recomputes five resources but recreates none of them; the certificate and the identity permissions each stay confined to one resource. The same table shows whether a read value is cheap or expensive: what decides it is not how many resources read it, but whether the field is on any of their forcing lists.

The fourth column gives outage risk. If the network or quota source becomes unreadable, seven resources each cannot be planned; six for the image source, one each for identity and certificate. The record depends on the availability of five items it does not manage, and it does not own that dependency. The owner column names this directly: network team, image team, security team, cloud ops. The team authorized to change the record and the team authorized to change the value the record reads are not the same team.

Where the Scan Does not Look

The last two lines count the record’s scope. The drift scan compares eight of thirteen items; coverage is sixty-two percent. The five items it does not compare are the ones the record reads but does not manage, and the reason they are not compared is simple: the scan compares what the record says against reality, and the record says nothing at all about these five items. If there is no desired value in the record, there is no way to say reality has drifted from it either.

The consequence of this is in the last line. The network team changes the subnet identifier, and cloud ops changes the machine cap. The drift scan runs and reports no drift; all eight resources it manages match the record, because none of them has changed. At the same moment, the next plan run is about to recompute seven resources and recreate all seven of them. Two sentences are true for the same environment at the same time: there is no drift, and seven items of infrastructure are about to change. The difference is that drift is defined within the record’s scope — a change the record does not cover is not drift, it is a surprise.

The practical payoff of this measure is knowing the number of external items and their propagation radius. Two of the seven used external fields are on their own capable of recreating seven resources; three trigger no recreation at all. Every new external value that gets read lowers the scan’s coverage and grows the surprise surface; every external value that does not get read has to be copied into the record by hand instead, and this time it is the copy itself that drifts.

Read or Copy

// copy.mjs — reading an external value versus copying it into the record: a fictional 90-day calendar (model).
import { EXTERNAL } from "./external.mjs";
import { RESOURCE, propagate } from "./record.mjs";

const CALENDAR = [                                    // who changed which external item, and when
  [12, "image.version"], [30, "quota.machine_cap"], [40, "network.subnet"],
  [55, "image.version"], [70, "quota.machine_cap"], [78, "network.gateway"],
];
const REFRESH = [30, 60, 90], DAYS = 90;            // days for a manual review of the copy
const externalFields = new Set(Object.entries(EXTERNAL).flatMap(([a, v]) => Object.keys(v).map((f) => `${a}.${f}`)));
const copied = new Set(Object.values(RESOURCE).flatMap((r) => r.input).filter((g) => externalFields.has(g)));

let readRecreations = 0, copyRecreations = 0, mismatchDays = 0, delay = 0;
for (const [day, field] of CALENDAR) {
  const { recomputed, recreated } = propagate(field), t = REFRESH.find((g) => g >= day) ?? DAYS;
  readRecreations += recreated.size; copyRecreations += recreated.size;
  delay += t - day; mismatchDays += (t - day) * recomputed.size;
}
console.log(`${CALENDAR.length} external changes in ${DAYS} days; fields to copy ${copied.size}`);
console.log(`by reading   : 0 days delay, ${readRecreations} recreations, 0 resource-days record-reality mismatch`);
console.log(`by copying   : ${(delay / CALENDAR.length).toFixed(1)} days average delay, ` +
  `${copyRecreations} recreations, ${mismatchDays} resource-days mismatched`);
console.log(`in both approaches the drift scan compares ${Object.keys(RESOURCE).length} items; ` +
  `${Object.keys(EXTERNAL).length} external items stay outside it either way`);

6 external changes in 90 days; fields to copy 7
by reading   : 0 days delay, 16 recreations, 0 resource-days record-reality mismatch
by copying   : 12.5 days average delay, 16 recreations, 390 resource-days mismatched
in both approaches the drift scan compares 8 items; 5 external items stay outside it either way

The alternative to reading is writing external values into the record by hand. The fictional calendar holds six external changes across ninety days, and both approaches produce the same sixteen recreations in the end — if the outside world changed, the infrastructure will change sooner or later too. The difference is in timing. A record that reads picks up the change on the next plan run; a record that copies depends on a monthly review and lags twelve and a half days behind on average. The cost of this delay is three hundred ninety resource-days: the total of how many resources were affected for how long the record misrepresented reality.

The visible gain of copying is zeroing out the read load; its invisible cost is keeping a frozen copy of seven fields in the record. This copy does enter the drift scan, but what the scan compares is the record’s own copy against the resource built from it — the external item’s real value is never compared in either approach. Copying does not widen the scan’s coverage, it only renames the drift: a surprise in the record that reads, a stale record in the one that copies.

Summary

• The record managed 8 resources; 5 data sources brought back 9 external fields, and only 7 of them were used — 2 fields were read on every plan run but never fed into a resource.
• 10 plan runs a day took 50 external reads and 8.2 seconds; 18,250 reads and 50 minutes a year. Caching once a day cuts reads to 5 and raises the invisibility window from 2.4 hours to 24.
• When the subnet identifier changed, 7 of 8 resources were recomputed and 7 were recreated; the image version recomputed 6 resources and recreated 4. The quota cap recomputed 5 resources but recreated none: what decides it is not the number of readers but whether the field is on the forcing list.
• If the network or quota source cannot be read, 7 resources each cannot be planned; the record depends on the availability of 5 items owned by other teams.
• The drift scan compared 8 of 13 items (62% coverage). When 2 external fields changed, the scan said “no drift” while the next plan was, at the same moment, about to recreate 7 resources.
• 6 external changes across 90 days produced 16 recreations in both approaches; the copying record lagged 12.5 days behind on average and misrepresented reality for 390 resource-days.

Next Step

Up to this point, the record described a single environment, and the values it read came from outside. When the same record has to run across three environments, the question changes: which value should vary by environment, which should stay the same in every one, and who writes that distinction. The next lesson measures parameterization: when the same definition is applied to three environments, how many values actually vary by environment, how many stand able to vary when they should not have to, how the odds of giving a wrong value grow as the input surface grows, and how many dependencies outputs create in other definitions.