Providers and Resources

The previous lesson tied the record to a language, but the target system facing the record was a five-line function: it took a value, wrote it, or rejected it. In reality a resource is created, read, updated, and deleted; the layer that knows these four operations, translates the record’s field names into the target system’s counterparts, and knows which field can be changed in place is called the provider. This lesson models the provider as an interface and counts three things: the gap between the field the record writes and the field that actually stands on the target, how well a change’s plan predicts the apply, and how much shifts while the record stays completely unchanged when the provider version changes.

The measurement runs on the west environment of the regional measurement network and is fictional; the provider, the resource type catalog, and the target store are a model built with node. No real product or target system is called.

DP7. The provider defines five resource types: machine, queue, data store, network, object store. DP8. Every field has three properties: its type, whether it can be updated in place, and its default. A field with no default is required. DP9. If a field cannot be updated in place, the resource is deleted and recreated when that field changes, and this carries a fixed downtime in minutes per type. DP10. The plan phase knows only the in-place-updatability flag in the catalog. DP11. The target system refuses to grow a numeric field in place if it is grown to more than double; this rule is not written in the catalog. DP12. The provider fills fields not written in the record with its own default, and the value it fills in appears nowhere in the record.

The Provider Interface and Resource Types

// provider.mjs — fictional provider: resource type catalog, create/read/update/delete interface (model).
const A = (type, inPlace, default_) => ({ type, inPlace, default: default_ });
export const CATALOG = {                            // provider version 1
  machine: { region: A("string", false), count: A("number", true), cores: A("number", false),
    memory_gb: A("number", true), disk_gb: A("number", true, 100), image: A("string", false, "base-2"),
    monitoring: A("string", true, "on") },
  queue: { region: A("string", false), partitions: A("number", false), retention_days: A("number", true, 7),
    format: A("string", false, "binary"), encryption: A("string", true, "off") },
  data_store: { region: A("string", false), size_gb: A("number", true), version: A("string", false, "3"),
    backup_days: A("number", true, 1), encryption: A("string", false, "off") },
  network: { region: A("string", false), subnet: A("number", false), gateways: A("number", true, 1) },
  object_store: { region: A("string", false), class: A("string", true, "hot"),
    versioning: A("string", false, "off"), retention_days: A("number", true, 365) },
};
export const DOWNTIME = { machine: 4, queue: 9, data_store: 25, network: 12, object_store: 30 };

export const RECORD = {                              // fictional record of the west environment
  "machine.collector": { region: "west", count: 6, cores: 4, memory_gb: 8 },
  "machine.verifier": { region: "west", count: 2, cores: 8, memory_gb: 16, disk_gb: 200, image: "base-2" },
  "queue.meter": { region: "west", partitions: 12 },
  "queue.alert": { region: "west", partitions: 3, format: "text" },
  "data_store.metrics": { region: "west", size_gb: 300 },
  "network.backbone": { region: "west", subnet: 3 },
  "object_store.archive": { region: "west" },
};

export const defaults = (t) => Object.fromEntries(Object.entries(t)
  .filter(([, a]) => a.default !== undefined).map(([f, a]) => [f, a.default]));

export function makeProvider(catalog) {             // interface: create / read / update / delete
  const store = new Map(), calls = { read: 0, create: 0, update: 0, delete: 0 };
  return { catalog, calls, store,
    read: (t, a) => (calls.read++, store.get(`${t}.${a}`) ?? null),
    create: (t, a, fields) => (calls.create++, store.set(`${t}.${a}`, { ...defaults(catalog[t]), ...fields })),
    update: (t, a, fields) => (calls.update++, Object.assign(store.get(`${t}.${a}`), fields)),
    delete: (t, a) => (calls.delete++, store.delete(`${t}.${a}`)) };
}

const OVERSIZED = (a, old, next) => a.type === "number" && next > old * 2;   // growth that cannot happen in place
export function decide(p, key, desired, plan) {  // the plan phase does not know the OVERSIZED rule
  const [t, a] = key.split(".");
  const actual = p.read(t, a);
  if (!actual) return ["create", []];
  const diff = Object.keys(desired).filter((f) => actual[f] !== desired[f]);
  const forcing = diff.filter((f) => !p.catalog[t][f].inPlace ||
    (!plan && OVERSIZED(p.catalog[t][f], actual[f], desired[f])));
  return forcing.length ? ["recreate", forcing]
    : diff.length ? ["update", diff] : ["unchanged", []];
}

export const PAST_TENSE = { create: "created", recreate: "recreated",
  update: "updated", unchanged: "unchanged", delete: "deleted" };

export function apply(p, record) {
  const result = [];
  for (const [k, desired] of Object.entries(record)) {
    const [t, a] = k.split("."), [decision, fields] = decide(p, k, desired, false);
    if (decision === "create") p.create(t, a, desired);
    else if (decision === "recreate") { p.delete(t, a); p.create(t, a, desired); }
    else if (decision === "update") p.update(t, a, Object.fromEntries(fields.map((f) => [f, desired[f]])));
    result.push([k, PAST_TENSE[decision], fields]);
  }
  for (const k of [...p.store.keys()]) if (!(k in record)) { const [t, a] = k.split("."); p.delete(t, a);
    result.push([k, "deleted", []]); }
  return result;
}

The provider consists of four operations, and all of them are counted; the record’s contact with the target system is the sum of these four calls. A resource type, in turn, is a field table: every field’s type, whether it can be updated in place, and its default are written down. Of these three properties, the second sets the record’s cost, and the third sets the record’s silence. The decision function uses both: if even one of the changed fields cannot be updated in place, the resource is deleted and recreated; otherwise a single update call is enough.

The Gap Between Plan and Apply

// change.mjs — the catalog is counted, the record applied, then a change set is planned and applied.
import { CATALOG, DOWNTIME, RECORD, makeProvider, decide, apply, PAST_TENSE } from "./provider.mjs";

const allFields = Object.values(CATALOG).flatMap(Object.values);
const print = (...s) => console.log(s.map((x, i) => String(x).padEnd(i ? 22 : 23)).join("").trimEnd());

console.log(`provider v1: ${Object.keys(CATALOG).length} resource types, ${allFields.length} fields ` +
  `(${(allFields.length / Object.keys(CATALOG).length).toFixed(1)} per type); ` +
  `${allFields.filter((a) => a.default === undefined).length} required, ` +
  `${allFields.filter((a) => !a.inPlace).length} fields cannot update in place`);

const narrow = (...s) => console.log(s.map((x) => String(x).padEnd(14)).join("").trimEnd());
narrow("type", "fields", "required", "not in place", "downtime");
for (const [t, f] of Object.entries(CATALOG)) { const v = Object.values(f);
  narrow(t, v.length, v.filter((a) => a.default === undefined).length,
    v.filter((a) => !a.inPlace).length, `${DOWNTIME[t]} min`); }
console.log();

const p = makeProvider(CATALOG);
apply(p, RECORD);
const written = Object.values(RECORD).reduce((t, r) => t + Object.keys(r).length, 0);
const actualFields = [...p.store.values()].reduce((t, r) => t + Object.keys(r).length, 0);
console.log(`record writes ${written} fields for ${Object.keys(RECORD).length} resources; ` +
  `${actualFields} fields exist on target, ${actualFields - written} of them filled by provider defaults`);
console.log(`first apply calls: ${JSON.stringify(p.calls)}\n`);

const RECORD2 = structuredClone(RECORD);              // the change set
RECORD2["machine.collector"].count = 10;
RECORD2["machine.collector"].cores = 8;
RECORD2["machine.verifier"].memory_gb = 48;
RECORD2["queue.meter"].partitions = 16;
RECORD2["data_store.metrics"].size_gb = 600;
delete RECORD2["queue.alert"];
RECORD2["machine.archiver"] = { region: "west", count: 1, cores: 2, memory_gb: 4 };

Object.assign(p.calls, { read: 0, create: 0, update: 0, delete: 0 });
const plan = Object.entries(RECORD2).map(([k, r]) => [k, ...decide(p, k, r, true)])
  .concat([...p.store.keys()].filter((k) => !(k in RECORD2)).map((k) => [k, "delete", []]));
const planReads = p.calls.read;
const result = apply(p, RECORD2);

print("resource", "plan", "apply", "forcing field");
let drift = 0;
for (const [k, decision, fields] of plan) {
  const g = result.find((x) => x[0] === k);
  if (PAST_TENSE[decision] !== g[1]) drift++;
  print(k, decision, g[1], (g[2].length ? g[2] : fields).join(", "));
}
const recreated = (l) => l.filter((x) => x[1].startsWith("recreate")).map((x) => DOWNTIME[x[0].split(".")[0]]);
const rp = recreated(plan), ru = recreated(result);
console.log(`\nplan predicted ${rp.length} recreation, apply did ${ru.length}; ` +
  `${plan.length - drift}/${plan.length} items held`);
console.log(`downtime: ${rp.reduce((a, b) => a + b, 0)} min in plan, ${ru.reduce((a, b) => a + b, 0)} min in apply`);
console.log(`plan did ${planReads} reads; change calls ${JSON.stringify(p.calls)}`);

provider v1: 5 resource types, 24 fields (4.8 per type); 11 required, 13 fields cannot update in place
type          fields        required      not in place  downtime
machine       7             4             3             4 min
queue         5             2             3             9 min
data_store    5             2             3             25 min
network       3             2             2             12 min
object_store  4             1             2             30 min

record writes 20 fields for 7 resources; 36 fields exist on target, 16 of them filled by provider defaults
first apply calls: {"read":7,"create":7,"update":0,"delete":0}

resource               plan                  apply                 forcing field
machine.collector      recreate              recreated             cores
machine.verifier       update                recreated             memory_gb
queue.meter            recreate              recreated             partitions
data_store.metrics     update                updated               size_gb
network.backbone       unchanged             unchanged
object_store.archive   unchanged             unchanged
machine.archiver       create                created
queue.alert            delete                deleted

plan predicted 2 recreation, apply did 3; 7/8 items held
downtime: 13 min in plan, 17 min in apply
plan did 7 reads; change calls {"read":14,"create":4,"update":1,"delete":4}

The first table gives the record’s scope. Five types define twenty-four fields in total; eleven are required, and thirteen cannot be updated in place. The split varies by type, and so does its cost: three of the machine’s seven fields and two of the network’s three fields cannot be updated in place, but recreating a machine takes four minutes and recreating an object store takes thirty. The share of fields that cannot be updated in place and the downtime in minutes are two separate measures, and they need to be read together: the network’s share is high but its resource is cheap; the object store’s share is low, and its resource is expensive.

The second line gives the gap between the record and reality. The record writes twenty fields for seven resources, but after apply thirty-six fields stand on the target. The provider has filled the sixteen fields in between with its own defaults, and these values are written nowhere in the record. Someone reading the record cannot learn from it which image the machines were built with, whether the queue is encrypted, or how many days the data store is backed up for. The infrastructure the record does not cover starts here: nearly half the fields of the resources the same record built run on values outside the record.

The third measure is the plan’s predictive accuracy. In an eight-item change set, the plan gets seven items right and misses one. The item it misses is this: the verifier machines’ memory is raised from sixteen to forty-eight, and the catalog has marked this field as updatable in place, so the plan says “will update.” The target system, however, will not accept growth of more than double in place, so it deletes the resource and recreates it. The result: the plan predicts two recreations, apply does three; predicted downtime is thirteen minutes, actual downtime is seventeen. Where the plan gets it wrong is where the catalog falls short — and the catalog belongs to the provider, not the record.

The last line is the call load. For eight items, the provider takes fourteen reads, four creates, one update, and four deletes. Half the reads belong to the plan itself: producing a plan means reading the target system once, start to finish; as the resource count grows, the plan’s cost grows right along with it. The three recreated resources alone account for three of the four deletes and three of the four creates. Three fields that cannot be updated in place turn into six calls and seventeen minutes of downtime.

When the Provider Version Changes

// version.mjs — when the provider version changes: shifting defaults, protected fields, changing decisions.
import { CATALOG, DOWNTIME, RECORD, makeProvider, decide, apply } from "./provider.mjs";

const CATALOG2 = structuredClone(CATALOG);          // provider version 2
CATALOG2.machine.image.default = "base-3";
CATALOG2.queue.encryption.default = "on";
CATALOG2.data_store.backup_days.default = 7;
CATALOG2.object_store.class.default = "cold";
CATALOG2.queue.retention_days.inPlace = false;         // in-place updatability changed too
CATALOG2.object_store.access = { type: "string", inPlace: true, default: "private" };

const setUp = (k) => { const p = makeProvider(k); apply(p, RECORD); return p; };
const a = setUp(CATALOG), b = setUp(CATALOG2), diff = [];
for (const [k, r] of a.store) for (const f of new Set([...Object.keys(r), ...Object.keys(b.store.get(k))]))
  if (r[f] !== b.store.get(k)[f]) diff.push([k, f, r[f], b.store.get(k)[f]]);
console.log(`same record set up with v1 and v2: ${diff.length} fields differ, ` +
  `${new Set(diff.map((x) => x[0])).size}/${a.store.size} resources affected`);
for (const [k, f, x, y] of diff) console.log(`  ${k.padEnd(23)}${f.padEnd(13)}${x} -> ${y}`);

const protectedFields = Object.entries(RECORD).flatMap(([k, r]) => Object.keys(r)
  .filter((f) => CATALOG[k.split(".")[0]][f].default !== CATALOG2[k.split(".")[0]][f].default)
  .map((f) => `${k}.${f}`));
console.log(`fields that do not shift because the record writes them: ${protectedFields.length} (${protectedFields.join(", ")})\n`);

const CHANGE = structuredClone(RECORD);                 // a one-line change in the record
CHANGE["queue.meter"].retention_days = 30;
for (const [name, cat] of [["v1", CATALOG], ["v2", CATALOG2]]) {
  const p = setUp(cat), pl = Object.entries(CHANGE).map(([k, r]) => [k, ...decide(p, k, r, true)]);
  const y = pl.filter((x) => x[1].startsWith("recreate"));
  console.log(`${name}: same change -> ${pl.filter((x) => x[1] === "update").length} update, ` +
    `${y.length} recreate, ${y.reduce((t, x) => t + DOWNTIME[x[0].split(".")[0]], 0)} min downtime`);
}

same record set up with v1 and v2: 6 fields differ, 5/7 resources affected
  machine.collector      image        base-2 -> base-3
  queue.meter            encryption   off -> on
  queue.alert            encryption   off -> on
  data_store.metrics     backup_days  1 -> 7
  object_store.archive   class        hot -> cold
  object_store.archive   access       undefined -> private
fields that do not shift because the record writes them: 1 (machine.verifier.image)

v1: same change -> 1 update, 0 recreate, 0 min downtime
v2: same change -> 0 update, 1 recreate, 9 min downtime

A version change is the record’s quietest risk, because nothing in the record changes at all. When the same seven-resource record is set up with two versions of the provider, the state on the target diverges in six fields, and five of the seven resources are affected: the machine’s image, the encryption on two queues, the data store’s backup frequency, the object store’s storage class, and a brand-new access field the second version adds. The last one also shows that a version does not just change existing defaults — it can add new fields that have no counterpart anywhere in the record.

The one field that does not shift is the one explicitly written in the record: the verifier machines write their own image, so they are not affected by the version change. The rule follows from this: if the record writes a field, that field is protected; if it does not, it is left to the provider’s version. A record that leaves sixteen fields at their defaults has left sixteen fields to the provider’s release notes. This is the second face of the sixteen from the first section: the same number is a readability gain in one place and a drift surface in another.

The version also changes the decision itself. The one-line change that raises the queue’s retention from seven days to thirty ends, in the first version, with one update call and zero minutes of downtime; in the second version, the same line recreates the resource and produces nine minutes of downtime. The record is the same, the change is the same, the outcome is not. Whether a field can be updated in place is knowledge that belongs to the provider, not the record; that knowledge changes with the version, and the only place that reports the change is the plan’s output — a change applied without reading the plan learns this difference only during the downtime.

Summary

• Provider v1’s 5 resource types defined 24 fields; 11 were required, 13 could not be updated in place. Recreation cost ranged from 4 to 30 minutes by type.
• The record wrote 20 fields for 7 resources, and 36 fields ended up on the target: the provider default filled 16 fields, and these values never appeared in the record.
• In an 8-item change, the plan got 7 items right; 2 recreations were predicted, 3 happened, and downtime came out to 17 minutes instead of 13. The plan knows only what is written in the catalog.
• Call load came to 14 reads, 4 creates, 1 update, 4 deletes; 7 of the reads belonged to the plan itself.
• When the provider version changed, 6 fields drifted with the record completely unchanged, and 5 of 7 resources were affected; the 1 field explicitly written in the record was protected.
• The same one-line change produced 1 update and 0 minutes in v1, and 1 recreation and 9 minutes of downtime in v2.

Next Step

Up to this point, the record built every resource on the target. Real environments do not work that way: part of the network, an identity definition, or a store another team built stands outside the record, and the record can connect to them only by reading. The next lesson measures that connection: how many reads the data sources that read existing infrastructure make, how often, how many resources get recomputed when a value they read changes, and why the items the record does not cover never show up in drift detection.