Resource Modification and Import

The previous lesson measured the plan for resources written in the record. On the measurement network, though, there are also resources never written in the record at all: an object store opened by hand at initial setup, a DNS zone, two old collector machines, a default security group, an old data store, and a queue. These seven resources never enter a plan, their drift never shows, and they never pass through a review. The planner does not see them, because the planner only compares what the record and state know.

This lesson’s job is bringing those seven under management. The operation has two parts, and both are measured separately: matching a resource’s real identity to an address, and hand-writing that resource’s counterpart in the record. The first is mechanical. The second is a copying job, and the copying job’s error rate is this lesson’s central number.

The measurement runs on the regional measurement network and is fictional; what runs is a model built with node.

DP43. The seven resources sitting at the provider are resources no record knows about. DP44. The record is written by hand; values are read from the provider interface. It is wrong in four places: two values were misread, one value was assumed to be the default, one value was written in a different letter case. DP45. Import writes the entirety of the resource’s real fields to state; only the hand-written fields enter the record. The gap between the two is the fields left out of the record. DP46. The planner and the provider carry over the previous lesson’s rules: image, size, and subnet require recreation, and the provider stores strings without case sensitivity. DP47. Recreation downtime is 105 seconds for a machine, 900 for a data store; a meter-reading batch is missed every 15 seconds. DP48. A recreated resource loses the fields not written in the record and reverts to the provider default. Computed fields the provider generates on its own are outside this.

Bringing Seven Resources Under Management

// import.mjs — bringing out-of-record resources under management, and measuring the first plan
// after import. Model; no call goes to a real provider.

// DP43: seven resources sitting at the provider that no record knows about. Opened by hand at initial setup.
const ACTUAL = {
  "object_store.old_metrics": { name: "old-metrics", region: "north",
    version_protection: "on", encryption: "provider" },
  "dns_zone.metrics": { name: "metrics.internal", ttl: 3600, record_count: 41 },
  "machine.old_collector_1": { subnet: "subnet-01", image: "base-4", size: "large",
    tag: "collector-old-1" },
  "machine.old_collector_2": { subnet: "subnet-01", image: "base-4", size: "large",
    tag: "collector-old-2" },
  "security_group.default": { network: "network-01", ingress: "10.20.0.0/16", egress: "0.0.0.0/0" },
  "data_store.old_meter": { subnet: "subnet-01", size: "medium", version: "3.9", backup: "daily" },
  "queue.old_stream": { subnet: "subnet-01", retention: 3, type: "fifo" },
};

// DP44: the record is written by hand; values are read from the provider interface. Wrong in four places.
const MANUAL = {
  "object_store.old_metrics": { name: "old-metrics", region: "north" },
  "dns_zone.metrics": { name: "metrics.internal", ttl: 300 },                    // assumed to be the default
  "machine.old_collector_1": { subnet: "subnet-01", image: "base-4-x86", size: "large",
    tag: "collector-old-1" },                                       // misread
  "machine.old_collector_2": { subnet: "subnet-01", image: "base-4", size: "large",
    tag: "collector-old-2" },
  "security_group.default": { network: "network-01", ingress: "10.20.1.0/24" }, // misread
  "data_store.old_meter": { subnet: "subnet-01", size: "MEDIUM", version: "3.9" },  // letter case
  "queue.old_stream": { subnet: "subnet-01", retention: 3, type: "fifo" },
};

const COMPUTED = ["record_count"];            // the provider produces this, it is not written to the record
const FORCES_RECREATE = ["image", "size", "subnet"];     // DP46: previous lesson's schema rules
const DOWNTIME = { machine: 105, data_store: 900 };  // DP47: recreation downtime, seconds
const NORMALIZE = (v) => (typeof v === "string" ? v.toLowerCase() : v);
const K = (v) => JSON.stringify(v);

// DP45: import writes the entirety of reality's fields to state; only what is written enters the record.
const STATE = structuredClone(ACTUAL);

function plan(record, baseline, { normalize }) {
  const equal = normalize ? (a, b) => K(NORMALIZE(a)) === K(NORMALIZE(b)) : (a, b) => K(a) === K(b);
  const items = new Map();
  for (const [address, y] of Object.entries(record)) {
    const fields = Object.keys(y).filter((k) => !equal(y[k], baseline[address][k]));
    if (fields.length)
      items.set(address, { fields,
        decision: fields.some((k) => FORCES_RECREATE.includes(k)) ? "recreate" : "update" });
  }
  return items;
}

const written = (address) => Object.keys(MANUAL[address]).length;
const manageable = (address) =>
  Object.keys(ACTUAL[address]).filter((k) => !COMPUTED.includes(k)).length;
const uncovered = (address) => manageable(address) - written(address);
const addresses = Object.keys(ACTUAL);

console.log("imported resources");
console.log("address                      real fields  hand-written  uncovered  first plan");
const PLAN = plan(MANUAL, STATE, { normalize: false });
const APPLY = plan(MANUAL, ACTUAL, { normalize: true });
for (const address of addresses)
  console.log(`${address.padEnd(28)}${String(manageable(address)).padStart(11)}` +
    `${String(written(address)).padStart(14)}${String(uncovered(address)).padStart(13)}` +
    `  ${PLAN.get(address) ? `${PLAN.get(address).decision} (${PLAN.get(address).fields.join(",")})` : "-"}`);

const total = (f) => addresses.reduce((s, a) => s + f(a), 0);
console.log(`\ntotal: ${addresses.length} items, ${total(manageable)} manageable fields, ` +
  `${total(written)} hand-written, ${total(uncovered)} fields stayed out of the record`);

console.log(`\nfirst plan after import: ${PLAN.size} items ` +
  `(${[...PLAN.values()].filter((v) => v.decision === "recreate").length} recreate)`);
const phantom = [...PLAN.keys()].filter((a) => !APPLY.has(a));
console.log(`  happened at apply: ${APPLY.size} items; did not happen ${phantom.length}` +
  ` (${phantom.join(", ")})`);
const recreations = (m) => [...m].filter(([, v]) => v.decision === "recreate").map(([a]) => a);
const downtime = (m) => recreations(m).reduce((s, a) => s + (DOWNTIME[a.split(".")[0]] ?? 0), 0);
console.log(`  recreation: ${recreations(PLAN).length} in plan, ` +
  `${recreations(APPLY).length} at apply; downtime ${downtime(PLAN)} s in plan, ` +
  `${downtime(APPLY)} s at apply (${(downtime(APPLY) / 15).toFixed(0)} missed meter-reading batches)`);
const lost = (m) => recreations(m).reduce((s, a) => s + uncovered(a), 0);
console.log(`  field reverting to default on recreation: ${lost(PLAN)} in plan, ` +
  `${lost(APPLY)} at apply`);

// Correction round: read every field the plan shows back from the provider and write it into the record.
const CORRECTED = structuredClone(MANUAL);
let corrected = 0;
for (const [address, v] of PLAN)
  for (const k of v.fields) { CORRECTED[address][k] = ACTUAL[address][k]; corrected++; }
const SECOND = plan(CORRECTED, STATE, { normalize: false });
console.log(`\ncorrection round: ${corrected} fields read back -> ` +
  `second plan ${SECOND.size} items`);

console.log("\nthe record's coverage: three options");
console.log("option               entered in record (drift visible)  unmanaged  " +
  "first plan  lost on recreation");
const option = [
  ["leave out of record", 0, total(manageable), 0, 0],
  ["partial import", total(written), total(uncovered), PLAN.size, total(uncovered)],
  ["full import", total(manageable), 0, 0, 0],
];
for (const [name, ...s] of option)
  console.log(name.padEnd(21) + s.map((x, i) => String(x).padStart([29, 14, 10, 27][i])).join(""));

imported resources
address                      real fields  hand-written  uncovered  first plan
object_store.old_metrics              4             2            2  -
dns_zone.metrics                      2             2            0  update (ttl)
machine.old_collector_1               4             4            0  recreate (image)
machine.old_collector_2               4             4            0  -
security_group.default                3             2            1  update (ingress)
data_store.old_meter                  4             3            1  recreate (size)
queue.old_stream                      3             3            0  -

total: 7 items, 24 manageable fields, 20 hand-written, 4 fields stayed out of the record

first plan after import: 4 items (2 recreate)
  happened at apply: 3 items; did not happen 1 (data_store.old_meter)
  recreation: 2 in plan, 1 at apply; downtime 1005 s in plan, 105 s at apply (7 missed meter-reading batches)
  field reverting to default on recreation: 1 in plan, 0 at apply

correction round: 4 fields read back -> second plan 0 items

the record's coverage: three options
option               entered in record (drift visible)  unmanaged  first plan  lost on recreation
leave out of record                              0            24         0                          0
partial import                                  20             4         4                          4
full import                                     24             0         0                          0

The top table is the copying job’s balance sheet. Seven resources have twenty-four manageable fields; twenty of them were hand-written. Four fields were never written at all: the object store’s version protection and encryption settings, the security group’s egress rule, the data store’s backup arrangement. These were not forgotten — the person writing the record either did not see them in the provider interface or passed over them assuming they were the default. The record’s coverage here is exactly eighty-three percent, and the remaining seventeen percent is infrastructure the record never knew about.

The coverage ratio varies from resource to resource. Two machines were written with all four of their four fields and the queue with all three of its three; the object store with two of four. The difference depends on how visible the fields are: the machine’s subnet and image are on the interface’s first screen, the object store’s encryption setting is on its third tab. The record’s coverage is determined by how far down the person writing it scrolled.

Import itself does exactly one thing: it binds the real identity at the provider to an address in the record, and writes all of that resource’s fields to state. What it does not do matters more — it does not write the record. State knows reality exactly; the record only knows what the human wrote. For the first time in this course, record and state start from the same moment and are born with separate contents, and the gap between them shows up immediately in the first plan.

The chosen address is a decision too, and it connects to this topic’s earlier measure. The seven resources here were addressed by their own names; if they had been addressed by an index, every future addition and removal would shift the addresses, and the imported resources would start pointing at different real resources. Import is where the price of address stability is highest: a wrong address means not just a needless recreation, but the recreation of the wrong resource.

The First Plan After Import

After import finishes, nothing is expected to have changed: the resources are the same, state is an exact copy of reality. Despite this, the first plan shows four items, and every one of them comes from the record not matching reality — not one of them is a wanted change.

Three of the four happen at apply. The item that does not is the data store: because the record writes size in uppercase, the planner sees a difference, but because the provider does not distinguish case, nothing happens at apply. This item will keep coming back on every plan run and will stay permanent noise unless someone touches it.

The danger in these four items is not their size, it is that they run counter to expectation. The plan is expected to come out empty after import; when it does not, two readings are possible — either the record describes reality wrong, or a change is genuinely wanted. Both look the same in the plan. The review load measured in the previous lesson is at its worst here: the person reading the plan of a freshly imported resource has nothing in hand to tell which kind an item is. The only way is to read every shown value back from the provider — that is, to do import’s copying job one more time.

The real number is in the recreation row. The plan says two resources will be recreated; at apply, one does. The gap between them is large in the measurement network’s unit: the plan’s promised downtime is 1005 seconds, the actual is 105. Nine hundred of the thousand five belong to the data store, and that item was born from nothing more than a letter-case difference. Because the person writing the record wrote MEDIUM instead of medium, the plan showed that a data store carrying forty-one days of meter data would be recreated.

Even the 105 seconds that did happen did not come free. One of the old collector machines’ image had been misread; the plan mistook this for a change and recreated the machine, the machine was out of service for 105 seconds, and seven meter-reading batches were missed in that subregion. The resource had never changed; the only thing that changed was its description in the record. Import’s downtime comes from the copy’s error, not the resource’s.

The correction round closes this section. When the four fields the plan shows are read back from the provider and written into the record, the second plan comes out to zero items. So import does not finish in a single step: once the address is bound, settling the record against reality needs at least one more round, and that round’s input is the plan itself. Here the plan works not as a warning, but as a correction list.

Fields Lost on Recreation

The four fields left out of the record look harmless right after import. The plan does not show them, apply does not touch them, the resource keeps working as it is. Their harmlessness rests on a single condition: the resource not being recreated.

Recreation removes that condition. When a resource is destroyed and recreated, the new resource is built from the record, and every field not written in the record reverts to the provider default. In this measurement, that is one field in the two planned recreations, zero in the one that actually happened — a small number, but the field at risk of loss is the data store’s backup arrangement. A data store with daily backups could have ended up without backups, with nobody wanting it and no plan ever showing it.

The shape of the danger here is worth noting. An out-of-record field does not show up as drift either, because drift can only be measured on fields that can be compared against the record. A field the record does not know about does not know its own drift either. Four of the seven resources’ twenty-four fields enter none of the differences this course measures — not in the plan, not in the drift count, not in review.

The Record’s Coverage: Three Options

The last table opens the decision between three options, and the first column gives away the misleading one. Leaving the resource out of the record asks for zero lines of record, produces a zero-item plan, loses zero fields on recreation. Every column is zero — but the first column is zero too: no field’s drift is visible. None of the twenty-four fields is ever compared. An out-of-record resource is not cheap, it is unmeasured.

Partial import makes twenty fields visible and leaves four outside, and in return asks for a four-item first plan and a correction round. Full import writes all twenty-four fields; its first plan is zero and no field is lost on recreation, but the number of fields to write rises by twenty percent, and each of those fields is a line that will need maintenance going forward.

There is a fourth option not shown in the table: import it, then remove it from management. The resource stays in the record, but apply never touches it. This option combines the first three’s downsides — the record needs maintenance, drift is still invisible — and its only gain is that the resource’s existence is written down. On the measurement network, its one legitimate use is the old data store, which is dangerous to destroy.

Put in this course’s measure: the wider the record, the more measurable the gap between it and reality, and the more measurable the gap, the more manageable it is. An unmeasured item is not failing to be fixed — nobody knows it is broken. Four of the seven resources’ fields are correct today; when someone changes one tomorrow, there is no plan anywhere that will show it.

Summary

• 20 of seven out-of-record resources’ 24 manageable fields were hand-written, 4 stayed out of the record: coverage 83 percent. What determines coverage is how visible the field is in the provider interface.
• The first plan after import showed 4 items, and none of them was a wanted change; 3 happened at apply, and 1 will keep coming back on every run because it was born from a letter-case difference.
• The plan promised 2 recreations, apply did 1: downtime was 1005 seconds in the plan, 105 at apply, with 7 missed meter-reading batches. The cause of the downtime was not the resource changing, it was the copy being misread.
• The correction round read 4 fields back from the provider and brought the second plan down to 0 items; import does not finish in a single step, and the correction’s input is the plan itself.
• The 4 fields left out of the record are harmless only as long as the resource is not recreated; once it is, they revert to the default. One of the fields at risk of loss was the data store’s backup arrangement.
• When the three options are compared, leaving a resource out of the record makes every column zero — including the column for fields whose drift is visible. An out-of-record resource is not cheap, it is unmeasured.

Next Step

This topic measured the record itself: order, iteration, the plan, and now the record’s coverage. In all four lessons, the same thing surfaced once more — the gap between plan and apply was measured, but the state the plan rests on was always taken as a given. Yet state is a file: it sits somewhere, someone writes it, someone reads it. This lesson’s import wrote to it directly too.

The next topic opens that file. What gets measured is clear: where state sits and who can access it, what is written inside it — sensitive values included — and what happens when two people apply at the same time. This last one is the most expensive question, because two applies overwriting each other never shows up in any plan.