State File

The gap between the plan and the result of the apply was measured: the plan showed one item, the apply produced a different outcome. That comparison rested on a silent assumption. To be able to say “this resource already exists” or “this resource does not exist,” the plan assumed that reality was written down somewhere. Where does that written form live?

This lesson actually writes that file, actually reads it, and counts what cannot be done without it.

Definition, Apply and Two Outputs

A fictional regional measurement network will be used: three environments, in each environment two machines collecting meter readings, a queue, a data store, an object store, a subnet and network rules. Both the network and the provider are models; no real infrastructure is touched.

An apply produces two things: resources on the provider side, and a state file on its own side. The script below writes all three to disk.

// apply.mjs — the fictional "regional measurement network": builds the definition, models the apply,
// and writes the provider-side reality and the state file to disk. Both are models.
import { writeFileSync } from "node:fs";

const environments = ["test", "staging", "production"];
const definition = [];
const add = (type, name, env, fields, deps = []) =>
  definition.push({ name, type, env, fields, deps });

environments.forEach((e, i) => {
  add("subnet", `subnet_${e}`, e, { block: `10.${i}.0.0/16` });
  add("queue", `read_queue_${e}`, e, { retention_hours: 24, partitions: 4 });
  add("data_store", `measurement_store_${e}`, e, { size_gb: 200, backup: true });
  add("object_store", `archive_${e}`, e, { class: "infrequent" });
  for (const n of [1, 2])
    add("machine", `collector_${e}_${n}`, e, { cores: 2, memory_gb: 8 },
      [`subnet_${e}`, `read_queue_${e}`]);
  for (const port of [443, 5671, 9100])
    add("security_rule", null, e, { direction: "inbound", port }, [`subnet_${e}`]);
  add("route_table_entry", null, e, { destination: "0.0.0.0/0", gateway: "edge" }, [`subnet_${e}`]);
});
// Second port-9100 rule for the backup collector in production: the same item appears twice in the definition.
add("security_rule", null, "production", { direction: "inbound", port: 9100 }, ["subnet_production"]);

// Values the provider fills in on its own for fields the definition does not give.
const defaults = {
  machine: { disk_gb: 40, monitoring: "on" }, queue: { encryption: "on" },
  data_store: { maintenance_window: "sunday" }, object_store: { version_retention: false },
  subnet: { ipv6: false }, security_rule: {}, route_table_entry: {},
};
const unnamed = new Set(["security_rule", "route_table_entry"]);

const ids = definition.map((_, i) => `res-${String(i + 1).padStart(4, "0")}`);
const nameIndex = new Map(definition.map((k, i) => [k.name, ids[i]]));

const actual = [], state = [];
definition.forEach((k, i) => {
  const fields = { ...defaults[k.type], ...k.fields };
  const parentName = k.deps.find((b) => b.startsWith("subnet")) ?? null;
  actual.push({
    id: ids[i], type: k.type, fields,
    tag: unnamed.has(k.type) ? {} : { name: k.name, env: k.env },
    parent: parentName ? nameIndex.get(parentName) : null,
  });
  state.push({
    record_name: k.name ?? `${k.type}.${k.env}[${i}]`, type: k.type, env: k.env,
    id: ids[i], fields, deps: k.deps, provider_version: "3.4.0",
  });
});

writeFileSync("definition.json", JSON.stringify(definition, null, 1));
writeFileSync("provider.json", JSON.stringify(actual, null, 1));
writeFileSync("state.json", JSON.stringify(
  { format_version: 4, serial: 1, items: state }, null, 1));

const fieldCount = state.reduce((t, k) => t + Object.keys(k.fields).length, 0);
const definitionFields = definition.reduce((t, k) => t + Object.keys(k.fields).length, 0);
const edges = state.reduce((t, k) => t + k.deps.length, 0);
console.log(`definition: ${definition.length} items, ${definitionFields} field values`);
console.log(`state.json: ${state.length} items, ${fieldCount} field values, ` +
  `${edges} dependency edges, ${fieldCount - definitionFields} fields state-only`);
console.log(`record per item: id 1, fields ${(fieldCount / state.length).toFixed(1)}, ` +
  `deps ${(edges / state.length).toFixed(1)}, provider version 1`);
console.log(`sample item: ${JSON.stringify(state[4])}`);

SM1. The provider assigns ids in apply order; in the model, ids are assigned in sequence starting from res-0001. SM2. The security rule and the route table entry are types that carry no tag; the provider gives these items only an id. SM3. The provider fills fields the definition does not give with defaults, and these values come back in the apply’s response.

node apply.mjs

definition: 31 items, 56 field values
state.json: 31 items, 80 field values, 25 dependency edges, 24 fields state-only
record per item: id 1, fields 2.6, deps 0.8, provider version 1
sample item: {"record_name":"collector_test_1","type":"machine","env":"test","id":"res-0005","fields":{"disk_gb":40,"monitoring":"on","cores":2,"memory_gb":8},"deps":["subnet_test","read_queue_test"],"provider_version":"3.4.0"}

The first pair of numbers is interesting: the definition writes 56 field values, the state holds 80. In between are 24 fields that exist only in state; the definition did not write them, the provider set them. What the record “wants” and what reality “is” already diverge in the apply’s first second, and the state file is the only place that keeps that distinction.

After a Manual Intervention

Time passed since the apply. During a night shift, someone touched the infrastructure through the provider’s interface.

// intervention.mjs — models a manual touch on the provider-side reality: two items deleted,
// four items opened outside the record, one field changed. The state file is not touched.
import { readFileSync, writeFileSync } from "node:fs";
const g = JSON.parse(readFileSync("provider.json", "utf8"));

const idOf = (name) => g.find((k) => k.tag.name === name).id;
const subnetStaging = idOf("subnet_staging"), subnetProduction = idOf("subnet_production");
const remove = (cond) => { const i = g.findIndex(cond); const [k] = g.splice(i, 1); return k.id; };
const del1 = remove((k) => k.tag.name === "archive_test");
const del2 = remove((k) => k.type === "route_table_entry" && k.parent === subnetStaging);

g.push({ id: "res-9001", type: "machine", fields: { cores: 4, memory_gb: 16, disk_gb: 40 },
  tag: { name: "temp_collector", env: "production" }, parent: subnetProduction });
g.push({ id: "res-9002", type: "object_store", fields: { class: "hot", version_retention: false },
  tag: { name: "manual_archive", env: "production" }, parent: null });
g.push({ id: "res-9003", type: "security_rule", fields: { direction: "inbound", port: 22 },
  tag: {}, parent: subnetProduction });
g.push({ id: "res-9004", type: "security_rule", fields: { direction: "inbound", port: 443 },
  tag: {}, parent: subnetProduction });

const m = g.find((k) => k.tag.name === "collector_production_1");
m.fields.memory_gb = 16;

writeFileSync("provider.json", JSON.stringify(g, null, 1));
console.log(`deleted: ${del1}, ${del2} | opened outside record: 4 | field manually changed: 1`);
console.log(`item count on provider: ${g.length}`);

node apply.mjs > /dev/null   # setup rebuilt from scratch
node intervention.mjs

deleted: res-0004, res-0020 | opened outside record: 4 | field manually changed: 1
item count on provider: 33

Matching Without State

The question now is: if there had never been a state file at all, what would it take to map the record to reality? The only way is to scan the provider and match logical names against tags.

// matching.mjs — tries to map the record to reality without looking at the state file.
import { readFileSync } from "node:fs";
const definition = JSON.parse(readFileSync("definition.json", "utf8"));
const actual = JSON.parse(readFileSync("provider.json", "utf8"));
const unnamed = new Set(["security_rule", "route_table_entry"]);

const types = [...new Set(actual.map((k) => k.type))];
let listings = types.length;                     // one listing per type
let detail = actual.filter((k) => unnamed.has(k.type)).length;  // every unnamed candidate is read

// Matching by name: the listing response carries the tag, so no extra read is needed.
const nameIndex = new Map(actual.filter((k) => k.tag.name).map((k) => [k.tag.name, k]));
const used = new Set();
let byName = 0, byNameMissing = 0;
for (const k of definition.filter((k) => k.name)) {
  const found = nameIndex.get(k.name);
  if (found) { byName++; used.add(found.id); } else byNameMissing++;
}

// Unnamed items: distinguishable only by content and the subnet they are attached to.
const signature = (type, parent, fields) =>
  `${type}|${parent}|${Object.keys(fields).sort().map((a) => `${a}=${fields[a]}`).join(",")}`;
const actualSig = new Map();
for (const k of actual.filter((k) => unnamed.has(k.type)))
  actualSig.set(signature(k.type, k.parent, k.fields), (actualSig.get(signature(k.type, k.parent, k.fields)) ?? 0) + 1);
const defSig = new Map();
for (const k of definition.filter((k) => unnamed.has(k.type))) {
  const parent = nameIndex.get(k.deps.find((b) => b.startsWith("subnet")))?.id ?? null;
  const i = signature(k.type, parent, k.fields);
  defSig.set(i, (defSig.get(i) ?? 0) + 1);
}
let unique = 0, ambiguous = 0, unnamedMissing = 0;
for (const [i, count] of defSig) {
  const g = actualSig.get(i) ?? 0;
  if (g === 0) unnamedMissing += count;
  else if (count === 1 && g === 1) unique++;
  else ambiguous += count;
}
const unowned = actual.filter((k) => !used.has(k.id) && !unnamed.has(k.type)).length;
const unownedUnnamed = [...actualSig].reduce((t, [i, g]) => t + Math.max(0, g - (defSig.get(i) ?? 0)), 0);

console.log(`matching without state — reads: ${listings} listings + ${detail} detail = ${listings + detail}`);
console.log(`  matched by name: ${byName}/${definition.filter((k) => k.name).length}`);
console.log(`  matched uniquely by content: ${unique}/${definition.filter((k) => unnamed.has(k.type)).length}`);
console.log(`  never matched (unnamed, signature collides): ${ambiguous}`);
console.log(`  in record but not in reality: ${byNameMissing + unnamedMissing} -> deleted or never created: indistinguishable`);
console.log(`  in reality but not in record: ${unowned} named + ${unownedUnnamed} unnamed`);

SM4. One listing per type counts as one read; the tag of a named item comes back with the listing, while an unnamed item’s fields are only visible through a detail read.

node matching.mjs

matching without state — reads: 7 listings + 14 detail = 21
  matched by name: 17/18
  matched uniquely by content: 9/13
  never matched (unnamed, signature collides): 3
  in record but not in reality: 2 -> deleted or never created: indistinguishable
  in reality but not in record: 2 named + 2 unnamed

After twenty-one reads, the table left in hand is incomplete. 26 of the thirty-one items matched, 3 items could never be matched. All three are unnamed resources: the two port-9100 rules in production are identical twins, and the port-443 rule has two counterparts on the provider — one from the definition, one by hand. Between two items with identical content, there is no information to say which id belongs to which record line.

The second loss is heavier: for the two missing items, the distinction between “deleted” and “never created” cannot be made. The record sees both the same way and makes the same decision for both — create. It is never learned that the deleted resource was deleted; it is only recreated.

What the State File Holds

The sample item shows that state keeps four separate pieces of information per item: id (res-0005), field values (four; two from the definition, two from the provider default), dependencies (two record names), and provider version (3.4.0). Totals for thirty-one items: 31 ids, 80 field values, 25 dependency edges, 31 version stamps.

Each of these four pieces does a separate job. The id makes matching unique. Field values are the comparison baseline for drift — because the default the provider set is recorded too, a difference becomes visible if that default changes with the provider version. The dependency edge gives the deletion order; it is read from the applied state, not from the text of the record file. The version stamp makes it possible to detect that the same definition produces a different result on a different provider version.

Comparing With State

// state-read.mjs — compares the ids in the state file against reality on the provider.
import { readFileSync } from "node:fs";
const st = JSON.parse(readFileSync("state.json", "utf8"));
const actual = new Map(JSON.parse(readFileSync("provider.json", "utf8")).map((k) => [k.id, k]));

let reads = 1, matched = 0;                      // 1: the state file itself
const deleted = [], drifted = [];
for (const k of st.items) {
  reads++;                                       // one item read by id
  const g = actual.get(k.id);
  if (!g) { deleted.push(`${k.record_name} (${k.id})`); continue; }
  matched++;
  for (const [field, value] of Object.entries(k.fields))
    if (g.fields[field] !== value) drifted.push(`${k.record_name}.${field} ${value} -> ${g.fields[field]}`);
}
const stateIds = new Set(st.items.map((k) => k.id));
const outOfRecord = [...actual.values()].filter((g) => !stateIds.has(g.id));

console.log(`comparing with state — reads: 1 state file + ${reads - 1} ids = ${reads}`);
console.log(`  matched: ${matched}/${st.items.length}, never matched: 0`);
console.log(`  deleted (id in state, not on provider): ${deleted.length} -> ${deleted.join("; ")}`);
console.log(`  drifted field: ${drifted.length} -> ${drifted.join("; ")}`);
console.log(`  not covered by the record (on provider, not in state): ${outOfRecord.length} -> ` +
  outOfRecord.map((g) => g.tag.name ?? `${g.type}/${g.id}`).join(", "));

node state-read.mjs

comparing with state — reads: 1 state file + 31 ids = 32
  matched: 29/31, never matched: 0
  deleted (id in state, not on provider): 2 -> archive_test (res-0004); route_table_entry.staging[19] (res-0020)
  drifted field: 1 -> collector_production_1.memory_gb 8 -> 16
  not covered by the record (on provider, not in state): 4 -> temp_collector, manual_archive, security_rule/res-9003, security_rule/res-9004

The read count went from 21 to 32 — state does not cheapen matching, it makes it certain. Three numbers were gained: unmatched items dropped from 3 to 0, two deleted items were found by name and id, and one drifted field (memory_gb 8 instead of 16) surfaced. None of these three were visible in matching without state.

The fourth number gives the record’s coverage: 4 of the 33 items on the provider are inside no record at all. Two machines and an object store stand by name, two security rules stand only by id. The record neither plans, nor deletes, nor counts drift for these; this part of the infrastructure is outside the record, and it is known to be there only as long as no one deletes it without looking.

The Loss of State

The state file is a record as critical as the infrastructure itself. The cost of losing it can be counted.

// loss.mjs — counts what can be recovered if the state file is lost.
import { readFileSync, rmSync, existsSync } from "node:fs";
const d = JSON.parse(readFileSync("state.json", "utf8"));
const lost = { id: d.items.length, edge: 0, version: 0, field: 0 };
for (const k of d.items) {
  lost.edge += k.deps.length;
  lost.version += k.provider_version ? 1 : 0;
  lost.field += Object.keys(k.fields).length;
}
rmSync("state.json");
console.log(`state.json deleted; does the file still exist: ${existsSync("state.json")}`);
console.log(`lost information: ${lost.id} ids, ${lost.field} field values, ` +
  `${lost.edge} dependency edges, ${lost.version} provider versions`);
console.log(`the only path left: matching without state`);

node apply.mjs > /dev/null && node intervention.mjs > /dev/null   # setup rebuilt from scratch
node loss.mjs

state.json deleted; does the file still exist: false
lost information: 31 ids, 80 field values, 25 dependency edges, 31 provider versions
the only path left: matching without state

When the file is gone, the infrastructure is not gone; its manageability is. The only path left is the matching-without-state above, and its ceiling is known: 26 items can be recovered by importing them one by one, 3 items can never be matched by any means. The information that the two deleted items were deleted has also gone with the file. Until the record is rebuilt, every apply plans to recreate resources that already exist.

SM5. The state file sits in a single working directory, as plain text. This is the second face of the record’s own risk: the file holds all 80 field values as they are. Even if a connection string or access credential given through a variable was hidden in the definition, it is written here in its post-apply form, and anyone who can read the file reads it. SM6. The only thing that writes the file is the apply; editing it by hand genuinely pulls the state away from reality, because it has no counterpart on the provider side.

Summary

• The apply produced two outputs: 31 resources on the provider and the state file that records them; the definition wrote 56 field values while state held 80, with 24 fields set by the provider.
• Matching without state took 21 reads and 3 items were never matched; all three were unnamed resources whose signatures were identical.
• Without state, “deleted” cannot be separated from “never created”; with state, two deleted items were found by id and one drifted field surfaced.
• State keeps four pieces of information per item: id, field values, dependencies, provider version — for 31 items, 31 ids, 80 fields, 25 edges, 31 version stamps.
• 4 of the 33 items on the provider fell outside the record; the record neither plans them nor counts them.
• When state is lost, 26 items come back by importing, 3 do not; the loss does not destroy the infrastructure, it makes it unmanageable.

Next Step

This whole lesson assumed a single person working in a single directory. The state file sat on local disk, and only one process touched it at a time. That assumption falls the moment two people on a team start an apply against the same environment at the same time: both read the same state, both change it, and the second overwrites what the first wrote. The next lesson actually runs two concurrent applies, counts the write that is lost without a lock, measures the waiting rounds a lock produces, and deals with the stale lock a process leaves behind when it dies without releasing it.