Dependencies and Lifecycle

The previous lesson parameterized the definition: the same record became applicable to three environments, each with its own set of variables. The record now says which resource will exist with which values. There is one thing it does not say, and that thing gets asked in apply’s first second: in what order will these fourteen resources be created.

A declarative record does not write down order. Not writing it does not mean order does not exist — a subnet cannot exist before its own network. The order is inferred from within the record: if a resource’s field uses another resource’s output, there is an edge between them. What this lesson measures is the scope of that inference: how many of the real requirements the record sees, what happens when it cannot see them, and what every hand-written line to make it see costs.

The measurement again runs on the regional measurement network’s record and is fictional; what runs is a model built with node, and no call goes to a real provider.

DP25. A region’s record consists of fourteen resources: a network, subnet, security group, role, grant, object store, config object, data store, queue, two collector machines, load balancer, health check, and DNS record. DP26. The configuration language’s syntax is this lesson’s own small form: resource <type> "<name>" { field = expression }. If a field’s expression contains <type>.<name>.<output>, an implicit dependency is born; a linked = [...] list is an explicit declaration. DP27. Resources with resolved dependencies are created in the same round, i.e. in parallel; the next round does not start until one round finishes. DP28. The machine pulls its config file from the object store on boot and needs a grant for it. These two requirements appear in the record only in free text; neither enters an expression. DP29. Durations: creating a machine is 90 seconds, destroying is 10, adding to and removing from the load balancer is 5. Two machines are full capacity, one machine is half. DP30. Recreating the data store is irreversible: the 41 days of meter readings inside it are deleted.

The Order the Record Builds on Its Own

// graph.mjs — builds a dependency graph from the record text and resolves a topological order. Model;
// no real infrastructure is created. The configuration language's syntax is this lesson's own form.

export const RECORD = `
resource network "metrics" {
  region = "north"
  block  = "10.20.0.0/16"
}
resource object_store "config" {
  name = "metrics-config"
}
resource role "collector" {
  name = "collector-role"
}
resource grant "machine_read" {
  target = object_store.config.name
  to     = role.collector.id
}
resource subnet "collector" {
  network = network.metrics.id
  block   = "10.20.1.0/24"
}
resource security_group "collector" {
  network = network.metrics.id
  ingress = "10.20.1.0/24"
}
resource data_store "meter" {
  subnet   = subnet.collector.id
  security = security_group.collector.id
  size     = "medium"
}
resource queue "batch" {
  subnet = subnet.collector.id
}
resource object "collector_config" {
  store         = object_store.config.name
  data_address  = data_store.meter.address
  queue_address = queue.batch.address
}
resource machine "collector_1" {
  subnet   = subnet.collector.id
  security = security_group.collector.id
  role     = role.collector.id
  boot     = "pull config from the object store"
}
resource machine "collector_2" {
  subnet   = subnet.collector.id
  security = security_group.collector.id
  role     = role.collector.id
  boot     = "pull config from the object store"
}
resource load_balancer "collector" {
  subnet  = subnet.collector.id
  targets = [machine.collector_1.id, machine.collector_2.id]
  linked  = [security_group.collector]
}
resource health_check "collector" {
  target = load_balancer.collector.id
}
resource dns_record "collector" {
  address = load_balancer.collector.address
  linked  = [health_check.collector]
}
`;

// DP28: four requirements the record never sees. The machine pulls its config from the object
// store on boot and needs a grant for it; both live only in the "boot" text, not in an expression.
export const HIDDEN = [
  ["machine.collector_1", "object.collector_config"],
  ["machine.collector_1", "grant.machine_read"],
  ["machine.collector_2", "object.collector_config"],
  ["machine.collector_2", "grant.machine_read"],
];

export function parse(text) {
  const nodes = [], edges = new Map();                    // edge: "a->b" -> {implicit, explicit}
  const block = /resource\s+(\w+)\s+"(\w+)"\s*\{([\s\S]*?)\n\}/g;
  let m;
  while ((m = block.exec(text)) !== null) {
    const id = `${m[1]}.${m[2]}`;
    nodes.push(id);
    const explicit = m[3].match(/linked\s*=\s*\[([^\]]*)\]/);
    const rest = m[3].replace(/linked\s*=\s*\[[^\]]*\]/, "");
    const addEdge = (target, kind) => {
      const a = `${id}->${target}`;
      if (!edges.has(a)) edges.set(a, { implicit: false, explicit: false });
      edges.get(a)[kind] = true;
    };
    for (const r of rest.matchAll(/\b([a-z_]+)\.([a-z_0-9]+)\.[a-z_]+\b/g))
      addEdge(`${r[1]}.${r[2]}`, "implicit");                   // an expression uses another resource's output
    if (explicit)
      for (const r of explicit[1].matchAll(/\b([a-z_]+)\.([a-z_0-9]+)\b/g))
        addEdge(`${r[1]}.${r[2]}`, "explicit");                  // a hand-written declaration
  }
  return { nodes, edges };
}

// Resources with resolved dependencies are created in the same round, i.e. in parallel.
export function levelize({ nodes, edges }) {
  const deps = new Map(nodes.map((d) => [d, []]));
  for (const a of edges.keys()) {
    const [k, h] = a.split("->");
    if (deps.has(k) && deps.has(h)) deps.get(k).push(h);
  }
  const level = new Map();
  let remaining = [...nodes], round = 0;
  while (remaining.length) {
    const ready = remaining.filter((d) => deps.get(d).every((h) => level.has(h)));
    if (!ready.length) return { cycle: remaining, level };
    ready.forEach((d) => level.set(d, round));
    remaining = remaining.filter((d) => !ready.includes(d));
    round++;
  }
  return { cycle: null, level };
}

export function measure(text, hidden = HIDDEN) {
  const g = parse(text), d = levelize(g);
  const counts = { implicit: 0, explicitOnly: 0, both: 0 };
  for (const v of g.edges.values())
    v.implicit && v.explicit ? counts.both++ : v.implicit ? counts.implicit++ : counts.explicitOnly++;
  if (d.cycle) return { g, d, counts, cycle: d.cycle };
  const race = hidden.filter(([a, b]) => d.level.get(a) <= d.level.get(b));
  const round = Math.max(...d.level.values()) + 1;
  const width = Math.max(...[...Array(round).keys()].map((t) =>
    [...d.level.values()].filter((x) => x === t).length));
  return { g, d, counts, race, round, width, cycle: null, lines: text.trim().split("\n").length };
}

if (import.meta.url === `file://${process.argv[1]}`) {
  const print = (title, r) => {
    console.log(`\n${title}`);
    console.log(`  resources ${r.g.nodes.length}, edges ${r.g.edges.size}, record ${r.lines} lines`);
    console.log(`  implicit (from expression) ${r.counts.implicit} | explicit only ${r.counts.explicitOnly}` +
      ` | both ${r.counts.both}`);
    console.log(`  ${r.round} rounds, widest round ${r.width} resources`);
    for (const [a, b] of HIDDEN)
      console.log(`  ${a.padEnd(18)} <- ${b.padEnd(21)} round ${r.d.level.get(a)} vs ` +
        `${r.d.level.get(b)}  ${r.d.level.get(a) <= r.d.level.get(b) ? "RACE" : "order correct"}`);
    console.log(`  race: ${r.race.length}/${HIDDEN.length}`);
  };
  print("A) record as-is — order comes only from implicit dependencies", measure(RECORD));

  const WITH_EXPLICIT = RECORD.replace(/ {2}boot     = "pull config from the object store"/g,
    '  boot     = "pull config from the object store"\n' +
    "  linked   = [object.collector_config, grant.machine_read]");
  print("B) explicit declaration added to both machines (2 lines)", measure(WITH_EXPLICIT));

  const WITH_CYCLE = WITH_EXPLICIT.replace(/ {2}data_address  = data_store.meter.address/,
    "  data_address  = data_store.meter.address\n  linked        = [machine.collector_1]");
  const c = measure(WITH_CYCLE);
  console.log("\nC) 'linked = [machine.collector_1]' added to the config object");
  const mutual = [...c.g.edges.keys()]                 // pair waiting on each other
    .filter((a) => c.g.edges.has(a.split("->").reverse().join("->")));
  console.log(`  still unresolved: ${c.cycle.length} resources`);
  console.log(`  cycle: ${mutual[0].split("->").join(" and ")} are waiting on each other`);
}

A) record as-is — order comes only from implicit dependencies
  resources 14, edges 23, record 59 lines
  implicit (from expression) 21 | explicit only 2 | both 0
  6 rounds, widest round 4 resources
  machine.collector_1 <- object.collector_config round 2 vs 3  RACE
  machine.collector_1 <- grant.machine_read    round 2 vs 1  order correct
  machine.collector_2 <- object.collector_config round 2 vs 3  RACE
  machine.collector_2 <- grant.machine_read    round 2 vs 1  order correct
  race: 2/4

B) explicit declaration added to both machines (2 lines)
  resources 14, edges 27, record 61 lines
  implicit (from expression) 21 | explicit only 6 | both 0
  8 rounds, widest round 3 resources
  machine.collector_1 <- object.collector_config round 4 vs 3  order correct
  machine.collector_1 <- grant.machine_read    round 4 vs 1  order correct
  machine.collector_2 <- object.collector_config round 4 vs 3  order correct
  machine.collector_2 <- grant.machine_read    round 4 vs 1  order correct
  race: 0/4

C) 'linked = [machine.collector_1]' added to the config object
  still unresolved: 6 resources
  cycle: object.collector_config and machine.collector_1 are waiting on each other

The first line gives the ratio: the record found twenty-one of twenty-three edges without anyone writing them. The subnet writing network.metrics.id is enough to say the network has to be created before the subnet; nobody said “set up the network first.” The main source of ordering is the declaration itself, and it is free — a byproduct of an expression that had to be written anyway.

The two edges that exist only through explicit declaration are meaningful too. The load balancer does not use the security group’s output in any of its fields, but a load balancer opened before the security rule exists drops traffic; the DNS record does not use the health check’s output either, but if the name is published before the check is set up, requests go to a broken node. Both are links with no data flow but a time relationship, and the only way to write them is by hand. This is exactly the blind spot of implicit inference.

The Requirement the Record Does not See

The next section gives the other side of the ratio. The record knows fourteen resources and twenty-three edges; in reality there are twenty-seven requirements. Four never entered the record, because they are written in free text, not in an expression: the machine pulls its config file on boot and needs a grant for it. The record sees 23 of 27 of reality — eighty-five percent.

Two of the four requirements it does not see resolved into the correct order by coincidence. The grant is created in round one because it derives from the object store and the role; the machines are in round two. Nobody wanted this, it just worked out that way. The other two did not work out: the config object drops into round three because it uses the data store’s and the queue’s addresses, while the machines boot in round two. Both machines try to pull a file that does not exist yet. This is not a bug, it is a race — the record left the order of two resources free, and the provider created them without an order.

The fix is two lines, and section B shows its cost: the race was zeroed out, the edge count rose from 23 to 27, but the round count went from six to eight, and the widest round’s parallelism dropped from four to three. Because the machines now wait on the config object, the graph’s longest path grew by two rounds. The price of the correct order is a longer apply and narrower parallelism.

Section C counts explicit declaration’s other risk. When a single linked line is written in the wrong direction on the config object, a cycle closes: the config object waits on the machine, the machine waits on the config object. The resolver cannot produce a single round, and six resources — the two in the cycle and the four resources depending on them — never get created at all. An implicit dependency cannot form a cycle, because an expression cannot read an output that does not exist yet; a cycle can only be built with a hand-written declaration. Explicit declaration closes what implicit inference cannot see, and in return it opens a new class of failure.

How Lifecycle Rules Change the Plan

// lifecycle.mjs — how lifecycle rules change the plan and the apply. Model.
const OP = { remove: 5, destroy: 10, create: 90, add: 5 };   // DP29: seconds

function scenario({ createFirst, parallel }) {
  const steps = [], service = [], assets = [];
  let cursor = 0;
  for (const m of ["collector_1", "collector_2"]) {
    let t = parallel ? 0 : cursor;
    if (createFirst) {
      steps.push([m, "create (new name)", t, OP.create]); assets.push([t, +1]); t += OP.create;
      steps.push([m, "add to target", t, OP.add]); service.push([t + OP.add, +1]); t += OP.add;
      steps.push([m, "remove (old)", t, OP.remove]); service.push([t + OP.remove, -1]); t += OP.remove;
      steps.push([m, "destroy (old)", t, OP.destroy]); assets.push([t + OP.destroy, -1]); t += OP.destroy;
    } else {
      steps.push([m, "remove from target", t, OP.remove]); service.push([t + OP.remove, -1]); t += OP.remove;
      steps.push([m, "destroy", t, OP.destroy]); assets.push([t, -1]); t += OP.destroy;
      steps.push([m, "create", t, OP.create]); assets.push([t, +1]); t += OP.create;
      steps.push([m, "add to target", t, OP.add]); service.push([t + OP.add, +1]); t += OP.add;
    }
    cursor = t;
  }
  const duration = Math.max(...steps.map(([, , t, s]) => t + s));
  let zero = 0, half = 0, mostAssets = 2;
  for (let t = 0; t <= duration; t++) {
    const h = 2 + service.filter(([a]) => a <= t).reduce((s, [, d]) => s + d, 0);
    const v = 2 + assets.filter(([a]) => a <= t).reduce((s, [, d]) => s + d, 0);
    if (h === 0) zero++; else if (h === 1) half++;
    mostAssets = Math.max(mostAssets, v);
  }
  return { steps: steps.length, duration, zero, half, mostAssets };
}

console.log("two collector machines are being recreated (machine type changed)");
console.log("rule                                 steps  time   zero capacity   half  most machines");
for (const [name, k] of [
  ["default, sequential", { createFirst: false, parallel: false }],
  ["default, parallel", { createFirst: false, parallel: true }],
  ["create_first, sequential", { createFirst: true, parallel: false }],
  ["create_first, parallel", { createFirst: true, parallel: true }],
]) {
  const r = scenario(k);
  console.log(`${name.padEnd(35)}${String(r.steps).padStart(7)}${String(r.duration).padStart(6)}` +
    `${String(r.zero).padStart(16)}${String(r.half).padStart(7)}${String(r.mostAssets).padStart(15)}`);
}

// DP30: the second change — the data store's region changed in the record; cannot update in place.
const CHANGE = [
  ["network.metrics", "update", false],
  ["data_store.meter", "recreate", true],        // carries prevent_destroy
  ["machine.collector_1", "recreate", false],
  ["machine.collector_2", "recreate", false],
  ["load_balancer.collector", "update", false],
  ["dns_record.collector", "update", false],
];

const blocked = CHANGE.filter(([, k, e]) => e && k === "recreate");
console.log(`\nplan: ${CHANGE.length} items — ` +
  `${CHANGE.filter(([, k]) => k === "recreate").length} recreate, ` +
  `${CHANGE.filter(([, k]) => k === "update").length} update`);
console.log(`items carrying the prevent_destroy rule: ${blocked.length} ` +
  `(${blocked.map(([a]) => a).join(", ")})`);
console.log(`applied without the rule: ${CHANGE.length}/${CHANGE.length} items, ` +
  `1 unrecoverable (41 days of meter readings)`);
console.log(`applied with the rule: 0/${CHANGE.length} items — the plan is rejected in full; ` +
  `${CHANGE.length - blocked.length} harmless items also wait`);

two collector machines are being recreated (machine type changed)
rule                                 steps  time   zero capacity   half  most machines
default, sequential                      8   220               0    210              2
default, parallel                        8   110             105      0              2
create_first, sequential                 8   220               0      0              3
create_first, parallel                   8   110               0      0              4

plan: 6 items — 3 recreate, 3 update
items carrying the prevent_destroy rule: 1 (data_store.meter)
applied without the rule: 6/6 items, 1 unrecoverable (41 days of meter readings)
applied with the rule: 0/6 items — the plan is rejected in full; 5 harmless items also wait

This table’s real finding is the distance between what the plan shows and what apply does. The plan writes two items: two machines will be recreated. Apply runs eight steps, and in the first row it runs at half capacity for 210 seconds. Neither the eight steps nor the 210 seconds show up in the plan; the plan states an item’s decision, not its cost.

The four rows scan two axes at once. Parallelism cuts the duration in half — from 220 seconds to 110 — but combined with the default rule it drops service entirely: 105 seconds of zero capacity, meaning nobody collects meter readings. The create_first rule rescues both: the old machine is not removed until the new one is ready and in the target, and the zero- and half-capacity columns stay empty. Its bill is in the last column: the number of machines standing at the same time rises from two to three, and to four under parallel apply. The price of recreating without downtime is temporarily doubled resources — and the new machine’s name has to be generated so its resource name does not collide.

The lower block measures the second rule. When the data store’s region changes in the record, the plan produces six items; three are recreations, one of them the data store itself. If that item is applied, the 41 days of meter readings are deleted, never to return. The prevent_destroy rule stops this, but not the one item — the entire plan: zero of the six items apply, and the five harmless items wait too. The record’s protection is deducted from the record’s fluidity.

Summary

• The record knows 14 resources and 23 edges; 21 of the 23 edges are implicit — dependency that is a byproduct of an expression, that nobody wrote. The main source of ordering is free.
• The 2 edges that exist only through explicit declaration are time relationships with no data flow; this is implicit inference’s blind spot, and the only way to write them is by hand.
• Real requirements: 27; what the record sees: 23 — 85 percent. 2 of the 4 unseen requirements landed in the correct order by coincidence; 2 produced a race — machines pulled a config file that did not exist yet.
• 2 lines of explicit declaration zeroed out the race; in return the round count went from 6 to 8, and the widest round’s parallelism from 4 to 3. A single declaration written in the wrong direction built a cycle, and 6 resources never got created.
• The plan says 2 items, apply runs 8 steps and runs at half capacity for 210 seconds; neither shows up in the plan. create_first zeroes out the downtime; its cost is 3 machines standing at once (4 under parallel).
• prevent_destroy saves the 41 days of data but rejects the plan in full: 0 of 6 items apply, and the 5 harmless items wait too.

Next Step

This lesson measured order and lifecycle, but looking at the record itself, one thing stands out: the two collector machines were written as two separate blocks, four lines identical letter for letter. When the region grows and the machine count reaches ten, will this block get copied ten times? The next lesson writes the same ten resources in three separate forms — by hand, with count-based iteration, with set-based iteration — and compares the three on the same items: how many lines, how many resources get recreated when an element is removed from the middle, and what readability loses.