State Splitting

The lock rescued correctness but also built a queue: everyone writing to the same state file waits. The only way to shrink the area a lock covers is to split the state. This lesson puts the numbers from before and after splitting side by side, then works out the bill for splitting.

Infrastructure Growing in a Single State

The fictional measurement network grew over the course of a year. SM13. The model builds a structure that can be split into thirteen parts: network, data, compute and observability layers in each of the three environments, plus a regional backbone shared across all three.

// network-gen.mjs — the grown regional measurement network: thirteen parts, items and dependency edges.
// Output is written in two forms: a single state file, and one state file per part.
import { writeFileSync, mkdirSync, rmSync } from "node:fs";

const environments = ["test", "staging", "production"];
const layers = ["network", "data", "compute", "observability"];
const parts = [{ name: "shared", layer: "shared", count: 20 }];
for (const e of environments)
  for (const l of layers) parts.push({ name: `${e}_${l}`, env: e, layer: l, count: 50 });

// How many items a layer pulls information from other layers through.
const crossDeps = {
  network: [["shared", 2]], data: [["network", 3]],
  compute: [["network", 6], ["data", 4]], observability: [["compute", 5], ["data", 2]],
};
const targetPart = (p, layer) => (layer === "shared" ? "shared" : `${p.env}_${layer}`);

const items = [], innerEdges = [], crossEdges = [];
for (const p of parts) {
  for (let i = 0; i < p.count; i++) {
    items.push({ name: `${p.name}/i${i}`, part: p.name });
    if (i > 0) innerEdges.push([`${p.name}/i${i}`, `${p.name}/i${i - 1}`]);
  }
  for (const [layer, count] of crossDeps[p.layer] ?? [])
    for (let i = 0; i < count; i++)
      crossEdges.push([`${p.name}/i${i}`, `${targetPart(p, layer)}/i${i}`]);
}

rmSync("split", { recursive: true, force: true });
mkdirSync("split");
const item = (k) => ({ ...k, deps: [...innerEdges, ...crossEdges].filter((e) => e[0] === k.name).map((e) => e[1]) });
writeFileSync("single-state.json", JSON.stringify({ serial: 1, items: items.map(item) }));
for (const p of parts)
  writeFileSync(`split/${p.name}.json`, JSON.stringify(
    { serial: 1, items: items.filter((k) => k.part === p.name).map(item) }));

// Reality on the provider: the items in the record, plus three out-of-record resources in the production compute layer.
const provider = [...items.map((k) => k.name),
  "production_compute/manual-1", "production_compute/manual-2", "production_compute/manual-3"];
writeFileSync("provider.json", JSON.stringify(provider));

console.log(`parts: ${parts.length}, items: ${items.length}, ` +
  `inner edges: ${innerEdges.length}, cross-part edges: ${crossEdges.length}`);
console.log(`single state file: 1 x ${items.length} items | split: ` +
  `${parts.length} files, largest ${Math.max(...parts.map((p) => p.count))} items`);
console.log(`${provider.length} resources on provider, ${items.length} in record ` +
  `-> out of record ${provider.length - items.length}`);

parts: 13, items: 620, inner edges: 607, cross-part edges: 66
single state file: 1 x 620 items | split: 13 files, largest 50 items
623 resources on provider, 620 in record -> out of record 3

Six hundred twenty items, six hundred seventy-three edges. Only sixty-six of the edges cross a part boundary — the rest stay inside their parts. This ratio says why splitting is possible at all: ninety percent of the links are already local.

The last line sets up a difference that stays constant through the rest of the lesson: there are six hundred twenty-three resources on the provider, six hundred twenty in the record. Three resources are in no part’s state at all. Splitting does not change this number — infrastructure the record does not cover stays out of scope no matter how many files the record is split into. The only effect it will feel below is which connection style accidentally pulls those three items back in.

Plan Duration and Blast Radius

A plan refreshes every item in the record from the provider and compares it against the definition. SM14. One refresh read per item takes one millisecond; the real duration depends on the provider and network latency, but it stays linear in the number of items.

The measurement rule is the same as the previous lesson’s: the measured duration is not printed, only whether it falls between 0.7 and 2.0 times the expected value.

// plan.mjs — reads the state of the given scope, refreshes each item, and measures plan duration.
// Everything printed is run-independent: the refresh count and the expected duration derived
// from it, plus a band check on the measured duration. Raw milliseconds are not printed.
// usage: node plan.mjs single | node plan.mjs part <name>
import { readFileSync } from "node:fs";
const REFRESH_MS = 1, LOW = 0.7, HIGH = 2.0;     // modeled cost of a refresh read, and the band
const wait = (ms) => { const b = Date.now() + ms; while (Date.now() < b); };
const [scope, name] = [process.argv[2] ?? "single", process.argv[3]];
const file = scope === "single" ? "single-state.json" : `split/${name}.json`;

const t0 = Date.now();
const d = JSON.parse(readFileSync(file, "utf8"));
for (const k of d.items) wait(REFRESH_MS);       // one refresh read per item
const measured = Date.now() - t0;
const expected = d.items.length * REFRESH_MS;

console.log(`${(scope === "single" ? "single state" : name).padEnd(24)} | ` +
  `${String(d.items.length).padStart(3)} refreshes, expected plan ${String(expected).padStart(3)} ms | ` +
  `measured in band: ${measured >= LOW * expected && measured <= HIGH * expected ? "yes" : "no"} | ` +
  `items locked ${String(d.items.length).padStart(3)}`);

The work done is the same in both scopes: changing a single item in the observability layer.

node plan.mjs single
node plan.mjs part production_observability

single state             | 620 refreshes, expected plan 620 ms | measured in band: yes | items locked 620
production_observability |  50 refreshes, expected plan  50 ms | measured in band: yes | items locked  50

The carrier of the measurement is the refresh count; that number depends not on the run but on the scope’s item count. Changing one item in the single state requires six hundred twenty refreshes, fifty in the split form. Against one changed item, the wasted-read ratio is above ninety-nine percent. Expected plan duration is derived from this number — six hundred twenty milliseconds against fifty — and the measurement stayed inside the band in both scopes; duration tracks the refresh count directly.

Blast radius is sharper still, and does not depend on time at all: the lock covers all six hundred twenty items. Someone adding a threshold to the observability layer holds all three environments’ network, data store and archive locked for the duration of the plan.

In split form, the same work refreshes fifty items and locks fifty items. Plan duration drops to one-twelfth, blast radius drops from one hundred percent to eight percent. Everyone touching the other twelve parts can work at the same time; the concurrency ceiling that fell to one in the previous lesson rises to thirteen.

The difference accumulates in daily work. If the team triggers an apply twenty times a day, total plan duration in the single state climbs to twelve seconds, and all of that time passes under lock; in split form, most of the same twenty applies land on different parts, so each one is both shorter and does not wait on the others. The real meaning of the measurement is not duration, it is how many people are waiting: in the single state, every apply keeps the whole team waiting; in split form, only the people touching the same part.

Another effect is the readability of the plan output. Finding the one changed item in a plan that refreshed six hundred twenty items is harder than finding it in a plan of fifty; drift gets lost in the same pile. As scope shrinks, the odds that the plan actually gets read go up, and a read plan is the one thing that can still be stopped before it is applied.

The Cost of Splitting

Splitting does not eliminate dependencies, it makes them visible. Each of the sixty-six edges that cross a part boundary now has to be built between two separate state files.

// dependency.mjs — counts cross-part edges, how they are resolved, and reapply propagation.
import { readFileSync, readdirSync } from "node:fs";

const parts = readdirSync("split").map((f) => f.replace(".json", ""));
const state = {}, itemPart = new Map();
for (const p of parts) {
  state[p] = JSON.parse(readFileSync(`split/${p}.json`, "utf8")).items;
  for (const k of state[p]) itemPart.set(k.name, p);
}
// Parts that publish outputs: they broadcast their values by name. Others can only be queried.
const outputParts = new Set(parts.filter((p) => /(^shared$|_network$|_data$)/.test(p)));

const edges = [];
for (const p of parts)
  for (const k of state[p])
    for (const b of k.deps) { const h = itemPart.get(b); if (h !== p) edges.push([p, h]); }
const byOutput = edges.filter(([, h]) => outputParts.has(h)).length;

console.log(`cross-part dependencies: ${edges.length} edges, ` +
  `${new Set(edges.map((e) => e.join(">"))).size} distinct part pairs`);
console.log(`  resolved by reading an output: ${byOutput}, ` +
  `resolved by data source: ${edges.length - byOutput}`);
const incoming = {};
for (const [, h] of edges) incoming[h] = (incoming[h] ?? 0) + 1;
console.log(`  most connected: ` + Object.entries(incoming).sort((a, b) => b[1] - a[1])
  .slice(0, 3).map(([p, n]) => `${p} ${n} edges`).join(", "));

// A data source returns everything matching the query: the record's uncovered resource too.
const provider = JSON.parse(readFileSync("provider.json", "utf8"));
const returned = provider.filter((a) => a.startsWith("production_compute/")).length;
const recorded = state["production_compute"].length;
console.log(`  data source query (production_compute): ${returned} items returned, ` +
  `recorded ${recorded}, out of record ${returned - recorded}`);
console.log(`  same link built by reading an output: ${recorded} items, out of record 0`);

const dependents = {};                           // target part -> parts that depend on it
for (const [k, h] of edges) (dependents[h] ??= new Set()).add(k);
const propagate = (p) => {
  const seen = new Set([p]), stack = [p];
  while (stack.length) {
    const x = stack.pop();
    for (const y of dependents[x] ?? []) if (!seen.has(y)) { seen.add(y); stack.push(y); }
  }
  return seen;
};
for (const p of ["production_observability", "production_data", "production_network", "shared"]) {
  const g = propagate(p);
  const items = [...g].reduce((t, x) => t + state[x].length, 0);
  console.log(`  ${p.padEnd(24)} changes: ${String(g.size).padStart(2)} parts reapplied, ` +
    `${String(items).padStart(3)} items refreshed, ${String(g.size).padStart(2)} locks taken`);
}

SM15. The network and data layers, along with the regional backbone, publish their values by name as outputs; the compute layer does not publish, and connecting to it requires a data source query. SM16. Three resources the record does not cover sit in the production compute layer.

cross-part dependencies: 66 edges, 18 distinct part pairs
  resolved by reading an output: 51, resolved by data source: 15
  most connected: production_network 9 edges, staging_network 9 edges, test_network 9 edges
  data source query (production_compute): 53 items returned, recorded 50, out of record 3
  same link built by reading an output: 50 items, out of record 0
  production_observability changes:  1 parts reapplied,  50 items refreshed,  1 locks taken
  production_data          changes:  3 parts reapplied, 150 items refreshed,  3 locks taken
  production_network       changes:  4 parts reapplied, 200 items refreshed,  4 locks taken
  shared                   changes: 13 parts reapplied, 620 items refreshed, 13 locks taken

Sixty-six edges spread across eighteen part pairs. Fifty-one are resolved by reading an output: the target part publishes a value by name, the connecting part reads that name. The link is recorded and both sides are known; its cost is a version bond — if the target part removes an output, every edge that depends on it breaks at the same time, and the break is only learned at apply time.

The distribution of edges says something too: the three most connected parts are the network layers of the three environments, and each carries nine edges. These three parts are the split’s breaking point. A naming change in the network layer’s output list is written in a single part, felt across nine edges, and forces three parts to reapply. Output names are therefore the parts’ interface; resource names inside a part can change freely, output names cannot.

The remaining fifteen are resolved by data source: the target part has no idea it is being connected to, the connecting side queries the provider instead. The measurement shows the cost of this path: the query returned fifty-three items, and fifty are in the record. Three items are resources the record does not cover, and the data source pulls them in too. If the same link had been built by reading an output, the returned item count would be fifty, out of record zero. A query reads reality, an output reads the record; the two diverge by exactly these three items.

Reapply Propagation

The last four lines are the real bill for splitting. SM17. When a part’s output changes, the parts that depend on it reapply, and this is transitive; whoever depends on the dependent gets in line too. Changing the observability part forces one part to reapply; changing the network part forces four; changing the regional backbone forces all thirteen.

The last line deserves a careful read: when the shared part changes, six hundred twenty items refresh — the exact same number as the single state file — but this time thirteen separate locks are taken and thirteen applies are put in a queue. Splitting does not cheapen a wide change; it makes it more expensive. In the single state, one apply did everything in the right order; in split form, the person doing the work is now the one who has to build that order.

From this comes a rule: where to split is chosen by frequency of change. A layer that changes often should stay in its own part so blast radius stays small; a layer that changes rarely and that everyone connects to should be collected into a single part so its propagation is not triggered often. Split in the wrong place and two harms combine: the count of cross-part edges grows, and every change propagates to many parts.

The Multiplication of the Record

Splitting multiplies the record too, and that is the record’s own risk. One file became thirteen: thirteen locks, thirteen access points, thirteen separate loss scenarios. The previous lesson’s measurement applies to each of them separately. Losing one part’s state only makes that part unmanageable — which is a gain — but the loss becomes harder to notice, because the other twelve parts keep applying without a problem.

The second risk is the boundary itself. The part boundary is written down in no state file; which item belongs to which part is visible only in how the files are distributed. If an item ends up in two parts’ definitions by mistake, two records claim ownership of the same resource, and the two applies take turns undoing each other’s work. The lock does not prevent this: two different files have two different locks.

The third risk is changing the boundary. Splitting is not a one-time act; over time an item turns out to sit in the wrong part and has to be moved. Moving is not copying the definition from one file to another. The moment the item is removed from the old part’s definition, that part’s plan shows it as to be deleted; the new part’s plan shows it as to be created. Both plans are correct on their own, but if they are applied one after the other, the resource is really deleted and recreated. The right way is to remove the item’s record from the old state by id and add it to the new state by id — the reverse and continuation of the import from the first lesson. Because this touches two files at once, it needs both locks at the same time, and if it is caught in between, the item shows up in no record at all: the count of resources the record does not cover rises from three to four.

Summary

• Of the six-hundred-twenty-item network’s six hundred seventy-three edges, only sixty-six cross a part boundary; ninety percent of the links are already local.
• In the single state, a one-item change refreshed six hundred twenty items and locked six hundred twenty items; in split form, the same work refreshed fifty items and locked fifty items.
• Of the sixty-six cross-part edges, fifty-one are resolved by reading an output, fifteen by data source; the data source query returned fifty-three items, three of which were outside the record.
• The number of parts that reapply when one part changes ranges from one to thirteen; changing the regional backbone requires thirteen locks and six hundred twenty refreshes.
• Splitting shrinks the lock, grows coordination; where to split is chosen by frequency of change.
• Thirteen files mean thirteen loss scenarios, and the part boundary is written down in no state file.

Next Step

This lesson’s split was made by layer: network, data, compute, observability. Environment was also in the part names, but how the environment distinction itself is built was never examined. If the definition of the three environments is the same, does the same text have to be written three times, or does a single definition keep three instances? Both setups have a cost, and it collects in the same place: whether the person starting the apply knows which environment they are in. The next lesson builds the three environments in two separate forms, measures the risk of applying to the wrong environment by step count, counts environment-specific differences, and compares how many environments are forced to change together when the definition changes.