Remote State and Locking

The previous lesson left the state file in a single directory, in the hands of a single process. That assumption holds for a one-person team. It collapses the moment a second person arrives: two applies read the same state, both create resources, both write to the same file. This lesson actually produces that collision, counts its cost, and measures how much of it a lock resolves.

The optimistic and pessimistic solutions for concurrent updates were measured in The Data Access Layer and Business Logic course; the mechanics there are not repeated here. The difference is granular: there, what was locked was a row, and two clients writing to different rows never met; here, what is locked is the entire state file. If three environments sit in a single file, the person adding a machine to the test environment waits on the person growing the production queue; the concurrency ceiling drops to one.

Remote State Store and Two Appliers

The script below models a single apply: it reads the state, runs for the duration of the apply, adds its own items, and writes the state back. SM7. The remote state store is modeled with a directory; both processes access the same store/state.json file. SM8. Apply duration is modeled with a busy loop; in reality that time passes in calls made to the provider.

// applier.mjs — models a single apply against a remote state store.
// modes: setup | apply <name> <count> <lock|nolock> <duration_ms> [wait_deadline_ms] | summary
import { readFileSync, writeFileSync, appendFileSync, mkdirSync, rmSync,
  openSync, closeSync, unlinkSync, existsSync } from "node:fs";

const STORE = "store", STATE = "store/state.json", LOCK = "store/lock.json";
const mode = process.argv[2];
const wait = (ms) => { const b = Date.now() + ms; while (Date.now() < b); }; // model of apply duration

if (mode === "setup") {
  rmSync(STORE, { recursive: true, force: true });
  mkdirSync(STORE);
  writeFileSync(STATE, JSON.stringify({ serial: 1, items: [] }));
  process.exit(0);
}

if (mode === "summary") {
  const d = JSON.parse(readFileSync(STATE, "utf8"));
  const owner = {};
  for (const k of d.items) owner[k[0]] = (owner[k[0]] ?? 0) + 1;
  const distribution = Object.keys(owner).sort().map((a) => `${a}=${owner[a]}`).join(" ") || "none";
  const g = existsSync(`${STORE}/log.txt`) ? readFileSync(`${STORE}/log.txt`, "utf8") : "";
  for (const s of g.trim().split("\n").sort()) console.log(s);
  const created = [...g.matchAll(/(\d+) items written/g)].reduce((t, m) => t + Number(m[1]), 0);
  console.log(`state: ${d.items.length} items (${distribution}), serial ${d.serial}, ` +
    `lock ${existsSync(LOCK) ? "held" : "none"}`);
  console.log(`actually created ${created} resources, visible in record ${d.items.length} ` +
    `-> out of record ${created - d.items.length}`);
  process.exit(0);
}

if (mode !== "apply") {
  console.log("usage: setup | apply <name> <count> <lock|nolock> <duration_ms> | summary");
  process.exit(0);
}

const [name, count, lockMode, durationMs] =
  [process.argv[3], Number(process.argv[4]), process.argv[5], Number(process.argv[6])];
let round = 0;
if (lockMode === "lock") {
  const deadline = Date.now() + Number(process.argv[7] ?? 10000);
  for (;;) {
    try {                                        // "wx": errors if the file exists, atomic
      const f = openSync(LOCK, "wx");
      writeFileSync(f, JSON.stringify({ owner: name, at: "fictional-timestamp" }));
      closeSync(f); break;
    } catch {
      round++;
      if (Date.now() > deadline) { appendFileSync(`${STORE}/log.txt`,
        `${name}: lock not acquired, ${round} waiting rounds\n`); process.exit(1); }
      wait(25);
    }
  }
}
const d = JSON.parse(readFileSync(STATE, "utf8"));
writeFileSync(`${STORE}/${name}-read`, "");
if (lockMode === "nolock")                       // let both applies read before either writes
  while (!existsSync(`${STORE}/a-read`) || !existsSync(`${STORE}/b-read`)) wait(5);
wait(durationMs);
for (let i = 1; i <= count; i++) d.items.push(`${name}-item-${i}`);
d.serial += 1;
writeFileSync(STATE, JSON.stringify(d));
if (lockMode === "lock") unlinkSync(LOCK);
appendFileSync(`${STORE}/log.txt`,
  `${name}: ${count} items written, read serial ${d.serial - 1}, waiting rounds ${round}\n`);

Two people are adding resources to the fictional measurement network’s staging environment at the same time: a adds twelve items, b adds twelve items. SM9. They are synchronized with marker files to make sure both read before either writes; because b’s apply duration is sixty milliseconds longer, it is the one that writes last.

# nolock.sh — two applies change the same remote state at the same time, no lock
node applier.mjs setup
node applier.mjs apply a 12 nolock 0 &
node applier.mjs apply b 12 nolock 60 &
wait
node applier.mjs summary

a: 12 items written, read serial 1, waiting rounds 0
b: 12 items written, read serial 1, waiting rounds 0
state: 12 items (b=12), serial 2, lock none
actually created 24 resources, visible in record 12 -> out of record 12

Three numbers show three separate harms. The first is the lost write: both applies wrote twelve items each, but twelve items are left in the file; a’s write was overwritten by b. The second is the gap between record and reality: a really did create its resources, there are twenty-four resources on the provider, and the record says twelve — twelve resources fell outside the record and will now neither be planned nor deleted. The third is the inconsistent state: even though two applies ran, the serial number went up by one; both say “read serial 1.” The record stands as if only a single apply had run.

The Lock

The solution is to acquire the right to write before reading the state. The lock is acquired through the filesystem’s atomic “error if it exists” creation: even if two processes try at the same instant, only one can create the file. The other catches the error, waits twenty-five milliseconds, and tries again.

# lock.sh — the same two applies, this time with the state lock
node applier.mjs setup
node applier.mjs apply a 12 lock 250 &
until [ -f store/lock.json ]; do :; done   # a took the lock
node applier.mjs apply b 12 lock 250 &
wait
node applier.mjs summary

a: 12 items written, read serial 1, waiting rounds 0
b: 12 items written, read serial 2, waiting rounds 10
state: 24 items (a=12 b=12), serial 3, lock none
actually created 24 resources, visible in record 24 -> out of record 0

Twenty-four items, serial three, zero out of record. b spent ten waiting rounds — the two-hundred-fifty-millisecond hold divided by the twenty-five-millisecond poll — and read the current state: “read serial 2.” The trade-off between the lost write and waiting is plain — the only cost the lock produced is b’s two hundred fifty milliseconds, and what it bought is twelve resources entering the record.

The lock file is deleted after the apply: the lock is not part of the state, it is the gate to accessing the state.

Acquiring the lock before the read is not an arbitrary order, either. When b acquires the lock, it rereads the state and sees serial two; if it had read first and then tried to acquire the lock, it would have been left holding serial one and would have erased a’s twelve items the moment it wrote. The result from the lockless measurement comes back if the lock is acquired in the wrong place. There is one order: lock, read, apply, write, release.

Scanning Lock Hold Time

Waiting lasts exactly as long as the lock is held. If the apply takes longer, everyone behind it waits that much longer.

Measurement rule: the measured duration is not printed, it is only run through a band check — between 0.7 and 2.0 times the expected value. The lower bound confirms the queue actually forms; the upper bound is left wide because the twenty-five-millisecond poll grain and the process-startup overhead can approach a multiple of the expected value at short hold times.

// scan.mjs — how the queue for four applies grows as lock hold time grows.
// Everything printed is run-independent: the expected values come from queue arithmetic,
// the measured duration is only run through a band check and printed as yes/no.
import { spawn } from "node:child_process";
import { readFileSync } from "node:fs";

const run = (...a) => new Promise((c) =>
  spawn("node", ["applier.mjs", ...a], { stdio: "ignore" }).on("exit", c));
const N = 4, POLL = 25, LOW = 0.7, HIGH = 2.0;
console.log(`queue arithmetic (${N} applies, run-independent): ` +
  `total wait = ${(N * (N - 1)) / 2} x hold, longest = ${N - 1} x hold`);
console.log(`band: measured must fall between ${LOW}-${HIGH}x expected`);

const accuracy = [];
for (const hold of [50, 200, 800]) {
  await run("setup");
  await Promise.all(["a", "b", "c", "d"].map(
    (name) => run("apply", name, "12", "lock", String(hold))));
  const rounds = readFileSync("store/log.txt", "utf8").trim().split("\n")
    .map((s) => Number(/waiting rounds (\d+)/.exec(s)[1]));
  const d = JSON.parse(readFileSync("store/state.json", "utf8"));
  accuracy.push(`${d.items.length} items, serial ${d.serial}`);

  const expTotal = ((N * (N - 1)) / 2) * hold, expLongest = (N - 1) * hold;
  const inBand = (measured, expected) =>
    measured >= LOW * expected && measured <= HIGH * expected ? "yes" : "no";
  console.log(`hold ${String(hold).padStart(3)} ms | expected total ` +
    `${String(expTotal).padStart(4)} ms, longest ${String(expLongest).padStart(4)} ms | ` +
    `measured in band: ${inBand(rounds.reduce((x, y) => x + y, 0) * POLL, expTotal)} / ` +
    `${inBand(Math.max(...rounds) * POLL, expLongest)}`);
}
console.log(`accuracy (run-independent): ` +
  `${new Set(accuracy).size === 1 ? `${accuracy[0]} at all three hold times` : accuracy.join(" | ")}`);

queue arithmetic (4 applies, run-independent): total wait = 6 x hold, longest = 3 x hold
band: measured must fall between 0.7-2x expected
hold  50 ms | expected total  300 ms, longest  150 ms | measured in band: yes / yes
hold 200 ms | expected total 1200 ms, longest  600 ms | measured in band: yes / yes
hold 800 ms | expected total 4800 ms, longest 2400 ms | measured in band: yes / yes
accuracy (run-independent): 48 items, serial 5 at all three hold times

The first line comes not from measurement but from the queue itself: when four applies line up, the first waits not at all, the second waits one, the third two, the fourth three lock hold times. The total is six, the longest is three hold times. The lesson’s claim rests on this arithmetic — queue length is directly proportional to apply duration — and the total wait, three hundred milliseconds at a fifty-millisecond hold, climbs to four thousand eight hundred at eight hundred: sixteen times the hold, sixteen times the wait.

The measurement only confirms this: yes at all three hold times. The source of deviation is a fixed overhead — process startup and the poll grain — and its share shrinks as the hold grows; in this run, the eight-hundred-millisecond row measured about two percent above expected.

The last line is also run-independent: forty-eight items, serial five. The lock does not give correctness for free, it gives it in exchange for a queue; the length of that queue can be calculated before it is ever measured.

From this comes an operating rule: long applies keep an entire team waiting. The way to shorten the queue is not to make the lock faster, but to shrink the area the lock covers.

The Stale Lock

If the process holding the lock dies, the lock file is left behind. SM10. The setup below builds this directly: the lock file is written, and its owner disappears without ever writing.

// stale-lock.mjs — the lock a process leaves behind when it dies without releasing it, and opening it by hand.
import { writeFileSync, readFileSync, existsSync, unlinkSync, rmSync, mkdirSync } from "node:fs";
import { spawnSync } from "node:child_process";

rmSync("store", { recursive: true, force: true });
mkdirSync("store");
writeFileSync("store/state.json", JSON.stringify({ serial: 1, items: [] }));
// a took the lock and died before writing; the lock file was left behind.
writeFileSync("store/lock.json", JSON.stringify({ owner: "a", at: "fictional-timestamp" }));

const lastLine = () => readFileSync("store/log.txt", "utf8").trim().split("\n").at(-1);
const tryB = () => spawnSync("node",
  ["applier.mjs", "apply", "b", "12", "lock", "0", "1000"]).status;

console.log(`stale lock: ${readFileSync("store/lock.json", "utf8")}`);
console.log(`b tried (exit ${tryB()}) -> ${lastLine()}`);
console.log(`state: ${JSON.parse(readFileSync("store/state.json", "utf8")).items.length} items`);

unlinkSync("store/lock.json");                   // opening it by hand
console.log(`lock opened by hand; does the file still exist: ${existsSync("store/lock.json")}`);
console.log(`b tried again (exit ${tryB()}) -> ${lastLine()}`);
console.log(`state: ${JSON.parse(readFileSync("store/state.json", "utf8")).items.length} items`);
console.log(`fields in the lock record: 2 (owner, at) | field saying whether the owner is still running: 0`);

stale lock: {"owner":"a","at":"fictional-timestamp"}
b tried (exit 1) -> b: lock not acquired, 42 waiting rounds
state: 0 items
lock opened by hand; does the file still exist: false
b tried again (exit 0) -> b: 12 items written, read serial 1, waiting rounds 0
state: 12 items
fields in the lock record: 2 (owner, at) | field saying whether the owner is still running: 0

b waited forty-two rounds, filled the one-second deadline, and managed to write zero items. The lock does not open on its own; the process that would release it no longer exists. From here on, this is the record’s own risk: even though the infrastructure was never touched, it is unmanageable, and the only way out is to open the lock by hand.

Opening it by hand takes two steps, but the person making the call has only two fields to go on: the owner’s name and a timestamp. The number of fields that say whether the owner is still running is zero. If the call is wrong — if the owner is in the middle of a long apply — the lock loses its meaning and the measurement from the first section comes back: twelve items are lost, twelve resources fall out of record.

The Difference Between a Local File and a Remote Store

For the lock to work, everyone has to be looking at the same state. If state sits on each applier’s own disk, there is nothing shared to lock.

// local.mjs — state sitting on each applier's own disk: two records, one reality.
import { writeFileSync, readFileSync, rmSync, mkdirSync } from "node:fs";

rmSync("local", { recursive: true, force: true });
mkdirSync("local");
const actual = [];
for (const name of ["a", "b"]) {                 // two people worked on their own copy
  const own = [];
  for (let i = 1; i <= 12; i++) { own.push(`${name}-item-${i}`); actual.push(`${name}-item-${i}`); }
  writeFileSync(`local/state-${name}.json`, JSON.stringify({ serial: 2, items: own }));
}
const read = (name) => JSON.parse(readFileSync(`local/state-${name}.json`, "utf8")).items;
const a = read("a"), b = read("b");
const pct = (n) => `${Math.round((100 * n) / actual.length)}%`;

console.log(`actual resources on provider: ${actual.length}`);
console.log(`local file: 2 -> state-a ${a.length} items (coverage ${pct(a.length)}), ` +
  `state-b ${b.length} items (coverage ${pct(b.length)})`);
console.log(`shared items: ${a.filter((k) => b.includes(k)).length}, ` +
  `file that knows all of it: 0, sum of both files: ${a.length + b.length}`);
console.log(`if a drift scan runs with state-a, appears out of record: ` +
  `${actual.filter((k) => !a.includes(k)).length} items (all false)`);
console.log(`with a remote store: 1 file, ${actual.length} items, coverage 100%, false out-of-record 0`);

actual resources on provider: 24
local file: 2 -> state-a 12 items (coverage 50%), state-b 12 items (coverage 50%)
shared items: 0, file that knows all of it: 0, sum of both files: 24
if a drift scan runs with state-a, appears out of record: 12 items (all false)
with a remote store: 1 file, 24 items, coverage 100%, false out-of-record 0

SM11. Two people open twelve resources each with no communication between them. The result is two separate records: each is internally consistent, and neither knows more than half of reality. There is no lost write — because there is no shared file — but the gap is worse: because both records are correct within their own scope, no one notices that anything is wrong. If a drift scan runs against state-a, it declares twelve items out of record; all twelve are false alarms, the resources do have an owner, the owner is just another file.

This is the one thing a remote state store gives: a single record. Coverage rises to one hundred percent, an object exists to lock, and the false-out-of-record count drops to zero. SM12. The store assumes a service that can make file creation atomic; without that guarantee, there is no lock either.

The gain does not come for free, and its cost is the record’s own risk. Two local files were two separate points of failure; one remote record is one point of failure. If a write that broke off halfway corrupts the file, the previous lesson’s numbers come back: twenty-six items are recovered by importing, three cannot be. If the store keeps no versions, there is no copy to fall back to either. The second risk is access: the file no longer sits on one person’s disk, it sits somewhere the whole team can read, and all eighty field values counted in the previous lesson are there.

Summary

• Two lockless concurrent applies wrote twelve items each, and twelve items are left in the record: one write was lost, twelve resources fell out of record, and the serial went up by one despite two applies.
• A lock built with atomic file creation produced twenty-four items, serial three, and zero out-of-record on the same measurement; its cost is the second apply’s ten waiting rounds.
• The queue for four applies waits six lock hold times regardless of the run, with the longest wait at three; the measurement stayed inside the band at all three hold times, and correctness never changed.
• A process that died without releasing the lock produced forty-two waiting rounds that filled the one-second deadline and zero items written; the lock record has no field saying whether the owner is alive.
• Keeping state in a local file produced two records, both stuck at fifty percent coverage, and a drift scan produced twelve false out-of-record results; the remote store brought coverage to one hundred percent.

Next Step

The lock rescued correctness but brought a cost: if the area the lock covers is a single file, everyone writing to that file gets in line. If the entire measurement network sits in one state, someone who wants to change a security rule waits on the apply of someone scaling the data store. The next lesson splits this single file: it measures plan duration and how many items a single change locks in infrastructure that runs on one state, repeats the same measurements in the split form, and counts the cost of splitting — the dependencies between the pieces.