--- title: 'The Reconciliation Loop' source: 'https://academia.sh/en/courses/orchestration/reconciliation-loop' course: 'Container Orchestration' language: en updated: '2026-09-13T05:08:26+00:00' license: 'CC BY-SA 4.0' objectives: - 'Writes a reconciliation loop that runs at cluster scale: reports, delayed reports, the gap, the correction' - 'Counts unneeded copies opened, copies deleted for nothing, and oscillation width as report delay grows' - 'Gives the relationship between loop period and report delay as a number, and shows its boundary' --- # The Reconciliation Loop The continuous comparison of desired state against actual state is written as a loop running at cluster scale, and report delay is scanned: how many unneeded copies the system opens, how many copies it deletes for nothing, and how wide the oscillation runs. The relationship between loop period and report delay is tested across fifty-six cells, and the missing information item is shown to be the system's own history. This topic's first three lessons measured two sides separately. Now the thing that connects them gets written: the continuous comparison of desired state against actual state, and closing the gap — the **reconciliation loop**. The loop itself is not new — a reconciler running on a single machine was examined earlier: it read the desired state from a file and the actual state from disk, touched only the differing item, and did zero work on its second run. That measurement is taken here as an input and is not repeated. At cluster scale, exactly one thing changes. Actual state no longer gets read from disk; it comes from **reports** arriving from the nodes, carrying the delay the previous lesson measured. This entire lesson counts the cost of that one difference: how many wrong corrections do correct decisions, made by looking at a delayed picture, produce. The measurement runs, once again, on the **regional measurement network**, and it is **fictional**; what runs is a **model** built with `node`, not a real cluster. **CA19.** Desired state is a single item: the collector service's copy count is six, and it comes from a human-written entry. **CA20.** Actual state comes from node reports; the ledger sits G seconds behind the real world. **CA21.** A created copy becomes running B seconds later, and the fact that it is running lands on the ledger G seconds after that; a deleted copy also disappears from the ledger G seconds later. **CA22.** The loop period is U; every round the ledger gets read, the gap gets computed, a correction gets written. **CA23.** Events are scripted, there is no randomness: two copies die at second 300; the run lasts 900 seconds. **CA24.** Two controllers are compared — one that looks only at the ledger, and one that also counts its own pending actions. **CA25.** A delete command is given to whatever the ledger shows; a command given to a copy that is already dead counts as work but does not change the live copy count. ## The Reconciliation Loop at Cluster Scale ```js // reconcile.mjs — the reconciliation loop at cluster scale. Desired state comes from the report, // actual state comes from delayed reports. Model; a real cluster does not run. No randomness, events are scripted. export const DEATHS = [[300, 2]]; // scripted event: 2 copies die at second 300 export function run({ desired = 6, U = 15, G = 0, B = 5, duration = 900, intent = false }) { const actual = []; // {birth, ready, death} let created = 0, commands = 0, deleted = 0, work = 0, round = 0, overshoot = 0, lastDrift = 0; const alive = (t) => actual.filter((r) => r.ready <= t && (r.death === null || r.death > t)); const seenAt = (t) => actual.filter((r) => // the ledger is G seconds behind r.ready + G <= t && (r.death === null || r.death + G > t)); for (let t = 0; t <= duration; t++) { for (const [at, count] of DEATHS) // scripted death: the oldest copies drop if (t === at) alive(t).slice(0, count).forEach((r) => { r.death = t; }); if (t % U === 0) { round++; const seen = seenAt(t); const pending = intent ? actual.filter((r) => r.death === null && r.ready + G > t).length : 0; const diff = desired - seen.length - pending; // intent: count its own pending actions if (diff > 0) { for (let i = 0; i < diff; i++) actual.push({ birth: t, ready: t + B, death: null }); created += diff; work += diff; } else if (diff < 0) { commands += -diff; work += -diff; // delete command: given to what the ledger shows seen.slice(0, -diff).forEach((r) => { if (r.death === null) { r.death = t; deleted++; } }); } } const c = alive(t).length; overshoot = Math.max(overshoot, c - desired); if (c !== desired) lastDrift = t; } const required = desired + DEATHS.reduce((s, k) => s + k[1], 0); return { round, workRound: work / round, created, unneeded: created - required, commands, deleted, overshoot, stable: lastDrift === duration ? null : lastDrift + 1 }; } const row = (lead, r) => `${lead}${r.workRound.toFixed(2).padStart(12)}${String(r.created).padStart(11)}` + `${String(r.unneeded).padStart(10)}${String(r.commands).padStart(17)}${String(r.deleted).padStart(9)}` + `${String(r.overshoot).padStart(11)}${(r.stable === null ? "none" : r.stable).toString().padStart(13)}`; if (import.meta.url === `file://${process.argv[1]}`) { console.log("U=15 sec loop, B=5 sec startup, desired 6 copies, 2 deaths at sec 300, 900 sec run"); for (const intent of [false, true]) { console.log(`\ncontroller: ${intent ? "ledger + its own pending actions" : "ledger only"}`); console.log("G(sec) work/round created unneeded delete commands deleted overshoot stable(sec)"); for (const G of [0, 5, 10, 20, 30, 60]) console.log(row(String(G).padStart(6), run({ G, intent }))); } } ``` ``` U=15 sec loop, B=5 sec startup, desired 6 copies, 2 deaths at sec 300, 900 sec run controller: ledger only G(sec) work/round created unneeded delete commands deleted overshoot stable(sec) 0 0.13 8 0 0 0 0 305 5 0.13 8 0 0 0 0 320 10 0.13 8 0 0 0 0 320 20 0.52 16 8 16 8 6 375 30 0.92 24 16 32 16 12 405 60 2.75 40 32 128 32 24 495 controller: ledger + its own pending actions G(sec) work/round created unneeded delete commands deleted overshoot stable(sec) 0 0.13 8 0 0 0 0 305 5 0.13 8 0 0 0 0 320 10 0.13 8 0 0 0 0 320 20 0.13 8 0 0 0 0 335 30 0.13 8 0 0 0 0 335 60 0.13 8 0 0 0 0 365 ``` The top table's first three rows are familiar. At a delay of zero, five, or ten seconds, the loop created eight copies — six for the initial setup, two to replace the dead ones — deleted none for nothing, and produced no overshoot at all. Work per round is 0.13: almost all of the sixty-one rounds did zero work. The inefficiency measured on a single machine holds at cluster scale too. At the fourth row, the loop's behavior changes. At a twenty-second report delay, the loop created sixteen copies; eight of them were unneeded. Eight copies got deleted for nothing, and the live copy count rose from six to twelve. At a sixty-second delay, the table falls apart: forty copies were created, thirty-two of them unneeded; a hundred twenty-eight delete commands were issued; overshoot hit twenty-four, meaning **thirty copies** ran at once at one point. What is worth noticing in these rows is that the controller never made a mistake. Every round it read the ledger, computed the gap correctly, and wrote exactly the correction needed to close it. The rule was right, the arithmetic was right, the input was **stale**. The copies were created, but because they had not been reported yet, the controller could not see them on the next round and **saw the same shortfall again**. The source of the wrong correction is not the decision rule but the age of the picture the decision rests on. The gap between delete commands and copies actually deleted is the same blindness's other face. At the sixty-second delay, 128 delete commands were issued, and only 32 of them landed on a live copy; the remaining 96 commands went to copies that were already dead. The system sees not only what it created, but also **what it deleted**, with a delay. On the measurement network, the counterpart of this is concrete: the six-copy collector service ran with thirty copies at one point, its database behind it took five times the connections, and the same meter-reading batches got pulled by more than one copy. The oscillation is not a display problem, it is real load. And this load leaves no trace at all in the controller's final line, which says "six copies wanted, six copies present"; once the oscillation closes, all that remains is a correct final state. ## The Relationship Between Loop Period and Report Delay ```js // relation.mjs — the relationship between loop period U and report delay G, with numbers. import { run } from "./reconcile.mjs"; const B = 5, DESIRED = 6; const predict = (G, U) => DESIRED * (Math.ceil((G + B) / U) - 1); // the relation to be tested console.log("G=30 sec fixed, B=5 sec; scanning the loop period"); console.log("U(sec) ceil((G+B)/U) predicted overshoot measured overshoot unneeded deleted work/round"); for (const U of [5, 10, 15, 20, 30, 35, 60]) { const r = run({ U, G: 30, B }); console.log(`${String(U).padStart(6)}${String(Math.ceil(35 / U)).padStart(15)}` + `${String(predict(30, U)).padStart(21)}${String(r.overshoot).padStart(20)}` + `${String(r.unneeded).padStart(10)}${String(r.deleted).padStart(9)}` + `${r.workRound.toFixed(2).padStart(12)}`); } console.log("\novershoot table (row: report delay G, column: loop period U)"); const U_VALUES = [5, 10, 15, 20, 30, 45, 60, 90]; console.log("G\\U " + U_VALUES.map((u) => String(u).padStart(5)).join("")); let mismatches = 0, cells = 0; for (const G of [0, 10, 20, 30, 45, 60, 90]) { const s = U_VALUES.map((U) => { const a = run({ U, G, B }).overshoot; cells++; if (a !== predict(G, U)) mismatches++; return String(a).padStart(5); }); console.log(String(G).padStart(3) + " " + s.join("")); } console.log(`\nrelation: overshoot = desired x (ceil((G+B)/U) - 1); ${cells} cells, ${mismatches} mismatched`); ``` ``` G=30 sec fixed, B=5 sec; scanning the loop period U(sec) ceil((G+B)/U) predicted overshoot measured overshoot unneeded deleted work/round 5 7 36 36 48 48 1.90 10 4 18 18 24 24 1.14 15 3 12 12 16 16 0.92 20 2 6 6 8 8 0.70 30 2 6 6 8 8 0.77 35 1 0 0 0 0 0.31 60 1 0 0 0 0 0.50 overshoot table (row: report delay G, column: loop period U) G\U 5 10 15 20 30 45 60 90 0 0 0 0 0 0 0 0 0 10 12 6 0 0 0 0 0 0 20 24 12 6 6 0 0 0 0 30 36 18 12 6 6 0 0 0 45 54 24 18 12 6 6 0 0 60 72 36 24 18 12 6 6 0 90 108 54 36 24 18 12 6 6 relation: overshoot = desired x (ceil((G+B)/U) - 1); 56 cells, 0 mismatched ``` The top table holds one delay (30 seconds) fixed and scans the loop period, and it shows the unexpected direction: **speeding up the loop makes the damage worse.** At a five-second period, overshoot is thirty-six and unneeded copies are forty-eight; at a thirty-five-second period, both are zero. A loop that reconciles often reads the same stale picture more times while the delay stays fixed, and tries to close the same shortfall again on every reading. The bottom table gives the relationship. Overshoot has this value in every cell measured: **overshoot = desired copies × (⌈(G + B) / U⌉ − 1)** None of the fifty-six cells deviate. The relationship reads in a direct way: `(G + B) / U` is **how many rounds** run in the time between a copy being created and it appearing on the ledger. The first of those rounds closes the real shortfall; each of the rest closes the same shortfall one more time. Once the number of blind rounds is one — that is, `U ≥ G + B` — overshoot zeroes out, and the table's zero region sits exactly above this diagonal. This is the sibling of the previous lesson's liveness-threshold rule. There, the threshold had to exceed the longest temporary outage plus one reporting period; here, the loop period has to exceed the report delay plus the startup time. In both cases, the cluster's setting depends on a duration the cluster does not measure itself, and in both cases a wrong setting carries no alarm — the oscillation runs silently. The relationship itself therefore carries operational value for this reason: it is possible to compute it knowing only three durations, without ever observing the overshoot. G is not a free setting here; the previous lesson measured it. The four-stage reporting chain averaged 13.53 seconds, worst case 26.36 seconds. With a five-second startup time, `G + B` comes out to 18.5 on average, 31.4 seconds in the worst case. A fifteen-second loop period sits below both of these numbers — meaning a cluster running with the measured delays sits in the oscillation zone **at its default setting**, with nothing broken. The table's `U=15, G=20` row is not an edge case; it is an ordinary day. The loop's base cost also shows up here. Work per round is 0.13, and most rounds do zero work, but a zero-work round is not free: every round reads the ledger from the top. The previous lesson's verification round counted this read at 26 items. The loop's cost grows not with the corrections it makes but with **the ratio of item count to period**; shrinking the period grows both the read load and — if the delay stays fixed — the overshoot. ## The Missing Item Is the System's Own History The bottom table of the first code block shows the second controller, and the result is sharp: the controller that counts its own pending actions created eight copies at every delay value, zero unneeded, zero deletions, 0.13 work per round. Overshoot is zero on every row. The added information item is a single line, and it has nothing to do with the outside world: **a record of what I created in the last few rounds.** The controller no longer says "I see four copies on the ledger instead of six"; it says "I see four copies on the ledger, and I created two of them two rounds ago myself — they may not have shown up yet." This is where the course's measure lands: the item added so that a delegated decision can be made correctly is, most of the time, not a measurement but **the system's memory of its own actions** — and this item comes close to free, because it asks no one for a report. It comes close to free, but it does not solve everything. Comparing the two tables' last column shows this: even for the controller with an intent record, the moment it turns stable grows with the delay — from 305 seconds to 365. A replacement for the two dead copies cannot open until the death shows up on the ledger. The distinction comes down to one sentence: **delay postpones noticing, an unrecorded intent multiplies the work.** These are two separate problems; the second closes with one line, the first does not — it can only be reduced, expensively, by shrinking the reporting period and the network delay — and, as the previous lesson showed, the bill for shrinking those periods comes back as reporting load per node. ## Summary - At report delays of zero, 5, and 10 seconds, the loop created 8 copies, deleted none for nothing, and almost all of the 61 rounds did zero work; the inefficiency measured on a single machine holds at cluster scale too. - At a 20-second delay, 8 unneeded copies got opened and 8 copies got deleted for nothing, with the live count rising from 6 to 12; at 60 seconds, 32 unneeded copies, 128 delete commands, and an oscillation of 30 live copies resulted. - Only 32 of 128 delete commands landed on a live copy; the system sees both what it created and what it deleted with a delay. - Overshoot came out to `desired × (⌈(G+B)/U⌉ − 1)` across every scan, with zero deviation across 56 cells. The boundary where overshoot zeroes out is `U ≥ G + B` — the loop period exceeding the report delay plus the startup time. - Speeding up the loop made the damage worse: at a 30-second delay, overshoot was 36 at a 5-second period and 0 at a 35-second period. - The controller that counted its own pending actions produced 0 unneeded copies and 0 wasted deletions at every delay; the moment it turned stable still rose from 305 to 365 seconds — delay postpones noticing, an unrecorded intent multiplies the work. ## Next Step This lesson measured the loop itself: what it reads, how often it runs, how much wrong work it does by looking at a delayed picture. The topic's final lesson returns to an operational decision — whether you will run this plane yourself, or delegate its operation to someone else. What gets measured is the two options compared on the same items: how many operational items get delegated, which items remain, the version-upgrade window, the loss of access to the control plane, and how reversible the decision is.