Node Components

The previous lesson’s last scenario was where a single assumption broke: once a report could not be recorded, the ledger kept counting a dead copy as running. That assumption — that nodes correctly and promptly report what they are running — sits underneath every decision the control plane makes. This lesson splits the node side into its parts and measures the report itself.

There are three parts. The node agent is the node’s face toward the ledger: it reads the copies assigned to it, requests that they run, and reports the result back. The runtime is the part that actually runs the output; its isolation and resource limits were measured in the previous course, and here it stands only as the party that returns status to the node agent. The network proxy holds, on the node, the rule set that translates a service name into the copies currently running.

The measurement runs, once again, on the regional measurement network, and it is fictional; what runs is not a real cluster but a model built with node.

CA13. The node side is three parts: the node agent, the runtime, the network proxy. CA14. The node agent reports once every 10 seconds; a report carries 3 fixed items per node (health, total and used capacity) and 2 items per copy (status, resource usage). CA15. The share of items that actually change in one period is 0.02. CA16. The liveness threshold is E: a node with E seconds since its last report is considered unknown, and the copies on it are declared dead and re-created. CA17. In one day there are 4 permanent node failures and 30 temporary reporting outages (5–90 seconds); we wrote the generator ourselves, and the seed 20260403 appears in the output. CA18. The network proxy reads the ledger every 5 seconds; the rule count per node is the sum of the service entry and the copy routing rules, and the request rate per copy is 80 per second.

What the Node Agent Reports

// node.mjs — the information the node side reports and its load at cluster scale. Model.
export const P = 10;                     // reporting period (seconds)
const FIXED = 3;                         // health, total capacity, used capacity
export const items = (copies) => FIXED + 2 * copies;   // per copy: status + resource usage
export const CHANGE_RATE = 0.02;             // the share of items that actually change in one period

if (import.meta.url === `file://${process.argv[1]}`) {
  console.log("nodes  copies    reports/node  items/sec   daily items  writes/sec   changed-only/sec");
  for (const M of [1, 3, 10, 30, 100, 300]) {
    const copies = 8, k = items(copies), itemsSec = (M * k) / P;
    console.log(`${String(M).padStart(5)}${String(copies).padStart(8)}${String(k).padStart(16)}` +
      `${itemsSec.toFixed(1).padStart(11)}${(itemsSec * 86400).toFixed(0).padStart(14)}` +
      `${(M / P).toFixed(1).padStart(12)}${(itemsSec * CHANGE_RATE).toFixed(2).padStart(19)}`);
  }
}

nodes  copies    reports/node  items/sec   daily items  writes/sec   changed-only/sec
    1       8              19        1.9        164160         0.1               0.04
    3       8              19        5.7        492480         0.3               0.11
   10       8              19       19.0       1641600         1.0               0.38
   30       8              19       57.0       4924800         3.0               1.14
  100       8              19      190.0      16416000        10.0               3.80
  300       8              19      570.0      49248000        30.0              11.40

The content of a report is small: a node with eight copies sends 19 items every period. What creates the load is not the content but the frequency. In a three-hundred-node cluster, 570 items get reported per second, 49 million per day; to do this, nodes produce 30 writes per second.

The last column gives the real number. Of the 570 items reported, only 11.4 have actually changed — two percent. The remaining ninety-eight percent is the cost of saying that nothing changed. This measure is the node side’s counterpart to the previous lesson’s verification round: there, the control plane read 26 items for 0 writes; here, nodes send 558 items because nothing changed at all.

The place where cutting this load looks easy is a trap. Reporting only what changed would divide the item count by fifty, but the cluster’s liveness decision rests on the existence of the report, not its content. A silent node and a node saying “nothing changed” look identical on the ledger; the only thing that tells them apart is whether the report arrived at all. This is why a report gets sent even when it is empty, and most of the reporting load is paid not for the information itself, but for the proof of it.

The scope of the report is a separate limit. The node agent asks the runtime whether the copy is up or not and passes along the answer it gets; it does not ask whether the copy is doing its job. On the measurement network, a collector copy could be up and processing no meter readings at all — the process is alive, resource usage is low, the report says running. On the ledger this copy is healthy, the network proxy sends it traffic, and the controller manager does nothing because it sees no missing copy. All three components work correctly, and the outcome is wrong, because the item the decision depends on is not in the report. Widening the scope — asking the copy to report its own state separately — adds this item, but brings a separate question along with it: who produces that report, how often, and by looking at what.

How Long the Cluster Believes the Old Fact

// stale-truth.mjs — once a report is lost, how long the cluster believes the old fact.
import { P } from "./node.mjs";
const generator = (seed) => { let s = seed >>> 0;          // our own generator; seed visible
  return () => (s = (s * 1664525 + 1013904223) >>> 0) / 4294967296; };
const DAY = 86400, COPIES = 3, REQUEST = 80, V = 5;   // copies per node, requests/sec per copy,
const rnd = generator(20260403);                      // V: the network proxy's store-read period
const failure = Array.from({ length: 4 }, () => rnd() * DAY);            // permanent node failure
const outage = Array.from({ length: 30 }, () => ({ t: rnd() * DAY, d: 5 + rnd() * 85 }));

console.log(`longest outage ${Math.max(...outage.map((k) => k.d)).toFixed(1)} sec` +
  `   reporting period ${P} sec   proxy read period ${V} sec`);
console.log("threshold     avg-detect    worst   wrong-dead-decl  wasted-copies  lost-requests");
for (const E of [20, 30, 40, 60, 90, 100, 120, 300]) {
  const g = failure.map((t) => Math.floor(t / P) * P + E - t);           // duration of the stale fact
  const wrong = outage.filter((k) => Math.floor(k.t / P) * P + E < k.t + k.d).length;
  const lost = g.reduce((s, x) => s + (x + V) * COPIES * REQUEST, 0);
  console.log(`${String(E).padStart(9)}${(g.reduce((a, b) => a + b) / g.length).toFixed(1).padStart(15)}` +
    `${Math.max(...g).toFixed(1).padStart(9)}${String(wrong).padStart(18)}` +
    `${String(wrong * COPIES).padStart(15)}${lost.toFixed(0).padStart(15)}`);
}

longest outage 84.6 sec   reporting period 10 sec   proxy read period 5 sec
threshold     avg-detect    worst   wrong-dead-decl  wasted-copies  lost-requests
       20           17.3     19.9                26             78          21408
       30           27.3     29.9                21             63          31008
       40           37.3     39.9                19             57          40608
       60           57.3     59.9                13             39          59808
       90           87.3     89.9                 2              6          88608
      100           97.3     99.9                 0              0          98208
      120          117.3    119.9                 0              0         117408
      300          297.3    299.9                 0              0         290208

The first column is how long the cluster is wrong. Whatever the threshold, the cluster learns of a failure at least E minus one period, at most E seconds later: at a 40-second threshold, it counts a dead node as alive for an average of 37.3 seconds. This duration cannot be zeroed out, because there is no such thing as a failure reporting itself — the cluster’s only evidence is a report that does not arrive, and deciding when something that has not arrived has failed to arrive requires waiting.

The second and third columns are the cost of shortening the wait. Once the threshold is lowered to 20 seconds, twenty-six of thirty temporary outages were mistaken for permanent failures; the cluster wastefully declared 78 copies dead and opened replacements — even though the old copies had not died, the node came back a few seconds later. This number falls as the threshold grows: 13 at 60 seconds, 2 at 90 seconds, zero at 100 seconds.

Where the zero falls is not a coincidence. The longest outage is 84.6 seconds and the reporting period is 10 seconds; wrong declarations end at the first step where the threshold exceeds the sum of the two. The right threshold is derived not from the cluster’s own measurements but from the network’s behavior — from the longest tail of a distribution the cluster neither knows nor measures. If outages one day run 130 seconds instead of 84.6, the 100-second threshold quietly starts producing wrong declarations again, and the cluster has no alarm for this.

The last column is the other half of the payment. Because the network proxy’s rule only updates once a copy is declared dead, requests arriving before then get sent to a copy that is not running. At the 100-second threshold that zeroes out wrong declarations, 98,208 requests went to dead copies over the course of a day; at the 20-second threshold that number was 21,408. The choice between the two ends is not a setting but a split between two wrongs: a fast-acting cluster kills healthy copies, a patient cluster sends traffic to dead ones. No threshold zeroes out both, and the node holds no information that could zero out both at once. On the measurement network, the counterpart of this is concrete — 98 thousand meter-reading requests go unanswered, and the nightly batch job starts with incomplete data.

Where the Reporting Delay Accumulates

Between a copy going down and that fact landing on the ledger and turning into a decision, there are four separate waits, and each one imposes its own period.

// delay.mjs — one event's reporting delay, split into four parts. Each stage adds its own wait.
import { P } from "./node.mjs";
const generator = (seed) => { let s = seed >>> 0;
  return () => (s = (s * 1664525 + 1013904223) >>> 0) / 4294967296; };
const rnd = generator(20260404), N = 20000;
const C = 2, WRITE = 0.05, U = 15;        // runtime poll, store write, reconciliation period
const stage = [["runtime poll", C], ["node agent report", P],
  ["API server + store write", 0], ["controller manager round", U]];

let total = new Array(N).fill(0);
console.log("stage                          period  avg(sec)   worst(sec)  cumulative avg(sec)");
for (const [name, period] of stage) {
  const share = total.map(() => (period ? rnd() * period : WRITE));
  total = total.map((x, i) => x + share[i]);
  console.log(`${name.padEnd(29)}${(period || WRITE).toFixed(2).padStart(8)}` +
    `${(share.reduce((a, b) => a + b) / N).toFixed(2).padStart(10)}` +
    `${Math.max(...share).toFixed(2).padStart(13)}` +
    `${(total.reduce((a, b) => a + b) / N).toFixed(2).padStart(21)}`);
}
console.log(`events ${N}   worst end-to-end ${Math.max(...total).toFixed(2)} sec` +
  `   stage count ${stage.length}`);

stage                          period  avg(sec)   worst(sec)  cumulative avg(sec)
runtime poll                     2.00      1.00         2.00                 1.00
node agent report               10.00      4.99        10.00                 5.99
API server + store write         0.05      0.05         0.05                 6.04
controller manager round        15.00      7.49        15.00                13.53
events 20000   worst end-to-end 26.36 sec   stage count 4

The average end-to-end delay is 13.53 seconds, the worst is 26.36 seconds. The distribution is instructive: 1.00 second of the delay comes from the runtime’s poll, 4.99 seconds from the node agent’s reporting period, 0.05 seconds from the write operation, and 7.49 seconds from the controller manager’s round. So more than half the delay sits not on the node side but in the control plane’s own period.

This split determines which setting actually helps. Halving the node agent’s reporting period lowers the average by 2.5 seconds and doubles the reporting load; halving the controller manager’s round period lowers it by 3.75 seconds and puts no load at all on the nodes, but doubles the ledger’s read load. The same delay is sold at two separate places and two separate prices; which setting is cheap depends on where the cluster is running tight.

The Network Proxy’s Rule Count

// proxy.mjs — how the network proxy's rule count grows with service count; where change spreads.
const M = 30, r = 5, V = 5, CHANGE = 200;   // node, copies/service, read period, copies/day
const rule = (S) => S * (1 + r);               // per node: service entry + copy routing rules

console.log(`nodes ${M}   copies/service ${r}   proxy read period ${V} sec`);
console.log("service  copies   rules/node  cluster-rules   single-change   items-read/sec  daily-writes");
for (const S of [2, 4, 8, 16, 32, 64]) {
  console.log(`${String(S).padStart(7)}${String(S * r).padStart(8)}${String(rule(S)).padStart(13)}` +
    `${String(M * rule(S)).padStart(15)}${String(M).padStart(16)}` +
    `${((M * S * r) / V).toFixed(0).padStart(17)}${String(CHANGE * M).padStart(14)}`);
}

nodes 30   copies/service 5   proxy read period 5 sec
service  copies   rules/node  cluster-rules   single-change   items-read/sec  daily-writes
      2      10           12            360              30               60          6000
      4      20           24            720              30              120          6000
      8      40           48           1440              30              240          6000
     16      80           96           2880              30              480          6000
     32     160          192           5760              30              960          6000
     64     320          384          11520              30             1920          6000

Two columns grow along two separate axes. Rule count grows with service count: 12 rules per node at two services, 384 at sixty-four. The cluster-wide total is this multiplied by node count and reaches 11,520 rules, because the same rule set sits separately on each of the thirty nodes. The proxy’s ledger-reading load grows along the same axis too: from 60 items per second to 1,920.

The spread of a change, however, is independent of service count. One copy being born or dying, whatever the service count, produces thirty rule writes across thirty nodes — because every node needs to be able to reach that service. Two hundred copy changes per day come out to 6,000 rule writes, and that number stays the same even if service count grows thirty-two times over.

The distinction matters operationally: adding a new service grows the static cost (rule count, memory, read load); adding a node, or changing copies often, grows the flowing cost (rule writes, the propagation window). The lost-request count from the previous section also accumulates within this window: once a copy is declared dead, updating the rule is delayed further by the proxy’s read period. This five seconds closes separately on each of the thirty nodes, at different moments; a rule change does not reach the cluster at a single instant but spreads across a window.

Summary

• A node with eight copies reports 19 items every period; across 300 nodes this comes to 49 million items a day, and only 11.4 items per second of that has actually changed — the remaining ninety-eight percent is paid not for information but for proof that the node is still up.
• An event turning into a decision passes through four waits and takes an average of 13.53 seconds, at worst 26.36; 7.49 seconds of that comes not from the node but from the controller manager’s round period.
• The cluster learns of a failure at least the threshold minus one period, at most the full threshold, later; at a 40-second threshold it counts a dead node as alive for an average of 37.3 seconds, and this duration cannot be zeroed out.
• Once the threshold was lowered to 20 seconds, 26 of 30 temporary outages were mistaken for permanent failures and 78 copies were wastefully opened; wrong declarations zeroed out only at 100 seconds, once the threshold exceeded the sum of the longest outage (84.6 sec) and one reporting period.
• At the threshold that zeroed out wrong declarations, requests sent to dead copies rose to 98,208; at the 20-second threshold it was 21,408 — the choice between the two ends is a split between two wrongs.
• The network proxy’s rule count grows with service count (12 → 384 per node, 11,520 cluster-wide), while the spread of a change is independent of service count: one copy change produces 30 rule writes across 30 nodes, 6,000 a day.

Next Step

Two lessons measured two sides separately: the control plane produces the decision, the nodes report the state. What connects the two has not yet been named — the continuous comparison of desired state against actual state, and closing the gap. The next lesson is this topic’s center, and it genuinely writes the reconciliation loop at cluster scale: the convergence round, work per round, and the question this lesson leaves behind — as reporting delay grows, how many copies does the loop open unnecessarily, how many does it delete for nothing, and what relationship holds between the loop’s period and the reporting delay.