Multi-Cluster Management

Every number in the previous lesson belonged to a single cluster. A cluster is a single failure unit — one control plane, one state store, one upgrade calendar — and this topic’s second lesson measured that once the entire plane goes down, the cluster keeps running but becomes unable to decide.

Multi-cluster operation splits a workload across more than one cluster; its axis is environment, region, or tenant. The delegated decision moves up a level: which cluster a workload runs on gets decided not by a human but by an upper layer. The measurement is again over the regional measurement network and is fictional; no real cluster is run, and what is counted is a model built with node.

• OO55. The workload splits into eighteen cells: three environments, two regions, three tenants.
• OO56. Operating hours are an input from the cluster-architecture topic: 67 hours per cluster monthly, 39 hours belonging to the workload, 3 windows per cluster over thirty-six months.
• OO57. The desired state lives in a single store; each cluster pulls it on its own 15-round period. The pull moment and network lag are hand-written.
• OO58. One of six clusters’ admission control rejects the new declaration; that cluster stays drifted.
• OO59. The upper-layer model carries 24 workloads and 6 clusters; seed 20260901 is visible in the output.
• OO60. The outcome splits into three classes: correct decision, wrong decision, undecided.

Split Axis, Propagation, and Failure Class

// split.mjs — 18 workload cells split along different axes; fixed cost, propagation, failure class.
const ENV = 3, REGION = 2, CELL = ENV * REGION * 3;
const CLUSTER_HOURS = 67, WORKLOAD_HOURS = 39, UPGRADE = 3;   // 33+34 per cluster, 39 belongs to the workload
const AXIS = [["single cluster", 1], ["region", REGION], ["environment", ENV],
  ["env x region", ENV * REGION], ["env x region x tenant", CELL]];
const pad = (x, n) => String(x).padStart(n);

console.log(`${CELL} cells = ${ENV} environments x ${REGION} regions x 3 tenants`);
console.log("axis            clusters  cells  blast    hrs/mo  windows/36mo");
for (const [name, n] of AXIS) {
  const h = Math.ceil(CELL / n);
  console.log(name.padEnd(24) + pad(n, 4) + pad(h, 7) + pad(`${((100 * h) / CELL).toFixed(1)}%`, 9) +
    pad(WORKLOAD_HOURS + n * CLUSTER_HOURS, 12) + pad(n * UPGRADE, 17));
}

const ROUND = 120, PERIOD = 15, CHANGE = 10, READS = 26;   // 26: reads of an empty verification round
const offset = (i) => (i * 4) % PERIOD, lag = (i) => [0, 1, 2, 1, 2, 0][i % 6];   // hand-written
const REJECT = 3;                                    // this cluster's admission control rejects
function propagate(n) {
  const pull = [...Array(n).keys()].map((i) => {
    if (i % 6 === REJECT) return null;                        // drifted cluster: cannot apply at all
    let t = CHANGE;
    while ((t - offset(i)) % PERIOD !== 0) t++;
    return t + lag(i);
  });
  const arrived = pull.filter((x) => x !== null);
  const partial = [...Array(ROUND - CHANGE).keys()]
    .filter((k) => { const y = arrived.filter((x) => x <= CHANGE + k).length; return y > 0 && y < n; }).length;
  const convRounds = n * Math.floor(ROUND / PERIOD);
  return [Math.min(...arrived), Math.max(...arrived), partial, n - arrived.length, convRounds, convRounds * READS];
}
console.log("\ndesired state in a single store; change at round 10, period 15 rounds");
console.log("clus  first  last  window  partial rounds  drifted  conv rounds  reads");
for (const [, n] of AXIS) {
  const [i, s, k, sp, t, o] = propagate(n);
  console.log(pad(n, 4) + pad(i, 5) + pad(s, 5) + pad(s - i, 9) + pad(k, 19) + pad(sp, 7) + pad(t, 14) + pad(o, 16));
}

const FAILURE = [["control plane", 1], ["node loss", 1], ["version upgrade", 1],
  ["admission rule change", 1], ["bad declaration", 0], ["shared data store", 0]];
const isolated = FAILURE.filter((a) => a[1]);
console.log(`\nfailure classes ${FAILURE.length}: isolated by splitting ${isolated.length} (affected cells ${CELL}` +
  ` -> ${CELL / 2}); not isolated ${FAILURE.length - isolated.length} — ` +
  FAILURE.filter((a) => !a[1]).map((a) => a[0]).join(", ") + ` — ${CELL}/${CELL} in both clusters`);

18 cells = 3 environments x 2 regions x 3 tenants
axis            clusters  cells  blast    hrs/mo  windows/36mo
single cluster             1     18   100.0%         106                3
region                     2      9    50.0%         173                6
environment                3      6    33.3%         240                9
env x region               6      3    16.7%         441               18
env x region x tenant     18      1     5.6%        1245               54

desired state in a single store; change at round 10, period 15 rounds
clus  first  last  window  partial rounds  drifted  conv rounds  reads
   1   15   15        0                  0      0             8             208
   2   15   20        5                  5      0            16             416
   3   15   25       10                 10      0            24             624
   6   15   25       10                105      1            48            1248
  18   12   25       13                108      3           144            3744

failure classes 6: isolated by splitting 4 (affected cells 18 -> 9); not isolated 2 — bad declaration, shared data store — 18/18 in both clusters

The first table’s two columns move in opposite directions. Blast radius — the share of cells affected by losing one cluster — falls from a hundred percent to 5.6 percent; monthly fixed hours climb from 106 to 1,245, and the upgrade window from three to fifty-four. Splitting by region carries the radius to fifty percent and the load to 173 hours, splitting by environment to 33.3 percent and 240 hours: the only difference is the axis — the region axis buys failure isolation, the environment axis buys change isolation.

In the second table, the propagation window is zero for a single cluster, ten rounds for six clusters, and across those ten rounds the set of clusters is partly new, partly old. The real number is in the partial-success column: for six clusters it is not ten but a hundred five, because one cluster’s admission control rejects the declaration. A single drifted cluster turns a temporary window into a permanent split. The convergence workload gets multiplied too: 208 item reads climb to 3,744.

The last row gives the place multi-cluster does not pay off: four of six failure classes get isolated, two do not. A shared data store, and a bad declaration that leaves the single store and spreads to both clusters at once, keep stopping eighteen of eighteen cells. Splitting divides the failure unit; it does not divide the data layer.

Delegating the Decision to an Upper Layer

// upper-layer.mjs — the upper layer deciding which cluster a workload lands on, and what it does not know.
const ALL = ["measurement", "field", "reporting"];
const CLUSTER = [                            // name, environment, region, capacity, tenants accepted
  ["dev-r1", "development", "r1", 34, ALL], ["dev-r2", "development", "r2", 34, ALL],
  ["stg-r1", "staging", "r1", 36, ALL], ["stg-r2", "staging", "r2", 36, ALL],
  ["prod-r1", "production", "r1", 50, ["measurement", "field"]], ["prod-r2", "production", "r2", 26, ["reporting"]]];

const r = ((t) => () => (t = (t * 48271) % 2147483647) / 2147483647)(20260901);   // seed visible
const ENV = ["development", "staging", "production"];
const WORKLOAD = Array.from({ length: 24 }, (_, i) => ({ env: ENV[i % 3],
  region: r() < 0.5 ? "r1" : "r2", tenant: ALL[Math.floor(r() * 3)], units: 4 + Math.floor(r() * 9) }));

function run({ age = 0, regionKnown = true, tenantKnown = true }) {
  const filled = Object.fromEntries(CLUSTER.map(([a]) => [a, 0]));
  let ledger = { ...filled };
  const s = { correct: 0, full: 0, distant: 0, violation: 0, undecided: 0 };
  WORKLOAD.forEach((y, i) => {
    if (i % (age + 1) === 0) ledger = { ...filled };            // the ledger goes stale by `age` steps
    let candidate = CLUSTER.filter(([, e]) => e === y.env);
    if (tenantKnown) candidate = candidate.filter(([, , , , k]) => k.includes(y.tenant));
    if (regionKnown) candidate = [...candidate].sort((a, b) => (b[2] === y.region) - (a[2] === y.region));
    const chosen = candidate.find(([a, , , k]) => ledger[a] + y.units <= k);
    if (!chosen) { s.undecided++; return; }                     // no decision could be made
    const [name, , region, cap, tenants] = chosen;
    if (filled[name] + y.units > cap) { s.full++; return; }     // real capacity was not enough
    filled[name] += y.units;
    if (!tenants.includes(y.tenant)) s.violation++;
    else if (region !== y.region) s.distant++;
    else s.correct++;
  });
  return s;
}

const SCENARIO = [["full information", {}], ["capacity stale by 8 placements", { age: 8 }],
  ["region not reported", { regionKnown: false }], ["tenant constraint not reported", { tenantKnown: false }],
  ["all three missing", { age: 8, regionKnown: false, tenantKnown: false }]];
const pad = (x, n) => String(x).padStart(n);
console.log(`${WORKLOAD.length} workloads, ${CLUSTER.length} clusters (3 env x 2 regions); seed 20260901`);
console.log("scenario                        correct  wrong  undecided  full  distant  violation");
for (const [name, se] of SCENARIO) {
  const o = run(se);
  console.log(name.padEnd(31) + pad(o.correct, 5) + pad(o.full + o.distant + o.violation, 8) + pad(o.undecided, 12) +
    pad(o.full, 6) + pad(o.distant, 6) + pad(o.violation, 7));
}
console.log(`items: ${WORKLOAD.length} workloads x 3 + ${CLUSTER.length} clusters x 2 = ${WORKLOAD.length * 3 + CLUSTER.length * 2}` +
  `; cluster capacity is the only one that is measured`);

24 workloads, 6 clusters (3 env x 2 regions); seed 20260901
scenario                        correct  wrong  undecided  full  distant  violation
full information                  18       3           3     0     3      0
capacity stale by 8 placements    18       6           0     3     3      0
region not reported               13      10           1     0    10      0
tenant constraint not reported    18       4           2     0     0      4
all three missing                 13      11           0     0     8      3
items: 24 workloads x 3 + 6 clusters x 2 = 84; cluster capacity is the only one that is measured

The full-information row is not flawless: eighteen of twenty-four workloads place correctly, three land in a distant region, and three go undecided — the first from capacity, the second from the tenant constraint.

The capacity ledger going stale produces a familiar shift. Once the ledger is eight placements stale, wrong decisions rise from three to six and undecided drops from three to zero: the aging ledger shows the pool as emptier than it is, so the upper layer stops saying “no room” and picks a full cluster instead.

Two reported items are measured separately. When region is not reported, correct decisions drop to thirteen and distant placement rises to ten; none get rejected, the workloads run and only exceed the latency limit — a silent class. When the tenant constraint is not reported, four workloads go to a cluster that does not accept them, and this wrong decision is noisy, because the cluster’s admission control catches it. With all three missing at once, undecided drops to zero: an upper layer that knows nothing never hesitates. Only six of the eighty-four items are measured.

Summary

• As 18 cells split from 1 cluster to 18, blast radius falls from 100% to 5.6%; monthly operation climbs from 106 to 1,245 hours, and the upgrade window from 3 to 54.
• The propagation window is 0 for a single cluster, 10 rounds for six; one rejecting drifted cluster pushes partial success to 105 rounds. Convergence load multiplies with cluster count: 208 item reads climb to 3,744.
• 4 of 6 failure classes get isolated by splitting; a shared data store and a bad declaration spreading from the single store affect 18 of 18 cells in both clusters.
• With full information, the upper layer places 18 of 24 workloads correctly; once capacity is stale by 8 placements, wrong decisions go from 3 to 6 and undecided from 3 to 0, and when region goes unreported, 10 workloads silently land in a distant region.

Course Wrap-Up

Thirty rows are one rule’s worked examples: delegating a decision requires producing the information that decision rests on, continuously and on time; every item the system does not know produces a class of wrong decision. Four sources repeat in the last column. Aging information appears in ten rows: the number at decision time lags behind reality, and as the lag grows, undecided turns into wrong. A condition left out of scope appears in eight rows: the system behaves correctly where it looks, and oscillates or drifts silently where it does not. An unreported item appears in seven rows, an unverified declaration in five — and a wrongly written declaration usually looks like an improvement. The system does not make the wrong decision because it lacks information; it makes it because there was no item it could have known. This is the delegation cost that has been this course’s unit of measure: the gain shows up in step count, the cost accumulates in information items.

One question is left, and M22/K04, Continuous Integration and Delivery, takes it up. The cluster applies the desired state: it reads the declaration, closes the gap, runs it through admission control, checks privilege. But who changes that desired state, looking at what, and passed through which check? A human writes every one of the thirty rows’ declaration items; for all the cluster’s rigor, nowhere does it test the correctness of the declaration it was handed.