--- title: 'Multi-Cluster Management' source: 'https://academia.sh/en/courses/orchestration/multi-cluster-management' course: 'Container Orchestration' language: en updated: '2026-10-21T05:03:04+00:00' license: 'CC BY-SA 4.0' objectives: - 'Compares the split axis by two numbers: blast radius and fixed cost per cluster' - 'Counts the propagation window, partial success, and drifted clusters in cross-cluster convergence' - "Measures the upper layer's missing information in three classes: correct decision, wrong decision, undecided" --- # Multi-Cluster Management Splitting by region and environment is tied to numbers: for every split axis, cluster count, fixed operating hours, upgrade window, and blast radius are all measured together. When the desired state is pushed from one place to many clusters, the propagation window, partial success, and drifted-cluster count are counted; an upper layer that assigns workloads to clusters is modeled, and which wrong decision it makes because of what it does not know is shown. Every number in the previous lesson belonged to **a single cluster**. A cluster is a single failure unit — one control plane, one state store, one upgrade calendar — and this topic's second lesson measured that once the entire plane goes down, the cluster keeps running but becomes **unable to decide**. **Multi-cluster operation** splits a workload across more than one cluster; its axis is environment, region, or tenant. The delegated decision moves up a level: **which cluster a workload runs on** gets decided not by a human but by an upper layer. The measurement is again over the **regional measurement network** and is **fictional**; no real cluster is run, and what is counted is a **model** built with `node`. - **OO55.** The workload splits into eighteen cells: three environments, two regions, three tenants. - **OO56.** Operating hours are an input from the `cluster-architecture` topic: 67 hours per cluster monthly, 39 hours belonging to the workload, 3 windows per cluster over thirty-six months. - **OO57.** The desired state lives in a single store; each cluster pulls it on its own 15-round period. The pull moment and network lag are **hand-written**. - **OO58.** One of six clusters' admission control rejects the new declaration; that cluster stays drifted. - **OO59.** The upper-layer model carries 24 workloads and 6 clusters; seed `20260901` is visible in the output. - **OO60.** The outcome splits into three classes: correct decision, wrong decision, undecided. ## Split Axis, Propagation, and Failure Class ```js // split.mjs — 18 workload cells split along different axes; fixed cost, propagation, failure class. const ENV = 3, REGION = 2, CELL = ENV * REGION * 3; const CLUSTER_HOURS = 67, WORKLOAD_HOURS = 39, UPGRADE = 3; // 33+34 per cluster, 39 belongs to the workload const AXIS = [["single cluster", 1], ["region", REGION], ["environment", ENV], ["env x region", ENV * REGION], ["env x region x tenant", CELL]]; const pad = (x, n) => String(x).padStart(n); console.log(`${CELL} cells = ${ENV} environments x ${REGION} regions x 3 tenants`); console.log("axis clusters cells blast hrs/mo windows/36mo"); for (const [name, n] of AXIS) { const h = Math.ceil(CELL / n); console.log(name.padEnd(24) + pad(n, 4) + pad(h, 7) + pad(`${((100 * h) / CELL).toFixed(1)}%`, 9) + pad(WORKLOAD_HOURS + n * CLUSTER_HOURS, 12) + pad(n * UPGRADE, 17)); } const ROUND = 120, PERIOD = 15, CHANGE = 10, READS = 26; // 26: reads of an empty verification round const offset = (i) => (i * 4) % PERIOD, lag = (i) => [0, 1, 2, 1, 2, 0][i % 6]; // hand-written const REJECT = 3; // this cluster's admission control rejects function propagate(n) { const pull = [...Array(n).keys()].map((i) => { if (i % 6 === REJECT) return null; // drifted cluster: cannot apply at all let t = CHANGE; while ((t - offset(i)) % PERIOD !== 0) t++; return t + lag(i); }); const arrived = pull.filter((x) => x !== null); const partial = [...Array(ROUND - CHANGE).keys()] .filter((k) => { const y = arrived.filter((x) => x <= CHANGE + k).length; return y > 0 && y < n; }).length; const convRounds = n * Math.floor(ROUND / PERIOD); return [Math.min(...arrived), Math.max(...arrived), partial, n - arrived.length, convRounds, convRounds * READS]; } console.log("\ndesired state in a single store; change at round 10, period 15 rounds"); console.log("clus first last window partial rounds drifted conv rounds reads"); for (const [, n] of AXIS) { const [i, s, k, sp, t, o] = propagate(n); console.log(pad(n, 4) + pad(i, 5) + pad(s, 5) + pad(s - i, 9) + pad(k, 19) + pad(sp, 7) + pad(t, 14) + pad(o, 16)); } const FAILURE = [["control plane", 1], ["node loss", 1], ["version upgrade", 1], ["admission rule change", 1], ["bad declaration", 0], ["shared data store", 0]]; const isolated = FAILURE.filter((a) => a[1]); console.log(`\nfailure classes ${FAILURE.length}: isolated by splitting ${isolated.length} (affected cells ${CELL}` + ` -> ${CELL / 2}); not isolated ${FAILURE.length - isolated.length} — ` + FAILURE.filter((a) => !a[1]).map((a) => a[0]).join(", ") + ` — ${CELL}/${CELL} in both clusters`); ``` ``` 18 cells = 3 environments x 2 regions x 3 tenants axis clusters cells blast hrs/mo windows/36mo single cluster 1 18 100.0% 106 3 region 2 9 50.0% 173 6 environment 3 6 33.3% 240 9 env x region 6 3 16.7% 441 18 env x region x tenant 18 1 5.6% 1245 54 desired state in a single store; change at round 10, period 15 rounds clus first last window partial rounds drifted conv rounds reads 1 15 15 0 0 0 8 208 2 15 20 5 5 0 16 416 3 15 25 10 10 0 24 624 6 15 25 10 105 1 48 1248 18 12 25 13 108 3 144 3744 failure classes 6: isolated by splitting 4 (affected cells 18 -> 9); not isolated 2 — bad declaration, shared data store — 18/18 in both clusters ``` The first table's two columns move in opposite directions. Blast radius — the share of cells affected by losing one cluster — falls from a hundred percent to 5.6 percent; monthly fixed hours climb from 106 to 1,245, and the upgrade window from three to fifty-four. Splitting by region carries the radius to fifty percent and the load to 173 hours, splitting by environment to 33.3 percent and 240 hours: the only difference is the axis — the region axis buys failure isolation, the environment axis buys change isolation. In the second table, the propagation window is zero for a single cluster, ten rounds for six clusters, and across those ten rounds the set of clusters is partly new, partly old. The real number is in the partial-success column: for six clusters it is not ten but **a hundred five**, because one cluster's admission control rejects the declaration. **A single drifted cluster turns a temporary window into a permanent split.** The convergence workload gets multiplied too: 208 item reads climb to 3,744. The last row gives the place multi-cluster **does not pay off**: four of six failure classes get isolated, two do not. A shared data store, and a bad declaration that leaves the single store and spreads to both clusters at once, keep stopping eighteen of eighteen cells. **Splitting divides the failure unit; it does not divide the data layer.** ## Delegating the Decision to an Upper Layer ```js // upper-layer.mjs — the upper layer deciding which cluster a workload lands on, and what it does not know. const ALL = ["measurement", "field", "reporting"]; const CLUSTER = [ // name, environment, region, capacity, tenants accepted ["dev-r1", "development", "r1", 34, ALL], ["dev-r2", "development", "r2", 34, ALL], ["stg-r1", "staging", "r1", 36, ALL], ["stg-r2", "staging", "r2", 36, ALL], ["prod-r1", "production", "r1", 50, ["measurement", "field"]], ["prod-r2", "production", "r2", 26, ["reporting"]]]; const r = ((t) => () => (t = (t * 48271) % 2147483647) / 2147483647)(20260901); // seed visible const ENV = ["development", "staging", "production"]; const WORKLOAD = Array.from({ length: 24 }, (_, i) => ({ env: ENV[i % 3], region: r() < 0.5 ? "r1" : "r2", tenant: ALL[Math.floor(r() * 3)], units: 4 + Math.floor(r() * 9) })); function run({ age = 0, regionKnown = true, tenantKnown = true }) { const filled = Object.fromEntries(CLUSTER.map(([a]) => [a, 0])); let ledger = { ...filled }; const s = { correct: 0, full: 0, distant: 0, violation: 0, undecided: 0 }; WORKLOAD.forEach((y, i) => { if (i % (age + 1) === 0) ledger = { ...filled }; // the ledger goes stale by `age` steps let candidate = CLUSTER.filter(([, e]) => e === y.env); if (tenantKnown) candidate = candidate.filter(([, , , , k]) => k.includes(y.tenant)); if (regionKnown) candidate = [...candidate].sort((a, b) => (b[2] === y.region) - (a[2] === y.region)); const chosen = candidate.find(([a, , , k]) => ledger[a] + y.units <= k); if (!chosen) { s.undecided++; return; } // no decision could be made const [name, , region, cap, tenants] = chosen; if (filled[name] + y.units > cap) { s.full++; return; } // real capacity was not enough filled[name] += y.units; if (!tenants.includes(y.tenant)) s.violation++; else if (region !== y.region) s.distant++; else s.correct++; }); return s; } const SCENARIO = [["full information", {}], ["capacity stale by 8 placements", { age: 8 }], ["region not reported", { regionKnown: false }], ["tenant constraint not reported", { tenantKnown: false }], ["all three missing", { age: 8, regionKnown: false, tenantKnown: false }]]; const pad = (x, n) => String(x).padStart(n); console.log(`${WORKLOAD.length} workloads, ${CLUSTER.length} clusters (3 env x 2 regions); seed 20260901`); console.log("scenario correct wrong undecided full distant violation"); for (const [name, se] of SCENARIO) { const o = run(se); console.log(name.padEnd(31) + pad(o.correct, 5) + pad(o.full + o.distant + o.violation, 8) + pad(o.undecided, 12) + pad(o.full, 6) + pad(o.distant, 6) + pad(o.violation, 7)); } console.log(`items: ${WORKLOAD.length} workloads x 3 + ${CLUSTER.length} clusters x 2 = ${WORKLOAD.length * 3 + CLUSTER.length * 2}` + `; cluster capacity is the only one that is measured`); ``` ``` 24 workloads, 6 clusters (3 env x 2 regions); seed 20260901 scenario correct wrong undecided full distant violation full information 18 3 3 0 3 0 capacity stale by 8 placements 18 6 0 3 3 0 region not reported 13 10 1 0 10 0 tenant constraint not reported 18 4 2 0 0 4 all three missing 13 11 0 0 8 3 items: 24 workloads x 3 + 6 clusters x 2 = 84; cluster capacity is the only one that is measured ``` The full-information row is not flawless: eighteen of twenty-four workloads place correctly, three land in a distant region, and three go undecided — the first from capacity, the second from the tenant constraint. The capacity ledger going stale produces a familiar shift. Once the ledger is eight placements stale, wrong decisions rise from three to six and **undecided drops from three to zero**: the aging ledger shows the pool as emptier than it is, so the upper layer stops saying "no room" and picks a full cluster instead. Two reported items are measured separately. When region is not reported, correct decisions drop to thirteen and distant placement rises to ten; none get rejected, the workloads run and only exceed the latency limit — **a silent class**. When the tenant constraint is not reported, four workloads go to a cluster that does not accept them, and this wrong decision is noisy, because the cluster's admission control catches it. With all three missing at once, undecided drops to zero: **an upper layer that knows nothing never hesitates.** Only six of the eighty-four items are measured. ## Summary - As 18 cells split from 1 cluster to 18, blast radius falls from 100% to 5.6%; monthly operation climbs from 106 to 1,245 hours, and the upgrade window from 3 to 54. - The propagation window is 0 for a single cluster, 10 rounds for six; one rejecting drifted cluster pushes partial success to 105 rounds. Convergence load multiplies with cluster count: 208 item reads climb to 3,744. - 4 of 6 failure classes get isolated by splitting; a shared data store and a bad declaration spreading from the single store affect 18 of 18 cells in both clusters. - With full information, the upper layer places 18 of 24 workloads correctly; once capacity is stale by 8 placements, wrong decisions go from 3 to 6 and undecided from 3 to 0, and when region goes unreported, 10 workloads silently land in a distant region. ## Course Wrap-Up | Lesson | Delegated decision | Required information and its cost | Source of the wrong decision | |---|---|---|---| | The Need for Orchestration | placement | 6 items, 4 measured | unwritten constraint, 4 wrong spots | | Control Plane Components | request becoming a decision | 50 reads, 7 writes | unrecorded death, 3 presumed | | Node Components | is the node alive | 19 items in 10 seconds | threshold, 26 wrongly dead | | The Reconciliation Loop | closing the gap | delayed report, 0.13 actions/round | unrecorded intent, 32 unnecessary copies | | Local and Managed Clusters | operating the plane | 21 items, 106 hours | lost access, 5 settings missing | | Pod | packaging boundary | 3 items, one guessed | no replica ratio, 4 excess instances | | Replica Sets and Version Objects | replica count, rollout speed | 1 number, 2 settings | closed inside the pod, 6 wasted | | Service Objects | which pod the request goes to | 3 items, 1-round resolution | list age, 216 stale requests | | External Access | the outside request's inside service | 4 declarations, 14–36 surface | uncoverage unwritten, 6 wrong | | Configuration and Secret Objects | the setting reaching the pod | 3 declarations, 3-round propagation | read moment, 36 stale | | Storage and Volumes | matching demand to supply | 76 items, 72 declarations/round | inventory age, 134 wrong | | Stateful Applications | which replica, which data | 3 per replica, 15 items | name not persistent, 7 orphaned volumes | | Jobs and Scheduled Jobs | finished or retry | 4 per night, 120 items | duration unknown, 5 skipped nights | | Namespaces and Quotas | admitting the request | 3 per request, 180 items | usage invisible, 22 rejections | | Resource Requests and Limits | placement | 42 items, 36 declared | no usage, 5,503 conflicts | | Scheduling Constraints | eligible node set | 39 items, no measurement | wrong label, placed 15 → 16 | | Taints and Tolerations | which node admits whom | 27 items, no verifier | excess toleration, 3 foreign pods | | Eviction and Priorities | who leaves under pressure | 17 priorities, criticality 0 | value unmeasured, the bloater survives | | Horizontal Scaling | replica count | 120 metrics, 3 rounds stale | own decision invisible, 510 replica-rounds | | Vertical and Cluster Scaling | size, node count | 60 samples per round | cause absent from the metric, 12 excess events | | Deployment Patterns | advance or stop | 4 items, version label | sparse metric, 851 requests | | Health Checks | restart, open to traffic | 3 probes, 3 thresholds | startup inseparable, 113 kills | | Role-Based Access Control | who can make the request | 43 of 1344 cells | trace ages, 28 escalation paths | | Network Policies | is the connection legitimate | 18 rules or 6 | absent from trace, 119 unnecessary open | | Security Context | should the definition be admitted | one item, the definition itself | looks at declaration, 13 silently exposed | | Packaging and Templating | who writes the object set | 7 values, 3 branches | commonality unwritten, 3/5 diverged | | Custom Resource Definitions | is the declaration valid | 10 fields, 120 fields of burden | schema too narrow, 4 silently wrong | | Operator Pattern | what to do about the event | 5 items, 240 reads | maintenance invisible, 12 restarts | | Cluster Observability | which question gets answered | 10,131 series, 26.1 GB | uncollected detail, 5 of 12 | | Multi-Cluster Management | which cluster the workload runs on | 84 items, 1,245 hours | stale capacity, 6 wrong | Thirty rows are one rule's worked examples: **delegating a decision requires producing the information that decision rests on, continuously and on time; every item the system does not know produces a class of wrong decision.** Four sources repeat in the last column. **Aging information** appears in ten rows: the number at decision time lags behind reality, and as the lag grows, undecided turns into wrong. **A condition left out of scope** appears in eight rows: the system behaves correctly where it looks, and oscillates or drifts silently where it does not. **An unreported item** appears in seven rows, **an unverified declaration** in five — and a wrongly written declaration usually looks like an improvement. The system does not make the wrong decision because it lacks information; it makes it because **there was no item it could have known.** This is the delegation cost that has been this course's unit of measure: the gain shows up in step count, the cost accumulates in information items. One question is left, and **M22/K04, Continuous Integration and Delivery**, takes it up. The cluster applies the desired state: it reads the declaration, closes the gap, runs it through admission control, checks privilege. But **who** changes that desired state, **looking at what**, and **passed through which check**? A human writes every one of the thirty rows' declaration items; for all the cluster's rigor, nowhere does it test the correctness of the declaration it was handed.